Multi-source data automatic collection method based on large language model

Abstract

The application provides a multi-source data automatic acquisition method based on a large language model, which comprises the following steps: analyzing the feature distribution of historical acquisition data, combining the state of the current data source, and dynamically generating an optimal acquisition strategy; scheduling the acquisition task in real time to obtain a dynamically scheduled data stream; performing real-time anomaly detection and adaptive compensation on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream; analyzing the entity association and business logic of the self-fault-tolerant data stream, predicting potential data demand points; automatically expanding the data acquisition range to obtain knowledge-enhanced data assets; identifying the protocol characteristics of the data transmission stream of the data assets, parsing the data based on a protocol characteristic mapping parser to obtain protocol-independent standardized data; performing semantic noise filtering and cross-modal cleaning to obtain semantic pure data. In the application, the defects that the existing multi-source data acquisition technology cannot perform protocol adaptation and adaptive strategy optimization are overcome.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an automated method for acquiring multi-source data based on a large language model. Background Technology

[0002] Against the backdrop of the deep integration of big data and artificial intelligence, multi-source data acquisition has become a crucial step in the digital transformation of various industries. Currently, fields such as the Industrial Internet of Things, smart cities, and financial risk control all face the need for collaborative data acquisition from massive heterogeneous data sources (such as sensors, databases, API interfaces, and web pages). However, existing multi-source data acquisition technologies suffer from the following significant drawbacks:

[0003] First, there are limitations in protocol adaptability. Traditional data acquisition relies on pre-written protocol parsing rules (such as fixed parsing modules for HTTP and Modbus protocols), which are difficult to cope with new IoT protocols (such as LoRaWAN and custom industrial protocols) or scenarios with rapid protocol version iterations. When unknown protocols are encountered, a lot of manpower is required for protocol reverse analysis and parser development, resulting in poor scalability of the acquisition system and an inability to meet dynamically changing business needs.

[0004] Secondly, the data collection strategy is static. Most data collection systems use a fixed collection pattern (such as collecting log data once per hour), and cannot dynamically adjust the strategy according to data value density, data source load status, etc. This leads to the simultaneous occurrence of high-value data being missed and low-value data being redundantly collected, resulting in the dual problems of wasted computing resources and missing key information.

[0005] In summary, existing multi-source data acquisition technologies have significant shortcomings in terms of protocol adaptation and strategy optimization, and cannot meet the data acquisition needs of complex business scenarios. Summary of the Invention

[0006] The main objective of this invention is to provide an automated multi-source data acquisition method based on a large language model, which aims to overcome the shortcomings of existing multi-source data acquisition technologies that cannot perform protocol adaptation and adaptive strategy optimization.

[0007] To achieve the above objectives, this invention provides an automated multi-source data acquisition method based on a large language model, comprising the following steps:

[0008] Analyze the characteristic distribution of historical data and combine it with the current status of the data source to dynamically generate the optimal data collection strategy;

[0009] Based on the optimal acquisition strategy, acquisition tasks are scheduled in real time to obtain a dynamically scheduled data stream; real-time anomaly detection and adaptive compensation are performed on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream.

[0010] A domain knowledge graph is constructed based on a large language model. The entity relationships and business logic of the self-fault-tolerant data stream are analyzed to predict potential data demand points. Based on the predicted potential data demand points, the data collection scope is automatically expanded to obtain knowledge-enhanced data assets.

[0011] The protocol characteristics of the data transmission flow of the data asset are identified, and the data is parsed by a parser based on the protocol feature mapping to obtain protocol-independent standardized data; semantic noise filtering and cross-modal cleaning are performed on the protocol-independent standardized data to obtain semantically clean data.

[0012] Furthermore, the protocol-independent standardized data is data that has eliminated the communication protocol characteristics on which the original data depends and has been uniformly converted into an independent data format.

[0013] Furthermore, the acquisition strategy includes sampling frequency, batch size, and priority sorting.

[0014] Furthermore, by analyzing the characteristic distribution of historically collected data and combining it with the current state of the data source, the optimal collection strategy is dynamically generated, including:

[0015] The historical data is sliced, and the data volume fluctuation characteristics, value density distribution and field integrity characteristics of each slice are extracted to obtain the data feature vector;

[0016] Monitor the connection success rate, response timeliness, and server load pressure of each data source, and conduct a graded evaluation to obtain a data source status matrix;

[0017] A reinforcement learning model is constructed, and the data feature vector and the data source state matrix are subjected to multi-objective reinforcement learning optimization processing to generate the optimal parameter vector. The optimal acquisition strategy is obtained based on the optimal parameter vector.

[0018] Furthermore, based on a large language model, a domain knowledge graph is constructed to analyze the entity relationships and business logic of the self-fault-tolerant data stream, predicting potential data demand points, including:

[0019] Identify entities in the self-fault-tolerant data stream, construct semantic relationships between entities, and generate a set of entity relationship triples;

[0020] A domain knowledge graph is constructed based on a large language model. The set of entity relation triples is mapped to the domain knowledge graph, the similarity between entities is calculated, and the implicit relations are completed to obtain a dynamic knowledge graph.

[0021] Analyze the association paths of entities in the dynamic knowledge graph, extract business rules, and generate a business logic rule base by identifying event sequence patterns through time series analysis.

[0022] By using a Transformer-based demand prediction model, the business logic rule base and dynamic knowledge graph are analyzed to identify data gaps and business relationship blanks, and potential data demand points are output.

[0023] Furthermore, real-time anomaly detection and adaptive compensation are performed on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream, including:

[0024] The dynamically scheduled data stream is sliced, and numerical features, temporal features, and semantic features are extracted to construct a feature vector set;

[0025] Anomaly detection is performed based on the feature vector set to generate anomaly events containing anomaly type and confidence level;

[0026] Abnormal events are mapped to a pre-built fault knowledge graph, and the abnormal propagation path is analyzed through graph neural networks to locate the root cause of the problem and generate a diagnostic report.

[0027] The optimal compensation strategy is automatically matched based on the diagnostic report, and parallel compensation is performed on the dynamically scheduled data stream according to the optimal compensation strategy to obtain a self-fault-tolerant data stream.

[0028] Furthermore, the protocol characteristics of the data transmission stream of the data asset are identified, and a parser based on the protocol feature mapping parses the data to obtain protocol-independent standardized data, including:

[0029] The data transmission stream of the data asset is scanned bit by bit using a finite state automaton, and the protocol type and data structure characteristics are identified according to the preset protocol state transition rules.

[0030] The parser is retrieved from the parser rule base according to the corresponding protocol type and data structure characteristics, and the data is decapsulated according to the protocol and mapped to the fields to obtain protocol-independent standardized data containing data source tags, timestamps and business fields.

[0031] Furthermore, semantic noise filtering and cross-modal cleaning are performed on the protocol-independent standardized data to obtain semantically clean data, including:

[0032] Feature extraction is performed on the protocol-independent standardized data to obtain a neuromorphic feature map, including pulse timing features, synaptic weight features, and population coding features.

[0033] The neural morphological feature map is subjected to noise filtering to obtain an enhanced feature map;

[0034] The enhanced feature map is subjected to entity parsing, event extraction, and causal reasoning to construct a cognitive map;

[0035] The cognitive graph is subjected to cross-modal semantic alignment and meta-learning-driven cleaning to obtain semantically clean data.

[0036] This invention also provides an automated multi-source data acquisition device based on a large language model, comprising:

[0037] The analysis unit is used to analyze the characteristic distribution of historical data and dynamically generate the optimal data collection strategy by combining it with the current status of the data source.

[0038] The compensation unit is used to schedule acquisition tasks in real time based on the optimal acquisition strategy to obtain a dynamically scheduled data stream; and to perform real-time anomaly detection and adaptive compensation on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream.

[0039] The extension unit is used to construct a domain knowledge graph based on a large language model, analyze the entity associations and business logic of the self-fault-tolerant data stream, and predict potential data demand points; based on the predicted potential data demand points, it automatically expands the data collection scope to obtain knowledge-enhanced data assets.

[0040] The cleaning unit is used to identify the protocol characteristics of the data transmission flow of the data asset, and to parse the data based on the protocol feature mapping to obtain protocol-independent standardized data; the protocol-independent standardized data is then subjected to semantic noise filtering and cross-modal cleaning to obtain semantically clean data.

[0041] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0043] This invention provides an automated multi-source data acquisition method based on a large language model, comprising: analyzing the feature distribution of historically acquired data and dynamically generating an optimal acquisition strategy based on the current data source status; scheduling acquisition tasks in real time based on the optimal acquisition strategy to obtain a dynamically scheduled data stream; performing real-time anomaly detection and adaptive compensation on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream; constructing a domain knowledge graph based on a large language model, analyzing the entity associations and business logic of the self-fault-tolerant data stream, and predicting potential data demand points; automatically expanding the data acquisition scope based on the predicted potential data demand points to obtain knowledge-enhanced data assets; identifying the protocol characteristics of the data transmission stream of the data assets, parsing the data based on a protocol feature mapping parser to obtain protocol-independent standardized data; and performing semantic noise filtering and cross-modal cleaning on the protocol-independent standardized data to obtain semantically clean data. In this invention, by analyzing the feature distribution of historically acquired data and combining it with the current data source status, an optimal acquisition strategy is dynamically generated. Simultaneously, the data acquisition scope is automatically expanded based on the predicted potential data demand points. Furthermore, the data is parsed based on a protocol feature mapping parser to obtain protocol-independent standardized data. This overcomes the shortcomings of existing multi-source data acquisition technologies in terms of protocol adaptation and adaptive strategy optimization. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the steps of an automated multi-source data acquisition method based on a large language model in one embodiment of the present invention;

[0045] Figure 2 This is a structural block diagram of a multi-source data automated acquisition device based on a large language model according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0047] The implementation, functional features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] Reference Figure 1 One embodiment of the present invention provides an automated multi-source data acquisition method based on a large language model, comprising the following steps:

[0050] Step S1: Analyze the characteristic distribution of historical data collection and dynamically generate the optimal collection strategy based on the current data source status.

[0051] Step S2: Based on the optimal acquisition strategy, the acquisition tasks are scheduled in real time to obtain a dynamically scheduled data stream; real-time anomaly detection and adaptive compensation are performed on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream.

[0052] Step S3: Construct a domain knowledge graph based on a large language model, analyze the entity relationships and business logic of the self-fault-tolerant data stream, and predict potential data demand points; based on the predicted potential data demand points, automatically expand the data collection scope to obtain knowledge-enhanced data assets.

[0053] Step S4: Identify the protocol characteristics of the data transmission flow of the data asset, parse the data based on the protocol feature mapping parser to obtain protocol-independent standardized data; perform semantic noise filtering and cross-modal cleaning on the protocol-independent standardized data to obtain semantically clean data.

[0054] In this embodiment, as described in step S1 above, during the multi-source data acquisition process, historical data contains key information such as the data volume variation pattern and data value distribution, while the current data source status (such as server load and network latency) reflects the real-time acquisition environment. This step first performs in-depth analysis of the historical data, using time series analysis to extract features such as the fluctuation characteristics of data volume in different time periods (such as the difference between weekdays and holidays) and the frequency distribution of high-value data. At the same time, the real-time monitoring module obtains status parameters such as the response time, connection success rate, and server resource utilization rate of each data source. Then, using the historical data features and the current data source status information as input, and leveraging the data analysis and decision-making capabilities of the large language model, combined with reinforcement learning algorithms, the optimal acquisition strategy for different data sources is dynamically generated with the optimization goals of maximizing acquisition efficiency, minimizing resource consumption, and ensuring data integrity. This includes parameter configurations such as sampling frequency (e.g., high-frequency acquisition for real-time data streams and low-frequency batch acquisition for historical logs) and acquisition priority (e.g., priority acquisition of alarm data).

[0055] As described in step S2 above, based on the optimal acquisition strategy generated in step S1, the real-time scheduling module precisely allocates acquisition tasks to corresponding computing resource nodes according to priority and time schedule (e.g., in-memory computing nodes handle data with high real-time requirements, and distributed storage clusters handle batch data), achieving efficient execution of acquisition tasks and thus forming a dynamically scheduled data stream. During this process, to ensure the stability and reliability of the data stream, a real-time anomaly detection mechanism is deployed. This involves constructing a Transformer-based time-series analysis model and a semantic recognition model to monitor the dynamically scheduled data stream in real time. On one hand, for numerical data, the mechanism analyzes its changing trends and fluctuation range to detect abnormal jumps, missing data, and other issues. On the other hand, for textual data, it identifies semantic contradictions, erroneous expressions, and other noise. Once an anomaly is detected, an adaptive compensation mechanism is immediately activated. Based on the anomaly type (e.g., data source failure, network interruption, data format error), the corresponding compensation strategy is automatically executed, such as switching to a backup data source, adjusting data request parameters, and generating data repair rules using a large language model. This eliminates the impact of the anomaly, resulting in a self-fault-tolerant data stream that ensures the continuity and accuracy of data acquisition.

[0056] As described in step S3 above, the powerful knowledge understanding and graph construction capabilities of the large language model are leveraged to conduct in-depth analysis of the self-fault-tolerant data stream obtained in step S2. First, various entities (such as users, devices, and orders) are automatically extracted from the data stream, and the relationships between entities are determined through semantic association analysis (e.g., "user-owner-device" and "order-related-product"), thereby constructing a domain knowledge graph. Next, graph analysis algorithms are used to mine the knowledge graph, identifying potential patterns and business logic rules of entity associations in the data stream (e.g., the relationship between order volume of a certain type of product and user location within a specific time period). Then, based on these discovered business logics and historical data patterns, combined with the predictive capabilities of the large language model, forward-looking predictions are made of potential future data demand points (e.g., predicting new user attribute collection needs based on user growth trends). Finally, based on the prediction results, the data collection scope expansion process is automatically triggered, including adding new data source access configurations, adjusting web crawler crawling rules, and modifying API call parameters, incorporating new data into the collection scope to form knowledge-enhanced data assets, providing richer and more valuable data support for subsequent data analysis and decision-making.

[0057] As described in step S4 above, since data assets originate from various data sources, the protocols (such as HTTP, MQTT, Modbus) and data formats (JSON, XML, binary) used for data transmission differ, posing challenges to unified data processing and analysis. This step first identifies the protocol features of the data transmission streams in the data assets. By deploying an adaptive protocol parsing engine and utilizing deep packet inspection technology and machine learning classification algorithms, features such as protocol identifiers, message structures, and interaction sequences in the data transmission streams are extracted to accurately determine the protocol type used by the data. Then, based on the pre-established protocol-parser mapping relationship, the corresponding parser is automatically invoked to decapsulate and convert the data format, removing protocol-specific information and uniformly converting data from different protocols into a standardized format containing data source tags, timestamps, and business fields, achieving protocol independence. On this basis, semantic noise filtering and cross-modal cleaning are performed on the protocol-independent standardized data. For numerical data, statistical methods and anomaly detection algorithms are used to remove outliers and invalid data. For textual data, the semantic understanding capabilities of large language models are utilized to identify and filter duplicate content, erroneous expressions, and semantically ambiguous information. In cross-modal data processing, a unified semantic space is established to semantically align and fused unstructured data such as images and audio with structured data, ultimately obtaining semantically clean and formatted data, laying a solid foundation for subsequent data mining, analysis, and applications.

[0058] In this embodiment, by analyzing the characteristic distribution of historically collected data and combining it with the current state of the data source, the optimal collection strategy is dynamically generated. At the same time, the data collection range is automatically expanded by predicting potential data demand points. Then, the data is parsed based on the protocol feature mapping parser to obtain protocol-independent standardized data. This overcomes the shortcomings of existing multi-source data collection technologies that cannot perform protocol adaptation and adaptive strategy optimization.

[0059] In one embodiment, the protocol-independent standardized data is data that has eliminated the communication protocol characteristics on which the original data depends and has been uniformly converted into an independent data format.

[0060] In one embodiment, the acquisition strategy includes sampling frequency, batch size, and priority sorting.

[0061] In one embodiment, the characteristic distribution of historically collected data is analyzed, and combined with the current state of the data source, an optimal collection strategy is dynamically generated, including:

[0062] The historical data is sliced, and the data volume fluctuation characteristics, value density distribution and field integrity characteristics of each slice are extracted to obtain the data feature vector;

[0063] Monitor the connection success rate, response timeliness, and server load pressure of each data source, and conduct a graded evaluation to obtain a data source status matrix;

[0064] A reinforcement learning model is constructed, and the data feature vector and the data source state matrix are subjected to multi-objective reinforcement learning optimization processing to generate the optimal parameter vector. The optimal acquisition strategy is obtained based on the optimal parameter vector.

[0065] In this embodiment, to comprehensively analyze the potential patterns contained in historically collected data, the sliding window technique in time series analysis is first used to slice the historically collected data according to fixed time intervals (such as hours or days), dividing the continuous data stream into multiple independent time segments. For each slice of data, deep feature extraction is performed from three dimensions: First, the data volume fluctuation characteristics are calculated. By counting the number of data records within the slice and combining statistical indicators such as mean, standard deviation, and coefficient of variation, the fluctuation range and stability of the data volume in different time periods are quantified, and abnormal intervals of sudden increases or decreases in data volume are identified. Second, the value density distribution is evaluated. A large language model is used to perform semantic analysis on the data content within the slice. Through keyword extraction, sentiment judgment, and importance scoring, the proportion of high-value information (such as records containing key business indicators or abnormal alarms) in the data is marked, and a curve of value density changing over time is constructed. Third, the field integrity characteristics are detected. Each data record within the slice is traversed, and the number of missing business fields (such as "user ID" and "transaction amount") is counted. The field missing rate is calculated, and unstable data areas with long-term field missing problems are identified. Finally, the three types of features are normalized and integrated to form a multidimensional data feature vector, providing a quantitative data foundation for subsequent strategy generation.

[0066] To monitor the real-time operational status of the current data sources, a multi-dimensional data source monitoring module was deployed. First, the connection success rate of each data source is continuously monitored by periodically sending connection requests and calculating the ratio of successful connections to total requests to assess the network connectivity stability of the data source. Second, the response timeliness of the data source is measured by recording the time interval between sending and receiving each data request, distinguishing between different response levels such as real-time (response time less than 100ms), near real-time (100-500ms), and offline (more than 500ms). Simultaneously, load stress indicators of the servers hosting the data sources are collected, including CPU utilization, memory usage, and disk I / O throughput, to determine if server resources are overloaded. Based on the above monitoring data, according to preset threshold standards (e.g., connection success rate below 90% indicates instability, CPU utilization above 80% indicates high load), the three indicators of each data source are evaluated in three levels (high, medium, and low). Finally, the connection success rate, response timeliness, and server load pressure of each data source are stored in a structured matrix to construct a data source status matrix, which intuitively reflects the real-time health status and service capabilities of all data sources.

[0067] To achieve dynamic optimization of the data acquisition strategy, a decision model based on deep reinforcement learning was constructed. This model uses data feature vectors and the data source state matrix as inputs, with the optimization objectives of maximizing acquisition efficiency, minimizing resource consumption, and optimizing data value. During model training, the data feature vectors and the data source state matrix are mapped to a "state" space in reinforcement learning. Different acquisition strategy parameters (such as sampling frequency, batch size, and acquisition priority) are defined as "action" spaces. By designing a reasonable reward function (e.g., providing positive rewards for complete acquisition of high-value data and negative penalties for excessive resource consumption), the agent is guided to explore different action combinations in the state space. Classic reinforcement learning algorithms such as Actor-Critic are employed. Through continuous interaction with the environment, the agent learns the optimal decision strategy from historical data, gradually adjusting the strategy parameters to maximize long-term cumulative rewards. After multiple rounds of iterative training, when the model converges, it outputs an optimal parameter vector containing parameters such as the optimal sampling frequency, optimal batch size, and optimal data source priority. Finally, the optimal parameter vector is transformed into executable acquisition strategy instructions, which dynamically allocate acquisition resources for different data sources. This ensures the integrity and value of the data while achieving efficient utilization of computing resources, generating an optimal acquisition strategy that adapts to data changes and data source states.

[0068] In one embodiment, a domain knowledge graph is constructed based on a large language model to analyze the entity relationships and business logic of the self-fault-tolerant data stream and predict potential data demand points, including:

[0069] Identify entities in the self-fault-tolerant data stream, construct semantic relationships between entities, and generate a set of entity relationship triples;

[0070] A domain knowledge graph is constructed based on a large language model. The set of entity relation triples is mapped to the domain knowledge graph, the similarity between entities is calculated, and the implicit relations are completed to obtain a dynamic knowledge graph.

[0071] Analyze the association paths of entities in the dynamic knowledge graph, extract business rules, and generate a business logic rule base by identifying event sequence patterns through time series analysis.

[0072] By using a Transformer-based demand prediction model, the business logic rule base and dynamic knowledge graph are analyzed to identify data gaps and business relationship blanks, and potential data demand points are output.

[0073] In this embodiment, after acquiring the fault-tolerant data stream, the powerful natural language processing capabilities of the large language model are first utilized to scan various types of data, such as text and numerical values, in the data stream using Named Entity Recognition (NER) technology. This identifies entities with practical meaning, such as "User ID," "Product Model," and "Order Number." For numerical data, predefined mapping rules or contextual semantics are used to convert it into corresponding entity concepts (e.g., mapping "1001" to "Specific User ID"). After identifying entities, dependency parsing and semantic role labeling techniques are further employed to analyze the semantic relationships between entities, clarifying the role relationships of each entity in the business scenario, such as "User-Purchase-Product" and "Order-Include-Goods." Each group of entities and their relationships is structured and expressed as a triple (subject, verb, object), ultimately forming a set of triples containing a large number of entity relationships. This set clearly presents the direct association information between entities in the data stream, providing foundational data for subsequent knowledge graph construction.

[0074] Using an industry-specific business domain as a context, and leveraging the knowledge system pre-trained by a large language model, an initial domain knowledge graph framework is constructed. This framework covers common entity types, relationship categories, and basic business rules within the domain. Subsequently, the entity-relationship triplet set generated in the previous step is precisely mapped to corresponding nodes and connections in the domain knowledge graph according to entity type and relationship category, achieving the transformation from scattered triplet data to a structured knowledge network. During the mapping process, knowledge embedding algorithms (such as TransE and ComplEx) are used to map entities and relationships to a low-dimensional vector space. By calculating the cosine similarity or Euclidean distance between vectors, the semantic similarity between entities is quantified. Based on the entity similarity calculation results, combined with the knowledge reasoning capabilities of the large language model, implicit relationships are mined and completed for entities in the knowledge graph that are not yet clearly defined but have potential connections. For example, by inferring that "User A purchases product X" and "Product X belongs to brand Y", we can deduce that "User A may be interested in other products of brand Y". This implicit relationship is then added to the knowledge graph, thereby continuously enriching and improving the content of the knowledge graph and forming a dynamic knowledge graph that can be dynamically updated and reflects the evolution of knowledge in the business domain.

[0075] Deep graph analysis is performed on dynamic knowledge graphs, utilizing techniques such as graph neural networks (GNNs) to explore multi-hop association paths between entities and uncover complex indirect relationships and potential impacts. For example, by analyzing association paths such as "user-purchase-product," "product-category-product type," and "category-promotional activity-discount strategy," business rules such as "when a product category has a promotional activity, the number of users purchasing products in that category may increase" are extracted. Simultaneously, for time-series data in the data stream (such as order creation time and logistics status change time), time-series analysis algorithms (such as sliding window and Fourier transform) are used to identify regular event sequence patterns, such as the typical shopping process of "user browsing products → adding to cart → placing an order." The business rules extracted from the knowledge graph are integrated, verified, and supplemented with the event sequence patterns obtained through time-series analysis, removing duplicate or contradictory rules, ultimately forming a complete, accurate, and business-guiding business logic rule base. This rule base comprehensively reflects the core knowledge of entity relationships, business processes, and logical constraints in the business domain.

[0076] Finally, a demand prediction model based on the Transformer architecture is constructed. This model encodes the structured rules in the business logic rule base and the semantic information in the dynamic knowledge graph, transforming them into vector representations that the model can process. Leveraging the powerful attention mechanism of the Transformer model, global feature extraction and deep semantic understanding are performed on the input data. The matching between current data assets and business needs is analyzed from multiple dimensions, including the execution conditions of business logic rules, their scope of influence, and the tightness of relationships between entities in the knowledge graph. By comparing the knowledge structure under ideal business scenarios with the knowledge system constructed from existing data, missing fields (e.g., the lack of a "consumption habits" field in user profiles), uncovered entity types (e.g., the "virtual goods" entity appearing in emerging business scenarios), and unestablished business relationships (e.g., the potential connection between "user health data" and "product recommendation strategy") are identified. These identified data gaps and business relationship gaps are summarized, filtered, and prioritized, ultimately outputting a list of potential data requirements. This list clearly indicates the direction and content of data that needs to be further collected or supplemented to meet business development and data analysis needs, providing precise guidance for expanding the scope of data collection.

[0077] In one embodiment, real-time anomaly detection and adaptive compensation are performed on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream, including:

[0078] The dynamically scheduled data stream is sliced, and numerical features, temporal features, and semantic features are extracted to construct a feature vector set;

[0079] Anomaly detection is performed based on the feature vector set to generate anomaly events containing anomaly type and confidence level;

[0080] Abnormal events are mapped to a pre-built fault knowledge graph, and the abnormal propagation path is analyzed through graph neural networks to locate the root cause of the problem and generate a diagnostic report.

[0081] The optimal compensation strategy is automatically matched based on the diagnostic report, and parallel compensation is performed on the dynamically scheduled data stream according to the optimal compensation strategy to obtain a self-fault-tolerant data stream.

[0082] In this embodiment, to achieve comprehensive monitoring and analysis of dynamically scheduled data streams, a sliding time window technique is employed to slice the data stream according to preset time intervals (e.g., 1 minute, 5 minutes), dividing the continuous data stream into multiple discrete data segments. For each slice, deep feature extraction is performed from three dimensions: In terms of numerical feature extraction, statistical indicators such as the mean, standard deviation, maximum, and minimum values ​​of numerical data within the data segment are calculated to quantify the central tendency and dispersion of the data; for temporal features, algorithms such as Fourier transform and wavelet analysis are used to mine the periodicity and trend of data changes over time, identifying the frequency and amplitude characteristics of data fluctuations; in the semantic feature extraction stage, the natural language processing capabilities of large language models are utilized to perform word segmentation, part-of-speech tagging, and named entity recognition on text data, and the text is converted into high-dimensional semantic vectors through word vector models (such as BERT and Word2Vec) to capture the semantic information of the text content. After normalizing the extracted numerical features, temporal features, and semantic features, they are integrated into multidimensional feature vectors. Each feature vector corresponds to a data slice, and finally a complete set of feature vectors is constructed, providing rich analytical data for subsequent anomaly detection.

[0083] Using a constructed set of feature vectors as input, a deep learning-based anomaly detection model, such as an autoencoder or Transformer model, is deployed. This model first learns the normal pattern representation of the data by studying the feature vectors of normal data, learning the distribution patterns and inherent structure of the data in the feature space. In the real-time detection phase, new feature vectors are input into the model, and the reconstruction error or difference between them and the normal pattern is calculated. When this error exceeds a preset threshold (e.g., the mean plus three standard deviations), an anomaly is identified. To accurately identify the anomaly type, the model combines preset anomaly type classification rules (e.g., numerical anomalies, temporal anomalies, semantic anomalies) to classify and analyze the anomalous feature vectors. For example, if the mean in the numerical features deviates significantly from the historical mean, it is classified as a numerical jump anomaly; if the temporal features show periodic data disappearances, it is classified as a temporal interruption anomaly; if contradictory expressions or illegal words appear in the semantic features, it is classified as a semantic error anomaly. Simultaneously, the model calculates the similarity between the anomalous features and templates for each anomaly type, generating a confidence score for each detected anomalous event to quantify the reliability of the anomaly judgment. Finally, the detected anomaly types are integrated with their corresponding confidence levels to generate anomaly event records that include both the anomaly type and the confidence level, providing clear clues for subsequent problem diagnosis.

[0084] A fault knowledge graph is pre-constructed, covering various data sources, data processing stages, and common fault types. This graph uses entities (such as data source servers, data transmission protocols, and data processing modules) as nodes and relationships between entities (such as "server hosts data source" and "protocol affects data transmission") as edges, building a knowledge network for the fault domain. When an abnormal event is detected, key information from the abnormal event (such as the data source involved and the type of abnormality) is mapped to the corresponding node in the fault knowledge graph. Then, the powerful graph structure data processing capabilities of Graph Neural Networks (GNNs) are utilized to traverse and analyze the knowledge graph, exploring possible propagation paths of the abnormality along the relationship edges between nodes. For example, if a data transmission abnormality is detected, the GNN will analyze the state information of each node along the relationship path of "data transmission protocol → network device → data source server" to assess the impact of each node on the abnormality. By calculating the importance score of the nodes (such as using the PageRank algorithm), the nodes that play a key role in the occurrence of the abnormality are located, i.e., the root cause of the problem (such as server CPU overload causing data processing delay). Finally, the basic information of the abnormal event, the analysis process, and the root causes of the problem are summarized to generate a detailed diagnostic report, which clearly presents the cause of the abnormality, the scope of its impact, and relevant evidence, providing an accurate basis for the formulation of subsequent compensation strategies.

[0085] In this embodiment, a built-in compensation strategy library is implemented. This library pre-stores various compensation strategies for different anomaly types and root causes, each strategy including specific execution steps, applicable scenarios, and expected performance evaluation metrics. Upon receiving an anomaly diagnosis report, key information (such as anomaly type and root cause) is extracted from the report using natural language processing technology and matched against strategies in the compensation strategy library. A rule-based matching algorithm and similarity calculation method are used to select the compensation strategy that best matches the current anomaly. If multiple candidate strategies exist, multi-criteria decision analysis (such as the analytic hierarchy process) is used to comprehensively consider factors such as strategy execution cost, recovery efficiency, and impact on business operations to determine the optimal compensation strategy. For example, if the diagnosis report indicates that the anomaly is due to a network interruption of the data source server, the optimal compensation strategy might be to switch to a backup data source server and adjust the data request parameters. After determining the optimal compensation strategy, a parallel compensation mechanism is initiated, simultaneously implementing compensation operations at multiple dimensions, including the network layer, protocol layer, and data layer. At the network layer, communication links are automatically switched or network bandwidth allocation is adjusted. At the protocol layer, data transmission protocol parameters are modified or protocol types are switched. At the data layer, a large language model is used to generate data repair rules to complete or correct erroneous or missing data. By executing the compensation strategy in parallel, the impact of anomalies on the data stream is quickly eliminated, restoring the data stream to normal transmission and processing. Ultimately, a self-fault-tolerant data stream with self-fault tolerance capabilities is obtained, ensuring the continuity of the data acquisition process and the reliability of data quality.

[0086] In one embodiment, the protocol characteristics of the data transmission flow of the data asset are identified, and a parser based on the protocol characteristic mapping parses the data to obtain protocol-independent standardized data, including:

[0087] The data transmission stream of the data asset is scanned bit by bit using a finite state automaton, and the protocol type and data structure characteristics are identified according to the preset protocol state transition rules.

[0088] The parser is retrieved from the parser rule base according to the corresponding protocol type and data structure characteristics, and the data is decapsulated according to the protocol and mapped to the fields to obtain protocol-independent standardized data containing data source tags, timestamps and business fields.

[0089] In this embodiment, a Finite State Automaton (FSA) is deployed as the core identification tool when processing data transmission streams in data assets. Based on a pre-defined set of states and state transition rules, the FSA performs a byte-by-byte, bit-by-bit sequential scan of the data transmission stream. During the scan, the automaton starts from an initial state and matches the received data bytes with the preset rules, triggering corresponding state transitions. For example, when identifying the HTTP protocol, if the automaton encounters specific strings such as "GET" or "POST", it transitions from the "initial state" to the "request line parsing state"; if it subsequently receives fields conforming to the HTTP header format (such as "Host:" or "Content-Length:"), it continues to transition to the "header parsing state"; when a blank line separator is detected, it enters the "body parsing state". Through this state transition mechanism, the FSA can accurately determine the protocol type used by the data transmission stream (such as HTTP, MQTT, Modbus, etc.) and simultaneously identify data structure characteristics, including data frame format, field order, and separator type. Furthermore, for custom or unknown protocols, the system expands the state transition rule base and combines heuristic algorithms for pattern matching to identify complex protocol types and structures. This ultimately generates identification results containing protocol type identifiers and data structure descriptions, providing a basis for subsequent data parsing.

[0090] After obtaining the protocol type and data structure characteristics of the data transmission stream, a pre-built parser rule base is accessed. This rule base stores dedicated parsers for various standard protocols (such as TCP / IP and CoAP) and custom protocols. Each parser contains decapsulation logic and field mapping rules that match the protocol type and data structure. Based on the identification results from the first step, the corresponding parser is accurately retrieved from the rule base. For example, for a data stream identified as an MQTT protocol, the MQTT parser is invoked. The parser first performs protocol decapsulation on the data, stripping the transport layer (such as TCP / UDP) and application layer protocol header information (such as the fixed and variable headers of MQTT), retaining only the data payload. Next, according to predefined field mapping rules, protocol-specific fields (such as register addresses in the Modbus protocol and status codes in the HTTP protocol) are converted into general business fields (such as "device parameter number" and "request response status"). During this process, metadata information is automatically added to the parsed data, including data source tags (used to identify the data source device or system), timestamps (recording the accurate time of data collection or generation), and standardized business fields (such as "temperature" and "user ID"). Through the above-mentioned protocol decapsulation, field mapping, and metadata injection operations, the original data with specific protocol characteristics is transformed into standardized data in a unified format that does not depend on specific transmission protocols. This achieves the normalization of data from different protocols and provides standardized basic data for subsequent data cleaning, analysis, and fusion.

[0091] In one embodiment, semantic noise filtering and cross-modal cleaning are performed on the protocol-independent standardized data to obtain semantically clean data, including:

[0092] Feature extraction is performed on the protocol-independent standardized data to obtain a neuromorphic feature map, including pulse timing features, synaptic weight features, and population coding features.

[0093] The neural morphological feature map is subjected to noise filtering to obtain an enhanced feature map;

[0094] The enhanced feature map is subjected to entity parsing, event extraction, and causal reasoning to construct a cognitive map;

[0095] The cognitive graph is subjected to cross-modal semantic alignment and meta-learning-driven cleaning to obtain semantically clean data.

[0096] In this embodiment, a neuromorphic computing architecture is used for feature extraction when processing protocol-independent standardized data. First, a biological neuron model is constructed using a memristor array to convert numerical data into neuronal firing frequencies, forming pulse-time sequence features. This conversion simulates the information encoding method of biological nervous systems, transforming continuous numerical changes into discrete pulse sequences, making the data features more consistent with the characteristics of neural processing. Next, the neural connection strength between different fields is calculated to form synaptic weight features. The correlation between each data field in the business logic is analyzed; for example, the strong correlation between "order amount" and "product quantity." By quantifying this correlation strength, a connection weight network similar to biological synapses is constructed. Finally, semantic information is represented using neuronal cluster activity patterns, forming population encoding features. For textual data, its semantic information is encoded through the cooperative firing patterns of neuronal populations, with different semantic content corresponding to different neuronal firing combinations. The extracted pulse timing features, synaptic weight features, and population coding features are integrated to construct a neuromorphic feature map. This map represents data features in the manner of biological neural networks, preserving the temporal characteristics, semantic associations, and collaborative relationships of the data, and providing richer feature representations that are more in line with cognitive laws for subsequent noise filtering and semantic analysis.

[0097] A quantum-enhanced noise filtering mechanism is deployed for neuromorphic feature maps. First, data noise is mapped to the superposition state of qubits, utilizing the uncertainty principle in quantum mechanics to more accurately represent the probability distribution of noise. Quantum entanglement measurement technology is used to identify incoherent noise in the data features, i.e., random interference unrelated to the data's semantics. Quantum entanglement can capture subtle correlations between features, thus distinguishing between true signals and noise. Then, the noise threshold is adaptively adjusted using quantum phase transition properties. When a complex data environment and high noise levels are detected, the noise filtering threshold is automatically increased; conversely, the threshold is decreased to retain more useful information. During the filtering process, a quantum annealing algorithm can be used to find the optimal noise filtering scheme, determining which features are noise and which are valid signals by minimizing the energy function. After quantum-enhanced noise filtering, noise in the neuromorphic feature map is effectively suppressed, and useful signals are enhanced, forming an enhanced feature map, providing cleaner and more reliable feature data for subsequent cognitive analysis.

[0098] Based on enhanced feature maps, knowledge construction leverages the powerful semantic understanding capabilities of large language models. First, entity recognition in the data is achieved through referential resolution techniques, addressing the issue of different representations of the same entity in different contexts (e.g., "apple" can refer to both fruit and a technology company). Combined with named entity recognition and semantic role labeling, various entities (such as users, products, and orders) and their attributes are accurately extracted from the data. Next, business events are identified from time-series data; for example, by analyzing the sequence of order status changes ("creation → payment → shipment → receipt"), complete order processing events are extracted. Then, a causal relationship network between events is constructed, using graph neural networks (GNNs) to calculate the causal probabilities between events. For example, the strength of the causal relationship between "promotional activities" and "sales growth" is analyzed, and the complex interactions between events are represented through a graph structure. Entities, events, and their causal relationships are integrated to construct a cognitive map. This map not only contains entity information and event relationships from the data but also incorporates causal reasoning knowledge, enabling a deeper reflection of the inherent logic and patterns of the business domain, providing a rich knowledge foundation for cross-modal semantic alignment and data cleaning.

[0099] After obtaining the cognitive graph, cross-modal semantic alignment is performed. First, unstructured data such as images and audio are converted into vector representations using a feature extraction model, and then semantically matched with the structured data in the cognitive graph. For example, the visual features of product images are aligned with the semantic features of product description text to establish an "image-text" semantic association. For temporal and spatial data, a spatiotemporal alignment algorithm is used to associate events in the time series with entities in three-dimensional space, such as mapping time series data collected by sensors to corresponding geographical locations. During cross-modal alignment, an attention mechanism is used to focus on key semantic information in the data to improve alignment accuracy. After completing cross-modal semantic alignment, a meta-learning system is built for adaptive cleaning. The Model-Independent Meta-Learning (MAML) algorithm is used to extract common features from different data sources and different types of data, enabling the cleaning model to quickly adapt to new data distributions. Based on historical cleaning results, cleaning parameters (such as missing value imputation methods and outlier handling strategies) are dynamically adjusted, and the semantic similarity of data before and after cleaning is calculated through comparative learning to evaluate the cleaning effect. The data was also integrated with a brain-computer interface system for final verification, converting the cleaning results into neural stimulation signals. Subconscious reactions from domain experts were collected via EEG, and the neural feedback from multiple experts was integrated to form the final cleaning decision. After cross-modal semantic alignment and meta-learning-driven cleaning, semantic noise in the data was completely eliminated, and semantic unity was achieved for data from different modalities, ultimately resulting in semantically clean data. This provides a high-quality, unambiguous data foundation for subsequent data analysis and intelligent decision-making.

[0100] In one embodiment, after obtaining semantically clean data, the process includes:

[0101] Obtain attribute information from each data source, generate a character matrix based on the attribute information, and draw a polyline in a preset coordinate system based on the attribute information.

[0102] Extract the attribute information of the character matrix, and determine a target position in a preset Base64 encoding table based on the attribute information;

[0103] The character matrix is ​​superimposed on a preset Base64 encoding table; wherein the center of the character matrix overlaps with the target position, and each matrix element overlaps with a cell in the preset Base64 encoding table;

[0104] The XOR operation is performed on each element of the character matrix and the characters in the overlapping cells. The results are then combined sequentially to obtain the first character combination string. During the XOR operation, if the characters are of the same type, the result is 1; if the characters are of different types, the result is 0.

[0105] The polyline is superimposed onto a preset Base32 encoding table according to the rules. All characters in the cells crossed by the polyline are selected in the preset Base32 encoding table and combined to obtain a second character combination string.

[0106] The first character string and the second character string are interleaved and combined to obtain an encryption key, which is then used to encrypt and store the semantically clean data.

[0107] In this embodiment, firstly, the attribute information of each data source is acquired. This attribute information covers key characteristics such as the type of data source, update frequency, and data volume. Based on this attribute information, the system quantizes and encodes it, converting each attribute value into a corresponding character according to a specific mapping rule, and arranging these characters according to a preset row and column rule to generate a character matrix. Simultaneously, the numerical data in the data source attribute information is mapped onto the coordinate axes of a preset coordinate system, and the corresponding coordinate points are connected sequentially according to the association order of the attribute information to form a polyline.

[0108] Next, the generated character matrix is ​​further analyzed to extract its attribute information, including the number of rows and columns, element distribution patterns, and character frequency. Based on this matrix attribute information, a target position is determined in a pre-defined Base64 encoding table using specific calculation rules. This calculation rule comprehensively considers various matrix attributes, such as using the number of rows and columns as offsets to perform positioning calculations in the Base64 encoding table, thereby accurately locating the target position.

[0109] The generated character matrix is ​​then overlaid onto a pre-defined Base64 encoding table. During the overlay process, the center of the character matrix is ​​strictly controlled to completely overlap with the previously determined target position, ensuring that the matrix elements correspond one-to-one with the cells in the Base64 encoding table.

[0110] Next, an XOR operation is performed on each element in the character matrix and the character in its corresponding Base64 encoded cell. The specific calculation rule is as follows: if the character matrix element and the corresponding cell character have the same type (e.g., both are numbers or both are letters), it is counted as 1; if the character types are different, it is counted as 0. The XOR operation is performed sequentially according to the order of the matrix elements, and the results are combined in order to obtain the first character string. This string contains the characteristic difference information between the character matrix and the Base64 encoded table.

[0111] Simultaneously, the previously generated polylines are superimposed onto a pre-defined Base32 encoding table according to predetermined rules. These rules include the starting position and direction of the polylines within the encoding table. During the superposition process, each cell in the Base32 encoding table crossed by the polyline is examined, and all characters in these cells are selected and combined according to the order in which the polylines cross them, thus obtaining a second character string. This string records the characteristic information of the interaction between the polylines and the Base32 encoding table.

[0112] Finally, the first character string and the second character string are interleaved and combined. Specifically, this interleaving can be done by alternately selecting characters from the two strings; for example, taking the first character of the first string, then the first character of the second string, and so on. Through this interleaving and combining operation, a complete encryption key is obtained. This encryption key integrates information from multiple aspects, including data source attributes, character matrix characteristics, and Base64 and Base32 encoding table features, and is used to encrypt and store semantically clean data, ensuring the security and confidentiality of the data during storage.

[0113] In one embodiment, after obtaining semantically clean data, the process includes:

[0114] Obtain attribute information from various data sources, convert the attribute information into transfinite ordinal representations, and construct transfinite ordinal chains using Cantor's transfinite ordinal theory;

[0115] The data structure of semantically pure data is abstracted into objects in a category, and the data relations are abstracted into morphisms; dual categories are generated through the duality principle of category theory, and the semantically pure data is transformed into a dual representation to obtain a dual data stream.

[0116] The superfindable ordinal chain and the dual data stream are subjected to stacked tensor product processing to generate a stacked tensor product matrix.

[0117] A logical paradox structure is embedded in a stacked tensor product matrix, and contradiction points are generated through self-reference operations. The position coordinates of the contradiction points in the matrix are extracted, and the coordinate sequence is used to generate an encryption key for encrypting semantically clean data.

[0118] In this embodiment, firstly, attribute information from each data source is collected, including key parameters such as data type, update frequency, and security level. Then, this attribute information is converted into a transboundary ordinal representation. Transboundary ordinals are a mathematical concept in Cantor's set theory used to describe infinite ordinals. The system maps attributes to transboundary ordinal expressions based on their numerical value, importance, and other characteristics. For example, frequently updated data source attributes are mapped to higher-order transboundary ordinals, and infrequently updated attributes to lower-order ones. Based on this, utilizing the operational rules and order relationships of transboundary ordinals, the transboundary ordinals corresponding to all attributes are arranged sequentially according to the association logic of the data sources, constructing a transboundary ordinal chain reflecting attribute hierarchy and dependencies. This ordinal chain not only preserves the original information of the data source attributes but also forms a data encoding with complex hierarchical relationships through the unique mathematical structure of transboundary ordinals.

[0119] Next, the semantically pure data undergoes category-theoretic abstraction. Basic units in the data, such as records and fields, are abstracted as "objects" within the category; relationships between data units, such as parent-child relationships and reference relationships, are abstracted as "morphisms" within the category. In this way, the originally structured data is transformed into an abstract mathematical model within the category-theoretic framework. Then, using the duality principle of category theory, the constructed categories are reversed. Specifically, the direction of all morphisms is reversed, and the logical relationships between objects are adjusted to generate a new category dual to the original data structure. In this process, the logical relationships of the data are reorganized, forming a semantically equivalent but structurally completely different dual representation—the dual data flow. This dual transformation breaks the conventional structure of the original data, increasing its complexity and confidentiality.

[0120] Subsequently, the transfinite ordinal chain and the dual data stream are processed using stacked tensor products. Tensor product is an operation that combines different mathematical objects. The system treats the transfinite ordinal chain and the dual data stream as elements in a tensor space, and deeply integrates their structure and information through multi-level tensor product operations. In each level of tensor product operation, the tensor basis is dynamically adjusted according to the characteristics of the data, allowing different levels of operation to capture different dimensions of data information. For example, the basic information of the data source attributes is integrated in the first level of tensor product, and the complex relationships of the data structure are gradually incorporated in subsequent levels. After multiple levels of tensor product operations, a stacked tensor product matrix containing rich semantic and structural information is generated. This matrix highly integrates the data source attributes and data structure at the mathematical level, forming the core data carrier for encryption processing.

[0121] Finally, a logical paradox structure is embedded within the stacked tensor product matrix. A logical paradox is a proposition containing self-referential statements that leads to contradiction. Through a pre-defined algorithm, a self-referential logical structure similar to the "liar's paradox" is constructed within the matrix. For example, an element is placed at a specific position in the matrix whose value depends on the results of operations on other elements, which in turn affect the element, thus forming a logical loop. By operating on this self-referential structure, the system induces contradictory points—the positions of matrix elements that cannot satisfy logical consistency. The row and column coordinates of these contradictory points are extracted from the matrix, and the coordinate sequences are converted into character or number combinations according to specific rules, ultimately generating an encryption key. This key integrates data source attributes, dual transformations of the data structure, and the contradictory characteristics of logical paradoxes, and is used to encrypt semantically clean data, ensuring data security during encrypted storage and transmission.

[0122] In one embodiment, after obtaining semantically clean data, the process includes:

[0123] Obtain the attribute information from each data source, map the attribute information to the control point coordinates of the Bézier curve, and generate the Bézier curve.

[0124] Using the line connecting the start and end points of the Bézier curve as the axis, the region enclosed by the curve is recursively divided into a binary tree structure, forming a curve binary tree;

[0125] Based on the aforementioned curve binary tree, a number sequence is generated according to the rules;

[0126] The order of the digital sequence is determined by the overall curvature of the Bézier curve. The grouping method of the digital sequence is controlled by the number of inflection points of the curve. Modulo operation and character mapping are performed on the grouped digital sequences. The results are combined according to the level order traversal of the curve binary tree to generate an encryption key for encrypting semantically clean data.

[0127] In this embodiment, firstly, the system acquires the attribute information of each data source, which includes key parameters such as data source type, data volume, and update frequency. After numerical processing of this attribute information, the system maps it to the coordinates of control points on a Bézier curve. Specifically, each data source attribute corresponds to one or more control points on the curve; the numerical value of the attribute determines the position of the control point in the coordinate system, and the logical relationships between attributes affect the distribution pattern of the control points. Through precise calculation and mapping, the system generates a Bézier curve that fully reflects the attribute characteristics of the data source. The shape, curvature, and trend of this curve are all determined by the data source attribute information, forming the core foundation for subsequent encryption processing.

[0128] Next, using the line connecting the start and end points of the generated Bézier curve as the dividing axis, the region enclosed by the curve is recursively divided. The division process employs a binary tree structure. Starting from the root node, the region enclosed by the curve is divided into left and right sub-regions along the dividing axis each time. Each sub-region becomes a child node and continues to the next round of division until a preset termination condition is met (e.g., the area of ​​a sub-region is less than a threshold or the division level reaches its upper limit). During the division process, the position, direction, and feature information of each sub-region are recorded, ultimately forming a complete curve binary tree. The curve binary tree not only reflects the spatial division relationship of the Bézier curve region but also implies the geometrical distribution characteristics of the data source attributes.

[0129] Subsequently, based on the constructed curved binary tree, a numerical sequence is generated according to pre-defined rules. The rules are closely linked to the node attributes and structural features of the binary tree, using information such as the node's level number, left and right subtree identifiers, and the geometric parameters (such as area and perimeter) of the region represented by the node as the basis for number generation. Each node is traversed, and the corresponding number is calculated using a specific algorithm based on the node's specific attribute values. These numbers are then arranged in a certain order to form the initial numerical sequence. This numerical sequence contains the structural information of the curved binary tree and the geometric features of the Bézier curve region.

[0130] Finally, the order of the number sequence is determined based on the overall curvature of the Bézier curve. If the curve bends clockwise, the number sequence is arranged from left to right; if it bends counterclockwise, it is arranged from right to left. Simultaneously, the grouping method of the number sequence is controlled by counting the number of inflection points on the Bézier curve; for example, each inflection point corresponds to a group of numbers, and the number of inflection points determines the number of groups. After grouping, a modulo operation is performed on each group of numbers, and an appropriate modulus is selected to map the numbers to a specific range. The result of the modulo operation is then converted into characters using a character mapping table. Finally, following the level-order traversal of the curve binary tree, the converted characters are combined sequentially to generate the final encryption key. This key integrates data source attribute information, the geometric features of the Bézier curve, and the structural characteristics of the curve binary tree, and is used to encrypt semantically clean data, ensuring the security and confidentiality of the data during storage.

[0131] Reference Figure 2 Another embodiment of the present invention also provides an automated multi-source data acquisition device based on a large language model, comprising:

[0132] The analysis unit is used to analyze the characteristic distribution of historical data and dynamically generate the optimal data collection strategy by combining it with the current status of the data source.

[0133] The compensation unit is used to schedule acquisition tasks in real time based on the optimal acquisition strategy to obtain a dynamically scheduled data stream; and to perform real-time anomaly detection and adaptive compensation on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream.

[0134] The extension unit is used to construct a domain knowledge graph based on a large language model, analyze the entity associations and business logic of the self-fault-tolerant data stream, and predict potential data demand points; based on the predicted potential data demand points, it automatically expands the data collection scope to obtain knowledge-enhanced data assets.

[0135] The cleaning unit is used to identify the protocol characteristics of the data transmission flow of the data asset, and to parse the data based on the protocol feature mapping to obtain protocol-independent standardized data; the protocol-independent standardized data is then subjected to semantic noise filtering and cross-modal cleaning to obtain semantically clean data.

[0136] In this embodiment, the specific implementation of each unit in the above device embodiment is described in the above method embodiment, and will not be repeated here.

[0137] Reference Figure 3 This invention also provides a computer device, which can be a server, and its internal structure can be as follows: Figure 3As shown, the computer device includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores the data corresponding to this embodiment. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described method.

[0138] Those skilled in the art will understand that Figure 3 The structures shown are merely block diagrams of some structures related to the present invention and do not constitute a limitation on the computer devices on which the present invention is applied.

[0139] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0140] In summary, the automated multi-source data acquisition method based on a large language model provided in this embodiment of the invention includes: analyzing the feature distribution of historically acquired data and dynamically generating an optimal acquisition strategy based on the current state of the data source; scheduling acquisition tasks in real time based on the optimal acquisition strategy to obtain a dynamically scheduled data stream; performing real-time anomaly detection and adaptive compensation on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream; constructing a domain knowledge graph based on a large language model, analyzing the entity associations and business logic of the self-fault-tolerant data stream, and predicting potential data demand points; automatically expanding the data acquisition scope based on the predicted potential data demand points to obtain knowledge-enhanced data assets; identifying the protocol characteristics of the data transmission stream of the data assets, parsing the data based on a protocol feature mapping parser to obtain protocol-independent standardized data; and performing semantic noise filtering and cross-modal cleaning on the protocol-independent standardized data to obtain semantically clean data. In this invention, by analyzing the characteristic distribution of historically collected data and combining it with the current state of the data source, the optimal collection strategy is dynamically generated. At the same time, the data collection range is automatically expanded by predicting potential data demand points. Then, the data is parsed based on the protocol feature mapping parser to obtain protocol-independent standardized data. This overcomes the shortcomings of existing multi-source data collection technologies that cannot perform protocol adaptation and adaptive strategy optimization.

[0141] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the present invention and embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0142] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0143] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for automated acquisition of multi-source data based on a large language model, characterized in that, Includes the following steps: Analyzing the characteristic distribution of historically collected data and combining it with the current state of the data source, the optimal collection strategy is dynamically generated. This includes: slicing the historically collected data and extracting the data volume fluctuation characteristics, value density distribution, and field integrity characteristics of each slice to obtain a data feature vector; monitoring the connection success rate, response timeliness, and server load pressure of each data source and performing hierarchical evaluation to obtain a data source state matrix; constructing a reinforcement learning model to perform multi-objective reinforcement learning optimization processing on the data feature vector and the data source state matrix to generate an optimal parameter vector; and obtaining the optimal collection strategy based on the optimal parameter vector. Based on the optimal acquisition strategy, acquisition tasks are scheduled in real time to obtain a dynamically scheduled data stream; real-time anomaly detection and adaptive compensation are performed on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream. This process involves constructing a domain knowledge graph based on a large language model, analyzing entity relationships and business logic within the self-fault-tolerant data stream, and predicting potential data demand points. The process includes: identifying entities in the self-fault-tolerant data stream, constructing semantic relationships between entities, and generating a set of entity relationship triples; constructing a domain knowledge graph based on a large language model, mapping the set of entity relationship triples to the domain knowledge graph, calculating the similarity between entities, and completing implicit relationships to obtain a dynamic knowledge graph; analyzing the relationship paths of entities in the dynamic knowledge graph, extracting business rules, and combining this with event sequence pattern identification through time-series analysis to generate a business logic rule base; and analyzing the business logic rule base and the dynamic knowledge graph using a Transformer-based demand prediction model to identify data gaps and gaps in business relationships, and outputting potential data demand points. Based on predicted potential data demand points, the scope of data collection is automatically expanded to obtain knowledge-enhanced data assets; The protocol characteristics of the data transmission flow of the data asset are identified, and the data is parsed by a parser based on the protocol feature mapping to obtain protocol-independent standardized data; semantic noise filtering and cross-modal cleaning are performed on the protocol-independent standardized data to obtain semantically clean data.

2. The automated multi-source data acquisition method based on a large language model according to claim 1, characterized in that, The protocol-independent standardized data refers to data that has eliminated the communication protocol characteristics on which the original data depended and has been uniformly converted into an independent data format.

3. The automated multi-source data acquisition method based on a large language model according to claim 1, characterized in that, The acquisition strategy includes sampling frequency, batch size, and priority sorting.

4. The automated multi-source data acquisition method based on a large language model according to claim 1, characterized in that, Real-time anomaly detection and adaptive compensation are performed on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream, including: The dynamically scheduled data stream is sliced, and numerical features, temporal features, and semantic features are extracted to construct a feature vector set; Anomaly detection is performed based on the feature vector set to generate anomaly events containing anomaly type and confidence level; Abnormal events are mapped to a pre-built fault knowledge graph, and the abnormal propagation path is analyzed through graph neural networks to locate the root cause of the problem and generate a diagnostic report. The optimal compensation strategy is automatically matched based on the diagnostic report, and parallel compensation is performed on the dynamically scheduled data stream according to the optimal compensation strategy to obtain a self-fault-tolerant data stream.

5. The automated multi-source data acquisition method based on a large language model according to claim 1, characterized in that, The protocol characteristics of the data transmission flow of the data asset are identified, and the data is parsed by a parser based on the protocol characteristic mapping to obtain protocol-independent standardized data, including: The data transmission stream of the data asset is scanned bit by bit using a finite state automaton, and the protocol type and data structure characteristics are identified according to the preset protocol state transition rules. The parser is retrieved from the parser rule base according to the corresponding protocol type and data structure characteristics, and the data is decapsulated according to the protocol and mapped to the fields to obtain protocol-independent standardized data containing data source tags, timestamps and business fields.

6. The automated multi-source data acquisition method based on a large language model according to claim 1, characterized in that, Semantic noise filtering and cross-modal cleaning are performed on the protocol-independent standardized data to obtain semantically clean data, including: Feature extraction is performed on the protocol-independent standardized data to obtain a neuromorphic feature map, including pulse timing features, synaptic weight features, and population coding features. The neural morphological feature map is subjected to noise filtering to obtain an enhanced feature map; The enhanced feature map is subjected to entity parsing, event extraction, and causal reasoning to construct a cognitive map; The cognitive graph is subjected to cross-modal semantic alignment and meta-learning-driven cleaning to obtain semantically clean data.

7. An automated data acquisition device based on a large language model, characterized in that, include: The analysis unit is used to analyze the feature distribution of historically collected data and dynamically generate the optimal collection strategy based on the current data source status. This includes: slicing the historically collected data and extracting the data volume fluctuation characteristics, value density distribution, and field integrity characteristics of each slice to obtain a data feature vector; monitoring the connection success rate, response timeliness, and server load pressure of each data source and performing hierarchical evaluation to obtain a data source status matrix; constructing a reinforcement learning model to perform multi-objective reinforcement learning optimization processing on the data feature vector and the data source status matrix to generate an optimal parameter vector; and obtaining the optimal collection strategy based on the optimal parameter vector. The compensation unit is used to schedule acquisition tasks in real time based on the optimal acquisition strategy to obtain a dynamically scheduled data stream; and to perform real-time anomaly detection and adaptive compensation on the dynamically scheduled data stream to obtain a self-fault-tolerant data stream. An extension unit is used to construct a domain knowledge graph based on a large language model, analyze the entity associations and business logic of the self-fault-tolerant data stream, and predict potential data demand points. This includes: identifying entities in the self-fault-tolerant data stream, constructing semantic associations between entities, and generating a set of entity relationship triples; constructing a domain knowledge graph based on a large language model, mapping the set of entity relationship triples to the domain knowledge graph, calculating the similarity between entities, and completing implicit relationships to obtain a dynamic knowledge graph; analyzing the association paths of entities in the dynamic knowledge graph, extracting business rules, and combining this with event sequence pattern identification through time-series analysis to generate a business logic rule base; analyzing the business logic rule base and the dynamic knowledge graph using a Transformer-based demand prediction model to identify data gaps and business association blanks, and outputting potential data demand points; and automatically expanding the data collection scope based on the predicted potential data demand points to obtain knowledge-enhanced data assets. The cleaning unit is used to identify the protocol characteristics of the data transmission flow of the data asset, and to parse the data based on the protocol feature mapping to obtain protocol-independent standardized data; the protocol-independent standardized data is then subjected to semantic noise filtering and cross-modal cleaning to obtain semantically clean data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images