Multi-source data fusion and analysis method and system for complex scene

Abstract

The invention discloses a multi-source data fusion and analysis method and system for a complex scene, belongs to the technical field of data processing, and aims to solve the contradiction between a static semantic model and dynamic data evolution. The method comprises the following steps: firstly, configuring a data pipeline, fully extracting source data from an original database to a target database to obtain a source data set, and respectively obtaining a first data set and a second data set through cleaning, dump and link validity judgment; analyzing the semantic constraint relationship of the entity attributes of the ontology model to obtain a rule set, and mapping the second data set field to the target entity attributes of the ontology model to obtain a preliminary alignment data set; generating temporary attributes for unmatched fields through context association analysis, and expanding the temporary attributes to the ontology model to obtain a semantic alignment data set; standardizing the numerical value to obtain a third data set; and finally, carrying out multi-modal feature conjoint analysis on the third data set to generate a dynamic decision index.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-source data fusion and analysis method and system for complex scenarios. Background Art

[0002] With the deepening of digital transformation, data sources are becoming multimodal, cross-platform, and highly dynamic. In complex scenarios such as smart cities and the Industrial Internet of Things, real-time fusion and analysis of multi-source data are the core links supporting intelligent decision-making. However, the heterogeneity of data semantics in complex scenarios is becoming increasingly prominent: there are significant differences in field naming rules, contextual meanings, and dynamic evolution logic between different sources, resulting in frequent manual intervention during the fusion process to adapt to semantic changes. For example, the same entity attribute may be expressed in completely different forms in different data sources (such as "GPS coordinates" and "geographic location identifiers"), and traditional methods rely on predefined static mapping rules or fixed semantic models, and are unable to perceive contextual associations in real time or dynamically expand semantic constraints.

[0003] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of this application. Summary of the Invention

[0004] This application provides a multi-source data fusion and analysis method and system for complex scenarios, which is used to resolve the contradiction between static semantic models and dynamic data evolution, and realize the autonomous evolution capability of semantic alignment in the process of multi-source heterogeneous data fusion.

[0005] To achieve the above objectives, the present application discloses the following technical solutions:

[0006] In a first aspect, embodiments of the present application provide a multi-source data fusion and analysis method for complex scenarios, comprising the following steps:

[0007] Configure a data pipeline from the original database to the target database, and extract all source data from the original database to the target database through the data pipeline to obtain the source dataset;

[0008] Performing a cleaning operation on the source dataset in the target database, including data deduplication, missing value filling, and outlier detection, to obtain a first dataset;

[0009] Dumping the first data set, determining the validity of the file links contained in the data set, dumping the files corresponding to the valid links to the local server and generating a hash primary key to obtain the second data set;

[0010] Parse the semantic constraint relationship of entity attributes in the predefined ontology model to obtain a set of semantic constraint rules;

[0011] According to a set of semantic constraint rules, the fields of different data sources in the second dataset are mapped to the target entity attributes in the ontology model to obtain a preliminary aligned dataset;

[0012] For the unmatched fields in the preliminary aligned dataset, temporary attributes are generated through contextual association analysis, and the temporary attributes are expanded into the ontology model to generate a semantically aligned dataset;

[0013] The semantically aligned dataset is numerically normalized, and the value range of the same field is mapped to a unified scale to obtain the third dataset;

[0014] Perform a multimodal feature joint analysis on the third data set, generate dynamic decision indicators based on preset scenario rules, and store them persistently in a distributed database.

[0015] In an embodiment of the present application, a full data pipeline is first established from the original data to the target database to ensure centralized processing of the physical layer of multi-platform data and provide a data foundation for semantic unification. After obtaining the second data set that has been cleaned and verified for validity, the field mapping of heterogeneous data sources is implemented based on the semantic constraint rules of the predefined ontology model, and the same entity attributes of different naming rules are mapped to standardized semantic expressions to form a preliminary semantic alignment. For dynamically added fields that cannot be handled by traditional static mapping methods, the technical solution uses runtime context association analysis to generate temporary attributes, and realizes the adaptive evolution of semantic structure through the dynamic expansion mechanism of the ontology model. In this way, when a new data source is connected and an undefined field appears, it can autonomously create a temporary semantic node and complete conflict verification, and maintain semantic consistency without manually updating the mapping rules. The multimodal data that has been numerically standardized is subjected to feature joint analysis under the expanded semantic framework, and finally generates dynamic decision indicators that can be directly used for intelligent decision-making.

[0016] In addition, by coordinating the initial semantic constraints of the ontology model with dynamic expansion at runtime, a flexible semantic adaptation system is formed, which effectively resolves the contradiction between the static semantic model and the dynamic data evolution, and realizes the autonomous evolution capability of semantic alignment in the process of multi-source heterogeneous data fusion.

[0017] In a second aspect, the embodiments of the present application provide a multi-source data fusion and analysis system for complex scenarios, including:

[0018] The data pipeline configuration module is used to configure the data pipeline from the original database to the target database, and extract the source data from the original database to the target database through the data pipeline to obtain the source data set;

[0019] A data cleaning module is used to perform cleaning operations on the source data set in the target database, including data deduplication, missing value filling and outlier detection, to obtain a first data set;

[0020] A data dump module is used to dump the first data set, determine the validity of the file links contained in the data set, dump the files corresponding to the valid links to the local server and generate a hash primary key to obtain the second data set;

[0021] Semantic rule acquisition module, used to parse the semantic constraint relationship of entity attributes in the predefined ontology model and obtain the semantic constraint rule set;

[0022] a data field mapping module, configured to map fields from different data sources in the second data set to target entity attributes in the ontology model according to a set of semantic constraint rules, to obtain a preliminary aligned data set;

[0023] The temporary attribute generation module is used to generate temporary attributes for the unmatched fields in the preliminary aligned dataset through contextual association analysis, and then expand the temporary attributes into the ontology model to generate a semantically aligned dataset;

[0024] The numerical standardization module is used to perform numerical standardization on the semantically aligned dataset, mapping the value range of the same field to a unified scale to obtain the third dataset;

[0025] The multimodal analysis and decision-making module is used to perform multimodal feature joint analysis on the third data set, generate dynamic decision indicators based on preset scenario rules, and store them persistently in a distributed database.

[0026] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any technical solution of the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any technical solution of the first aspect is implemented.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any technical solution of the first aspect.

[0029] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0031] Figure 1 A flowchart of a multi-source data fusion and analysis method for complex scenarios provided in some embodiments of the present application;

[0032] Figure 2 A performance comparison chart of the multi-source data fusion and analysis method for complex scenarios provided by this application compared with traditional methods;

[0033] Figure 3 A schematic diagram of the structure of a multi-source data fusion and analysis system for complex scenarios provided in some embodiments of the present application;

[0034] Figure 4 It is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present application. DETAILED DESCRIPTION

[0035] Specific embodiments of the present invention will now be mentioned in detail. Although the present invention is described in conjunction with these specific embodiments, it should be appreciated that the present invention is not intended to be limited to these specific embodiments. On the contrary, these embodiments are intended to cover substitutions, changes, or equivalent embodiments that may be included within the spirit and scope of the invention defined by the claims. In the following description, a large number of specific details are set forth in order to provide a comprehensive understanding of the present invention. The present invention may be implemented without some or all of these specific details.

[0036] When used in conjunction with "including," "methods comprising," or similar language in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0037] Application Overview: With the deepening of digital transformation, data sources are becoming multimodal, cross-platform, and highly dynamic. In complex scenarios such as smart cities and the Industrial Internet of Things, real-time fusion and analysis of multi-source data are the core links supporting intelligent decision-making. However, the heterogeneity of data semantics in complex scenarios is becoming increasingly prominent: there are significant differences in field naming rules, contextual meanings, and dynamic evolution logic from different sources, resulting in frequent manual intervention during the fusion process to adapt to semantic changes. For example, the same entity attribute may be expressed in completely different forms in different data sources (such as "GPS coordinates" and "geographic location identifiers"), and traditional methods rely on predefined static mapping rules or fixed semantic models, and are unable to perceive contextual associations in real time or dynamically expand semantic constraints.

[0038] In response to the above technical problems, the overall idea of the technical solution provided by this application is as follows: a multi-source data fusion and analysis method for complex scenarios, including the following steps: configuring a data pipeline from the original database to the target database, extracting the source data from the original database to the target database in full through the data pipeline, and obtaining a source data set; performing a cleaning operation on the source data set in the target database, including data deduplication, missing value filling and outlier detection, to obtain a first data set; dumping the first data set, judging the validity of the file links contained in the data set, dumping the files corresponding to the valid links to the local server and generating a hash primary key, to obtain a second data set; parsing the predefined local The semantic constraint relationship of entity attributes in the ontology model is obtained to obtain a set of semantic constraint rules; according to the set of semantic constraint rules, the fields of different data sources in the second data set are mapped to the target entity attributes in the ontology model to obtain a preliminary aligned data set; for the unmatched fields in the preliminary aligned data set, temporary attributes are generated through context association analysis, and the temporary attributes are extended to the ontology model to generate a semantic aligned data set; the semantic aligned data set is numerically normalized, the value range of the same field is mapped to a unified scale to obtain a third data set; the third data set is jointly analyzed with multimodal features, dynamic decision indicators are generated according to preset scenario rules, and persistently stored in a distributed database.

[0039] The method first establishes a full data pipeline from the original data to the target database to ensure the centralized physical layer processing of multi-platform data and provide a data foundation for semantic unification. After obtaining the second data set that has been cleaned and verified for validity, the field mapping of heterogeneous data sources is implemented based on the semantic constraint rules of the predefined ontology model, and the same entity attributes with different naming rules are mapped to standardized semantic expressions to form a preliminary semantic alignment. For dynamically added fields that cannot be handled by traditional static mapping methods, the technical solution uses runtime context association analysis to generate temporary attributes, and realizes the adaptive evolution of semantic structure through the dynamic expansion mechanism of the ontology model. In this way, when an undefined field appears when a new data source is connected, a temporary semantic node can be created autonomously and conflict verification can be completed, so that semantic consistency can be maintained continuously without manually updating the mapping rules. The multimodal data that has been numerically standardized is subjected to feature joint analysis under the expanded semantic framework, and finally generates dynamic decision indicators that can be directly used for intelligent decision-making.

[0040] In addition, this method forms a flexible semantic adaptation system by coordinating the initial semantic constraints of the ontology model with dynamic expansion at runtime, effectively resolving the contradiction between static semantic models and dynamic data evolution, and realizing the autonomous evolution capability of semantic alignment in the process of multi-source heterogeneous data fusion.

[0041] After introducing the basic principles of this application, various non-limiting implementation methods of this application will be specifically introduced in conjunction with the accompanying drawings. Figure 1 , the embodiment of the present application provides a multi-source data fusion and analysis method for complex scenarios, including the following steps:

[0042] S101: Configure a data pipeline from the original database to the target database, and extract all source data from the original database to the target database through the data pipeline to obtain a source data set;

[0043] For example, the data pipeline can use Apache Kafka as the messaging middleware, configure a Debezium connector to capture the original database's change log in real time, and set the parallelism to twice the number of CPU cores in the cluster to optimize throughput. When extracting full data, enable paging queries with a page size of 1,000 records per page, manage connection resources through a JDBC connection pool, and set a timeout of 300 seconds.

[0044] S102: performing a cleaning operation on the source dataset in the target database, including data deduplication, missing value filling, and outlier detection, to obtain a first dataset;

[0045] For example, data deduplication uses a primary key hash comparison algorithm to retain the entry with the latest timestamp for duplicate records; missing value filling uses a random forest regression model to predict missing fields, and the feature input is other attribute values of the same entity and time series context window data; outlier detection is based on the isolation forest algorithm.

[0046] S103: Dumping the first data set, determining the validity of the file links contained in the data set, dumping the files corresponding to the valid links to the local server and generating a hash primary key to obtain the second data set;

[0047] For example, the validity of the file link is verified by using a HEAD request to detect the HTTP status code, setting a timeout of 5 seconds and a retry count of 3 times; the hash key is generated using the SHA-256 algorithm, the file content is divided into blocks (1MB per block), and the hash values are calculated in parallel and then concatenated into the final key.

[0048] S104: parsing the semantic constraint relationship of entity attributes in the predefined ontology model to obtain a set of semantic constraint rules;

[0049] Specifically, in some embodiments, the execution subject of the multi-source data fusion and analysis method for complex scenarios can parse the semantic constraint relationship of entity attributes in a predefined ontology model through the following steps to obtain a set of semantic constraint rules:

[0050] In the first step, the class hierarchical relationship between entities is constructed using the OWL ontology modeling framework based on the predefined ontology model;

[0051] For example, an example of constructing a class hierarchy relationship is to define the "device" class as the parent class of the "sensor" class, declare the inheritance relationship through owl:subClassOf, and limit the scope of application of the attribute in rdfs:domain.

[0052] The second step is to define the transitive dependency constraints between attributes by parsing the RDF triples in the predefined ontology model;

[0053] Exemplarily, the transitive dependency constraint resolution uses SPARQL query language to extract all transitive properties.

[0054] The third step is to parse the data type constraints and attribute value range restrictions according to the SWRL rules of the predefined ontology model;

[0055] The fourth step is to integrate class hierarchical relationships, transitive dependency constraints and value range restrictions to generate a set of semantic constraint rules.

[0056] In this way, a structured semantic framework for entity attributes can be established through class hierarchical relationships constructed within the OWL framework, providing a hierarchical reasoning foundation for inter-attribute association constraints. Within this framework, RDF triple parsing activates the dynamic identification of attribute transitive dependencies, making implicit cross-hierarchical attribute associations explicit. SWRL rule parsing further incorporates fine-grained numerical logic, transforming abstract semantic constraints into executable data validation conditions. These three elements form a progressive constraint system of "structure-relationship-logic": class-level constraints define the legal scope of attribute associations, transitive dependency constraints reveal implicit association paths between attributes, and range constraints lock in the logical boundaries of specific values. This hierarchical constraint mechanism enables the ontology model to simultaneously possess both structured semantic expression and dynamic rule execution capabilities. During the data mapping phase, it can filter illegal attribute associations through class inheritance relationships, derive implicit association rules across data sources through transitive dependencies, and, in real time, prevent anomalous mappings that violate business logic using range constraints.

[0057] S105: Mapping the fields of different data sources in the second dataset to the target entity attributes in the ontology model according to the semantic constraint rule set to obtain a preliminary aligned dataset;

[0058] Specifically, in some embodiments, the execution entity may map fields from different data sources in the second dataset to target entity attributes in the ontology model according to a set of semantic constraint rules through the following steps to obtain a preliminary aligned dataset:

[0059] In the first step, a regular expression pattern library of field names is established based on the semantic constraint rule set, and the cosine similarity algorithm is used to calculate the semantic similarity between the fields of the second dataset and the target entity attributes;

[0060] For example, the regular expression pattern library includes rules such as ^[AZ]{3}_(\d+) for stripping field prefixes;

[0061] In the second step, when the semantic similarity exceeds the preset threshold, the attribute mapping relationship is automatically established;

[0062] Exemplarily, the preset threshold may be, but is not limited to, 0.85, and is determined by ROC curve analysis.

[0063] The third step is to verify the logical consistency of the mapping relationship through the ontology reasoning engine;

[0064] For example, the consistency-check command may be executed by using a Pellet inference engine to verify whether the mapped attributes violate class hierarchy or value range constraints.

[0065] The fourth step is to integrate the verified mapping relationships to generate a preliminary alignment dataset. The field name regular expression pattern library enables structured feature extraction, removing non-core semantic elements such as business prefixes and abbreviations from the original fields to form standardized feature vectors for matching. The cosine similarity algorithm calculates semantic relevance based on the preprocessed feature vectors, effectively overcoming the semantic drift caused by differences in naming conventions in traditional text matching.

[0066] A preset threshold mechanism implements dynamic filtering after semantic similarity calculation, preventing low-quality mappings from wasting verification resources while retaining potentially high-value matches for subsequent processing. The ontology reasoner verifies the logical consistency of mappings that pass the threshold screening. Combining the class hierarchy and attribute constraints defined in the ontology model, it identifies and removes erroneous mappings that violate data type or value range restrictions. Finally, the verified mappings are integrated to form a preliminary alignment dataset.

[0067] In addition, the combination of regular expression feature extraction and cosine similarity calculation can enhance the core semantic capture capability, and the connection between threshold filtering and ontology reasoning forms a dual guarantee mechanism for optimized allocation of computing resources and mapping quality, which enables massive field mapping to improve processing efficiency while ensuring accuracy.

[0068] S106: For the unmatched fields in the preliminary aligned dataset, generate temporary attributes through context association analysis, and expand the temporary attributes into the ontology model to generate a semantically aligned dataset;

[0069] Specifically, in some embodiments, the execution entity may generate temporary attributes for unmatched fields in the preliminary aligned dataset through contextual analysis, and extend the temporary attributes into the ontology model to generate semantically aligned data through the following steps:

[0070] In the first step, the graph attention network is used to analyze the contextual association path of the unmatched fields in the knowledge graph, and the key path is identified by calculating the node association weight. The node association weight calculation formula is:

[0071]

[0072] Where, and is the n-dimensional feature vector of nodes i and j, is the trainable weight matrix, is the attention coefficient vector, represents the set of adjacent nodes of node i, ‖ represents the vector concatenation operation, LeakyReLU is the linear rectification function with leakage parameters; T represents vector transposition; represents a set of real numbers; n-node feature vector Dimensions; m weight matrix number of rows;

[0073] In the second step, a temporary attribute is generated when the association path weight value reaches the preset association threshold;

[0074] The third step is to perform the ontology model version tagging and verify the naming conflicts between the temporary attributes and the existing attributes;

[0075] The fourth step is to automatically expand the ontology model attribute set when the conflict verification passes;

[0076] In the fifth step, attribute mapping is re-executed based on the expanded ontology model to generate a semantically aligned dataset.

[0077] In this way, the efficiency of handling unmatched fields can be improved through semantic association analysis and dynamic ontology expansion driven by a graph attention network. Specifically, the graph attention network performs weighted analysis of the contextual association paths of unmatched fields in the knowledge graph. Using node feature vectors and a trainable parameter matrix to calculate association weights, it can accurately identify potential semantic connections across entity attributes, overcoming the limitations of traditional rule engines in complex association reasoning. When high-weight association paths are detected, temporary attribute nodes are automatically generated. The ontology model version tagging and naming conflict verification mechanisms ensure the security of semantic expansion and avoid logical inconsistencies caused by dynamic attribute additions. The expanded ontology model re-executes attribute mapping to form a closed-loop optimization, enabling autonomous semantic digestion of undefined fields. This allows for the continuous evolution of semantic alignment capabilities while maintaining the stability of the ontology model structure, significantly reducing the risk of semantic gaps caused by dynamic changes in data sources and ensuring the consistency of semantic expression throughout multiple rounds of data fusion.

[0078] S107: performing numerical standardization on the semantically aligned dataset, mapping the value range of the same field to a unified scale, and obtaining a third dataset;

[0079] Specifically, in some embodiments, the execution entity may obtain the third data set through the following steps:

[0080] In the first step, the Z-score normalization algorithm is used to process the discrete distribution of numerical fields in the semantic alignment dataset. The calculation formula is:

[0081]

[0082] In the formula, x is the original value, μ is the field mean, σ is the field standard deviation, and x ′ To standardize the results;

[0083] The second step is to apply the binning method to merge the value ranges of the categorical fields in the semantic alignment dataset;

[0084] The third step is to dynamically select the normalization or quantile transformation method based on the field value distribution characteristics;

[0085] The fourth step is to uniformly map the processed values to the [0,1] interval to generate the third data set.

[0086] In this way, the scale of multi-source heterogeneous data values is unified through the adaptation of differentiated data features. Specifically, the Z-score normalization algorithm is used to eliminate dimensional differences for numerical fields, and the discrete distribution is converted into a statistical standard form with zero mean and unit variance through mean and standard deviation calculations, ensuring that data of different dimensions have a basis for joint analysis. Binning is implemented on categorical fields to merge value ranges and reduce semantic noise caused by differences in classification systems. Based on the dynamic selection of normalization or quantile transformation methods based on field value distribution characteristics, the optimal conversion strategy can be self-adapted according to the data form, avoiding the distortion effect of a single normalization method on skewed distribution data. Finally, all numerical values are mapped to the [0,1] interval to form a unified metric space, which not only eliminates the interference of multi-source data value range differences on the analysis model, but also provides a standardized input paradigm for subsequent machine learning algorithms.

[0087] S108: Perform multimodal feature joint analysis on the third data set, generate dynamic decision indicators according to preset scenario rules, and store them persistently in a distributed database.

[0088] Specifically, in some embodiments, the execution entity may perform a multimodal feature joint analysis on the third data set through the following steps, generate a dynamic decision indicator according to preset scenario rules, and persistently store the indicator in a distributed database, including:

[0089] The first step is to use the Drools rule engine to parse the multimodal feature association conditions in the scenario rules and perform association condition analysis on the multimodal features in the third dataset.

[0090] In the second step, the entropy weight method is used to calculate the initial weight value of each characteristic index in the third data set. The information entropy calculation formula is:

[0091]

[0092] Where, x ij is the normalized value of the jth feature of the i-th sample, m is the total number of samples, E j ∈[0,1] is the information entropy of feature j;

[0093] The third step is to adjust the feature weight distribution ratio in real time based on the data quality assessment results of the third dataset;

[0094] In the fourth step, the LSTM network is used to predict the time series trend of the characteristic indicators in the third data set; the hidden state update formula is:

[0095] h t =o t ⊙tanh(c t );

[0096] in, is the output gate vector at time t, is the cell state vector at time t, ⊙ represents element-by-element multiplication, and tanh is the hyperbolic tangent function;

[0097] Step 5: Integrate the adjusted feature weights with the trend prediction results of the third data set to generate dynamic decision indicators;

[0098] The sixth step is to persist the dynamic decision indicators into a distributed database using the consistent hashing algorithm according to the timestamp range sharding strategy.

[0099] In this solution, the Drools rule engine parses pre-set scenario rules to implement explicit control of business logic, ensuring that multimodal feature correlation analysis complies with domain knowledge constraints and avoiding the logical transgression risks associated with purely data-driven approaches. The entropy weight method dynamically calculates indicator weights based on the information entropy of the eigenvalue distribution. This allows for differentiated weight distribution for time-sensitive indicators in traffic flow monitoring and spatially distributed indicators in meteorological data, strengthening the quantitative expression of the data's intrinsic value. The LSTM network's ability to extract temporal features from standardized data effectively captures the nonlinear variations in industrial equipment sensor data, improving the time dimension accuracy of equipment status predictions. The dynamic decision indicator generation process integrates a triple validation mechanism: rule constraints, entropy weight quantification, and trend prediction, ensuring that the decision logic maintains both the structure of domain knowledge and sensitivity to data characteristics.

[0100] See also Figure 2 , this application intuitively demonstrates the performance advantages of this technical solution over the traditional method through two sets of comparative experimental data. In the semantic alignment and processing performance comparison diagram, as the data scale expands from thousands to hundreds of thousands, the semantic alignment accuracy of this application (blue curve) remains stable in the range of 93% to 98.7%, while the accuracy of the traditional method (red curve) continues to drop from 82.4% to 75%, especially when the data scale reaches 50,000. This application expands the accuracy advantage to 23.7 percentage points through a dynamic attribute expansion mechanism. In terms of processing throughput, the real-time processing capability of this application (pink curve) grows logarithmically with the data scale, reaching a processing capacity of 220,000 items per second at a data scale of 100,000, an increase of 182% compared to the 78,000 items of the traditional method (green curve). This is due to the coordinated optimization of Kafka's sharding strategy and parallel computing architecture.

[0101] The dynamic decision delay comparison chart reveals the core innovative value of this application: when the data scale exceeds 10,000, the decision delay of the traditional method (red curve) shows an exponential growth trend, reaching 4200 milliseconds at 100,000 data items, while the delay of this application (blue curve) continues to decrease from the initial 180 milliseconds and stabilizes in the 50 millisecond range due to the adaptive optimization of the LSTM prediction model. This performance inversion phenomenon verifies the synergy between the dynamic rule engine and the machine learning model. Especially in large-scale data scenarios, this application improves end-to-end decision efficiency by two orders of magnitude through real-time weight adjustment and distributed storage optimization. The two sets of experimental data jointly confirmed that this technical solution shows stronger robustness, scalability and intelligent features in the process of data scale expansion, and realizes the simultaneous optimization of three key indicators: accuracy, throughput and latency.

[0102] See also Figure 3 Based on the same inventive concept as the multi-source data fusion and analysis method for complex scenarios in the aforementioned embodiment, the present embodiment provides a multi-source data fusion and analysis system for complex scenarios, including:

[0103] The data pipeline configuration module 201 is used to configure a data pipeline from the original database to the target database, and extract the source data from the original database to the target database through the data pipeline to obtain a source data set;

[0104] The data cleaning module 202 is used to perform cleaning operations on the source data set in the target database, including data deduplication, missing value filling and outlier detection, to obtain a first data set;

[0105] The data dump module 203 is used to dump the first data set, determine the validity of the file links contained in the data set, dump the files corresponding to the valid links to the local server and generate a hash primary key to obtain the second data set;

[0106] Semantic rule acquisition module 204, used to parse the semantic constraint relationship of entity attributes in the predefined ontology model and obtain a set of semantic constraint rules;

[0107] A data field mapping module 205 is configured to map fields from different data sources in the second data set to target entity attributes in the ontology model according to a set of semantic constraint rules, thereby obtaining a preliminary aligned data set;

[0108] A temporary attribute generation module 206 is used to generate temporary attributes for unmatched fields in the preliminary aligned dataset through context association analysis, and to extend the temporary attributes into the ontology model to generate a semantically aligned dataset;

[0109] A numerical standardization module 207 is used to perform numerical standardization on the semantically aligned dataset, mapping the value range of the same field to a unified scale to obtain a third dataset;

[0110] The multimodal analysis and decision module 208 is used to perform a multimodal feature joint analysis on the third data set, generate dynamic decision indicators according to preset scenario rules, and store them persistently in a distributed database.

[0111] In some embodiments, the semantic rule acquisition module 204 is specifically used to: construct a class hierarchical relationship between entities based on a predefined ontology model using the OWL ontology modeling framework; define transitive dependency constraints between attributes by parsing RDF triples in the predefined ontology model; parse data type constraints and attribute value range restrictions according to the SWRL rules of the predefined ontology model; and integrate class hierarchical relationships, transitive dependency constraints, and value range restrictions to generate a set of semantic constraint rules.

[0112] In some embodiments, the data field mapping module 205 is specifically used to: establish a field name regular expression pattern library based on a set of semantic constraint rules, and use a cosine similarity algorithm to calculate the semantic similarity between the second data set field and the target entity attribute; automatically establish an attribute mapping relationship when the semantic similarity exceeds a preset threshold; verify the logical consistency of the mapping relationship through an ontology reasoning engine; and integrate the verified mapping relationships to generate a preliminary aligned data set.

[0113] In some embodiments, the temporary attribute generation module 206 is specifically configured to: use a graph attention network to analyze the context association path of the unmatched field in the knowledge graph, and identify the key path by calculating the node association weight, wherein the node association weight calculation formula is:

[0114]

[0115] Where, and is the n-dimensional feature vector of nodes i and j, is the trainable weight matrix, is the attention coefficient vector, represents the set of adjacent nodes of node i, ‖ represents the vector concatenation operation, LeakyReLU is the linear rectification function with leakage parameters; T represents vector transposition; represents a set of real numbers; n-node feature vector Dimensions; m weight matrix number of rows;

[0116] When the association path weight value reaches the preset association threshold, a temporary attribute is generated; the ontology model version is marked and the naming conflict between the temporary attribute and the existing attribute is verified; when the conflict verification passes, the ontology model attribute set is automatically expanded; based on the expanded ontology model, the attribute mapping is re-executed to generate a semantic alignment dataset.

[0117] In some embodiments, the numerical normalization module 207 is specifically configured to use a Z-score normalization algorithm to process the discrete distribution of numerical fields in the semantic alignment dataset. The calculation formula is:

[0118]

[0119] In the formula, x is the original value, μ is the field mean, σ is the field standard deviation, and x ′ To standardize the results;

[0120] The binning method is applied to merge the value ranges of the categorical fields in the semantic alignment dataset; the normalization or quantile transformation method is dynamically selected based on the field value distribution characteristics; the processed values are uniformly mapped to the [0,1] interval to generate the third dataset.

[0121] In some embodiments, the multimodal analysis and decision module 208 is specifically used to: parse the multimodal feature association conditions in the scenario rules through the Drools rule engine, and perform association condition analysis on the multimodal features in the third data set; and calculate the initial weight value of each feature index in the third data set using the entropy weight method. The information entropy calculation formula is:

[0122]

[0123] Where, x ij is the normalized value of the jth feature of the i-th sample, m is the total number of samples, E j ∈[0,1] is the information entropy of feature j;

[0124] Based on the data quality assessment results of the third data set, the feature weight distribution ratio is adjusted in real time. The LSTM network is used to predict the time series change trend of the feature indicators in the third data set. The hidden state update formula is:

[0125] h t =o t ⊙tanh(c t );

[0126] in, is the output gate vector at time t, is the cell state vector at time t, ⊙ represents element-by-element multiplication, and tanh is the hyperbolic tangent function;

[0127] The adjusted feature weights are integrated with the trend prediction results of the third data set to generate dynamic decision indicators. The dynamic decision indicators are persistently stored in a distributed database using a consistent hashing algorithm based on a timestamp range sharding strategy.

[0128] It is understandable that the modules and references recorded in this multi-source data fusion and analysis system for complex scenarios Figure 1 The steps in the multi-source data fusion and analysis method for complex scenarios described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the multi-source data fusion and analysis system for complex scenarios and the modules contained therein, and will not be repeated here.

[0129] See also Figure 4 , based on the inventive concept of a multi-source data fusion and analysis method for complex scenes in the aforementioned embodiment, an embodiment of the present application provides an electronic device. The electronic device may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device includes a processing device 301 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in ROM 302 (read-only memory) or the program loaded from the storage device 308 into RAM 303 (random access memory). Various programs and data required for the operation of the electronic device are also stored in RAM 303. The processing device 301, ROM 302, and RAM 303 are connected to each other via a bus 304. The input / output interface (i.e., I / O interface 305) is also connected to the bus 304.

[0130] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data.

[0131] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present application are performed.

[0132] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present application, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In some embodiments of the present application, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0133] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.

[0134] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can implement the method steps of any of the above technical solutions.

[0135] Computer program code for performing the operations of some embodiments of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0137] The modules described in some embodiments of the present application may be implemented in software or hardware. The modules described may also be provided in a processor. It is understood that the names of these modules do not limit the modules themselves.

[0138] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0139] Some embodiments of the present application further provide a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned multi-source data fusion and analysis methods for complex scenarios.

[0140] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A multi-source data fusion and analysis method for complex scenarios, characterized by: The following steps are involved: Configure a data pipeline from the original database to the target database, and extract all source data from the original database to the target database through the data pipeline to obtain the source dataset; Performing a cleaning operation on the source dataset in the target database, including data deduplication, missing value filling, and outlier detection, to obtain a first dataset; Dumping the first data set, determining the validity of the file links contained in the data set, dumping the files corresponding to the valid links to the local server and generating a hash primary key to obtain the second data set; Parse the semantic constraint relationship of entity attributes in the predefined ontology model to obtain a set of semantic constraint rules; According to the set of semantic constraint rules, mapping the fields of different data sources in the second data set to the target entity attributes in the ontology model to obtain a preliminary aligned data set; For unmatched fields in the preliminary aligned dataset, temporary attributes are generated through context association analysis, and the temporary attributes are expanded into the ontology model to generate a semantically aligned dataset; performing numerical standardization on the semantically aligned dataset, mapping the value range of the same field to a unified scale, and obtaining a third dataset; Perform a multimodal feature joint analysis on the third data set, generate dynamic decision indicators based on preset scenario rules, and store them persistently in a distributed database.

2. The multi-source data fusion and analysis method for complex scenarios according to claim 1 is characterized in that: The steps of parsing the semantic constraint relationship of entity attributes in the predefined ontology model and obtaining a set of semantic constraint rules include: Based on the predefined ontology model, the class hierarchical relationship between entities is constructed using the OWL ontology modeling framework. Defining transitive dependency constraints between attributes by parsing RDF triples in the predefined ontology model; Parsing data type constraints and attribute value range restrictions according to the SWRL rules of the predefined ontology model; The class hierarchical relationship, transitive dependency constraints and value range restrictions are integrated to generate a semantic constraint rule set.

3. The multi-source data fusion and analysis method for complex scenarios according to claim 2 is characterized in that: The step of mapping the fields of different data sources in the second data set to target entity attributes in the ontology model according to the set of semantic constraint rules to obtain a preliminary aligned data set includes: Establishing a field name regular expression pattern library based on the semantic constraint rule set, and using a cosine similarity algorithm to calculate the semantic similarity between the second data set field and the target entity attribute; When the semantic similarity exceeds the preset threshold, the attribute mapping relationship is automatically established; Verify the logical consistency of the mapping relationship through the ontology reasoning engine; The verified mapping relationships are integrated to generate a preliminary alignment dataset.

4. The multi-source data fusion and analysis method for complex scenarios according to claim 3 is characterized in that: For the unmatched fields in the preliminary aligned data set, temporary attributes are generated through context association analysis, and the temporary attributes are expanded into the ontology model to generate semantically aligned data, including the following steps: The graph attention network is used to analyze the contextual association path of unmatched fields in the knowledge graph, and the key path is identified by calculating the node association weight. The node association weight calculation formula is: Where, and is the n-dimensional feature vector of nodes i and j, is the trainable weight matrix, is the attention coefficient vector, represents the set of adjacent nodes of node i, || represents the vector concatenation operation, LeakyReLU is a linear rectification function with leakage parameters; T represents vector transposition; represents a set of real numbers; n-node feature vector Dimensions; m weight matrix number of rows; When the weight value of the association path reaches the preset association threshold, a temporary attribute is generated; Performs ontology model version tagging and verifies naming conflicts between temporary attributes and existing attributes; Automatically expand the ontology model attribute set when the conflict verification passes; Re-execute attribute mapping based on the expanded ontology model to generate a semantically aligned dataset.

5. The multi-source data fusion and analysis method for complex scenarios according to claim 4 is characterized in that: The steps of performing numerical standardization on the semantically aligned dataset and mapping the value range of the same field to a unified scale to obtain the third dataset include: The Z-score normalization algorithm is used to process the discrete distribution of numerical fields in the semantic alignment dataset. The calculation formula is: In the formula, x is the original value, μ is the field mean, σ is the field standard deviation, and x ′ To standardize the results; The binning method is applied to merge the value ranges of the categorical fields in the semantic alignment dataset; Dynamically select normalization or quantile transformation methods based on field value distribution characteristics; The processed values are uniformly mapped to the interval [0,1] to generate the third data set.

6. The multi-source data fusion and analysis method for complex scenarios according to claim 5 is characterized in that: The steps of performing a multimodal feature joint analysis on the third data set and generating a dynamic decision indicator according to preset scenario rules include: The Drools rule engine is used to parse the multimodal feature association conditions in the scenario rules, and the association conditions of the multimodal features in the third data set are parsed; The entropy weight method is used to calculate the initial weight value of each characteristic index in the third data set. The information entropy calculation formula is: Where, x ij is the normalized value of the jth feature of the i-th sample, m is the total number of samples, E j ∈[0,1] is the information entropy of feature j; Based on the data quality assessment results of the third data set, the feature weight distribution ratio is adjusted in real time; The LSTM network is used to predict the time series trend of the characteristic indicators in the third data set; the hidden state update formula is: h t =o t ⊙tanh(c t ) in, is the output gate vector at time t, is the cell state vector at time t, ⊙ represents element-by-element multiplication, and tanh is the hyperbolic tangent function; The adjusted feature weights are integrated with the trend prediction results of the third data set to generate dynamic decision indicators.

7. A multi-source data fusion and analysis system for complex scenarios, characterized by: include: The data pipeline configuration module is used to configure the data pipeline from the original database to the target database, and extract the source data from the original database to the target database through the data pipeline to obtain the source data set; A data cleaning module is used to perform cleaning operations on the source data set in the target database, including data deduplication, missing value filling and outlier detection, to obtain a first data set; A data dump module is used to dump the first data set, determine the validity of the file links contained in the data set, dump the files corresponding to the valid links to the local server and generate a hash primary key to obtain the second data set; Semantic rule acquisition module, used to parse the semantic constraint relationship of entity attributes in the predefined ontology model and obtain the semantic constraint rule set; a data field mapping module, configured to map fields from different data sources in the second data set to target entity attributes in the ontology model according to the set of semantic constraint rules, to obtain a preliminary aligned data set; A temporary attribute generation module is used to generate temporary attributes for unmatched fields in the preliminary aligned dataset through context association analysis, and to expand the temporary attributes into the ontology model to generate a semantically aligned dataset; a numerical standardization module, configured to perform numerical standardization on the semantically aligned dataset, mapping the value range of the same field to a unified scale, and obtaining a third dataset; The multimodal analysis and decision-making module is used to perform multimodal feature joint analysis on the third data set, generate dynamic decision indicators based on preset scenario rules, and store them persistently in a distributed database.

8. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which implements the method according to any one of claims 1 to 6 when executed by a processing device.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processing device, the method according to any one of claims 1 to 6 is implemented.

Images