Data collection method and device based on semantic analysis and storage medium

Abstract

The present application relates to the technical field of semantic analysis, and more particularly to a data collection method and device based on semantic analysis and a storage medium, which constructs a contradictory pair by means of semantic contradiction of real-time multi-source data, traces its evolution path and inheritance relationship in historical data by means of a semantic evolution prediction model, finally locates the contradictory bifurcation source and generates a contradictory attribution data set, realizes automatic tracing and attribution of data semantic contradiction, and changes traditional data quality post-passive detection into active risk interception and mechanism optimization driven by semantic consistency, effectively suppresses analysis and decision deviation caused by cross-source semantic conflict under the condition of not significantly increasing data governance investment, and significantly improves the data reliability and intelligent level of a multi-source data fusion system.

Description

Technical Field

[0001] This invention relates to the field of semantic analysis technology, and in particular to data acquisition methods, devices and storage media based on semantic analysis. Background Technology

[0002] In data-driven decision support systems, the fusion and utilization of multi-source heterogeneous data is a core component. These systems typically require continuous data collection from various sources, including databases, sensors, documents, reports, and third-party APIs. However, a common and challenging problem is that data from different sources often contains contradictory semantic information. For example, the attributes of the same entity may be recorded inconsistently in different systems, or there may be conflicts in the temporal and causal descriptions of the same event. If these semantic contradictions are not identified and traced back to their source, they will directly contaminate the data analysis results, leading to decision-making biases or even serious errors.

[0003] Existing data quality management technologies largely focus on the morphological quality dimension of the data itself, such as checking data format, value range, completeness, or uniqueness through predefined rules. These traditional methods have a fundamental flaw: they can only assess the "compliance" of data within a single dataset, completely ignoring the deeper "semantic consistency" between cross-source data. A piece of data that conforms to format specifications and has a normal value range may logically conflict with other equally "compliant" data. Simply relying on rule-based verification cannot trace the source and evolution of these contradictions, creating a blind spot in data quality attribution and risk warning, and failing to systematically intercept and attribute semantic contradictions at the source of data fusion and utilization.

[0004] Therefore, it is necessary to propose a data acquisition method, device, and storage medium based on semantic analysis to solve at least one of the above-mentioned technical problems. Summary of the Invention

[0005] To address this, the present invention provides a data acquisition method, apparatus, and storage medium based on semantic analysis, in order to overcome the problems in the prior art where it is impossible to trace back and utilize cross-source data semantic consistency for data quality and risk management, resulting in delayed identification and difficulty in attribution of semantic contradictions.

[0006] Firstly, this disclosure provides a data acquisition method based on semantic analysis, including:

[0007] Obtain historical multi-source datasets and real-time multi-source datasets, perform semantic analysis on the real-time multi-source datasets, and extract semantic contradictions from the real-time multi-source datasets;

[0008] Based on the semantic contradiction, a contradictory dual is generated, which includes a first semantic endmember and a second semantic endmember that conflict with each other.

[0009] Semantic fingerprints of the first semantic endmember and the second semantic endmember are extracted respectively, and a semantic evolution prediction model is used to infer the first evolution path and the second evolution path corresponding to the first semantic endmember and the second semantic endmember based on the semantic fingerprints. The semantic fingerprints include entity attribute vectors, context feature vectors, source credibility weights and assertion structure signatures.

[0010] Along the first evolution path and the second evolution path, historical semantic data corresponding to the first semantic endmember and the second semantic endmember are extracted from the historical multi-source dataset, and the semantic inheritance degree between the historical semantic data and the corresponding semantic endmember is calculated.

[0011] Based on the semantic inheritance degree, the continuity of the first evolution path and the second evolution path is determined. In response to the fact that both the first evolution path and the second evolution path remain continuous, the semantic consistency value of the historical semantic data on the first evolution path and the second evolution path is calculated.

[0012] In response to a semantic consistency value falling below a consistency threshold, the first inconsistent node appearing in the historical multi-source dataset is identified as a source of conflicting forks.

[0013] Contextual data of the contradictory bifurcation source is collected, and a contradictory attribution dataset is generated based on the contextual data. The contradictory attribution dataset is used to characterize the causal path of semantic evolution from a consistent state to a contradictory state.

[0014] Furthermore, the generation of contradictory duals based on the semantic contradiction includes:

[0015] Semantic triples are extracted from the structured and / or unstructured data in the multi-source dataset to obtain at least one semantic assertion;

[0016] Perform a logical consistency check on the semantic assertions to obtain mutually exclusive semantic assertions;

[0017] Based on the mutually exclusive semantic assertion, the first semantic terminator and the second semantic terminator are determined, and the contradictory duality is constructed.

[0018] Further, the step of extracting historical semantic data corresponding to the first semantic endmember and the second semantic endmember from the historical multi-source dataset along the first evolutionary path and the second evolutionary path, and calculating the semantic inheritance degree between the historical semantic data and the corresponding semantic endmember, includes:

[0019] For the first evolutionary path and the second evolutionary path, the historical time step interval is extracted from the historical multi-source dataset to obtain the candidate dataset;

[0020] Each candidate data in the candidate dataset is matched with the first semantic terminus and the second semantic terminus for semantic feature distribution, and the candidate data with a matching degree exceeding the preset inheritance threshold is selected as the historical semantic data of the corresponding historical time step.

[0021] Calculate the semantic vector similarity between the historical semantic data of the historical time step and the historical semantic data of adjacent historical time steps, and use it as the semantic inheritance degree;

[0022] In response to the failure to collect the historical semantic data for a consecutive preset number of historical time steps, the corresponding evolution path is determined to be interrupted and the first historical time step in which the historical semantic data cannot be collected is recorded as the interruption point.

[0023] Further, the step of determining the continuity of the first evolutionary path and the second evolutionary path based on the semantic inheritance degree, and calculating the semantic consistency value of historical semantic data on the first evolutionary path and the second evolutionary path in response to both the first evolutionary path and the second evolutionary path remaining continuous, includes:

[0024] Confirm whether the first evolution path and the second evolution path have recorded the interruption point. If there is no interruption point, it is determined that the first evolution path and the second evolution path are both continuous.

[0025] In response to the fact that both the first evolutionary path and the second evolutionary path remain continuous, for each historical time step, the first historical semantic data on the first evolutionary path and the second historical semantic data on the second evolutionary path are extracted;

[0026] Extract the first semantic assertion and the second semantic assertion from the first historical semantic data and the second historical semantic data, respectively;

[0027] Determine whether there is a mutual exclusion relationship between the first semantic assertion and the second semantic assertion. The mutual exclusion relationship includes attribute value conflict, temporal conflict, spatial conflict or causal chain conflict.

[0028] If the first semantic assertion and the second semantic assertion have the mutual exclusion relationship, then the historical time step is marked as the inconsistency node;

[0029] The percentage of inconsistent nodes in the historical time step is statistically analyzed to obtain the semantic consistency value of historical semantic data on the first evolution path and the second evolution path.

[0030] Further, the step of collecting contextual data of the contradictory bifurcation source and generating a contradictory attribution dataset based on the contextual data includes:

[0031] The historical time step corresponding to the source of the contradiction is determined as the central node. With the central node as the center, a preset number of historical time steps are collected forward and backward to form a context collection window.

[0032] Within the context acquisition window, data associated with the first semantic terminus and the second semantic terminus is extracted. The associated data includes at least one of data metadata, data modification records, and data source identifiers.

[0033] Based on the associated data, the semantic change events of the context acquisition window are determined, thereby generating the contradictory attribution dataset.

[0034] Furthermore, the method also includes:

[0035] Identify the causal triggering conditions of the semantic change event, wherein the causal triggering conditions include at least one of the following: credibility decay of a specific data source, assertion ambiguity in a specific context, and attribute value overwriting caused by data modification operations;

[0036] Based on the causal triggering conditions, adjust the collection frequency, weight allocation, and / or verification strength of the real-time multi-source dataset.

[0037] Secondly, this disclosure provides a data acquisition device based on semantic analysis, including:

[0038] The acquisition module is configured to acquire historical multi-source datasets and real-time multi-source datasets, perform semantic analysis on the real-time multi-source datasets, and extract semantic contradictions from the real-time multi-source datasets.

[0039] The end-member generation module is configured to generate a contradictory dual body based on the semantic contradiction, wherein the contradictory dual body includes a first semantic end-member and a second semantic end-member that conflict with each other.

[0040] The path estimation module is configured to extract the semantic fingerprints of the first semantic endmember and the second semantic endmember respectively, and use the semantic evolution prediction model to estimate the first evolution path and the second evolution path corresponding to the first semantic endmember and the second semantic endmember based on the semantic fingerprints. The semantic fingerprints include entity attribute vectors, context feature vectors, source credibility weights and assertion structure signatures.

[0041] The calculation module is configured to extract historical semantic data corresponding to the first semantic endmember and the second semantic endmember from the historical multi-source dataset along the first evolution path and the second evolution path, and to calculate the semantic inheritance degree between the historical semantic data and the corresponding semantic endmember.

[0042] The judgment module is configured to determine the continuity of the first evolution path and the second evolution path based on the semantic inheritance degree, and in response to the fact that both the first evolution path and the second evolution path remain continuous, calculate the semantic consistency value of the historical semantic data on the first evolution path and the second evolution path.

[0043] The comparison module is configured to identify the first inconsistent node in the historical multi-source dataset as a conflict fork source in response to a semantic consistency value being lower than a consistency threshold.

[0044] The generation module is configured to collect context data of the contradictory bifurcation source and generate a contradictory attribution dataset based on the context data. The contradictory attribution dataset is used to characterize the causal path of semantic evolution from a consistent state to a contradictory state.

[0045] Thirdly, this disclosure provides a storage medium, including:

[0046] One or more processors;

[0047] Storage device, on which one or more programs are stored,

[0048] When the above-described one or more programs are executed by the above-described one or more processors, the above-described one or more processors implement the method as described in any embodiment of the first aspect of this disclosure.

[0049] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any embodiment of the first aspect of this disclosure.

[0050] Fifthly, this disclosure provides a computer program product including a computer program / instructions that, when executed by a processor, implement the method described in any embodiment of the first aspect of this disclosure.

[0051] Compared with existing technologies, the beneficial effects of this invention are that by constructing a contradiction duality based on the semantic contradictions of real-time multi-source data, and using a semantic evolution prediction model to trace its evolution path and inheritance relationship in historical data, the source of contradiction bifurcation is finally located and a contradiction attribution dataset is generated. This realizes the automated tracing and attribution of data semantic contradictions, transforming the traditional passive detection of data quality after the fact into an active risk interception and mechanism optimization driven by semantic consistency. Without significantly increasing the investment in data governance, it effectively curbs the bias in analysis and decision-making caused by cross-source semantic conflicts, and significantly improves the data credibility and intelligence level of the multi-source data fusion system. Attached Figure Description

[0052] Figure 1 is a system architecture diagram of one embodiment of the data acquisition method based on semantic analysis according to the present disclosure;

[0053] Figure 1A This is a flowchart of an embodiment of the semantic analysis-based data acquisition method disclosed herein;

[0054] Figure 1B This is an exploded flowchart of one embodiment of step 102 of this disclosure;

[0055] Figure 1C This is an exploded flowchart of one embodiment of step 104 of this disclosure;

[0056] Figure 1D This is an exploded flowchart of an embodiment of step 105 of this disclosure;

[0057] Figure 2 This is a schematic diagram of a structure of an embodiment of a semantic analysis-based data acquisition device according to the present disclosure. Detailed Implementation

[0058] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0059] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0060] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0061] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0062] Please refer to Figure 1A , Figure 1AA flow 100 of an embodiment of a semantic analysis-based data acquisition method according to the present disclosure is shown. The flow 100 includes the following steps:

[0063] Step 101: Obtain historical multi-source datasets and real-time multi-source datasets, perform semantic analysis on the real-time multi-source datasets, and extract semantic contradictions from the real-time multi-source datasets.

[0064] In this embodiment, the execution entity of the semantic analysis-based data acquisition method first acquires two types of datasets. Historical multi-source datasets are collections of multi-source data stored in a data warehouse or data lake, covering a long past period; they serve as the foundational corpus for contradiction source tracing analysis. Real-time multi-source datasets are snapshots of current data accessed in real-time from various data sources through a data stream processing engine; they are the inputs that trigger contradiction detection. Semantic analysis refers to the process of understanding and formalizing the deeper meanings beyond the structural definitions in a dataset using natural language processing and information extraction techniques. Semantic contradictions refer to patterns identified from real-time multi-source datasets where two or more data items exhibit logically irreconcilable conflicts at the semantic level; for example, the "status" field of the same order might be "received" in the transaction system and "in transit" in the logistics system.

[0065] By acquiring both historical and real-time multi-source datasets and performing semantic analysis on real-time data to extract semantic contradictions, the system achieves instant and automated perception of semantic conflicts across data sources, thereby shifting the focus of data quality monitoring from single-point compliance to global semantic consistency.

[0066] Step 102: Generate contradictory duals based on semantic contradictions.

[0067] Here, the contradictory duality contains conflicting first and second semantic endmembers.

[0068] In some alternative implementations, step 102 may include, for example: Figure 1B The following steps 1021 to 1023 are shown:

[0069] Step 1021: Extract semantic triples from the structured and / or unstructured data in the multi-source dataset to obtain at least one semantic assertion.

[0070] In this embodiment, structured data refers to row and column data stored in a relational database with a predefined schema. Unstructured data refers to information such as free text, logs, and reports that do not have a predefined data model. Semantic triple extraction refers to the process of identifying and extracting the smallest knowledge units with a structure of "entity-attribute-value" or "entity-relationship-entity" from data. Semantic assertion refers to a triple that expresses a clear statement of fact after extraction and formal representation.

[0071] By extracting semantic triples from structured and unstructured data in a unified manner, data of different forms are normalized into a unified set of machine-understandable semantic assertions, thus achieving semantic alignment of cross-modal data and providing a standardized processing object for subsequent logical consistency verification.

[0072] Step 1022: Perform logical consistency verification on the semantic assertions to obtain mutually exclusive semantic assertions.

[0073] In this embodiment, logical consistency verification refers to the process of automatically reasoning and verifying the extracted set of semantic assertions based on a preset knowledge ontology (such as a domain ontology) and axiomatic rules (such as "an entity cannot have two mutually exclusive states at the same time"). Mutually exclusive semantic assertions refer to two or more semantic assertions that are judged by the logical reasoning engine to violate consistency rules during the verification process, and they jointly assert a logically impossible state.

[0074] By performing logical consistency checks on semantic assertions based on knowledge ontology and axiomatic rules, the system accurately identifies mutually exclusive assertion pairs from a massive amount of semantic assertions. This achieves automated judgment from formal knowledge to logical conflicts, thereby eliminating interference from non-critical differences and accurately pinpointing the core of the true semantic contradiction.

[0075] Step 1023: Determine the first semantic endmember and the second semantic endmember based on mutually exclusive semantic assertions, and construct the contradictory dual body.

[0076] In this embodiment, the first semantic endmember and the second semantic endmember refer to the semantic content context anchors representing the contradictory sides, extracted from mutually exclusive semantic assertion pairs. Each endmember not only contains the attribute values ​​constituting the conflict but also encapsulates its corresponding entity, timestamp, source, and other complete context. The contradictory duality is thus a complete data structure composed of the first semantic endmember and the second semantic endmember.

[0077] By identifying and encapsulating the two contradictory parties as the first and second semantic end units from mutually exclusive semantic assertions to construct contradictory duality, scattered logical conflicts are organized into contradictory behavior units containing complete context, realizing the structuring and context preservation of contradictory information, thus providing a precise and self-contained analytical starting point for subsequent path tracing and fingerprint extraction.

[0078] Step 103: Extract the semantic fingerprints of the first semantic endmember and the second semantic endmember respectively, and use the semantic evolution prediction model to infer the corresponding first evolution path and second evolution path based on the semantic fingerprints for the first semantic endmember and the second semantic endmember.

[0079] Here, semantic fingerprints refer to compact numerical vectors or data structures generated for each semantic endmember through embedding models and feature engineering, uniquely and robustly representing its core semantics and contextual information. Semantic fingerprints include entity attribute vectors, contextual feature vectors, source credibility weights, and assertion structure signatures. Specifically, the entity attribute vector is a dense vector encoded by the key attribute values ​​of the core entity, representing "what the object is"; the contextual feature vector is a vector capturing the time, interaction background, and other contextual information when the semantic endmember was generated, representing "under what circumstances it was said"; the source credibility weight is a scalar value representing the historical reliability score of the data source that generated the semantic endmember, representing "how credible the source is"; and the assertion structure signature is a hash value or feature code describing the logical structure of the assertion itself (such as a typed signature of an "entity-attribute-value" triple), representing "what the skeleton structure of the statement is." The semantic evolution prediction model is a generative model trained on graph neural networks or time-series point processes, internally encoding common paradigms of knowledge evolution across cross-source data. The first and second evolution paths refer to the routing sequences that the model predicts for each end-member in the historical data knowledge graph, using semantic fingerprints as input, and which are most likely to evolve backward in time for that semantic state. They are represented as chains of historical data nodes arranged in chronological order.

[0080] By extracting semantic fingerprints containing multi-dimensional information such as entity attribute vectors and source credibility, and using semantic evolution prediction models to infer evolution paths, reverse routing from immediate points of conflict to historical evolution is achieved. This transforms isolated conflicts into time-series tracing chains, providing efficient and intelligent navigation for accurately retrieving relevant evidence from vast historical data.

[0081] Step 104: Along the first evolution path and the second evolution path, extract historical semantic data corresponding to the first semantic endmember and the second semantic endmember from the historical multi-source dataset, and calculate the semantic inheritance degree between the historical semantic data and the corresponding semantic endmember.

[0082] Here, semantic inheritance is used to quantify the smoothness or abruptness of semantic changes between historical semantic data at adjacent time steps. A high value indicates natural semantic inheritance, while a low value indicates drastic semantic changes.

[0083] In some alternative implementations, step 104 may include, for example: Figure 1C The following steps 1041 to 1044 are shown:

[0084] Step 1041: For the first evolutionary path and the second evolutionary path, extract the candidate dataset from the historical multi-source dataset at historical time steps to obtain the candidate dataset.

[0085] In this embodiment, the historical time step refers to the predefined time granularity used when sampling historical data, such as "day," "week," or "hour," and its setting depends on the frequency of change of business data. The alternative dataset refers to a subset of all data snapshots retrieved from historical multi-source datasets within a time interval corresponding to a specific historical time step.

[0086] By sampling the historical dataset along the predicted evolution path at a fixed historical time step, the continuous evolution process is discretized into candidate datasets for subsequent matching. This achieves standardized and operable sampling of the infinite historical data stream, thereby effectively reducing the computational complexity of subsequent semantic matching.

[0087] Step 1042: Perform semantic feature distribution matching between each candidate data in the candidate dataset and the first semantic endmember and the second semantic endmember, and select the candidate data with a matching degree exceeding the preset inheritance threshold as the historical semantic data of the corresponding historical time step.

[0088] In this embodiment, semantic feature distribution matching refers to calculating the distribution similarity measure, such as cosine similarity or bulldozer distance, between the semantic fingerprint of candidate data and the semantic fingerprint of the target endmember, to assess the degree of similarity between the two in core semantics and context. The preset inheritance threshold is an empirically determined similarity threshold; only when the matching degree exceeds this threshold can the candidate data be considered a valid continuation of the endmember semantics at that historical node. Historical semantic data refers to representative data records that successfully pass the threshold screening and are confirmed to belong to a historical state on this evolutionary path.

[0089] By matching the semantic feature distribution of candidate data with semantic endmembers and filtering historical semantic data with an inheritance threshold, records with real semantic inheritance relationships can be accurately identified from a large number of candidate data. This ensures that each node on the constructed evolutionary path has a solid semantic association and avoids the contamination of evolutionary analysis by noisy data.

[0090] Step 1043: Calculate the semantic vector similarity between historical semantic data at a historical time step and historical semantic data at adjacent historical time steps, as the semantic inheritance degree.

[0091] In this embodiment, semantic vector similarity refers to the cosine similarity value calculated after converting historical semantic data at two adjacent time steps into sentence vectors or graph vectors through a semantic encoder. This value is directly used as a quantitative indicator of semantic inheritance.

[0092] By calculating the semantic vector similarity of adjacent historical semantic data and using it as the semantic inheritance degree, the continuity and abruptness of semantic evolution along the timeline are accurately quantified, thus providing a numerical basis for subsequent judgment on whether the path is interrupted or whether the semantics have undergone a step change.

[0093] Step 1044: In response to the failure to collect historical semantic data for a consecutive preset number of historical time steps, determine that the corresponding evolution path is interrupted and record the first historical time step in which historical semantic data cannot be collected as the interruption point.

[0094] In this embodiment, the consecutive preset number is a tolerance parameter (e.g., 3 steps) used to prevent misjudgment of path interruption due to temporary data loss. An interruption refers to a state in which an evolutionary path cannot be traced due to data loss or excessive semantic jumps. The first historical time step in which historical semantic data cannot be collected is accurately recorded as the interruption point, which is a key marker for subsequent determination of evolutionary boundaries.

[0095] By setting tolerance and continuously detecting missing historical semantic data, path interruptions are determined and interruption points are recorded, thus achieving automatic definition of the limits of evolutionary path tracing. This avoids invalid calculations of blank data areas and accurately marks the start and end points of the complete evolutionary chain that can be used for subsequent analysis.

[0096] Step 105: Based on semantic inheritance, determine the continuity of the first evolution path and the second evolution path. In response to the fact that both the first evolution path and the second evolution path remain continuous, calculate the semantic consistency value of the historical semantic data on the first evolution path and the second evolution path.

[0097] In this embodiment, the aforementioned execution entity first confirms whether there are no interruptions in either evolutionary path, which is a prerequisite for subsequent comparison. Only when both sides have completed backtracking can their corresponding nodes at different historical time steps be cross-compared to discover when the contradiction began. The semantic consistency value is a global indicator used to measure the duration or proportion of nodes on the two evolutionary paths that remain in a conflict-free state throughout the entire traceable historical period.

[0098] In some alternative implementations, step 105 may include, for example: Figure 1D The following steps 1051 to 1056 are shown:

[0099] Step 1051: Confirm whether there are any interruption points recorded in the first evolution path and the second evolution path. If there are no interruption points, it is determined that both the first evolution path and the second evolution path are continuous.

[0100] By confirming that there are no interruptions in the two evolutionary paths to determine their continuity, it is ensured that subsequent consistency calculations are performed only within a complete and traceable historical window, thereby eliminating analytical biases caused by incomplete data.

[0101] Step 1052: In response to the fact that both the first evolutionary path and the second evolutionary path remain continuous, for each historical time step, extract the first historical semantic data on the first evolutionary path and the second historical semantic data on the second evolutionary path.

[0102] In this embodiment, the first historical semantic data refers to the data node representing the historical state of the first semantic end-member extracted from the first evolution path at a specific historical time step. Similarly, the second historical semantic data represents the historical state of the second semantic end-member at the same time step.

[0103] By aligning and extracting the first and second historical semantic data at the same historical time step, a synchronous slice of the historical states of the conflicting parties is achieved, thus providing temporally comparable sample pairs for subsequent point-to-point semantic conflict comparison.

[0104] Step 1053: Extract the first semantic assertion and the second semantic assertion from the first historical semantic data and the second historical semantic data, respectively.

[0105] In this embodiment, the first semantic assertion refers to a triplet statement extracted from the first historical semantic data that can characterize its core semantics at that point in time. The second semantic assertion refers to a triplet statement extracted from the second historical semantic data that can characterize its core semantics at that point in time.

[0106] By refining the granularity of comparison from "data" to "semantic assertion", the focus of semantic conflict is further purified, so that subsequent mutual exclusion judgments can directly act on the knowledge ontology, rather than the vague data representation.

[0107] Step 1054: Determine whether there is a mutual exclusion relationship between the first semantic assertion and the second semantic assertion.

[0108] Here, the determination of mutual exclusion is based on the disjoint axiom in knowledge ontology. Mutual exclusion includes attribute value conflicts (such as the status of the same order cannot be both "signed" and "in transit"), temporal conflicts (such as the end time of an event being earlier than its start time), spatial conflicts (such as the current location of the same package being in two geographically disjoint warehouses at the same time), or causal chain conflicts [such as event A being the cause of event B (A occurring before B), but another description states that B is the cause of A (B occurring before A)].

[0109] By judging various types of mutual exclusion relationships based on the axioms of knowledge ontology, a comprehensive and multi-dimensional identification of semantic contradictions is achieved, so as not to miss any type of conflict such as attribute, tense, space or logic.

[0110] Step 1055: If the first semantic assertion and the second semantic assertion are mutually exclusive, then mark the historical time step as an inconsistent node.

[0111] Here, an inconsistency node refers to a node that is precisely marked on the historical timeline, at which point the semantic assertions of the contradictory parties begin to show a mutually exclusive relationship.

[0112] By marking time steps with mutual exclusion relationships as inconsistent nodes, the discretization of conflict triggering moments on a continuous evolution path is achieved, thus transforming the consistency judgment problem of long sequences into a statistical node state identification problem.

[0113] Step 1056: Calculate the proportion of inconsistent nodes in the historical time step to obtain the semantic consistency value of historical semantic data on the first evolution path and the second evolution path.

[0114] In this embodiment, the semantic consistency value ranges from 0 to 1. The higher the value, the longer the data of the two paths have remained consistent within the historical backtracking period.

[0115] By calculating the semantic consistency value by statistically analyzing the proportion of inconsistent nodes, the overall degree of conflict between two sequences is objectively quantified, thus providing a clear and adjustable numerical threshold for determining whether attribution analysis is necessary.

[0116] Step 106: In response to the semantic consistency value being lower than the consistency threshold, the first inconsistent node in the historical multi-source dataset is identified as the source of the conflict fork.

[0117] In this embodiment, the consistency threshold is a preset critical value based on the strictness of the business scenario, such as 0.9, indicating that when only 10% or less of the historical nodes are inconsistent, the system considers the contradiction to be a recent, sporadic occurrence. The contradiction bifurcation source refers to the historical time step and its corresponding dataset that is first detected as semantically mutually exclusive on two originally consistent evolutionary paths. It is considered the origin of the contradiction from nothing to something and is the core object of attribution analysis.

[0118] By identifying the first inconsistent node in a sequence below the consistency threshold as the source of the conflict, the system enables automatic tracing and precise location of the "first scene" of the conflict, thereby narrowing the vast range of historical data into a specific time slice and data set, creating a prerequisite for efficient and accurate attribution analysis.

[0119] Step 107: Collect contextual data of conflicting bifurcation sources and generate a conflict attribution dataset based on the contextual data.

[0120] In this embodiment, contextual data refers to relevant operations and metadata within a time window surrounding the point in time before and after the point in time where the contradiction bifurcation originates. The contradiction attribution dataset is a structured collection of evidence used to explain "why a contradiction arose at this moment," with the goal of representing the causal path of semantic evolution from a consistent state to a contradictory state.

[0121] In some alternative implementations, step 107 may include steps one through three:

[0122] Step 1: Determine the historical time step corresponding to the source of the conflict as the central node. Using the central node as the center, collect a preset number of historical time steps forward and backward to form a context collection window.

[0123] By expanding the collection of data forward and backward from the source of the conflict to form a contextual collection window, a complete capture of the causal chain before and after the outbreak of the conflict is achieved, thus avoiding the loss of attribution clues caused by observing only a single moment.

[0124] Step 2: In the context acquisition window, extract the associated data with the first semantic endmember and the second semantic endmember.

[0125] In this embodiment, associated data refers to all data that shares the same entity identifier or relationship identifier with the two semantic endpoints that constitute a conflict. This includes not only the conflicting attributes but also all operations involving these entities within the specified time period. Associated data includes at least one of data metadata, data modification records, and data source identifiers. Data metadata refers to information such as the data's creation time, creator, and last modification time. Data modification records refer to audit logs of data value updates, insertions, and deletions. Data source identifiers are unique identifiers of the data source system from which each piece of data originates.

[0126] By extracting multidimensional related data containing metadata, modification records, and source identifiers within the context window, a complete information profile of the moment of conflict bifurcation is constructed, thereby enabling attribution analysis to be conducted based on an evidence space that is informative and comprehensive.

[0127] Step 3: Determine the semantic change events of the context collection window based on the associated data, thereby generating a conflict attribution dataset.

[0128] In this embodiment, semantic change events refer to atomic operations or source switching behaviors identified from associated data that may lead to changes in semantic assertions, such as "data updates from system B overwrite attribute values ​​from system A" or "data source S is marked as downgraded around this point in time." Arranging these events in chronological order forms a causal event chain from consistency to contradiction, i.e., a contradictory attribution dataset.

[0129] By identifying and serializing semantic change events from associated data to generate a contradictory attribution dataset, fragmented pieces of evidence are strung together into a logically coherent causal storyline, achieving a cognitive leap from "what happened" to "why it happened," and providing an interpretable and well-evidenced causal path for understanding the root causes of contradictions.

[0130] After performing step 107, you can continue with steps 108 and 109:

[0131] Step 108: Identify the causal triggering conditions of semantic change events.

[0132] In this embodiment, causal triggering conditions refer to the fundamental patterns and conditions that lead to the high-frequency occurrence of semantic change events, summarized from multiple contradictory attribution datasets. Causal triggering conditions include at least one of the following: credibility decay of a specific data source (e.g., the data quality provided by a data source systematically declines after a certain point in time due to maintenance errors), assertion ambiguity in a specific context (e.g., during a promotional activity, the definition of "order amount" is ambiguous when calculating the final price and the page price), and attribute value overwriting caused by data modification operations (e.g., a manual repair work order incorrectly overwrites the original data automatically collected by the system).

[0133] By summarizing specific semantic change events into abstract, reproducible causal triggering conditions, a shift from case-specific attribution to pattern discovery is achieved, enabling solutions to resolve current conflicts to be used to prevent similar large-scale problems from occurring in the future.

[0134] Step 109: Adjust the collection frequency, weight allocation, and / or verification strength of the real-time multi-source dataset according to the causal triggering conditions.

[0135] In this embodiment, adjusting the collection frequency refers to reducing the data retrieval frequency for a data source marked as having diminishing credibility. Adjusting the weight allocation refers to dynamically reducing the weight of data from contexts prone to assertion ambiguity in the data fusion model. Adjusting the verification strength refers to adding a stricter business logic verification rule at the upstream write interface for data fields that have experienced overwriting errors.

[0136] By dynamically adjusting the collection frequency, weight allocation, and verification intensity of real-time data based on causal triggering conditions, a closed-loop feedback mechanism for data quality governance is achieved, moving from passive detection to proactive immunity. This allows for precise policy implementation at the source of data collection, preventing the recurrence of similar semantic contradictions and ultimately improving the robustness and credibility of the multi-source data system.

[0137] The semantic analysis-based data acquisition method provided in the above embodiments of this disclosure constructs a contradiction duality for semantic contradictions in real-time multi-source data, and uses a semantic evolution prediction model to trace its evolution path and inheritance relationship in historical data. Finally, it locates the source of contradiction bifurcation and generates a contradiction attribution dataset, realizing automated source tracing and attribution of data semantic contradictions. It transforms the traditional passive detection of data quality after the fact into an active risk interception and mechanism optimization driven by semantic consistency. Without significantly increasing the investment in data governance, it effectively curbs the bias in analysis and decision-making caused by cross-source semantic conflicts, and significantly improves the data credibility and intelligence level of the multi-source data fusion system.

[0138] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a data acquisition device based on semantic analysis, which is similar to... Figure 1A Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0139] like Figure 2As shown, the semantic analysis-based data acquisition device 200 of this embodiment includes: an acquisition module 201, an endmember generation module 202, a path estimation module 203, a calculation module 204, a judgment module 205, a comparison module 206, and a generation module 207. The acquisition module 201 is configured to acquire historical multi-source datasets and real-time multi-source datasets, perform semantic analysis on the real-time multi-source datasets, and extract semantic contradictions from the real-time multi-source datasets. The endmember generation module 202 is configured to generate contradictory pairs based on semantic contradictions, the contradictory pairs containing mutually conflicting first and second semantic endmembers. The path estimation module 203 is configured to extract semantic fingerprints from the first and second semantic endmembers respectively, and use a semantic evolution prediction model to estimate the corresponding first and second evolution paths based on the semantic fingerprints. The semantic fingerprints include entity attribute vectors, context feature vectors, source credibility weights, and assertion structure signatures. The calculation module 204 is configured to calculate the semantic fingerprints along the first and second evolution paths in the historical multi-source datasets. The system extracts historical semantic data corresponding to the first and second semantic endmembers and calculates the semantic inheritance degree between the historical semantic data and the corresponding semantic endmembers. The judgment module 205 is configured to judge the continuity of the first evolution path and the second evolution path based on the semantic inheritance degree. In response to the continuity of both the first and second evolution paths, the system calculates the semantic consistency value of the historical semantic data on the first and second evolution paths. The comparison module 206 is configured to determine the first inconsistent node in the historical multi-source dataset as the source of the contradiction bifurcation in response to the semantic consistency value being lower than the consistency threshold. The generation module 207 is configured to collect the context data of the source of the contradiction bifurcation and generate a contradiction attribution dataset based on the context data. The contradiction attribution dataset is used to characterize the causal path of semantic evolution from a consistent state to a contradictory state.

[0140] It should be noted that the implementation details and technical effects of each module and unit in the semantic analysis-based data acquisition device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0141] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A data acquisition method based on semantic analysis, characterized in that, include: Obtain historical multi-source datasets and real-time multi-source datasets, perform semantic analysis on the real-time multi-source datasets, and extract semantic contradictions from the real-time multi-source datasets; Based on the semantic contradiction, a contradictory dual is generated, which includes a first semantic endmember and a second semantic endmember that conflict with each other. Semantic fingerprints of the first semantic endmember and the second semantic endmember are extracted respectively, and a semantic evolution prediction model is used to infer the first evolution path and the second evolution path corresponding to the first semantic endmember and the second semantic endmember based on the semantic fingerprints. The semantic fingerprints include entity attribute vectors, context feature vectors, source credibility weights and assertion structure signatures. Along the first evolution path and the second evolution path, historical semantic data corresponding to the first semantic endmember and the second semantic endmember are extracted from the historical multi-source dataset, and the semantic inheritance degree between the historical semantic data and the corresponding semantic endmember is calculated. Based on the semantic inheritance degree, the continuity of the first evolution path and the second evolution path is determined. In response to the fact that both the first evolution path and the second evolution path remain continuous, the semantic consistency value of the historical semantic data on the first evolution path and the second evolution path is calculated. In response to a semantic consistency value falling below a consistency threshold, the first inconsistent node appearing in the historical multi-source dataset is identified as a source of conflicting forks. Contextual data of the contradictory bifurcation source is collected, and a contradictory attribution dataset is generated based on the contextual data. The contradictory attribution dataset is used to characterize the causal path of semantic evolution from a consistent state to a contradictory state.

2. The method according to claim 1, characterized in that, The generation of contradictory duals based on the semantic contradiction includes: Semantic triples are extracted from the structured and / or unstructured data in the multi-source dataset to obtain at least one semantic assertion; Perform a logical consistency check on the semantic assertions to obtain mutually exclusive semantic assertions; Based on the mutually exclusive semantic assertion, the first semantic terminator and the second semantic terminator are determined, and the contradictory duality is constructed.

3. The method according to claim 1, characterized in that, The step of extracting historical semantic data corresponding to the first and second semantic endmembers from the historical multi-source dataset along the first and second evolution paths, and calculating the semantic inheritance degree between the historical semantic data and the corresponding semantic endmembers, includes: For the first evolutionary path and the second evolutionary path, the historical time step interval is extracted from the historical multi-source dataset to obtain the candidate dataset; Each candidate data in the candidate dataset is matched with the first semantic terminus and the second semantic terminus for semantic feature distribution, and the candidate data with a matching degree exceeding the preset inheritance threshold is selected as the historical semantic data of the corresponding historical time step. Calculate the semantic vector similarity between the historical semantic data of the historical time step and the historical semantic data of adjacent historical time steps, and use it as the semantic inheritance degree; In response to the failure to collect the historical semantic data for a consecutive preset number of historical time steps, the corresponding evolution path is determined to be interrupted and the first historical time step in which the historical semantic data cannot be collected is recorded as the interruption point.

4. The method according to claim 3, characterized in that, The step of determining the continuity of the first evolutionary path and the second evolutionary path based on the semantic inheritance degree, and calculating the semantic consistency value of historical semantic data on the first evolutionary path and the second evolutionary path in response to both the first evolutionary path and the second evolutionary path remaining continuous, includes: Confirm whether the first evolution path and the second evolution path have recorded the interruption point. If there is no interruption point, it is determined that the first evolution path and the second evolution path are both continuous. In response to the fact that both the first evolutionary path and the second evolutionary path remain continuous, for each historical time step, the first historical semantic data on the first evolutionary path and the second historical semantic data on the second evolutionary path are extracted; Extract the first semantic assertion and the second semantic assertion from the first historical semantic data and the second historical semantic data, respectively; Determine whether there is a mutual exclusion relationship between the first semantic assertion and the second semantic assertion. The mutual exclusion relationship includes attribute value conflict, temporal conflict, spatial conflict or causal chain conflict. If the first semantic assertion and the second semantic assertion have the mutual exclusion relationship, then the historical time step is marked as the inconsistency node; The percentage of inconsistent nodes in the historical time step is statistically analyzed to obtain the semantic consistency value of historical semantic data on the first evolution path and the second evolution path.

5. The method according to claim 4, characterized in that, The process of collecting contextual data of the contradictory bifurcation source and generating a contradictory attribution dataset based on the contextual data includes: The historical time step corresponding to the source of the contradiction is determined as the central node. With the central node as the center, a preset number of historical time steps are collected forward and backward to form a context collection window. Within the context acquisition window, data associated with the first semantic terminus and the second semantic terminus is extracted. The associated data includes at least one of data metadata, data modification records, and data source identifiers. Based on the associated data, the semantic change events of the context acquisition window are determined, thereby generating the contradictory attribution dataset.

6. The method according to claim 5, characterized in that, The method further includes: Identify the causal triggering conditions of the semantic change event, wherein the causal triggering conditions include at least one of the following: credibility decay of a specific data source, assertion ambiguity in a specific context, and attribute value overwriting caused by data modification operations; Based on the causal triggering conditions, adjust the collection frequency, weight allocation, and / or verification strength of the real-time multi-source dataset.

7. A data acquisition device based on semantic analysis, characterized in that, include: The acquisition module is configured to acquire historical multi-source datasets and real-time multi-source datasets, perform semantic analysis on the real-time multi-source datasets, and extract semantic contradictions from the real-time multi-source datasets. The end-member generation module is configured to generate a contradictory dual body based on the semantic contradiction, wherein the contradictory dual body includes a first semantic end-member and a second semantic end-member that conflict with each other. The path estimation module is configured to extract the semantic fingerprints of the first semantic endmember and the second semantic endmember respectively, and use the semantic evolution prediction model to estimate the first evolution path and the second evolution path corresponding to the first semantic endmember and the second semantic endmember based on the semantic fingerprints. The semantic fingerprints include entity attribute vectors, context feature vectors, source credibility weights and assertion structure signatures. The calculation module is configured to extract historical semantic data corresponding to the first semantic endmember and the second semantic endmember from the historical multi-source dataset along the first evolution path and the second evolution path, and to calculate the semantic inheritance degree between the historical semantic data and the corresponding semantic endmember. The judgment module is configured to determine the continuity of the first evolution path and the second evolution path based on the semantic inheritance degree, and in response to the fact that both the first evolution path and the second evolution path remain continuous, calculate the semantic consistency value of the historical semantic data on the first evolution path and the second evolution path. The comparison module is configured to identify the first inconsistent node in the historical multi-source dataset as a conflict fork source in response to a semantic consistency value being lower than a consistency threshold. The generation module is configured to collect context data of the contradictory bifurcation source and generate a contradictory attribution dataset based on the context data. The contradictory attribution dataset is used to characterize the causal path of semantic evolution from a consistent state to a contradictory state.

8. A storage medium, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

Images