Big data analytics driven ai application system

Abstract

The application discloses an AI application system driven by big data analysis and relates to the technical field of big data and artificial intelligence, which is used for solving the problems of disconnection between search content and business state, poor index update consistency and difficult responsibility tracing in search enhancement generation architecture, and comprises a data access and incremental capture module, a joint index storage engine module, a bloodline perception fusion search module and an Agent access control and fuse tracing module; the joint index storage engine module physically binds vector data, subgraph adjacency data and bloodline metadata in the same index block; the fusion search module sorts according to the product score of semantic similarity, node center weight and bloodline confidence attenuation value; the access control module generates evidence reference identification and executes isolation processing when the score is lower than a threshold value, and can be applied to financial risk control, medical diagnosis and other scenes.

Description

Technical Field

[0001] This invention relates to the fields of big data and artificial intelligence technology, specifically to an AI application system driven by big data analysis. Background Technology

[0002] With the rapid development of large language models and generative artificial intelligence technologies, enterprise-level artificial intelligence applications are gradually evolving from single-point dialogue tools to artificial intelligence application platforms driven by big data analysis. The retrieval-enhanced generative technology architecture has been widely used in this evolution process. Its basic principle is as follows: when a user query request is received, the system first retrieves content fragments related to the query semantics from an external knowledge base, and then inputs the retrieved content fragments and the original query into the large language model. The large language model combines the retrieved context content to generate the final answer, thereby alleviating to some extent the problem of knowledge obsolescence or content inaccuracy caused by the large language model relying solely on internal parameter memory.

[0003] The underlying technologies supporting the operation of the retrieval enhancement generation architecture mainly include: a vector database based on the approximate nearest neighbor index algorithm, used for similarity retrieval of embedded vectors of unstructured data such as text and images; a relational database or big data storage platform for carrying structured business data of enterprises; a knowledge graph database for storing the topology of relationships between entities and assisting multi-hop reasoning; and an extraction, transformation and loading tool for synchronizing data between the above heterogeneous storage systems. In the traditional architecture, the above four types of underlying technologies are independently carried by different storage systems and are connected to each other through the calling logic of the application layer.

[0004] The closest existing technical solution to this invention is a retrieval enhancement generation architecture that combines a relational database or data lake with an independently deployed vector database through periodic data synchronization. This solution typically includes the following process: the business system writes structured data into the relational database or data lake; through batch data synchronization, the business data is extracted and processed by text slicing and encoding models to generate embedded vectors, which are then written into an independently deployed vector database; in scenarios involving relational reasoning, a knowledge graph is separately constructed using a graph database; graph updates also rely on periodic batch processing tasks; when a user query is received, the retrieval module performs similarity retrieval based on vector distance only in the vector database; if necessary, it separately queries the graph database to obtain relational information; the two results are combined at the application layer; the combined retrieval result is used as contextual prompts and input into a large language model along with the original query; the large language model generates the final answer; some solutions use post-processing rules or manual review to perform secondary verification of the generated content after the large language model outputs the results, in order to discover potential errors.

[0005] In the above-mentioned existing technical solutions, the business database, vector database, and graph database are physically independent and logically loosely coupled. They are asynchronously transferred to each other through batch data synchronization pipelines. During retrieval, they rely on multi-way query splicing at the application layer. Moreover, none of the storage systems carry data lineage and confidence labels, and the retrieval system cannot perceive the business timeliness and lineage credibility of the data.

[0006] The aforementioned existing technical solutions have the following technical defects: First, because existing information retrieval systems can only return content based on string or semantic similarity, they cannot determine whether the retrieved content is invalid or contradictory in terms of business logic. This leads to large language models blindly accepting retrieved content fragments and easily generating answers that do not conform to the objective business situation. Furthermore, due to the lack of data lineage tracing capabilities, it is difficult to trace responsibility after errors occur in the generated content. This problem is particularly prominent in high-value scenarios such as financial risk control and medical diagnosis. Second, because existing big data artificial intelligence platforms mostly adopt heterogeneous splicing solutions, data... Synchronization typically involves time delays and makes it difficult to guarantee the consistency of updates between knowledge graph relationships and vector semantics. When concurrent retrieval is involved in massive knowledge bases, the computational resource consumption of rebuilding the entire index is high, while the simple incremental append method leads to a decrease in retrieval accuracy. Thirdly, current database systems and file systems lack permission convergence and fine-grained access isolation mechanisms for autonomously running artificial intelligence agents. Once an erroneous decision occurs, it is difficult to quickly locate and block it at the data structure level, and it is also difficult to block low-confidence information in the early stages of content dissemination. Therefore, we propose a big data analytics-driven AI application system. Summary of the Invention

[0007] The purpose of this invention is to provide an AI application system driven by big data analytics.

[0008] To address the shortcomings of the existing technologies, the technical problem this invention aims to solve is that existing big data and artificial intelligence application platforms, when supporting large language models and AI agents for retrieval enhancement generation, suffer from several issues. These issues arise because the underlying data storage architecture uses a loosely coupled relational database or data lake and vector database model, leading to the retrieval system's inability to perceive the business timeliness and lineage reliability of the data. This results in interconnected technical problems such as a disconnect between the retrieved content and the actual business state of the data, structural contradictions in incremental updates of multi-source heterogeneous data and index consistency maintenance, and a lack of fine-grained accountability and permission-based circuit breaker mechanisms when AI agents access big data resources across domains.

[0009] To address the aforementioned technical issues, this invention provides a big data analytics-driven AI application system. This system comprises four parts: a data access and incremental capture module, a federated index storage engine module, a lineage-aware fusion retrieval module, and an Agent access control and circuit breaker tracing module. These four parts work together to form a complete closed-loop link from the data source to the AI-generated output.

[0010] The data access and incremental capture module is used to monitor the data change logs of multiple business data sources in real time, capture change events including add, modify, and delete operations, and generate fields to be encoded, graph node information to be updated, and lineage metadata based on the change events, so as to achieve second-level perception of changes in business data.

[0011] The composite index storage engine module is used to physically bind vector data, subgraph adjacency data, and lineage metadata within the same index storage unit to form a composite index block. The composite index block includes a vector array area, a subgraph adjacency table area, and a lineage metadata area. The vector array area is used to store the embedded vector numerical sequence of entities or document fragments. The subgraph adjacency table area is used to store the direct adjacent node identifiers, edge types, and edge weight lists of the entity in the knowledge graph. The lineage metadata area is used to store the data source identifier, the timestamp of the most recent business confirmation, the initial confidence level, the decay rate parameter, and the validity flag. This module changes the technical pattern of physically separating the storage of vector databases, graph databases, and relational databases from the bottom layer of the storage structure.

[0012] The lineage-aware fusion retrieval module is used to receive query requests, calculate the semantic similarity between the query vector and the candidate entity embedding vector, the node centrality weight of the candidate entity in the knowledge graph, and the confidence decay value of the lineage label of the candidate entity based on the joint index block, multiply the three to obtain the fusion retrieval score of the candidate entity, sort the candidate entities according to the fusion retrieval score, and output the candidate results.

[0013] The Agent access control and circuit breaker tracing module is used to verify the identity and permissions of the Agents accessing the federated index storage engine module, generate evidence reference identifiers for candidate results that pass the verification, record the binding relationship between the generated content and the evidence reference identifiers, and perform isolation processing on the candidate entity if the fusion retrieval score of the candidate entity is less than the preset circuit breaker threshold.

[0014] This invention also provides a big data analytics-driven AI application method corresponding to the above system. This method achieves incremental updates of composite index blocks through a change data capture micro-batch streaming synchronization mechanism: when a composite index block corresponding to a certain entity is detected to already exist, the lineage confidence decay timer of all directly associated adjacent nodes of that entity is reset to reflect the impact of the entity's state change on the confidence of surrounding associated nodes. When a composite index block corresponding to that entity is detected to not exist, a new index block insertion operation is performed. After the index is written, the compaction processing of the underlying storage engine and the incremental update of the vector graph index are asynchronously triggered, enabling the retrieval side to perceive the latest data status within a second-level time window, avoiding the resource overhead caused by a full index reconstruction.

[0015] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0016] 1. This invention physically binds vector data, graph topology data, and lineage tags within the same storage unit. The retrieval results naturally carry business timeliness information, which can suppress the phenomenon of large language models generating content that does not conform to objective facts from the source of retrieval. Compared with the existing loose splicing architecture, it reduces generation errors caused by the disconnect between retrieval content and business status. It is especially suitable for application scenarios with high requirements for content accuracy, such as financial risk control and medical diagnosis.

[0017] 2. This invention adopts a change data capture micro-batch streaming synchronization and incremental writing mechanism with a composite index structure, which can compress the data update delay to within a second-level time window, while avoiding the computational resource consumption caused by full index reconstruction. It solves the structural contradiction between incremental update consistency and retrieval delay in the prior art, and takes into account both retrieval accuracy and update timeliness.

[0018] 3. This invention natively embeds Agent behavior marking and evidence citation mechanisms at the data structure input and output level, enabling rapid backtracking of evidence between the retrieval and generation links. It can also automatically perform isolation processing in scenarios where the confidence of candidate entities is low. Compared with the processing method of post-rule filtering based on the output of large language models in the prior art, it can block the spread of low-confidence content earlier at the data structure level, thus improving the efficiency of responsibility tracing.

[0019] 4. In this invention, the vector data, subgraph adjacency data, and lineage metadata of the candidate entity are physically bound to the same composite index block. During retrieval, only one input-output access is needed to read the three types of information, avoiding the network input-output latency caused by the association query between multiple independent storage systems in the prior art, improving the overall retrieval efficiency, and reducing the complexity of system deployment and maintenance. Attached Figure Description

[0020] Fig. 1 This is a schematic diagram of the overall four-layer module architecture of the system in this embodiment of the invention;

[0021] Fig. 2 This is a schematic diagram of the data incremental update writing process to the composite index in an embodiment of the present invention;

[0022] Fig. 3 This is a schematic diagram of the entire process of query retrieval, circuit breaking, and AI generation in an embodiment of the present invention. Detailed Implementation

[0023] The following is in conjunction with the appendix Figs. 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0024] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The big data analysis-driven AI application system provided in this embodiment of the invention includes a data access and incremental capture module, a federated index storage engine module, a lineage-aware fusion retrieval module, and an Agent access control and circuit breaker tracing module. The four modules work together to form a complete closed-loop link from the data source to the artificial intelligence-generated output. The multi-source heterogeneous data sources connected to the system include business data sources such as relational databases, document systems, and log streams.

[0026] The data access and incremental capture module is equipped with a change data capture component, which is used to monitor the add, modify, and delete operation logs of the above data sources in real time, forming a change event stream. For each captured change event, the data access and incremental capture module parses the business field content it carries, determines whether the event involves unstructured fields such as text and images, and if the determination result is yes, the field content is sent to the multimodal encoding unit for processing to generate the corresponding embedding vector. At the same time, the data access and incremental capture module parses the relationship between the business entity involved in the change event and other known entities, generates or updates the adjacent node list and edge weight of the entity in the knowledge graph, forms a subgraph update fragment, and generates or updates the lineage metadata for the entity. The lineage metadata includes the data source identifier of this change, the change timestamp, the initial confidence level, and the validity flag. The initial confidence level is a value greater than 0 and less than or equal to 1, which can be preset according to the confidence level of the data source. The validity flag is set to the valid state when it is generated.

[0027] The composite index storage engine module is implemented based on a modified log structure merged tree key-value storage engine. Its core is to maintain composite triple index blocks. The composite triple index block includes three parallel load areas within the same index storage unit: a vector array area, used to store the embedded vector numerical sequence of the entity or document fragment; a subgraph adjacency list area, used to store the direct adjacent node identifiers, edge types, and edge weight lists of the node in the knowledge graph; and a lineage metadata area, used to store the data source identifier, the most recent business confirmation timestamp, the initial confidence level, the decay rate parameter, and the validity flag. The above three load areas are physically adjacent to each other in a nested rectangular layout within the composite triple index block, corresponding to the composite serialization format of the value field of the key-value structure in the underlying key-value storage engine. The storage primary key of this composite triple index block is composed of the entity identifier and the version number.

[0028] The specific construction method of the composite index storage engine module is as follows: it is expanded on the basis of the open-source key-value storage engine, and its value field load structure is upgraded from a single text load or vector load to a composite structure that includes vector arrays, subgraph adjacency lists, and lineage metadata. Only the serialization and deserialization protocols of the index storage unit need to be adjusted, without changing the overall architecture of the underlying storage engine, so that atomic-level read and write and consistency guarantees of the three types of information, namely vector data, graph topology data and lineage data, can be achieved.

[0029] The data access and federated index construction method's processing steps include: A change data capture component deployed in the access layer monitors change logs from various business data sources in real time, capturing operation events including additions, modifications, and deletions to form a change event stream. For each captured change event, the business field content it carries is parsed. If the event involves unstructured fields such as text or images, it is sent to a multimodal encoding unit for processing to generate corresponding embedding vectors. For the business entity involved in the change event, its relationships with other known entities are parsed, generating or updating the entity's adjacent node list and edge weights in the knowledge graph, forming a subgraph update fragment. Lineage metadata is generated or updated for the entity, recording the data source identifier, change timestamp, and initial confidence of this change. The degree and validity flag are used to serialize the above-generated embedded vector, subgraph update fragment and lineage metadata according to the aforementioned composite structure to form the update load of the joint triplet index block. It is determined whether the entity already has a historical index block in the joint index storage engine module. If the result is that it exists, an atomic overwrite is performed and the lineage confidence decay timer of all directly related adjacent nodes of the entity is reset to reflect the impact of the entity's state change on the confidence of surrounding related nodes. If the result is that it does not exist, a new index block insertion operation is performed. After the index is written, the compaction processing of the underlying log structure merging tree storage engine and the incremental update of the vector graph index are asynchronously triggered, so that the retrieval side can perceive the latest data status within a second-level time window.

[0030] When performing semantic retrieval, the bloodline-aware fusion retrieval module abandons the traditional distance metric method that only considers semantic relevance and constructs a joint metric function that integrates semantic similarity, node centrality weight and bloodline confidence decay factor.

[0031] Specifically, semantic similarity is calculated using the cosine similarity between the query vector and the candidate entity embedding vector, and its expression is:

[0032] ;

[0033] in, This represents the query vector obtained after encoding the user's query. Indicates candidate entities Embedded vector, and These represent the magnitudes of the query vector and the candidate entity embedding vector, respectively. Represents the query and candidate entities The semantic similarity score between them is a value that is greater than or equal to negative 1 and less than or equal to positive 1.

[0034] Node centrality weight is used to measure the structural importance of a candidate entity within its knowledge subgraph, and its expression is:

[0035] ;

[0036] in, Indicates candidate entities The corresponding node in the knowledge graph This represents the centrality score calculated based on the local subgraph where the entity resides. It can be obtained using degree centrality or PageRank-like algorithms. This represents the upper limit of the centrality score within the subgraph involved in the current query, used for normalizing the centrality score. This is a preset smoothing baseline weight coefficient, with values ​​ranging from 0 to 1, used to prevent the weights of isolated nodes from dropping to zero. Indicates candidate entities The corresponding node centrality weights;

[0037] The lineage confidence decay function is used to quantify the degree of confidence decay of a candidate entity's lineage label over time or due to business status failure. Its expression is:

[0038] ;

[0039] in, Represents a node Initial confidence level of kinship tagged This represents the preset decay rate constant, which can be configured in different domains according to the timeliness requirements of data in different business areas. For example, the decay rate constant for financial market data can be configured to be greater than that for historical archive data. This represents the time interval between the current query time and the timestamp of the most recent business confirmation for the lineage tag of this node. This represents the validity indicator factor, taking a value of 0 if the upstream business data corresponding to this node is marked as invalid, and a value of 1 if the upstream business data corresponding to this node is not marked as invalid. Represents a node At any moment The confidence level of blood ties decreased.

[0040] This invention multiplies and fuses the above three factors to obtain a fusion retrieval score for candidate entities. Existing multi-factor weighted ranking techniques are limited by the physically separated storage architecture of vector and graph libraries, making it impossible to synchronously detect lineage failure status and atomically execute isolation during retrieval. It is generally believed in the art that retrieval ranking and data governance should belong to different processing stages. This invention, through the physical binding architecture of the joint index block, enables the fusion calculation of semantic similarity, node centrality weight, and lineage confidence decay value, as well as the circuit breaker isolation determination, to be completed atomically within the same index read. This overcomes the above-mentioned technical bias and produces a non-obvious technical effect of governance at the retrieval source, the expression of which is:

[0041] ;

[0042] in, Candidate entities For query The obtained fusion retrieval score is used by the bloodline perception fusion retrieval module to sort all candidate entities in descending order and extract a preset number of candidate entities at the top of the sort as the recall result.

[0043] The processing steps of the lineage-aware fusion retrieval method include: receiving user query requests, converting the query text into query vectors using an encoding model, performing initial recall in the vector index layer of the joint index storage engine module to obtain a candidate entity set, and for each candidate entity in the candidate entity set, simultaneously reading three types of information from its respective joint triplet index block: vector array, subgraph adjacency list, and lineage metadata. Since the three are physically bound to the same index block, this step only requires one input / output access, avoiding the network input / output latency caused by multi-database association queries in traditional architectures. Based on the aforementioned expressions, the semantic similarity, node centrality weight, and lineage of the candidate entity are calculated respectively. The confidence decay value is calculated by multiplying three factors to obtain the fusion retrieval score of the candidate entity. A preset circuit breaker threshold is compared with the fusion retrieval score. If the fusion retrieval score is not less than the circuit breaker threshold, the candidate entity is retained and included in the sorting queue. If the fusion retrieval score is less than the circuit breaker threshold, the candidate entity is isolated and does not participate in the subsequent sorting and generation process. The candidate entities after isolation are sorted in descending order according to the fusion retrieval score. A preset number of candidate entities at the top of the sort are selected as the retrieval results and returned to the upper-layer caller along with the original data source identifier, timestamp and other lineage information of each candidate entity.

[0044] The Agent access control and circuit breaker tracing module is used to solve the problem of lack of permission governance when artificial intelligence agents access big data resources across domains. It embeds the Agent behavior tag structure in the database access control layer, so that the retrieval link and the generation link naturally contain evidence references, and that each sentence fragment in the generated content can be mapped back to a row in the source structured form or a certain original document.

[0045] The agent access control and accountability mechanism's processing steps include: When an agent initiates a retrieval request, it carries its identity and pre-authorized data access domain scope. The agent access control and circuit breaker tracing module, based on a preset permission convergence strategy, verifies the data domain to which the federated triplet index blocks involved in the query belong, removing index blocks exceeding the authorized scope from the initial candidate entity recall results. For candidate entities that pass permission verification, the agent access control and circuit breaker tracing module generates a corresponding evidence reference identifier for each candidate entity while returning the results. This evidence reference identifier contains the original data source identifier, line number, or document offset information recorded in the lineage metadata area of ​​the federated triplet index block to which the candidate entity belongs. During the process of generating the response content, the large language model synchronously records... The system establishes an evidence citation chain by binding each sentence fragment in the generated content with the evidence citation identifiers it relies on. The system performs low-confidence patchwork detection on the generated content, determining whether all evidence citations relied upon by a particular sentence fragment meet the circuit breaker threshold requirements. If a sentence fragment in the generated content references evidence citations that have already been isolated, the system blocks the output of that sentence fragment at the data structure input / output level. Similarly, if a sentence fragment in the generated content lacks traceable evidence citation support, the system also blocks the output of that sentence fragment. The generated content that passes the circuit breaker check, along with its complete evidence citation chain, is returned to the user. Users or auditors can then use the evidence citation identifiers to locate and trace back to the original structured forms or document records, achieving traceability of responsibility.

[0046] As a specific application example, in a financial risk control business scenario, after a credit transaction data is captured by the change data capture component, the federated index storage engine module generates a federated triplet index block for the entity corresponding to the transaction, recording its embedding vector, the subgraph adjacency relationship with the associated customer entity, and the timestamp and initial confidence level of this data confirmation. When the risk control AI agent initiates a relevant query, the lineage-aware fusion retrieval module calculates the fusion retrieval score for the candidate entity corresponding to the transaction. If the transaction data is subsequently marked as invalid by the upstream system, the validity indicator factor is set to 0, and the fusion retrieval score is correspondingly reduced or even judged to be less than the circuit breaker threshold. The Agent access control and circuit breaker tracing module then performs isolation processing on the candidate evidence to prevent the large language model from referencing invalidated credit data to generate risk control conclusions. At the same time, each generated risk control conclusion carries an evidence citation identifier, making it easy for auditors to trace back to the original business record.

[0047] In another embodiment of the present invention, the federated index storage engine module can also be implemented through a loosely coupled middleware layer: without changing the underlying storage engines of the existing independently deployed vector database and graph database, a unified middleware synchronization and query proxy service is built on top of the vector database and graph database. This middleware is responsible for accessing the vector database and graph database in parallel at the time of query, and performs joint calculation and circuit breaker determination processing of semantic similarity, node centrality weight and lineage confidence decay factor at the application layer. Although this implementation does not achieve a strong physical layer binding of vector data, graph topology data and lineage tags, the retrieval governance effect of the present invention can still be achieved through the logical binding of the middleware layer. It is suitable for application scenarios where there are already a large number of existing vector databases and graph databases deployed and it is inconvenient to modify the underlying storage engine in the short term.

[0048] In another embodiment of the present invention, the bloodline confidence decay value can be calculated not by a continuous exponential decay function, but by a number of preset confidence levels: multiple confidence levels are preset, and the business rule engine maps the bloodline tag confidence to the corresponding level according to the change event type of the data source, and a fixed weight coefficient is preset for each level to participate in the calculation of the fusion retrieval score. The computational complexity of this implementation is relatively low, and it is suitable for application scenarios that are sensitive to computational delay and have relatively clear and enumerable business timeliness rules. It can also realize the downweighting or isolation of invalid data, and can serve as an effective supplement to the aforementioned continuous decay function implementation.

[0049] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A big data analytics-driven AI application system, characterized in that, include: The data access and incremental capture module is used to monitor the data change logs of multiple business data sources in real time, capture change events containing add, modify, and delete operations, and generate fields to be encoded, graph node information to be updated, and lineage metadata based on the change events. The composite index storage engine module is used to physically bind vector data, subgraph adjacency data, and lineage metadata within the same index storage unit to form a composite index block. The composite index block includes a vector array area, a subgraph adjacency table area, and a lineage metadata area. The vector array area is used to store the embedding vector of an entity. The subgraph adjacency table area is used to store the adjacent node identifier, edge type, and edge weight of the entity in the knowledge graph. The lineage metadata area is used to store the data source identifier, business confirmation timestamp, initial confidence level, decay rate parameter, and validity flag. The lineage-aware fusion retrieval module is used to receive query requests, calculate the semantic similarity between the query vector and the candidate entity embedding vector, the node centrality weight of the candidate entity in the knowledge graph, and the confidence decay value of the lineage tag of the candidate entity based on the federated index block, multiply the semantic similarity, the node centrality weight, and the confidence decay value to obtain the fusion retrieval score of the candidate entity, sort the candidate entities according to the fusion retrieval score, and output the candidate results. The Agent access control and circuit breaker tracing module is used to verify the identity and permissions of the Agent accessing the federated index storage engine module, generate evidence reference identifiers for candidate results that pass the verification, record the binding relationship between the generated content and the evidence reference identifiers, and perform isolation processing on the candidate entity if the fusion retrieval score of the candidate entity is less than the preset circuit breaker threshold.

2. The big data analytics-driven AI application system according to claim 1, characterized in that: The composite index storage engine module is implemented based on the log structure merge tree key-value storage engine. The storage primary key of the composite index block is composed of the entity identifier and the version number. The composite index block corresponds to the composite serialization format of the value field of the key-value structure in the key-value storage engine.

3. The big data analytics-driven AI application system according to claim 1, characterized in that: The bloodline perception fusion retrieval module calculates the confidence decay value of the bloodline tag of the candidate entity using the following formula: ； in, Indicates candidate entities At the time of query The confidence decay value, Indicates candidate entities Initial confidence level of kinship tagged This represents the preset decay rate constant. Indicates the query time and candidate entities The time interval between the most recent business confirmation timestamp of the lineage tag Indicators of validity: in candidate entities If the corresponding upstream business data is marked as invalid, A value of 0 is used in candidate entities. If the corresponding upstream business data is not marked as invalid, The value is 1.

4. The big data analytics-driven AI application system according to claim 1, characterized in that: The node centrality weight It is calculated in the following way: ,in The smoothing baseline weighting coefficient, The centrality score of the local subgraph where the candidate entity is located. The centrality score within the local subgraph is the upper limit value, and the smoothing baseline weight coefficient is a value greater than 0 and less than 1.

5. The big data analytics-driven AI application system according to claim 1, characterized in that: When the data access and incremental capture module detects that a composite index block corresponding to an entity already exists, it resets the lineage confidence decay timer for all directly associated adjacent nodes of that entity. When it detects that a composite index block corresponding to that entity does not exist, it performs a new index block insertion operation.

6. The big data analytics-driven AI application system according to claim 1, characterized in that: The Agent access control and circuit breaker tracing module is also used to perform low-confidence splicing detection on the generated content: if the evidence reference on which the statement fragment in the generated content depends is determined to have been isolated, the output of the statement fragment is blocked; if the statement fragment lacks traceable evidence references to support it, the output of the statement fragment is also blocked.

7. A big data analytics-driven AI application method, applied to the system according to any one of claims 1 to 6, characterized in that, include: Monitor data change logs from multiple business data sources in real time and capture change events; Based on the change event, the entity's embedding vector, subgraph adjacency update fragment, and lineage metadata are generated. The embedding vector, the subgraph adjacency update fragment, and the lineage metadata are serialized according to a preset composite structure to form the update load of the composite index block, and the update load is written into the composite index storage engine. Receive a query request, convert the query request into a query vector, and perform preliminary recall of candidate entities based on the vector index; Read the embedding vector, subgraph adjacency data, and lineage metadata from the joint index block to which the candidate entity belongs. Calculate the semantic similarity between the query vector and the candidate entity embedding vector, the node centrality weight of the candidate entity in the knowledge graph, and the confidence decay value of the candidate entity's lineage tag. Multiply the semantic similarity, the node centrality weight, and the confidence decay value to obtain the fusion retrieval score of the candidate entity. The fusion retrieval score is compared with a preset circuit breaker threshold: if the fusion retrieval score is less than the circuit breaker threshold, the candidate entity is isolated; if the fusion retrieval score is not less than the circuit breaker threshold, the candidate entity is included in the candidate result set. The candidate entities in the candidate result set are sorted according to the fusion search score, and the preset number of candidate entities at the top of the sort are output as the search results. A corresponding evidence citation identifier is generated for the search results. The search results, along with the evidence citation identifier, are input into the large language model to generate the answer content, and the binding relationship between the answer content and the evidence citation identifier is recorded.

8. The AI ​​application method driven by big data analytics according to claim 7, characterized in that: Before receiving a query request, the process also includes: obtaining the identity of the Agent that initiated the query request and the scope of the pre-authorized data access domain; verifying the data domain to which the composite index block involved in the query request belongs; and removing composite index blocks that exceed the scope of the pre-authorized data access domain from the preliminary recall results of candidate entities.

9. The AI ​​application method driven by big data analytics according to claim 7, characterized in that, Also includes: Low-confidence piecing detection is performed on the response content: if the candidate entity corresponding to the evidence reference identifier on which the statement fragment in the response content depends has been isolated, the output of the statement fragment is blocked.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 7 to 9.

Images