Multi-dimensional service data quality analysis method and device, equipment and medium

Abstract

The invention relates to the technical field of data analysis, and discloses a multi-dimensional business data quality analysis method and device, equipment and a medium, and the method comprises the steps: obtaining a set of all data related to financial business activities, and obtaining a business data set, the business module is used for identifying text fragments describing customer demands, transaction rules, operation rules and business tasks in the business data set to obtain business statements, identifying semantic integrity coefficients of the business data set, obtaining business data checked in advance, obtaining check business data and identifying the check business data; and obtaining business data corresponding to the business module in the business data set to obtain to-be-verified business data, performing consistency analysis on the verification business data and the to-be-verified business data to obtain a consistency coefficient, fusing the semantic integrity coefficient and the consistency coefficient to obtain a data quality coefficient, and verifying the data quality coefficient. And the reliability of data quality analysis is improved.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a method, apparatus, equipment and medium for multidimensional business data quality analysis. Background Technology

[0002] In financial operations, data serves as the core basis for business decisions and risk control, and its quality directly impacts business efficiency and security. Financial business data encompasses various types of information, including contract texts, customer service records, and transaction rules. Among these, unstructured data accounts for a high proportion, has complex formats, and contains numerous technical terms and logical connections, demanding extremely high precision in data processing.

[0003] Existing technologies have significant limitations in processing financial business data: On the one hand, the processing of unstructured data often remains at the level of simple extraction without refined cleaning. Redundant information (such as special symbols and irrelevant greetings) interferes with subsequent analysis, leading to errors in the recognition of business statements. On the other hand, semantic missing detection only focuses on the omission of explicit elements (such as "amount" and "term"), ignoring implicit logical connection defects (such as the logical gap of "loan application" not being associated with "repayment rules"), failing to comprehensively assess the semantic integrity of the data. Furthermore, data quality assessment often relies on single consistency comparisons (such as numerical matching) without integrating semantic-level validity analysis, making it difficult for the assessment results to reflect the reliability of the data at the business logic level. Summary of the Invention

[0004] This invention provides a method, apparatus, computer equipment, and medium for multidimensional business data quality analysis, in order to solve the problem of low reliability of existing business data quality analysis methods on the market.

[0005] Firstly, a multi-dimensional business data quality analysis method is provided, including: Obtain a collection of all data related to financial business activities to obtain a business data set; identify text fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks to obtain business statements. The semantic integrity coefficient of the business data set is identified by performing explicit missing detection and implicit semantic missing analysis on the business statements. Acquire pre-verified business data, obtain verified business data, and identify the business module of the verified business data; Obtain the business data corresponding to the business module from the business data set to obtain the business data to be verified. A consistency analysis is performed on the verification business data and the business data to be verified to obtain a consistency coefficient. The semantic integrity coefficient and the consistency coefficient are then combined to obtain a data quality coefficient.

[0006] Secondly, a multi-dimensional business data quality analysis device is provided, including: The text recognition module is used to acquire a set of all data related to financial business activities, obtain a business data set, and recognize text fragments in the business data set that describe customer needs, transaction rules, operation rules and business tasks to obtain business statements. The semantic analysis module is used to identify the semantic integrity coefficient of the business data set by performing explicit missing detection and implicit semantic missing analysis on the business statements; The module identification module is used to acquire pre-verified business data, obtain verified business data, and identify the business module of the verified business data. The data acquisition module is used to acquire the business data corresponding to the business module in the business data set, and obtain the business data to be verified. The consistency analysis module is used to perform consistency analysis on the verification business data and the business data to be verified, obtain a consistency coefficient, and fuse the semantic integrity coefficient and the consistency coefficient to obtain a data quality coefficient.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned multidimensional business data quality analysis method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned multidimensional business data quality analysis method.

[0009] The aforementioned multi-dimensional business data quality analysis method, apparatus, computer equipment, and storage medium can achieve the following: A business data set is obtained by acquiring a collection of all data related to financial business activities; text fragments describing customer needs, transaction rules, operational rules, and business tasks within the business data set are identified to obtain business statements; explicit missing data detection and implicit semantic missing data analysis are performed on these business statements to identify the semantic integrity coefficient of the business data set; pre-verified business data is obtained to obtain verified business data; the business modules of the verified business data are identified; business data corresponding to the business modules in the business data set is obtained to obtain business data to be verified; consistency analysis is performed between the verified business data and the business data to be verified to obtain a consistency coefficient; and the semantic integrity coefficient and the consistency coefficient are fused to obtain a data quality coefficient. This improves the reliability of data quality analysis. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a multidimensional business data quality analysis method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a multi-dimensional business data quality analysis method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a multi-dimensional business data quality analysis device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The multidimensional business data quality analysis method provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain a set of all data related to financial business activities from the client, resulting in a business data set. It then identifies text fragments describing customer needs, transaction rules, operational rules, and business tasks within the business data set to obtain business statements. By performing explicit missing data detection and implicit semantic missing data analysis on these business statements, it identifies the semantic integrity coefficient of the business data set. It then obtains pre-verified business data, resulting in verified business data. Next, it identifies the business modules within the verified business data, obtains the business data corresponding to those modules from the business data set, resulting in business data to be verified. It performs consistency analysis on the verified business data and the business data to be verified to obtain a consistency coefficient. Finally, it merges the semantic integrity coefficient and the consistency coefficient to obtain a data quality coefficient. This improves the reliability of data quality analysis. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the multidimensional business data quality analysis method provided in this embodiment of the invention includes the following steps: S1. Obtain a set of all data related to financial business activities to obtain a business data set. Identify text fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks to obtain business statements.

[0015] In this embodiment of the invention, the step of identifying text fragments describing customer needs, transaction rules, operational rules, and business tasks in the business data set to obtain business statements includes: The unstructured text in the business data set is processed to obtain structured text; The structured text is filtered for useless information to obtain a clean text set; The clean text set is subjected to text segmentation processing to obtain a set of sentence fragments; Extract the semantic features of each statement fragment in the set of statement fragments; The semantic features of each sentence fragment are matched with the preset key features of the sentence type to obtain the matching results; Based on the matching results, identify the sentence fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks, and obtain business sentences with business type labels.

[0016] In detail, the unstructured text may refer to texts such as PDF contracts, customer service chat logs, and meeting minutes.

[0017] In detail, the process of structuring unstructured text in the business data set to obtain structured text involves converting it into a unified format that can be directly processed by a computer through technical means. Specific operations include: extracting plain text content from documents (removing formatting codes, images, and other irrelevant elements), extracting text fields from databases or APIs, and integrating them into plain text data with a unified encoding (such as UTF-8). The final output is "structured text," which is a standardized text set that retains only the text content after removing complex formatting (e.g., converting a Word document to a TXT file, or extracting plain text dialogue content from a JSON dialogue log).

[0018] In detail, the filtering of useless information in the structured text refers to removing redundant information ("noise data") that does not affect the business semantics. Specific filtering includes: deleting special symbols (such as "★" and "©"), emoticons, irrelevant punctuation (such as multiple consecutive periods), duplicate blank lines, page numbers / titles in headers and footers, etc.; simultaneously standardizing text format (such as unifying full-width and half-width characters, and case conversion), and filtering out greetings (such as "Hello" and "Thank you") or system prompts (such as "[Message received]") that have no actual business meaning. The filtered text results in a "clean text set".

[0019] In detail, the text segmentation process for the clean text set refers to breaking down the "clean text set" into the smallest units (sentence fragments) with independent semantics to accurately identify business statements. Specifically, this is done by using natural language punctuation (such as periods, question marks, and exclamation marks) and semantic logic (such as parallel or explanatory relationships separated by semicolons and colons) to break down long paragraphs into individual sentences or short sentences (for example, breaking down "Customers applying for loans need to provide their ID card and proof of income, and their age must be between 22 and 60 years old" into two independent sentences). For special texts without obvious punctuation (such as continuous rule descriptions), further segmentation is performed using logical connectors such as "at the same time," "if," and "in addition." The final output is a "set of sentence fragments."

[0020] In detail, extracting the semantic features of each sentence fragment in the set of sentence fragments may refer to extracting keyword features.

[0021] In detail, the preset statement type key features can refer to the key features of several business types such as customer needs, transaction rules, operation rules, and business tasks. For example, customer needs include keywords such as "hope", "application", and "need"; transaction rules include logical words such as "must", "must not", and "if...then..."; operation rules include words such as "system automatic", "process", and "standard"; and business tasks include words such as "complete", "execute", and "need".

[0022] S2. Identify the semantic integrity coefficient of the business data set by performing explicit missing detection and implicit semantic missing analysis on the business statements.

[0023] In this embodiment of the invention, the step of identifying the semantic integrity coefficient of the business data set by performing explicit missing detection and implicit semantic missing analysis on the business statements includes: Perform semantic analysis on the business statement to obtain the semantic analysis results; Perform explicit missing data detection on the business statements to obtain explicit missing data detection results; Based on the semantic analysis results, the implicit semantic missing analysis is performed on the business data set using a graph neural network to obtain the implicit semantic analysis results. The semantic integrity coefficient of the business data set is calculated based on the explicit missing detection results and the implicit semantic analysis results.

[0024] In detail, the semantic analysis of the business statements involves using Natural Language Processing (NLP) technology to perform deep semantic parsing on the labeled business statements (such as customer needs, transaction rules, etc.) to extract their core semantic elements and logical relationships. Specifically, this includes word segmentation, part-of-speech tagging, and dependency parsing to identify entities (such as "customer" and "loan amount"), attributes (such as "annual interest rate of 5%), actions (such as "apply" and "adjust"), and logical relationships; determining the semantic roles of each component in the statement (such as "customer" being the subject of the "apply" action, and "300,000 yuan" being the object of the "application"); and combining this with a financial business dictionary (such as "credit limit" and "overdue penalty interest") to convert general semantics into business-specific semantic elements (such as mapping "borrow money" to "apply for a loan").

[0025] In this embodiment of the invention, the step of performing explicit missing detection on the business statement to obtain explicit missing detection results includes: Retrieve required element templates from the preset template library based on the business type; Natural language processing technology is used to extract the key elements contained in the business statement. The key elements are compared one by one with the required element templates to obtain the comparison results; Based on the comparison results, the explicit missing elements are located to obtain the location results; Based on the location results, the total frequency of missing elements, the missing statement rate, and the missing type rate in the business statements are quantitatively statistically analyzed to obtain the explicit missing detection results.

[0026] In detail, the step of retrieving required element templates from a pre-set template library based on business type involves retrieving the corresponding list of required elements from the pre-set template library according to the type of business statement (such as customer needs, transaction rules, operational rules, etc.). The template library is pre-built based on business domain knowledge, clearly defining the core elements required for different types of business statements to complete semantic expression (such as "customer applying for a loan" must include "amount," "term," and "purpose"). For example, for "transaction rule" statements, the template may stipulate that it must include three elements: "condition," "action to be performed," and "constraint object."

[0027] In detail, the use of natural language processing technology to extract key elements contained in the business statement includes locating core entities in the statement (such as "customer", "300,000 yuan", "July 2025"), determining the role of the entity in the business action (such as "customer" being the subject of "application" and "300,000 yuan" being the object of "application"), and identifying business domain terms (such as "credit grant" and "overdue") and logical words (such as "must" and "if...then...").

[0028] In this embodiment of the invention, the implicit semantic missing analysis performed on the business data set based on the semantic analysis results using a graph neural network to obtain the implicit semantic analysis results includes: Construct a business semantic association graph based on the semantic analysis results; Extract the structural features of the semantic association graph; Based on the structural features and semantic analysis results, a domain benchmark semantic graph model is constructed using a graph neural network. Obtain the node embedding count and edge embedding count in the business semantic association graph; Using the domain benchmark semantic graph model, abnormal associations in the business semantic association graph are identified based on the node embedding amount and the edge embedding amount, thereby obtaining a set of abnormal associations. Based on the aforementioned set of abnormal associations, implicit semantic missing elements are defined; The degree of implicit semantic missing is quantified based on the aforementioned implicit semantic missing elements, and the implicit semantic analysis results are obtained.

[0029] In detail, constructing a business semantic association graph based on the semantic analysis results can be achieved by defining entities (such as "customer", "loan product", "interest rate"), concepts (such as "transaction rules", "application process"), and actions (such as "application" and "execution") as nodes in the graph, and defining the semantic relationships between entities (such as "customer-application-loan product" and "rule-constraint-execution action") as edges in the graph, with the edge weights reflecting the strength of the relationships (e.g., high-frequency co-occurrence relationships have higher weights). The final "business semantic association graph" is thus formed.

[0030] In detail, the extraction of structural features of the semantic association graph may include statistical distribution of node types and node degree; statistical distribution of edge types and edge weights; identification of common subgraph structures (such as "condition-action-constraint object" triple subgraph, "customer-demand-rule" chain subgraph), and recording the frequency of occurrence and connection patterns of subgraphs.

[0031] In detail, the construction of the domain benchmark semantic graph model based on the graph neural network according to the structural features and the semantic analysis results can be achieved by inputting the structural features and the semantic analysis results into the graph neural network for training, so that the model learns the typical association patterns of nodes and edges in the benchmark graph (for example, in the "loan application" scenario, the "customer" node must be connected to the "loan amount" and "term" nodes through strong association edges). The model outputs a "domain benchmark semantic graph model" containing normal semantic association features, and stores parameters such as node embedding distribution, edge weight threshold, and typical subgraph structure as a standard for judging whether there are implicit missing features in the current business graph.

[0032] In detail, the acquisition of node embeddings and edge embeddings in the business semantic association graph is achieved by performing embedding learning on the current business semantic association graph through a graph neural network, transforming nodes and edges into high-dimensional vectors (embeddings) to capture their deep semantic association features.

[0033] In detail, the step of using the domain benchmark semantic graph model to identify abnormal associations in the business semantic association graph based on the node embedding amount and the edge embedding amount can be achieved by comparing the similarity between the node / edge embeddings in the current business graph and the corresponding type of node / edge embeddings in the benchmark model through cosine distance or Euclidean distance. If the difference exceeds a preset threshold, it is judged as abnormal. Alternatively, the subgraph structure in the business graph can be compared with the typical subgraph in the benchmark model (such as the "rule condition - execution action - constraint object" triple). If a key node (such as "constraint object") or key edge (such as "condition-action" edge with too low weight) is missing, it is marked as structurally abnormal.

[0034] In detail, the quantification of the degree of implicit semantic missing elements based on the implicit semantic missing elements can be achieved by scoring each missing element based on the importance of the element in the business logic (e.g., the influence of core rule elements is 1, and the influence of auxiliary explanatory elements is 0.3), combined with the difference in abnormal association (the greater the difference, the higher the influence), and summing the influence of all implicit missing elements and dividing by the total influence (or total number) of key elements in the business semantic graph to obtain the quantitative value of the degree of implicit missing elements.

[0035] In detail, calculating the semantic integrity coefficient of the business data set based on the explicit missing detection results and the implicit semantic analysis results may refer to quantifying the implicit semantic analysis results and the explicit missing detection results into probability values, and then performing weighted fusion.

[0036] In this embodiment of the invention, extracting the structural features of the semantic association graph includes: Obtain the topological statistics of the semantic association graph, and construct a node centrality matrix based on the semantic association graph; Shortest path analysis and reachability analysis are performed on the semantic association graph to obtain the shortest path length matrix and reachability matrix; The topological statistics, the node centrality matrix, the shortest path length matrix, and the reachability matrix are subjected to eigenvectorization to obtain topological statistics vector, node centrality vector, shortest path length vector, and reachability vector; The structural features are obtained by dimensionality reduction and fusion of the topological statistics vector, the node center vector, the shortest path length vector, and the reachability vector.

[0037] In detail, obtaining the topological statistics of the semantic association graph is to compute the global basic attributes of the graph, including the total number of nodes, the total number of edges, the edge density, the average node degree and the degree distribution. At the same time, the connectivity of the graph is analyzed to form a holistic quantitative description of the graph structure.

[0038] In detail, the construction of the node centrality matrix based on the semantic association graph is achieved by using indicators such as degree centrality (measuring the direct connectivity of nodes), betweenness centrality (measuring the importance of nodes as information bridges), and eigenvector centrality (measuring the strength of the association between nodes and high-influence nodes) to construct a matrix that records the multi-dimensional centrality value of each node.

[0039] In detail, the semantic association graph is subjected to shortest path analysis and reachability analysis to obtain a shortest path length matrix and a reachability matrix. The shortest path length between all node pairs is calculated to generate a shortest path length matrix. Indicators such as average path length and graph diameter (longest and shortest path) are statistically analyzed. A reachability matrix is ​​constructed for the directed graph (matrix element Aij=1 indicates that node i can reach j, otherwise it is 0). The proportion of reachable node pairs is calculated to reveal the dependencies and logical connectivity between nodes in the graph.

[0040] S3. Obtain the pre-verified business data, obtain the verified business data, and identify the business module of the verified business data.

[0041] In this embodiment of the invention, the business data that has been pre-verified refers to data that has been manually verified and reviewed in advance to ensure that the data is accurate and to lay the foundation for subsequent comparative analysis.

[0042] In this embodiment of the invention, the identification of the business module for the verification business data is achieved by extracting core entities, key attributes, and business rule descriptions from the data; then, based on a preset business module feature library, the verification business data is matched with preset module features through keyword matching, field mapping, or process logic judgment; finally, the matching results are verified in conjunction with the experience of business experts to correct module attribution deviations caused by data overlap or differences in expression, and the business module corresponding to the verification business data is finally determined.

[0043] S4. Obtain the business data corresponding to the business module from the business data set to obtain the business data to be verified.

[0044] In this embodiment of the invention, each piece of business data in the business data set is labeled with a business module, and the business data to be verified in the business data set can be directly determined based on the business module.

[0045] S5. Perform consistency analysis on the verification business data and the business data to be verified to obtain a consistency coefficient. Combine the semantic integrity coefficient and the consistency coefficient to obtain a data quality coefficient.

[0046] In this embodiment of the invention, the step of performing consistency analysis on the verification business data and the business data to be verified to obtain a consistency coefficient includes: The verification business data and the business data to be verified are processed by word segmentation to obtain the verification business word segmentation set and the business data to be verified word segmentation set. The word segmentation results in the verification business word segmentation set and the business word segmentation set to be verified are compared sequentially to obtain the word segmentation consistency comparison result. The word segmentation consistency coefficient is calculated based on the word segmentation consistency comparison result. Extract the numerical fields from the verification business data and the business data to be verified to obtain the set of verification numerical fields and the set of numerical fields to be verified. The numerical fields in the set of numerical fields to be checked and the set of numerical fields to be checked are compared sequentially to obtain the numerical consistency comparison results. The numerical consistency coefficient is then calculated based on the numerical consistency comparison results. The average of the word segmentation consistency coefficient and the numerical consistency coefficient is calculated to obtain the consistency coefficient.

[0047] In this embodiment of the invention, the process of fusing the semantic integrity coefficient and the consistency coefficient to obtain the data quality coefficient can be achieved by weighted fusion, assigning a weight of 0.5 to each of the semantic integrity coefficient and the consistency coefficient.

[0048] As can be seen, the above solution first acquires a financial business data set, converts unstructured text such as PDF contracts and customer service chat logs into structured text, filters out useless information such as special symbols and greetings, and segments it into sentence fragments. Semantic features of each fragment are extracted and matched with key features of the business type, such as pre-defined customer needs and transaction rules, to obtain business sentences labeled with the business type. Next, a semantic integrity coefficient is calculated through explicit missing element detection and implicit semantic missing element analysis: explicit detection retrieves the required element template based on the business type and quantifies the missing key elements extracted from the business sentences; implicit analysis constructs a business semantic association graph containing entities, action nodes, and semantic relationship edges based on a graph neural network, extracts structural features, establishes a domain benchmark semantic graph model, identifies abnormal associations and quantifies missing elements, and then weights and fuses the results to obtain the semantic integrity coefficient. Then, accurate business data that has been pre-verified manually is acquired, core entities and key attributes are extracted, and the corresponding business module is determined by combining the business module feature library and expert verification. Finally, based on the business module labeling, business data to be verified is selected from the business data set. Finally, the word segmentation consistency coefficient is calculated for the verification data and the data to be verified, and the numerical field is extracted to calculate the numerical consistency coefficient. The average value is taken as the consistency coefficient. Then, the semantic integrity coefficient and the consistency coefficient are each assigned a weight of 0.5 and weighted and merged to finally obtain the data quality coefficient.

[0049] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] In one embodiment, a multi-dimensional business data quality analysis device is provided, which corresponds one-to-one with the multi-dimensional business data quality analysis method in the above embodiments. For example... Figure 3As shown, the multidimensional business data quality analysis device includes a text recognition module 101, a semantic analysis module 102, a module recognition module 103, a data acquisition module 104, and a consistency analysis module 105. Detailed descriptions of each functional module are as follows: The text recognition module 101 is used to acquire a set of all data related to financial business activities, obtain a business data set, and recognize text fragments in the business data set that describe customer needs, transaction rules, operation rules and business tasks to obtain business statements. The semantic analysis module 102 is used to identify the semantic integrity coefficient of the business data set by performing explicit missing detection and implicit semantic missing analysis on the business statements; Module identification module 103 is used to acquire pre-verified business data, obtain verified business data, and identify the business module of the verified business data; Data acquisition module 104 is used to acquire business data corresponding to the business module in the business data set to obtain business data to be verified; The consistency analysis module 105 is used to perform consistency analysis on the verification business data and the business data to be verified, obtain a consistency coefficient, and fuse the semantic integrity coefficient and the consistency coefficient to obtain a data quality coefficient.

[0051] In one embodiment, the text recognition module 101, when performing the process of recognizing text fragments describing customer needs, transaction rules, operational rules, and business tasks in the business data set to obtain business statements, is specifically used for: The unstructured text in the business data set is processed to obtain structured text; The structured text is filtered for useless information to obtain a clean text set; The clean text set is subjected to text segmentation processing to obtain a set of sentence fragments; Extract the semantic features of each statement fragment in the set of statement fragments; The semantic features of each sentence fragment are matched with the preset key features of the sentence type to obtain the matching results; Based on the matching results, identify the sentence fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks, and obtain business sentences with business type labels.

[0052] In one embodiment, the semantic analysis module 102, when performing the step of identifying the semantic integrity coefficient of the business data set by performing explicit missing detection and implicit semantic missing analysis on the business statements, is specifically used for: Perform semantic analysis on the business statement to obtain the semantic analysis results; Perform explicit missing data detection on the business statements to obtain explicit missing data detection results; Based on the semantic analysis results, the implicit semantic missing analysis is performed on the business data set using a graph neural network to obtain the implicit semantic analysis results. The semantic integrity coefficient of the business data set is calculated based on the explicit missing detection results and the implicit semantic analysis results.

[0053] In one embodiment, when the semantic analysis module 102 performs explicit missing detection on the business statement and obtains the explicit missing detection result, it is specifically used for: Retrieve required element templates from the preset template library based on the business type; Natural language processing technology is used to extract the key elements contained in the business statement. The key elements are compared one by one with the required element templates to obtain the comparison results; Based on the comparison results, the explicit missing elements are located to obtain the location results; Based on the location results, the total frequency of missing elements, the missing statement rate, and the missing type rate in the business statements are quantitatively statistically analyzed to obtain the explicit missing detection results.

[0054] In one embodiment, the semantic analysis module 102, when performing the implicit semantic missing analysis on the business data set based on the semantic analysis result using a graph neural network to obtain the implicit semantic analysis result, is specifically used for: Construct a business semantic association graph based on the semantic analysis results; Extract the structural features of the semantic association graph; Based on the structural features and semantic analysis results, a domain benchmark semantic graph model is constructed using a graph neural network. Obtain the node embedding count and edge embedding count in the business semantic association graph; Using the domain benchmark semantic graph model, abnormal associations in the business semantic association graph are identified based on the node embedding amount and the edge embedding amount, thereby obtaining a set of abnormal associations. Based on the aforementioned set of abnormal associations, implicit semantic missing elements are defined; The degree of implicit semantic missing is quantified based on the aforementioned implicit semantic missing elements, and the implicit semantic analysis results are obtained.

[0055] In one embodiment, the semantic analysis module 102, when performing the extraction of structural features from the semantic association graph, is specifically used for: Obtain the topological statistics of the semantic association graph, and construct a node centrality matrix based on the semantic association graph; Shortest path analysis and reachability analysis are performed on the semantic association graph to obtain the shortest path length matrix and reachability matrix; The topological statistics, the node centrality matrix, the shortest path length matrix, and the reachability matrix are subjected to eigenvectorization to obtain topological statistics vector, node centrality vector, shortest path length vector, and reachability vector; The structural features are obtained by dimensionality reduction and fusion of the topological statistics vector, the node center vector, the shortest path length vector, and the reachability vector.

[0056] In one embodiment, the consistency analysis module 105, when performing consistency analysis on the verification business data and the business data to be verified to obtain a consistency coefficient, is specifically used for: The verification business data and the business data to be verified are processed by word segmentation to obtain the verification business word segmentation set and the business data to be verified word segmentation set. The word segmentation results in the verification business word segmentation set and the business word segmentation set to be verified are compared sequentially to obtain the word segmentation consistency comparison result. The word segmentation consistency coefficient is calculated based on the word segmentation consistency comparison result. Extract the numerical fields from the verification business data and the business data to be verified to obtain the set of verification numerical fields and the set of numerical fields to be verified. The numerical fields in the set of numerical fields to be checked and the set of numerical fields to be checked are compared sequentially to obtain the numerical consistency comparison results. The numerical consistency coefficient is then calculated based on the numerical consistency comparison results. The average of the word segmentation consistency coefficient and the numerical consistency coefficient is calculated to obtain the consistency coefficient.

[0057] This invention provides a multi-dimensional business data quality analysis device. First, it acquires a financial business data set, converting unstructured text such as PDF contracts and customer service chat logs into structured text. After filtering out useless information such as special symbols and greetings, the data is segmented into sentence fragments. Semantic features of each fragment are extracted and matched with preset key features of business types, such as customer needs and transaction rules, to obtain business sentences labeled with business types. Second, it calculates the semantic integrity coefficient through explicit missing element detection and implicit semantic missing element analysis: explicit detection retrieves required element templates based on business types and quantifies missing key elements extracted from business sentences; implicit analysis constructs a business semantic association graph containing entities, action nodes, and semantic relationship edges based on graph neural networks, extracts structural features, establishes a domain benchmark semantic graph model, identifies abnormal associations, quantifies missing elements, and then weights and fuses the results to obtain the semantic integrity coefficient. Next, it acquires accurate business data that has been pre-verified manually, extracts core entities and key attributes, and combines a business module feature library with expert verification to determine the corresponding business module. Then, based on the business module labeling, it selects business data to be verified from the business data set. Finally, the word segmentation consistency coefficient is calculated for the verification data and the data to be verified, and the numerical field is extracted to calculate the numerical consistency coefficient. The average value is taken as the consistency coefficient. Then, the semantic integrity coefficient and the consistency coefficient are each assigned a weight of 0.5 and weighted and merged to finally obtain the data quality coefficient.

[0058] Specific limitations regarding the multidimensional business data quality analysis device can be found in the limitations of the multidimensional business data quality analysis method described above, and will not be repeated here. Each module in the aforementioned multidimensional business data quality analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0059] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multi-dimensional business data quality analysis method on the server side.

[0060] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a multi-dimensional business data quality analysis method. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain a collection of all data related to financial business activities to obtain a business data set; identify text fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks to obtain business statements. The semantic integrity coefficient of the business data set is identified by performing explicit missing detection and implicit semantic missing analysis on the business statements. Acquire pre-verified business data, obtain verified business data, and identify the business module of the verified business data; Obtain the business data corresponding to the business module from the business data set to obtain the business data to be verified. A consistency analysis is performed on the verification business data and the business data to be verified to obtain a consistency coefficient. The semantic integrity coefficient and the consistency coefficient are then combined to obtain a data quality coefficient.

[0061] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain a collection of all data related to financial business activities to obtain a business data set; identify text fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks to obtain business statements. The semantic integrity coefficient of the business data set is identified by performing explicit missing detection and implicit semantic missing analysis on the business statements. Acquire pre-verified business data, obtain verified business data, and identify the business module of the verified business data; Obtain the business data corresponding to the business module from the business data set to obtain the business data to be verified. A consistency analysis is performed on the verification business data and the business data to be verified to obtain a consistency coefficient. The semantic integrity coefficient and the consistency coefficient are then combined to obtain a data quality coefficient.

[0062] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0063] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0065] Finally, it should be noted that if any software tools or components not belonging to this company appear in the embodiments of the application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method of multi-dimensional business data quality analysis, characterized by, include: Obtain a collection of all data related to financial business activities to obtain a business data set; identify text fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks to obtain business statements. The semantic integrity coefficient of the business data set is identified by performing explicit missing detection and implicit semantic missing analysis on the business statements. Acquire pre-verified business data, obtain verified business data, and identify the business module of the verified business data; Obtain the business data corresponding to the business module from the business data set to obtain the business data to be verified. A consistency analysis is performed on the verification business data and the business data to be verified to obtain a consistency coefficient. The semantic integrity coefficient and the consistency coefficient are then combined to obtain a data quality coefficient.

2. The method of multi-dimensional business data quality analysis of claim 1, wherein, The process involves identifying text fragments describing customer needs, transaction rules, operational rules, and business tasks within the business data set to obtain business statements, including: The unstructured text in the business data set is processed to obtain structured text; The structured text is filtered for useless information to obtain a set of clean text. The clean text set is subjected to text segmentation to obtain a set of sentence fragments; Extract the semantic features of each sentence fragment in the set of sentence fragments; The semantic features of each sentence fragment are matched with the preset key features of the sentence type to obtain the matching results; Based on the matching results, identify the sentence fragments in the business data set that describe customer needs, transaction rules, operational rules, and business tasks, and obtain business sentences with business type labels.

3. The method of multi-dimensional business data quality analysis of claim 1, wherein, The step of identifying the semantic integrity coefficient of the business data set by performing explicit missing detection and implicit semantic missing analysis on the business statements includes: Perform semantic analysis on the business statement to obtain the semantic analysis results; Perform explicit missing data detection on the business statements to obtain explicit missing data detection results; Based on the semantic analysis results, the implicit semantic missing analysis is performed on the business data set using a graph neural network to obtain the implicit semantic analysis results. The semantic integrity coefficient of the business data set is calculated based on the explicit missing detection results and the implicit semantic analysis results.

4. The method of multi-dimensional business data quality analysis of claim 3, wherein, The step of performing explicit missing detection on the business statement to obtain explicit missing detection results includes: Retrieve required element templates from the preset template library based on the business type; Natural language processing technology is used to extract the key elements contained in the business statement. The key elements are compared one by one with the required element templates to obtain the comparison results; Based on the comparison results, the explicit missing elements are located to obtain the location results; Based on the location results, the total frequency of missing elements, the missing statement rate, and the missing type rate in the business statements are quantitatively statistically analyzed to obtain the explicit missing detection results.

5. The method of multi-dimensional business data quality analysis of claim 3, wherein, The implicit semantic missing analysis is performed on the business data set based on the semantic analysis results using a graph neural network, resulting in implicit semantic analysis results, including: Construct a business semantic association graph based on the semantic analysis results; Extract the structural features of the semantic association graph; Based on the structural features and semantic analysis results, a domain benchmark semantic graph model is constructed using a graph neural network. Obtain the node embedding count and edge embedding count in the business semantic association graph; Using the domain benchmark semantic graph model, abnormal associations in the business semantic association graph are identified based on the node embedding amount and the edge embedding amount, thereby obtaining a set of abnormal associations. Based on the aforementioned set of abnormal associations, implicit semantic missing elements are defined; The degree of implicit semantic missing is quantified based on the aforementioned implicit semantic missing elements, and the implicit semantic analysis results are obtained.

6. The method for multi-dimensional business data quality analysis of claim 5, wherein, The extraction of structural features from the semantic association graph includes: Obtain the topological statistics of the semantic association graph, and construct a node centrality matrix based on the semantic association graph; Shortest path analysis and reachability analysis are performed on the semantic association graph to obtain the shortest path length matrix and reachability matrix; The topological statistics, the node centrality matrix, the shortest path length matrix, and the reachability matrix are subjected to eigenvectorization to obtain topological statistics vector, node centrality vector, shortest path length vector, and reachability vector; The structural features are obtained by dimensionality reduction and fusion of the topological statistics vector, the node center vector, the shortest path length vector, and the reachability vector.

7. The multidimensional business data quality analysis method as described in claim 1, characterized in that, The consistency analysis of the verification business data and the business data to be verified to obtain a consistency coefficient includes: The verification business data and the business data to be verified are processed by word segmentation to obtain the verification business word segmentation set and the business data to be verified word segmentation set. The word segmentation results in the verification business word segmentation set and the business word segmentation set to be verified are compared sequentially to obtain the word segmentation consistency comparison result. The word segmentation consistency coefficient is calculated based on the word segmentation consistency comparison result. Extract the numerical fields from the verification business data and the business data to be verified to obtain the set of verification numerical fields and the set of numerical fields to be verified. The numerical fields in the set of numerical fields to be checked and the set of numerical fields to be checked are compared sequentially to obtain the numerical consistency comparison results. The numerical consistency coefficient is then calculated based on the numerical consistency comparison results. The average of the word segmentation consistency coefficient and the numerical consistency coefficient is calculated to obtain the consistency coefficient.

8. A multidimensional business data quality analysis device, characterized in that, include: The text recognition module is used to acquire a set of all data related to financial business activities, obtain a business data set, and recognize text fragments in the business data set that describe customer needs, transaction rules, operation rules and business tasks to obtain business statements. The semantic analysis module is used to identify the semantic integrity coefficient of the business data set by performing explicit missing detection and implicit semantic missing analysis on the business statements; The module identification module is used to acquire pre-verified business data, obtain verified business data, and identify the business module of the verified business data. The data acquisition module is used to acquire the business data corresponding to the business module in the business data set, and obtain the business data to be verified. The consistency analysis module is used to perform consistency analysis on the verification business data and the business data to be verified, obtain a consistency coefficient, and fuse the semantic integrity coefficient and the consistency coefficient to obtain a data quality coefficient.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multidimensional business data quality analysis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multidimensional business data quality analysis method as described in any one of claims 1 to 7.

Images