A bank data access semantic modeling method

Abstract

This application provides a semantic modeling method for bank data access, relating to the field of data processing technology, including: receiving structured business files, determining the target character set and parsing to obtain header fields and data rows; performing normalized mapping on the header fields and generating field data profiles based on sample values; generating field semantic profiles using a banking business knowledge base; identifying fields with multiple candidate business expressions or inconsistent with the field data profiles, receiving user confirmation or correction to obtain confirmed field profiles; generating a target physical table structure based on physical column names and confirmed field profiles and writing it into data rows, saving the mapping relationship between the original field names, physical column names, confirmed field profiles, and user confirmation records; generating a question data schema for intelligent question data services to call based on the mapping relationship; and updating the banking business knowledge base based on user confirmation records and supplementary content.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, specifically relating to a semantic modeling method for bank data access. Background Technology

[0002] In business scenarios such as bank operations analysis, deposit business analysis, loan risk analysis, customer profiling analysis, compliance monitoring, and ad-hoc thematic analysis, business departments often need to connect branch office reporting forms, core system exported forms, thematic analysis detail forms, or manually summarized forms to the analysis platform. These data files are usually in CSV, Excel, TSV, or similar tabular format. Field names may be in Chinese, abbreviations, business-specific descriptions, or headers with spaces and special characters. Furthermore, the same business indicator may have different names in different institutions or different themes.

[0003] Existing data access tools can perform file reading, field type mapping, automatic table creation, and batch writing to a certain extent, but their processing focus is usually on the physical storage layer. For banking personnel, completing the physical data entry does not mean that the new table can be directly understood by intelligent analysis or natural language processing systems. Especially when temporary thematic tables are not pre-integrated into indicator platforms, semantic layers, or knowledge graphs, the system often only knows the database type of the fields, but cannot determine the indicator meaning, dimension role, business line affiliation, or natural language alias of the fields in banking operations.

[0004] Existing intelligent query or semantic data modeling solutions typically assume the existence of a stable data model, indicator definition configuration, or knowledge graph. When business personnel frequently upload temporary topic files, if technical personnel still manually configure field aliases, indicator definitions, dimensional relationships, and query schemas after the data is entered into the database, it will lead to a disconnect between the access process and the semantic modeling process. There will be high maintenance costs and time delays between adding a table from "already entered into the database" to "available for querying".

[0005] In the banking sector, a more specific issue exists: numerical fields are not necessarily indicator fields. For example, customer ID, institution ID, product code, and account number may meet numerical parsing rules, but their analytical role is essentially that of a dimension or identifier. Conversely, business expressions such as loan-to-deposit ratio, non-performing loan ratio, provision coverage ratio, corporate deposit balance, and retail AUM may appear in the table header as abbreviations, aliases, or internal institutional terms. If automatic modeling is performed solely based on field type, it is easy to incorrectly classify identifier fields as indicators, or fail to map the questions raised by business personnel to the underlying fields. Summary of the Invention

[0006] This application provides a semantic modeling method for bank data access to solve one of the aforementioned technical problems.

[0007] The technical solution adopted in this application is as follows: This application provides a semantic modeling method for bank data access, including: S100: Receives structured business files, determines the target character set, and parses them to obtain header fields and data rows; S200, perform normalized mapping on each of the header fields to generate physical column names, and generate field data profiles based on the sample values ​​of each field. The field data profiles include storage type, data distribution characteristics, and anomaly resolution identifiers. S300: Using a banking business knowledge base, the field name, the field data profile, and banking business terms, indicator aliases, and business line labels are matched to generate a field semantic profile. The field semantic profile includes the analysis role, business semantic label, candidate business expression, and matching confidence. S400: Based on the semantic profile of the field, identify fields with multiple candidate business expressions or inconsistent with the data profile of the field, generate a mapping item to be confirmed, and receive confirmation or correction from the user for the mapping item to be confirmed, thereby obtaining a confirmed field profile. S500, Generate a target physical table structure based on the physical column name and the confirmation field profile, write the data row into the target physical table, and save the mapping relationship between the original field name, the physical column name, the confirmation field profile, and the user confirmation record; S600, Generate a query schema for intelligent query service to call according to the mapping relationship. The query schema associates natural language business expressions, banking business terms or indicator aliases with underlying physical column names. S700: When a supplementary knowledge base or supplementary structured business file is received, the banking business knowledge base is updated based on the user confirmation record and the supplementary content, and the updated banking business knowledge base is used to perform semantic extension on existing mapping relationships or newly accessed files.

[0008] According to one embodiment of this application, step S100 includes: reading the first byte sequence of the structured business file, determining whether the first byte sequence meets the UTF-8 encoding rules; using UTF-8 parsing when the UTF-8 encoding rules are met, using GBK parsing when the UTF-8 encoding rules are not met, using a preset default character set to backtrack parsing when parsing is abnormal; and determining the first row of data as the header field when the structured parsing component does not recognize the header.

[0009] According to one embodiment of this application, step S200 includes: establishing a candidate storage type set for each field that includes at least integer, floating-point, date, and text types; traversing sample values ​​within a preset sample range and eliminating candidate storage types that do not meet the corresponding parsing rules; and simultaneously calculating the field null value rate, unique value ratio, numerical range, date format consistency, and whether it contains leading zeros to form the data distribution characteristics.

[0010] According to one embodiment of this application, S200 further includes: when the field sample value meets the numerical parsing rules but the field name contains any of the identification features such as number, code, code, account, customer number or organization number, or the proportion of unique values ​​is higher than the preset uniqueness threshold, the corresponding field is marked as a candidate field of the identification category dimension, instead of directly determining it as an indicator field.

[0011] According to one embodiment of this application, the banking business knowledge base includes at least one of banking operation indicator terminology, business line terminology, indicator alias dictionary, field mapping dictionary, indicator definition description, and subject domain tags; S300 includes: performing word segmentation, unit identification, and banking terminology matching on the field display name, and weightedly fusing the field name matching result, field data profile matching result, and business line tag matching result to generate the matching confidence score.

[0012] According to one embodiment of this application, S400 includes: when the same field matches two or more candidate business expressions, the same candidate business expression matches two or more fields, the field data profile indicates an identifier dimension while the field semantic profile indicates an amount or ratio indicator, or the field data profile indicates a numerical type while the field semantic profile indicates a time field, the corresponding relationship is determined as a mapping item to be confirmed.

[0013] According to one embodiment of this application, in step S500, normalizing the mapping of the header field includes: removing invisible characters and BOM characters, escaping or replacing spaces, special characters and database reserved words, appending a sequence number or hash fragment when physical column names are duplicated, and saving the original field name as a display name to the field-level metadata.

[0014] According to one embodiment of this application, in step S500, writing the data row into the target physical table includes: performing null value conversion on empty strings, and performing dequotation processing on quoted field values; ignoring the excess part when the number of data rows and columns is greater than the number of fields in the target physical table, and writing null values ​​to the missing columns when the number of data rows and columns is less than the number of fields in the target physical table, and recording the abnormal row number and abnormal type.

[0015] According to one embodiment of this application, the query schema includes table-level description and field-level description; the table-level description includes physical table name, display name, subject domain label, number of record rows, uploading user, and creation time; the field-level description includes physical column name, display name, storage type, analysis role, business semantic label, candidate business expression, indicator definition, sensitive marker, sample value summary, and matching confidence.

[0016] According to one embodiment of this application, step S700 includes: writing the candidate business expressions, field tags, or indicator definitions confirmed by the user into the banking business knowledge base as local knowledge rules, and configuring source identifiers and version identifiers for the local knowledge rules; when a field in a subsequent access file matches the local knowledge rules, increasing the matching confidence of the corresponding candidate business expression.

[0017] A second aspect of this application provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps described in the method.

[0018] A third aspect of this application provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described.

[0019] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows: This application couples the physical table creation action in the structured business file access phase with the semantic modeling action for intelligent data analysis, so that the newly accessed table can have field types, analysis roles, business tags and natural language mapping relationships that can be recognized by the analysis service after being entered into the database, thereby shortening the time from file upload to data analysis.

[0020] This application generates semantic profiles of fields by combining field data profiling with a banking business knowledge base, and confirms or corrects fields with multiple candidate mappings and inconsistent type roles. This can reduce the risk of numerical identifier fields being misjudged as indicator fields and bank abbreviation fields failing to match business terms.

[0021] This application preserves the mapping relationship between the original field names, physical column names, confirmation field profiles, and user confirmation records, ensuring both the executability of Chinese fields, special character fields, and database reserved word fields at the database level, and preserving business readability and the interpretability of subsequent queries.

[0022] This application uses user confirmation records and supplementary knowledge bases as the source for updating the banking business knowledge base, enabling the system to continuously expand for branch-customized indicators, new product terminology, and special topic definitions, rather than remaining within the scope of the initial preset rules.

[0023] The question schema generated by this application can provide a stable relationship between table-level description, field-level description, business expression, matching confidence and underlying physical column names to the intelligent question counting service, which is conducive to improving the accuracy and maintainability of the mapping from natural language questions to database fields. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a system overall architecture diagram according to an embodiment of the present invention; Figure 2 This is the main flowchart of data access and automatic modeling in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the field profile generation process in an embodiment of the present invention. Figure 4 This is a diagram illustrating the physical structure and semantic mapping link of an embodiment of the present invention. Figure 5 This is a diagram illustrating the linkage between knowledge supplementation and question reuse in embodiments of the present invention. Detailed Implementation

[0025] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0026] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0027] In this application, unless otherwise expressly specified and limited, the "above" or "below" of the second feature can mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0028] Example 1 like Figure 1 As shown, the system includes a data access and parsing module, a field profile generation module, a bank knowledge matching module, a mapping confirmation module, a table creation and database entry module, a query data schema generation module, and a knowledge expansion module. Specifically, the data access and parsing module receives structured business files, determines the target character set, and parses them to obtain header fields and data rows; the field profile generation module performs normalized mapping on header fields to generate physical column names and generates field data profiles based on sample values; the bank knowledge matching module uses a banking business knowledge base to match field names, field data profiles, and banking business terms, indicator aliases, and business line tags to generate field semantic profiles; and the mapping confirmation module identifies fields with multiple candidate business expressions or inconsistencies with the field data profiles, generates mapping items to be confirmed, and receives user confirmation or correction to obtain confirmation. The module consists of three parts: Field Profiling and Table Creation / Database Input. The Field Profiling module generates the target physical table structure based on physical column names and confirmed field profiles, writes data rows to the target physical table, and saves the mapping relationship between the original field names, physical column names, confirmed field profiles, and user confirmation records. The Question Schema Generation module generates a Question Schema for the intelligent question service to use based on the mapping relationship. The Knowledge Extension module updates the banking business knowledge base based on user confirmation records and supplementary content upon receiving supplementary knowledge base or supplementary structured business documents, and uses the updated banking business knowledge base to perform semantic extensions on existing mapping relationships or newly accessed documents. The data flow and calling relationships between the modules are shown by arrows in the figure.

[0029] like Figure 1 and Figure 2 As shown, this embodiment provides a semantic modeling method for bank data access. This method can run between the data access service, metadata service, knowledge base service, database service, and intelligent query service of a bank's intelligent analysis platform. It is used to transform temporary structured business files uploaded by business personnel into data assets that simultaneously possess physical storage structure, field data profiles, field semantic profiles, and query schemas.

[0030] In this embodiment, the structured business file can be a CSV file, an Excel file, a TSV file, or other tabular file with a row and column structure. The structured business file can originate from reports exported from the bank's core system, documents submitted by branch offices, detailed business analysis reports, detailed risk monitoring reports, customer-specific analysis reports, or manually compiled summary reports. The structured business file typically includes header fields and corresponding data rows.

[0031] S100 receives the structured business file, determines the target character set, and parses it to obtain the header fields and data rows.

[0032] In this embodiment, the system first receives the structured business file uploaded by the user and performs pre-parsing processing on the structured business file. Since internal bank structured business files may be generated by different systems, branches, or office software, their character encoding may not be consistent. Directly reading them according to a fixed encoding can easily lead to problems such as garbled Chinese field names, incorrect field value parsing, or failure to recognize table headers. To address the aforementioned issues, upon receiving a structured business file, the system first reads the initial byte sequence of the file and determines the applicable target character set based on this sequence. Once the target character set is determined, the system parses the structured business file using that set to obtain the header fields and data rows.

[0033] Specifically, when the structured business file is a CSV or TSV file, the system can read the text content according to the target character set and identify the header fields and data rows based on the delimiters. When the structured business file is an Excel file, the system can read the worksheet content through the table parsing component and parse a specified row or the first row as header fields. For files where the header cannot be clearly identified, the system can determine the first row of data as the header field and the subsequent rows as data rows.

[0034] Through this step, the system converts bank structured business files from different sources, with different encoding formats, and different table formats into a unified set of header fields and data rows, providing a unified data input for subsequent field profile generation, physical table creation, semantic matching, and question schema generation.

[0035] S200, normalize and map each of the header fields to generate physical column names, and generate field data profiles based on the sample values ​​of each field.

[0036] In this embodiment, after obtaining the header fields and data rows, the system performs normalized mapping on each header field to generate physical column names suitable for database execution. The original header fields in the bank's structured business files may contain Chinese characters, spaces, parentheses, forward slashes, percent signs, special symbols, or database reserved words. If directly used as database column names, it may cause table creation statements to fail or subsequent query statements to be difficult to generate stably.

[0037] To address the aforementioned issues, the system performs normalization processing on each header field, converting the original field names into database-recognizable, referential, and conflict-free physical column names. For example, the system removes invisible characters and BOM characters, replaces spaces and special characters, escapes database reserved words, and appends ordinal numbers or hash fragments when duplicate physical column names exist. Simultaneously, the system retains the original field names as the field display names for front-end presentation, business explanations, and intelligent query result descriptions.

[0038] While generating physical column names, the system generates field data profiles based on sample values ​​of each field within a preset sample range. These field data profiles include storage type, data distribution characteristics, and anomaly resolution identifiers. Storage type determines the storage type of the database column; data distribution characteristics help determine the field's analytical role and business attributes; and anomaly resolution identifiers record instances of inconsistent formatting, abnormal null value ratios, date format conflicts, or candidate type conflicts within the field samples.

[0039] For example, for the field "Customer Number," its sample values ​​may all be numbers, but the system also counts the proportion of unique values ​​in this field, whether there are leading zeros, and whether the field name contains "Customer Number," thus avoiding misclassification as a monetary or quantitative indicator field based solely on its numerical form. For the field "Corporate Deposit Balance," the system can initially form a candidate profile for monetary indicators based on whether its sample values ​​meet numerical rules and whether the field name contains features such as "Corporate," "Deposit," and "Balance."

[0040] Through this step, the system not only obtains the physical column names that can be used to create database tables, but also forms a field data profile that can support subsequent semantic judgments, thus providing a data foundation for solving the problem of "physical fields can be created but the analysis roles are unclear".

[0041] S300 uses a banking business knowledge base to match field names, field data profiles, and banking business terms, indicator aliases, and business line labels to generate field semantic profiles.

[0042] In this embodiment, after obtaining the field data profile, the system calls the banking business knowledge base to match the field name, field data profile with banking business terms, indicator aliases and business line tags to generate a field semantic profile.

[0043] The banking business knowledge base may include one or more of the following: banking operation indicator terminology, deposit business terminology, loan business terminology, customer business terminology, risk management terminology, compliance management terminology, indicator alias dictionary, field mapping dictionary, indicator definition explanation, and subject domain tags. Through this banking business knowledge base, the system can identify common professional expressions in the banking field, such as "loan-to-deposit ratio," "non-performing loan ratio," "provision coverage ratio," "corporate deposits," "retail AUM," and "on- and off-balance sheet credit," and establish a correspondence between these business expressions and field names and field data profiles.

[0044] Specifically, the system can perform word segmentation, unit identification, indicator suffix identification, and business line trigger word identification on field display names. For example, for the field "Retail AUM Balance," the system can identify terms such as "retail," "AUM," and "balance," and, combined with the field data profile where the sample value is a monetary value, match it as a retail customer asset management related indicator field. For the field "Institution Number," the system can identify the identifier attributes corresponding to "institution" and "number," and, combined with the data distribution characteristics of a high proportion of unique values, match it as an institution dimension field.

[0045] The field semantic profile may include analysis roles, business semantic tags, candidate business expressions, and matching confidence scores. Analysis roles represent the field's function in the analysis, such as metrics, dimensions, time-related fields, or identifier fields. Business semantic tags represent the business theme to which the field belongs, such as deposits, loans, customers, risks, compliance, or institutional operations. Candidate business expressions represent natural language question expressions that correspond to the field. Matching confidence scores represent the reliability of the system's semantic matching results for that field.

[0046] This step expands the fields from simple database fields to semantic fields with banking business meaning, solving the problem that existing data access only completes physical entry into the database but cannot be understood by the intelligent data query system.

[0047] S400 identifies fields with multiple candidate business expressions or inconsistent with the field data profile, generates a mapping item to be confirmed, and receives user confirmation or correction to obtain a confirmed field profile.

[0048] In this embodiment, after generating a field semantic profile, the system further determines whether there is any uncertainty or conflict in the field semantic profile. Because banking business fields may have abbreviations, aliases, internal institutional names, or multiple meanings, the candidate business expressions automatically matched by the system may not be unique. Furthermore, there may be inconsistencies between the field data profile and the field semantic profile; for example, a field sample value may appear numerical, but its business meaning is actually an identifier field such as customer number, account number, or institution number.

[0049] To address the aforementioned issues, the system identifies fields with multiple candidate business expressions or those inconsistent with the field data profile, and generates a mapping item to be confirmed. This mapping item may include the original field name, physical column name, field data profile, candidate business expression, business semantic label, matching confidence level, and reason for conflict.

[0050] For example, when the field "Balance" may be matched as "Deposit Balance", "Loan Balance" or "Wealth Management Balance", the system will include the field in the pending confirmation mapping item; when the sample values ​​of the field "Customer Number" all meet the numerical parsing rules, but the field name and unique value ratio point to the identifier dimension, if the semantic matching result identifies it as a numerical indicator, the system will include the field in the pending confirmation mapping item; when the sample values ​​of the field "Date" have multiple date formats or some cannot be parsed, the system can also include it in the pending confirmation mapping item.

[0051] The system displays the mapping items to be confirmed to the user through an interactive interface and receives the user's confirmation or correction of candidate business expressions, analysis roles, business semantic tags, indicator definitions, or sensitive markers. Based on the user's confirmation or correction, the system generates a confirmed field profile. Compared to automatically generated field semantic profiles, the confirmed field profile has higher business credibility and can serve as the basis for subsequent table creation and database entry, metadata storage, question schema generation, and knowledge base updates.

[0052] This step combines automated modeling with confirmation from business personnel, reducing the risk of semantic misjudgment and natural language mapping errors in the banking sector.

[0053] S500, generate a target physical table structure based on the physical column name and the confirmation field profile, write the data row into the target physical table, and save the mapping relationship between the original field name, physical column name, confirmation field profile and user confirmation record.

[0054] In this embodiment, the system generates a target physical table structure based on the physical column names and confirmation field profiles. The target physical table structure includes the target physical table name, primary key field, names of each physical column, and the database storage type corresponding to each physical column name.

[0055] Specifically, the system can automatically generate a unique physical table name and determine the database column type of each field based on the storage type in the field data profile. For example, an integer field can be mapped to an integer type, a floating-point field can be mapped to a decimal or floating-point type, a date field can be mapped to a date or timestamp type, and a text field can be mapped to a character type. Subsequently, the system assembles and executes the table creation statement according to the field order to create the target physical table.

[0056] After the target physical table is created, the system writes the data rows from the structured business file into the target physical table. During the writing process, the system can convert data values ​​according to the field storage type and handle cases such as empty strings, quoted content, and inconsistent row and column numbers. For data that cannot be written normally, the system can record the abnormal row number, abnormal field, and abnormal type for subsequent traceability.

[0057] Simultaneously, the system stores the mapping relationship between the original field names, physical column names, confirmation field profiles, and user confirmation records. This mapping relationship can be stored in a metadata table, semantic mapping table, or knowledge base records. Through this mapping relationship, the system can ensure the execution of database queries using physical column names, and ensure business readability and query interpretability using original field names and confirmation field profiles.

[0058] Through this step, this embodiment completes the process of importing structured business files into the target physical table, and binds the physical table creation results with the semantic modeling results to avoid the separation of data access and semantic configuration.

[0059] S600, Generate a question schema for the intelligent question service to call based on the mapping relationship.

[0060] In this embodiment, the system generates a query schema for the intelligent query service to call based on the mapping relationship stored in S500. The query schema is used to describe to the intelligent query service the table-level information, field-level information, and the correspondence between natural language business expressions and underlying physical column names of the target physical table.

[0061] Specifically, the query schema can include table-level descriptions and field-level descriptions. Table-level descriptions can include physical table name, display name, subject area label, number of rows, uploading user, and creation time. Field-level descriptions can include physical column name, display name, storage type, analysis role, business semantic label, candidate business expression, metric definition, sensitivity marker, sample value summary, and matching confidence level.

[0062] When the intelligent question data service receives a user's natural language question, it can invoke the question data schema and recall candidate fields based on banking terminology, indicator aliases, or business line expressions in the question. For example, when a user enters "query the ranking of corporate deposit balances of each branch this month," the intelligent question data service can identify "branch" as an institutional dimension field, "corporate deposit balance" as a deposit business indicator field, and "this month" as a time-limited condition based on the question data schema, and map them to the underlying physical column names respectively.

[0063] Through this step, the system provides the metadata and semantic mapping results generated during the data access phase to the intelligent data query service, enabling newly accessed structured business files to be recognized and called by the natural language data query system after they are entered into the database.

[0064] S700: When a supplementary knowledge base or supplementary structured business file is received, the banking business knowledge base is updated based on the user confirmation record and the supplementary content, and the updated banking business knowledge base is used to perform semantic extension on existing mapping relationships or newly accessed files.

[0065] In this embodiment, the system further supports the continuous expansion of the banking business knowledge base. Because banking business scenarios may include branch-defined indicators, new product terminology, temporary thematic definitions, or internal abbreviations, a pre-built banking business knowledge base may not be able to cover all business expressions in its initial state.

[0066] To address the aforementioned issues, when the system receives supplementary knowledge bases or supplementary structured business documents uploaded by users, the system parses these documents and writes new terms, indicator aliases, field mapping relationships, business line tags, or indicator definitions into the banking business knowledge base. The supplementary knowledge base can be banking policy documents, product manuals, indicator definitions, thematic analysis materials, or manually compiled field mapping tables.

[0067] Meanwhile, the system uses user confirmation records generated in S400 as a source for knowledge base updates. For example, if a user confirms the field "corporate deposit balance" as "corporate bank deposit balance", the system can write this confirmation relationship into the banking business knowledge base as a local knowledge rule; when similar fields such as "corporate deposit balance" and "corporate bank balance" appear again in subsequent access files, the system can increase the confidence level of matching it to "corporate bank deposit balance".

[0068] Through this step, the system will convert user-confirmed experience and supplementary information into reusable knowledge rules, enabling the banking business knowledge base to continuously evolve with actual use, thereby improving the semantic matching accuracy and automation level when accessing subsequent structured business documents.

[0069] In some implementations, the target character set determination process in S100 includes encoding judgment and fallback parsing. The system reads the first byte sequence of the structured business file and determines whether the first byte sequence conforms to the UTF-8 encoding rules. When the judgment result is that the UTF-8 encoding rules are met, the system uses UTF-8 to parse the structured business file; when the judgment result is that the UTF-8 encoding rules are not met, the system uses GBK to parse the structured business file; when an exception occurs during the parsing process, the system uses a preset default character set for fallback parsing.

[0070] Furthermore, when the structured parsing component fails to recognize the table header, the system identifies the first row of data as the header field and the data following the first row as the data rows. In this way, the system can adapt to UTF-8, GBK, and other Chinese-encoded report files commonly used within banks, reducing the impact of garbled Chinese field names, missing headers, or parsing failures on subsequent automatic modeling processes.

[0071] In some implementations, the field data profiling process in S200 includes candidate storage type elimination and data distribution feature statistics. The system establishes a candidate storage type set for each field, which includes at least integer, floating-point, date, and text types. The system iterates through the sample values ​​of the field within a preset sample range and gradually eliminates candidate storage types that do not meet the conditions based on whether each sample value satisfies the corresponding parsing rules.

[0072] For example, when all non-empty sample values ​​of a field satisfy the integer parsing rules, the system retains integer candidates; when the non-empty sample values ​​of a field satisfy the numerical parsing rules but contain decimals, the system retains floating-point candidates; when the non-empty sample values ​​of a field satisfy the preset date format or can be recognized by the date parser, the system retains date candidates; when none of the above candidate types are satisfied, the system uses text as the candidate or final storage type for that field.

[0073] Simultaneously, the system statistically analyzes field null value rate, unique value ratio, numerical range, date format consistency, and whether leading zeros are included to form data distribution characteristics. Through this method, field data profiling not only supports database table creation but also assists in subsequent determinations of whether a field belongs to an indicator, dimension, time, or identifier category.

[0074] In some implementations, the system does not directly identify fields that meet the numerical parsing rules as indicator fields in S200, but further identifies candidate fields for the identifier dimension by combining field names and data distribution characteristics.

[0075] Specifically, when a field sample value meets the numerical parsing rules, but the field name contains any of the identifying features such as "number", "encoding", "code", "account number", "customer number", or "organization number", the system marks the field as a candidate field for the identifier dimension. Alternatively, when the proportion of unique values ​​in a field is higher than a preset uniqueness threshold, the system can also mark the field as a candidate field for the identifier dimension.

[0076] For example, the sample values ​​for the "Customer Number" field may all be numbers, but this field is used to identify customer objects and not for summation, averaging, or trend analysis. Directly using it as an indicator field would lead to misuse by the intelligent query service when generating statistical queries. Therefore, this implementation method uses both field name characteristics and unique value ratio characteristics to jointly identify identifier fields, reducing the probability of numeric number fields being misidentified as indicator fields in banking scenarios.

[0077] In some implementations, the banking business knowledge base used in S300 includes at least one of banking operation indicator terminology, business line terminology, indicator alias dictionary, field mapping dictionary, indicator definition description, and subject domain labels.

[0078] When generating semantic profiles for fields, the system can perform word segmentation, unit identification, and banking terminology matching on the field display names. For example, the system can identify indicator suffixes such as "balance," "transaction amount," "ratio," "number of accounts," and "number of transactions" in field names, as well as banking business terms such as "deposit," "loan," "customer," "institution," "risk," "non-performing," and "provision." The system further weights and merges the field name matching results, field data profile matching results, and business line tag matching results to generate a matching confidence score.

[0079] For example, the field "Corporate Deposit Balance" matches "corporate," "deposit," and "balance" in its name, and is represented as an monetary value in the field data profile, falling under the deposit business subject domain. Therefore, the system can assign it a high match confidence level. In contrast, if a field is only similar to "balance" in name, but its sample value is text or a number, its match confidence level can be lowered. This approach reduces false matches resulting from relying solely on field name similarity.

[0080] In some implementations, the unconfirmed mapping item in S400 is used to handle ambiguity and conflict in the automatic semantic matching process.

[0081] Specifically, when the same field matches more than two candidate business expressions, the system includes the corresponding candidate business expressions in the pending confirmation mapping items. For example, the field "balance" may correspond to deposit balance, loan balance, wealth management balance, or account balance, and the system requires business personnel to confirm its true meaning. When the same candidate business expression matches more than two fields, the system also includes it in the pending confirmation mapping items. For example, if both fields are matched as "non-performing loan balance," the system can prompt the user to confirm whether the two correspond to different definitions, different institutional levels, or different statistical periods.

[0082] Furthermore, when there is a conflict between the field data profile and the field semantic profile, the system also generates a mapping item to be confirmed. For example, the field data profile indicates that a field is an identifier dimension, while the field semantic profile indicates that it is a monetary or ratio indicator; or the field data profile indicates that a field is numerical, while the field semantic profile indicates that it is a time field. By submitting the conflicting fields to the user for confirmation, the system can prevent incorrect semantics from directly entering the query schema.

[0083] In some implementations, the field normalization mapping process in S500 includes database executability processing of the original header fields. The system removes invisible characters and BOM characters from the original field names, and escapes or replaces spaces, special characters, and database reserved words to generate physical column names that conform to database identifier rules.

[0084] When different original field names result in the same physical column name after normalization, the system can append a sequence number or hash fragment to the physical column name to avoid column name conflicts. For example, the original fields "Customer Number" and "Customer-Number" may result in the same physical column name after replacing special characters. The system can ensure that the two are unique in the database by appending a sequence number or hash fragment.

[0085] Meanwhile, the system saves the original field names as display names in the field-level metadata. In this way, the system ensures that table creation and querying at the database level are executable, while retaining field names familiar to business users, which facilitates front-end display, result interpretation, and intelligent query semantic mapping.

[0086] In some implementations, the data row writing process in S500 includes field value cleaning, column number adaptation, and exception recording. Before writing the data row to the target physical table, the system performs null value conversion on empty strings, dequotes on quoted field values, and performs type conversion on field values ​​according to the storage type of the target field.

[0087] When the number of rows and columns in the data exceeds the number of fields in the target physical table, the system can ignore the excess to avoid the entire row of data failing to import due to a single abnormal column. When the number of rows and columns in the data is less than the number of fields in the target physical table, the system can write null values ​​to the missing columns and record the abnormal row number and the abnormal type. The abnormal type can include insufficient columns, excessive columns, type conversion failure, abnormal date format, or value out of bounds.

[0088] In this way, the system can improve the fault tolerance of importing temporary structured business files, while retaining anomaly records to facilitate subsequent data governance and manual review.

[0089] In some implementations, the query schema generated by S600 includes table-level descriptions and field-level descriptions. Table-level descriptions may include the physical table name, display name, subject domain label, number of rows, uploading user, and creation time, describing the source, subject, and basic size of the target physical table. Field-level descriptions may include physical column names, display names, storage type, analysis role, business semantic labels, candidate business expressions, metric definition descriptions, sensitivity markers, sample value summaries, and matching confidence levels.

[0090] The physical column names in the field-level description are used to generate database query statements, the display names are used to show and explain the meaning of the fields to users, the analysis roles are used to determine whether the fields can be used as indicators, dimensions or time conditions, the business semantic tags and candidate business expressions are used to support the recall of natural language questions to fields, and the matching confidence is used to sort or disambiguate among multiple candidate fields.

[0091] Through this question schema, the intelligent question service can complete question parsing, field matching, and query generation by utilizing the metadata and semantic mapping relationships formed during the data access phase without reconfiguring the semantic layer manually.

[0092] In some implementations, the knowledge expansion process in S700 includes writing the user confirmation result into the banking business knowledge base. The system writes the candidate business expressions, field tags, or indicator definitions confirmed by the user into the banking business knowledge base as local knowledge rules, and configures source identifiers and version identifiers for the local knowledge rules.

[0093] The source identifier can be used to indicate that the local knowledge rule originates from a user confirmation, a file uploaded by a branch office, a supplementary policy document, or a specific business topic. The version identifier can be used to record the generation time, update batch, or applicable version of the local knowledge rule.

[0094] When fields in subsequent access files match the aforementioned local knowledge rules, the system increases the matching confidence of the corresponding candidate business expressions. For example, if a user has previously confirmed that "public deposit balance" corresponds to "corporate deposit balance," then when "public deposit balance" or similar fields appear again in subsequent files, the system can prioritize matching them to "corporate deposit balance." In this way, the system can transform the confirmation experience of business personnel into reusable semantic rules, improving the accuracy of subsequent automatic modeling.

[0095] like Figure 3As shown, during the field profile generation process, the system can extract features from both field names and field sample values ​​simultaneously. On the field name side, it can perform functions such as removing invisible characters, word segmentation, suffix recognition, unit recognition, and recognition of bank terminology trigger words; on the field sample value side, it can perform functions such as candidate storage type elimination, null value rate statistics, unique value ratio statistics, numerical range statistics, date format consistency judgment, and leading zero recognition.

[0096] For example, the sample values ​​for the field "Customer Number" may consist entirely of numbers. If judged solely based on storage type, this field might be mistakenly identified as a numerical indicator. This embodiment combines the "Customer Number" identifier feature in the field name with the unique value ratio feature to mark it as a candidate field for the identifier category dimension. For the field "Corporate Deposit Balance," the system can combine banking terms such as "corporate," "deposit," and "balance" with the distribution of monetary samples to identify it as an indicator field and assign it a deposit business theme label.

[0097] like Figure 4 As shown, the system establishes a continuous mapping link between the original field names, physical column names, confirmation field profiles, business expressions, and question data schema. The original field names are used to maintain readability for business personnel, the physical column names are used to ensure database execution, the confirmation field profiles are used to describe the field type, analysis role, and business tags, the business expressions are used to incorporate bank terminology, indicator aliases, and question expressions, and the question data schema is used for intelligent question data services to retrieve and generate query statements.

[0098] This mapping link can be persisted using metadata tables. For example, field-level metadata tables can store physical column names, display names, storage types, analysis roles, business semantic tags, sensitivity markers, and sample value summaries; semantic mapping tables can store business expressions, field tags, metric definitions, matching confidence levels, confirmation status, and confirmation sources.

[0099] like Figure 5 As shown, when business personnel modify field labels, candidate business expressions, or indicator definitions on the confirmation interface, the system writes the modification results back to the banking business knowledge base as local knowledge rules. When users upload policy documents, product manuals, definition explanations, or supplementary structured business documents, the system can parse the supplementary content and incorporate the new terms, aliases, field mappings, and topic tags into the knowledge rule version.

[0100] In the subsequent intelligent data collection process, the intelligent data collection service reads the data collection schema. When a user inputs a natural language question, the system can retrieve candidate fields based on banking terminology, indicator aliases, and business line expressions in the question, and further utilize storage type, analysis role, sample value summary, and matching confidence to perform semantic disambiguation. As a result, newly added structured business documents can quickly enter the data collection state.

[0101] In some implementations, such as Figure 1 As shown, this invention also provides a structured business document access and automatic modeling system for bank intelligent analysis platforms. The system includes a data access and parsing module, a field profile generation module, a bank knowledge matching module, a mapping confirmation module, a table creation and database entry module, a question data schema generation module, and a knowledge expansion module.

[0102] The data access and parsing module executes S100, receiving the structured business file, determining the target character set, and parsing to obtain the header fields and data rows. The field profile generation module executes S200, performing normalized mapping on the header fields, generating physical column names, and generating field data profiles based on sample values. The bank knowledge matching module executes S300, generating field semantic profiles using the banking business knowledge base. The mapping confirmation module executes S400, identifying mapping items to be confirmed and receiving user confirmation or correction. The table creation and database entry module executes S500, generating the target physical table structure, writing data rows, and saving the mapping relationship. The question data schema generation module executes S600, generating a question data schema for use by the intelligent question data service. The knowledge expansion module executes S700, updating the banking business knowledge base based on user confirmation records and supplementary content.

[0103] In some embodiments, the present invention also provides an electronic device. The electronic device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the steps of the above-described method for accessing and automatically modeling structured business documents.

[0104] In some embodiments, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-described method for accessing and automatically modeling structured business files.

[0105] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0106] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A semantic modeling method for bank data access, characterized in that, include: S100: Receives structured business files, determines the target character set, and parses them to obtain header fields and data rows; S200, perform normalized mapping on each of the header fields to generate physical column names, and generate field data profiles based on the sample values ​​of each field. The field data profiles include storage type, data distribution characteristics, and anomaly resolution identifiers. S300: Using a banking business knowledge base, the field name, the field data profile, and banking business terms, indicator aliases, and business line labels are matched to generate a field semantic profile. The field semantic profile includes the analysis role, business semantic label, candidate business expression, and matching confidence. S400: Based on the semantic profile of the field, identify fields with multiple candidate business expressions or inconsistent with the data profile of the field, generate a mapping item to be confirmed, and receive confirmation or correction from the user for the mapping item to be confirmed, thereby obtaining a confirmed field profile. S500, Generate a target physical table structure based on the physical column name and the confirmation field profile, write the data row into the target physical table, and save the mapping relationship between the original field name, the physical column name, the confirmation field profile, and the user confirmation record; S600, Generate a query schema for intelligent query service to call according to the mapping relationship. The query schema associates natural language business expressions, banking business terms or indicator aliases with underlying physical column names. S700: When a supplementary knowledge base or supplementary structured business file is received, the banking business knowledge base is updated based on the user confirmation record and the supplementary content, and the updated banking business knowledge base is used to perform semantic extension on existing mapping relationships or newly accessed files.

2. The method according to claim 1, characterized in that, S100 includes: reading the first byte sequence of the structured business file, determining whether the first byte sequence meets the UTF-8 encoding rules; using UTF-8 parsing when the UTF-8 encoding rules are met, using GBK parsing when the UTF-8 encoding rules are not met, using a preset default character set to backtrack parsing when parsing is abnormal; and determining the first row of data as the header field when the structured parsing component does not recognize the header.

3. The method according to claim 1, characterized in that, S200 includes: establishing a candidate storage type set for each field that includes at least integer, floating-point, date, and text types; traversing sample values ​​within a preset sample range and eliminating candidate storage types that do not meet the corresponding parsing rules; and simultaneously calculating the field null value rate, unique value ratio, numerical range, date format consistency, and whether it contains leading zeros to form the data distribution characteristics.

4. The method according to claim 3, characterized in that, S200 further includes: when the field sample value meets the numerical parsing rules but the field name contains any of the identification features such as number, code, code, account, customer number or organization number, or the proportion of unique values ​​is higher than the preset uniqueness threshold, the corresponding field is marked as a candidate field of the identification category dimension, instead of being directly determined as an indicator field.

5. The method according to claim 1, characterized in that, The banking business knowledge base includes at least one of banking operation indicator terminology, business line terminology, indicator alias dictionary, field mapping dictionary, indicator definition description, and subject domain tags; S300 includes: performing word segmentation, unit identification, and banking terminology matching on the field display name, and weightedly fusing the field name matching result, field data profile matching result, and business line tag matching result to generate the matching confidence score.

6. The method according to claim 1, characterized in that, S400 includes: when the same field matches two or more candidate business expressions, the same candidate business expression matches two or more fields, the field data profile indicates an identifier dimension while the field semantic profile indicates an amount or ratio indicator, or the field data profile indicates a numerical type while the field semantic profile indicates a time field, the corresponding relationship is determined as a mapping item to be confirmed.

7. The method according to claim 1, characterized in that, In S500, the normalization mapping of the header field includes: removing invisible characters and BOM characters, escaping or replacing spaces, special characters and database reserved words, appending a sequence number or hash fragment when physical column names are duplicated, and saving the original field name as the display name to the field-level metadata.

8. The method according to claim 1, characterized in that, In step S500, writing the data row into the target physical table includes: performing null value conversion on empty strings, and performing dequotation processing on quoted field values; ignoring the excess part when the number of data rows and columns is greater than the number of fields in the target physical table, and writing null values ​​to missing columns when the number of data rows and columns is less than the number of fields in the target physical table, and recording the abnormal row number and abnormal type.

9. The method according to claim 1, characterized in that, The query schema includes table-level descriptions and field-level descriptions; the table-level descriptions include physical table name, display name, subject domain label, number of record rows, uploading user, and creation time; the field-level descriptions include physical column name, display name, storage type, analysis role, business semantic label, candidate business expression, indicator definition, sensitivity marker, sample value summary, and matching confidence.

10. The method according to claim 1, characterized in that, The S700 includes: writing the candidate business expressions, field tags, or indicator definitions confirmed by the user into the banking business knowledge base as local knowledge rules, and configuring source identifiers and version identifiers for the local knowledge rules; when a field in a subsequent access file matches the local knowledge rule, increasing the matching confidence of the corresponding candidate business expression.

Images