Relational data importing method and device suitable for retrieval enhancement generation system, equipment and medium

Abstract

The invention discloses a relational data importing method and device suitable for a retrieval enhancement generation system, equipment and a medium, and relates to the technical field of data processing and artificial intelligence crossing. The method comprises the following steps: analyzing metadata in a relational database to obtain table structure information, field attribute information and inter-table association relationship information; constructing a knowledge set for describing data semantics, wherein the knowledge set comprises metadata description knowledge generated on the basis of table structure information and field attribute information, association rule knowledge generated on the basis of association relationship information between tables, and instance semantization knowledge generated on the basis of field attribute information and business data content; and converting the knowledge set into an unstructured knowledge unit which can be retrieved by the RAG system, and updating the unstructured knowledge unit based on an incremental synchronization mechanism. By means of the technical scheme, the problems that the adaptability of relational data and an RAG system is poor, and knowledge extraction is insufficient can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing and artificial intelligence, in particular to a relational data import method and device suitable for a retrieval-augmented generation system, equipment and medium. BACKGROUND

[0002] As an artificial intelligence system combining retrieval and generation capabilities, the RAG (Retrieval-Augmented Generation) system can generate accurate answers based on external knowledge, and its performance is highly dependent on the quality and coverage of the input knowledge. Currently, the knowledge sources of the RAG system are mainly text, documents and other unstructured data, while relational databases, as the core carriers for storing structured information, contain a large amount of associated data with business value, but are difficult to be directly utilized by the RAG system due to format differences.

[0003] In the prior art, the way to import relational data into the RAG system is mainly simple data export and splicing, which has the following defects: first, the inter-table association relationship is ignored, resulting in knowledge fragmentation, for example, the association information between the order table and the user table is lost, and the complete business logic cannot be formed; second, only field values are extracted, while metadata such as table structure and constraint conditions are omitted, limiting the understanding of the RAG system on the data background; third, there is a lack of dynamic updating mechanism, when the relational data is added, deleted or modified, the RAG system cannot be synchronized in real time, resulting in knowledge lag. Therefore, there is an urgent need for a method that can fully exploit the value of relational data and achieve efficient import. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a relational data import method and device suitable for a retrieval-augmented generation system, which can solve the problems of poor adaptability of relational data to the RAG system and insufficient knowledge extraction. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a relational data import method suitable for a retrieval-augmented generation system, comprising:

[0006] Parsing the metadata in the relational database to obtain table structure information, field attribute information and inter-table association relationship information;

[0007] Based on the table structure information, the field attribute information and the inter-table association relationship information, a knowledge set for describing data semantics is constructed; the knowledge set includes: metadata description knowledge generated based on the table structure information and the field attribute information, association rule knowledge generated based on the inter-table association relationship information, and instance semantic knowledge generated based on the field attribute information and business data content;

[0008] The knowledge set is converted into unstructured knowledge units that can be retrieved by the retrieval enhancement generation system, and an incremental synchronization mechanism based on data change events of the relational database is established so as to update the unstructured knowledge units based on the incremental synchronization mechanism.

[0009] Optionally, parsing the metadata in the relational database to obtain table structure information, field attribute information, and inter-table relationship information includes:

[0010] Establish a connection with a relational database through a database connection interface to obtain access permissions to the system tables in the relational database;

[0011] The system tables are queried based on preset filtering conditions to filter out target business tables, and the target business tables are classified by importance based on preset business rules to determine the table structure information.

[0012] The system table extracts the field attribute information of the target business table, and performs semantic transformation on the target format field attribute information to generate enhanced field attribute information with clear business meaning; the field attribute information includes at least the field name, data type, and constraints; wherein, the semantic transformation includes: converting the numeric format timestamp into a date and time string in a preset format, and / or, converting the enumeration class field or the set class field into the corresponding business option description text, and / or, parsing the field validation rules and converting them into constraints described in natural language;

[0013] Based on the foreign keys in the relational database, the inter-table relationship information between the target business tables is identified and extracted. The relationship strength is scored based on at least one dimension, namely, data interaction frequency, business logic dependency, and field association cardinality.

[0014] Optionally, the metadata description knowledge is constructed, including:

[0015] Based on the table structure information and the field attribute information, natural language description text containing table-level metadata and field-level metadata is generated according to a predefined metadata template to obtain the metadata description knowledge. The table-level metadata includes the table's basic attributes and dynamic attributes, wherein the dynamic attributes include data volume statistics and / or data update frequency information. The field-level metadata includes the field's definition attributes and business examples, wherein the business examples are descriptions of the business meaning of field values ​​based on predefined rules.

[0016] Optionally, the association rule knowledge is constructed, including:

[0017] Based on the inter-table relationship information, corresponding natural language association rule description text is generated according to the association type to obtain the association rule knowledge; the association types include one-to-one, one-to-many, many-to-one, and many-to-many.

[0018] Optionally, constructing the semantic knowledge of the instance includes:

[0019] Read the business data content from the relational database;

[0020] According to predefined semantic mapping rules, the original data values ​​of the target fields in the business data content are converted into natural language description text with business meaning; the semantic mapping rules include enumerated value mapping based on dynamic mapping dictionary, and mapping that extends the date field with time dimension.

[0021] The descriptive text is associated and combined with the field attribute information corresponding to the target field to generate instance semantic knowledge of the business data content.

[0022] Optionally, after constructing the knowledge set for describing data semantics, the method further includes:

[0023] The constructed knowledge set is scored with confidence, and knowledge below the confidence threshold is removed based on the scoring results; the confidence score is based on at least one dimension of metadata integrity, association rule accuracy and semantic mapping consistency.

[0024] Optionally, converting the knowledge set into unstructured knowledge units that the retrieval-enhanced generation system can retrieve includes:

[0025] For a single business table, a single table document object is constructed using the metadata description knowledge and the instance semantic knowledge, and a data statistics node is added to the single table document object to record the data distribution characteristics of the single business table;

[0026] For multiple business tables that are related, the content of the related parties in the multiple business tables is merged according to the association strength score in the association rule knowledge to generate an associated document object;

[0027] The single-table document object or the associated document object is imported into the retrieval enhancement generation system to obtain unstructured knowledge units that the retrieval enhancement generation system can retrieve, and the unstructured knowledge units are optimized and configured. The optimization configuration includes at least one of the following: configuring a custom word segmentation strategy for the text content field of the unstructured knowledge unit, establishing a thesaurus of business keywords for the unstructured knowledge unit, and setting a document lifespan field for the unstructured knowledge unit.

[0028] Optionally, an incremental synchronization mechanism based on data change events of the relational database is established to update the unstructured knowledge units based on the incremental synchronization mechanism, including:

[0029] The incremental data acquisition component monitors the transaction log of the relational database to capture data change events for the target business table; the data change events include insertion, update, and deletion.

[0030] For the captured data change events, based on the event type and change content, the step of constructing a knowledge set to describe the data semantics based on the table structure information, the field attribute information, and the inter-table relationship information is re-executed to generate incremental knowledge;

[0031] After converting the incremental knowledge into unstructured knowledge units that the retrieval enhancement generation system can retrieve, the corresponding existing knowledge units in the retrieval enhancement generation system are inserted, updated, or deleted.

[0032] Optionally, the relational data import method suitable for retrieval enhancement generation systems further includes:

[0033] The system periodically verifies the data consistency between the relational database and the retrieval enhancement generation system, monitors the size of the retrieval index in the retrieval enhancement generation system, and outputs an alarm message when the index size exceeds a preset threshold.

[0034] Secondly, this application discloses a relational data import device suitable for a retrieval enhancement generation system, comprising:

[0035] The metadata parsing module is used to parse metadata in relational databases to obtain table structure information, field attribute information, and inter-table relationship information;

[0036] The knowledge construction module is used to construct a knowledge set for describing data semantics based on the table structure information, the field attribute information, and the inter-table relationship information; the knowledge set includes: metadata description knowledge generated based on the table structure information and the field attribute information, association rule knowledge generated based on the inter-table relationship information, and instance semantic knowledge generated based on the field attribute information and business data content.

[0037] The format conversion module is used to convert the knowledge set into unstructured knowledge units that the retrieval enhancement generation system can retrieve;

[0038] The incremental synchronization module is used to establish an incremental synchronization mechanism based on data change events of the relational database, so as to update the unstructured knowledge units based on the incremental synchronization mechanism.

[0039] Thirdly, this application discloses an electronic device comprising a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the relational data import method for a retrieval enhancement generation system as described above.

[0040] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the relational data import method for a retrieval-enhanced generation system as described above.

[0041] The beneficial effects of this application are as follows: By acquiring table structure information, field attribute information, and inter-table relationship information, the metadata information of relational data is fully preserved, providing a precise framework for subsequent knowledge construction. Based on this, a multi-dimensional, semantic knowledge system of "metadata-association rules-business data" is constructed, effectively overcoming the problems of knowledge fragmentation and loss of background information caused by traditional simple export methods, enabling the retrieval enhancement generation system to understand the business logic and context behind the data. By accurately converting structured data into suitable unstructured knowledge units, the format conflict between the two is resolved, improving the efficiency and accuracy of knowledge retrieval. Furthermore, the introduced incremental synchronization mechanism ensures that the knowledge base in the retrieval enhancement generation system remains consistent with the source database in real time, avoiding generation errors caused by outdated data. Therefore, the overall retrieval enhancement generation system significantly enhances the depth of knowledge utilization, the accuracy of responses, and the system's practicality in processing structured data scenarios.

[0042] In addition, the relational data import device, equipment and storage medium provided in this application are applicable to the above-mentioned relational data import method applicable to the retrieval enhancement generation system, and have the same effect. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This application discloses a flowchart of a relational data import method applicable to a retrieval enhancement generation system;

[0045] Figure 2 This is a flowchart illustrating the deep analysis of a relational data structure disclosed in this application;

[0046] Figure 3 This application discloses a flowchart for converting structured data to unstructured data.

[0047] Figure 4 This is a schematic diagram of a relational data import device suitable for a retrieval enhancement generation system disclosed in this application;

[0048] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Currently, importing relational data into RAG systems suffers from issues such as knowledge fragmentation, metadata loss, and untimely updates. To address these problems, this application provides a relational data import solution suitable for retrieval enhancement generation systems. Through data structure parsing, multi-dimensional knowledge extraction, structured-to-unstructured processing, and incremental update mechanisms, it transforms table structures, field relationships, and business data from relational databases into knowledge units that can be efficiently retrieved by the RAG system. This achieves accurate and dynamic import of relational data into the RAG system, improving the RAG system's efficiency in utilizing structured data and the accuracy of generated responses.

[0051] This invention discloses a method for importing relational data into a retrieval enhancement generation system. Taking the commonly used relational database MySQL and ElasticSearch, which is often used for knowledge storage in the RAG system, as examples, the method achieves accurate import of relational data through the following stage process. (See attached document.) Figure 1 As shown, the method includes:

[0052] Step S1: Parse the metadata in the relational database to obtain table structure information, field attribute information, and inter-table relationship information.

[0053] The first phase addresses the issues of lost metadata and fragmented relationships through in-depth analysis of relational data structures. This step builds a structured understanding of relational data, laying the foundation for subsequent knowledge extraction. Specifically, for example... Figure 2 As shown, the process of parsing metadata in a relational database includes the following steps:

[0054] Step S11: Establish a connection with the relational database through the database connection interface to obtain access permissions to the system tables in the relational database.

[0055] First, select the appropriate database connection interface, such as ODBC or JDBC, to establish a connection with the relational database and obtain access permissions to the database system tables. Ensure that the database connection user has permission to access the table metadata or the system tables that store table metadata, such as the `information_schema` system table in MySQL.

[0056] Step S12: Query the system tables based on preset filtering conditions to filter out the target business tables, and classify the target business tables by importance based on preset business rules to determine the table structure information.

[0057] Secondly, when importing relational data into the RAG system, target business tables are selected by setting filtering conditions, achieving intelligent table structure filtering. For example, when querying information_schema.TABLES, the table type filter condition is set to TABLE_TYPE='BASE TABLE' to exclude views; the table comment filter condition is set to TABLE_COMMENT NOT LIKE '%test%' to remove test tables. After simultaneously satisfying the above two filtering conditions, excluding views and test tables, the target business tables are determined. Then, the filtered tables are classified according to business importance, providing a basis for subsequent resource allocation; for example, the importance level of the master order table is higher than that of the order log table.

[0058] Step S13: Extract the field attribute information of the target business table from the system table, and perform semantic transformation on the field attribute information in the target format to generate enhanced field attribute information with clear business meaning.

[0059] Furthermore, the field attribute information of the target business table is extracted, and the data with internal system conventions and specific formats is converted into a representation with clear business meaning. The field attribute information includes at least the field name, data type, and constraints. The semantic conversion includes: converting numeric timestamps into date and time strings in a preset format, and / or converting enumerated or set-type fields into corresponding business option description text, and / or parsing field validation rules and converting them into constraints described in natural language.

[0060] For example, COLUMNS is a core system table in the built-in MySQL database information_schema. It acts as the database's "data dictionary," storing the field (column) definition information for every table in the database. This table records extremely detailed metadata. When extracting field information from the COLUMNS table, system-standard format data is converted into expressions with clear business meaning. For example, a timestamp in numeric format is converted into the standard string YYYY-MM-DD HH24:MM:SS. Enumeration types (ENUM) and set types (SET) are handled specially; for example, the ENUM('pending', 'paid', 'shipped') of the order_status field is converted into "status options: pending payment, paid, shipped". Field validation rules are parsed through the CHECK_CONSTRAINTS table, such as the constraint "amount>0" being converted into "the amount field value must be greater than 0". CHECK_CONSTRAINTS is a core system table in the built-in MySQL database information_schema. It is specifically used to store and query detailed information about all check constraints defined in the database. It is the database's "rulebook," recording which columns must meet what conditions.

[0061] Step S14: Based on foreign keys in the relational database, identify and extract inter-table relationship information between target business tables, and score the relationship strength based on at least one dimension of data interaction frequency, business logic dependency, and field association cardinality.

[0062] Finally, quantitative modeling of relationships is performed. An ER (Entity-Relationship) graph generated based on foreign keys is used to introduce a relationship strength scoring mechanism to obtain information on inter-table relationships. These relationships are evaluated from three dimensions: ① data interaction frequency (e.g., the number of JOIN queries between the daily order and user tables); ② business logic dependency (e.g., the order table must be joined with the user table to generate complete order information); ③ field association cardinality, which focuses on the number or proportion of associated records to quantify relationship strength (e.g., one-to-many relationships have a higher weight than one-to-one relationships).

[0063] For example, the scoring range is 0-10 points, with 0-3 points assigned to the data interaction frequency dimension, 0-4 points to the business logic dependency dimension, and 0-3 points to the field association cardinality dimension. The scores of each dimension are added together to obtain the association strength score. Furthermore, the association strength score can be used to divide the association into three levels: strong, medium, and weak. A score of 8 or above is considered a strong association (e.g., order_main and user_account); 5-7 points is a medium association (e.g., order_main and coupon); and below 4 points is a weak association (e.g., order_main and system_config). The scoring result affects the priority of subsequent knowledge fusion.

[0064] Step S2: Based on the table structure information, the field attribute information, and the inter-table relationship information, construct a knowledge set for describing data semantics; the knowledge set includes: metadata description knowledge generated based on the table structure information and the field attribute information, association rule knowledge generated based on the inter-table relationship information, and instance semantic knowledge generated based on the field attribute information and business data content.

[0065] In this step, to achieve the second stage of relational data import, multi-dimensional knowledge is extracted from the metadata layer, association layer, and business data layer to address the deficiency of insufficient knowledge extraction. By constructing a three-layer knowledge system of "metadata-association rules-business data," the semantic transformation of structured data is achieved.

[0066] In the first specific implementation, metadata description knowledge is generated based on table structure information and field attribute information to describe the data framework and address the issue of missing metadata such as table structure and constraints. In this implementation, the table structure information of the target business tables (excluding views and test tables) selected in step S12 and the hierarchical results determines the priority of metadata extraction (tables with high business importance are extracted first); the field attribute information extracted in step S13 (such as optional values ​​for enumeration / set types, field validation rules, and timestamp format conversion results) is directly filled into the metadata template to generate the final metadata description knowledge unit. Specifically, the process of constructing metadata description knowledge includes:

[0067] Based on table structure information and field attribute information, natural language description text containing table-level metadata and field-level metadata is generated according to a predefined metadata template to obtain metadata description knowledge.

[0068] In this implementation, metadata describes knowledge by structuring the metadata knowledge and filling the extracted table and field information into a preset template by developing a metadata template engine.

[0069] Example, table-level template example: [Table Name] {TABLE_NAME};

Business Meaning

Update Frequency

[0070] The table-level metadata in the table-level template includes the table's basic attributes and dynamic attributes. The dynamic attributes include data volume statistics and / or data update frequency information.

[0071] For example, a field-level template includes "field type + constraint rule + business example", such as: [Field Name] user_phone; [Type] VARCHAR(20);

Constraints

Business Example

[0072] The field-level metadata in the field-level template includes the field's definition attributes and business examples. The business examples are descriptions of the business meaning of the field values ​​based on predefined rules.

[0073] In the second specific implementation, association rule knowledge is generated based on inter-table relationship information to describe data logic and solve the problem of knowledge fragmentation caused by ignoring inter-table relationships. In this implementation, the association model built based on foreign keys and the three-dimensional association strength score (strong / medium / weak level) in step S14 are the core basis for the semanticization of association rules. In this implementation, corresponding natural language rules need to be generated according to the association type (one-to-one / one-to-many / many-to-one / many-to-many) and association strength. At the same time, the association cardinality and business dependency information in step S14 can also be used to enrich the business instances in the association rules. Specifically, the process of constructing association rule knowledge includes:

[0074] Based on the inter-table relationship information, corresponding natural language association rule description text is generated according to the association type to obtain association rule knowledge.

[0075] In this implementation, association rule knowledge refers to the semanticization of association rules by generating natural language rules for different association types. Association types describe the logical structure of the relationships between tables and determine the template type for knowledge representation. Association types include one-to-one, one-to-many, many-to-one, and many-to-many.

[0076] Example: Many-to-one association rule template: "When it is necessary to retrieve the information of the {main table} corresponding to the {sub-table field} in the {sub-table}, the association is established by {sub-table foreign key} = {main table primary key}, for example, {sub-table instance} is associated with {main table instance}." Example of a many-to-many association rule (e.g., user and role through the user_role intermediate table): "Users and roles have a many-to-many relationship. The association is established through the user_id and role_id fields of the intermediate table user_role. A user can have multiple roles (e.g., user ID=1001 is associated with role IDs=201 and 202)."

[0077] In the third specific implementation, instance semantic knowledge generated based on field attribute information and business data content is used to deeply describe the data content, solving the problem of only extracting field values. It combines metadata description knowledge, reflecting knowledge fusion. In this implementation, step S13's standardization of field formats (such as converting timestamps to strings and enumeration values ​​to business definitions) and the parsing results of validation rules provide a mapping foundation for enhancing the semantics of business data. Specifically, the process of constructing instance semantic knowledge includes:

[0078] Read the business data content from the relational database;

[0079] According to predefined semantic mapping rules, the original data values ​​of the target fields in the business data content are converted into natural language description text with business meaning; the semantic mapping rules include enumerated value mapping based on dynamic mapping dictionary, and mapping that extends the date field with time dimension.

[0080] The descriptive text is associated and combined with the field attribute information corresponding to the target field to generate instance semantic knowledge of the business data content.

[0081] In this implementation, business data is first acquired by reading it from a relational database. Next, semantic transformation of the data values ​​is performed, converting the original field values ​​into semantic descriptions according to predefined semantic mapping rules. Finally, an instance description is generated.

[0082] Predefined semantic mapping rules may be extracted from field attribute information (such as the meaning of each value in an enumeration type) or configured externally (such as an Excel mapping table). In one feasible implementation, a dynamic mapping dictionary is developed to store the correspondence between field values ​​and semantics, supporting Excel import and updates. For example, the mapping rules for the `user_level` field are shown in Table 1: Table 1 Mapping rules for the user_level field The date field is expanded with a time dimension, such as create_time='2023-10-01' is transformed into "Creation time: October 1, 2023 (Sunday, National Day holiday)", which enhances the time semantics.

[0083] In the actual process of generating semantic knowledge for instances, the first step is to parse field attribute information (such as field names, data types, constraints, enumeration value lists, etc.). The second step is to translate or map the field attributes into business semantic rules. For example, the enumeration values ​​(1, 2, 3) of the field `user_level` are mapped to the rules: 1 → Regular Member, 2 → Gold Member, 3 → Diamond Member. The third step is to apply these business semantic rules to specific business data values ​​to generate the final semantic description. For example, if the business data `user_level=2`, after applying the rules, the semantic knowledge "Gold Member" is generated.

[0084] It should be noted that this application embodiment also provides a knowledge confidence scoring mechanism. Specifically, the constructed knowledge set is scored with confidence, and knowledge below the confidence threshold is removed based on the scoring results; the confidence scoring is based on at least one dimension of metadata integrity, association rule accuracy, and semantic mapping consistency.

[0085] The knowledge quality verification is evaluated across three dimensions: ① Metadata completeness (whether it includes comments for all required fields); for example, for the metadata completeness dimension, a full score is awarded if the metadata knowledge unit includes all required fields such as table business meaning and field constraint rules, and a deduction is made for each missing field; ② Accuracy of association rules (whether circular associations exist); for example, for the accuracy of association rules, a full score is awarded if there are no circular associations or incorrect association fields, and a low confidence level is determined if there are logical errors in association; ③ Semantic consistency of business data (whether the mapping is unique); for example, for the semantic consistency of business data, a full score is awarded if the semantic description of the business data does not conflict with the field verification rules, and a deduction is made for any conflicts. Finally, invalid knowledge is eliminated based on the combined scores of the three dimensions.

[0086] As can be seen, the three-layer knowledge system of "metadata - association rules - business data" constructed in this step is a templated and semantic transformation completed based on the parsing results of step S1. The knowledge confidence scoring mechanism relies on dimensions such as the accuracy of association rules to accurately filter invalid knowledge. Compared with the traditional simple splicing method, it significantly improves the completeness and accuracy of knowledge and eliminates the negative impact of knowledge fragmentation on RAG's generated answers. At the same time, it extracts metadata, association rules, and business data to avoid knowledge omissions, enabling the RAG system to understand the data background and logic.

[0087] Step S3: Convert the knowledge set into unstructured knowledge units that the retrieval enhancement generation system can retrieve, and establish an incremental synchronization mechanism based on data change events of the relational database, so as to update the unstructured knowledge units based on the incremental synchronization mechanism.

[0088] This step is to achieve the third stage of relational data import (conversion of structured knowledge to RAG retrieval format) and the fourth stage (incremental update and consistency verification based on database logs).

[0089] In this embodiment of the application, taking the conversion of structured data to ElasticSearch format as an example, efficient adaptation of knowledge units to the retrieval system is achieved through format conversion and index optimization. For example... Figure 3 As shown, the process of converting the knowledge set into unstructured knowledge units that can be retrieved by the retrieval enhancement generation system includes the following steps:

[0090] Step S31: For a single business table, construct a single table document object using the metadata description knowledge and the instance semantic knowledge, and add a data statistics node to the single table document object to record the data distribution characteristics of the single business table.

[0091] In this step, a single-table document object is generated in JSON format. The single-table JSON document contains the following three core nodes: _meta (metadata), _fields (field information), and _records (business data). A new _statistics node is added to record data distribution characteristics, for example: { "_statistics": { "gender_distribution": {"male": 62%, "female": 38%}, "age_range": {"18-25": 23%, "26-35": 51%, "36+": 26%} } }

[0092] This statistical information can help the RAG system generate more accurate answers (such as "62% of users in the system are male"). Specifically, the single-table document object generated for a single business table includes header information generated based on metadata description knowledge, core content generated based on instance semantic knowledge, and feature information generated based on business data statistics.

[0093] Step S32: For multiple business tables with related relationships, the content of the related parties in the multiple business tables is merged according to the association strength score in the association rule knowledge to generate a related document object.

[0094] In this step, associated document objects are generated. An associated document object is a document generated by merging documents from two or more associated tables based on the association strength and varying fusion depths. The associated document object includes a `_association` node, which records information such as association type, association strength, update time, and association source.

[0095] Specifically, when constructing associated document objects based on the association strength score: if the association is strong, all core knowledge content of both parties is deeply integrated; if the association is medium, some core knowledge content of both parties is selectively integrated; if the association is weak, the association relationship is only recorded in the associated information node. For example, strongly associated documents undergo deep integration: for instance, the associated document between an order and a user includes all fields from both parties; medium-related documents only integrate core fields: for instance, the association between an order and a coupon only integrates the coupon amount and validity period; weak-related documents only record the association relationship without integrating content. The integrated document is marked with a _association node to indicate the association information. "_association": { "type": "many-to-one", "strength": 9, "update_time": "2023-10-01T12:00:00", "source": "FK_ORDER_USER }

[0096] Step S33: Import the single-table document object or the associated document object into the retrieval enhancement generation system to obtain unstructured knowledge units that the retrieval enhancement generation system can retrieve, and optimize the configuration of the unstructured knowledge units.

[0097] In this step, when importing the generated document object into the retrieval enhancement generation system, at least one of the following configurations is required for the unstructured knowledge unit in the retrieval enhancement generation system: configure a custom word segmentation strategy for the unstructured knowledge unit, establish a thesaurus of business keywords for the unstructured knowledge unit, map the synonyms of business keywords to the same root word, and set a document lifespan field for the unstructured knowledge unit to automatically clean up documents that have not been updated for more than a preset time.

[0098] For example, fine-grained configuration of an ElasticSearch index: the index mysql_rag_index uses a custom word segmentation strategy, with the following settings for the content field: "analyzer": { "custom_ik": { "type": "custom", "tokenizer": "ik_max_word", "filter": ["lowercase", "word_delimiter"] } } "search_analyzer": "ik_smart".

[0099] Create a thesaurus for business keywords (such as "order status" and "user level"), for example, "order placement time" = "order creation time" to improve search recall. Set the _ttl field (time to live) to automatically clean up historical data documents that have not been updated for more than 365 days.

[0100] In summary, by converting structured data to unstructured data, the format conflict between relational data and the RAG system is resolved, thereby improving retrieval efficiency.

[0101] Furthermore, in this embodiment, the fourth stage of importing relational data is a binlog-based incremental update process. By constructing a real-time, reliable incremental synchronization mechanism, consistency between the RAG system knowledge and the relational database data is ensured. Specifically, establishing an incremental synchronization mechanism based on data change events in the relational database, so that the process of updating the unstructured knowledge units based on the incremental synchronization mechanism includes the following steps:

[0102] The incremental data acquisition component monitors the transaction log of the relational database to capture data change events for the target business table; the data change events include insertion, update, and deletion.

[0103] For the captured data change events, based on the event type and change content, the step of constructing a knowledge set to describe the data semantics based on the table structure information, the field attribute information, and the inter-table relationship information is re-executed to generate incremental knowledge;

[0104] After converting the incremental knowledge into unstructured knowledge units that the retrieval enhancement generation system can retrieve, the corresponding existing knowledge units in the retrieval enhancement generation system are inserted, updated, or deleted.

[0105] In this embodiment, incremental data is continuously read using an incremental data acquisition component. Commonly used incremental acquisition components such as Debezium and Canel support multiple databases. The main objective is to obtain the add, delete, and modify events of the tables of interest, and these events need to be filtered.

[0106] Specifically, the captured data change events need to be re-executed in step S2 to construct semantic knowledge and generate incremental knowledge; the precise processing procedure for data change events is illustrated below:

[0107] (1) For NSERT events: extract all field values ​​of the new record, call the instance semantic knowledge in step S2 to generate a complete semantic description, for example, when adding a user record, generate "User ID=2001, Name=Wang Wu, Registration time 2023-10-02, Department=Technical Department";

[0108] (2) For UPDATE events: The "field-level differential" algorithm is used to extract only the changed fields (such as order_status changing from 1 to 2) and generate an incremental description "the status of order ID=1001 is updated from 'pending payment' to 'paid', update time 2023-10-02 15:30:00", avoiding the waste of resources from full updates;

[0109] (3) For the DELETE event: record the primary key and association of the deleted record, generate the deletion command "delete all associated documents of user ID=1008 (including user basic information, order records, membership level)", and simultaneously delete related documents in ElasticSearch.

[0110] In summary, the incremental update mechanism ensures that the knowledge and relational data of the RAG system are synchronized, avoiding incorrect answers due to data lag; compared with full import, incremental update reduces the amount of data processing and lowers the consumption of system resources.

[0111] It should be noted that the data consistency between the relational database and the retrieval enhancement generation system needs to be verified periodically to ensure the atomicity of incremental data in the relational database and operations on the knowledge base. Additionally, the size of the retrieval index in the retrieval enhancement generation system needs to be monitored, and an alarm message should be output when the index size exceeds a preset threshold to prevent performance degradation caused by an excessively large single index.

[0112] Through the collaborative processing of the above four stages, the full initialization and incremental update of relational data to the RAG system were realized, providing the RAG system with high-quality structured knowledge input.

[0113] The beneficial effects of this application are as follows: By acquiring table structure information, field attribute information, and inter-table relationship information, the metadata information of relational data is fully preserved, providing a precise framework for subsequent knowledge construction. Based on this, a multi-dimensional, semantic knowledge system of "metadata-association rules-business data" is constructed, effectively overcoming the problems of knowledge fragmentation and loss of background information caused by traditional simple export methods, enabling the retrieval enhancement generation system to understand the business logic and context behind the data. By accurately converting structured data into suitable unstructured knowledge units, the format conflict between the two is resolved, improving the efficiency and accuracy of knowledge retrieval. Furthermore, the introduced incremental synchronization mechanism ensures that the knowledge base in the retrieval enhancement generation system remains consistent with the source database in real time, avoiding generation errors caused by outdated data. Therefore, the overall retrieval enhancement generation system significantly enhances the depth of knowledge utilization, the accuracy of responses, and the system's practicality in processing structured data scenarios.

[0114] Accordingly, embodiments of this application also disclose a relational data import device suitable for retrieval enhancement generation systems, see [link to relevant documentation]. Figure 4 As shown, the device includes:

[0115] Metadata parsing module 11 is used to parse metadata in relational databases to obtain table structure information, field attribute information, and inter-table relationship information;

[0116] The knowledge construction module 12 is used to construct a knowledge set for describing data semantics based on the table structure information, the field attribute information, and the inter-table relationship information; the knowledge set includes: metadata description knowledge generated based on the table structure information and the field attribute information, association rule knowledge generated based on the inter-table relationship information, and instance semantic knowledge generated based on the field attribute information and business data content.

[0117] Format conversion module 13 is used to convert the knowledge set into unstructured knowledge units that can be retrieved by the retrieval enhancement generation system;

[0118] The incremental synchronization module 14 is used to establish an incremental synchronization mechanism based on data change events of the relational database, so as to update the unstructured knowledge unit based on the incremental synchronization mechanism.

[0119] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0120] Therefore, the above-described solution in this embodiment, by acquiring table structure information, field attribute information, and inter-table relationship information, completely preserves the metadata information of relational data, providing a precise framework for subsequent knowledge construction. Based on this, a multi-dimensional, semantic knowledge system of "metadata-association rules-business data" is constructed, effectively overcoming the fragmentation of knowledge and loss of background information caused by traditional simple export methods. This enables the retrieval enhancement generation system to understand the business logic and context behind the data. By accurately converting structured data into suitable unstructured knowledge units, the format conflict between the two is resolved, improving the efficiency and accuracy of knowledge retrieval. Furthermore, the introduced incremental synchronization mechanism ensures that the knowledge base in the retrieval enhancement generation system remains consistent with the source database in real time, avoiding generation errors caused by outdated data. This significantly enhances the depth of knowledge utilization, accuracy of responses, and system practicality of the retrieval enhancement generation system in processing structured data scenarios.

[0121] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0122] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the relational data import method for retrieval enhancement generation systems disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0123] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0124] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it can include an operating system 221, computer programs 222, and data 223, etc. The data 223 can include various types of data. The storage method can be temporary storage or permanent storage.

[0125] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the relational data import method for the retrieval enhancement generation system disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0126] Furthermore, this application also discloses a computer-readable storage medium, which includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, optical disks, or any other form of storage medium known in the art. When the computer program is executed by a processor, it implements the aforementioned relational data import method suitable for a retrieval-enhanced generation system. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0127] Furthermore, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the above-described methods for importing relational data into a retrieval-enhanced generation system.

[0128] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0129] The steps of the relational data import method or algorithm for retrieval enhancement generation systems described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0130] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The foregoing has provided a detailed description of the relational data import method, apparatus, device, and medium applicable to retrieval enhancement generation systems provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for importing relational data into a retrieval enhancement generation system, characterized in that, include: Parse metadata in a relational database to obtain table structure information, field attribute information, and inter-table relationships; Based on the table structure information, the field attribute information, and the inter-table relationship information, a knowledge set is constructed to describe the semantics of the data. The knowledge set includes: metadata description knowledge generated based on the table structure information and the field attribute information, association rule knowledge generated based on the inter-table relationship information, and instance semantic knowledge generated based on the field attribute information and business data content. The knowledge set is converted into unstructured knowledge units that can be retrieved by the retrieval enhancement generation system, and an incremental synchronization mechanism based on data change events of the relational database is established so as to update the unstructured knowledge units based on the incremental synchronization mechanism.

2. The relational data import method for a retrieval enhancement generation system according to claim 1, characterized in that, The metadata in the relational database is parsed to obtain table structure information, field attribute information, and inter-table relationship information, including: Establish a connection with a relational database through a database connection interface to obtain access permissions to the system tables in the relational database; The system tables are queried based on preset filtering conditions to filter out target business tables, and the target business tables are classified by importance based on preset business rules to determine the table structure information. The system table extracts the field attribute information of the target business table, and performs semantic transformation on the target format field attribute information to generate enhanced field attribute information with clear business meaning; the field attribute information includes at least the field name, data type, and constraints; wherein, the semantic transformation includes: converting the numeric format timestamp into a date and time string in a preset format, and / or, converting the enumeration class field or the set class field into the corresponding business option description text, and / or, parsing the field validation rules and converting them into constraints described in natural language; Based on the foreign keys in the relational database, the inter-table relationship information between the target business tables is identified and extracted. The relationship strength is scored based on at least one dimension, namely, data interaction frequency, business logic dependency, and field association cardinality.

3. The relational data import method for a retrieval enhancement generation system according to claim 1, characterized in that, Constructing the metadata description knowledge includes: Based on the table structure information and the field attribute information, natural language description text containing table-level metadata and field-level metadata is generated according to a predefined metadata template to obtain the metadata description knowledge. The table-level metadata includes the table's basic attributes and dynamic attributes, wherein the dynamic attributes include data volume statistics and / or data update frequency information. The field-level metadata includes the field's definition attributes and business examples, wherein the business examples are descriptions of the business meaning of field values ​​based on predefined rules.

4. The relational data import method for a retrieval enhancement generation system according to claim 1, characterized in that, Constructing the association rule knowledge includes: Based on the inter-table relationship information, corresponding natural language association rule description text is generated according to the association type to obtain the association rule knowledge; the association types include one-to-one, one-to-many, many-to-one, and many-to-many.

5. The relational data import method for a retrieval enhancement generation system according to claim 1, characterized in that, Constructing the semantic knowledge of the instance includes: Read the business data content from the relational database; According to predefined semantic mapping rules, the original data values ​​of the target fields in the business data content are converted into natural language description text with business meaning; the semantic mapping rules include enumerated value mapping based on dynamic mapping dictionary, and mapping that extends the date field with time dimension. The descriptive text is associated and combined with the field attribute information corresponding to the target field to generate instance semantic knowledge of the business data content.

6. The relational data import method for a retrieval enhancement generation system according to claim 1, characterized in that, After constructing the knowledge set used to describe data semantics, the method further includes: The constructed knowledge set is scored with confidence, and knowledge below the confidence threshold is removed based on the scoring results; the confidence score is based on at least one dimension of metadata integrity, association rule accuracy and semantic mapping consistency.

7. The relational data import method for a retrieval enhancement generation system according to claim 1, characterized in that, The process of converting the knowledge set into unstructured knowledge units that the retrieval-enhanced generation system can retrieve includes: For a single business table, a single table document object is constructed using the metadata description knowledge and the instance semantic knowledge, and a data statistics node is added to the single table document object to record the data distribution characteristics of the single business table; For multiple business tables that are related, the content of the related parties in the multiple business tables is merged according to the association strength score in the association rule knowledge to generate an associated document object; The single-table document object or the associated document object is imported into the retrieval enhancement generation system to obtain unstructured knowledge units that the retrieval enhancement generation system can retrieve, and the unstructured knowledge units are optimized and configured. The optimization configuration includes at least one of the following: configuring a custom word segmentation strategy for the text content field of the unstructured knowledge unit, establishing a thesaurus of business keywords for the unstructured knowledge unit, and setting a document lifespan field for the unstructured knowledge unit.

8. The relational data import method for a retrieval enhancement generation system according to claim 2, characterized in that, Establish an incremental synchronization mechanism based on data change events of the relational database, so as to update the unstructured knowledge units based on the incremental synchronization mechanism, including: The incremental data acquisition component monitors the transaction log of the relational database to capture data change events for the target business table; the data change events include insertion, update, and deletion. For the captured data change events, based on the event type and change content, the step of constructing a knowledge set to describe the data semantics based on the table structure information, the field attribute information, and the inter-table relationship information is re-executed to generate incremental knowledge; After converting the incremental knowledge into unstructured knowledge units that the retrieval enhancement generation system can retrieve, the corresponding existing knowledge units in the retrieval enhancement generation system are inserted, updated, or deleted.

9. The method for importing relational data into a retrieval enhancement generation system according to any one of claims 1 to 8, characterized in that, Also includes: The system periodically verifies the data consistency between the relational database and the retrieval enhancement generation system, monitors the size of the retrieval index in the retrieval enhancement generation system, and outputs an alarm message when the index size exceeds a preset threshold.

10. A relational data import device suitable for a retrieval enhancement generation system, characterized in that, include: The metadata parsing module is used to parse metadata in relational databases to obtain table structure information, field attribute information, and inter-table relationship information; The knowledge construction module is used to construct a knowledge set to describe the semantics of the data based on the table structure information, the field attribute information, and the inter-table relationship information. The knowledge set includes: metadata description knowledge generated based on the table structure information and the field attribute information, association rule knowledge generated based on the inter-table relationship information, and instance semantic knowledge generated based on the field attribute information and business data content. The format conversion module is used to convert the knowledge set into unstructured knowledge units that the retrieval enhancement generation system can retrieve; The incremental synchronization module is used to establish an incremental synchronization mechanism based on data change events of the relational database, so as to update the unstructured knowledge units based on the incremental synchronization mechanism.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the relational data import method for a retrieval enhancement generation system as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the relational data import method for a retrieval-enhanced generation system as described in any one of claims 1 to 9.

Images