Structured data storage method based on incremental synchronization

Abstract

The invention discloses a structured data storage method based on incremental synchronization. The method comprises the steps of obtaining change data related to a data change event and associated information to which the change data belongs by responding to the data change event of at least one database; converting the change data into a data document which can be retrieved from the vector database, and determining a document identifier of the data document and document content written into the data document based on the associated information; according to the change data, document index information used for retrieving the data document is determined; wherein the document index information comprises sparse features, vector features and metadata features corresponding to the change data. According to the method and the device, the change data is synchronously updated to the corresponding vector database, so that computing resources are saved, the real-time performance of data synchronization is improved, and feedback response corresponding to the latest data is conveniently given subsequently based on the index information in the vector database.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a structured data storage method based on incremental synchronization. Background Technology

[0002] In today's digital operations environment, data has become an asset. Various business systems store relevant business data in relational databases. Meanwhile, with the development of artificial intelligence technology, all parties hope to leverage large-scale models to achieve functions such as intelligent question answering, intelligent customer service, and data analysis decision support, thereby improving operational efficiency and optimizing user experience.

[0003] Currently, the primary approach is to export all data from relational databases using periodic batch synchronization and store it in a database corresponding to the large model, facilitating subsequent indexing. However, this full export method not only requires processing all data in the relational database each time, resulting in significant redundant computation and wasted storage resources, but it is also time-consuming. Since data in relational databases is updated extremely frequently, the data stored in the database corresponding to the large model experiences a delay, making it impossible to provide feedback consistent with the latest data. Furthermore, existing technologies directly vectorize the structured data in relational databases, resulting in ambiguous semantic information in the vectorized data, which is detrimental to providing accurate feedback responses from the large model. Summary of the Invention

[0004] This invention provides a structured data storage method based on incremental synchronization, which realizes the synchronous update of changed data to the corresponding vector database, saves computing resources, improves the real-time performance of data synchronization, and facilitates subsequent feedback responses based on the index information in the vector database to provide the latest data.

[0005] According to one aspect of the present invention, a structured data storage method based on incremental synchronization is provided, the method comprising:

[0006] In response to a data change event in at least one database, the system retrieves the changed data related to the data change event and the associated information of the changed data. The associated information includes at least one or more of the following: table name, business index, operation type, transaction progress information, and data change completion time.

[0007] The changed data is converted into data documents that can be retrieved from the vector database, and the document identifier of the data document and the document content written into the data document are determined based on the association information.

[0008] Based on the changed data, determine the document index information used to retrieve data documents;

[0009] The document index information includes sparse features, vector features, and metadata features corresponding to the changed data.

[0010] According to another aspect of the present invention, a structured data storage device based on incremental synchronization is provided, the device comprising:

[0011] The data acquisition module is used to respond to a data change event in at least one database and acquire the changed data related to the data change event and the associated information of the changed data. The associated information includes at least one or more of the following: table name, business index, operation type, transaction progress information, and data change completion time.

[0012] The data conversion module is used to convert changed data into data documents that can be retrieved from the vector database, and to determine the document identifier of the data document and the document content to be written into the data document based on the association information.

[0013] The index information determination module is used to determine the document index information used to retrieve data documents based on the changed data.

[0014] The document index information includes sparse features, vector features, and metadata features corresponding to the changed data.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory that is communicatively connected to at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to execute the incremental synchronization-based structured data storage method of any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement the incremental synchronization-based structured data storage method of any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements a structured data storage method based on incremental synchronization as described in any embodiment of the present invention.

[0021] The technical solution of this invention, in response to a data change event in at least one database, acquires the changed data related to the data change event and the associated information of the changed data. The changed data is converted into a data document that can be retrieved from a vector database, and the document identifier and content of the data document are determined based on the associated information. Based on this, the changed data in the database is synchronized to the vector database, providing data support for subsequent feedback responses based on the data in the vector database. Document index information for indexing the data document is determined based on the changed data, facilitating the subsequent determination of the data document matching the request based on the document index information, thereby providing the best feedback response. This invention solves the problem of wasted computing and storage resources caused by full-data export synchronization in the prior art. By synchronizing only the changed data in the database, computing resources are saved and data synchronization time is shortened. Correspondingly, it solves the problem that data stored in the vector database has a certain delay, making it impossible to provide the best feedback response. By synchronizing the changed data and providing the corresponding document index information, it is convenient to quickly determine the data document matching the request based on the document index information, thereby providing a feedback response corresponding to the latest data document, improving the user experience, and ensuring the real-time nature and accuracy of the feedback.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a structured data storage method based on incremental synchronization provided in an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of a structured data storage method based on incremental synchronization provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of a structured data storage device based on incremental synchronization provided in an embodiment of the present invention;

[0027] Figure 4This is a schematic diagram of the structure of an electronic device that implements the incremental synchronization-based structured data storage method of this invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a structured data storage method based on incremental synchronization provided in Embodiment 1 of the present invention. This embodiment is applicable to synchronizing changed data to convert the changed data into data documents for storage in a database and provide corresponding document index information to facilitate subsequent request-based feedback responses. This method can be executed by a structured data storage device based on incremental synchronization, which can be implemented in hardware and / or software and can be configured in electronic devices such as mobile phones, computers, or servers. Figure 1 As shown, the method includes:

[0032] S110. In response to a data change event in at least one database, obtain the changed data related to the data change event and the associated information of the changed data.

[0033] Among these, a database can be used to store business data. In practical applications, there may be multiple business databases storing business data, and correspondingly, at least one of these databases may experience data changes. It should be noted that the business data in the database (business database) is typically stored in tabular form.

[0034] Data change events are related to database transactions. When a database transaction is detected to be in a completed state, the changed data corresponding to that completed transaction is determined. It should be noted that a database transaction consists of one or more database operations (such as insert, update, delete, query, etc.), which typically revolve around a business logic (such as resource transfer, order processing, etc.). The changed data is the business data that has been changed or updated corresponding to the most recently completed database transaction.

[0035] The associated information can be information related to the changed data. The associated information includes at least one or more of the following: table name, business index, operation type, transaction progress information, and data change completion time. The table name can be understood as the data table to which the changed data belongs. The business index can be the primary key or business key corresponding to the changed data. The operation type can be the database operation performed to obtain the changed data; for example, it can be the changed data obtained through an update database operation. Optionally, the operation type can be identified by I (Insert), U (Update), and D (Delete). The transaction progress information can be used to characterize the transaction status of the database transaction corresponding to the changed data; that is, the transaction progress information can determine whether the database transaction is in a completed state. Optionally, the transaction progress information can include: transaction identifier, log sequence number of the database change log corresponding to the database transaction, etc. The transaction identifier is used to uniquely identify the database transaction. The data change completion time can be used to characterize the time when the changed data was obtained. Optionally, the data change completion time can be the database transaction commit (completion) time.

[0036] Specifically, if data changes are detected in at least one database related to the business, the database change logs corresponding to each database are obtained, and the change logs of at least one database are analyzed to determine the changed data and the associated information corresponding to the changed data.

[0037] For example, for at least one database, a CDC (Change Data Capture) event is captured from the change log of each database to obtain the changed data and the associated information corresponding to the changed data. The associated information includes: table (table name), business_key (primary key or business key), op_type (operation type), before / after, txid (transaction identifier) / lsn (log sequence number), commit_ts (data change completion time), and partition_key (key used for data partitioning). The before and after events are typically used to describe the state of the data before and after the change. In the CDC event or database change log, before corresponds to the data before the change, and after corresponds to the data after the change, i.e., the changed data.

[0038] Optionally, since the obtained correlation information corresponds to at least one database, to ensure data consistency, the correlation information can be normalized based on a unified event model to obtain normalized correlation information. Correspondingly, a trace identifier (trace_id) can be set to identify the changed data and the corresponding data document. An idempotent key can also be set. It should be noted that an idempotent key is an identifier used to ensure that the same operation produces the same result when executed multiple times. Idempotent keys can avoid duplicate processing and prevent dirty reads. Optionally, if problems arise during the normalization of correlation information based on the unified event model, replay processing can be performed, and corresponding replay markers can be added. Replay markers are used to characterize the data source or processing status. Optionally, the trace identifier, idempotent key, and replay markers can also be used as correlation information for the changed data.

[0039] S120. Convert the changed data into a data document that can be retrieved from the vector database, and determine the document identifier of the data document and the document content to be written into the data document based on the association information.

[0040] The vector database is a database system specifically designed for storing, managing, and querying high-dimensional vector data. In this embodiment of the invention, the vector database stores data documents that have undergone vectorization processing (Embedding) and their corresponding vector representations to facilitate similarity retrieval. Optionally, the vector database may include multiple vectorized data documents. These data documents are primarily unstructured or semi-structured documents, stored in the vector database as high-dimensional vectors using embedding technology. When a request is received based on the large model, the corresponding data document can be determined based on the data documents stored in the vector database, allowing the large model to provide appropriate feedback based on the data document.

[0041] A document identifier can consist of two parts: a name representing the data document and a version representing the data document. The document content written into the data document can include semantically meaningful text representing the changes and data content corresponding to associated information.

[0042] Specifically, the changed data is converted into semantically meaningful text, which is then written into the data document as part of the document content. Based on the association information, a document identifier corresponding to the data document and its corresponding data content are determined. The data content corresponding to the association information is also written into the data document as part of the document content.

[0043] In this embodiment of the invention, the method of converting changed data into a data document may be: receiving pre-configured document conversion parameters; converting the changed data into a data document adapted to the document conversion parameters; wherein, the document conversion parameters include any one of line document parameters, aggregate document parameters, semantic template parameters, and indicator document parameters.

[0044] Document transformation parameters characterize how the data documents corresponding to changed data are stored in the vector database. Optionally, document transformation parameters include any one of the following: row document parameters, aggregated document parameters, semantic template parameters, and indicator document parameters. Row document parameters convert each row of changed data into a data document; aggregated document parameters can be used to generate medium-granularity documents by aggregating multiple row documents according to a specific dimension (such as time or business category); indicator document parameters typically convert changed data corresponding to the relevant business dimension into data documents. Semantic template parameters convert changed data into semantically meaningful text. It should be noted that document transformation parameters can be pre-configured or determined based on the analysis of the correlation information of the changed data using a unified event model.

[0045] Specifically, it receives pre-configured document transformation parameters. Based on these parameters, it determines how the structured change data will be stored in the vector database, storing the change data as data documents adapted to the transformation parameters.

[0046] For example, based on configurable granularity (document transformation parameters), such as line document granularity, aggregate document granularity, or indicator document granularity, it is determined whether the changed data is stored in the vector database as a line document, aggregate document, and / or indicator document, and the changed data is converted into semantically meaningful text through semantic template parameters for storage in the corresponding data document.

[0047] Optionally, the method to convert the changed data into a data document that is compatible with the document conversion parameters can be: formatting the changed data based on a preset template, and converting the formatted changed data into semantically meaningful text based on the document conversion parameters to obtain part of the document content in the data document.

[0048] The preset template can be a pre-set template used to format and adjust the changed data. Formatting and adjusting can transform the structured fields corresponding to the changed data into readable natural language. Semantic text can be the text obtained by concatenating the formatted and adjusted changed data based on document transformation parameters. It should be noted that here, document transformation parameters can be semantic template parameters. These semantic template parameters are used to concatenate the formatted and adjusted changed data to obtain semantic text.

[0049] Optionally, semantically meaningful text should include at least: field aliases, definitions, and standardized units. Field aliases convert the field names corresponding to the changed data into readable names. For example, converting `user_id` to `user ID`. Definitions are used to characterize the business calculation logic corresponding to the changed data. Standardized units can be units in a uniform format, such as a uniform time format of YYYY-MM-DD.

[0050] Specifically, since the changed data is usually structured fields, to facilitate accurate feedback from the subsequent large model based on the data document, the changed data can be formatted and adjusted using a preset template to obtain formatted changed data. The formatted changed data is then concatenated to obtain semantically meaningful text that matches the document conversion parameters, and this semantically meaningful text is used as part of the document content in the data document.

[0051] For example, if the changed data is {order_id: 123, status: 1}, it is converted into natural language text by combining the preset template and the semantic template parameter in the document conversion parameters: "The status of order number 123 has been changed to paid".

[0052] In this embodiment of the invention, the method for determining the document identifier and document content based on the association information may be as follows: the document identifier of the data document is determined based on the table name, business index, transaction progress information and data change completion time in the association information; wherein, the document identifier is used to determine the data document corresponding to the data change event in the vector database; the association information is analyzed to determine at least one metadata related to the business dimension and written into the data document.

[0053] The document identifier can consist of two parts: a name representing the data document and a version representing the corresponding data document. The name of the data document can be determined by the table name and business index, while the version of the data document can be determined by transaction progress information and data change completion time.

[0054] Metadata for at least one business dimension must include one or more of the following: applicable time range, accessible user permissions, the database and / or storage block to which the changed data belongs, and the business product corresponding to the changed data. The applicable time range characterizes the period during which the changed data can be used. Within the applicable time range, the changed data is the latest and valid business data. Accessible user permissions can be understood as the users who can access the changed data. That is, accessible user permissions limit which users are associated with the changed data and which users can access it. The storage block is the original storage area or storage space corresponding to the changed data. The business product is the business product corresponding to the changed data.

[0055] Specifically, based on the table name and business index in the association information, determine the data document name; based on the transaction progress information and data change completion time in the association information, determine the data document version. Based on the data document name and version, determine the document identifier of the data document. Analyze the association information to identify at least one metadata related to a business dimension. Write at least one metadata related to a business dimension into the data document.

[0056] For example, referring to the above example, the name of a data document, `doc_id`, can be determined by the table name `table` and the business index `business_key`, i.e., `doc_id = hash(table, business_key)`; the version corresponding to the data document, `version`, can be determined by the transaction identifier `txid` and the data change completion time `commit_ts` in the transaction progress information, i.e., `version = (commit_ts, txid)`. The associated information is then analyzed to determine the business dimension metadata corresponding to the changed data, and this business dimension metadata is written into the data document. For example, business dimension metadata may include: time (applicable time range), organization (accessible user permissions), region (the database and / or storage block to which the changed data belongs), product (the business product corresponding to the changed data), status, and categorical labels, etc.

[0057] S130. Based on the changed data, determine the document index information used to retrieve data documents.

[0058] The document index information includes sparse features, vector features, and metadata features corresponding to the changed data. Sparse features refer to feature vectors where the number of non-zero values ​​is much smaller than the total number of dimensions. Vector features are the feature vectors corresponding to the changed data. It should be noted that both vector features and sparse features are determined based on the document content corresponding to the changed data within the data document. Metadata features are at least one piece of metadata related to the business dimension, determined based on association information.

[0059] Specifically, the changed data is analyzed to determine the sparse and vector features corresponding to it. The associated information is also analyzed to identify at least one metadata element relevant to the business dimension, which is then used as a metadata feature. The sparse features, vector features, and metadata features are then used as document index information for the data document, facilitating rapid retrieval of the corresponding data document based on the document index information.

[0060] In this embodiment of the invention, the document index information can be determined by: determining sparse features based on keywords and word segmentation information corresponding to the changed data; obtaining vector features corresponding to the changed data based on the embedding model and the changed data; and determining document index information for retrieving data documents based on sparse features, vector features, and metadata features determined based on association information.

[0061] It should be noted that the document index information is determined after the data document is obtained. Therefore, the keywords, word segmentation information, sparse features, and vector features of the changed data can all be determined based on the semantically meaningful text in the data document corresponding to the changed data.

[0062] Keywords can be representative words or phrases extracted from the semantically meaningful text corresponding to the changed data. Word segmentation information can be obtained by segmenting the semantically meaningful text corresponding to the changed data. Optionally, the semantically meaningful text corresponding to the changed data can be segmented using the BM25 segmentation algorithm. The embedding model is used to vectorize the changed data or the semantically meaningful text corresponding to the changed data.

[0063] Specifically, keyword extraction is performed on the semantically meaningful text corresponding to the changed data to obtain keywords. The semantically meaningful text corresponding to the changed data is then segmented using the BM25 segmentation algorithm to obtain segmentation information. Sparse features are generated based on the segmentation information and keywords. Correspondingly, the changed data or the semantically meaningful text corresponding to the changed data is vectorized using an embedding model to obtain vector features. The association information with the changed data is analyzed to determine the metadata features corresponding to the association information. These metadata features, vector features, and sparse features are used as document index information for retrieving data documents.

[0064] The technical solution of this embodiment, in response to a data change event in at least one database, obtains the changed data related to the data change event and the associated information of the changed data. The changed data is converted into a data document that can be retrieved from the vector database, and the document identifier and content of the data document are determined based on the associated information. Based on this, the changed data in the database is synchronized to the vector database, providing data support for subsequent feedback responses based on the data in the vector database. According to the changed data, document index information for indexing the data document is determined, facilitating the subsequent determination of the data document matching the request based on the document index information, thereby providing the best feedback response. This invention solves the problem of wasted computing and storage resources caused by full-data export synchronization in the prior art. By synchronizing only the changed data in the database, computing resources are saved and data synchronization time is shortened. Correspondingly, it solves the problem that data stored in the vector database has a certain delay, making it impossible to provide the best feedback response. By synchronizing the changed data and providing the corresponding document index information, it is convenient to quickly determine the data document matching the request based on the document index information, thereby providing a feedback response corresponding to the latest data document, improving the user experience, and ensuring the real-time nature and accuracy of the feedback.

[0065] Example 2

[0066] Figure 2 This is a flowchart of a structured data storage method based on incremental synchronization provided in Embodiment 2 of the present invention. This embodiment, based on the above embodiments, allows for the verification and writing of data documents into the vector database after obtaining data documents that can be retrieved from the vector database and corresponding document index information. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2 As shown, the method includes:

[0067] S210. Cache the data document in the temporary cache area.

[0068] The temporary cache can be a storage space used to temporarily store data documents. Optionally, the temporary cache can be a temporary buffer message queue (such as Kafka) or a memory cache.

[0069] Specifically, the data document is cached in a temporary cache area. When the data document meets the document requirements in the temporary cache area, the data document is written to the vector database.

[0070] S220. Verify the data document based on the document identifier to which the data document belongs and the data change completion time.

[0071] Specifically, based on the document identifier to which the data document belongs and the data change completion time, version verification and data content verification are performed on the data document to obtain the verification result. If the verification result is successful, the data document is written to the vector database. Based on this, the problems of duplicate writing and incorrect writing of data documents can be avoided.

[0072] For example, the temporary cache area can be the preparation area of ​​the vector database. That is, the data document can be written to the preparation area of ​​the vector database first, and the data document can be processed according to the Offset Checkpoint and the High Watermark (HW) of the version. When the requirements for writing to the searchable area of ​​the vector database are met, the data document can be written to the searchable area of ​​the vector database, so that the data document can be retrieved based on the document index information later.

[0073] S230. Write the data document into the vector database and retrieve all first historical documents in the vector database to determine the second historical document that is associated with the data document.

[0074] The data documents stored in the vector database are used as the first historical documents. The association relationship includes at least fields whose field similarity meets a preset field similarity threshold. Field similarity can be used to characterize the degree of similarity between fields in the first historical document stored in the vector database and the currently written data document. The preset field similarity threshold can be a pre-set standard value for field similarity. The second historical document can be the first historical document in the vector database that has an association relationship with the data document.

[0075] Specifically, when a data document is written to the vector database, all first historical documents in the vector database can be retrieved based on the transaction identifier in the data change completion time and transaction progress information corresponding to the data document. This allows for the identification of a first historical document to be processed that matches the transaction identifier and / or the data change completion time of the data document. A similarity assessment is then performed between the fields in this first historical document and the fields in the data document to determine the field similarity. If the field similarity exceeds a preset field similarity threshold, a relationship is established between the first historical document and the data document, and this first historical document is designated as a second historical document.

[0076] S240. The vector database is updated by merging the second historical document and the data document.

[0077] Specifically, the second historical document and the data document are merged to group them into one category and store them in the same storage space, enabling updates to the vector database. This facilitates faster searching of similar documents within the same storage space of the vector database, improving the efficiency of subsequent response requests.

[0078] Optionally, embodiments of the present invention further include: retrieving the first historical document in the vector database based on the data change completion time `commit_ts` and the transaction identifier `txid` in the transaction progress information to determine whether there are any historical documents that conflict with the data document. Accordingly, for historical documents and data documents that conflict with each other, only the document with the latest data change completion time can be retained based on the data change completion time corresponding to the historical documents and data documents, or, based on the business databases corresponding to the historical documents and data documents, the document corresponding to the higher-priority business database can be retained. The above processing is recorded accordingly for easy subsequent traceability.

[0079] Additionally, it should be noted that only data documents with the latest version corresponding to their document identifier are retained in the vector database. That is, when writing a data document to the vector database, the document identifier corresponding to the data document is matched with the document identifiers corresponding to each first historical document in the vector database to identify at least one first historical document with the same doc_id but a different version. Based on the version of the data document and the version corresponding to at least one first historical document, at least one historical document is updated or replaced according to the data document to store the document corresponding to the latest version in the vector database. Preferably, if the version corresponding to the data document is determined to be the latest version, the data document is stored in the vector database, and at least one first historical document with the same doc_id but a different version is deleted. This deletion event is recorded. Correspondingly, the corresponding data in at least one original database can also be deleted.

[0080] Optionally, the method further includes: in response to receiving a request to query business data, searching for target data from a vector database based on the descriptive information carried in the request and providing feedback; wherein the descriptive information includes query time information.

[0081] Specifically, a request to query business data can be generated based on descriptive information entered by the user in the corresponding session interface. This descriptive information can characterize the business data the user expects to obtain. The target data can be feedback information corresponding to the descriptive information.

[0082] Specifically, when the large model associated with the vector database receives a request to query business data, it parses the request to determine the descriptive information carried by the request. The descriptive information and the corresponding query time information are then vectorized to obtain vectorized descriptive information. Based on the vectorized descriptive information, the corresponding target data is retrieved from the vector database and displayed in the appropriate session interface.

[0083] It should be noted that if there is no target data in the vector database that matches the applicable time range with the query time information, the data corresponding to the applicable time range that is closest to the query time information can be used as the target data, and the applicable time range corresponding to the target data can be returned.

[0084] The vector database includes multiple first historical documents, each containing document index information. Correspondingly, the method for finding target data based on description information can be: determining the original metadata that matches the request; determining the third historical document based on the metadata features of each first historical document and the original metadata; determining the features to be matched based on the request description information, and then determining the fourth historical document from the third historical documents based on these features; and finally, determining the target data based on the document content of the fourth historical document and providing feedback.

[0085] The original metadata can be the metadata corresponding to the requested description information. The third historical document can be the first historical document whose metadata features match the original metadata. The features to be matched can be vector features and / or sparse features corresponding to the description information. The fourth historical document can be the third historical document whose vector features and / or sparse features match the features to be matched. The target data can be determined based on the document content in the fourth historical document.

[0086] Specifically, the original metadata corresponding to the requested description information is determined. This original metadata is then matched against the metadata features corresponding to each first historical document in the vector database to determine at least one third historical document from the first historical documents based on the metadata matching results. Based on the requested description information, the description information is vectorized to determine the features to be matched corresponding to the description information.

[0087] When the feature to be matched is a vector feature, the vector feature corresponding to the descriptive information is matched with the vector feature corresponding to each third historical document to determine the fourth historical document that matches the vector feature.

[0088] When the feature to be matched is a sparse feature, the sparse feature corresponding to the description information is matched with the sparse feature corresponding to each third historical document to determine the fourth historical document that matches the sparse feature.

[0089] Accordingly, when the features to be matched include vector features and sparse features, feature matching is performed between the vector features corresponding to the descriptive information and the vector features corresponding to each third historical document to determine the third historical document whose vector features match. The sparse features corresponding to the third historical document whose vector features match are then matched with the sparse features corresponding to the descriptive information to obtain the fourth historical document.

[0090] Based on the content of the fourth historical document obtained, the target data corresponding to the request is determined and fed back. By determining the fourth historical document in this way, precise matching with the descriptive information is achieved, improving retrieval accuracy.

[0091] For example, in the retrieval phase, the original metadata adapted to the request is determined. Based on the original metadata, metadata filtering is performed on multiple first historical documents in the vector database to obtain at least one third historical document. Vector retrieval processing is performed on the at least one third historical document to determine the third historical document that matches the vector features of the description information. The BM25 word segmentation algorithm is used to determine the word segmentation information corresponding to the description information, and keywords corresponding to the description information are also determined. Based on the word segmentation information and keywords corresponding to the description information, sparse features corresponding to the description information are determined. Based on the sparse features corresponding to the description information, a fourth historical document matching the sparse features is determined from the third historical document matching the vector features of the description information. Based on the fourth historical document, explanatory information corresponding to the semantically meaningful text in the fourth historical document is determined, i.e., the target data. Correspondingly, if multiple fourth historical documents exist, the explanatory information can be determined based on the matching fields between the semantically meaningful text and the description information in the fourth historical document, or, based on the number of matching fields, the explanatory information can be determined sequentially from high to low number of matching fields and fed back sequentially.

[0092] It should be noted that if the request is a question-and-answer request based on business data, the target data returned can be determined based on the document content of the fourth historical document and the data in the business database corresponding to the fourth historical document. Accordingly, the target data includes: feedback information corresponding to the document content of the fourth historical document and data information corresponding to the data in the business database. It should also be noted that if the target data includes both of the above parts, a consistency arbiter can be enabled to perform consistency verification on the feedback information and the data information. If the verification result shows that the feedback information and the data information are consistent, the target data will be returned.

[0093] Additionally, upon receiving the description information, timing processing can be performed based on the query time information corresponding to the description information. If the target data is not determined within a preset time period after the query time information is detected, retrieval degradation processing can be performed to ensure the timeliness of the returned target data. This retrieval degradation processing can employ one or two of the following methods to obtain the target data: metadata-based matching, vector feature-based matching, and sparse feature-based matching.

[0094] Optionally, the description information includes query time information. The method for determining the third historical document may be based on the metadata characteristics of the first historical document, the applicable time range corresponding to the document identifier, and the original metadata.

[0095] Specifically, the process involves matching the original metadata corresponding to the descriptive information with the metadata features of multiple first historical documents in the vector database to obtain at least one first historical document with matching metadata. Based on the document identifier of the at least one first historical document with matching metadata, the applicable time range for each first historical document is determined. If the applicable time range of a first historical document includes the query time information, then that first historical document with matching metadata is designated as the third historical document. If the applicable time range of no first historical document includes the query time information, then the first historical document with the latest applicable time range is determined and designated as the third historical document.

[0096] Optionally, the method further includes recording the storage process of the changed data and the full-link information corresponding to the search process, so as to facilitate subsequent application in different large models and improve the reusability of the solution. For example, recording the causal link of data change event → data document → document index information → retrieval → target data can facilitate subsequent tracing, replay and solution reuse, and ensure the interpretability of the processing.

[0097] Optionally, the method further includes: determining the first frequently accessed historical document in the vector database based on the target data fed back within a preset number of times; caching the first frequently accessed historical document in the vector database to a hotspot dimension storage partition, so as to facilitate the subsequent priority determination of target data from the first historical document in the hotspot dimension partition, thereby improving the efficiency of target data feedback. Correspondingly, determining the first frequently accessed historical document in the vector database; merging and compressing the first historical document in the low-frequency access to cache in another storage partition of the vector database, thereby reducing storage costs.

[0098] It should be noted that the solutions corresponding to the embodiments of the present invention are applied in scenarios such as operation analysis, operation and maintenance alarms, data quality inspection, real-time Q&A of multi-source data, and related retrieval, so as to realize the storage of changed data in the corresponding scenarios and the corresponding target data based on the request feedback, thus ensuring the applicability of multiple scenarios.

[0099] The technical solution of this embodiment caches data documents in a temporary cache area and verifies the data documents based on their document identifier and data change completion time to obtain verification results. This avoids the problems of duplicate and erroneous data document writing, while ensuring the real-time nature and consistency of the data documents. The data documents are written to a vector database, and all first historical documents in the vector database are retrieved to identify second historical documents associated with the data documents. The vector database is updated by merging the second historical documents and the data documents. This ensures the consistency of documents stored in the vector database, avoids business conflicts during data document storage, and facilitates subsequent feedback of target data corresponding to requests.

[0100] Example 3

[0101] Figure 3 This is a schematic diagram of a structured data storage device based on incremental synchronization provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a data acquisition module 310, a data conversion module 320, and an index information determination module 330.

[0102] The data acquisition module 310 is used to acquire, in response to a data change event in at least one database, change data related to the data change event and associated information of the change data, wherein the associated information includes at least one or more of the following: table name, business index, operation type, transaction progress information, and data change completion time; the data conversion module 320 is used to convert the change data into a data document that can be retrieved from the vector database, and determine the document identifier of the data document and the document content written into the data document based on the associated information; the index information determination module 330 is used to determine document index information for retrieving the data document based on the change data; wherein the document index information includes sparse features, vector features, and metadata features corresponding to the change data.

[0103] The technical solution of this embodiment, in response to a data change event in at least one database, obtains the changed data related to the data change event and the associated information of the changed data. The changed data is converted into a data document that can be retrieved from the vector database, and the document identifier and content of the data document are determined based on the associated information. Based on this, the changed data in the database is synchronized to the vector database, providing data support for subsequent feedback responses based on the data in the vector database. According to the changed data, document index information for indexing the data document is determined, facilitating the subsequent determination of the data document matching the request based on the document index information, thereby providing the best feedback response. This invention solves the problem of wasted computing and storage resources caused by full-data export synchronization in the prior art. By synchronizing only the changed data in the database, computing resources are saved and data synchronization time is shortened. Correspondingly, it solves the problem that data stored in the vector database has a certain delay, making it impossible to provide the best feedback response. By synchronizing the changed data and providing the corresponding document index information, it is convenient to quickly determine the data document matching the request based on the document index information, thereby providing a feedback response corresponding to the latest data document, improving the user experience, and ensuring the real-time nature and accuracy of the feedback.

[0104] Based on the above embodiments, optionally, the data conversion module includes a data document determination unit, which includes: a parameter receiving subunit for receiving pre-configured document conversion parameters; and a data document determination subunit for converting the changed data into a data document adapted to the document conversion parameters; wherein the document conversion parameters include any one of line document parameters, aggregated document parameters, semantic template parameters, and indicator document parameters.

[0105] Optionally, a data document determination subunit is used to format and adjust the changed data based on a preset template, and to convert the formatted and adjusted changed data into semantically meaningful text based on the document conversion parameters, so as to obtain part of the document content in the data document.

[0106] Optionally, the data conversion module includes: a document content determination unit, used to determine the document identifier of the data document based on the table name, business index, transaction progress information, and data change completion time in the associated information; wherein the document identifier is used to determine the data document corresponding to the data change event in the vector database; the associated information is analyzed to determine at least one piece of metadata related to the business dimension and written into the data document; wherein the metadata includes at least one or more of the following: applicable time range, accessible user permissions, the database and / or storage block to which the changed data belongs, and the business product corresponding to the changed data.

[0107] Optionally, the index information determination module is used to determine sparse features based on keywords and word segmentation information corresponding to the changed data; obtain vector features corresponding to the changed data based on the embedding model and the changed data; and determine document index information for retrieving the data document based on the sparse features, the vector features, and metadata features determined based on the association information.

[0108] Optionally, the device further includes: a data document verification module, used to cache the data document in a temporary cache area; and to verify the data document based on the document identifier to which the data document belongs and the data change completion time.

[0109] Optionally, the device further includes: a vector database update module, configured to write the data document into the vector database, and retrieve all first historical documents in the vector database to determine a second historical document that is associated with the data document, wherein the association relationship includes at least: fields whose field similarity satisfies a preset field similarity threshold; and to update the vector database by merging the second historical document and the data document.

[0110] Optionally, the device further includes: a request feedback module, used to respond to a received request for querying business data, search for target data from the vector database according to the description information carried in the request, and provide feedback; wherein the description information includes query time information.

[0111] Optionally, the device further includes: the vector database includes multiple first historical documents, each first historical document including document index information; a request feedback module, including: an original metadata determination unit, configured to determine original metadata adapted to the request; a third historical document determination unit, configured to determine a third historical document based on the metadata features of each first historical document and the original metadata; a fourth historical document determination unit, configured to determine matching features based on the description information of the request, so as to determine a fourth historical document from the third historical documents based on the matching features; and a target data feedback unit, configured to determine and feedback the target data based on the document content of the fourth historical document.

[0112] Optionally, the description information includes query time information and a third historical document determination unit, which is used to determine the third historical document based on the metadata characteristics of the first historical document, the applicable time range corresponding to the document identifier, and the original metadata.

[0113] The incremental synchronization-based structured data storage device provided in this embodiment of the invention can execute the incremental synchronization-based structured data storage method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0114] Example 4

[0115] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0116] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0117] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0118] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the incremental synchronization-based structured data storage method.

[0119] In some embodiments, the incremental synchronization-based structured data storage method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the incremental synchronization-based structured data storage method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the incremental synchronization-based structured data storage method by any other suitable means (e.g., by means of firmware).

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] Computer programs for implementing the incremental synchronization-based structured data storage method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0123] Example 5

[0124] Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a structured data storage method based on incremental synchronization, the method comprising:

[0125] In response to a data change event in at least one database, the system acquires the change data related to the data change event and the associated information of the change data, wherein the associated information includes at least one or more of the following: table name, business index, operation type, transaction progress information, and data change completion time; the system converts the change data into a data document that can be retrieved from a vector database, and determines the document identifier of the data document and the document content written into the data document based on the associated information; and determines document index information for retrieving the data document based on the change data, wherein the document index information includes sparse features, vector features, and metadata features corresponding to the change data.

[0126] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0128] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0129] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0130] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0131] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A structured data storage method based on incremental synchronization, characterized in that, include: In response to a data change event in at least one database, the system obtains the changed data related to the data change event and the associated information of the changed data, wherein the associated information includes at least one or more of the following: table name, business index, operation type, transaction progress information, and data change completion time. The changed data is converted into a data document that can be retrieved from the vector database, and the document identifier of the data document and the document content written into the data document are determined based on the association information. Based on the changed data, determine the document index information used to retrieve the data document; The document index information includes sparse features, vector features, and metadata features corresponding to the changed data.

2. The method according to claim 1, characterized in that, The step of converting the changed data into a data document that can be retrieved from the vector database includes: Receive pre-configured document conversion parameters; The changed data is converted into a data document that is compatible with the document conversion parameters; The document conversion parameters include any one of the following: line document parameters, aggregated document parameters, semantic template parameters, and indicator document parameters.

3. The method according to claim 2, characterized in that, The step of converting the changed data into a data document adapted to the document conversion parameters includes: The changed data is formatted and adjusted based on a preset template, and the formatted and adjusted changed data is converted into semantically meaningful text based on the document conversion parameters to obtain part of the document content in the data document.

4. The method according to claim 1, characterized in that, The step of determining the document identifier of the data document and the document content written into the data document based on the association information includes: Based on the table name, business index, transaction progress information, and data change completion time in the associated information, the document identifier of the data document is determined; wherein, the document identifier is used to determine the data document corresponding to the data change event in the vector database; The associated information is analyzed to identify at least one metadata related to the business dimension, and then written into the data document. The metadata includes at least one or more of the following: applicable time range, accessible user permissions, the database and / or storage block to which the changed data belongs, and the business product corresponding to the changed data.

5. The method according to claim 1, characterized in that, The step of determining the document index information for retrieving the data document based on the changed data includes: Based on the keywords and word segmentation information corresponding to the changed data, sparse features are determined; Based on the embedding model and the changed data, vector features corresponding to the changed data are obtained; Based on the sparse features, the vector features, and the metadata features determined based on the association information, document index information for retrieving the data documents is determined.

6. The method according to claim 1, characterized in that, The method further includes: The data document is cached in a temporary cache area; The data document is verified based on its document identifier and the time when the data change was completed.

7. The method according to claim 1 or 6, characterized in that, The method further includes: The data document is written into the vector database, and all first historical documents in the vector database are retrieved to determine the second historical document that is associated with the data document, wherein the association relationship includes at least: fields whose field similarity meets a preset field similarity threshold; The vector database is updated by merging the second historical document and the data document.

8. The method according to claim 1, characterized in that, The method further includes: In response to receiving a request to query business data, the system searches for the target data in the vector database based on the description information carried in the request and returns the result. The description information includes query time information.

9. The method according to claim 8, characterized in that, The vector database includes multiple first historical documents, each containing document index information. The step of retrieving target data from the vector database and providing feedback based on the description information carried in the request includes: Determine the original metadata that is compatible with the request; The third historical document is determined based on the metadata characteristics of each first historical document and the original metadata; Based on the description information of the request, determine the features to be matched, so as to determine the fourth historical document from the third historical document based on the features to be matched; Based on the content of the fourth historical document, the target data is determined and fed back.

10. The method according to claim 9, characterized in that, The description information includes query time information. Determining the third historical document based on the metadata characteristics of each first historical document and the original metadata includes: The third historical document is determined based on the metadata characteristics of the first historical document, the applicable time range corresponding to the document identifier, and the original metadata.

Images