Semi-structured document data schema real-time recommendation method

Abstract

The present application relates to the field of database application, and particularly to a semi-structured document data schema real-time recommendation method, comprising the following steps: step S1: semi-structured document data schema embedding, converting the semi-structured document data schema into a low-dimensional vector representation and taking it as the input of real-time recommendation, step S2: real-time recommendation, predicting the content to be created by the user next according to the data schema content being created by the user and the current input field and making a recommendation, helping the user to quickly create high-quality data schema. The method can reduce the operation frequency of two-thirds of the user in creating data schema, greatly reduce the burden of the user in creating data schema, guide the user to use homogeneous data schema at the same time, reduce the heterogeneity of database data schema, and improve the data utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of database applications, specifically a method for real-time recommendation of semi-structured document data patterns. Background Technology

[0002] Semi-structured document data is a data type that lies between structured and unstructured data. Unlike structured data, it lacks a strict data model or architecture, but it contains certain structural information that allows the data to be parsed and processed. Unlike unstructured data, it contains some form of metadata or tags, enabling the data to be classified and organized. In recent years, semi-structured documents have gained increasing attention due to their flexibility, and they have been widely used in collaborative online information platforms, scientific data management systems, and other fields.

[0003] However, in practical applications, due to the massive scale, complexity, and diversity of data, as well as issues related to data quality and integrity, users face many difficulties and challenges when creating data schemas. First, users need to manually input each field when creating a data schema, a time-consuming and labor-intensive process. Second, users may forget some fields during creation, compromising the quality and integrity of the data schema. Furthermore, different users may use different fields to describe the same attribute, leading to a surge in data schemas and reducing query efficiency and quality.

[0004] Existing embedding algorithms struggle to directly process semi-structured document data patterns and fail to capture the hierarchical relationships between fields within the data pattern. Some recommendation systems with similar scenarios suffer from low efficiency, lack of generality, and limited recommendation results; these existing methods cannot adequately address these issues. Summary of the Invention

[0005] To address the problems in existing technologies, this invention provides a real-time recommendation method for semi-structured document data patterns.

[0006] The technical solution adopted by this invention to solve its technical problem is: a real-time recommendation method for semi-structured document data patterns, comprising the following steps:

[0007] Step S1: Embedding semi-structured document data patterns, which transforms semi-structured document data patterns into low-dimensional vector representations and uses them as input for real-time recommendations. This includes four stages: data pattern splitting, field vector generation, weighted average pooling, and encoding.

[0008] Step S2: Real-time recommendation. Based on the data pattern content that the user is creating and the current input fields, predict the content that the user will create next and make recommendations to help the user quickly create high-quality data patterns. A deep neural network is used, which concatenates the vector of the input fields, the embedding vector of the currently created content, and the embedding vector of the candidate template fragment as input. The output is the probability that the user will use the candidate template fragment, and a recommendation list is generated after sorting according to the probability. This includes building a recommendation model training dataset, training the recommendation model, and deployment and application.

[0009] Specifically, the process in step S1 is as follows:

[0010] Step S11: Prepare the dataset by splitting all data patterns into a dataset for training data pattern embedding vectors.

[0011] Step S12: Train field vectors. Based on the position of the field in the data pattern, generate positive samples for training field word vectors according to different strategies, and randomly generate negative samples to train field word vectors.

[0012] Step S13: Weighted average pooling and encoding.

[0013] Specifically, the process of step S2 is as follows:

[0014] Step S21: Construct the training dataset for the recommendation model;

[0015] Step S22: Train the recommendation model using the current data pattern embedding vector, data pattern word vector, and data pattern fragment embedding vector as input, and the predicted click-through rate as input.

[0016] Step S23: Deployment and application, deploy and apply the recommendation model.

[0017] Specifically, the process for step S13 is as follows:

[0018] Step S131: Based on field depth, required fields, and frequency of occurrence, set weights for the word vectors of each data pattern field and perform weighted average pooling.

[0019] Step S132: Encode and reduce the dimensionality of the average pooled vector to serve as the final data pattern embedding vector.

[0020] Specifically, step S21 involves simulating the data pattern creation process, generating multiple intermediate states for each data pattern, which serve as the training set for the recommendation model.

[0021] The beneficial effects of this invention are: it can reduce the number of operations required for users to create data patterns by two-thirds, greatly reducing the burden on users in creating data patterns, while guiding users to use homogeneous data patterns, reducing the heterogeneity of database data patterns, and improving data utilization efficiency. Attached Figure Description

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Figure 1 Embedded module architecture diagram for data patterns;

[0024] Figure 2 This is a recommended model diagram. Detailed Implementation

[0025] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0026] like Figures 1-2 As shown, the real-time recommendation method for semi-structured document data patterns according to the present invention includes the following steps:

[0027] Step S1: Embedding semi-structured document data patterns;

[0028] Step S2: Real-time recommendations consist of two parts.

[0029] Embedding algorithms are primarily responsible for transforming semi-structured document data patterns into low-dimensional vector representations, which serve as input for real-time recommendations. This significantly reduces feature complexity and computational and storage overhead. This part mainly consists of four stages: data pattern splitting, field vector generation, weighted average pooling, and encoding. Data pattern splitting employs a hierarchical traversal approach, dividing the data pattern into pattern fragments and feeding them into the field vector generation model. The field vector generation model then generates different samples based on the field's position within the data pattern, training field vectors. The weighted average pooling stage assigns weights to each field based on its importance and generates data pattern vectors. Finally, the data pattern vectors are encoded as the output of this part.

[0030] The semi-structured document data schema can be represented by a tree, where the root node represents the entire data schema, and other nodes represent each field within the schema. Due to the potential for nested structures within the data schema, this tree typically has multiple branches. From a semantic perspective, each subsequent node is often a detailed description of its parent node or a component of its parent node, while nodes on different branches often lack semantic similarity. When performing word embedding, these semantically dissimilar contents should not be used as positive sample inputs; instead, the focus should be on similar fields within the same branch, while also considering the hierarchical information within that branch. Data schema decomposition effectively addresses this issue. Furthermore, the decomposed data schema fragments can serve as content for real-time recommendations, corresponding to subsequent recommendation tasks.

[0031] For a semi-structured document data schema, field types can be divided into basic data types (such as strings, numbers, etc.) and nested types (corresponding to objects in JSON). Basic data types correspond to leaf nodes in the schema tree, while nested types correspond to non-leaf nodes. For all nested types, the path from the root node to that node and all its child nodes are considered as a subtree of that data schema. Thus, each data schema will be split into N subtrees, where N is the number of non-leaf nodes.

[0032] Based on the position of fields in the data schema, we can divide the data schema into four categories and process the fields of different categories separately. Based on the position of nodes in the schema fragment subtree, nodes can be divided into the following four categories:

[0033] Root node: The root node exists in all subtrees and represents the data schema in which this field is located.

[0034] Central node: The nested node selected when generating a subtree.

[0035] Intermediate nodes: Intermediate nodes on the path from the root node to the center node, which may not exist.

[0036] Leaf nodes: All child nodes of the central node.

[0037] Different strategies are needed to generate positive samples for different types of nodes. First, for the root node, it represents the entire data pattern, so it can be considered to be related to all fields. In other words, this node determines all the nodes that make up the data pattern, so the root node is combined with every other node to form a positive sample.

[0038] Secondly, for the central node, it is often a detailed description or component of the next level node. Therefore, the "preceding" information of the central node can be considered as its parent node, and the "following" information is all its child nodes. Thus, the central node, its parent node, and each child node are combined to form a positive sample.

[0039] For intermediate nodes, it provides the path from the root node to the center node, so we only need to consider the hierarchical relationship between adjacent nodes, that is, to form two positive samples with each intermediate node, its parent node, and its child node.

[0040] Finally, for leaf nodes, two aspects need to be considered: on the one hand, they can be considered as components of their parent nodes, so each leaf node and its parent node should be combined into a positive sample; on the other hand, since leaf nodes on the same branch at the same level have similarity, and there is an order relationship when users create these nodes, this system uses a sliding window of size 2 to process each leaf node and its two preceding and following sibling nodes into a positive sample.

[0041] By selecting different strategies for different types of nodes, this system can effectively capture the hierarchical relationships between nodes in a data pattern and generate more accurate word vector representations. After generating positive samples for each pattern subtree, an optimization objective P(Context(w)|w) can be set. Then, the field word vectors of the data pattern are obtained through training and used as input for subsequent pattern vector generation.

[0042] After obtaining the word vectors for each field, the final data pattern vector can be concatenated based on these field vectors. This system employs a weighted average pooling method, using three factors—the field's depth in the pattern tree, its frequency of occurrence in the entire dataset, and whether the field is required—to generate weights for each field. The word vectors of these fields are then combined to create the data pattern vector. Following this, the pooled vector is re-encoded using an autoencoder to generate the final low-dimensional embedding vector of the data pattern.

[0043] The real-time data pattern recommendation module utilizes a deep neural network. Its input layer consists of the embedding vector of the data pattern created by the current user, the embedding vector of the last input field, and the embedding vector of the data pattern fragment. The output is the predicted probability of the user using each data pattern fragment. When training the real-time data pattern recommendation model, several intermediate states need to be randomly constructed based on the existing data patterns as training samples. After the recommendation model is trained, the data patterns are sorted according to their predicted probabilities, and Top-k recommendations are performed.

[0044] Real-time recommendations predict and recommend content the user will create next based on the data pattern they are currently creating and the current input fields, helping users quickly create high-quality data patterns. This part mainly uses a deep neural network, which concatenates the vector of the input fields, the embedding vector of the currently created content, and the embedding vector of the candidate template fragments as input, and outputs the probability of the user using the candidate template fragments. The recommendations are then generated by sorting the probabilities.

[0045] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time recommendation method for semi-structured document data patterns, characterized in that, Includes the following steps: Step S1: Embedding semi-structured document data patterns, which transforms semi-structured document data patterns into low-dimensional vector representations and uses them as input for real-time recommendations. This includes four stages: data pattern splitting, field vector generation, weighted average pooling, and encoding. The data pattern splitting is as follows: a semi-structured document data pattern is represented by a tree, where the root node of the tree represents the entire data pattern, and other nodes represent each field in the data pattern; basic data types correspond to leaf nodes on the pattern tree, and nested types correspond to non-leaf nodes on the pattern tree; for each nested type, the path from the root node to the non-leaf node and all child nodes of the non-leaf node are taken as a subtree of the data pattern, and each data pattern will be split into N subtrees, where N is the number of non-leaf nodes; The field vector generation specifically involves: classifying nodes into four categories based on their position in the pattern fragment subtree—root node: representing the entire data pattern and existing in all subtrees; center node: the nested node selected when generating the subtree; intermediate node: the intermediate node in the path from the root node to the center node; leaf node: all child nodes of the center node; and generating positive samples for training field word vectors using different strategies for different types of nodes: for the root node, forming a positive sample with the root node and every other node; for the center node, forming a positive sample with the center node, its parent node, and every child node. For intermediate nodes, each intermediate node is combined with its parent node and child nodes to form a positive sample; for leaf nodes, each leaf node is combined with its parent node to form a positive sample, and a sliding window of size 2 is used to combine each leaf node with its two sibling nodes before and after it to form a positive sample; after generating positive samples by selecting different strategies for different types of nodes, an optimization target is set, and the field word vectors of the data pattern are trained. The weighted average pooling is specifically implemented by: assigning weights to the word vectors of each data pattern field based on three factors: the depth of the field in the pattern tree, the frequency of the field in the entire dataset, and whether the field is a required field, and then performing weighted average pooling. The encoding specifically involves: re-encoding the average pooled vector using an autoencoder to generate the final low-dimensional embedding vector of the data pattern. Step S2: Real-time recommendation. Based on the data pattern content being created by the user and the current input fields, predict the content the user will create next and make recommendations to help the user quickly create high-quality data patterns. A deep neural network is used, which concatenates the vector of the input fields, the embedding vector of the currently created content, and the embedding vector of the candidate template fragment as input. The output is the probability that the user will use the candidate template fragment, and a recommendation list is generated after sorting according to the probability. This includes building a recommendation model training dataset, training the recommendation model, and deployment and application. The construction of the recommendation model training dataset specifically involves simulating the data pattern creation process and generating multiple intermediate states for each data pattern as the training set for the recommendation model.

2. The real-time recommendation method for semi-structured document data patterns according to claim 1, characterized in that, In step S1, before splitting the data patterns, prepare the dataset by splitting all data patterns to serve as the dataset for training data pattern embedding vectors. When training field vectors, positive samples for training field word vectors are generated according to different strategies based on the position of the field in the data pattern, and negative samples are randomly generated to train field word vectors.

3. The real-time recommendation method for semi-structured document data patterns according to claim 1, characterized in that, In step S2, the recommendation model is trained by taking the current data pattern embedding vector, data pattern word vector, and data pattern fragment embedding vector as input and the predicted click rate as output. After training, the recommendation model is deployed and applied.

4. The real-time recommendation method for semi-structured document data patterns according to claim 2, characterized in that, In the weighted average pooling process, weights are assigned to the word vectors of each data pattern field based on field depth, required fields, and frequency of occurrence, and then weighted average pooling is performed. The average pooled vectors are then encoded and dimensionality reduced to serve as the final data pattern embedding vectors.

Images