Similarity calculation method, device, medium and equipment for JSON format data
Through the neural network model, JSON format data is converted into graph structure, features are extracted and similarity is calculated, and the accuracy and efficiency of JSON format data similarity calculation in the prior art is solved, and efficient and accurate similarity evaluation is achieved.
Patent Information
- Application Number
- CN202411411597.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-10-11
AI Technical Summary
The prior art calculates the similarity between JSON format data, which is low accuracy and time-consuming, making it difficult to process large-scale data, affecting the user experience.
The neural network model is used to convert JSON format data into graph structures, extract node-level and graph-level features, and calculate the graph-level and node-level comparison result vectors through the pre-trained neural tensor network, and perform the final similarity calculation in combination.
It greatly improves the accuracy and efficiency of JSON format data similarity calculation, adapts to different types and structures of JSON data, and improves user experience and computer adaptability.
Smart Images

Figure CN118916717B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method and device for calculating similarity of JSON format data, a storage medium, and a computer device. Background Art
[0002] JSON is a common semi-structured data format, which is currently widely used in document storage, network data transmission and other scenarios. With the increasing amount of JSON format data, many databases store a large amount of JSON format data. Currently, it is necessary to calculate the similarity between JSON format data in many scenarios. For example, in order to avoid duplicate JSON format data in the database, the storage system usually queries whether the same JSON format data already exists in the database before storing new JSON format data. At this time, it is necessary to calculate the similarity between JSON format data, so as to quickly and effectively identify and delete duplicate JSON format data; for example, when a user queries other documents similar to the target document, the target document and the document to be matched are both stored in JSON format. At this time, it is necessary to filter and sort the search results according to the similarity measurement in a large-scale JSON format data set to find the ideal matching document. In these scenarios, it is of great significance to accurately and efficiently calculate the similarity between JSON format data.
[0003] The structural forms of JSON format data are diverse, such as nested structures, flat structures, etc., so it is very challenging to calculate the similarity between JSON format data. At present, traditional algorithms have proposed a reliable method for calculating the structural differences between JSON format data based on tree edit distance. This method proposes the concepts of JSON tree and JSON edit distance, which can more accurately calculate the structural differences between JSON format data, but its problem is that only the differences in the node literals of the JSON tree are considered during comparison. Therefore, the existing methods have low accuracy when calculating the similarity between JSON format data. In addition, when faced with the challenges of large-scale data processing and complex information comparison, traditional algorithms become time-consuming and limited, and may require a long calculation time, which directly affects the efficiency of similarity calculation of JSON format data and cannot guarantee the user experience. Summary of the invention
[0004] In view of this, the present application provides a method and device for calculating the similarity of JSON format data, a storage medium, and a computer device. The similarity between JSON format data is calculated through a similarity calculation model. Since the neural network model can quickly extract node-level features and graph-level features from the graph structure obtained by converting JSON format data, the present application can process the similarity calculation between large-scale JSON format data in a short time, which greatly improves the efficiency of similarity calculation and helps to improve user experience. In addition, when performing similarity calculation, the similarity calculation model takes into account both node-level features and graph-level features, realizes the similarity comparison of JSON format data at multiple levels, and can greatly improve the accuracy of similarity calculation of JSON format data. After receiving the JSON format data, the similarity calculation model first converts the JSON format data into a graph structure form, and then performs similarity calculation, so it can adapt to JSON data of different types and structures. The enhancement of generalization ability makes the similarity calculation model more practical and adaptable.
[0005] According to one aspect of the present application, a method for calculating similarity of data in JSON format is provided, comprising:
[0006] Obtain first JSON format data and second JSON format data to be used for similarity calculation;
[0007] Through the similarity calculation model, the first JSON format data and the second JSON format data are respectively converted into graph structures to obtain the first graph structure and the second graph structure; for each graph structure in the first graph structure and the second graph structure, according to the semantic feature vector corresponding to each node in the graph structure, the first weight of each node relative to the graph structure and the second weight relative to another graph structure are calculated, according to the first weight and the second weight, the final weight corresponding to each node is calculated, according to the semantic feature vector corresponding to each node and the final weight, the graph vector corresponding to the graph structure is determined, and the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure are obtained; according to the first graph vector and the second graph vector, through the pre-trained neural tensor network, the graph-level comparison result vector between the first JSON format data and the second JSON format data is calculated; according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure, the node-level comparison result vector corresponding to the first JSON format data and the second JSON format data is calculated; the graph-level comparison result vector and the node-level comparison result vector are spliced, and the final similarity between the first JSON format data and the second JSON format data is calculated according to the splicing result.
[0008] According to another aspect of the present application, a similarity calculation device for JSON format data is provided, comprising:
[0009] A data acquisition module, used to acquire first JSON format data and second JSON format data to be used for similarity calculation;
[0010] A similarity calculation module is used to convert the first JSON format data and the second JSON format data into graph structures respectively through a similarity calculation model to obtain a first graph structure and a second graph structure; for each graph structure in the first graph structure and the second graph structure, according to the semantic feature vector corresponding to each node in the graph structure, calculate the first weight of each node relative to the graph structure and the second weight relative to another graph structure, calculate the final weight corresponding to each node according to the first weight and the second weight, determine the graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node and the final weight, and obtain the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure; according to the first graph vector and the second graph vector, calculate the graph-level comparison result vector between the first JSON format data and the second JSON format data through a pre-trained neural tensor network; according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure, calculate the node-level comparison result vector corresponding to the first JSON format data and the second JSON format data; splice the graph-level comparison result vector and the node-level comparison result vector, and calculate the final similarity between the first JSON format data and the second JSON format data according to the splicing result.
[0011] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the similarity calculation method of the above-mentioned JSON format data is implemented.
[0012] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned JSON format data similarity calculation method when executing the program.
[0013] By means of the above technical scheme, the present application provides a method and device for calculating the similarity of JSON format data, a storage medium, and a computer device. First, two JSON format data to be calculated for similarity are selected, namely, the first JSON format data and the second JSON format data. Afterwards, the pre-trained similarity calculation model can be used to input the first JSON format data and the second JSON format data into the similarity calculation model, and the first JSON format data and the second JSON format data are respectively converted into graph structures by the similarity calculation model, so that the first graph structure corresponding to the first JSON format data and the second graph structure corresponding to the second JSON format data can be obtained. Further, the converted graph structure can be represented in the form of a vector, and the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure can be obtained. Then, by comparing the two graph vectors, a vector representing the similarity between the two at the graph level can be obtained, which is called a graph-level comparison result vector. In addition, each node can be assigned a semantic feature vector, and a node-level comparison result vector can be obtained based on the comparison of the semantic feature vectors between the nodes in the two graph structures. Finally, the graph-level comparison result vector and the node-level comparison result vector are spliced to obtain a splicing vector, and the final similarity between the two JSON format data is calculated according to the obtained splicing vector. The embodiment of the present application calculates the similarity between JSON format data through a similarity calculation model. Since the neural network model can quickly extract node-level features and graph-level features from the graph structure obtained by converting JSON format data, the present application can process the similarity calculation between large-scale JSON format data in a short time, greatly improving the efficiency of similarity calculation and helping to improve user experience. In addition, when performing similarity calculation, the similarity calculation model considers both node-level features and graph-level features, realizes the similarity comparison of JSON format data at multiple levels, and can greatly improve the similarity calculation accuracy of JSON format data. After receiving the JSON format data, the similarity calculation model first converts the JSON format data into a graph structure form, and then performs similarity calculation, so it can adapt to JSON data of different types and structures, and the enhancement of generalization ability makes the similarity calculation model more practical and adaptable.
[0014] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0016] Figure 1 A schematic diagram showing a process of calculating similarity of JSON format data provided in an embodiment of the present application;
[0017] Figure 2 A schematic diagram of JSON format data provided in an embodiment of the present application is shown;
[0018] Figure 3 A schematic diagram showing another JSON format data provided in an embodiment of the present application;
[0019] Figure 4 A schematic diagram of a JSON tree corresponding to JSON format data provided in an embodiment of the present application is shown;
[0020] Figure 5 A schematic diagram of the structure of a device for calculating similarity of JSON format data provided in an embodiment of the present application is shown;
[0021] Figure 6 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0022] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.
[0023] In this embodiment, a method for calculating the similarity of JSON format data is provided, such as Figure 1 As shown, the method includes:
[0024] Step 101, obtaining first JSON format data and second JSON format data to be calculated for similarity;
[0025] Step 102: Convert the first JSON format data and the second JSON format data into graph structures respectively through a similarity calculation model to obtain a first graph structure and a second graph structure; for each graph structure in the first graph structure and the second graph structure, calculate the first weight of each node relative to the graph structure and the second weight relative to another graph structure according to the semantic feature vector corresponding to each node in the graph structure, calculate the final weight corresponding to each node according to the first weight and the second weight, determine the graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node and the final weight, and obtain the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure; calculate the graph-level comparison result vector between the first JSON format data and the second JSON format data through a pre-trained neural tensor network according to the first graph vector and the second graph vector; calculate the node-level comparison result vector between the first JSON format data and the second JSON format data according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure; perform splicing processing on the graph-level comparison result vector and the node-level comparison result vector, and calculate the final similarity between the first JSON format data and the second JSON format data according to the splicing processing result.
[0026] The embodiment of the present application calculates the similarity between JSON format data through a similarity calculation model, and can accurately and quickly calculate the similarity results between JSON format data. First, two JSON format data to be similarly calculated can be selected, and the two JSON format data can be named as the first JSON format data and the second JSON format data respectively.
[0027] Afterwards, the pre-trained similarity calculation model can be used to calculate the final similarity between the first JSON format data and the second JSON format data. Here, the similarity calculation model can be a graph neural network model. Specifically, the first JSON format data and the second JSON format data can be input into the similarity calculation model, and the following operations are performed through the similarity calculation model: the first JSON format data and the second JSON format data are respectively converted into graph structures, and then a first graph structure corresponding to the first JSON format data and a second graph structure corresponding to the second JSON format data can be obtained. In the conversion process, the key and value in the JSON format data can be used as nodes in the graph structure, and the edges represent the relationship between these key-value pairs. Converting JSON format data into a graph structure helps to capture the intrinsic structure and semantic relationship of JSON format data.
[0028] After obtaining the first graph structure and the second graph structure, the converted graph structure can be further represented in vector form, so as to obtain a first graph vector corresponding to the first graph structure and a second graph vector corresponding to the second graph structure. The process of determining the graph vector is detailed in the following embodiment. Next, by comparing the two graph vectors through a pre-trained neural tensor network, a vector representing the similarity between the two at the graph level can be obtained, which is called a graph-level comparison result vector.
[0029] In addition to the comparison at the graph level, the embodiment of the present application also considers the similarity comparison at the node level in the two graph structures. Specifically, each node can be assigned a semantic feature vector, and a node-level comparison result vector can be obtained based on the comparison of the semantic feature vectors between the nodes in the two graph structures. Through the node-level comparison result vector, more fine-grained similarity information can be captured.
[0030] Finally, in order to obtain a comprehensive similarity evaluation, the graph-level comparison result vector and the node-level comparison result vector are concatenated to obtain a concatenated vector that contains both global (graph-level) and local (node-level) similarity information. Afterwards, the final similarity between the two JSON format data is calculated based on the concatenated vector.
[0031] By applying the technical solution of this embodiment, first, two JSON format data to be similarity calculated are selected, namely, the first JSON format data and the second JSON format data. After that, the pre-trained similarity calculation model can be used to input the first JSON format data and the second JSON format data into the similarity calculation model, and the first JSON format data and the second JSON format data are respectively converted into graph structures by the similarity calculation model, so as to obtain the first graph structure corresponding to the first JSON format data and the second graph structure corresponding to the second JSON format data. Further, the converted graph structure can be represented in the form of a vector, and the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure can be obtained. Then, by comparing the two graph vectors, a vector representing the similarity between the two at the graph level can be obtained, which is called the graph-level comparison result vector. In addition, each node can be assigned a semantic feature vector, and a node-level comparison result vector can be obtained according to the comparison of the semantic feature vectors between the nodes in the two graph structures. Finally, the graph-level comparison result vector and the node-level comparison result vector are spliced to obtain a splicing vector, and the final similarity between the two JSON format data is calculated according to the obtained splicing vector. The embodiment of the present application calculates the similarity between JSON format data through a similarity calculation model. Since the neural network model can quickly extract node-level features and graph-level features from the graph structure obtained by converting JSON format data, the present application can process the similarity calculation between large-scale JSON format data in a short time, greatly improving the efficiency of similarity calculation and helping to improve user experience. In addition, when performing similarity calculation, the similarity calculation model takes into account both node-level features and graph-level features, realizes the similarity comparison of JSON format data at multiple levels, and can greatly improve the similarity calculation accuracy of JSON format data. After receiving the JSON format data, the similarity calculation model first converts the JSON format data into a graph structure form, and then performs similarity calculation, so it can adapt to JSON data of different types and structures. The enhancement of generalization ability makes the similarity calculation model more practical and adaptable.
[0032] In an embodiment of the present application, optionally, the step 102 of "calculating a first weight of each node relative to the graph structure and a second weight relative to another graph structure according to the semantic feature vector corresponding to each node in the graph structure, calculating a final weight corresponding to each node according to the first weight and the second weight, and determining a graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node and the final weight" includes: determining a target neighbor node corresponding to each node in the graph structure, and calculating a semantic feature vector corresponding to each node according to an initial semantic vector of each node and an initial semantic vector of the corresponding target neighbor node, wherein the initial semantic vector is calculated based on the node label; determining a first initial graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node in the graph structure and a learning weight matrix in the similarity calculation model, and respectively Calculate the inner product between the semantic feature vector corresponding to each node in the graph structure and the first initial graph vector to obtain a first weight corresponding to each node; for each node in the graph structure, calculate the node similarity between the node and each node in another graph structure based on the semantic feature vector corresponding to the node and the semantic feature vector corresponding to each node in another graph structure, and calculate the second initial graph vector of the other graph structure relative to the node based on the node similarity and the semantic feature vector corresponding to each node in the other graph structure, calculate the inner product between the semantic feature vector corresponding to the node and the second initial graph vector to obtain a second weight corresponding to the node; calculate the final weight corresponding to the node in the graph structure based on the first weight and the second weight, and determine the graph vector corresponding to the graph structure based on the semantic feature vector corresponding to each node in the graph structure and the final weight.
[0033] In this embodiment, regardless of the first graph structure or the second graph structure, the graph vector corresponding to each graph structure can be obtained by the following steps: wherein the graph vector corresponding to the first graph structure is the first graph vector, and the graph vector corresponding to the second graph structure is the second graph vector.
[0034] For each node in the graph structure, find all the neighbor nodes directly connected to it. After that, all of these neighbor nodes can be used as target neighbor nodes as needed, or only some of them can be used as target neighbor nodes. After determining the target neighbor nodes, the initial semantic vector of each node and the initial semantic vector of its corresponding target neighbor node can be used to calculate the semantic feature vector of each node through methods such as mean pooling and maximum pooling. The vector combines the information of the node itself and the information of its target neighbor node. Here, the initial semantic vector can be calculated based on the node label. For example, if the node label is "chemical composition", then "chemical composition" can be input into the BERT model, and the output of the BERT model can be used as the initial semantic vector corresponding to the node label. In this way, the initial semantic vector corresponding to each node can be calculated. In one embodiment, the semantic feature vector of a node can be obtained by summing the initial semantic vector of the node and the initial semantic vector of its target neighbor node and then calculating the average value. For example, if a node has 5 target neighbor nodes, the initial semantic vector of the node and the initial semantic vectors of the 5 target neighbor nodes can be added, and then the vector obtained after the addition can be divided by 6. The obtained vector can be used as the semantic feature vector of the node. In this way, the semantic feature vector corresponding to each node in the graph structure can be obtained.
[0035] Furthermore, the semantic feature vectors of all nodes in the graph structure and the learning weight matrix in the similarity calculation model can be used to calculate the first initial graph vector of the entire graph structure by the following formula:
[0036] ;
[0037] in, represents the first initial graph vector, represents the learning weight matrix, Indicates n The semantic feature vector of a node, N Indicates the total number of nodes in the graph structure. Here, the learning weight matrix can be obtained after the similarity calculation model training is completed, and can be a part of the model parameters of the similarity calculation model. The learning weight matrix can include the same number of learning weights as the number of dimensions of the semantic feature vector.
[0038] After obtaining the first initial graph vector, the inner product between the semantic feature vector of each node in the graph structure and the first initial graph vector can be calculated to obtain the first weight corresponding to each node, which reflects the similarity or contribution of the node to the global representation of the graph structure. For example, the semantic feature vector of node i in the graph structure is , the first initial graph vector corresponding to this graph structure is , then the first weight corresponding to node i can be . In this way, the first weight corresponding to each node can be calculated.
[0039] In addition, for each node in the graph structure, the similarity between its semantic feature vector and the semantic feature vectors of all nodes in another graph structure can also be calculated. Here, the similarity calculation method may include a cosine similarity calculation method, a Euclidean distance calculation method, etc. It should be noted that another graph structure refers to a graph structure corresponding to another JSON format data to be similarly calculated; the semantic feature vectors corresponding to each node in another graph structure can also be calculated according to the above method. For node i in a graph structure, assuming that another graph structure includes 30 nodes, the semantic feature vector corresponding to node i can be used to calculate the similarity with the semantic feature vector corresponding to each of the 30 nodes. Then, for node i in the graph structure, the node similarity between node i and each node in another graph structure can be calculated based on the node similarity between node i and each node in another graph structure, and the proportion of the sum of the node similarities of all 30 nodes, and the semantic feature vector of each node in another graph structure, and the corresponding similarity proportion, are calculated in a weighted manner for the second initial graph vector of another graph structure for node i. In this way, each node in the graph structure can correspond to a second initial graph vector. Afterwards, the inner product between the semantic feature vector of node i and the second initial graph vector is calculated to obtain the second weight corresponding to node i, which reflects the similarity between node i and another graph structure.
[0040] After obtaining the first weight and the second weight corresponding to each node, the final weight corresponding to each node in the graph structure can be calculated based on the first weight and the second weight. Next, the semantic feature vectors of each node in the graph structure and their final weights are used to calculate the graph vector of the entire graph structure in a weighted manner. For example, if the graph structure includes 35 nodes, then the final weight of each node can be multiplied by the semantic feature vector of the node to obtain the first result corresponding to each node, and then the first results corresponding to the 35 nodes are added together to obtain the graph vector corresponding to the graph structure. This graph vector not only contains the local and global information of the nodes in the graph, but also incorporates the relevant information of another graph structure by considering cross-graph similarity. Therefore, when performing graph-level comparisons based on the graph vector, the accuracy of the comparison results can be greatly improved, thereby improving the accuracy of the similarity calculation of JSON format data.
[0041] In an embodiment of the present application, optionally, the step 102 of "calculating a graph-level comparison result vector between the first JSON format data and the second JSON format data through a pre-trained neural tensor network based on the first graph vector and the second graph vector" includes: inputting the first graph vector and the second graph vector into a pre-trained neural tensor network in the similarity calculation model, and calculating a graph-level comparison result vector between the first JSON format data and the second JSON format data through the neural tensor network.
[0042] In this embodiment, the neural tensor network refers to a deep learning model that combines a neural network and tensor decomposition. It enables the model to capture the complex interactive relationship between input data by introducing a tensor layer in the neural network layer. The tensor layer learns the nonlinear relationship between inputs through tensor decomposition, thereby improving the expressiveness and generalization ability of the model. The neural tensor network in this embodiment is a pre-trained network, that is, the neural tensor network has been trained using a large amount of data and has learned effective parameters and feature representations. Specifically, the graph vector converted from two JSON format data can be used as the input of the neural tensor network. The neural tensor network performs complex nonlinear transformations and interactions on the two graph vectors through its internal neural network structure and tensor layer, thereby capturing the similarities and differences between them. Finally, the neural tensor network outputs a graph-level comparison result vector, which can reflect the graph-level similarity between two JSON format data.
[0043] The Neural Tensor Network works as follows:
[0044]
[0045] in, Represents the graph structure i and graph structure j The corresponding graph vector, is the graph-level comparison result vector, is the weight tensor, [] represents the concatenation operation, is the weight vector, is the bias vector, and is the activation function. is a hyperparameter. In the pre-trained neural tensor network, the weight tensor, weight vector, bias vector, etc. are all known. By inputting the graph vectors corresponding to the two graph structures, the graph-level comparison result vector can be obtained.
[0046] The embodiment of the present application converts JSON format data into graph vectors and uses a pre-trained neural tensor network to perform graph-level comparison, which can capture the complex structure and relationship between JSON format data, thereby providing a more accurate and comprehensive similarity evaluation for JSON format data.
[0047] In an embodiment of the present application, optionally, the "calculating the node-level comparison result vector corresponding to the first JSON format data and the second JSON format data according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure" in step 102 includes: for each graph structure in the first graph structure and the second graph structure, performing the following operations to calculate the feature vector to obtain a first feature vector of the first JSON format data and a second feature vector of the second JSON format data: determining a reference node among the nodes included in the graph structure, and determining a reference feature vector according to the semantic feature vector corresponding to the reference node, and comparing the semantic feature vector corresponding to each node in the graph structure with the reference feature vector; Similarity between feature vectors, sorting each node in the graph structure, and according to the sorted order, inputting the semantic feature vector corresponding to each node into the pre-trained recurrent neural network in the similarity calculation model for feature vector calculation; calculating the vector difference between the first feature vector and the second feature vector, and determining the node-level comparison result vector according to the vector difference; accordingly, the "calculating the final similarity between the first JSON format data and the second JSON format data according to the splicing processing result" in step 102 includes: inputting the splicing processing result into the pre-trained fully connected layer in the similarity calculation model to obtain the final similarity between the first JSON format data and the second JSON format data.
[0048] In this embodiment, when calculating the node-level comparison result vector between JSON format data, the first feature vector of the first JSON format data and the second feature vector of the second JSON format data can be calculated first. Specifically, both the first feature vector and the second feature vector can be calculated using the following steps:
[0049] First, one or more reference nodes can be determined in each graph structure. These reference nodes can be special nodes in the graph structure, such as nodes with a node label of null. Specifically, the node labels of object type nodes and array nodes can be set to null. For example, Figure 2The following shows a JSON format data. The keys are "title", "running time", "cast", and "type", the value of "cast" is an object, the value of "type" is an array, and the other values are strings and numbers. Therefore, the node labels of the value nodes of "cast" and "type" can be set to null.
[0050] After the reference nodes are determined, the semantic feature vectors of these reference nodes can be further used to determine the reference feature vector. If there is only one reference node in the graph structure, the semantic feature vector corresponding to the reference node can be used as the reference feature vector; if the graph structure includes multiple reference nodes, the semantic feature vectors corresponding to these reference nodes can be added and averaged, and the averaged vector can be used as the reference feature vector. After the reference feature vector is determined, the nodes in the graph structure can be sorted based on the similarity between the semantic feature vector corresponding to each node in the graph structure and the reference feature vector. Here, the similarity can be calculated by metrics such as cosine similarity and Euclidean distance. Afterwards, the semantic feature vectors corresponding to the sorted nodes are sequentially input into the pre-trained recurrent neural network (Recurrent Neural Network, also known as RNN) for feature vector calculation. Specifically, they can be sorted in order from high to low similarity. RNN is a neural network suitable for sequence data processing, which can capture time dependency or sequence information in a sequence. In this embodiment, RNN is used to integrate the sequence information and semantic features of the nodes in the graph structure to generate a feature vector for the entire graph structure.
[0051] The first eigenvector and the second eigenvector can be calculated through the above process. Then, the vector difference between the first eigenvector and the second eigenvector can be calculated. This vector difference reflects the difference between the two graph structures in the feature space, that is, in which features they differ and the size of the difference. Further, the vector difference can be directly used as a node-level comparison result vector. This vector is used to describe the similarity or difference between the two graph structures at the node level. The embodiment of the present application implements a process of node-level comparison of graph structures corresponding to two JSON format data by determining a reference node, calculating node sorting, using RNN to calculate eigenvectors, and calculating vector differences. This process combines graph theory, natural language processing, and deep learning technology, and can capture the complex information in the graph structure and the relationship between nodes, thereby providing a more in-depth and comprehensive comparison result.
[0052] After obtaining the node-level comparison result vector, the graph-level comparison result vector and the node-level comparison result vector can be concatenated to form a concatenated vector. Here, the concatenation process can be a simple vector concatenation. Then, the concatenated vector is input into the pre-trained fully connected layer in the similarity calculation model, and the final similarity between the first JSON format data and the second JSON format data can be obtained. Among them, the fully connected layer is a common component in neural networks, which can perform linear transformation on the input data, introduce nonlinearity through the activation function, and finally output the result.
[0053] In the embodiment of the present application, optionally, before step 101, the method further includes: obtaining multiple groups of training sample data, wherein each group of training sample data includes two JSON format sample data; for the two JSON format sample data included in each group of training sample data, converting each JSON format sample data into a first sample data represented in the form of a JSON tree, and converting each JSON format sample data into a second sample data represented in the form of a text, calculating the semantic edit distance similarity corresponding to each group of training sample data according to the first sample data corresponding to the two JSON format sample data, and calculating the semantic text similarity corresponding to each group of training sample data according to the second sample data corresponding to the two JSON format sample data, and calculating each The sample data is obtained based on the semantic edit distance similarity and the semantic text similarity. The method comprises the following steps: obtaining a true similarity corresponding to a group of training sample data; inputting each group of training sample data into an initial similarity calculation model to obtain a predicted similarity corresponding to each group of training sample data; calculating a model loss value corresponding to the initial similarity calculation model according to the true similarity corresponding to each group of training sample data and the predicted similarity through a target loss function, and adjusting the model parameters in the initial similarity calculation model according to the model loss value, until the model loss value is less than a preset loss threshold, and obtaining the similarity calculation model, wherein the model parameters include parameters of an initial weight matrix, parameters of an initial neural tensor network, parameters of an initial recurrent neural network, and parameters of an initial fully connected layer, and the similarity calculation model includes a learning weight matrix, a pre-trained neural tensor network, a pre-trained recurrent neural network, and a pre-trained fully connected layer.
[0054] In this embodiment, the initial similarity calculation model may be trained first to obtain the similarity calculation model used above. The training process of the initial similarity calculation model is as follows:
[0055] First, multiple groups of training sample data are obtained, and each group of training sample data may contain two sample data in JSON format. For each group of training sample data, each sample data in JSON format is converted into a JSON tree form to obtain the first sample data. This structured representation helps to capture the hierarchy and structure information of the JSON format data; at the same time, each sample data in JSON format is converted into a text form to obtain the second sample data. This form helps to perform natural language processing (Natural Language Processing, also known as NLP) tasks, such as semantic text similarity calculation. It should be noted that due to the various forms of JSON format data, when the JSON format sample data in the training sample data itself is in the form of a JSON tree, the JSON format sample data in the form of a JSON tree is directly used as the first sample data. Similarly, when the JSON format sample data in the training sample data itself is in the form of a text, the JSON format sample data in the form of a text is directly used as the second sample data.
[0056] The process of converting JSON format sample data into JSON tree form can be as follows: (1) Determine the root node. In the JSON format sample data, the outermost curly brackets {} represent an object, that is, the root node of the tree. (2) Parse the key-value pair. In the JSON format sample data, each pair of common key-value pairs corresponds to a node in the tree, where the key can be used as a key node, whose label is its name, and the value can be used as a leaf node, whose label is its value itself. Each key node has a subtree whose label is its value. (3) Process arrays. If the value of a key-value pair is an array, the array becomes an array node, and each array element is a child node of the array node, and the node label of the array node is set to null. (4) Nested objects. If the value of a key-value pair is another JSON object, then the JSON object is used as an object node, its node label is set to null, and it is used as a child node of the key node, and then the above steps are repeated to parse its internal structure. (5) Build a tree structure. Recursively build each node and its child nodes according to the above steps until all the data in the JSON format sample data is converted into a tree structure. Figure 3 As shown, a JSON format sample data is given, and Figure 4 The first sample data in the form of a JSON tree corresponding to the sample data in the JSON format is given.
[0057] The process of converting JSON format sample data into text form can be as follows: first, flatten the JSON tree corresponding to the JSON format sample data contained in each set of training sample data. The flattening process can be to convert the JSON tree into a set of key-value pairs, which can eliminate the influence of the tree structure and make it easier to compare the differences between the two JSON tree contexts. The flattening operation can be implemented by depth-first search, traversing the JSON tree level by level, and adding the generated key-value pairs to the set. It should be noted that the key of each key-value pair in the final set is a composite key concatenated from the top-level nesting to the target level (excluding the "root" node). This method will cause the key name of the higher level to be repeated more times, thereby affecting the final semantic vector of the key, which is consistent with the cognition that the higher level keys are more important in actual scenarios. After flattening the two JSON format sample data respectively, two sets containing key-value pairs will be obtained, representing the contents of the two JSON trees. Then these key-value pairs are converted into text pattern form. In the pattern, each key is regarded as a subject corresponding to a value. Specifically, each pattern is represented as a text in which the key and value are connected by a specific predicate. In this way, the text collections of the two JSON trees are obtained. Since the same recursive traversal algorithm is used to process the JSON tree, the order of the text collection also contains the structural information of some JSON format sample data.
[0058] In this way, after the above process, each JSON format sample data can exist in two forms, one is a JSON tree form, and the other is a text form. Then, the semantic edit distance between the two JSON format sample data is calculated using the first sample data in the JSON tree form corresponding to the two JSON format sample data, so as to obtain the semantic edit distance similarity, which reflects the structural similarity of the JSON format sample data. In addition, the semantic text similarity between the two JSON format sample data can also be calculated using the second sample data in the text form corresponding to the two JSON format sample data, which reflects the similarity in the content of the JSON format sample data. Afterwards, based on the semantic edit distance similarity and the semantic text similarity, the true similarity of each set of training sample data is calculated in a weighted manner. The semantic edit distance similarity can capture the subtle differences in the JSON tree structure, while the semantic text similarity can more comprehensively consider the content factors of JSON. By combining the two, the similarity differences between the JSON format sample data can be more accurately measured. Here, the weights corresponding to the semantic edit distance similarity and the semantic text similarity can be determined according to actual needs, or they can be set to 0.5 respectively.
[0059] After determining the true similarity corresponding to each group of training sample data, the initial similarity calculation model can be trained. The initial similarity calculation model may include an initial weight matrix, an initial neural tensor network, an initial recurrent neural network, and an initial fully connected layer, etc. These components together constitute the basic structure of the model, which is used to learn the characteristics of the data and calculate the similarity. When training the initial similarity calculation model, each group of training sample data is input into the initial similarity calculation model, and the model calculates the predicted similarity between the two JSON format sample data according to the two JSON format sample data input. After the predicted similarity corresponding to each group of training sample data is calculated by the initial similarity calculation model, the target loss function can be used to evaluate the model loss value of the model. Here, the target loss function can be a mean square error loss function, etc. After obtaining the model loss value, the size relationship between the model loss value and the preset loss threshold can be compared. If the model loss value is greater than or equal to the preset loss threshold, then at this time, according to the model loss value, the parameters in the model can be adjusted through the back propagation algorithm, including the parameters of the initial weight matrix, the parameters of the initial neural tensor network, the parameters of the initial recurrent neural network, the parameters of the initial fully connected layer, etc., to minimize the model loss value. Afterwards, each group of training sample data can be input into the initial similarity calculation model after parameter adjustment again, and the model loss value can be calculated again through the model until the model loss value is less than the preset loss threshold, and the training process ends, thereby obtaining a trained similarity calculation model. At this point, the parameters of the initial weight matrix, the parameters of the initial neural tensor network, the parameters of the initial recurrent neural network, the parameters of the initial fully connected layer, etc. are all determined, and the corresponding learning weight matrix, pre-trained neural tensor network, pre-trained recurrent neural network and pre-trained fully connected layer can be obtained. The trained similarity calculation model can be used to evaluate the similarity between new JSON format data. The embodiment of the present application obtains the true similarity corresponding to each group of training sample data by combining the similarity calculation in JSON tree form and text form, and finally uses the true similarity to train the initial similarity calculation model, so as to obtain a model that can accurately evaluate the similarity of JSON format data.
[0060] In an embodiment of the present application, optionally, the "calculating the semantic edit distance similarity corresponding to each group of training sample data based on the first sample data corresponding to the two JSON format sample data" includes: according to a preset matching rule, matching the tree nodes in the two first sample data corresponding to each group of training sample data to determine the matching node pairs; for each node pair, calculating the cosine similarity corresponding to the node pair based on the initial semantic vector corresponding to each node in the node pair; when the cosine similarity is greater than a preset similarity threshold, using the cosine similarity as the semantic edit distance corresponding to the node pair; when the cosine similarity is less than or equal to the preset similarity threshold, using the complement of the cosine similarity as the semantic edit distance corresponding to the node pair; according to the semantic edit distance corresponding to each node pair, the semantic edit distance similarity corresponding to each group of training sample data is obtained.
[0061] In this embodiment, first, the tree nodes in the two JSON trees corresponding to each set of training sample data can be matched according to the preset matching rules, so as to find the node pairs that match each other in the set of training sample data. It should be noted that the semantic edit distance is calculated only between the matching node pairs. Specifically, the preset matching rules can be set as follows:
[0062] Assume there are two JSON trees , and there exists a node pair , , , the match is valid if and only if the following five rules are met: (1) and [One-to-one]; (2) yes Ancestors and yes Ancestors [ancestor restriction]; (3) and of the same type [type restriction]; (4) if The order is less than ,So The order must be less than [array order restriction]; (5) If and The common ancestor is ,So and The common ancestor is [Structural restrictions]. It should be noted that in (1) above, nodes with consistent semantics (or semantic differences less than a preset threshold) are considered "="; in (2) above, ancestors refer to all nodes on the path from the root node of the JSON tree to the node; in (3) above, type specifically refers to field type, such as numeric type, string type, array type, container type, image type, etc. In database storage, these types have corresponding id unique identifiers. You can determine whether the types are the same by comparing whether the id unique identifiers are the same.
[0063] Among the above preset matching rules, the first two rules are the basic conditions for calculating the tree edit distance; the above (3) is intended to ensure that the types of a group of mapped node pairs are the same to avoid ignoring type differences; the above (4) restricts the mapping of elements in the array to be in order, while the child nodes within the nested object are not restricted by this rule; the above (5) forces the nested structure of JSON format data to be retained, and matching across subtrees or rearranging the JSON tree structure is not allowed.
[0064] When there are nodes in the two JSON trees that meet the preset matching rules above, they can be mapped into node pairs. , , Calculate the cosine similarity. For nodes in the JSON tree that do not meet the preset matching rules, there is no need to calculate the cosine similarity.
[0065] In this embodiment, while keeping the operations of adding and deleting nodes unchanged, the node labels can be calculated based on the BERT model to obtain the initial semantic vector corresponding to each node. After that, the cosine familiarity corresponding to each pair of node pairs can be calculated, and the size relationship between the calculated cosine similarity and the preset similarity threshold can be compared. If the cosine familiarity is greater than the preset similarity threshold, then the cosine similarity can be directly used as the semantic edit distance corresponding to the pair of node pairs; if the cosine familiarity is less than or equal to the preset similarity threshold, then the complement of the cosine similarity can be used as the semantic edit distance corresponding to the pair of node pairs. For example, the preset similarity threshold is set to 0.8. If the cosine similarity A between the initial semantic vectors of two nodes in a node pair is greater than 0.8, the node labels corresponding to the two nodes are considered to be synonyms, and it is deemed that there is no need to reassign values, and the semantic edit distance is the cosine similarity A; if it is less than or equal to 0.8, it is considered that the difference between it and 1 is the editing cost of re-assigning labels, and 1-A is used as the semantic edit distance.
[0066] After obtaining the semantic edit distance corresponding to each node pair, the semantic edit distances corresponding to all node pairs can be added together, and the final result is the semantic edit distance of the set of training sample data. Then, the semantic edit distance similarity corresponding to each set of training sample data can be calculated using the following formula:
[0067] Based on the two JSON trees corresponding to each set of training sample data, the number of nodes of each JSON tree is determined, and the larger value of the number of nodes is used to calculate the semantic edit distance similarity JEDS corresponding to each set of training sample data through the following formula.
[0068]
[0069] in, Representing two JSON trees respectively, d Represents the semantic edit distance corresponding to this group of training sample data, Represents the number of nodes in two JSON trees respectively. The embodiment of the present application improves the comparison between nodes from the simple determination of whether the strings are the same in the prior art to the comparison of semantic similarity, improves the JSON tree edit distance to the calculation of the JSON semantic edit distance, and incorporates the comparison of semantic factors into the structural comparison, so as to more reasonably characterize the results of the structural comparison of the sample data in the JSON format.
[0070] In an embodiment of the present application, optionally, the "calculating the semantic text similarity corresponding to each group of training sample data based on the second sample data corresponding to the two JSON format sample data" includes: calculating the semantic vector corresponding to each text in the second sample data; averaging the semantic vectors corresponding to each text to obtain the text semantic vector corresponding to the second sample data; calculating the cosine similarity between the text semantic vectors based on the text semantic vectors of the two second sample data corresponding to each group of training sample data, and using the cosine similarity as the semantic text similarity corresponding to each group of training sample data.
[0071] In this embodiment, the second sample data is actually a text collection. When calculating the semantic text similarity, the BERT model can be used to convert the text collection into a text vector collection. Specifically, each text in the text collection can be input into the BERT model, and the semantic vector corresponding to the text can be output. It should be noted that when calculating the semantic vector corresponding to each text in the text collection, for each key of a key-value pair, since it is represented as a composite key spliced from the top-level nesting to the key of the target level, the semantic vector of each text output is more realistic. Further calculate the average value of the semantic vector corresponding to the text contained in each text collection, and you can get an average vector representing the entire JSON format sample data, that is, the text semantic vector. Finally, use cosine similarity to measure the similarity between the two text semantic vectors to obtain semantic text similarity.
[0072] Further, as Figure 1The specific implementation of the method, the embodiment of the present application provides a similarity calculation device for JSON format data, such as Figure 5 As shown, the device comprises:
[0073] A data acquisition module, used to acquire first JSON format data and second JSON format data to be used for similarity calculation;
[0074] A similarity calculation module is used to convert the first JSON format data and the second JSON format data into graph structures respectively through a similarity calculation model to obtain a first graph structure and a second graph structure; for each graph structure in the first graph structure and the second graph structure, according to the semantic feature vector corresponding to each node in the graph structure, calculate the first weight of each node relative to the graph structure and the second weight relative to another graph structure, calculate the final weight corresponding to each node according to the first weight and the second weight, determine the graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node and the final weight, and obtain the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure; according to the first graph vector and the second graph vector, calculate the graph-level comparison result vector between the first JSON format data and the second JSON format data through a pre-trained neural tensor network; according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure, calculate the node-level comparison result vector corresponding to the first JSON format data and the second JSON format data; splice the graph-level comparison result vector and the node-level comparison result vector, and calculate the final similarity between the first JSON format data and the second JSON format data according to the splicing result.
[0075] Optionally, the similarity calculation module is used to:
[0076] Determine the target neighbor node corresponding to each node in the graph structure, and calculate the semantic feature vector corresponding to each node according to the initial semantic vector of each node and the initial semantic vector of the corresponding target neighbor node, wherein the initial semantic vector is calculated based on the node label;
[0077] Determine a first initial graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node in the graph structure and the learning weight matrix in the similarity calculation model, and respectively calculate the inner product between the semantic feature vector corresponding to each node in the graph structure and the first initial graph vector to obtain a first weight corresponding to each node;
[0078] For each node in the graph structure, according to the semantic feature vector corresponding to the node and the semantic feature vectors corresponding to the nodes in the other graph structure, calculate the node similarity between the node and the nodes in the other graph structure, and according to the node similarity and the semantic feature vectors corresponding to the nodes in the other graph structure, calculate the second initial graph vector of the other graph structure relative to the node, calculate the inner product between the semantic feature vector corresponding to the node and the second initial graph vector, and obtain a second weight corresponding to the node;
[0079] According to the first weight and the second weight, the final weight corresponding to the node in the graph structure is calculated, and according to the semantic feature vector corresponding to each node in the graph structure and the final weight, the graph vector corresponding to the graph structure is determined.
[0080] Optionally, the similarity calculation module is used to:
[0081] For each graph structure in the first graph structure and the second graph structure, the following operations are performed to calculate feature vectors to obtain a first feature vector of the first JSON format data and a second feature vector of the second JSON format data: a reference node is determined among the nodes included in the graph structure, and a reference feature vector is determined according to a semantic feature vector corresponding to the reference node, and each node in the graph structure is sorted based on a similarity between a semantic feature vector corresponding to each node in the graph structure and the reference feature vector, and according to the sorted order, the semantic feature vector corresponding to each node is input into a pre-trained recurrent neural network in the similarity calculation model to perform feature vector calculation;
[0082] Calculating a vector difference between the first eigenvector and the second eigenvector, and determining a node-level comparison result vector according to the vector difference;
[0083] Accordingly, the similarity calculation module is also used for:
[0084] The concatenation result is input into the pre-trained fully connected layer in the similarity calculation model to obtain the final similarity between the first JSON format data and the second JSON format data.
[0085] Optionally, the device further comprises:
[0086] The data acquisition module is further used to acquire multiple sets of training sample data before acquiring the first JSON format data and the second JSON format data to be subjected to similarity calculation, wherein each set of training sample data includes two pieces of JSON format sample data;
[0087] A first calculation module is used to convert each piece of JSON format sample data contained in each group of training sample data into first sample data represented in the form of a JSON tree, and convert each piece of JSON format sample data into second sample data represented in the form of a text, calculate the semantic edit distance similarity corresponding to each group of training sample data according to the first sample data corresponding to the two JSON format sample data, and calculate the semantic text similarity corresponding to each group of training sample data according to the second sample data corresponding to the two JSON format sample data, and calculate the true similarity corresponding to each group of training sample data based on the semantic edit distance similarity and the semantic text similarity;
[0088] The second calculation module is used to input each set of training sample data into the initial similarity calculation model to obtain the predicted similarity corresponding to each set of training sample data;
[0089] A model training module is used to calculate the model loss value corresponding to the initial similarity calculation model according to the real similarity corresponding to each group of training sample data and the predicted similarity through a target loss function, and adjust the model parameters in the initial similarity calculation model according to the model loss value until the model loss value is less than a preset loss threshold, thereby obtaining the similarity calculation model, wherein the model parameters include parameters of an initial weight matrix, parameters of an initial neural tensor network, parameters of an initial recurrent neural network, and parameters of an initial fully connected layer, and the similarity calculation model includes a learning weight matrix, a pre-trained neural tensor network, a pre-trained recurrent neural network, and a pre-trained fully connected layer.
[0090] Optionally, the first computing module is used to:
[0091] According to the preset matching rules, matching processing is performed on the tree nodes in the two first sample data corresponding to each group of training sample data to determine the matching node pairs;
[0092] For each node pair, according to the initial semantic vector corresponding to each node in the node pair, the cosine similarity corresponding to the node pair is calculated;
[0093] When the cosine similarity is greater than a preset similarity threshold, the cosine similarity is used as the semantic edit distance corresponding to the node pair;
[0094] When the cosine similarity is less than or equal to the preset similarity threshold, taking the complement of the cosine similarity as the semantic edit distance corresponding to the node pair;
[0095] According to the semantic edit distance corresponding to each node pair, the semantic edit distance similarity corresponding to each group of training sample data is obtained.
[0096] Optionally, the first computing module is used to:
[0097] Calculating a semantic vector corresponding to each text in the second sample data;
[0098] averaging the semantic vectors corresponding to the respective texts to obtain the text semantic vector corresponding to the second sample data;
[0099] According to the text semantic vectors of the two second sample data corresponding to each group of training sample data, the cosine similarity between the text semantic vectors is calculated, and the cosine similarity is used as the semantic text similarity corresponding to each group of training sample data.
[0100] It should be noted that for other corresponding descriptions of the functional units involved in the similarity calculation device for JSON format data provided in the embodiment of the present application, reference can be made to Figures 1 to 4 The corresponding description in the method will not be repeated here.
[0101] The present application also provides a computer device, which may be a personal computer, a server, a network device, etc. Figure 6 As shown, the computer device includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.
[0102] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0103] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0104] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0107] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0108] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for calculating similarity of JSON format data, characterized in that: include: Obtain first JSON format data and second JSON format data to be used for similarity calculation; Through the similarity calculation model, the first JSON format data and the second JSON format data are respectively converted into graph structures to obtain the first graph structure and the second graph structure; for each graph structure in the first graph structure and the second graph structure, according to the semantic feature vector corresponding to each node in the graph structure, the first weight of each node relative to the graph structure and the second weight relative to another graph structure are calculated, according to the first weight and the second weight, the final weight corresponding to each node is calculated, according to the semantic feature vector corresponding to each node and the final weight, the graph vector corresponding to the graph structure is determined, and the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure are obtained; according to the first graph vector and the second graph vector, through the pre-trained neural tensor network, the graph-level comparison result vector between the first JSON format data and the second JSON format data is calculated; according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure, the node-level comparison result vector corresponding to the first JSON format data and the second JSON format data is calculated; the graph-level comparison result vector and the node-level comparison result vector are spliced, and the final similarity between the first JSON format data and the second JSON format data is calculated according to the splicing result; The step of calculating, according to the semantic feature vector corresponding to each node in the graph structure, a first weight of each node relative to the graph structure and a second weight relative to another graph structure, calculating, according to the first weight and the second weight, a final weight corresponding to each node, and determining, according to the semantic feature vector corresponding to each node and the final weight, a graph vector corresponding to the graph structure, includes: Determine the target neighbor node corresponding to each node in the graph structure, and calculate the semantic feature vector corresponding to each node according to the initial semantic vector of each node and the initial semantic vector of the corresponding target neighbor node, wherein the initial semantic vector is calculated based on the node label; Determine a first initial graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node in the graph structure and the learning weight matrix in the similarity calculation model, and respectively calculate the inner product between the semantic feature vector corresponding to each node in the graph structure and the first initial graph vector to obtain a first weight corresponding to each node; For each node in the graph structure, according to the semantic feature vector corresponding to the node and the semantic feature vectors corresponding to the nodes in the other graph structure, calculate the node similarity between the node and the nodes in the other graph structure, and according to the node similarity and the semantic feature vectors corresponding to the nodes in the other graph structure, calculate the second initial graph vector of the other graph structure relative to the node, calculate the inner product between the semantic feature vector corresponding to the node and the second initial graph vector, and obtain a second weight corresponding to the node; Calculating a final weight corresponding to the node in the graph structure according to the first weight and the second weight, and determining a graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node in the graph structure and the final weight; The step of calculating the node-level comparison result vector corresponding to the first JSON format data and the second JSON format data according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure includes: For each graph structure in the first graph structure and the second graph structure, the following operations are performed to calculate feature vectors to obtain a first feature vector of the first JSON format data and a second feature vector of the second JSON format data: a reference node is determined among the nodes included in the graph structure, and a reference feature vector is determined according to a semantic feature vector corresponding to the reference node, and each node in the graph structure is sorted based on a similarity between a semantic feature vector corresponding to each node in the graph structure and the reference feature vector, and according to the sorted order, the semantic feature vector corresponding to each node is input into a pre-trained recurrent neural network in the similarity calculation model to perform feature vector calculation; Calculating a vector difference between the first eigenvector and the second eigenvector, and determining a node-level comparison result vector according to the vector difference; Accordingly, calculating the final similarity between the first JSON format data and the second JSON format data according to the concatenation processing result includes: The concatenation result is input into the pre-trained fully connected layer in the similarity calculation model to obtain the final similarity between the first JSON format data and the second JSON format data.
2. The method according to claim 1, characterized in that Before obtaining the first JSON format data and the second JSON format data to be subjected to similarity calculation, the method further includes: Obtain multiple sets of training sample data, where each set of training sample data includes two pieces of sample data in JSON format; For the two JSON format sample data included in each group of training sample data, each JSON format sample data is converted into first sample data represented in the form of a JSON tree, and each JSON format sample data is converted into second sample data represented in the form of a text, and the semantic edit distance similarity corresponding to each group of training sample data is calculated according to the first sample data corresponding to the two JSON format sample data, and the semantic text similarity corresponding to each group of training sample data is calculated according to the second sample data corresponding to the two JSON format sample data, and the real similarity corresponding to each group of training sample data is calculated based on the semantic edit distance similarity and the semantic text similarity; Input each set of training sample data into the initial similarity calculation model to obtain the predicted similarity corresponding to each set of training sample data; Through the target loss function, according to the real similarity corresponding to each group of training sample data and the predicted similarity, the model loss value corresponding to the initial similarity calculation model is calculated, and the model parameters in the initial similarity calculation model are adjusted according to the model loss value until the model loss value is less than the preset loss threshold, thereby obtaining the similarity calculation model, wherein the model parameters include parameters of the initial weight matrix, parameters of the initial neural tensor network, parameters of the initial recurrent neural network and parameters of the initial fully connected layer, and the similarity calculation model includes a learning weight matrix, a pre-trained neural tensor network, a pre-trained recurrent neural network and a pre-trained fully connected layer.
3. The method according to claim 2, characterized in that The calculating the semantic edit distance similarity corresponding to each set of training sample data according to the first sample data corresponding to the two JSON format sample data includes: According to the preset matching rules, matching processing is performed on the tree nodes in the two first sample data corresponding to each group of training sample data to determine the matching node pairs; For each node pair, according to the initial semantic vector corresponding to each node in the node pair, the cosine similarity corresponding to the node pair is calculated; When the cosine similarity is greater than a preset similarity threshold, the cosine similarity is used as the semantic edit distance corresponding to the node pair; When the cosine similarity is less than or equal to the preset similarity threshold, taking the complement of the cosine similarity as the semantic edit distance corresponding to the node pair; According to the semantic edit distance corresponding to each node pair, the semantic edit distance similarity corresponding to each group of training sample data is obtained.
4. The method according to claim 2, characterized in that: The calculating the semantic text similarity corresponding to each set of training sample data according to the second sample data corresponding to the two JSON format sample data includes: Calculating a semantic vector corresponding to each text in the second sample data; averaging the semantic vectors corresponding to the respective texts to obtain the text semantic vector corresponding to the second sample data; According to the text semantic vectors of the two second sample data corresponding to each group of training sample data, the cosine similarity between the text semantic vectors is calculated, and the cosine similarity is used as the semantic text similarity corresponding to each group of training sample data.
5. A similarity calculation device for JSON format data, characterized in that: include: A data acquisition module, used to acquire first JSON format data and second JSON format data to be used for similarity calculation; A similarity calculation module, used to convert the first JSON format data and the second JSON format data into graph structures respectively through a similarity calculation model to obtain a first graph structure and a second graph structure; for each graph structure in the first graph structure and the second graph structure, according to the semantic feature vector corresponding to each node in the graph structure, calculate the first weight of each node relative to the graph structure and the second weight relative to another graph structure, calculate the final weight corresponding to each node according to the first weight and the second weight, determine the graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node and the final weight, and obtain the first graph vector corresponding to the first graph structure and the second graph vector corresponding to the second graph structure; according to the first graph vector and the second graph vector, calculate the graph-level comparison result vector between the first JSON format data and the second JSON format data through a pre-trained neural tensor network; according to the semantic feature vector corresponding to each node in the first graph structure and the second graph structure, calculate the node-level comparison result vector corresponding to the first JSON format data and the second JSON format data; splice the graph-level comparison result vector and the node-level comparison result vector, and calculate the final similarity between the first JSON format data and the second JSON format data according to the splicing result; The similarity calculation module is used to: Determine the target neighbor node corresponding to each node in the graph structure, and calculate the semantic feature vector corresponding to each node according to the initial semantic vector of each node and the initial semantic vector of the corresponding target neighbor node, wherein the initial semantic vector is calculated based on the node label; Determine a first initial graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node in the graph structure and the learning weight matrix in the similarity calculation model, and respectively calculate the inner product between the semantic feature vector corresponding to each node in the graph structure and the first initial graph vector to obtain a first weight corresponding to each node; For each node in the graph structure, according to the semantic feature vector corresponding to the node and the semantic feature vectors corresponding to the nodes in the other graph structure, calculate the node similarity between the node and the nodes in the other graph structure, and according to the node similarity and the semantic feature vectors corresponding to the nodes in the other graph structure, calculate the second initial graph vector of the other graph structure relative to the node, calculate the inner product between the semantic feature vector corresponding to the node and the second initial graph vector, and obtain a second weight corresponding to the node; Calculating a final weight corresponding to the node in the graph structure according to the first weight and the second weight, and determining a graph vector corresponding to the graph structure according to the semantic feature vector corresponding to each node in the graph structure and the final weight; The similarity calculation module is used to: For each graph structure in the first graph structure and the second graph structure, the following operations are performed to calculate feature vectors to obtain a first feature vector of the first JSON format data and a second feature vector of the second JSON format data: a reference node is determined among the nodes included in the graph structure, and a reference feature vector is determined according to a semantic feature vector corresponding to the reference node, and each node in the graph structure is sorted based on a similarity between a semantic feature vector corresponding to each node in the graph structure and the reference feature vector, and according to the sorted order, the semantic feature vector corresponding to each node is input into a pre-trained recurrent neural network in the similarity calculation model to perform feature vector calculation; Calculating a vector difference between the first eigenvector and the second eigenvector, and determining a node-level comparison result vector according to the vector difference; Accordingly, the similarity calculation module is also used for: The concatenation result is input into the pre-trained fully connected layer in the similarity calculation model to obtain the final similarity between the first JSON format data and the second JSON format data.
6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
7. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.
