Unstructured data storage method and system

Abstract

The present invention relates to the technical field of unstructured data storage, and specifically discloses an unstructured data storage method and system. The method comprises obtaining unstructured comment information, converting the comment information into a word vector matrix according to a Word2Vec word vector model; converting the comment information into data nodes, calculating the similarity of any two comment information according to the word vector matrix, and determining the connection relationship between the data nodes according to the similarity; clustering the data nodes based on the connection vectors to create a node cluster, and constructing a feature matrix of the node cluster according to the word vector matrix of each node in the node cluster; updating the connection relationship of each node based on the node cluster; the present invention converts the comment information into data nodes, stores the comment information based on a graph structure, and synchronously constructs the connection relationship of each data node in the storage stage, thereby generating a storage architecture similar to a knowledge graph, which is highly readable and convenient for subsequent analysis and processing.

Description

Technical Field

[0001] The present invention relates to the technical field of unstructured data storage, and in particular to an unstructured data storage system and method. Background Art

[0002] Unstructured data is data with an irregular or incomplete structure, no predefined data model, and is not convenient to represent using a two-dimensional logical table in a database. In the context of existing technology, unstructured data generally refers to unstructured text data, that is, text data without a pre-set format. Product review information is the most typical unstructured text data.

[0003] The existing storage method for this type of data is to use a table with no size limit to sort the comments according to the time or first word of the comment information, and does not consider the internal connection of the comment information. In subsequent processing, it is difficult to quickly locate similar comments, and often an external algorithm is required for classification, which is relatively cumbersome. How to improve the readability of unstructured text data during storage to facilitate subsequent analysis and processing is the technical problem that the present invention aims to solve. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for storing unstructured data to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A first aspect of the present invention provides a method for storing unstructured data, the method comprising:

[0007] Obtain unstructured review information and convert it into a word vector matrix based on the Word2Vec word vector model;

[0008] Convert the comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between the data nodes based on the similarity; the connection relationship includes the names of the two mutually connected data nodes and the connection vector; the module length of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment;

[0009] Cluster the data nodes based on the connection vectors to create node clusters, and construct the feature matrix of the node cluster based on the word vector matrix of each node in the node cluster;

[0010] Update the connection relationship of each node based on the node cluster.

[0011] Furthermore, the step of obtaining unstructured review information and converting the review information into a word vector matrix according to the Word2Vec word vector model includes:

[0012] Obtain unstructured comment information, perform part-of-speech analysis on the comment information, and locate the words in the comment information;

[0013] Convert the word into a word vector according to the Word2Vec word vector model;

[0014] Get the order of words in the comment information, arrange the word vectors according to the order of words in the comment information, and obtain the word vector matrix.

[0015] Furthermore, the steps of converting the comment information into data nodes, calculating the similarity between any two comment information according to the word vector matrix, and determining the connection relationship between the data nodes according to the similarity include:

[0016] Read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0017] Calculate the average vector distance between each word vector and other word vectors, select the word vectors whose average vector distance is less than the preset distance threshold, read the word corresponding to the word vector as the node name;

[0018] Calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node name into the data node;

[0019] Calculate the similarity between any two review messages based on the word vector matrix, determine the direction based on the time sequence of the two review messages, and create a connection vector with a modulus length of similarity based on the direction; the direction is from the previous moment to the next moment;

[0020] Connect the data nodes corresponding to the two review information based on the connection vector;

[0021] The calculation process of the vector distance is as follows:

[0022] Where, is a vector and vector The vector distance, is a vector No. parameters, is a vector No. parameters; is a vector and vector Dimensions;

[0023] The similarity calculation process is:

[0024] Where, is the similarity between two word vector matrices, Indicates the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns. Represents the number of columns in the word vector matrix with a smaller number of columns; is the total number of columns of the word vector matrix with a smaller number of columns, is the total number of columns of the word vector matrix with a larger number of columns; is the first word in the word vector matrix with a smaller number of columns column vectors, is the first column in the word vector matrix with a larger number of columns column vectors; for and The vector distance of Choose a function for the minimum value.

[0025] Furthermore, the steps of clustering the data nodes based on the connection vectors to create a node cluster, and constructing a feature matrix of the node cluster according to the word vector matrix of each node in the node cluster include:

[0026] Performing spectral clustering on the data nodes based on the connection vectors to obtain node clusters;

[0027] Determine the node cluster name based on the names of the data nodes in the node cluster;

[0028] Read the word vector matrix of each data node in the node cluster, connect the word vector matrix of each data node as the feature matrix of the node cluster.

[0029] Furthermore, the step of updating the connection relationship of each node based on the node cluster includes:

[0030] Select node clusters in sequence, traverse the data nodes in the node clusters, and read the connection relationships of the data nodes;

[0031] Query the node name of another data node in the connection relationship to determine whether it belongs to the same node cluster;

[0032] When the node name of the other data node does not belong to the same node cluster, querying the node cluster name of the node cluster where the other data node is located;

[0033] Replace the node name of another data node in the connection relationship with the node cluster name;

[0034] Generate an upper layer vector connected to the node cluster based on the connection vector.

[0035] Furthermore, the step of generating an upper layer vector connected to the node cluster based on the connection vector includes:

[0036] Read the feature matrix of the node cluster;

[0037] Read the word vector matrix corresponding to the data node, traverse the feature matrix through the word vector matrix, and calculate the maximum similarity;

[0038] The direction of the connection vector is retained, the maximum similarity is used as the modulus, and a vector from the data node to the node cluster is generated as the upper vector of the connection vector;

[0039] The upper layer vector nests the connection vector.

[0040] The technical solution of the present invention further provides an unstructured data storage system, the system comprising:

[0041] The word vector conversion module is used to obtain unstructured review information and convert the review information into a word vector matrix based on the Word2Vec word vector model;

[0042] A connection relationship construction module is used to convert comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between the data nodes based on the similarity; the connection relationship includes the names of the two mutually connected data nodes and the connection vector; the module length of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment;

[0043] The data node clustering module is used to cluster data nodes based on connection vectors, create node clusters, and construct the feature matrix of the node cluster based on the word vector matrix of each node in the node cluster;

[0044] The connection relationship updating module is used to update the connection relationship of each node based on the node cluster.

[0045] Furthermore, the word vector conversion module includes:

[0046] A word location unit is used to obtain unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information;

[0047] A word conversion unit, configured to convert the word into a word vector according to a Word2Vec word vector model;

[0048] The word vector sorting unit is used to obtain the order of words in the comment information, arrange the word vectors according to the order of the words in the comment information, and obtain a word vector matrix.

[0049] Furthermore, the connection relationship building module includes:

[0050] A distance calculation unit is used to read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0051] A name generation unit is used to calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, and read the word corresponding to the word vector as the node name;

[0052] The node creation unit is used to calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node name into the data node;

[0053] A vector generation unit is used to calculate the similarity between any two review information based on the word vector matrix, determine the direction based on the time sequence of the two review information, and create a connection vector with a modulus length of similarity based on the direction; the direction points from the previous moment to the next moment;

[0054] A connection execution unit, configured to connect data nodes corresponding to two review information based on a connection vector;

[0055] The calculation process of the vector distance is as follows:

[0056] Where, is a vector and vector The vector distance, is a vector No. parameters, is a vector No. parameters; is the dimension of vector A and vector B;

[0057] The similarity calculation process is:

[0058] Where, is the similarity between two word vector matrices, Indicates the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns. Represents the number of columns in the word vector matrix with a smaller number of columns; is the total number of columns of the word vector matrix with a smaller number of columns, is the total number of columns of the word vector matrix with a larger number of columns; is the first word in the word vector matrix with a smaller number of columns column vectors, is the first column in the word vector matrix with a larger number of columns column vectors; for and The vector distance of Choose a function for the minimum value.

[0059] Furthermore, the data node clustering module includes:

[0060] a clustering execution unit, configured to perform spectral clustering on the data nodes based on the connection vectors to obtain node clusters;

[0061] A name statistics unit, used to determine the node cluster name based on the names of the data nodes in the node cluster;

[0062] The feature extraction unit is used to read the word vector matrix of each data node in the node cluster, connect the word vector matrix of each data node as the feature matrix of the node cluster.

[0063] Compared with the existing technology, the beneficial effects of the present invention are: the present invention converts comment information into data nodes, stores comment information based on a graph structure, and synchronously constructs the connection relationship between each data node in the storage stage, thereby generating a storage architecture similar to a knowledge graph, which is highly readable and facilitates subsequent analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention.

[0065] Figure 1 The figure is a flowchart of an unstructured data storage method.

[0066] Figure 2 This is a structural block diagram of an unstructured data storage system. DETAILED DESCRIPTION

[0067] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] Figure 1 This is a flowchart of a method for storing unstructured data. An embodiment of the present invention provides a method for storing unstructured data, the method comprising:

[0069] Step S100: Obtain unstructured review information and convert the review information into a word vector matrix based on the Word2Vec word vector model;

[0070] Word2vec, also known as word embeddings (or "word vectors" in Chinese), converts words in natural language into dense vectors that computers can understand. This application targets product reviews. The reviews are broken down into sets of words. The Word2Vec word embedding model converts these words into word vectors, with each word corresponding to a specific word vector. A review contains multiple words, and therefore multiple word vectors. These word vectors are then tallied to form a word vector matrix. The final result is a single word vector matrix for each review.

[0071] Step S200: Convert the comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between the data nodes based on the similarity; the connection relationship includes the names of the two interconnected data nodes and the connection vector; the module length of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment;

[0072] The purpose of the technical solution of the present invention is to store comment information in a graph structure, convert each comment information into a data node, and then calculate the similarity between the comment information as the similarity between the data nodes. The connection vector is determined by the similarity, and each data node can be connected.

[0073] Specifically, after each data node and its connection vector are determined, the name of the data node and the connection vector are collectively referred to as a connection relationship; the module length of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment.

[0074] Step S300: clustering the data nodes based on the connection vectors to create node clusters, and constructing a feature matrix of the node cluster based on the word vector matrix of each node in the node cluster;

[0075] After connection vectors are generated between all data nodes, the modulus of the connection vectors can reflect the similarity between each data node. The modulus of the connection vectors is used as the inverse of the distance to cluster the data nodes, and similar data nodes can be classified into one category, called a node cluster. After the node cluster is generated, the feature matrix of the node cluster can be generated based on the word vector matrix of each node in the node cluster. The simplest feature matrix generation scheme is to connect the word vector matrix in a preset order, and the resulting augmented matrix is the feature matrix.

[0076] Step S400: updating the connection relationship of each node based on the node cluster.

[0077] After the node cluster is generated, the connection relationship of each node is updated synchronously. The specific process is that for each data node, the connection relationship between it and another data node is updated to the connection relationship between it and another node cluster (the node cluster to which the other data node belongs).

[0078] In some examples of the technical solution of the present invention, the step of obtaining unstructured review information and converting the review information into a word vector matrix according to the Word2Vec word vector model includes:

[0079] Obtain unstructured comment information, perform part-of-speech analysis on the comment information, and locate the words in the comment information;

[0080] Convert the word into a word vector according to the Word2Vec word vector model;

[0081] Get the order of words in the comment information, arrange the word vectors according to the order of words in the comment information, and obtain the word vector matrix.

[0082] The above content provides a specific solution for converting comment information into a word vector matrix. When the comment information is obtained, since the format of the comment information is not pre-specified, it has many formats. The comment information in different formats is first identified, and the individual words in the comment information are located. Then, the words are converted into word vectors according to the Word2Vec word vector model. Finally, the word vectors are arranged according to the order of the words in the comment information to obtain a matrix, which is called a word vector matrix.

[0083] In some examples of the technical solution of the present invention, the steps of converting the comment information into data nodes, calculating the similarity between any two comment information based on the word vector matrix, and determining the connection relationship between the data nodes based on the similarity include:

[0084] Read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0085] Calculate the average vector distance between each word vector and other word vectors, select the word vectors whose average vector distance is less than the preset distance threshold, read the word corresponding to the word vector as the node name;

[0086] Calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node name into the data node;

[0087] Calculate the similarity between any two review messages based on the word vector matrix, determine the direction based on the time sequence of the two review messages, and create a connection vector with a modulus length of similarity based on the direction; the direction is from the previous moment to the next moment;

[0088] Connect the data nodes corresponding to the two review information based on the connection vector;

[0089] For the word vector matrix of each comment information, each word vector in the word vector matrix corresponds to a word. By calculating the distance between each word vector, the degree of relevance between two words can be determined (the nature of the Word2Vec word vector model itself is that when creating word vectors, the word vectors of related words have a high similarity). For any word vector, the average value of the vector distance with other word vectors is used to obtain the average vector distance. The average vector distance reflects the average degree of relevance of a word vector with all other word vectors. One or several word vectors with the smallest average relevance are selected to represent the comment information as the node name. This process can be understood as extracting keywords.

[0090] Then, obtain the data volume of the word vector matrix (the number of rows and columns. Generally, the number of rows corresponds to the dimension of the word vector. All word vector matrices are the same, but the number of columns is proportional to the number of words). The size of the data node is determined by the data volume. The data node in this application is a circle, so only the radius needs to be determined by the data volume. Combined with the node size and node name, the data node can be constructed.

[0091] Finally, it is necessary to determine the connection vector between the two data nodes. The direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment. The modulus of the connection vector is the similarity between the word vector matrices of the comment information corresponding to the two data nodes (which is a different concept from the distance between any two word vectors in the word vector matrix).

[0092] The calculation process of the vector distance is as follows:

[0093] Where, is a vector and vector The vector distance, is a vector No. parameters, is a vector No. parameters; is the dimension of vector A and vector B;

[0094] The similarity calculation process is:

[0095] Where, is the similarity between two word vector matrices, Indicates the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns. Represents the number of columns in the word vector matrix with a smaller number of columns; is the total number of columns of the word vector matrix with a smaller number of columns, is the total number of columns of the word vector matrix with a larger number of columns; is the first word in the word vector matrix with a smaller number of columns column vectors, is the first column in the word vector matrix with a larger number of columns column vectors; for and The vector distance of Choose a function for the minimum value.

[0096] The calculation principle of the above content is not complicated. The vector distance is limited to the Euclidean distance of the two vectors. Then, for the similarity of the two word vector matrices, since the number of columns of the matrices is different (related to the number of words in the comment information), there is a traversal process in which the small matrix traverses the large matrix. During each traversal, the corresponding column vector distances are calculated one by one, and the average value is calculated. The smallest average value among all traversal results is selected as the minimum distance, and then the reciprocal is calculated as the similarity. The smaller the minimum distance, the higher the similarity, and the more similar the two word vector matrices are.

[0097] In some examples of the technical solution of the present invention, the steps of clustering data nodes based on connection vectors to create node clusters, and constructing a feature matrix of the node cluster based on the word vector matrix of each node in the node cluster include:

[0098] Performing spectral clustering on the data nodes based on the connection vectors to obtain node clusters;

[0099] Determine the node cluster name based on the names of the data nodes in the node cluster;

[0100] Read the word vector matrix of each data node in the node cluster, connect the word vector matrix of each data node as the feature matrix of the node cluster.

[0101] Spectral clustering, an algorithm derived from graph theory, has been widely used in clustering. Its key idea is to treat all data as points in space, connected by edges. Edge weights between distant points are low, while those between closer points are high. Clustering is achieved by slicing the graph of all data points, ensuring that the sum of edge weights between different subgraphs is as low as possible, while the sum of edge weights within a subgraph is as high as possible.

[0102] This application has converted the comment information into data nodes, and the modulus length of the connection vector between the data nodes is the similarity. Therefore, the spectral clustering algorithm can be directly applied with extremely high adaptability.

[0103] After spectral clustering, data nodes that are grouped into the same category are called node clusters. The names of the data nodes in the node cluster are counted and used as the node cluster name (the node name with the most occurrences can also be selected as the node cluster name, which only serves as a label). Then, the word vector matrix of each data node in the node cluster is read and the word vector matrices of each data node are concatenated to form the feature matrix of the node cluster.

[0104] In some examples of the technical solution of the present invention, the step of updating the connection relationship of each node based on the node cluster includes:

[0105] Select node clusters in sequence, traverse the data nodes in the node clusters, and read the connection relationships of the data nodes;

[0106] Query the node name of another data node in the connection relationship to determine whether it belongs to the same node cluster;

[0107] When the node name of the other data node does not belong to the same node cluster, querying the node cluster name of the node cluster where the other data node is located;

[0108] Replace the node name of another data node in the connection relationship with the node cluster name;

[0109] Generate an upper layer vector connected to the node cluster based on the connection vector.

[0110] Not all data connection relationships require a replacement process. When two data nodes belong to the same node cluster, there is no need to perform a replacement process.

[0111] Specifically, each node cluster is analyzed separately, and the data nodes in the node cluster are traversed to query which data node the other end points to and which node cluster the pointed data node belongs to. The node name of the pointed data node in the connection relationship is replaced with the node cluster name, and the connection vector is updated synchronously.

[0112] Specifically, the step of generating an upper layer vector connected to the node cluster based on the connection vector includes:

[0113] Read the feature matrix of the node cluster;

[0114] Read the word vector matrix corresponding to the data node, traverse the feature matrix through the word vector matrix, and calculate the maximum similarity;

[0115] The direction of the connection vector is retained, the maximum similarity is used as the modulus, and a vector from the data node to the node cluster is generated as the upper vector of the connection vector;

[0116] The upper layer vector nests the connection vector.

[0117] The above content limits the update process of the connection vector. The update process in the technical solution of this application is a nested process. It does not directly delete the connection vector, but introduces a vector pointing to the node cluster on the basis of the connection vector as a new connection vector. The new connection vector and the original connection vector are in an inclusion relationship, and the direction of the new connection vector is the same as the direction of the original connection vector. The module length adopts the maximum similarity of the word vector matrix and the feature matrix. The meaning of the maximum similarity is that since the size of the word vector matrix is smaller than that of the feature matrix, only a part can be selected for comparison in each traversal process, and multiple similarities are obtained. The maximum value of the similarity is selected as the final similarity.

[0118] Figure 2 is a structural block diagram of an unstructured data storage system. Another object of an embodiment of the present invention is to provide an unstructured data storage system, wherein the system 10 includes:

[0119] A word vector conversion module 11 is used to obtain unstructured comment information and convert the comment information into a word vector matrix according to the Word2Vec word vector model;

[0120] A connection relationship construction module 12 is used to convert comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between the data nodes based on the similarity; the connection relationship includes the names of the two interconnected data nodes and the connection vector; the module length of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment;

[0121] The data node clustering module 13 is used to cluster the data nodes based on the connection vectors, create node clusters, and construct a feature matrix of the node cluster based on the word vector matrix of each node in the node cluster;

[0122] The connection relationship updating module 14 is configured to update the connection relationship of each node based on the node cluster.

[0123] Furthermore, the word vector conversion module 11 includes:

[0124] A word location unit is used to obtain unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information;

[0125] A word conversion unit, configured to convert the word into a word vector according to a Word2Vec word vector model;

[0126] The word vector sorting unit is used to obtain the order of words in the comment information, arrange the word vectors according to the order of the words in the comment information, and obtain a word vector matrix.

[0127] Specifically, the connection relationship building module 12 includes:

[0128] A distance calculation unit is used to read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0129] A name generation unit is used to calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, and read the word corresponding to the word vector as the node name;

[0130] The node creation unit is used to calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node name into the data node;

[0131] A vector generation unit is used to calculate the similarity between any two review information based on the word vector matrix, determine the direction based on the time sequence of the two review information, and create a connection vector with a modulus length of similarity based on the direction; the direction points from the previous moment to the next moment;

[0132] A connection execution unit, configured to connect data nodes corresponding to two review information based on a connection vector;

[0133] The calculation process of the vector distance is as follows:

[0134] Where, is a vector and vector The vector distance, is a vector No. parameters, is a vector No. parameters; is a vector and vector Dimensions;

[0135] The similarity calculation process is:

[0136] Where, is the similarity between two word vector matrices, Indicates the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns. Represents the number of columns in the word vector matrix with a smaller number of columns; is the total number of columns of the word vector matrix with a smaller number of columns, is the total number of columns of the word vector matrix with a larger number of columns; is the first word in the word vector matrix with a smaller number of columns column vectors, is the first column in the word vector matrix with a larger number of columns column vectors; for and The vector distance of Choose a function for the minimum value.

[0137] Furthermore, the data node clustering module 13 includes:

[0138] a clustering execution unit, configured to perform spectral clustering on the data nodes based on the connection vectors to obtain node clusters;

[0139] A name statistics unit, used to determine the node cluster name based on the names of the data nodes in the node cluster;

[0140] The feature extraction unit is used to read the word vector matrix of each data node in the node cluster, connect the word vector matrix of each data node as the feature matrix of the node cluster.

[0141] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for storing unstructured data, characterized in that: The method comprises: Obtain unstructured review information and convert it into a word vector matrix based on the Word2Vec word vector model. Each word vector in the word vector matrix corresponds to a word. Convert the comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between the data nodes based on the similarity; the connection relationship includes the names of the two mutually connected data nodes and the connection vector; the module length of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment; Cluster the data nodes based on the connection vectors to create node clusters, and construct the feature matrix of the node cluster based on the word vector matrix of each node in the node cluster; Update the connection relationship of each node based on the node cluster; The steps of converting the comment information into data nodes, calculating the similarity between any two comment information according to the word vector matrix, and determining the connection relationship between the data nodes according to the similarity include: Read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix; Calculate the average vector distance between each word vector and other word vectors, select the word vectors whose average vector distance is less than the preset distance threshold, read the word corresponding to the word vector as the node name; Calculate the data volume of the word vector matrix, determine the node size based on the data volume, create a data node based on the node size, and insert the node name into the data node; the number of rows in the word vector corresponds to the dimension of the word vector, and the number of columns in the word vector is proportional to the number of words. The data volume is determined by the number of columns. The data node is a circle, and the node size is the radius. The similarity calculation process is: Where, S is the similarity between the two word vector matrices, i Indicates the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns. j Represents the number of columns in the word vector matrix with a smaller number of columns; M is the total number of columns of the word vector matrix with a smaller number of columns, is the total number of columns of the word vector matrix with a larger number of columns; is the first word in the word vector matrix with a smaller number of columns column vectors, is the first column in the word vector matrix with a larger number of columns column vectors; for and The vector distance of Select the function for the minimum value; The step of updating the connection relationship of each node based on the node cluster includes: Select node clusters in sequence, traverse the data nodes in the node clusters, and read the connection relationships of the data nodes; Query the node name of another data node in the connection relationship to determine whether it belongs to the same node cluster; When the node name of the other data node does not belong to the same node cluster, querying the node cluster name of the node cluster where the other data node is located; Replace the node name of another data node in the connection relationship with the node cluster name; Generate an upper layer vector connected to the node cluster based on the connection vector; The step of generating an upper layer vector connected to the node cluster based on the connection vector includes: Read the feature matrix of the node cluster; Read the word vector matrix corresponding to the data node, traverse the feature matrix through the word vector matrix, and calculate the maximum similarity; The direction of the connection vector is retained, the maximum similarity is used as the modulus, and a vector from the data node to the node cluster is generated as the upper vector of the connection vector; The upper layer vector nests the connection vector.

2. The unstructured data storage method according to claim 1, characterized in that: The steps of clustering the data nodes based on the connection vectors to create a node cluster, and constructing a feature matrix of the node cluster according to the word vector matrix of each node in the node cluster include: Performing spectral clustering on the data nodes based on the connection vectors to obtain node clusters; Determine the node cluster name based on the names of the data nodes in the node cluster; Read the word vector matrix of each data node in the node cluster, connect the word vector matrix of each data node as the feature matrix of the node cluster.

3. An unstructured data storage system, characterized in that: The system comprises: The word vector conversion module is used to obtain unstructured review information and convert the review information into a word vector matrix based on the Word2Vec word vector model. Each word vector in the word vector matrix corresponds to a word. A connection relationship construction module is used to convert comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between the data nodes based on the similarity; the connection relationship includes the names of the two mutually connected data nodes and the connection vector; the module length of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous moment to the data node of the comment information at the next moment; The data node clustering module is used to cluster data nodes based on connection vectors, create node clusters, and construct the feature matrix of the node cluster based on the word vector matrix of each node in the node cluster; A connection relationship updating module is used to update the connection relationship of each node based on the node cluster; The steps of converting the comment information into data nodes, calculating the similarity between any two comment information based on the word vector matrix, and determining the connection relationship between the data nodes based on the similarity include: Read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix; Calculate the average vector distance between each word vector and other word vectors, select the word vectors whose average vector distance is less than the preset distance threshold, read the word corresponding to the word vector as the node name; Calculate the data volume of the word vector matrix, determine the node size based on the data volume, create a data node based on the node size, and insert the node name into the data node; the number of rows in the word vector corresponds to the dimension of the word vector, and the number of columns in the word vector is proportional to the number of words. The data volume is determined by the number of columns. The data node is a circle, and the node size is the radius. The similarity calculation process is: Where, S is the similarity between two word vector matrices, i Indicates the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns. j Represents the number of columns in the word vector matrix with a smaller number of columns; M is the total number of columns of the word vector matrix with a smaller number of columns, is the total number of columns of the word vector matrix with a larger number of columns; is the first word in the word vector matrix with a smaller number of columns column vectors, is the first column in the word vector matrix with a larger number of columns column vectors; for and The vector distance of Select the function for the minimum value; The content of updating the connection relationship of each node based on the node cluster includes: Select node clusters in sequence, traverse the data nodes in the node clusters, and read the connection relationships of the data nodes; Query the node name of another data node in the connection relationship to determine whether it belongs to the same node cluster; When the node name of the other data node does not belong to the same node cluster, querying the node cluster name of the node cluster where the other data node is located; Replace the node name of another data node in the connection relationship with the node cluster name; Generate an upper layer vector connected to the node cluster based on the connection vector; The content of generating an upper layer vector connected to the node cluster based on the connection vector includes: Read the feature matrix of the node cluster; Read the word vector matrix corresponding to the data node, traverse the feature matrix through the word vector matrix, and calculate the maximum similarity; The direction of the connection vector is retained, the maximum similarity is used as the modulus, and a vector from the data node to the node cluster is generated as the upper vector of the connection vector; The upper layer vector nests the connection vector.

4. The unstructured data storage system according to claim 3, wherein: The data node clustering module includes: a clustering execution unit, configured to perform spectral clustering on the data nodes based on the connection vectors to obtain node clusters; A name statistics unit, used to determine the node cluster name based on the names of the data nodes in the node cluster; The feature extraction unit is used to read the word vector matrix of each data node in the node cluster, connect the word vector matrix of each data node as the feature matrix of the node cluster.

Images