Hierarchical classification storage method and system for data

Abstract

The application relates to the technical field of hierarchical classified storage of data, and particularly discloses a hierarchical classified storage method and system of data, the method comprising the following steps: obtaining unstructured comment information, converting the comment information into a word vector matrix; converting the comment information into data nodes, calculating the similarity of any two pieces of comment information according to the word vector matrix, and determining the connection relationship between the data nodes according to the similarity; clustering the data nodes based on a connection vector, creating a node cluster, constructing a feature matrix of the node cluster according to the word vector matrix of each node in the node cluster; and updating the connection relationship of each node based on the node cluster; the comment information is converted into data nodes, the comment information is stored based on a graph structure, the connection relationship of each data node is synchronously constructed in the storage stage, and then a storage architecture similar to a knowledge graph is generated, which is extremely high in readability and is convenient for subsequent analysis and processing.

Description

[0001] This application is a divisional application of the invention application filed on July 3, 2024, with Chinese application number 2024108817902 and entitled "A Method and System for Unstructured Data Storage". Technical Field

[0002] This invention relates to the field of hierarchical and classified data storage technology, specifically a hierarchical and classified data storage system and method. Background Technology

[0003] Text data refers to data composed of characters, with basic units such as letters, numbers, punctuation marks, and spaces. It can be words, sentences, paragraphs, or complete documents. Text data is a very important type of media data, and storage needs are extremely common. Currently, most methods of storing text data use very simple classification and storage architectures, such as storing each article or paragraph in an independent storage space. This approach is feasible, but the text data in different storage spaces is too independent. In reality, there are many connections between text data in the same scenario, and the applicability of different text data is also different. In other words, different text data have different levels. Therefore, how to classify and classify text data for storage based on the inherent content of the text is the technical problem that this invention aims to solve. Summary of the Invention

[0004] The purpose of this invention is to provide a hierarchical and categorized data storage method, the method comprising:

[0005] Obtain unstructured comment information and convert it into a word vector matrix; each word vector in the word vector matrix corresponds to a word.

[0006] Comment information is converted into data nodes. The similarity between any two comment information is calculated based on the word vector matrix. The connection relationship between data nodes is determined based on the similarity. The connection relationship includes the names of the two interconnected data nodes and the connection vector. The magnitude of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous time step to the data node of the comment information at the next time step.

[0007] Clustering of data nodes based on connection vectors creates node clusters, and feature matrices of node clusters are constructed based on word vector matrices of each node in the cluster.

[0008] Update the connection relationships of each node based on the node cluster;

[0009] The similarity S between the two word vector matrices is as follows:

[0010] In the formula, i represents the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns, j represents the column number in the word vector matrix with a smaller number of columns; M is the total number of columns in the word vector matrix with a smaller number of columns, and N is the total number of columns in the word vector matrix with a larger number of columns; A j B is the j-th column vector in a word vector matrix with a relatively small number of columns. j+i d(A) is the (j+i)th column vector in a word vector matrix with a relatively large number of columns; j B j+i ) is A j and B j+i The vector distance; MIN{} is the minimum value selection function.

[0011] Furthermore, the step of obtaining unstructured comment information and converting the comment information into a word vector matrix includes:

[0012] Obtain unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information;

[0013] The words are converted into word vectors according to the Word2Vec word vector model;

[0014] Obtain the order of words in the comment information, arrange the word vectors according to the order of words in the comment information, and obtain the word vector matrix.

[0015] Furthermore, the steps of converting comment information into data nodes, calculating the similarity between any two comment pieces based on the word vector matrix, and determining the connection relationship between data nodes based on the similarity include:

[0016] Read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0017] Calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, read the words corresponding to the word vectors, and use them as node names;

[0018] Calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node names into the data nodes;

[0019] The similarity between any two comments is calculated based on the word vector matrix. The direction is determined based on the time sequence of the two comments. A connection vector with a modulus equal to the similarity is created based on the direction. The direction is from one moment to the next.

[0020] Connect the data nodes corresponding to two comment information based on the connection vector;

[0021] The calculation process for the vector distance is as follows:

[0022] In the formula, d(A,B) is the vector distance between vectors A and B, and x(A) is the vector distance between them. k Let x(B) be the k-th parameter of vector A. k Let be the k-th parameter of vector B; L is the dimension of vectors A and B.

[0023] Furthermore, the step of clustering data nodes based on connection vectors to create node clusters, and constructing the feature matrix of the node clusters based on the word vector matrices of each node in the node clusters includes:

[0024] Based on the connection vectors, spectral clustering is performed on the data nodes to obtain node clusters;

[0025] The name of the node cluster is determined based on the names of the data nodes in the node cluster;

[0026] Read the word vector matrix of each data node in the node cluster, and concatenate the word vector matrices of each data node to form the feature matrix of the node cluster.

[0027] Furthermore, the step of updating the connection relationships of each node based on the node cluster includes:

[0028] Select node clusters sequentially, traverse the data nodes in the node clusters, and read the connection relationships of the data nodes;

[0029] Query the name of another data node in the connection relationship to determine whether it belongs to the same node cluster;

[0030] When the node name of the other data node does not belong to the same node cluster, query the node cluster name of the node cluster to which the other data node belongs;

[0031] Replace the node name of the other data node in the connection relationship with the node cluster name;

[0032] Generate upper-level vectors that are connected to the node cluster based on the connection vector.

[0033] Furthermore, the step of generating the upper-level vector connected to the node cluster based on the connection vector includes:

[0034] Read the feature matrix of the node cluster;

[0035] Read the word vector matrix corresponding to the data node, traverse the feature matrix using the word vector matrix, and calculate the maximum similarity.

[0036] Preserve the direction of the connection vector, use the maximum similarity as the modulus, and generate a vector from the data node to the node cluster, which serves as the upper-level vector of the connection vector;

[0037] The upper-level vector is nested within the connection vector.

[0038] The present invention also provides a hierarchical and classified data storage system, the system comprising:

[0039] The word vector conversion module is used to obtain unstructured comment information and convert it into a word vector matrix; each word vector in the word vector matrix corresponds to a word.

[0040] The connection relationship construction module is used to convert comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between data nodes based on the similarity. The connection relationship includes the names of two interconnected data nodes and a connection vector. The magnitude of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous time step to the data node of the comment information at the next time step.

[0041] The data node clustering module is used to cluster data nodes based on connection vectors, create node clusters, and construct the feature matrix of the node cluster based on the word vector matrix of each node in the node cluster.

[0042] The connection relationship update module is used to update the connection relationships of each node based on the node cluster;

[0043] The similarity S between the two word vector matrices is as follows:

[0044] In the formula, i represents the starting column number of the word vector matrix with a smaller number of columns in the word vector matrix with a larger number of columns, j represents the column number in the word vector matrix with a smaller number of columns; M is the total number of columns in the word vector matrix with a smaller number of columns, and N is the total number of columns in the word vector matrix with a larger number of columns; A j B is the j-th column vector in a word vector matrix with a relatively small number of columns. j+i d(A) is the (j+i)th column vector in a word vector matrix with a relatively large number of columns; j B j+i ) is A j and B j+i The vector distance; MIN{} is the minimum value selection function.

[0045] Furthermore, the word vector conversion module includes:

[0046] The word localization unit is used to acquire unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information;

[0047] The word conversion unit is used to convert the words into word vectors according to the Word2Vec word vector model;

[0048] The word vector sorting unit is used to obtain the order of words in the comment information, and arrange the word vectors according to the order of words in the comment information to obtain the word vector matrix.

[0049] Furthermore, the connection relationship construction module includes:

[0050] The distance calculation unit is used to read the word vector matrix of comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0051] The name generation unit is used to calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, read the words corresponding to the word vectors, and use them as node names;

[0052] The node creation unit is used to calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node name into the data node;

[0053] The vector generation unit is used to calculate the similarity between any two comment messages based on the word vector matrix, determine the direction based on the time sequence of the two comment messages, and create a connection vector with a magnitude equal to the similarity based on the direction; the direction is from the previous time to the next time.

[0054] The connection execution unit is used to connect data nodes corresponding to two comment information based on the connection vector;

[0055] The calculation process for the vector distance is as follows:

[0056] In the formula, d(A,B) is the vector distance between vectors A and B, and x(A) is the vector distance between them. k Let x(B) be the k-th parameter of vector A. k Let be the k-th parameter of vector B; L is the dimension of vectors A and B.

[0057] Furthermore, the data node clustering module includes:

[0058] A clustering execution unit is used to perform spectral clustering on data nodes based on the connection vector to obtain node clusters;

[0059] The name statistics unit is used to determine the name of the node cluster based on the names of the data nodes in the node cluster.

[0060] The feature extraction unit is used to read the word vector matrix of each data node in the node cluster, and connect the word vector matrices of each data node to form the feature matrix of the node cluster.

[0061] Compared with the prior art, the beneficial effects of the present invention are: the present invention converts comment information into data nodes, stores comment information based on a graph structure, and synchronously constructs the connection relationship between each data node during the storage stage, thereby generating a storage architecture similar to a knowledge graph, which has extremely high readability and facilitates subsequent analysis and processing, thus solving the technical problems involved in the background art. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0063] Figure 1 This is a flowchart illustrating a hierarchical and categorized data storage method.

[0064] Figure 2 This is a structural block diagram of a hierarchical and classified data storage system. Detailed Implementation

[0065] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0066] Figure 1 The flowchart illustrates a hierarchical and categorized data storage method. This embodiment of the invention provides a hierarchical and categorized data storage method, the method comprising:

[0067] Step S100: Obtain unstructured comment information and convert it into a word vector matrix using the Word2Vec word vector model;

[0068] Word2vec, also known as word embeddings, is used to convert words in natural language into dense vectors that computers can understand. This application addresses product review information by breaking down the review information into sets of words. The Word2Vec word vector model converts these words into word vectors, with each word corresponding to a word vector. Since a review contains multiple words, it generates multiple word vectors. These word vectors are then analyzed to obtain a word vector matrix. The final result is that each review corresponds to a single word vector matrix.

[0069] Step S200: Convert comment information into data nodes, calculate the similarity between any two comment pieces based on the word vector matrix, and determine the connection relationship between data nodes based on the similarity; the connection relationship includes the names of the two interconnected data nodes and the connection vector; the magnitude of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous time step to the data node of the comment information at the next time step;

[0070] The purpose of this invention is to store comment information in a graph structure, convert each comment information into a data node, calculate the similarity between comment information as the similarity between data nodes, determine the connection vector based on the similarity, and then connect the various data nodes.

[0071] Specifically, after each data node and its connection vector are determined, the names of the data nodes and the connection vectors are collectively referred to as connection relationships; the magnitude of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous time to the data node of the comment information at the next time.

[0072] Step S300: Cluster the data nodes based on the connection vectors to create node clusters, and construct the feature matrix of the node clusters based on the word vector matrix of each node in the node cluster;

[0073] After generating connection vectors between all data nodes, the magnitude of the connection vectors can reflect the similarity between the data nodes. Using the magnitude of the connection vectors as the inverse of the distance, clustering the data nodes allows similar data nodes to be grouped into one class, called a node cluster. After generating the node cluster, the feature matrix of the node cluster can be generated based on the word vector matrix of each node in the node cluster. The simplest feature matrix generation scheme is to connect the word vector matrices in a preset order, and the resulting augmented matrix is ​​the feature matrix.

[0074] Step S400: Update the connection relationships of each node based on the node cluster.

[0075] After generating the node cluster, the connection relationships of each node are updated synchronously. Specifically, for each data node, the connection relationship between it and another data node is updated to the connection relationship between it and another node cluster (the node cluster to which the other data node belongs).

[0076] In some embodiments of the technical solution of this invention, the step of obtaining unstructured comment information and converting the comment information into a word vector matrix according to the Word2Vec word vector model includes:

[0077] Obtain unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information;

[0078] The words are converted into word vectors according to the Word2Vec word vector model;

[0079] Obtain the order of words in the comment information, arrange the word vectors according to the order of words in the comment information, and obtain the word vector matrix.

[0080] The above provides a specific scheme for converting comment information into a word vector matrix. When comment information is obtained, since the format of comment information is not predefined and there are many formats, content recognition is first performed on comment information of different formats to locate each word in the comment information. Then, the words are converted into word vectors according to the Word2Vec word vector model. Finally, the word vectors are arranged according to the order of the words in the comment information to obtain a matrix called the word vector matrix.

[0081] In some embodiments of the technical solution of this invention, the steps of converting comment information into data nodes, calculating the similarity between any two comment pieces based on the word vector matrix, and determining the connection relationship between data nodes based on the similarity include:

[0082] Read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0083] Calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, read the words corresponding to the word vectors, and use them as node names;

[0084] Calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node names into the data nodes;

[0085] The similarity between any two comments is calculated based on the word vector matrix. The direction is determined based on the time sequence of the two comments. A connection vector with a modulus equal to the similarity is created based on the direction. The direction is from one moment to the next.

[0086] Connect the data nodes corresponding to two comment information based on the connection vector;

[0087] For each comment's word vector matrix, each word vector corresponds to a word. Calculating the distance between word vectors can determine the relevance between two words (due to the nature of the Word2Vec word vector model, related words have high similarity when creating word vectors). For any given word vector, the average vector distance is calculated by taking the average distance between it and other word vectors. The average vector distance reflects the average relevance of a word vector to all other word vectors. Selecting one or more word vectors with the lowest average relevance to represent the comment information and serve as the node name can be understood as keyword extraction.

[0088] Then, obtain the data volume of the word vector matrix (number of rows and columns; generally, the number of rows corresponds to the dimension of the word vector, and all word vector matrices are the same, but the number of columns is proportional to the number of words). The size of the data node is determined by the data volume. In this application, the data node is a circle, so only the radius needs to be determined by the data volume. The data node can be constructed by combining the node size and the node name.

[0089] Finally, it is necessary to determine the connection vector between the two data nodes. The direction of the connection vector is from the data node of the comment information at the previous time step to the data node of the comment information at the next time step. The magnitude of the connection vector is the similarity between the word vector matrices of the comment information corresponding to the two data nodes (which is a different concept from the distance between any two word vectors in the word vector matrix).

[0090] The calculation process for the vector distance is as follows:

[0091] In the formula, d(A,B) is the vector distance between vectors A and B, and x(A) is the vector distance between them. k Let x(B) be the k-th parameter of vector A. k Let be the k-th parameter of vector B; L is the dimension of vectors A and B;

[0092] The similarity calculation process is as follows:

[0093] In the formula, S represents the similarity between the two word vector matrices, i represents the starting column of the smaller word vector matrix in the larger word vector matrix, j represents the column number in the smaller word vector matrix; M is the total number of columns in the smaller word vector matrix, and N is the total number of columns in the larger word vector matrix; A j B is the j-th column vector in a word vector matrix with a relatively small number of columns. j+i d(A) is the (j+i)th column vector in a word vector matrix with a relatively large number of columns; j B j+i ) is A j and B j+iThe vector distance; MIN{} is the minimum value selection function.

[0094] The calculation principle described above is not complicated. The vector distance is limited to the Euclidean distance between two vectors. Then, for the similarity of two word vector matrices, since the number of columns in the matrices is different (related to the number of words in the comment information), there is a traversal process. The smaller matrix traverses the larger matrix. During each traversal, the distance of the corresponding column vectors is calculated one by one, and the average value is calculated. The smallest average value among all traversal results is selected as the minimum distance. Then, the reciprocal is calculated, which can be used as the similarity. The smaller the minimum distance, the higher the similarity, and the more similar the two word vector matrices are.

[0095] In some embodiments of the technical solution of this invention, the step of clustering data nodes based on connection vectors to create node clusters, and constructing the feature matrix of the node clusters based on the word vector matrices of each node in the node clusters includes:

[0096] Based on the connection vectors, spectral clustering is performed on the data nodes to obtain node clusters;

[0097] The name of the node cluster is determined based on the names of the data nodes in the node cluster;

[0098] Read the word vector matrix of each data node in the node cluster, and concatenate the word vector matrices of each data node to form the feature matrix of the node cluster.

[0099] Spectral clustering is an algorithm that evolved from graph theory and has since been widely applied in clustering. Its main idea is to treat all data points as points in a space, connected by edges. Edges between points that are far apart have lower weights, while those between points that are close together have higher weights. By slicing the graph composed of all data points, the goal is to minimize the sum of edge weights between different subgraphs and maximize the sum of edge weights within each subgraph, thus achieving the purpose of clustering.

[0100] This application has converted comment information into data nodes, and the magnitude of the connection vector between data nodes is the similarity. Therefore, it can be directly applied when using spectral clustering algorithm, with extremely high adaptability.

[0101] After performing spectral clustering, data nodes that are grouped into the same category are called node clusters. The names of each data node in a node cluster are counted and can be used as the node cluster name (or the name of the node that appears most frequently can be selected as the node cluster name, which only serves as a label). Then, the word vector matrix of each data node in the node cluster is read, and the word vector matrices of each data node are concatenated to form the feature matrix of the node cluster.

[0102] In some embodiments of the technical solution of the present invention, the step of updating the connection relationship of each node based on the node cluster includes:

[0103] Select node clusters sequentially, traverse the data nodes in the node clusters, and read the connection relationships of the data nodes;

[0104] Query the name of another data node in the connection relationship to determine whether it belongs to the same node cluster;

[0105] When the node name of the other data node does not belong to the same node cluster, query the node cluster name of the node cluster to which the other data node belongs;

[0106] Replace the node name of the other data node in the connection relationship with the node cluster name;

[0107] Generate upper-level vectors that are connected to the node cluster based on the connection vector.

[0108] Not all data connections require a replacement process. When two data nodes belong to the same node cluster, a replacement process is not necessary.

[0109] Specifically, each node cluster is analyzed separately. The data nodes in the cluster are traversed to find which data node the other end points to and which node cluster the pointed-to data node belongs to. The node name of the pointed-to data node in the connection relationship is replaced with the node cluster name, and the connection vector is updated synchronously.

[0110] Specifically, the step of generating the upper-level vector connected to the node cluster based on the connection vector includes:

[0111] Read the feature matrix of the node cluster;

[0112] Read the word vector matrix corresponding to the data node, traverse the feature matrix using the word vector matrix, and calculate the maximum similarity.

[0113] Preserve the direction of the connection vector, use the maximum similarity as the modulus, and generate a vector from the data node to the node cluster, which serves as the upper-level vector of the connection vector;

[0114] The upper-level vector is nested within the connection vector.

[0115] The above content defines the update process of the connection vector. The update process in this application is a nested process. It does not directly delete the connection vector, but introduces a vector pointing to the node cluster as a new connection vector based on the connection vector. The new connection vector is contained in the original connection vector, and the direction of the new connection vector is the same as the direction of the original connection vector. The modulus is the maximum similarity between the word vector matrix and the feature matrix. The meaning of maximum similarity is that, since the size of the word vector matrix is ​​smaller than that of the feature matrix, only a part can be selected for comparison in each traversal process, thus obtaining multiple similarities. The maximum similarity is selected as the final similarity.

[0116] Figure 2 This is a structural block diagram of a hierarchical and classified data storage system. Another objective of this invention is to provide a hierarchical and classified data storage system, wherein system 10 includes:

[0117] The word vector conversion module 11 is used to obtain unstructured comment information and convert the comment information into a word vector matrix according to the Word2Vec word vector model;

[0118] The connection relationship construction module 12 is used to convert comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between data nodes based on the similarity. The connection relationship includes the names of two interconnected data nodes and a connection vector. The magnitude of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous time to the data node of the comment information at the next time.

[0119] The data node clustering module 13 is used to cluster data nodes based on connection vectors, create node clusters, and construct the feature matrix of the node clusters based on the word vector matrix of each node in the node cluster.

[0120] The connection relationship update module 14 is used to update the connection relationship of each node based on the node cluster.

[0121] Furthermore, the word vector conversion module 11 includes:

[0122] The word localization unit is used to acquire unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information;

[0123] The word conversion unit is used to convert the words into word vectors according to the Word2Vec word vector model;

[0124] The word vector sorting unit is used to obtain the order of words in the comment information, and arrange the word vectors according to the order of words in the comment information to obtain the word vector matrix.

[0125] Specifically, the connection relationship construction module 12 includes:

[0126] The distance calculation unit is used to read the word vector matrix of comment information and calculate the vector distance between any two word vectors in the word vector matrix;

[0127] The name generation unit is used to calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, read the words corresponding to the word vectors, and use them as node names;

[0128] The node creation unit is used to calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node name into the data node;

[0129] The vector generation unit is used to calculate the similarity between any two comment messages based on the word vector matrix, determine the direction based on the time sequence of the two comment messages, and create a connection vector with a magnitude equal to the similarity based on the direction; the direction is from the previous time to the next time.

[0130] The connection execution unit is used to connect data nodes corresponding to two comment information based on the connection vector;

[0131] The calculation process for the vector distance is as follows:

[0132] In the formula, d(A,B) is the vector distance between vectors A and B, and x(A) is the vector distance between them. k Let x(B) be the k-th parameter of vector A. k Let be the k-th parameter of vector B; L is the dimension of vectors A and B;

[0133] The similarity calculation process is as follows:

[0134] In the formula, S represents the similarity between the two word vector matrices, i represents the starting column of the smaller word vector matrix in the larger word vector matrix, j represents the column number in the smaller word vector matrix; M is the total number of columns in the smaller word vector matrix, and N is the total number of columns in the larger word vector matrix; A j B is the j-th column vector in a word vector matrix with a relatively small number of columns. j+i d(A) is the (j+i)th column vector in a word vector matrix with a relatively large number of columns; j B j+i ) is A j and B j+i The vector distance; MIN{} is the minimum value selection function.

[0135] Furthermore, the data node clustering module 13 includes:

[0136] A clustering execution unit is used to perform spectral clustering on data nodes based on the connection vector to obtain node clusters;

[0137] The name statistics unit is used to determine the name of the node cluster based on the names of the data nodes in the node cluster.

[0138] The feature extraction unit is used to read the word vector matrix of each data node in the node cluster, and connect the word vector matrices of each data node to form the feature matrix of the node cluster.

[0139] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A hierarchical and classified data storage method, characterized in that, The method includes: Obtain unstructured comment information and convert it into a word vector matrix; each word vector in the word vector matrix corresponds to a word. Comment information is converted into data nodes. The similarity between any two comment information is calculated based on the word vector matrix. The connection relationship between data nodes is determined based on the similarity. The connection relationship includes the names of the two interconnected data nodes and the connection vector. The magnitude of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous time step to the data node of the comment information at the next time step. Clustering of data nodes based on connection vectors creates node clusters, and feature matrices of node clusters are constructed based on word vector matrices of each node in the node cluster. Update the connection relationships of each node based on the node cluster; The similarity S between the two word vector matrices is as follows: In the formula, i represents the starting column number of the word vector matrix with fewer columns in the traversal of the word vector matrix with more columns, j represents the column number in the word vector matrix with fewer columns, and M is the total number of columns in the word vector matrix with fewer columns. It represents the total number of columns in a word vector matrix with a relatively large number of columns; The word vector matrix with a smaller number of columns is the first one. column vectors, The first word in a word vector matrix with a large number of columns Column vectors; for and Vector distance; Select a function for the minimum value; The steps for updating the connection relationships of each node based on the node cluster include: Each node cluster is analyzed separately. The data nodes in the node cluster are traversed to find which data node its other end points to and which node cluster the pointed-to data node belongs to. The node name of the pointed-to data node in the connection relationship is replaced with the node cluster name, and the connection vector is updated synchronously. The process of synchronously updating the connection vector involves introducing a new vector pointing to the node cluster as the new connection vector. The new connection vector is contained within the original connection vector, and the direction of the new connection vector is the same as that of the original connection vector. The modulus is the maximum similarity between the word vector matrix and the feature matrix.

2. The hierarchical and classified data storage method according to claim 1, characterized in that, The steps of obtaining unstructured comment information and converting it into a word vector matrix include: Obtain unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information; The words are converted into word vectors according to the Word2Vec word vector model; Obtain the order of words in the comment information, arrange the word vectors according to the order of words in the comment information, and obtain the word vector matrix.

3. The hierarchical and classified data storage method according to claim 1, characterized in that, The steps of converting comment information into data nodes, calculating the similarity between any two comment pieces based on the word vector matrix, and determining the connection relationship between data nodes based on the similarity include: Read the word vector matrix of the comment information and calculate the vector distance between any two word vectors in the word vector matrix; Calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, read the words corresponding to the word vectors, and use them as node names; Calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node names into the data nodes; The similarity between any two comments is calculated based on the word vector matrix. The direction is determined based on the time sequence of the two comments. A connection vector with a modulus equal to the similarity is created based on the direction. The direction is from one moment to the next. Connect the data nodes corresponding to two comment information based on the connection vector; The calculation process for the vector distance is as follows: In the formula, For vectors sum vector vector distance, For vectors The One parameter, For vectors The One parameter; For vectors sum vector Dimensions.

4. The hierarchical and classified data storage method according to claim 1, characterized in that, The steps of clustering data nodes based on connection vectors to create node clusters, and constructing the feature matrix of the node clusters based on the word vector matrices of each node in the node clusters include: Based on the connection vectors, spectral clustering is performed on the data nodes to obtain node clusters; The name of the node cluster is determined based on the names of the data nodes in the node cluster; Read the word vector matrix of each data node in the node cluster, and concatenate the word vector matrices of each data node to form the feature matrix of the node cluster.

5. The hierarchical and classified data storage method according to claim 1, characterized in that, The steps for updating the connection relationships of each node based on the node cluster include: Select node clusters sequentially, traverse the data nodes in the node clusters, and read the connection relationships of the data nodes; Query the name of another data node in the connection relationship to determine whether it belongs to the same node cluster; When the node name of the other data node does not belong to the same node cluster, query the node cluster name of the node cluster to which the other data node belongs; Replace the node name of the other data node in the connection relationship with the node cluster name; Generate upper-level vectors that are connected to the node cluster based on the connection vector.

6. The hierarchical and classified data storage method according to claim 5, characterized in that, The step of generating the upper-level vector connected to the node cluster based on the connection vector includes: Read the feature matrix of the node cluster; Read the word vector matrix corresponding to the data node, traverse the feature matrix using the word vector matrix, and calculate the maximum similarity. Preserve the direction of the connection vector, use the maximum similarity as the modulus, and generate a vector from the data node to the node cluster, which serves as the upper-level vector of the connection vector; The upper-level vector is nested within the connection vector.

7. A hierarchical and classified data storage system, characterized in that, The system includes: The word vector conversion module is used to obtain unstructured comment information and convert it into a word vector matrix; each word vector in the word vector matrix corresponds to a word. The connection relationship construction module is used to convert comment information into data nodes, calculate the similarity between any two comment information based on the word vector matrix, and determine the connection relationship between data nodes based on the similarity. The connection relationship includes the names of two interconnected data nodes and a connection vector. The magnitude of the connection vector is the similarity, and the direction of the connection vector is from the data node of the comment information at the previous time step to the data node of the comment information at the next time step. The data node clustering module is used to cluster data nodes based on connection vectors, create node clusters, and construct the feature matrix of the node cluster based on the word vector matrix of each node in the node cluster. The connection relationship update module is used to update the connection relationships of each node based on the node cluster; The similarity S between the two word vector matrices is as follows: In the formula, i represents the starting column number of the word vector matrix with fewer columns in the traversal of the word vector matrix with more columns, j represents the column number in the word vector matrix with fewer columns, and M is the total number of columns in the word vector matrix with fewer columns. It represents the total number of columns in a word vector matrix with a relatively large number of columns; The word vector matrix with a smaller number of columns is the first one. column vectors, The first word in a word vector matrix with a large number of columns Column vectors; for and Vector distance; Select a function for the minimum value; The content of updating the connection relationships of each node based on the node cluster includes: Each node cluster is analyzed separately. The data nodes in the node cluster are traversed to find which data node its other end points to and which node cluster the pointed-to data node belongs to. The node name of the pointed-to data node in the connection relationship is replaced with the node cluster name, and the connection vector is updated synchronously. The process of synchronously updating the connection vector involves introducing a new vector pointing to the node cluster as the new connection vector. The new connection vector is contained within the original connection vector, and the direction of the new connection vector is the same as that of the original connection vector. The modulus is the maximum similarity between the word vector matrix and the feature matrix.

8. The hierarchical and classified data storage system according to claim 7, characterized in that, The word vector conversion module includes: The word localization unit is used to acquire unstructured comment information, perform part-of-speech analysis on the comment information, and locate words in the comment information; The word conversion unit is used to convert the words into word vectors according to the Word2Vec word vector model; The word vector sorting unit is used to obtain the order of words in the comment information, and arrange the word vectors according to the order of words in the comment information to obtain the word vector matrix.

9. The hierarchical and classified data storage system according to claim 8, characterized in that, The connection relationship construction module includes: The distance calculation unit is used to read the word vector matrix of comment information and calculate the vector distance between any two word vectors in the word vector matrix; The name generation unit is used to calculate the average vector distance between each word vector and other word vectors, select word vectors whose average vector distance is less than a preset distance threshold, read the words corresponding to the word vectors, and use them as node names; The node creation unit is used to calculate the data volume of the word vector matrix, determine the node size based on the data volume, create data nodes based on the node size, and insert the node name into the data node; The vector generation unit is used to calculate the similarity between any two comment messages based on the word vector matrix, determine the direction based on the time sequence of the two comment messages, and create a connection vector with a magnitude equal to the similarity based on the direction; the direction is from the previous time to the next time. The connection execution unit is used to connect data nodes corresponding to two comment information based on the connection vector; The calculation process for the vector distance is as follows: In the formula, For vectors sum vector vector distance, For vectors The One parameter, For vectors The One parameter; Let A and B be the dimensions of vectors A and B.

10. The hierarchical and classified data storage system according to claim 8, characterized in that, The data node clustering module includes: A clustering execution unit is used to perform spectral clustering on data nodes based on the connection vector to obtain node clusters; The name statistics unit is used to determine the name of the node cluster based on the names of the data nodes in the node cluster. The feature extraction unit is used to read the word vector matrix of each data node in the node cluster, and connect the word vector matrices of each data node to form the feature matrix of the node cluster.

Images