Data storage method and device

Abstract

The invention provides a data storage method and device, and the method comprises the steps: carrying out the slicing of a to-be-processed file based on a data slicing rule, and obtaining the slicing data; converting the slice data into a slice data vector, and determining index hierarchy information of the slice data vector based on slice description information of the slice data; storing the slice data vector to a vector database, and determining a data feature corresponding to the vector database; and determining an index type according to the data features, constructing a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information, and performing distributed storage on the hierarchical index and the vector database. In combination with distributed storage, a vector database and hierarchical indexes, an efficient retrieval method is provided. By means of distributed storage and a vector database, data can be quickly positioned and retrieved, and retrieval efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a data storage method and device. Background Art

[0002] With the advent of the big data era, the demand for data processing is growing. As the types of business in different fields become increasingly diverse, the efficiency of accessing massive data is particularly important. Reasonable data storage methods often improve the efficiency of data retrieval and query.

[0003] In the prior art, document retrieval mainly relies on keyword search. When the text library reaches tens of thousands or even tens of millions of documents, keyword search will no longer be applicable. On the one hand, the user may not know what the specific keywords to be searched are, and on the other hand, the user may not know which file to search in. As the amount of text data increases, the search time will increase exponentially, seriously affecting the search efficiency. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a data storage method. This specification also relates to a data storage device, a computing device, a computer-readable storage medium and a computer program product to solve the above problems existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data storage method is provided, including: Slice the file to be processed based on the data slicing rule to obtain slice data; Convert the slice data into a slice data vector, and determine index level information of the slice data vector based on slice description information of the slice data; Storing the slice data vector in a vector database, and determining data features corresponding to the vector database; An index type is determined according to the data feature, and a hierarchical index corresponding to the vector database is constructed based on the index type and the index hierarchical information, and the hierarchical index and the vector database are distributedly stored.

[0006] Optionally, the slicing of the file to be processed based on the data slicing rule to obtain slice data includes: Determine symbol rules, paragraph rules, format rules and title rules based on data slicing rules; The file to be processed is sliced ​​based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain the slice data.

[0007] Optionally, the slicing the to-be-processed file based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain the slicing data includes: Slicing the file to be processed based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain initial slicing data; A data identifier is assigned to the initial slice data based on an identifier assignment rule, and the initial slice data and the data identifier are used as the slice data.

[0008] Optionally, converting the slice data into a slice data vector comprises: Determining an analytical algorithm and a matrix size for the slice data; The slice data is parsed based on the parsing algorithm and the matrix size to obtain the slice data vector.

[0009] Optionally, the determining the index level information of the slice data vector based on the slice description information of the slice data includes: Determine the title level information of the slice data based on the file to be processed, and use the title level information as the slice description information; A title level corresponding to the slice data vector is determined based on the slice description information, and index level information of the slice data vector is generated based on the title level.

[0010] Optionally, constructing a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information includes: Determining primary index information and secondary index information in the index hierarchical information; Constructing a primary index matching the index type for the primary index information, and constructing a secondary index matching the index type for the secondary index information; The primary index and the secondary index are used as hierarchical indexes corresponding to the vector database.

[0011] Optionally, after the distributed storage of the hierarchical index and the vector database, the method further includes: Receiving a data query request, and parsing the data query request to obtain problem information; Convert the question information into a question vector, and determine target index information associated with the question vector in the hierarchical index of distributed storage; Answer information corresponding to the question information is determined in the vector database stored in a distributed manner based on the target index information.

[0012] Optionally, determining the target index information associated with the question vector in the hierarchical index of distributed storage includes: Determining a parent node index associated with the question vector in the hierarchical index of distributed storage; A child node index is determined based on the parent node index and the question vector, and the parent node index and the child node index are used as the target index information.

[0013] Optionally, after the distributed storage of the hierarchical index and the vector database, the method further includes: In the case where the to-be-processed file is updated, determining update data of the to-be-processed file; The hierarchical index of the distributed storage is updated based on the updated data.

[0014] According to a second aspect of an embodiment of this specification, there is provided a data storage device, including: A slicing module is configured to perform slicing processing on the file to be processed based on the data slicing rule to obtain slicing data; a determination module configured to convert the slice data into a slice data vector, and determine index level information of the slice data vector based on slice description information of the slice data; A storage module, configured to store the slice data vector into a vector database and determine data features corresponding to the vector database; A construction module is configured to determine an index type according to the data characteristics, and to construct a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information, and to perform distributed storage on the hierarchical index and the vector database.

[0015] According to a third aspect of an embodiment of this specification, a computing device is provided, comprising a memory, a processor, and a computer program or instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the data storage method when executing the computer program or instructions.

[0016] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps of the data storage method are implemented.

[0017] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned data storage method when executed by a processor.

[0018] The data storage method provided in this specification slices the processed file based on the data slicing rule to obtain slice data. The slice data is converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data. The slice data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index type and the index level information, and the hierarchical index and the vector database are distributedly stored.

[0019] An embodiment of this specification provides an efficient retrieval method by combining distributed storage, vector database and hierarchical index. By using distributed storage and vector database, data can be quickly located and retrieved, improving retrieval efficiency. By constructing hierarchical index, massive data is sliced ​​and processed, reducing the retrieval complexity in the subsequent retrieval process. It can be applied to a variety of scenarios such as big data processing, information retrieval and data mining, and has a wide range of applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of a data storage method provided by an embodiment of this specification; Figure 2 is a flow chart of a data storage method provided by an embodiment of this specification; Figure 3 It is a schematic diagram of a hierarchical index construction method of a data storage method provided in an embodiment of this specification; Figure 4 This is a distributed storage schematic diagram of a data storage method provided in an embodiment of this specification; Figure 5 It is a processing flow chart of a data storage method applied to book data storage provided by an embodiment of this specification; Figure 6 It is a structural schematic diagram of a data storage device provided by an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0021] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0022] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0024] First, the terms involved in one or more embodiments of this specification are explained.

[0025] HDFS (Hadoop Distributed File System): A distributed file system mainly used to store large-scale data. It has high fault tolerance, high throughput and scalability, and is suitable for processing large data sets. HDFS stores data in blocks on multiple nodes and ensures data reliability and availability through a replication mechanism.

[0026] Hierarchical index: A multi-level index structure that organizes and indexes data in a hierarchical manner, which can significantly improve retrieval efficiency. Common hierarchical indexes include B-tree, R-tree, KD-tree, etc.

[0027] B-tree: A self-balancing tree data structure widely used in file systems and database indexes. Its characteristics are that all leaf nodes are on the same level, and each node can have multiple child nodes.

[0028] R-tree: A data structure used for multidimensional spatial data indexing, mainly used to process geographic information and other multidimensional data.

[0029] KD tree (K-Dimensional Tree): A binary tree data structure used for multidimensional space data indexing. It is mainly used to solve the nearest neighbor search problem in multidimensional space.

[0030] Vector database: used to process high-dimensional vector data, such as feature vectors of text, images, and audio data. Vector database supports efficient similarity search, and commonly used techniques include Approximate Nearest Neighbor (ANN) search.

[0031] LSH (Locality Sensitive Hashing): Locality sensitive hashing is a technique for approximate nearest neighbor search, especially suitable for large-scale data sets in high-dimensional space. The main idea is to design a special family of hash functions so that the probability of similar data points being mapped to the same bucket is much higher than the probability of dissimilar data points being mapped to the same bucket.

[0032] Inverted index: A data structure widely used in information retrieval and search engines. It can quickly find a list of documents containing specific keywords. The traditional forward index stores the list of words contained in the document in units of documents, while the inverted index stores the list of all documents containing the word in units of words.

[0033] PQ (Product Quantization): Product Quantization is an efficient quantization technique for compressing and approximately retrieving high-dimensional vectors. It decomposes a high-dimensional vector into multiple low-dimensional sub-vectors and then applies quantization to each sub-vector independently.

[0034] Approximate Nearest Neighbor (ANN) search: An algorithm for quickly finding the closest point to a given query data point in a large dataset.

[0035] milvus: An open source vector database management system, mainly used for efficient management and retrieval of large-scale vector data. It supports multiple vector similarity search algorithms and can be integrated into the existing big data ecosystem.

[0036] qdrant: An open source vector database designed specifically for vector data, providing high-performance vector storage and search capabilities.

[0037] Snowflake Algorithm: A distributed ID generation algorithm that can generate incremental IDs.

[0038] TF-IDF (Term Frequency-Inverse Document Frequency): is a statistical method used to evaluate the importance of a word to a document in a document set.

[0039] BM25: An information retrieval model that computes a relevance score between a document and a query.

[0040] BGE (Bi-Encoder with General Embeddings): A deep learning-based method for generating dense vector representations of documents and queries. Methods that combine BM25 and BGE usually use BM25 to retrieve the most relevant subset of documents first, and then use BGE to re-rank these documents to improve retrieval quality.

[0041] Figure 1 A schematic diagram of a data storage method provided according to an embodiment of the present specification is shown. Current document retrieval mainly relies on keyword query, but when the text library reaches tens of thousands or even tens of millions, keyword retrieval will no longer be applicable. On the one hand, the user may not know what the specific keyword to be retrieved is, and on the other hand, the user may not know in which file to query the question. After the amount of text data increases, the retrieval time will increase exponentially, and the retrieval efficiency will be significantly reduced. In the data storage method provided in this embodiment, a large number of data files can be stored, and data retrieval can be performed after storage. In specific implementation, the to-be-processed files to be stored are determined, and the to-be-processed files are sliced ​​based on the data slicing rules to obtain slice data. The slice data is converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data. The slice data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index type and the index level information, and the hierarchical index and the vector database are distributedly stored.

[0042] Combining distributed storage, vector database and hierarchical index, an efficient retrieval method is provided. Using distributed storage and vector database, data can be quickly located and retrieved, improving retrieval efficiency. By constructing hierarchical index, massive data is sliced ​​and processed, reducing the retrieval complexity in the subsequent retrieval process. It can be applied to a variety of scenarios such as big data processing, information retrieval and data mining, and has a wide range of applicability.

[0043] In this specification, a data storage method is provided. This specification also relates to a data storage device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0044] Figure 2 A flowchart of a data storage method provided according to an embodiment of the present specification is shown, which specifically includes the following steps: Step 202: Slice the file to be processed based on the data slicing rule to obtain slice data.

[0045] Specifically, the data slicing rule is used to segment the data contained in the file to be processed, which is essentially to segment the data in the file to be processed. The sliced ​​data is the data segment obtained after the file to be processed is sliced. The data slicing rule determines the amount of data contained in the sliced ​​data or the number of sliced ​​data. Slicing the file to be processed can obtain at least two sliced ​​data. The formats of the files to be processed include but are not limited to .pdf, .txt, .doc, .docx and other formats.

[0046] Based on this, determine the file to be processed and the data slicing rule corresponding to the file to be processed. Determine the data contained in the file to be processed, and slice the file to be processed based on the data slicing rule for the data contained in the file to be processed to obtain at least two slices of data.

[0047] Furthermore, considering that a large amount of data is stored in the file to be processed, when slicing the file to be processed, appropriate data slicing rules can be selected according to actual needs. The specific implementation is as follows: Determine symbol rules, paragraph rules, format rules and title rules based on data slicing rules; slice the file to be processed based on the symbol rules, the paragraph rules, the format rules or the title rules to obtain the slice data.

[0048] Specifically, the symbol rules correspond to punctuation marks or special markers contained in the file to be processed; the paragraph rules correspond to the paragraphs of data in the file to be processed; the format rules correspond to the format of the text in the file to be processed, such as first line indentation, bold font and font size; the title rules correspond to the content title in the file to be processed.

[0049] Based on this, the symbol rule, paragraph rule, format rule and title rule are determined based on the data slicing rule. The attribute information of the data in the to-be-processed file is determined, and a target rule is determined in the symbol rule, paragraph rule, format rule and title rule according to the attribute information, and the to-be-processed file is sliced ​​based on the target rule to obtain at least two sliced ​​data. The data slicing rule can also be a word count rule, and a word count threshold is set, and the to-be-processed file is sliced ​​according to the word count threshold.

[0050] For example, in the enterprise knowledge storage scenario, enterprise knowledge is generally stored in the form of files. Enterprise knowledge bases often contain large-scale files. Retrieval of large-scale files is inefficient. Therefore, the storage method of large-scale files can be changed. After determining the files to be processed in various formats such as .pdf, .txt, .doc, .docx, etc., the files to be processed can be sliced. Data slicing rules include but are not limited to symbol rules, paragraph rules, format rules, and title rules. Data slicing rules can be selected according to the storage method of data in the files to be processed. When the target rule is the word count rule, 500 words are determined as a slice, and the slice is cut to the first punctuation mark after 500 words.

[0051] In summary, the file to be processed is sliced ​​based on symbol rules, paragraph rules, format rules or title rules to obtain slice data, so that data slicing rules can be flexibly determined to improve the availability of slice data.

[0052] Furthermore, considering that after slicing the file to be processed, multiple slice data can be obtained, in order to facilitate the storage and retrieval of the slice data, a data identifier can be assigned to the slice data, which is specifically implemented as follows: The file to be processed is sliced ​​based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain initial slice data; a data identifier is assigned to the initial slice data based on an identifier assignment rule, and the initial slice data and the data identifier are used as the slice data.

[0053] Specifically, the identifier allocation rule refers to a strategy for allocating data identifiers to slice data, and is used to allocate different data identifiers to each slice data. The identifier allocation rule may correspond to an identifier auto-increment allocation.

[0054] Based on this, the file to be processed is sliced ​​based on symbol rules, paragraph rules, format rules or title rules to obtain initial slice data. Data identifiers are assigned to the initial slice data based on the identifier assignment rule, and the initial slice data and the data identifier are used as slice data. For multiple initial slice data obtained by slicing the file to be processed, data identifiers are assigned one by one in an identifier self-incrementing manner, and the initial slice data and the corresponding data identifiers form the slice data.

[0055] Following the above example, after the enterprise knowledge document is divided into 100 initial slice data, data identifiers are assigned to the 100 initial slice data one by one starting from data identifier 1 in an auto-incremental manner. The slice data is composed of the initial slice data and the data identifier. The data on the slice is uniformly converted into json format.

[0056] In summary, a data identifier is assigned to the initial slice data based on the identifier assignment rule, and the initial slice data and the data identifier are used as slice data to facilitate subsequent storage and retrieval of the slice data.

[0057] Step 204: Convert the slice data into a slice data vector, and determine index level information of the slice data vector based on the slice description information of the slice data.

[0058] Specifically, after slicing the file to be processed based on the data slicing rule and obtaining the slice data, the slice data can be converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data, wherein the slice data vector is a vector matrix of the slice data, which is slice data in vector expression. The slice description information can be the title level or chapter level to which the slice data belongs in the file to be processed. The index level information corresponds to the slice description information, and the index level information can be a title level index corresponding to the title level, or a chapter level index corresponding to the chapter level.

[0059] Based on this, after slicing the file to be processed based on the data slicing rule and obtaining the slice data, the slice data is converted into a slice data vector based on the data parsing algorithm. The slice description information of the slice data is determined based on the correspondence between the slice data and the file to be processed. The slice description information of the slice data is converted into the index level information of the slice data vector. The index level information and the slice description information are title level information or chapter level information of different dimensions. In the case where the slice description information is the title level of the slice data in the file to be processed, the index level information is the index level corresponding to the title level. For example, if the title level is a first-level title, the index level is a first-level index.

[0060] Furthermore, considering that directly storing slice data requires a large storage space and is not conducive to subsequent data retrieval, the slice data can be converted into a vector expression for subsequent storage, which is specifically implemented as follows: Determine an analytical algorithm and a matrix size for the slice data; and parse the slice data based on the analytical algorithm and the matrix size to obtain the slice data vector.

[0061] Specifically, the parsing algorithm may be a BM25+BGE parsing algorithm or a TF-IDF algorithm. The parsing algorithm is used to convert the slice data into a vector expression. The matrix size refers to the size of the matrix, that is, the number of rows and columns of the matrix.

[0062] Based on this, an analytical algorithm and a matrix size are determined for the slice data, and the slice data is analyzed based on the analytical algorithm and the matrix size, and the slice data is converted into a slice data vector. The slice data is represented in the form of a vector matrix.

[0063] Using the above example, for the slice data in json format, the BM25+BGE parsing method is used to parse each slice data into a matrix of size = [512, 1]. Other parsing methods such as TF-IDF can also be used for parsing, and the corresponding parsing is a vector matrix of the corresponding size.

[0064] In summary, the slice data is parsed based on the parsing algorithm and the matrix size, and the slice data is parsed into slice data vectors to facilitate subsequent data storage.

[0065] Furthermore, considering that the data in the to-be-processed file has a certain distribution structure, the distribution structure of the data in the to-be-processed file can be referred to when generating the index level information, which is specifically implemented as follows: Determine the title level information of the slice data based on the file to be processed, and use the title level information as the slice description information; determine the title level corresponding to the slice data vector based on the slice description information, and generate index level information of the slice data vector based on the title level.

[0066] Specifically, the title level information refers to the title level to which the slice data belongs in the to-be-processed file. The title level refers to the title level corresponding to the slice data vector.

[0067] Based on this, the title level information of the slice data is determined based on the distribution structure of the data in the to-be-processed file, and the title level information is used as the slice description information. The title level corresponding to the slice data vector is determined based on the slice description information, and the index level information of the slice data vector is generated based on the title level so that the index level information corresponds to the distribution structure of the slice data in the to-be-processed file.

[0068] Using the above example, the enterprise knowledge document contains multiple title levels such as the first-level title, the second-level title, and the third-level title. After the enterprise knowledge document is divided into multiple slice data, the title level to which the slice data belongs in the enterprise knowledge document is the title level information of the slice data. This title level information corresponds to the title level of the slice data vector. The title level can be directly used as the index level of the slice data vector to generate index level information.

[0069] In summary, the title level corresponding to the slice data vector is determined based on the slice description information, and the index level information of the slice data vector is generated based on the title level, so that the index level information is associated with the title level corresponding to the slice data vector, so as to facilitate rapid positioning and data retrieval in the subsequent retrieval stage.

[0070] Step 206: Store the slice data vector into a vector database, and determine data features corresponding to the vector database.

[0071] Specifically, after the slice data is converted into a slice data vector and the index level information of the slice data vector is determined based on the slice description information of the slice data, the slice data vector can be stored in a vector database, and the data features corresponding to the vector database can be determined, wherein the vector database is used to process high-dimensional vector data and support efficient similarity search. It is used to store slice data vectors. Data features refer to the data dimensions, data distribution, data query mode, and data access frequency of the slice data vectors stored in the vector database. After the slice data vectors are stored in the vector database, a vector space in the form of a two-dimensional array is formed. The dimension of the vector space can increase with the complexity of the data.

[0072] Based on this, after converting the slice data into slice data vectors and determining the index level information of the slice data vectors based on the slice description information of the slice data, the slice data vectors are stored in the vector database to form a vector space in the form of a two-dimensional array, and the data dimension data distribution, data query mode and data access frequency of the data in the vector space are used as the data features corresponding to the vector database.

[0073] Step 208: Determine an index type according to the data feature, and construct a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information, and perform distributed storage on the hierarchical index and the vector database.

[0074] Specifically, after the slice data vectors are stored in the vector database and the data features corresponding to the vector database are determined, the index type can be determined according to the data features, and the hierarchical index corresponding to the vector database can be constructed based on the index type and index hierarchical information, and the hierarchical index and the vector database can be distributedly stored, wherein the index type refers to index technology, including but not limited to KD tree, LSH (local sensitive hashing), inverted index, PQ (product quantization), etc. Hierarchical index refers to a hierarchical index, which includes at least two index levels. Distributed storage can rely on a distributed file system to store the hierarchical index and the vector database in a distributed file system.

[0075] Based on this, after the sliced ​​data vectors are stored in the vector database and the data features corresponding to the vector database are determined, the index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index hierarchical information according to the index type. The hierarchical index and the vector database are distributedly stored and stored in a distributed file system.

[0076] In practical applications, when constructing hierarchical indexes corresponding to vector databases, the data in the vector databases can be hierarchically divided based on the distributed system. Figure 3 As shown, the original file is the data file to be processed. The slice data is obtained by slicing and parsing the data file to be processed, and the slice data is stored in the vector database. The slice data stored in the vector database is hierarchically divided, and the primary index belonging to the parent node vector is determined according to the title level of the slice data. Further, based on the primary index, the secondary index belonging to the child node vector is determined to achieve parent node segmentation and child node segmentation of the slice data stored in the vector database.

[0077] After constructing the hierarchical index and vector database, the hierarchical index and vector database can be distributed and stored to construct a distributed file system. Figure 4 As shown in the figure, the vector level index is the level index. After writing the vector level index, block operation is performed on each vector level index to read the node name of the vector level index and the id of the node name. Block operation is performed on the node name, and the vector level index is copied at the same time to build a vector level index copy.

[0078] Furthermore, when constructing the hierarchical index corresponding to the vector database, it can be determined layer by layer based on the index hierarchical information, and the specific implementation is as follows: Determine the primary index information and the secondary index information in the index hierarchy information; construct a primary index matching the index type for the primary index information, and construct a secondary index matching the index type for the secondary index information; use the primary index and the secondary index as hierarchical indexes corresponding to the vector database.

[0079] Specifically, the first-level index information refers to the index information of the field dimension, and the second-level index information refers to the index information of the data block dimension. The first-level index is the index of the parent node dimension, and the second-level index is the index of the child node dimension. The second-level index is the index level under the first-level index.

[0080] Based on this, the primary index information and the secondary index information are determined in the index hierarchy information, a primary index matching the index type is constructed for the primary index information, and a secondary index matching the index type is constructed for the secondary index information, and the secondary index is the index hierarchy under the primary index. The primary index and the secondary index are used as the hierarchical index corresponding to the vector database.

[0081] Following the above example, after determining the vector space in the form of a two-dimensional array corresponding to the vector database, select the inverted index type in combination with the vector space. Based on the index level information, the title level or chapter level can be determined. The title level or chapter level is used as the primary index, and the data block under the title level or chapter level is used as the secondary index. The primary index is a coarse-grained index, corresponding to the first dimension in the vector space, that is, the field dimension, and the secondary index is a fine-grained index, corresponding to the second dimension in the vector space, that is, the data block dimension.

[0082] In summary, the primary index and the secondary index are used as hierarchical indexes corresponding to the vector database. By constructing a hierarchical index, data retrieval and positioning can be quickly performed in subsequent retrieval nodes.

[0083] Furthermore, after the vector database and hierarchical index are distributedly stored, a retrieval service can be provided based on the hierarchical index and the vector database, which is specifically implemented as follows: Receive a data query request, and parse the data query request to obtain question information; convert the question information into a question vector, and determine the target index information associated with the question vector in the hierarchical index of distributed storage; determine the answer information corresponding to the question information in the vector database of distributed storage based on the target index information.

[0084] Specifically, the data query request may be a search request submitted by a search user, the data query request carries question information, the question information is a question sentence, and the search user hopes to obtain relatively accurate answer information corresponding to the question information. The question vector is a vector expression of the question information. The target index information may be determined by performing similarity matching between the hierarchical index and the question vector.

[0085] Based on this, a data query request is received and parsed to obtain question information. The question information is generalized and converted into a question vector. The target index information associated with the question vector is determined by similarity matching in the hierarchical index of distributed storage. The answer information corresponding to the question information is determined in the vector database of distributed storage based on the target index information.

[0086] Using the above example, after the search user determines the question "What is corporate culture", a data query request is submitted to the distributed system based on this question. The question information "What is corporate culture" is obtained by parsing the data query request. The question information is generalized and converted into a question vector. Retrieval is performed based on the question vector, and the parent node information "Organization and Policy" with a high similarity to the question vector is determined by similarity matching. Then, based on the parent node information and the question vector, similarity matching is further performed with the hierarchical index to determine the child node information with a high similarity to the question vector, and then the answer information corresponding to the question information is retrieved in the vector database based on the child node information.

[0087] In summary, the answer information corresponding to the question information is determined based on the hierarchical index and vector database of distributed storage, thereby improving the retrieval efficiency and the accuracy of the answer information.

[0088] Furthermore, when searching for question information, you can perform a level-by-level search based on the hierarchical index, which is specifically implemented as follows: Determine a parent node index associated with the question vector in the hierarchical index of distributed storage; determine a child node index based on the parent node index and the question vector, and use the parent node index and the child node index as the target index information.

[0089] Specifically, the parent node index refers to the parent node information with a higher similarity to the question vector in the hierarchical index; the child node index refers to the child node information with a higher similarity to the question vector under the parent node information.

[0090] Based on this, the parent node index with higher similarity to the question vector is determined by similarity matching in the hierarchical index of distributed storage. Under the parent node index, the child node index with higher similarity to the question vector is determined by similarity matching, and the parent node index and the child node index are used as target index information.

[0091] In summary, combining hierarchical index and vector database to retrieve answer information corresponding to question information improves the retrieval efficiency of answer information.

[0092] Furthermore, considering that the data in the files to be processed are updated in real time, the hierarchical index of distributed storage also needs to be updated synchronously. The specific implementation is as follows: In the case where the to-be-processed file is updated, update data of the to-be-processed file is determined; and the hierarchical index of the distributed storage is updated based on the update data.

[0093] Specifically, the update of the to-be-processed file may be the addition, deletion and modification of data in the to-be-processed file. The updated data may be newly added data, modified data or deleted data.

[0094] Based on this, when data in the file to be processed is deleted and the file to be processed is updated, the deleted data is used as the updated data; when data in the file to be processed is modified and the file to be processed is updated, the modified data is used as the updated data; when new data is added to the file to be processed and the file to be processed is updated, the new data is used as the updated data; the hierarchical index of the distributed storage is updated based on the updated data.

[0095] In actual applications, the performance of hierarchical indexes can be monitored in real time and adjusted according to actual conditions, for example, the hierarchical structure, parameter settings, or storage format of the index can be adjusted.

[0096] In summary, when the to-be-processed file is updated, the hierarchical index of the distributed storage is updated based on the updated data, thereby ensuring the accuracy of the hierarchical index.

[0097] In one embodiment of the present specification, a file to be processed is sliced ​​based on a data slicing rule to obtain slice data. The slice data is converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data. The slice data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and a hierarchical index corresponding to the vector database is constructed based on the index type and the index level information, and the hierarchical index and the vector database are distributedly stored. Combining distributed storage, vector database and hierarchical index, an efficient retrieval method is provided. By using distributed storage and vector database, data can be quickly located and retrieved, thereby improving retrieval efficiency. By constructing a hierarchical index, massive data is sliced ​​to reduce the retrieval complexity in the subsequent retrieval process. It can be applied to a variety of scenarios such as big data processing, information retrieval and data mining, and has a wide range of applicability.

[0098] The following combination Figure 5 Taking the application of the data storage method provided in this specification in book data storage as an example, the data storage method is further described. Figure 5 A processing flow chart of a data storage method for book data storage provided in an embodiment of the present specification is shown, which specifically includes the following steps: Step 502: Determine data files in multiple file formats.

[0099] The data storage method provided in this embodiment can be applied to the scenario of book data storage. The book data is stored in the form of a file, and the book data may include a book overview and book content. The file of the book data may be in the format of .pdf, .txt, .doc, .docx, etc.

[0100] Step 504: Determine data slicing rules, and perform slicing processing on the data file based on the data slicing rules to obtain sliced ​​data.

[0101] Data slicing rules can be punctuation, format, paragraph, etc. Slice the data file according to punctuation, format or paragraph and convert it into slice data in json format. Each slice data can use the snowflake algorithm to form a unique id and paragraph content.

[0102] Step 506: parse the slice data to obtain slice data vectors, and determine the level information of the slice data vectors in the data file.

[0103] The sliced ​​data file is parsed using a preset parsing method (BM25+BGE parsing method), and each slice data in json format is parsed into a matrix of size=[512, 1] to obtain a slice data vector. The parsing method can also be TF-IDF, etc., and the slice data is parsed into a vector matrix of corresponding size accordingly. This embodiment does not impose any limitation on the selection of the parsing method.

[0104] Step 508: construct a vector database based on the slice data vectors.

[0105] After parsing the slice data into slice data vectors, feature analysis is performed on the slice data vectors to determine data features such as data dimension, data distribution, query mode, and access frequency corresponding to the slice data vectors. After preprocessing the slice data vectors such as dimensionality reduction, standardization, or normalization, a vector database is constructed to form a vector space. The vector database can be a milvus vector database or a qdrant vector database.

[0106] Step 510: Determine the hierarchical index corresponding to the vector database based on the hierarchical information.

[0107] Hierarchical information can be the title level or chapter level of the data file. The first-level title is the first level, the second-level title is the second level, and so on. The data is hierarchically divided based on the hierarchical information to build a hierarchical index. The hierarchical index can divide the data into multiple levels, each level contains a certain number of data blocks, which is convenient for fast positioning and retrieval of data. When building a hierarchical index, you can choose a suitable indexing method according to actual needs, such as: inverted index, KD tree, LSH (local sensitive hashing), PQ (product quantization), etc.

[0108] In practical applications, coarse-grained indexes and fine-grained indexes can be established for vector databases respectively. Coarse-grained indexes can be established based on the first dimension of the vector space for partitioning. The first dimension can be the field dimension stored in the vector database. For example, given the fields passed into the milvus vector database, a coarse-grained index is established. Fine-grained indexes are established. On the basis of the coarse-grained indexes, fine-grained indexes are established for each partition. These indexes can be established based on the second dimension of the vector space, which can quickly locate specific data sets. The second dimension can be the number of segmented texts. For example, parse each piece of data passed into the milvus vector database.

[0109] Step 512: Distributed storage of the vector database and hierarchical index.

[0110] After determining the vector database and hierarchical index, you can build a distributed file system. Taking advantage of HDFS's high reliability, high throughput, and high scalability, massive data can be distributed and stored on multiple nodes to improve data storage and access efficiency. For the storage of hierarchical indexes, the fields of coarse-grained indexes and each piece of data in fine-grained indexes can be stored in HDFS. In addition, hierarchical indexes can be updated regularly to reflect data changes, monitor index performance, and make adjustments based on actual conditions, such as adjusting the hierarchical structure, parameter settings, or storage format of the hierarchical index.

[0111] Step 514: Receive question information, and search for answer information corresponding to the question information based on the question information, the hierarchical index and the vector database.

[0112] After the distributed file system is built, the retrieval task can be performed. The user provides the question information to be retrieved. The question information is generalized and vectorized to obtain the question vector. The parent node and child node data blocks corresponding to the question vector are determined based on the hierarchical index in the distributed file system. The question vector is matched with the vector data in the child node data block to obtain the retrieval result corresponding to the question information.

[0113] In summary, a distributed file system, hierarchical index and vector database are combined to propose an efficient retrieval method. By taking advantage of HDFS and vector database, data can be quickly located and retrieved, thereby improving retrieval efficiency. By constructing a hierarchical index, massive data can be divided into multiple levels, thereby reducing the complexity of the retrieval process. The distributed storage mechanism based on HDFS can easily cope with the continuous growth of data volume and has good scalability. The data storage method provided in this embodiment can be applied to a variety of scenarios such as big data processing, information retrieval, and data mining, and has high versatility.

[0114] Corresponding to the above method embodiment, this specification also provides a data storage device embodiment, Figure 6 FIG. 1 shows a schematic diagram of a data storage device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises: The slicing module 602 is configured to perform slicing processing on the to-be-processed file based on the data slicing rule to obtain slicing data; A determination module 604 is configured to convert the slice data into a slice data vector, and determine index level information of the slice data vector based on slice description information of the slice data; The storage module 606 is configured to store the slice data vector into a vector database and determine data features corresponding to the vector database; The construction module 608 is configured to determine the index type according to the data characteristics, and to construct a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information, and to perform distributed storage on the hierarchical index and the vector database.

[0115] In an optional embodiment, the slicing module 602 is further configured to: Determine symbol rules, paragraph rules, format rules and title rules based on data slicing rules; The file to be processed is sliced ​​based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain the slice data.

[0116] In an optional embodiment, the slicing module 602 is further configured to: Slicing the file to be processed based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain initial slicing data; A data identifier is assigned to the initial slice data based on an identifier assignment rule, and the initial slice data and the data identifier are used as the slice data.

[0117] In an optional embodiment, the determining module 604 is further configured to: Determining an analytical algorithm and a matrix size for the slice data; The slice data is parsed based on the parsing algorithm and the matrix size to obtain the slice data vector.

[0118] In an optional embodiment, the determining module 604 is further configured to: Determine the title level information of the slice data based on the file to be processed, and use the title level information as the slice description information; A title level corresponding to the slice data vector is determined based on the slice description information, and index level information of the slice data vector is generated based on the title level.

[0119] In an optional embodiment, the construction module 608 is further configured to: Determining primary index information and secondary index information in the index hierarchical information; Constructing a primary index matching the index type for the primary index information, and constructing a secondary index matching the index type for the secondary index information; The primary index and the secondary index are used as hierarchical indexes corresponding to the vector database.

[0120] In an optional embodiment, the construction module 608 is further configured to: Receiving a data query request, and parsing the data query request to obtain problem information; Convert the question information into a question vector, and determine target index information associated with the question vector in the hierarchical index of distributed storage; Answer information corresponding to the question information is determined in the vector database stored in a distributed manner based on the target index information.

[0121] In an optional embodiment, the construction module 608 is further configured to: Determining a parent node index associated with the question vector in the hierarchical index of distributed storage; A child node index is determined based on the parent node index and the question vector, and the parent node index and the child node index are used as the target index information.

[0122] In an optional embodiment, the construction module 608 is further configured to: In the case where the to-be-processed file is updated, determining update data of the to-be-processed file; The hierarchical index of the distributed storage is updated based on the updated data.

[0123] In one embodiment of the present specification, a file to be processed is sliced ​​based on a data slicing rule to obtain slice data. The slice data is converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data. The slice data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and a hierarchical index corresponding to the vector database is constructed based on the index type and the index level information, and the hierarchical index and the vector database are distributedly stored. Combining distributed storage, vector database and hierarchical index, an efficient retrieval method is provided. By using distributed storage and vector database, data can be quickly located and retrieved, thereby improving retrieval efficiency. By constructing a hierarchical index, massive data is sliced ​​to reduce the retrieval complexity in the subsequent retrieval process. It can be applied to a variety of scenarios such as big data processing, information retrieval and data mining, and has a wide range of applicability.

[0124] The above is a schematic scheme of a data storage device of this embodiment. It should be noted that the technical scheme of the data storage device and the technical scheme of the data storage method described above are of the same concept, and the details of the technical scheme of the data storage device that are not described in detail can be found in the description of the technical scheme of the data storage method described above.

[0125] Figure 7 The structure block diagram of a computing device 700 provided according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0126] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0127] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0128] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0129] Among them, when the processor 720 executes the computer program or instruction, the steps of the data storage method are implemented.

[0130] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data storage method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data storage method.

[0131] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program or instruction, which implements the steps of the data storage method as described above when executed by a processor.

[0132] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data storage method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data storage method.

[0133] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above data storage method when executed by a processor.

[0134] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data storage method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above data storage method.

[0135] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0136] The computer program or instruction includes a computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0137] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0138] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data storage method, characterized in that: include: Slice the file to be processed based on the data slicing rule to obtain slice data; Convert the slice data into a slice data vector, and determine index level information of the slice data vector based on slice description information of the slice data; Storing the slice data vector in a vector database, and determining data features corresponding to the vector database; An index type is determined according to the data feature, and a hierarchical index corresponding to the vector database is constructed based on the index type and the index hierarchical information, and the hierarchical index and the vector database are distributedly stored.

2. The data storage method according to claim 1, characterized in that: The slicing of the file to be processed based on the data slicing rule to obtain the slice data includes: Determine symbol rules, paragraph rules, format rules and title rules based on data slicing rules; The file to be processed is sliced ​​based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain the slice data.

3. The data storage method according to claim 2, characterized in that: The slicing process of the to-be-processed file based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain the slicing data includes: Slicing the file to be processed based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain initial slicing data; A data identifier is assigned to the initial slice data based on an identifier assignment rule, and the initial slice data and the data identifier are used as the slice data.

4. The data storage method according to claim 1, characterized in that: The step of converting the slice data into a slice data vector comprises: Determining an analytical algorithm and a matrix size for the slice data; The slice data is parsed based on the parsing algorithm and the matrix size to obtain the slice data vector.

5. The data storage method according to claim 1, characterized in that: The determining the index level information of the slice data vector based on the slice description information of the slice data includes: Determine the title level information of the slice data based on the file to be processed, and use the title level information as the slice description information; A title level corresponding to the slice data vector is determined based on the slice description information, and index level information of the slice data vector is generated based on the title level.

6. The data storage method according to claim 1, characterized in that: The constructing a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information includes: Determining primary index information and secondary index information in the index hierarchical information; Constructing a primary index matching the index type for the primary index information, and constructing a secondary index matching the index type for the secondary index information; The primary index and the secondary index are used as hierarchical indexes corresponding to the vector database.

7. The data storage method according to claim 1, characterized in that: After the hierarchical index and the vector database are distributedly stored, the method further includes: Receiving a data query request, and parsing the data query request to obtain problem information; Convert the question information into a question vector, and determine target index information associated with the question vector in the hierarchical index of distributed storage; Answer information corresponding to the question information is determined in the vector database stored in a distributed manner based on the target index information.

8. The data storage method according to claim 7, characterized in that: The determining the target index information associated with the question vector in the hierarchical index of the distributed storage includes: Determining a parent node index associated with the question vector in the hierarchical index of distributed storage; A child node index is determined based on the parent node index and the question vector, and the parent node index and the child node index are used as the target index information.

9. The data storage method according to claim 1, characterized in that: After the hierarchical index and the vector database are distributedly stored, the method further includes: In the case where the to-be-processed file is updated, determining update data of the to-be-processed file; The hierarchical index of the distributed storage is updated based on the updated data.

10. A data storage device, characterized in that: include: A slicing module is configured to perform slicing processing on the file to be processed based on the data slicing rule to obtain slicing data; a determination module configured to convert the slice data into a slice data vector, and determine index level information of the slice data vector based on slice description information of the slice data; A storage module, configured to store the slice data vector into a vector database and determine data features corresponding to the vector database; A construction module is configured to determine an index type according to the data characteristics, and to construct a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information, and to perform distributed storage on the hierarchical index and the vector database.

11. A computing device comprising a memory, a processor, and a computer program or instruction stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program or instructions, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing a computer program or instruction, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images