Data storage method and apparatus

Abstract

The present specification provides a data storage method and device, wherein the data storage method comprises: performing slicing processing on a to-be-processed file based on a data slicing rule to obtain sliced data; converting the sliced data into a sliced data vector, and determining index level information of the sliced data vector based on slicing description information of the sliced data; storing the sliced data vector into a vector database, and determining data features corresponding to the vector database; determining an index type according to the data features, and constructing a hierarchical index corresponding to the vector database based on the index type and the index level information, and performing distributed storage on the hierarchical index and the vector database. In combination with distributed storage, a vector database and a hierarchical index, an efficient retrieval method is provided. By using distributed storage and a vector database, data can be quickly located and retrieved, and retrieval efficiency is improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to data storage methods and devices. Background Technology

[0002] With the advent of the big data era, the demand for data processing is growing rapidly. As business types across different fields become increasingly diverse, the efficiency of storing and retrieving massive amounts of data becomes particularly important. A reasonable data storage method often improves the efficiency of data retrieval and querying.

[0003] In current technologies, document retrieval primarily relies on keyword searches. However, when the text database contains tens of thousands or even tens of millions of documents, keyword searches become ineffective. On one hand, users may not know the specific keywords they are searching for; on the other hand, they may not know which file to search in. As the volume of text data increases, retrieval time rises exponentially, severely impacting retrieval efficiency. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data storage method. This specification also relates to a data storage device, a computing device, a computer-readable storage medium, and a computer program product, to address the aforementioned problems existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data storage method is provided, comprising:

[0006] The file to be processed is sliced ​​based on data slicing rules to obtain sliced ​​data;

[0007] The slice data is converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data;

[0008] The sliced ​​data vectors are stored in a vector database, and the data features corresponding to the vector database are determined.

[0009] The index type is determined based on the data characteristics, and a hierarchical index corresponding to the vector database is constructed based on the index type and the index hierarchy information. The hierarchical index and the vector database are then stored in a distributed manner.

[0010] Optionally, the step of slicing the file to be processed based on data slicing rules to obtain sliced ​​data includes:

[0011] The rules for determining symbols, paragraphs, formatting, and headings are based on the data slicing rules.

[0012] slice the to-be-processed file based on the symbol rule, the paragraph rule, the format rule, or the title rule to obtain the slice data.

[0013] Optionally, the slicing the to-be-processed file based on the symbol rule, the paragraph rule, the format rule, or the title rule to obtain the slice data comprises:

[0014] slice the to-be-processed file based on the symbol rule, the paragraph rule, the format rule, or the title rule to obtain initial slice data;

[0015] assign a data identifier to the initial slice data based on an identifier assignment rule, and take the initial slice data and the data identifier as the slice data.

[0016] Optionally, the converting the slice data into a slice data vector comprises:

[0017] determining a parsing algorithm and a matrix size for the slice data;

[0018] parsing the slice data based on the parsing algorithm and the matrix size to obtain the slice data vector.

[0019] Optionally, the determining the index level information of the slice data vector based on slice description information of the slice data comprises:

[0020] determining title level information of the slice data based on the to-be-processed file, and taking the title level information as the slice description information;

[0021] determining a title level corresponding to the slice data vector based on the slice description information, and generating the index level information of the slice data vector based on the title level.

[0022] Optionally, the constructing the hierarchical index corresponding to the vector database based on the index type and the index level information comprises:

[0023] determining first-level index information and second-level index information in the index level information;

[0024] constructing a first-level index matching the index type for the first-level index information, and constructing a second-level index matching the index type for the second-level index information;

[0025] taking the first-level index and the second-level index as the hierarchical index corresponding to the vector database.

[0026] Optionally, after the distributed storage of the hierarchical index and the vector database, the method further comprises:

[0027] receive a data query request and parse the data query request to obtain question information;

[0028] convert the question information into a question vector and determine target index information associated with the question vector in a distributed stored hierarchical index;

[0029] determine answer information corresponding to the question information in the distributed stored vector database based on the target index information.

[0030] Optionally, the determining the target index information associated with the question vector in the distributed stored hierarchical index comprises:

[0031] determining a parent node index associated with the question vector in the distributed stored hierarchical index;

[0032] determining a child node index based on the parent node index and the question vector, and taking the parent node index and the child node index as the target index information.

[0033] Optionally, after the distributed storage of the hierarchical index and the vector database, the method further comprises:

[0034] in the case that the to-be-processed file is updated, determining update data of the to-be-processed file;

[0035] performing index update on the distributed stored hierarchical index based on the update data.

[0036] According to a second aspect of the embodiments of the present specification, a data storage device is provided, comprising:

[0037] a slicing module configured to perform slicing processing on a to-be-processed file based on a data slicing rule to obtain sliced data;

[0038] a determining module configured to convert the sliced data into a sliced data vector and determine index hierarchical information of the sliced data vector based on slicing description information of the sliced data;

[0039] a storage module configured to store the sliced data vector to a vector database and determine data features corresponding to the vector database;

[0040] a constructing module configured to determine an index type according to the data features and construct a hierarchical index corresponding to the vector database based on the index type and the index hierarchical information, and perform distributed storage on the hierarchical index and the vector database.

[0041] According to a third aspect of the embodiments of the present specification, a computing device is provided, comprising a memory, a processor, and a computer program or instructions stored on the memory and executable on the processor, wherein the processor implements the steps of the data storage method when executing the computer program or instructions.

[0042] According to a fourth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the data storage method.

[0043] According to a fifth aspect of the embodiments of the present specification, a computer program product is provided, comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the data storage method.

[0044] The data storage method provided by the present specification performs slicing processing on the to-be-processed file based on the data slicing rule to obtain sliced data. The sliced data is converted into a sliced data vector, and the index level information of the sliced data vector is determined based on the slicing description information of the sliced data. The sliced data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index type and the index level information, and the hierarchical index and the vector database are stored in a distributed manner.

[0045] An embodiment of the present specification combines distributed storage, a vector database, and a hierarchical index to provide an efficient retrieval method. By using distributed storage and a vector database, data can be quickly located and retrieved, improving retrieval efficiency. By constructing a hierarchical index, massive data is sliced, reducing the complexity of subsequent retrieval processes. It can be applied to various scenarios such as big data processing, information retrieval, and data mining, and has wide applicability. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a schematic diagram of a data storage method provided by an embodiment of the present specification;

[0047] Figure 2 is a flowchart of a data storage method provided by an embodiment of the present specification;

[0048] Figure 3 is a hierarchical index construction schematic diagram of a data storage method provided by an embodiment of the present specification;

[0049] Figure 4 is a distributed storage schematic diagram of a data storage method provided by an embodiment of the present specification;

[0050] Figure 5is a processing flowchart of a data storage method applied to book data storage provided by an embodiment of the present specification;

[0051] Figure 6 is a structural schematic diagram of a data storage device provided by an embodiment of the present specification;

[0052] Figure 7 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0053] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples described herein, and it is understood that the scope of the present specification is not limited to the details below.

[0054] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0055] It will be understood that, although the terms first, second, etc. can be employed in describing various information, such information should not be limited to these terms. These terms are only used to differentiate one piece of information from another. For example, without departing from the scope of one or more embodiments of the present specification, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining."

[0056] First, the noun terms related to one or more embodiments of the present specification are explained.

[0057] HDFS (Hadoop Distributed File System): a distributed file system mainly used for storing large-scale data. It has high fault tolerance, high throughput, and scalability, and is suitable for processing large data sets. HDFS stores data in blocks on multiple nodes, and through the replication mechanism, it guarantees the reliability and availability of data.

[0058] Hierarchical Index: A multi-level index structure that organizes and indexes data in a hierarchical manner, significantly improving retrieval efficiency. Common hierarchical indexes include B-tree, R-tree, KD-tree, etc.

[0059] B-tree (B-Tree): A self-balancing tree data structure widely used in file systems and database indexing. Its characteristics are that all leaf nodes are at the same level, and each node can have multiple child nodes.

[0060] R-tree (R-Tree): A data structure for indexing multi-dimensional spatial data, mainly used for processing geographic information and other multi-dimensional data.

[0061] KD-tree (K-Dimensional Tree): A binary tree data structure for indexing multi-dimensional spatial data. It is mainly used to solve the nearest neighbor search problem in multi-dimensional space.

[0062] Vector Database: A database used to handle high-dimensional vector data such as text, image and audio feature vectors. Vector databases support efficient similarity search, and common techniques include Approximate Nearest Neighbor (ANN) search.

[0063] LSH (Locality Sensitive Hashing): A technique for approximate nearest neighbor search, especially suitable for large-scale data sets in high-dimensional space. Its main idea is to design a special family of hash functions, so that the probability of similar data points being mapped to the same bucket is much higher than the probability of dissimilar data points being mapped to the same bucket.

[0064] Inverted Index: A data structure widely used in information retrieval and search engines. It can quickly find a list of documents containing a specific keyword. Traditional forward index is to store the list of words contained in the document based on the document, while inverted index is to store the list of all documents containing the word based on the word.

[0065] PQ (Product Quantization): A high-efficiency quantization technique used to compress and approximate retrieval of high-dimensional vectors. It decomposes high-dimensional vectors into multiple low-dimensional sub-vectors, and then applies quantization to each sub-vector independently.

[0066] Approximate Nearest Neighbor (ANN) Search: An algorithm that quickly finds the nearest point to a given query data point in a large-scale data set.

[0067] Milvus: An open-source vector database management system designed for efficient management and retrieval of large-scale vector data. It supports multiple vector similarity search algorithms and can be integrated into existing big data ecosystems.

[0068] Qdrant: An open-source vector database designed for vector data, providing high-performance vector storage and search capabilities.

[0069] Snowflake Algorithm: A distributed ID generation algorithm that generates incremental IDs.

[0070] TF-IDF (Term Frequency-Inverse Document Frequency): A statistical method used to evaluate the importance of a word to a document in a document set.

[0071] BM25: An information retrieval model used to calculate the relevance score between a document and a query.

[0072] BGE (Bi-Encoder with General Embeddings): A deep learning-based method for generating dense vector representations of documents and queries. Combining BM25 and BGE methods usually first uses BM25 to retrieve the most relevant document subset, and then uses BGE to reorder these documents to improve retrieval quality.

[0073] Figure 1 A schematic diagram of a data storage method according to an embodiment of the present specification is shown. Current document retrieval mainly relies on the way of keyword query, but when the text library reaches tens of thousands or even hundreds of millions, keyword retrieval will no longer be applicable. On the one hand, the user may not know the specific keyword to be retrieved, and on the other hand, the user may not know which file to query. After the amount of text data increases, the retrieval time will increase exponentially, and the retrieval efficiency will be significantly reduced. The data storage method provided in the embodiment can store a large amount of data files and perform data retrieval after storage. In specific implementation, the file to be processed that needs to be stored is determined, the file to be processed is sliced based on a data slicing rule to obtain sliced data. The sliced data is converted into a sliced data vector, and the index level information of the sliced data vector is determined based on the sliced description information of the sliced data. The sliced data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index type and the index level information. The hierarchical index and the vector database are stored in a distributed manner.

[0074] In combination with distributed storage, vector database and hierarchical index, an efficient retrieval method is provided. By using distributed storage and vector database, data can be quickly located and retrieved, improving retrieval efficiency. By constructing hierarchical index, massive data is sliced, reducing the complexity of subsequent retrieval process. It can be applied to various scenarios such as big data processing, information retrieval and data mining, and has wide applicability.

[0075] In the present specification, a data storage method is provided, and the present specification also relates to a data storage device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail in the following embodiments.

[0076] Figure 2 A flowchart of a data storage method according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0077] Step 202: Slice the to-be-processed file based on the data slicing rule to obtain slice data.

[0078] Specifically, the data slicing rule is used to divide the data contained in the to-be-processed file, which is essentially a segmentation process of the data in the to-be-processed file. The slice data is the data segment obtained after the to-be-processed file is sliced. The data slicing rule determines the amount of data contained in the slice data or the number of slice data. The to-be-processed file is sliced to obtain at least two slice data. The format of the to-be-processed file includes but is not limited to.pdf,.txt,.doc,.docx, etc.

[0079] Based on this, the to-be-processed file and the data slicing rule corresponding to the to-be-processed file are determined. The data contained in the to-be-processed file is determined, and the to-be-processed file is sliced based on the data slicing rule for the data contained in the to-be-processed file, to obtain at least two slice data.

[0080] Further, considering that a large amount of data is stored in the to-be-processed file, a suitable data slicing rule can be selected according to actual needs when the to-be-processed file is sliced, which is specifically implemented as follows:

[0081] Based on the data slicing rule, a symbol rule, a paragraph rule, a format rule and a title rule are determined; the to-be-processed file is sliced based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain the slice data.

[0082] Specifically, the symbol rule corresponds to a punctuation mark or a special mark included in the to-be-processed file; the paragraph rule corresponds to a paragraph of data in the to-be-processed file; the format rule corresponds to a format of a character in the to-be-processed file, for example, first line indentation, font bolding, and font size; and the title rule corresponds to a content title in the to-be-processed file.

[0083] Based on this, the symbol rule, the paragraph rule, the format rule, and the title rule are determined based on the data slicing rule. The attribute information of the data in the to-be-processed file is determined, a target rule is determined in the symbol rule, the paragraph rule, the format rule, and the title rule according to the attribute information, and the to-be-processed file is sliced based on the target rule to obtain at least two slice data. The data slicing rule can also be a word number rule, a word number threshold is set, and the to-be-processed file is sliced according to the word number threshold.

[0084] For example, in the enterprise knowledge storage scenario, enterprise knowledge is generally stored in the form of a file. The enterprise knowledge base often contains large-scale files. The efficiency of searching for large-scale files is low. Therefore, the storage method of large-scale files can be changed. After determining the to-be-processed file in.pdf,.txt,.doc,.docx, and other formats, the to-be-processed file can be sliced. The data slicing rule includes but is not limited to the symbol rule, the paragraph rule, the format rule, and the title rule, and the data slicing rule can be selected according to the storage method of the data in the to-be-processed file. In the case of a word number rule as a target rule, 500 words are determined as a slice, and the first punctuation mark after 500 words.

[0085] In summary, the to-be-processed file is sliced based on the symbol rule, the paragraph rule, the format rule, or the title rule to obtain slice data, so that the data slicing rule can be flexibly determined, and the usability of the slice data is improved.

[0086] Further, considering that multiple slice data can be obtained after the to-be-processed file is sliced, in order to facilitate the storage and retrieval of the slice data, the slice data can be allocated a data identifier, and the specific implementation is as follows:

[0087] The to-be-processed file is sliced based on the symbol rule, the paragraph rule, the format rule, or the title rule to obtain initial slice data; the initial slice data is allocated a data identifier based on an identifier allocation rule, and the initial slice data and the data identifier are taken as the slice data.

[0088] Specifically, the identifier allocation rule refers to a strategy for allocating a data identifier to slice data, which is used to allocate different data identifiers to each slice data. The identifier allocation rule can correspond to an identifier self-increment allocation.

[0089] Based on this, the to-be-processed file is sliced based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain initial slice data. The initial slice data is assigned a data identifier based on an identifier assignment rule, and the initial slice data and the data identifier are taken as slice data. The multiple initial slice data obtained by slicing the to-be-processed file are assigned data identifiers one by one in an identifier self-increment manner, and the slice data is composed of the initial slice data and the corresponding data identifier.

[0090] In the above example, after the enterprise knowledge document is sliced into 100 initial slice data, the 100 initial slice data are assigned data identifiers one by one in a self-increment manner starting from data identifier 1. The slice data is composed of the initial slice data and the data identifier. The slice data is uniformly converted into a json format.

[0091] In summary, the initial slice data is assigned a data identifier based on the identifier assignment rule, and the initial slice data and the data identifier are taken as slice data, so as to facilitate the storage and retrieval of the slice data in the subsequent process.

[0092] Step 204: converting the slice data into a slice data vector and determining index level information of the slice data vector based on slice description information of the slice data.

[0093] Specifically, after the to-be-processed file is sliced based on the data slicing rule to obtain the slice data, the slice data can be converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data. The slice data vector is a vector matrix of the slice data, which is a vector expression form of the slice data. The slice description information can be a title level or a chapter level to which the slice data belongs in the to-be-processed file. The index level information corresponds to the slice description information, and the index level information can be a title level index corresponding to the title level, or a chapter level index corresponding to the chapter level.

[0094] Based on this, after the to-be-processed file is sliced based on the data slicing rule to obtain the slice data, the slice data is converted into a slice data vector based on a data parsing algorithm. The slice description information of the slice data is determined based on the corresponding relationship between the slice data and the to-be-processed file. The slice description information of the slice data is converted into index level information of the slice data vector. The index level information and the slice description information are different dimensions of title level information or chapter level information. In the case where the slice description information is a title level of the slice data in the to-be-processed file, the index level information is an index level corresponding to the title level. For example, the title level is a first-level title, and the index level is a first-level index.

[0095] Further, considering that directly storing the slice data requires occupying a large storage space and is not conducive to subsequent data retrieval, the slice data can be converted into a vector expression form for subsequent storage, and the specific implementation is as follows:

[0096] An analysis algorithm and a matrix size are determined for the slice data, and the slice data is analyzed based on the analysis algorithm and the matrix size to obtain the slice data vector.

[0097] Specifically, the analysis algorithm can be a BM25+BGE analysis algorithm, and can also be a TF-IDF algorithm. The analysis algorithm is used to convert the slice data into a vector expression form. The matrix size refers to the size of the matrix, that is, the number of rows and columns of the matrix.

[0098] Based on this, the analysis algorithm and the matrix size are determined for the slice data, and the slice data is analyzed based on the analysis algorithm and the matrix size to convert the slice data into a slice data vector. The slice data is represented in the form of a vector matrix.

[0099] Using the above example, for the slice data in json format, the BM25+BGE analysis method is used to analyze each slice data into a matrix with size = [512, 1]. Other analysis methods such as TF-IDF can also be used for analysis, and the corresponding analysis is a vector matrix with a corresponding size.

[0100] As described above, the slice data is analyzed based on the analysis algorithm and the matrix size to convert the slice data into a slice data vector for subsequent data storage.

[0101] Further, considering that the data in the to-be-processed file has a certain distribution structure, when generating the index level information, the distribution structure of the data in the to-be-processed file can be referred to, and the specific implementation is as follows:

[0102] The title level information of the slice data is determined based on the to-be-processed file, and the title level information is used as the slice description information. The title level corresponding to the slice data vector is determined based on the slice description information, and the index level information of the slice data vector is generated based on the title level.

[0103] Specifically, the title level information refers to the title level to which the slice data belongs in the to-be-processed file. The title level refers to the title level corresponding to the slice data vector.

[0104] Based on this, the title level information of the slice data is determined based on the distribution structure of the data in the to-be-processed file, and the title level information is taken as the slice description information. The title level corresponding to the slice data vector is determined based on the slice description information, and the index level information of the slice data vector is generated based on the title level, so that the index level information corresponds to the distribution structure of the slice data in the to-be-processed file.

[0105] In the above example, the enterprise knowledge document contains multiple title levels such as a first-level title, a second-level title, and a third-level title. After the enterprise knowledge document is divided into multiple slice data, the title level of the slice data in the enterprise knowledge document is the title level information of the slice data. The title level information corresponds to the title level of the slice data vector. The title level can be directly taken as the index level of the slice data vector to generate the index level information.

[0106] In summary, the title level corresponding to the slice data vector is determined based on the slice description information, and the index level information of the slice data vector is generated based on the title level. The index level information is associated with the title level corresponding to the slice data vector, so as to facilitate positioning and data retrieval in the subsequent retrieval stage.

[0107] Step 206: store the slice data vector into a vector database, and determine the data features corresponding to the vector database.

[0108] Specifically, after the slice data is converted into the slice data vector and the index level information of the slice data vector is determined based on the slice description information of the slice data, the slice data vector can be stored into the vector database, and the data features corresponding to the vector database are determined. The vector database is used to process high-dimensional vector data and supports efficient similarity search. The slice data vector is stored. The data features refer to the data dimensions, data distribution, data query mode, and data access frequency of the slice data vector stored in the vector database. After the slice data vector is stored into the vector database, a two-dimensional array form of vector space is formed. The dimension of the vector space can increase with the complexity of the data.

[0109] Based on this, after the slice data is converted into the slice data vector and the index level information of the slice data vector is determined based on the slice description information of the slice data, the slice data vector is stored into the vector database to form a two-dimensional array form of vector space. The data dimensions, data distribution, data query mode, and data access frequency of the data in the vector space are taken as the data features corresponding to the vector database.

[0110] Step 208: determine the index type according to the data features, and construct the hierarchical index corresponding to the vector database based on the index type and the index level information, and perform distributed storage on the hierarchical index and the vector database.

[0111] Specifically, after storing the slice data vector into the vector database and determining the data features corresponding to the vector database, the index type can be determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index type and the index level information, and the hierarchical index and the vector database are stored in a distributed manner, wherein the index type refers to an index technology, including but not limited to KD tree, LSH (locality sensitive hashing), inverted index, PQ (product quantization) and the like. The hierarchical index refers to a hierarchical index, which includes at least two index levels. The distributed storage can rely on a distributed file system, and the hierarchical index and the vector database are stored in the distributed file system.

[0112] Based on this, after storing the slice data vector into the vector database and determining the data features corresponding to the vector database, the index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed according to the index type based on the index level information, and the hierarchical index and the vector database are stored in a distributed manner, and the hierarchical index and the vector database are stored in the distributed file system.

[0113] In practical applications, when constructing the hierarchical index corresponding to the vector database, the data in the vector database can be hierarchically divided based on a distributed system. As shown in Figure 3 The original file is the data file to be processed, the slice data is obtained by slicing and parsing the data file to be processed, and the slice data is stored in the vector database. The slice data stored in the vector database is hierarchically divided, and the first index belonging to the parent node vector is determined according to the title level of the slice data, and further, the second index belonging to the child node vector is determined based on the first index, so as to realize the parent node division and the child node division of the slice data stored in the vector database.

[0114] After constructing the hierarchical index and the vector database, the hierarchical index and the vector database can be stored in a distributed manner, and a distributed file system is constructed. As shown in Figure 4 The vector hierarchical index is the hierarchical index, and after writing the vector hierarchical index, block operation is performed on each vector hierarchical index, and the node name of the vector hierarchical index and the id of the node name and other data are read. The node name is subjected to block operation, and the vector hierarchical index is rewritten to construct a copy of the vector hierarchical index.

[0115] Further, when constructing the hierarchical index corresponding to the vector database, it can be determined layer by layer based on the index level information, and the specific implementation is as follows:

[0116] determine first index information and second index information in the index level information; construct a first index matching the index type for the first index information, and construct a second index matching the index type for the second index information; and take the first index and the second index as the hierarchical index corresponding to the vector database.

[0117] Specifically, the first index information refers to field dimension index information, and the second index information refers to data block dimension index information. The first index is a parent node dimension index, and the second index is a child node dimension index. The second index is an index level below the first index.

[0118] Based on this, the first index information and the second index information are determined in the index level information, the first index matching the index type is constructed for the first index information, and the second index matching the index type is constructed for the second index information. The second index is an index level below the first index. The first index and the second index are taken as the hierarchical index corresponding to the vector database.

[0119] In the above example, after determining the two-dimensional array form of the vector space corresponding to the vector database, the inverted index is selected as the index type in combination with the vector space. Based on the index level information, the title level or the chapter level can be determined. The title level or the chapter level is taken as the first index, and the data block under the title level or the chapter level is taken as the second index. The first index is a coarse-grained index, corresponding to the first dimension in the vector space, i.e., the field dimension. The second index is a fine-grained index, corresponding to the second dimension in the vector space, i.e., the data block dimension.

[0120] In summary, the first index and the second index are taken as the hierarchical index corresponding to the vector database. The hierarchical index is constructed, and data retrieval positioning can be quickly performed in subsequent retrieval nodes.

[0121] Further, after the vector database and the hierarchical index are stored in a distributed manner, retrieval services can be provided based on the hierarchical index and the vector database, and the specific implementation is as follows:

[0122] receiving a data query request, parsing the data query request to obtain problem information, converting the problem information into a problem vector, and determining target index information associated with the problem vector in the distributed hierarchical index; and determining answer information corresponding to the problem information in the distributed vector database based on the target index information.

[0123] Specifically, the data query request can be a retrieval request submitted by a user, the data query request carries question information, the question information is a question, and the retrieval user hopes to obtain relatively accurate answer information corresponding to the question information. The question vector is a vector expression of the question information. The target index information can be determined by similarity matching of the hierarchical index and the question vector.

[0124] Based on this, the data query request is received and analyzed to obtain the question information. The question information is converted into a question vector after generalization processing. The target index information associated with the question vector is determined in the distributed storage hierarchical index through similarity matching. Based on the target index information, the answer information corresponding to the question information is determined in the distributed storage vector database.

[0125] In the above example, after the retrieval user determines the question "What is corporate culture?", the data query request is submitted to the distributed system based on this question. The question information "What is corporate culture?" is obtained by analyzing the data query request. The question vector is converted after generalization processing of the question information. Based on the question vector, the parent node information "Organization and policy" with high similarity to the question vector is determined through similarity matching. Based on the parent node information and the question vector, further similarity matching with the hierarchical index is performed to determine the child node information with high similarity to the question vector, and then the answer information corresponding to the question information is retrieved in the vector database based on the child node information.

[0126] In summary, the answer information corresponding to the question information is determined based on the distributed storage hierarchical index and vector database, improving the retrieval efficiency and accuracy of the answer information.

[0127] Further, when retrieving the question information, the hierarchical index can be used for step-by-step retrieval, which is implemented as follows:

[0128] The parent node index associated with the question vector is determined in the distributed storage hierarchical index; the child node index is determined based on the parent node index and the question vector, and the parent node index and the child node index are used as the target index information.

[0129] Specifically, the parent node index refers to the parent node information with high similarity to the question vector in the hierarchical index; and the child node index refers to the child node information with high similarity to the question vector under the parent node information.

[0130] Based on this, the parent node index with high similarity to the question vector is determined in the distributed storage hierarchical index through similarity matching. The child node index with high similarity to the question vector is determined under the parent node index through similarity matching, and the parent node index and the child node index are used as the target index information.

[0131] In summary, the combination of hierarchical index and vector database retrieval of question information corresponding to answer information improves the retrieval efficiency of answer information.

[0132] Further, considering that the data in the to-be-processed file is updated in real time, the distributed storage hierarchical index also needs to be updated synchronously, and the specific implementation is as follows:

[0133] In the case of updating the to-be-processed file, the update data of the to-be-processed file is determined, and the distributed storage hierarchical index is indexed and updated based on the update data.

[0134] Specifically, the update of the to-be-processed file can be the addition, deletion, and modification of data in the to-be-processed file. The update data can be newly added data, modified data, or deleted data.

[0135] Based on this, in the case that the data in the to-be-processed file is deleted to update the to-be-processed file, the deleted data is used as the update data; in the case that the data in the to-be-processed file is modified to update the to-be-processed file, the modified data is used as the update data; in the case that new data is added to the to-be-processed file to update the to-be-processed file, the new data is used as the update data; and the distributed storage hierarchical index is indexed and updated based on the update data.

[0136] In practical applications, the performance of the hierarchical index can be monitored in real time, and adjustments can be made according to actual conditions, such as adjusting the hierarchical structure, parameter settings, or storage format of the index.

[0137] In summary, in the case of updating the to-be-processed file, the distributed storage hierarchical index is indexed and updated based on the update data, thereby ensuring the accuracy of the hierarchical index.

[0138] In an embodiment of the present specification, the to-be-processed file is sliced based on a data slicing rule to obtain sliced data. The sliced data is converted into a sliced data vector, and the index level information of the sliced data vector is determined based on the slice description information of the sliced data. The sliced data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index type and the index level information. The hierarchical index and the vector database are distributed stored. Combining distributed storage, vector database, and hierarchical index, an efficient retrieval method is provided. Using distributed storage and vector database, data can be quickly located and retrieved, improving retrieval efficiency. By constructing a hierarchical index, massive data is sliced, reducing the complexity of subsequent retrieval processes. It can be applied to various scenarios such as big data processing, information retrieval, and data mining, and has wide applicability.

[0139] The data storage method provided in the present specification is further described below in combination with the accompanying drawings Figure 5 The data storage method provided in the present specification is further described below in combination with the accompanying drawings Figure 5 A processing flowchart of the data storage method applied to book data storage is shown, which specifically includes the following steps:

[0140] Step 502: Determine data files in multiple file formats.

[0141] The data storage method provided in the present embodiment can be applied to the scenario of book data storage. Book data is stored in the form of files, and the book data can include book summaries and book content. The files of the book data can be in formats such as.pdf,.txt,.doc,.docx, etc.

[0142] Step 504: Determine a data slicing rule, and perform slicing processing on the data files based on the data slicing rule to obtain sliced data.

[0143] The data slicing rule can be a punctuation symbol, a format, a paragraph, etc. The data files are sliced and processed according to the punctuation symbol, the format, or the paragraph, and are converted into sliced data in json format. Each sliced data can adopt a snowflake algorithm to form a unique id and paragraph content.

[0144] Step 506: Analyze the sliced data to obtain a sliced data vector and determine the hierarchical information of the sliced data vector in the data file.

[0145] The sliced data files are analyzed by using a preset analysis method (BM25+BGE analysis method), each sliced data in json format is analyzed into a matrix with size = [512, 1], and a sliced data vector is obtained. The analysis method can also be TF-IDF, etc., and the sliced data is analyzed into a vector matrix with a corresponding size. The present embodiment does not make any limitation on the selection of the analysis method.

[0146] Step 508: Construct a vector database based on the sliced data vector.

[0147] After the sliced data is analyzed into a sliced data vector, feature analysis is performed on the sliced data vector to determine data features such as data dimensions, data distribution, query mode, and access frequency corresponding to the sliced data vector. After the sliced data vector is preprocessed by dimension reduction, standardization, or normalization, etc., a vector database is constructed to form a vector space. The vector database can select a milvus vector database or a qdrant vector database.

[0148] Step 510: Determine the hierarchical index corresponding to the vector database based on the hierarchical information.

[0149] The hierarchical information can be the title level or chapter level of the data file. The first-level title is the first level, the second-level title is the second level, and so on. Based on the hierarchical information, the data is hierarchically divided to construct the hierarchical index. The hierarchical index can divide the data into multiple levels, and each level contains a certain number of data blocks, which facilitates fast positioning and retrieval of data. When constructing the hierarchical index, appropriate index methods can be selected according to actual needs, such as: inverted index, KD tree, LSH (locality-sensitive hashing), PQ (product quantization), etc.

[0150] In practical applications, coarse-grained index and fine-grained index can be established for the vector database respectively. The coarse-grained index can be established based on the first dimension of the vector space, which can be the field dimension stored in the vector database. For example, given the field of the incoming milvus vector database, a coarse-grained index is established. The fine-grained index is established on the basis of the coarse-grained index, and a fine-grained index is established for each partition. These indexes can be established based on the second dimension of the vector space, which can quickly locate a specific data set, and the second dimension can be the number of segmented texts. For example, analyze each piece of data in the incoming milvus vector database.

[0151] Step 512: Distributively store the vector database and the hierarchical index.

[0152] After determining the vector database and the hierarchical index, a distributed file system can be constructed. By taking advantage of the high reliability, high throughput, and high scalability of HDFS, massive data can be stored in multiple nodes in a distributed manner, improving the storage and access efficiency of data. For the storage of the hierarchical index, the fields of the coarse-grained index and each piece of data of the fine-grained index can be stored in HDFS. In addition, for the hierarchical index, it can be updated regularly to reflect changes in data, and the performance of the index can be monitored to make adjustments according to actual conditions, such as adjusting the hierarchical structure, parameter settings, or storage format of the hierarchical index.

[0153] Step 514: Receive question information and search answer information corresponding to the question information based on the question information, the hierarchical index, and the vector database.

[0154] After the distributed file system is constructed, the retrieval task can be performed. The user provides the question information that needs to be retrieved. After generalization and vectorization processing of the question information, the question vector is obtained. Based on the hierarchical index in the distributed file system, the parent node and child node data blocks corresponding to the question vector are determined. The question vector is matched using the vector data in the child node data block to obtain the retrieval result corresponding to the question information.

[0155] In summary, by combining the distributed file system, hierarchical index and vector database, an efficient retrieval method is proposed. By utilizing the advantages of HDFS and vector database, data can be quickly located and retrieved, improving retrieval efficiency. By constructing hierarchical index, massive data can be divided into multiple levels, reducing the complexity of the retrieval process. Based on the distributed storage mechanism of HDFS, the increasing amount of data can be easily handled, and the system has good scalability. The data storage method provided by the embodiment can be applied to various scenarios such as big data processing, information retrieval and data mining, and has high universality.

[0156] Corresponding to the method embodiments described above, the present specification also provides data storage device embodiments, Figure 6 The structure of a data storage device provided by an embodiment of the present specification is shown in the structural schematic diagram. As shown in the figure, Figure 6 The device comprises:

[0157] The slicing module 602 is configured to slice the to-be-processed file based on a data slicing rule to obtain sliced data;

[0158] The determination module 604 is configured to convert the sliced data into a sliced data vector, and determine index level information of the sliced data vector based on slicing description information of the sliced data;

[0159] The storage module 606 is configured to store the sliced data vector to a vector database, and determine data features corresponding to the vector database;

[0160] The construction module 608 is configured to determine an index type according to the data features, and construct a hierarchical index corresponding to the vector database based on the index type and the index level information, and perform distributed storage on the hierarchical index and the vector database.

[0161] In an optional embodiment, the slicing module 602 is further configured to:

[0162] Determine a symbol rule, a paragraph rule, a format rule and a title rule based on a data slicing rule;

[0163] Slice the to-be-processed file based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain the sliced data.

[0164] In an optional embodiment, the slicing module 602 is further configured to:

[0165] Slice the to-be-processed file based on the symbol rule, the paragraph rule, the format rule or the title rule to obtain initial sliced data;

[0166] assign a data identifier to the initial slice data based on the identifier assignment rule, and take the initial slice data and the data identifier as the slice data.

[0167] In an optional embodiment, the determining module 604 is further configured to:

[0168] determine a parsing algorithm and a matrix size for the slice data;

[0169] parse the slice data based on the parsing algorithm and the matrix size to obtain the slice data vector.

[0170] In an optional embodiment, the determining module 604 is further configured to:

[0171] determine title level information of the slice data based on the to-be-processed file, and take the title level information as the slice description information;

[0172] determine a title level corresponding to the slice data vector based on the slice description information, and generate index level information of the slice data vector based on the title level.

[0173] In an optional embodiment, the constructing module 608 is further configured to:

[0174] determine first index information and second index information in the index level information;

[0175] construct a first index matching the index type for the first index information, and construct a second index matching the index type for the second index information;

[0176] take the first index and the second index as the corresponding level index of the vector database.

[0177] In an optional embodiment, the constructing module 608 is further configured to:

[0178] receive a data query request, and parse the data query request to obtain question information;

[0179] convert the question information into a question vector, and determine target index information associated with the question vector in the distributedly stored level index;

[0180] determine answer information corresponding to the question information in the distributedly stored vector database based on the target index information.

[0181] In an optional embodiment, the constructing module 608 is further configured to:

[0182] determine a parent node index associated with the problem vector in the hierarchical index stored in a distributed manner;

[0183] determine a child node index based on the parent node index and the problem vector, and use the parent node index and the child node index as the target index information.

[0184] In an optional embodiment, the construction module 608 is further configured to:

[0185] In the case of an update to the to-be-processed file, determine update data of the to-be-processed file;

[0186] perform index update on the hierarchical index stored in a distributed manner based on the update data.

[0187] In an embodiment of the present specification, the to-be-processed file is sliced based on a data slicing rule to obtain slice data. The slice data is converted into a slice data vector, and the index level information of the slice data vector is determined based on the slice description information of the slice data. The slice data vector is stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined according to the data features, and the hierarchical index corresponding to the vector database is constructed based on the index type and the index level information. The hierarchical index and the vector database are stored in a distributed manner. In combination with the distributed storage, the vector database and the hierarchical index, an efficient retrieval method is provided. By using the distributed storage and the vector database, the data can be quickly located and retrieved, and the retrieval efficiency is improved. By constructing the hierarchical index, the massive data is sliced, and the retrieval complexity in the subsequent retrieval process is reduced. It can be applied to various scenarios such as big data processing, information retrieval and data mining, and has wide applicability.

[0188] The above is a schematic scheme of a data storage device according to an embodiment of the present specification. It should be noted that the technical scheme of the data storage device and the technical scheme of the data storage method described above belong to the same concept. The technical scheme of the data storage device is not described in detail, and the details can be referred to the description of the technical scheme of the data storage method.

[0189] Figure 7 A structural block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 through a bus 730, and a database 750 is used to save data.

[0190] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 740 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, or the like.

[0191] In one embodiment of the present specification, the above-mentioned components of the computing device 700 and other components not shown in the Figure 7 may be connected to each other, for example, through a bus. It should be understood that Figure 7 The computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed.

[0192] The computing device 700 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 can also be a mobile or stationary server.

[0193] wherein the processor 720 implements the steps of the data storage method when executing the computer program or instructions.

[0194] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the data storage method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data storage method.

[0195] An embodiment of the present specification also provides a computer readable storage medium storing computer programs or instructions, which, when executed by a processor, implement the steps of the data storage method described above.

[0196] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data storage method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the data storage method.

[0197] An embodiment of the present specification also provides a computer program product comprising computer programs or instructions, which, when executed by a processor, implement the steps of the data storage method described above.

[0198] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the data storage method described above belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the data storage method.

[0199] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0200] The computer programs or instructions include computer program codes, which can be in the form of source codes, object codes, executable files, or some intermediate forms of codes, etc. The computer readable medium can include any entity or device capable of carrying the computer program codes, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0201] It should be noted that, for the aforementioned method embodiments, the sequences of the actions are described for ease of description. However, it is to be understood that the sequences of the actions can be changed according to the embodiments of the present application, and more actions can be added, or existing actions can be removed, depending on the actual conditions. Moreover, the embodiments described in the specification are preferred embodiments only and do not limit the scope of the application.

[0202] In the above embodiments, the description of each embodiment is focused on a certain aspect, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0203] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, according to the content of the present application, many modifications and changes can be made. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited by the claims and their full scope and equivalents.

Claims

1. A data storage method, characterized in that, include: The file to be processed is sliced ​​based on data slicing rules to obtain sliced ​​data; The slice data is converted into a slice data vector, and the title level information of the slice data is determined based on the file to be processed. The title level corresponding to the slice data vector is determined based on the title level information, and the index level information of the slice data vector is generated based on the title level. The sliced ​​data vectors are stored in a vector database, and the data features corresponding to the vector database are determined. The index type is determined based on the data characteristics, and a hierarchical index corresponding to the vector database is constructed based on the index type and the index hierarchy information. The hierarchical index and the vector database are then stored in a distributed manner.

2. The data storage method according to claim 1, characterized in that, The process of slicing the file to be processed based on data slicing rules to obtain sliced ​​data includes: The rules for determining symbols, paragraphs, formatting, and headings are based on the data slicing rules. The file to be processed is sliced ​​based on the symbol rules, paragraph rules, format rules, or heading rules to obtain the slice data.

3. The data storage method according to claim 2, characterized in that, The step of slicing the file to be processed based on the symbol rules, the paragraph rules, the format rules, or the heading rules to obtain the slice data includes: The file to be processed is sliced ​​based on the symbol rules, the paragraph rules, the format rules, or the heading rules to obtain initial slice data. Data identifiers are assigned to the initial slice data based on the identifier allocation rules, and the initial slice data and the data identifiers are used as the slice data.

4. The data storage method according to claim 1, characterized in that, The step of converting the slice data into a slice data vector includes: The parsing algorithm and matrix size are determined for the sliced ​​data; The slice data is parsed based on the parsing algorithm and the matrix size to obtain the slice data vector.

5. The data storage method according to claim 1, characterized in that, The construction of the hierarchical index corresponding to the vector database based on the index type and the index hierarchy information includes: The first-level index information and the second-level index information are determined from the index hierarchy information; A first-level index matching the index type is constructed based on the first-level index information, and a second-level index matching the index type is constructed based on the second-level index information; The first-level index and the second-level index are used as the hierarchical indexes corresponding to the vector database.

6. The data storage method according to claim 1, characterized in that, After distributing the hierarchical index and the vector database, the method further includes: Receive a data query request and parse the data query request to obtain the problem information; The problem information is converted into a problem vector, and the target index information associated with the problem vector is determined in the hierarchical index of the distributed storage; Based on the target index information, the answer information corresponding to the question information is determined in the distributed storage vector database.

7. The data storage method according to claim 6, characterized in that, Determining the target index information associated with the problem vector in the hierarchical index of the distributed storage includes: Determine the parent node index associated with the problem vector in the hierarchical index of the distributed storage; The child node index is determined based on the parent node index and the question vector, and the parent node index and the child node index are used as the target index information.

8. The data storage method according to claim 1, characterized in that, After distributing the hierarchical index and the vector database, the method further includes: If the file to be processed is updated, determine the updated data of the file to be processed; The hierarchical index of the distributed storage is updated based on the updated data.

9. A data storage device, characterized in that, include: The slicing module is configured to slice the file to be processed based on data slicing rules to obtain sliced ​​data. The determining module is configured to convert the slice data into a slice data vector, determine the title level information of the slice data based on the file to be processed, determine the title level corresponding to the slice data vector based on the title level information, and generate the index level information of the slice data vector based on the title level. The storage module is configured to store the sliced ​​data vectors into a vector database and determine the data features corresponding to the vector database; The construction module is configured to determine the index type based on the data characteristics, construct the hierarchical index corresponding to the vector database based on the index type and the index hierarchy information, and perform distributed storage on the hierarchical index and the vector database.

10. A computing device, comprising a memory, a processor, and a computer program or instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program or instructions, it implements the steps of the method according to any one of claims 1-8.

11. A computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1-8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1-8.

Images