Data storage method and device and electronic equipment
Through segmented index structure and parallel search, the problem of user operation blocking during data index drop-off is solved, the rapid persistence of incremental data and the improvement of retrieval efficiency is achieved, and the time spent on drop-off when the data volume is large is solved and search efficiency challenges.
Patent Information
- Application Number
- CN202310539183.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
When processing unstructured data, the data index will block the operation of the user during the drop-off process, which will affect the retrieval efficiency. Especially when the data volume is huge, how to reduce the time-consuming drop-off without affecting the retrieval efficiency and improve the retrieval efficiency. Real-time user experience is an urgent problem.
Using a segmented index structure, when obtaining incremental data, the target cluster is filtered by the similarity between the encoded vector and the cluster in memory, the incremental data is written into the corresponding writable index, and a new one is created when the preset conditions are met. Indexing to achieve rapid persistence of incremental data; parallel search and compression merging strategies to improve retrieval efficiency.
It greatly reduces the delay in dropping, reduces service blocking time, improves real-time performance and retrieval efficiency, ensures real-time experience for users, and avoids index bloat and invalid retrieval.
Smart Images

Figure CN118939838A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information technology (IT) technology, and in particular to a data storage method, device and electronic device. Background Art
[0002] With the rapid growth of unstructured data (such as images), storing and querying these data face huge challenges. Generally, these data are often described in a high-dimensional space and stored in an indexed manner. In order to avoid frequent reading of data from the disk, the index of these data is often stored in memory. However, for the high reliability of data, it is often necessary to write the index to the disk in real time. During the process of indexing, in order to ensure data consistency, the data stored in this part of the index cannot be modified. Therefore, in this process, the user's request to add, delete or modify data (that is, requests to add, delete or change data) will be blocked. As the amount of data continues to grow from tens of millions, hundreds of millions, and tens of billions, the time it takes to write the full amount of data to the disk will be significantly longer, causing users to be unable to operate for a long time. Therefore, how to reduce the time spent on disk and improve the user's real-time experience without affecting the retrieval efficiency is a technical problem that needs to be solved urgently. Summary of the invention
[0003] The present application provides a data storage method, device, vector index structure, electronic device, computer storage medium and computer product, which can reduce the time consumption of storing data on disk without affecting the retrieval efficiency.
[0004] In a first aspect, the present application provides a data storage method, comprising: obtaining a coding vector of the incremental data when incremental data is obtained; screening out a target cluster from the first index based on the similarity between the coding vector of the incremental data and the coding vector of the cluster stored in the first index in the memory, wherein the cluster in the first index is obtained by clustering; writing the coding vector and identifier of the incremental data into the target cluster in the Nth second index in the memory, wherein the status of the indexes in the memory except the Nth second index are all read-only and have been persisted to the storage medium, N≥1. Exemplarily, the Nth second index may be the most recently created index in the memory. Exemplarily, the first index may be understood as the index described below, and the second index may be understood as the segment described below.
[0005] In this way, through the segmented index structure, when there is incremental data, real-time writing and storage can only involve the index where the incremental data is located, without involving other indexes. Since the amount of data in the index where the incremental data is located is very small compared to the full amount of data, the storage latency can be greatly reduced when the incremental data is stored on the disk, improving the storage efficiency, thereby reducing the service blocking time and improving the real-time performance, and realizing the rapid persistence of incremental data.
[0006] In a possible implementation, the method further includes: updating the data volume in the target cluster in the Nth second index, so that the data volume in each cluster in the Nth second index is consistent with the actual data volume, which facilitates subsequent statistics of the total data volume in the index.
[0007] In a possible implementation, it further includes: when the creation time of the Nth second index reaches a preset time, or the amount of data in the Nth second index reaches a preset threshold, creating the N+1th second index in the memory, changing the state of the Nth second index from writable to read-only, and persisting the Nth second index to the storage medium. In this way, the data storage time can be controlled, and the rapid persistence of incremental data is achieved.
[0008] In a possible implementation, the method further includes: obtaining a query statement input by a user; selecting f clusters from the first index based on the similarity between the query statement and the encoding vectors of the clusters stored in the first index; searching in parallel in the data stored in the f clusters in the first index and at least one second index to obtain m search results; and outputting the m search results. In this way, searching in multiple indexes in a parallel search manner greatly improves the search efficiency.
[0009] In a possible implementation, the method further includes: obtaining a query statement and query conditions input by a user; selecting f clusters from the first index based on the similarity between the query statement and the encoding vectors of the clusters stored in the first index; selecting q indexes that meet the query conditions from the first index and the second index based on the query conditions; searching in parallel in the data stored in the f clusters in the q indexes to obtain m search results; and outputting the m search results. In this way, by performing parallel searches in the indexes that meet the user's query conditions, excessive invalid searches can be avoided, greatly improving the search efficiency.
[0010] In a possible implementation, it also includes: based on a preset compression and merging strategy, k second indexes that meet the compression and merging strategy are screened out from the storage medium; when k ≥ 1, a third index is created in the storage medium; the data stored in the clusters in the k second indexes are read, and the data not marked as deleted is written into the cluster in the third index for storing the data not marked as deleted, and the amount of data stored in the cluster in the third index is written into the third index; the third index is loaded into the memory, and the k second indexes are released in the memory and the storage medium. In this way, the indexes that meet the conditions are compressed and merged according to the data structure of the index, and the deleted data in the index can be quickly cleaned up, and since the data structure after compression and merging has not changed, the subsequent retrieval efficiency can be guaranteed. In addition, after the compression and merging of the indexes are completed, the switching between the old and new indexes can also ensure that the user's add, delete, modify and query requests will not be blocked. Exemplarily, for writing data not marked for deletion into a cluster in the third index for storing the data not marked for deletion, for example, when data A is data not marked for deletion and the data is stored in cluster 1, the data A can be written into cluster 1 in the third index. Exemplarily, the third index can be understood as a segment newly created in the compression and merging process described below.
[0011] In a possible implementation, the time when the amount of data stored in the cluster in the third index is written into the third index is later than the time when the data not marked as deleted is written into the third index, thereby avoiding frequent data reading by delaying the backfilling of data and reducing the time of compression and merging.
[0012] In one possible implementation, the compression and merging strategy includes one or more of the following: the proportion of data marked for deletion in the index reaches a preset value, the creation time of the index reaches a preset duration, or the amount of data in the index is less than a preset value.
[0013] In a second aspect, the present application provides a data storage device, comprising: an acquisition module and a processing module. The acquisition module is used to obtain the encoding vector of the incremental data when the incremental data is acquired. The processing module is used to filter out the target cluster from the first index based on the similarity between the encoding vector of the incremental data and the encoding vector of the cluster stored in the first index in the memory, wherein the cluster is obtained by clustering. The processing module is also used to write the encoding vector and identifier of the incremental data into the target cluster in the Nth second index in the memory, wherein the status of the indexes in the memory except the Nth second index are all read-only and have been persisted to the storage medium, N≥1.
[0014] In a possible implementation manner, the processing module is further configured to: update the amount of data in the target cluster in the Nth second index.
[0015] In one possible implementation, the processing module is also used to: when the creation time of the Nth second index reaches a preset time, or the amount of data in the Nth second index reaches a preset threshold, create the N+1th second index in the memory, change the status of the Nth second index to read-only, and persist the Nth second index to the storage medium.
[0016] In a possible implementation, the acquisition module is further used to: acquire a query statement input by a user. The processing module is further used to select f clusters from the first index based on the similarity between the query statement and the encoding vectors of the clusters stored in the first index, and to search in parallel in the data stored in the f clusters in the first index and at least one second index to obtain m search results, and output the m search results.
[0017] In a possible implementation, the acquisition module is further used to: acquire a query statement and a query condition input by a user. The processing module is further used to select f clusters from the first index based on the similarity between the query statement and the encoding vectors of the clusters stored in the first index; and, based on the query condition, select q indexes that meet the query condition from the first index and the second index; and search in parallel in the data stored in the f clusters in the q indexes to obtain m search results, and output the m search results.
[0018] In one possible implementation, the processing module is also used to: based on a preset compression and merging strategy, filter out k second indexes that meet the compression and merging strategy from the storage medium; when k ≥ 1, create a third index in the storage medium; read the data stored in the clusters in the k second indexes, write the data not marked as deleted into the cluster in the third index for storing the data not marked as deleted, and write the amount of data stored in the cluster in the third index into the third index; load the third index into the memory, and release the k second indexes in the memory and the storage medium.
[0019] In a possible implementation, the time when the amount of data stored in the cluster within the third index is written into the third index is later than the time when the data not marked as deleted is written into the third index.
[0020] In one possible implementation, the compression and merging strategy includes one or more of the following: the proportion of data marked for deletion in the index reaches a preset value, the creation time of the index reaches a preset duration, or the amount of data in the index is less than a preset value.
[0021] In a third aspect, the present application provides a vector index structure, including: a first index and N second indexes. The first index is used to store the identifier and encoding vector of the cluster obtained by clustering, and the status of the first index is read-only. The second index is used to store the identifier, encoding vector and data volume of the data in the cluster, and different second indexes are created at different times. Among the N second indexes, the status of the Nth index is writable, and the status of the indexes except the Nth index are all read-only.
[0022] In one possible implementation, the data stored in at least one second index includes incremental data.
[0023] In a fourth aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory; wherein, when the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.
[0024] In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.
[0025] In a sixth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.
[0026] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a schematic diagram of a vector index structure provided in an embodiment of the present application;
[0028] Figure 2 is a schematic diagram of another vector index structure provided in an embodiment of the present application;
[0029] Figure 3 This is a schematic diagram of a process of real-time writing of incremental data and storage of incremental data on disk provided by an embodiment of the present application;
[0030] Figure 4 It is a schematic diagram of data changes stored in a segment before and after writing data in the segment provided by an embodiment of the present application;
[0031] Figure 5 is a schematic diagram of a data retrieval process provided by an embodiment of the present application;
[0032] Figure 6 It is a schematic diagram of the steps of compressing and merging segments provided in an embodiment of the present application;
[0033] Figure 7 It is a schematic diagram of a process of compressing and merging two segments provided in an embodiment of the present application;
[0034] Figure 8 is a structural schematic diagram of a data storage device provided in an embodiment of the present application;
[0035] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0036] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.
[0037] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.
[0038] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0039] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.
[0040] First, the technical terms that may be involved in this application are introduced.
[0041] (1) Clustering
[0042] Clustering is to divide a data set into different clusters according to a specific standard (such as distance, etc.), so that the similarity of data objects in the same cluster is as large as possible, and the difference of data objects in different clusters is as large as possible. In clustering, each data is regarded as a vector, which can be obtained by encoding the data. The clustering algorithm uses a distance metric to calculate the similarity between each data and assigns data with high similarity to the same cluster. Common distance metrics include: Euclidean distance, Manhattan distance, and cosine similarity.
[0043] (2) Vector Index
[0044] Vector indexing is a data indexing technology based on vector representation. In vector indexing, each data is represented as one or more vectors, and each vector is stored in the index, thus obtaining a vector index. For example, if the data is 10 pictures, a neural network (NN) can be used to extract the feature vectors (also called encoding vectors) of each picture, and these feature vectors can be stored in the index, thus obtaining the vector indexes of these 10 pictures. In general, in order to improve retrieval efficiency, data vectors are often clustered, and the clustered results are stored in the vector index.
[0045] (3) Vector retrieval
[0046] Vector retrieval is to represent data as vectors and implement data retrieval by calculating the similarity between vectors. When you need to query certain data, you can calculate the similarity between the query vector and the vectors stored in the vector index to find similar data.
[0047] Next, the technical solution provided by the embodiments of the present application is introduced.
[0048] For example, Figure 1 FIG. 1 is a schematic diagram of a vector index structure provided by an embodiment of the present application. Figure 1 As shown, the structure of the vector index in memory and disk can be mainly composed of header information and inverted sequence. The header information may include: the identification and encoding vector of each cluster obtained by clustering all data. Exemplarily, the identification and encoding vector of a cluster can be understood as the identification and encoding vector of the center of the cluster. The inverted sequence may include: the identification of the data stored in each cluster (i.e., the data contained in ids), the encoding vector corresponding to each data, and the amount of data stored in each cluster (i.e., the number of data), etc. Among them, the encoding vector of the data stored in each cluster can be obtained by NN extraction, one-hot encoding, or ordinal encoding, etc., which can be determined according to the actual situation and is not limited here.
[0049] in, Figure 1 The vector index shown is stored continuously in both memory and disk. Since the vector index of this structure is stored continuously in both memory and disk, when new data is written and needs to be persisted, it is often necessary to write all the data in memory to disk in sequence, which will cause a long delay in writing to disk, and during this period, the real-time services of adding, deleting, modifying and querying will be blocked.
[0050] In view of this, the present application embodiment provides a vector index structure, which Figure 1 The overall continuous storage index is split into a collection of multiple indexes, so that when there is incremental data, real-time writing and storage only involves the index where the incremental data is located, without involving other indexes. Since the amount of data in the index where the incremental data is located is very small compared to the full amount of data, the storage latency can be greatly reduced when the incremental data is stored on the disk, and the storage efficiency can be improved, thereby reducing the service blocking time and improving the real-time performance, and realizing the rapid persistence of incremental data.
[0051] For example, Figure 2 FIG. 2 shows a schematic diagram of another vector index structure provided by an embodiment of the present application. Figure 2 As shown, both the memory and the storage medium include: an index, at least one segment, and a segment meta (segment.Meta). Exemplarily, a segment can also be referred to as an index. Exemplarily, the storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. Among them, the available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, DVDs), or semiconductor media (for example, solid state disks (SSDs)), etc. In some embodiments, both index and segment can be understood as a substructure or subindex in a vector index.
[0052] The index in memory can be an index built offline, and its structure can be Figure 1 The index structure described in the above includes header information and inverted sequence. The inverted sequence in the index may contain data already in the electronic device to which the memory belongs, or may be empty, which is not limited here. In some embodiments, the index may also be referred to as a "first index".
[0053] The segment in the memory is mainly used to store the encoding vector of the data (i.e., incremental data) written in real time after the index; of course, the segment can also be used to store existing data, which is not limited here. The segment can be created in real time, and its data structure is the same as the inverted sequence in the index. Among them, the data in the segment is stored according to the clustering results. In some embodiments, the segment can also be called the "second index".
[0054] In memory, when the creation time of a segment reaches the preset time, or the amount of data in the segment reaches the preset threshold, another segment can be created and the previously created segment can be persisted to the storage medium. In this way, when there is incremental data, real-time writing and storage can only involve the segment where the incremental data is located, without involving other segments. Since the amount of data in a segment is very small compared to the full amount of data, the storage latency can be greatly reduced, the storage efficiency can be improved, thereby reducing service blocking time and improving real-time performance. In other words, Figure 2 The vector index structure shown is Figure 1 The overall continuous storage index is split into a collection of multiple segments to achieve fast persistence of incremental data. At the same time, storing data in segments according to clustering results can also improve retrieval efficiency. In addition, when searching, you can search in multiple segments in parallel, which also greatly improves retrieval efficiency.
[0055] The segment.Meta in the memory is mainly used to store the meta information of each segment, such as the type of each segment (writable or read-only, etc.), the data marked for deletion in the segment, etc. After the segment.Meta in the memory is updated, it can be persisted to the storage medium. In this way, when the electronic device where the memory is located has an abnormal power outage and restart, the memory can load the segment.Meta from the storage medium and restore the data based on the segment.Meta, thereby ensuring the reliability of the data. For example, the information recorded in the segment.Meta can be used to know whether the restored data is complete.
[0056] Each data in the storage medium can be obtained by persisting the data in the memory.
[0057] The above is an introduction to the vector index structure provided in the embodiment of the present application. To facilitate understanding, the following introduces the process of writing incremental data in real time and storing incremental data on disk under the encoding vector index structure, as well as the data retrieval process.
[0058] (1) The process of writing incremental data in real time and storing incremental data on disk
[0059] For example, Figure 3 A schematic diagram of a process of real-time writing of incremental data and storage of incremental data on disk provided by an embodiment of the present application is shown. Figure 3 As shown, when there is incremental data to be written, the encoding vector of the incremental data can be first extracted through NN, etc. Then, the multi-segment management unit in the memory can calculate the similarity between the encoding vector of the incremental data and the encoding vector of each cluster in the index to search for the cluster used to store the incremental data. Exemplarily, the multi-segment management unit can be a data structure for managing the index, segment and segment.Meta in the memory. The incremental data can be, but is not limited to, pictures or videos taken by the user using the electronic device to which the memory belongs, or pictures, videos or files downloaded from the Internet by the user using the electronic device to which the memory belongs.
[0060] Then, the encoding vector and identifier of the incremental data can be written (insert) into the segment in the writable state (i.e. Figure 3 Segment 2) in the cluster is used to store the incremental data. The segment in the writable state can be the latest segment created in the memory, and the segments created before this segment have been persisted to the storage medium and are in the read-only state. For example, Figure 4 As shown, Figure 4 (A) shows the data structure and data stored in segment 2 before data is written to segment 2. Figure 4 In (A), area 41 shows the number of data stored in each cluster (i.e., data volume) from left to right, area 42 shows the identifier (i.e., id) and the encoding vector (i.e., code) of the data stored in cluster 0, area 43 shows the identifier (i.e., id) and the encoding vector (i.e., code) of the data stored in cluster 1, and area 44 shows the identifier (i.e., id) and the encoding vector (i.e., code) of the data stored in cluster n. When the incremental data is data 98, if the id of the cluster to which data 98 belongs is 0, the identifier and encoding vector of data 98 can be written in cluster 0 in segment 2, and the following can be obtained after writing: Figure 4 In addition, after the incremental data is written to segment 2, the data volume of the cluster used to store the incremental data in segment 2 can be updated. For example, continue to refer to Figure 4 ,exist Figure 4In (B), the amount of data corresponding to cluster 0 in region 41 may be changed from 1 to 2. Exemplarily, the process of writing incremental data into a segment may be understood as a data storage process.
[0061] When the new segmentation conditions are met in the memory, for example, the amount of data in segment 2 reaches the preset threshold, or the creation time of segment 2 reaches the preset time, segment 2 can be persisted to the storage medium and marked as read-only (stable) in the memory, and a new segment is created to receive subsequent data write requests. After completing the disk drop of segment 2, segment.Meta can be updated and the updated segment.Meta can be persisted to the storage medium. In addition, when persisting the segment to the storage medium, the corresponding segment can be persisted to the storage medium according to the segment's data structure.
[0062] (2) Data retrieval process
[0063] For example, Figure 5 FIG. 1 is a schematic diagram showing a data retrieval process provided by an embodiment of the present application. Figure 5 As shown, after obtaining the query statement (query) input by the user, the multi-segment management unit in the memory can first calculate the similarity between the encoding vector of the query and the encoding vectors of each cluster in the index, in order to search for at least one cluster that is most similar to the query. Then, the multi-segment management unit in the memory can search in the index and segment in parallel. When searching, you can only search in the data stored in the cluster that is most similar to the query, and get the search result (result). After getting the result, the multi-segment management unit in the memory can output the result. In this way, when searching, you can search in multiple segments in parallel, which greatly improves the retrieval efficiency.
[0064] In addition, in this embodiment, the user can also set query conditions when searching. When searching later, the index and / or segment that meet the query conditions can be filtered out from the index and segment in the memory based on the query conditions set by the user. Afterwards, the search can be performed in parallel in the filtered index and / or segment. For example, when the strategy for creating a new segment in the memory is timed, such as: creating a new segment every day and storing the data of that day. At this time, the user can specify a date query when searching. When searching, the segments that meet the specified date can be filtered out first, and the search can be performed only in the filtered segments, thereby improving the search efficiency.
[0065] The above is an introduction to the process of writing incremental data in real time and storing incremental data on disk, as well as the data retrieval process provided by the embodiment of the present application. The following is an introduction to deleting data in memory and storage medium.
[0066] Generally, for the user's real-time deletion operation, if the data in the index is traversed and physical deletion is performed, the performance of the system will be seriously affected. Therefore, the soft deletion method is often used, that is, the actual data will not be removed from the memory and index immediately. However, as data is continuously inserted and deleted, the memory and storage medium resources occupied by the index will only increase, which will lead to problems such as index expansion and continuous growth of memory usage in the long run. Since the data that needs to be calculated during retrieval also increases accordingly, the retrieval performance will be reduced. Especially under the constraints of poor computing resource performance on the end side and small storage space, it is necessary to clean up the junk data that has been marked for deletion in the memory and storage medium in a timely manner to reduce space occupancy. In view of this, an embodiment of the present application provides a compression and merging solution to solve the problem of memory and / or storage medium expansion caused by marked deletion.
[0067] Specifically, Figure 6 As shown, the compression and merging process may include the following steps:
[0068] S601. Based on a preset compression and merging strategy, k segments satisfying the compression and merging strategy are selected from a storage medium.
[0069] In this embodiment, the compression and merging strategy can be preset, such as: 1. The proportion of data marked for deletion in a single segment reaches a threshold, 2. The aging time is set to regularly delete expired segments, 3. The minimum data volume threshold of the segment is set to merge small segments into large segments to reduce fragmentation, etc. In addition, in order not to block the system's external add, delete, modify and query services, the status of the segments to be compressed and merged can all be read-only. The electronic device to which the storage medium belongs can scan the segments in the memory in real time or periodically (for example, every hour, every day or every week, etc.) to filter out k segments that meet the compression and merging strategy from the storage medium. In this embodiment, there is no limit on the number of segments to be compressed and merged, which can be 1 or more.
[0070] S602: When k≥1, create a new segment in the storage medium.
[0071] In this embodiment, when k≥1, it indicates that there are segments that need to be compressed and merged. At this time, a new segment can be created in the storage medium. The newly created segment is mainly used to store the identifiers and encoding vectors of the data that are not marked as deleted in the k segments, so as to realize the compression and merging of the k segments. In some embodiments, the newly created segment can also be referred to as the "third index".
[0072] S603. Read the data in each of the k segments respectively, and filter out the encoding vectors and identifiers of the data marked as deleted, and write the encoding vectors and identifiers of the data not marked as deleted into the newly created segment, and write the number of data under each cluster in the new segment.
[0073] In this embodiment, after a new segment is created, the data in each of the k segments can be read separately, and the encoding vectors and identifiers of the data marked as deleted can be filtered out during the reading process. At the same time, the encoding vectors and identifiers of the read data that are not marked as deleted can be written to the newly created segment, and the number of data under each cluster can be written to the new segment. In this way, the compression and merging of the k segments is completed.
[0074] For example, if Figure 7As shown, if the segments to be compressed and merged are segment 1 and segment 2, the newly created segment is segment 3. At this time, you can first read and filter the data marked for deletion in cluster 0 from segment 1, and add the data not marked for deletion in cluster 0 to the location of cluster 0 in segment 3; wherein, this process can be read and written at the same time, or read first and then write, which is not limited here. Then, read and filter the data marked for deletion in cluster 0 from segment 2, and add the data not marked for deletion in cluster 0 to the location of cluster 0 in segment 3; wherein, this process can also be read and written at the same time, or read first and then write, which is not limited here. In this way, the data in the clusters in segments 1 and 2 are read and filtered in turn, and added to segment 3. After reading and filtering the data in cluster 99 in segments 1 and 2, and adding the data in cluster 99 that is not marked for deletion to segment 3, you can backfill the amount of data contained in each cluster in segment 3, that is, backfill the header information of the segment. At this point, the compression and merging of segments 1 and 2 is completed. Figure 7 For the number of data stored in each cluster in segment 3, assume that the number of data stored in cluster 0 of segment 1 is 80, the number of data marked for deletion is 2, the number of data stored in cluster 1 is 20, the number of data marked for deletion is 5, the number of data stored in cluster 99 is 50, and the number of data marked for deletion is 2; the number of data stored in cluster 0 of segment 2 is 15, the number of data marked for deletion is 3, the number of data stored in cluster 1 is 60, the number of data marked for deletion is 5, the number of data stored in cluster 99 is 32, and the number of data marked for deletion is 0. After compression and merging, the number of data stored in cluster 0 of segment 3 is 90, the number of data stored in cluster 1 is 70, and the number of data stored in cluster 99 is 80.
[0075] In addition, during the compression and merging process, the identifiers of each data that is filtered out (i.e. deleted) can also be recorded. In this way, during the compression process, if there is a power outage, the compression and merging can be continued from the last recorded identifier after restart, avoiding the repetition of some data reading and filling operations. For example, continue to refer to Figure 7 During the compression and merging process, the identifiers of the filtered data can be recorded in the compactids file.
[0076] In some embodiments, during the compression and merging process, the data in k segments may be read first, and the amount of data in each cluster may be recorded in the newly created segment. Then, the data in the k segments may be read again, and the data that has not been deleted may be added to the newly created segment. Although this method can also achieve compression and merging, due to the existence of two data reading operations, data reading is too frequent and the compression and merging time is prolonged. Therefore, it is best to use a method of delaying the filling of the amount of data in each cluster for compression and merging. The specific method may be determined according to the actual situation and is not limited here.
[0077] S604: Load the newly created segment into the memory, and release the filtered k segments in both the memory and the storage medium.
[0078] In this embodiment, after completing the compression and merging of k segments on the storage medium, the newly created segment can be loaded from the storage medium into the memory, and then the k segments can be released in the memory and the storage medium, such as being deleted.
[0079] In this way, the compression and merging process of the storage medium and the segments in the memory is completed, the invalid data in the index is cleaned up, and the resource usage in the memory and storage medium is reduced. Since the entire compression and merging process only involves segments in a read-only state, and the data in the old segment will be released only after the new segment is loaded into the memory, through this new and old segment switching strategy, the data retrieval system can continue to provide external add, delete, modify and query services during the compression and merging process. That is, this compression and merging solution has little impact on the add, delete, modify and query services provided by the system, ensuring the real-time performance of the system.
[0080] The above is an introduction to the vector index structure, incremental data writing and storage, data retrieval process and data compression and merging process provided by the embodiment of the present application. Next, based on the above content, a data storage device provided by the embodiment of the present application is introduced.
[0081] For example, Figure 8 FIG. 1 is a schematic diagram showing a structure of a data storage device provided by an embodiment of the present application. Figure 8As shown, the data storage device 800 may include: an acquisition module 801 and a processing module 802. The acquisition module 801 is used to obtain the encoding vector of the incremental data when the incremental data is acquired. The processing module 802 is used to filter out the target cluster from the first index based on the similarity between the encoding vector of the incremental data and the encoding vector of the cluster stored in the first index in the memory, wherein the cluster is obtained by clustering. The processing module 802 is also used to write the encoding vector and identifier of the incremental data into the target cluster in the Nth second index in the memory, wherein the status of the indexes in the memory except the Nth second index are all read-only and have been persisted to the storage medium, N≥1.
[0082] In some embodiments, the processing module 802 is further used to: update the amount of data in the target cluster in the Nth second index.
[0083] In some embodiments, the processing module 802 is also used to: when the creation time of the Nth second index reaches a preset time, or the amount of data in the Nth second index reaches a preset threshold, create the N+1th second index in the memory, change the status of the Nth second index to read-only, and persist the Nth second index to the storage medium.
[0084] In some embodiments, the acquisition module 801 is further used to: acquire a query statement input by a user. The processing module 802 is further used to select f clusters from the first index based on the similarity between the query statement and the encoding vectors of the clusters stored in the first index, and to search in parallel in the data stored in the f clusters in the first index and at least one second index to obtain m search results, and output the m search results.
[0085] In some embodiments, the acquisition module 801 is further used to: acquire the query statement and query condition input by the user. The processing module 802 is further used to select f clusters from the first index based on the similarity between the query statement and the encoding vector of the cluster stored in the first index; and, based on the query condition, select q indexes that meet the query condition from the first index and the second index; and search in parallel in the data stored in the f clusters in the q indexes to obtain m search results, and output the m search results.
[0086] In some embodiments, the processing module 802 is also used to: based on a preset compression and merging strategy, filter out k second indexes that meet the compression and merging strategy from the storage medium; when k≥1, create a third index in the storage medium; read the data stored in the clusters in the k second indexes, write the data not marked as deleted into the cluster in the third index for storing the data not marked as deleted, and write the amount of data stored in the cluster in the third index into the third index; load the third index into the memory, and release the k second indexes in the memory and the storage medium.
[0087] In some embodiments, the compression and merging strategy includes one or more of the following: the proportion of data marked for deletion in the index reaches a preset value, the creation time of the index reaches a preset duration, or the amount of data in the index is less than a preset value.
[0088] It should be understood that the above-mentioned device is used to execute the data storage process in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned data storage process. The working process of the device can refer to the corresponding process in the above-mentioned data storage process, which will not be repeated here.
[0089] Based on the method in the above embodiment, the embodiment of the present application further provides an electronic device 900. Fig. 9 As shown, the electronic device 900 includes: a bus 902, a processor 904, a memory 906 and a communication interface 908. The processor 904, the memory 906 and the communication interface 908 communicate with each other through the bus 902. The electronic device 900 can be, but is not limited to, an electronic device to which the memory and / or disk described above belong, such as: a mobile phone, a computer, etc. It should be understood that the present application does not limit the number of processors and memories in the electronic device 900.
[0090] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 904 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 904 may include a path for transmitting information between various components of the electronic device 900 (eg, the memory 906, the processor 904, and the communication interface 908).
[0091] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0092] The memory 906 may include a volatile memory, such as a random access memory (RAM). The processor 904 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0093] The memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the aforementioned Figure 8 The functions of each module in the memory 906 are implemented to realize all or part of the steps in the data storage process described in the above embodiment. That is, the memory 906 stores instructions for executing all or part of the steps in the data storage process described in the above embodiment.
[0094] Alternatively, the memory 906 stores executable codes, and the processor 904 executes the executable codes to respectively implement the functions of the data storage device 800 described above, thereby implementing all or part of the steps in the data storage process described in the above embodiment. That is, the memory 906 stores instructions for executing all or part of the steps in the data storage process described in the above embodiment.
[0095] The communication interface 903 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the electronic device 900 and other devices or a communication network.
[0096] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the data storage process described in the above embodiment.
[0097] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the data storage process described in the above embodiment.
[0098] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or a software-based instruction in a processor.
[0099] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.
[0100] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (random access memory, memory), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable rom, PROM), erasable programmable read-only memory (erasablePROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.
[0101] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0102] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.
Claims
1. A data storage method, characterized in that: The method comprises: When the incremental data is obtained, obtaining a coding vector of the incremental data; Based on the similarity between the encoding vector of the incremental data and the encoding vector of the cluster stored in the first index in the memory, a target cluster is selected from the first index, wherein the cluster is obtained by clustering; The encoding vector and identifier of the incremental data are written into the target cluster within the Nth second index in the memory, wherein the status of the indexes in the memory except the Nth second index are all read-only and have been persisted to the storage medium, N≥1.
2. The method according to claim 1, characterized in that The method further comprises: The amount of data in the target cluster in the Nth second index is updated.
3. The method according to claim 1 or 2, characterized in that: The method further comprises: When the creation time of the Nth second index reaches a preset time, or the amount of data in the Nth second index reaches a preset threshold, an N+1th second index is created in the memory, the state of the Nth second index is changed to read-only, and the Nth second index is persisted to the storage medium.
4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Get the query statement entered by the user; Filtering f clusters from the first index based on the similarity between the query statement and the encoding vectors of the clusters stored in the first index; Searching in parallel within the data stored in the f clusters in the first index and at least one of the second indexes to obtain m search results; The m search results are output.
5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Get the query statement and query conditions entered by the user; Filtering f clusters from the first index based on the similarity between the query statement and the encoding vectors of the clusters stored in the first index; Based on the query condition, filter out q indexes that meet the query condition from the first index and the second index; Searching in parallel in the data stored in the f clusters in the q indexes to obtain m search results; The m search results are output.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Based on a preset compression and merging strategy, k second indexes satisfying the compression and merging strategy are selected from the storage medium; In the case where k≥1, creating a third index in the storage medium; Reading data stored in the clusters in the k second indexes, writing data not marked as deleted into the clusters in the third index for storing the data not marked as deleted, and writing the amount of data stored in the clusters in the third index into the third index; The third index is loaded into the memory, and the k second indexes are released in the memory and the storage medium.
7. The method according to claim 6, characterized in that The amount of data stored in the clusters within the third index is written into the third index at a time later than the data not marked for deletion is written into the third index.
8. The method according to claim 6 or 7, characterized in that: The compression and merging strategies include one or more of the following: The percentage of data marked for deletion in the index reaches the preset value, the index creation time reaches the preset duration, or the amount of data in the index is less than the preset value.
9. A vector index structure, characterized in that: include: A first index, used to store identifiers and encoding vectors of clusters obtained by clustering, wherein the state of the first index is read-only; N (N≥1) second indexes, where the second indexes are used to store the identifier, encoding vector and data volume of the data in the cluster, and different second indexes are created at different times, wherein the status of the Nth index among the N second indexes is writable, and the status of the indexes except the Nth index are all read-only.
10. The vector index structure according to claim 9, characterized in that: The data stored in at least one of the second indexes includes incremental data.
11. A data storage device, characterized in that: include: An acquisition module, configured to acquire, when the incremental data is acquired, an encoding vector of the incremental data; a processing module, configured to filter out a target cluster from the first index based on a similarity between a coding vector of the incremental data and a coding vector of a cluster stored in the first index in the memory, wherein the cluster is obtained by clustering; The processing module is also used to write the encoding vector and identifier of the incremental data into the target cluster within the Nth second index in the memory, wherein the status of the indexes in the memory except the Nth second index are all read-only and have been persisted to the storage medium, N≥1.
12. An electronic device, characterized in that: include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; Wherein, when the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1-8.
13. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 8.
14. A computer program product, characterized in that When the computer program product runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 8.
Citation Information
Cited By
Data indexing method, data reading method, data indexing device, data reading device and electronic equipment
CN119396840A
Data indexing method, data reading method, device and electronic equipment
CN119396840B
