Searching method and device, storage medium, electronic equipment and computer program product

Abstract

The application discloses a retrieval method and device, a storage medium, an electronic device and a computer program product, and relates to the technical field of computer storage. The method comprises the following steps: allocating a plurality of data blocks corresponding to a first file to a plurality of storage nodes; calculating a first shard vector corresponding to the data blocks in each storage node according to a vector generation algorithm, and aggregating a plurality of first shard vectors into a first file vector; establishing a first association index between the first shard vector and the data blocks, and establishing a second association index between the first file vector and the first file; and retrieving the first file based on the first association index and / or the second association index. That is, the application does not rely on the surface layer for file retrieval, but performs vector-level retrieval on the first file, so that the vector features of the first file can be quickly retrieved. Through the application, the problem of low data retrieval efficiency caused by relying on surface layer information for file retrieval in the related art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer storage, and in particular, to a retrieval method and device, a storage medium, an electronic device, and a computer program product. BACKGROUND

[0002] With the rapid development of information technology, the amount of data is growing explosively. Distributed storage systems have become an important choice for storing massive data due to their high scalability and high availability. However, the distributed storage systems in the related art mainly focus on optimization schemes based on metadata and hash in the distributed storage system in terms of Input / Output (IO) processing and data management. The efficiency of file retrieval is improved by improving the metadata index structure (for example, a distributed hash table and a B+ tree index). However, the essence is to rely on file name, size, creation time, and other surface information to classify and retrieve files. It is impossible to achieve accurate classification and semantic retrieval based on file content features. When retrieving, a large number of irrelevant files still need to be traversed, and the efficiency is limited. This approach is easily affected by factors such as non-standard file names, leading to chaotic classification, and full directory scanning is often required during retrieval, which is extremely inefficient.

[0003] Therefore, the related art has the problem of low data retrieval efficiency caused by relying on surface information for file retrieval.

[0004] To address the problem of low data retrieval efficiency caused by relying on surface information for file retrieval in the related art, an effective solution has not yet been proposed.

[0005] Therefore, it is necessary to improve the related art to overcome the defects in the related art. SUMMARY

[0006] The present application provides a retrieval method, device, storage medium, electronic device, and computer program product to at least solve the problem of low data retrieval efficiency caused by relying on surface information for file retrieval in the related art.

[0007] The present application provides a retrieval method, comprising: dividing a first file into a plurality of data blocks, and distributing the plurality of data blocks to a plurality of storage nodes; calculating a first shard vector corresponding to each data block in each storage node according to a vector generation algorithm, and aggregating a plurality of first shard vectors into a first file vector; establishing a first association index between each first shard vector and each data block, and establishing a second association index between the first file vector and the first file; and retrieving the first file based on the first association index and / or the second association index.

[0008] The application further provides a retrieval device, comprising: a division module, configured to divide a first file into a plurality of data blocks, and distribute the plurality of data blocks to a plurality of storage nodes; a calculation module, configured to calculate a first shard vector corresponding to a data block in each storage node according to a vector generation algorithm, and aggregate a plurality of first shard vectors into a first file vector; an establishment module, configured to establish a first association index between each first shard vector and each data block, and establish a second association index between the first file vector and the first file; and a retrieval module, configured to retrieve the first file based on the first association index and / or the second association index.

[0009] The application further provides an electronic device, comprising: a memory, configured to store a computer program; and a processor, configured to execute the computer program to implement the steps of any of the retrieval methods.

[0010] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the retrieval methods.

[0011] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the steps of any of the retrieval methods.

[0012] According to the application, the first file is divided into a plurality of data blocks, and the plurality of data blocks are distributed to a plurality of storage nodes; a first shard vector corresponding to a data block in each storage node is calculated according to a vector generation algorithm, and a plurality of first shard vectors are aggregated into a first file vector; a first association index between the first shard vector and the data block is established, and a second association index between the first file vector and the first file is established; and the first file is retrieved based on the first association index and / or the second association index. That is, the application does not rely on the surface layer to retrieve the file, but retrieves the first file at the vector level, so that the vector features of the first file can be quickly retrieved. The problem of low data retrieval efficiency caused by relying on the surface layer information to retrieve the file in the related art is solved, and similarity retrieval can be performed based on the vector without viewing the specific content of the first file, thereby improving the retrieval efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1is a hardware structure block diagram of the retrieval method of the embodiment of the present application.

[0015] Figure 2 is a flow chart of the retrieval method according to the embodiment of the present application.

[0016] Figure 3 is a system architecture diagram of a vector-based storage IO and data association optimization system according to an optional embodiment of the present application.

[0017] Figure 4 is a structure block diagram of a retrieval device according to the embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0019] It should be noted that, in the description of the present application, the terms “include”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0020] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0021] In combination with the specific application environment architecture or specific hardware architecture on which the retrieval method depends, the specific application environment architecture or specific hardware architecture is described here.

[0022] The method embodiments provided in the embodiments of the present application can be executed in a server device or similar computing device. Taking the case of running on a server device, Figure 1 is a hardware structure block diagram of the retrieval method of the embodiment of the present application. As Figure 1 shown, the server device can include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor (MPU) or programmable logic device (PLD)) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0023] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the retrieval method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0025] The embodiments of this application provide a retrieval method, and the method is described in detail below in conjunction with the execution flow of the retrieval method.

[0026] This embodiment provides a retrieval method. Figure 2 This is a flowchart of the retrieval method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0027] Step S202, dividing the first file into a plurality of data blocks, and distributing the plurality of data blocks into a plurality of storage nodes;

[0028] Step S204, calculating a first shard vector corresponding to the data block in each storage node according to a vector generation algorithm, and aggregating a plurality of first shard vectors into a first file vector;

[0029] Step S206, establishing a first association index between each first shard vector and each data block, and establishing a second association index between the first file vector and the first file;

[0030] Step S208, searching the first file based on the first association index and / or the second association index.

[0031] Through the above steps, by dividing the first file into a plurality of data blocks, and distributing the plurality of data blocks into a plurality of storage nodes; calculating a first shard vector corresponding to the data block in each storage node according to a vector generation algorithm, and aggregating a plurality of first shard vectors into a first file vector; establishing a first association index between the first shard vector and the data block, and establishing a second association index between the first file vector and the first file; searching the first file based on the first association index and / or the second association index. That is, the present application does not rely on the surface layer to search the file, but searches the first file at the vector level, so that the vector features of the first file can be quickly searched. The problem of low data search efficiency caused by relying on surface layer information for file search in related technologies is solved, and then similarity search can be performed based on the vector without viewing the specific content of the first file, thereby improving the search efficiency.

[0032] Optionally, the step S202 of dividing the first file into a plurality of data blocks comprises: obtaining file information of the first file, wherein the file information at least includes one of the following: file size of the first file, file type of the first file; determining a shard strategy corresponding to the first file according to the file information; and dividing the first file into the plurality of data blocks based on the shard strategy.

[0033] It can be understood that the method of dividing the first file into a plurality of data blocks can be:

[0034] 1) Obtain the file information of the first file: when receiving a write or update request for the first file, the basic information of the file will be obtained first, including but not limited to: file size, file type, etc. For example: assuming that a user uploads a 2GB high-definition video file to a distributed storage system, it needs to know that the size of this file is 2GB, the type is a video file, and there may be other metadata such as resolution, encoding format, etc.

[0035] 2) Determine the sharding strategy according to the file information: According to the obtained file information, such as file type and file size, determine the most suitable sharding strategy. Among them, the sharding strategy can include: fixed size sharding, sharding according to file content characteristics, etc. For example: it may choose a fixed size sharding strategy, such as 100MB per data block, and cut the 2GB video file into 20 data blocks. If the file type is a database table, it may choose a row or column-based sharding method.

[0036] 3) Divide the first file into multiple data blocks based on the sharding strategy: Once the sharding strategy is determined, the first file can be divided into several data blocks according to the sharding strategy, and these data blocks are stored in different storage nodes respectively, realizing distributed storage of data. For example: After dividing it into 20 data blocks of 100MB, these data blocks are evenly distributed to different storage nodes according to the consistent hashing algorithm or similar strategy, ensuring that there is no single point overload.

[0037] Through the above technical solution, the data blocks can be stored in a scattered manner, thereby avoiding the hot spot problem and making the storage pressure of each node tend to be average, optimizing the overall performance of the system.

[0038] Optionally, the step S204 of aggregating the plurality of first shard vectors into a first file vector comprises: determining the load information of each storage node, and determining a master computing node among the plurality of storage nodes based on the load information; sending the first shard vector in each storage node except the master computing node to the master computing node, so that the master computing node aggregates the plurality of first shard vectors according to a preset aggregation algorithm to generate the first file vector.

[0039] It can be understood that the plurality of first shard vectors can be aggregated into one or more first file vectors, specifically:

[0040] 1) Determine the load information of each storage node: In the entire distributed storage system, the load condition of each storage node (such as CPU usage, memory occupation, disk I / O, etc.) needs to be monitored and recorded in real time. These load information can be collected regularly to make more reasonable computing task allocation decisions. For example: Assuming that a distributed storage system has 10 storage nodes, each storage node has different load conditions. The CPU usage of node A is 40%, the memory occupation is 30%, while the CPU usage of node B is as high as 80%, and the memory occupation is 60%. According to these load information, it can be determined which node is most suitable for performing vector feature aggregation calculation task.

[0041] 2) Determine the master computing node based on load information: The load information of all storage nodes can be analyzed, and a node with relatively low load and sufficient computing resources is selected as the master computing node. The master computing node needs to be responsible for the final vector aggregation calculation to generate the vector features of the file. For example, if it is found that the load information of node A is the most ideal, then node A will be selected as the master computing node and be responsible for the subsequent vector aggregation calculation.

[0042] 3) Send the first shard vector to the master computing node: Among all the storage nodes, except for the node selected as the master computing node, other nodes will send the first shard vector calculated by themselves to the master computing node. For example, assuming that node B stores the shard vectors of the first 4 data blocks of the first file, then node B will send these 4 shard vectors to node A (if node A is the master computing node).

[0043] 4) The master computing node aggregates the shard vectors to generate the file vector: After receiving the first shard vectors from other nodes, the master computing node will aggregate the first shard vectors according to the preset aggregation algorithm (such as weighted average, time sequence fusion, attention mechanism, etc.) to finally generate the complete vector features of the first file. For example, node A (as the master computing node) can use the weighted average algorithm to calculate the weight of each shard vector according to the size or importance of each data block after receiving all the first shard vectors sent by other nodes, thereby aggregating to generate the vector features of the first file.

[0044] In summary, by selecting a node with low load as the master computing node, the waste of computing resources is avoided, and the overall computing efficiency and response speed of the system are improved. Moreover, by sending only the shard vector to the master computing node instead of the entire data block, the amount of data transmitted over the network is significantly reduced, and the network delay and bandwidth occupation are also reduced.

[0045] Optionally, the step S206 of establishing the first association index between each first shard vector and each data block comprises: obtaining a data block identifier corresponding to each data block, and establishing a first mapping relationship from the each first shard vector to the each data block identifier; encapsulating the each first shard vector and the each data block identifier into a first entry based on the first mapping relationship, and inserting the first entry into a first index data structure corresponding to a vector pool to generate a first index entry, wherein the vector pool is used to store the each first shard vector; establishing a second mapping relationship from the each data block identifier to the each first shard vector; encapsulating the each data block identifier and the each first shard vector into a second entry based on the second mapping relationship, and inserting the second entry into a second index data structure corresponding to a data pool to generate a second index entry, wherein the data pool is used to store the plurality of data blocks; and establishing the first association index according to the first index entry and the second index entry.

[0046] It can be understood that the present application needs to establish a first association index and a second association index, wherein the step of establishing the first association index comprises:

[0047] 1) Obtaining a data block identifier corresponding to each data block, and establishing a first mapping relationship: when the first file is fragmented and stored to different storage nodes, each data block will be assigned a unique data block identifier (ID). These identifier information can be captured, and a first mapping relationship between the data block identifier and the calculated first shard vector is established, that is, the association between the vector features in the vector pool and the data blocks in the data pool. For example: suppose a video file is divided into 10 data blocks, each data block has a unique identifier ID1-ID10. At the same time, each data block calculates the corresponding vector feature V1-V10. The mapping relationship to be established will be: ID1→V1, ID2→V2, …, ID10→V10.

[0048] 2) Encapsulating the first entry and inserting the vector pool: encapsulating each first shard vector and its corresponding data block identifier into a first entry, and inserting these first entries into the first index data structure corresponding to the vector pool. In this way, the vector pool can be used to reversely look up the corresponding data block identifier according to the vector feature. For example: the encapsulated first entry is, for example: {V1: ID1}, {V2: ID2}, …, {V10: ID10}, and is inserted into the index structure of the vector pool, for example, using the inverted file index method to insert into the index structure of the vector pool.

[0049] 3) Establishing a second mapping relationship: In order to realize the forward query of data block identification to vector feature, a second mapping relationship needs to be established, that is, the association from data block identification to vector feature. For example: the second mapping relationship can be: ID1→V1, ID2→V2, …, ID10→V10, which is essentially reciprocal with the first mapping relationship.

[0050] 4) Packaging the second entry and inserting into the data pool: Each data block identification and its corresponding first fragment vector are packaged as a second entry and inserted into the second index data structure in the data pool to complete the bidirectional association mechanism between the data pool and the vector pool. For example: the packaged second entry is like {ID1: V1}, {ID2: V2}, …, {ID10: V10}, and is inserted into the index structure of the data pool.

[0051] 5) Establishing a first association index: According to the first index entry in the vector pool and the second index entry in the data pool, a first association index is constructed to support bidirectional query based on vector features or data block identification, ensuring effective association and efficient retrieval of vector features and original files. For example: assuming that there are two data block identifications ID1 and ID2 in the system, and the associated vectors are V1 and V10 respectively, the first association index can be constructed as: {ID1: {V1}, ID2: {V10}} and {V1: {ID1}, V10: {ID2}}.

[0052] In summary, by establishing a first association index, whether based on vector features or data block identification, the specific content of the file can be quickly located, which can improve the retrieval efficiency and accuracy.

[0053] Optionally, the step S206 of establishing the second association index between the first file vector and the first file comprises: obtaining a file identification corresponding to the first file, and establishing a third mapping relationship from the first file vector to the file identification; packaging the first file vector and the file identification into a third entry based on the third mapping relationship, and inserting the third entry into a third index data structure corresponding to a vector pool to generate a third index entry, wherein the vector pool is used to store the first file vector; establishing a fourth mapping relationship from the file identification to the first file vector; packaging the file identification and the first file vector into a fourth entry based on the fourth mapping relationship, and inserting the fourth entry into a fourth index data structure corresponding to a data pool to generate a fourth index entry, wherein the data pool is used to store the first file; and establishing the second association index according to the third index entry and the fourth index entry.

[0054] It can be understood that the step of establishing the second association index can be:

[0055] 1) Obtain the file identifier of the first file, and establish a third mapping relationship: Each file has a unique file identifier in the distributed storage system. After the vector feature of the first file is calculated, a third mapping relationship between the file vector and the file identifier can be established to ensure that the file vector and its corresponding file can be accurately associated at any time. For example: Assuming that the file identifier of the first file is FI1 and the calculated file vector is FV1, the third mapping relationship is to establish the association of FV1 to FI1 in the vector pool.

[0056] 2) Package the third entry and insert it into the vector pool: Package the first file vector FV1 and the file identifier FI1 into the third entry, for example: {FV1: FI1}, and then insert this entry into the third index data structure of the vector pool. This index structure can be inverted file index and vector quantization (Inverted File, referred to as IVF), hierarchical navigable small world (Hierarchical Navigable Small World, referred to as HNSW) or other forms of vector index.

[0057] 3) Establish a fourth mapping relationship: In contrast to the third mapping relationship, a fourth mapping relationship from the file identifier FI1 to the first file vector FV1 also needs to be established, that is, the query path of the file identifier to the vector feature in the positive direction. For example: From the file identifier FI1, the file vector FV1 can be directly located, that is, FI1→FV1.

[0058] 4) Package the fourth entry and insert it into the data pool: Package the file identifier FI1 and the first file vector FV1 into the fourth entry, for example: {FI1: FV1}, and insert it into the fourth index data structure of the data pool to ensure that the file identifier can quickly locate the vector feature.

[0059] 5) Establish a second association index: Combine the third index entry and the fourth index entry to establish a second association index. The second association index not only contains the reverse query path from the vector to the file identifier, but also contains the positive query path from the file identifier to the vector, realizing the bidirectional fast positioning based on the file vector and the file identifier. For example: The second association index can be {FV1: FI1, FI1: FV1}, ensuring that from any direction, the target file can be quickly located.

[0060] In summary, the construction of the second association index can improve the query ability of the system to the file. Whether it is based on the reverse query of the vector feature or the positive query based on the file identifier, fast positioning can be achieved, improving the efficiency of file retrieval.

[0061] Optionally, the step S208 of retrieving the first file based on the first association index and / or the second association index includes:

[0062] (1) In the case of receiving a search request sent by a target object, determining a first type of a first search result to be searched corresponding to the search request, and obtaining a feature vector and a first parameter corresponding to the search request, wherein the search request is a request for searching the first file, the first type at least includes one of the following: data block type, file type, and the first parameter is used to indicate the target number of the first search result to be searched.

[0063] It can be understood that if the first file is searched, the search request needs to be parsed first: when receiving a search request sent by a target object, the request content is parsed to determine the first type of the search. It can be a data block type or a file type, or both. For example: the user may want to search for a certain segment (data block type) in a video file, or may want to find all files similar to a certain file (file type). Further, the feature vector and search parameters in the search request are obtained, for example: the number of similar results to be returned.

[0064] (2) Determining a target association index according to the first type, including: in the case of determining that the first type only includes the data block type, determining the first association index as the target association index; in the case of determining that the first type only includes the file type, determining the second association index as the target association index; in the case of determining that the first type includes the data block type and the file type, determining the first association index and the second association index as the target association index.

[0065] It can be understood that the target association index needs to be determined: according to the parsed search type, it is determined whether to use the first association index or the second association index, or both to search. For example: when the user only searches for the data block type, the first association index (vector to data block ID mapping) is located; if the user searches for the file type, the second association index (vector to file ID mapping) can be used; if the user needs to search for data blocks and files at the same time, the first and second association indexes can be combined to meet the demand.

[0066] (3) retrieving the first file according to the feature vector and the target association index to obtain the target number of first retrieval results, wherein the target association index at least includes one of the first association index and the second association index, and specifically: in a case where it is determined that the first type of the first retrieval result to be retrieved corresponding to the retrieval request includes a data block type, the feature vector is parsed to determine a vector dimension of the feature vector; a retrieval range corresponding to the retrieval request is determined according to the vector dimension, wherein the retrieval range includes one or more storage nodes; the feature vector is sent to each storage node in the retrieval range, so that each storage node in the retrieval range calculates a similarity score between the feature vector and a second shard vector corresponding to each storage node in the retrieval range according to a similarity algorithm, wherein the plurality of first shard vectors include the second shard vector; a plurality of similarity scores between the second shard vectors corresponding to one or more storage nodes in the retrieval range are sorted in descending order, and a target number of target shard vectors are determined according to the arrangement order of the plurality of similarity scores; a target number of target data blocks associated with the target shard vectors are determined according to the target association index, and the target number of target data blocks are determined as the target number of first retrieval results.

[0067] It can be understood that the retrieval range can be determined according to the vector dimension of the feature vector, that is, the set of storage nodes that need to be queried. For example: assuming that the dimension of the feature vector is 256, the shard with a 256-dimensional vector can be searched, which may involve multiple nodes in the storage cluster, because the shard may be distributed in different physical locations according to a consistent hashing algorithm. Then, the feature vector is broadcast to each storage node in the retrieval range, and the storage node calculates the similarity score between the feature vector and the locally stored second shard vector according to the preset similarity algorithm.

[0068] Through the above process, the present application can realize efficient retrieval of data blocks or files according to different types of retrieval requests by using the feature vector and the pre-constructed association index, greatly improving the retrieval speed and accuracy.

[0069] Optionally, after the first association index between each first shard vector and each data block and the second association index between the first file vector and the first file are established in step S206, the method further comprises: in a case where a first data block in the first file is updated to a second data block, calculating a third shard vector corresponding to the second data block according to the vector generation algorithm, and aggregating the third shard vector and a fourth shard vector in the plurality of first shard vectors except the first shard vector corresponding to the first data block into a second file vector; establishing a third association index between the third shard vector and the second data block, and updating the first association index according to the second association index; establishing a fourth association index between the second file vector and the updated first file, and updating the second association index according to the fourth association index.

[0070] It can be understood that after the first association index and the second association index are established, the first association index and the second association index can also be updated. Specifically, the first association index and the second association index can be updated in the following three scenarios:

[0071] Scenario one: a case where a first data block in the first file is updated to a second data block:

[0072] 1) Detect data block update: through a monitoring and logging mechanism, it is detected that a certain data block (first data block) in the first file has been updated to a new data block (second data block). For example, assume that there is a video file in which a certain scene (first data block) is re-shot and replaced with a new scene (second data block).

[0073] 2) Calculate vector features of updated data block: a preset vector generation algorithm can be used to calculate the vector features (third shard vector) corresponding to the updated data block (second data block).

[0074] 3) Incremental aggregation and update of file vector: the newly calculated third shard vector is aggregated with the vectors (fourth shard vector) of other non-updated data blocks in the file to generate a new file-level vector (second file vector). For example, if a video file has 10 shards and the third shard is updated, the vector features of the third shard can be recalculated and aggregated with the vector features of the first, second, fourth to tenth shards to obtain new video file vector features.

[0075] 4) Update the vector pool and index structure: Update the third fragment vector in the vector pool that is related to the second data block, and establish a third association index between the third fragment vector and the second data block to ensure the latest association between vector features and data blocks. When updating the first association index, adjust it according to the second association index (i.e., the index from file vector to file identifier) ​​to ensure consistency between the file identifier and the updated file vector.

[0076] 5) Update the file to vector index structure: Establish a fourth association index between the second file vector and the updated file, that is, point from the file identifier to the new file vector.

[0077] In summary, the updated index structure supports more accurate and faster retrieval. Even after the file content changes, it can quickly locate the updated vector features, improving retrieval efficiency and user experience.

[0078] Scenario 2: Adding a third data block to the first file:

[0079] If it is determined that a third data block has been added to the first file, the fifth fragment vector corresponding to the third data block is calculated according to the vector generation algorithm, and the fifth fragment vector and the plurality of first fragment vectors are aggregated into a third file vector; a fifth association index is established between the fifth fragment vector and the third data block, and the fifth association index is added to the first association index; a sixth association index is established between the third file vector and the first file after the third data block has been added, and the second association index is updated according to the sixth association index.

[0080] Understandably, in the scenario where a third data block is added to the first file, the steps for updating the first and second associated indexes can be as follows:

[0081] 1) File Append Detection: A new data block (third data block) has been detected appended to the first file. This could be due to natural growth of the file content or manual appending by the user. For example, in a log file, new log entries can be automatically appended over time, forming a third data block.

[0082] 2) Calculate the vector features of the appended data block: For the appended third data block, a preset vector generation algorithm can be used to calculate its corresponding vector features (the fifth slice vector). The vector generation algorithm can be: convolutional neural network feature extraction, hash vector generation, etc.

[0083] 3) Incremental aggregation to generate new file vector: the newly calculated fifth shard vector is aggregated with the vectors of other original data blocks (multiple first shard vectors) in the file to generate an updated file-level vector (third file vector). For example, if there are 10 data blocks in the log file before, a new 11th block is appended, then the vector features of the new 11th block can be calculated, and the features of the previous 10 blocks are integrated to obtain the updated vector features of the entire log file.

[0084] 4) Update the index structure of the vector to the data block: update the index structure in the vector pool, establish the fifth association index between the fifth shard vector and the third data block, and add it to the first association index (mapping of vector to data block ID).

[0085] 5) Update the index structure of the file to the vector: establish the sixth association index between the third file vector and the first file after adding the third data block, and update the second association index (i.e., the index of the file identifier to the file vector) according to the sixth association index.

[0086] In summary, the updated second association index can support faster and more accurate file retrieval, even if the file content changes frequently, the system can quickly locate the file most similar to the query vector, improving user search experience and system performance.

[0087] Scenario three: the fourth data block in the first file has been deleted:

[0088] In the case where it is determined that the fourth data block in the first file has been deleted, the other shard vectors in the plurality of first shard vectors except the sixth shard vector corresponding to the fourth data block are aggregated into a fourth file vector; the association index between the sixth shard vector and the fourth data block in the first association index is deleted; the seventh association index between the fourth file vector and the first file after deleting the fourth data block is established, and the second association index is updated according to the seventh association index.

[0089] It can be understood that in the scenario where the fourth data block in the first file has been deleted, the steps of updating the first association index and the second association index can be:

[0090] 1) Detect data block deletion: through log monitoring or file state checking mechanism, it is found that the fourth data block in the first file has been deleted. For example, in a database log file, a certain expired or no longer needed log entry (fourth data block) is manually deleted or removed by an automatic log cleaning strategy.

[0091] 2) Remove the vector features of the deleted data block: remove the sixth shard vector corresponding to the fourth data block from the original plurality of first shard vectors;

[0092] 3) Incremental aggregation to generate new file vector: aggregate the remaining other shard vectors to generate the updated file-level vector, i.e., the fourth file vector. For example: if the original log file is composed of 10 data blocks, and the 4th block is deleted, then the aggregation vector is recalculated according to the content of the remaining 9 blocks to ensure that the file vector reflects the true content of the current file.

[0093] 4) Update the index structure of the vector to the data block: delete the association index between the sixth shard vector and the fourth data block in the first association index (i.e., the mapping of the vector to the data block ID).

[0094] 5) Update the index structure of the file to the vector: establish the seventh association index between the fourth file vector and the first file after deleting the fourth data block, and update the second association index (i.e., the index of the file identifier to the file vector) according to the index.

[0095] In summary, the present application detects file content changes in real time, updates file vector features using an incremental aggregation strategy, and adjusts the index structure in a timely manner, ensuring high adaptability of the system to file content changes, improving retrieval efficiency and data management consistency.

[0096] Optionally, after the first file is retrieved based on the first association index and / or the second association index, the storage capacity of the vector pool can also be updated, specifically:

[0097] (1) Detect the storage capacity in the vector pool based on a preset period, wherein the vector pool is used to store target vectors stored in the plurality of storage nodes, and the target vectors at least include the first file vector and the plurality of first shard vectors.

[0098] Specifically: automatically detect the storage capacity in the vector pool based on a preset period (such as every day, every week) to evaluate the storage status of the current vector pool. For example: automatically check the vector pool every night and find that the storage capacity has reached 90%.

[0099] (2) If it is determined that the storage capacity is greater than or equal to a preset threshold, determine whether other file vectors except the first file vector are stored in the plurality of storage nodes.

[0100] Specifically: evaluate the storage capacity and the threshold, and when it is found that the storage capacity of the vector pool reaches or exceeds the preset threshold, further check whether other file vectors except the first file vector (the vector feature of the current concerned file) are stored.

[0101] (3) In the case where it is determined that the other file vectors have been stored in the plurality of storage nodes, the plurality of second files corresponding to a plurality of fifth file vectors are subjected to similar deduplication, specifically: determining a first similarity between each two files in the plurality of second files; in the case where it is determined that there is a second similarity greater than or equal to a similarity threshold value in the plurality of first similarities, a plurality of third files corresponding to the second similarity are divided into a same cluster group, wherein the plurality of second files include the plurality of third files; and the plurality of third files in the same cluster group are subjected to similar deduplication.

[0102] It can be understood that the similar deduplication operation can be performed, that is:

[0103] When it is confirmed that there are a plurality of file vectors, similarity judgment is performed based on vector similarity to identify similar files for classified management. For example: assuming that vector features of a plurality of video files are stored in the vector pool, it is found that the vector similarity of several groups of files is very high, which may be because they contain similar contents (such as advertisement paragraphs or repeated shots).

[0104] The similarity score (first similarity) between each two files in each group of files is determined, and a similarity threshold value, such as a cosine similarity of 0.9, is set to judge whether the files are similar enough.

[0105] For files with a similarity reaching or exceeding the threshold value (second similarity), they are divided into the same cluster group, and similar deduplication is performed, that is, a reference file and difference files of the remaining files are retained.

[0106] The specific steps of performing similar deduplication on the plurality of third files in the same cluster group can be: determining a reference file corresponding to the plurality of third files and a difference file corresponding to each third file, and replacing the plurality of third files with the reference file and the difference file to perform similar deduplication on the plurality of third files, wherein the reference file is a file composed of fifth data blocks with a similarity greater than or equal to a target similarity in the plurality of third files, the target similarity is greater than the similarity threshold value, and the difference file is a file composed of sixth data blocks other than the fifth data blocks in each third file;

[0107] It can be understood that first, the reference file is determined: one or more files are selected from the cluster group as the reference file, and these files are composed of data blocks with a similarity greater than or equal to a target similarity (a value higher than the threshold value) with other files in the group. Second, the difference file is generated: for each file in the group, the newly added or modified data blocks other than the data blocks already in the reference file are extracted to form a difference file. Finally, the original file is replaced: each file in the cluster group is replaced by a reference file and a set of difference files, thereby realizing the deduplication of similar files.

[0108] (4) updating the storage capacity of the vector pool according to the plurality of second files after the similarity deduplication, wherein the plurality of fifth file vectors include the first file vectors and the other file vectors stored in the plurality of storage nodes, and specifically: calculating a sixth file vector of the plurality of second files after the similarity deduplication according to the vector generation algorithm; updating the fifth file vectors in the vector pool according to the sixth file vector; determining a target storage capacity of the updated vector pool, and determining the target storage capacity as the updated storage capacity.

[0109] It can be understood that the vector pool and the storage capacity are updated, the vector features of the files after the similarity deduplication (the sixth file vector) are recalculated, and the fifth file vectors (all the file vectors already stored in the storage) in the vector pool are updated.

[0110] In summary, the application monitors the storage capacity of the vector pool, performs similarity deduplication based on the vector similarity, effectively reduces the occupation of the storage space, and improves the efficiency of data retrieval and the intelligent management level of the system.

[0111] Optionally, after the similarity deduplication of the plurality of second files corresponding to the plurality of fifth file vectors, the method further includes: determining whether the updated storage capacity is greater than or equal to a preset threshold; in the case where it is determined that the updated storage capacity is greater than or equal to the preset threshold, obtaining the plurality of first shard vectors stored in the plurality of storage nodes; performing similarity deduplication on the plurality of data blocks corresponding to the plurality of first shard vectors, and updating the updated storage capacity again according to the plurality of data blocks after the similarity deduplication.

[0112] It can be understood that after the similarity deduplication of the file vectors, the similarity deduplication of the shard vectors can also be performed, and specifically:

[0113] 1) evaluating the updated storage capacity: after any storage optimization operation (such as similarity file deduplication), it can be checked whether the current storage capacity of the vector pool still exceeds the preset threshold. For example: assuming that after the preliminary similarity file deduplication, the storage capacity of the vector pool is reduced from 95% to 90%, but the preset threshold is 85%, at this time, it can be further evaluated whether there is further optimization space.

[0114] 2) obtaining the shard vectors: if the updated storage capacity is still higher than the preset threshold, the plurality of first shard vectors stored in the plurality of nodes can be obtained.

[0115] 3) Perform data block level similarity deduplication: Perform similarity analysis on the data blocks corresponding to the plurality of first shard vectors obtained, and identify highly similar data blocks for deduplication. For example, in the storage scenario of video files, it is found that some video clips are repeatedly stored in different files, and the vectors corresponding to these clips are very similar. At this time, one of the clips can be selected as a reference, and the other clips can be converted into references to the reference clip, thereby avoiding storing duplicate data.

[0116] 4) Update storage capacity again: According to the result of data block level similarity deduplication, recalculate the storage capacity of the vector pool.

[0117] In summary, the present application realizes deep storage optimization through multi-level similarity deduplication from the file level to the data block level, effectively reduces the storage occupation, and improves the data retrieval efficiency and quality.

[0118] In order to better understand the process of the above retrieval method, the implementation method flow of the above retrieval method will be described in combination with the optional embodiments below, but it is not used to limit the technical scheme of the embodiments of the present application.

[0119] The following is an explanation of the proper nouns that may be involved in the optional embodiments of the present application:

[0120] 1. Distributed storage (Distributed Storage): A storage architecture that stores data in multiple nodes, forming a unified storage system through network connection. It has the characteristics of high scalability and high availability, and can cope with the storage demand of massive data. Common distributed storage systems include Ceph distributed storage system (Ceph Distributed Storage System, abbreviated as Ceph), Hadoop distributed file system (Hadoop Distributed File System, abbreviated as HDFS), etc. In the optional embodiments of the present application, the files of the distributed storage are usually stored in the form of shards in different nodes.

[0121] 2. Vector feature (Vector Feature): refers to the structured numerical vector converted from the content of the file (including text, binary and various data). It can be used to quantitatively describe the content features of the file. Its dimension can be set according to actual needs, for example: 128 dimensions, 256 dimensions, etc. The generation algorithm includes hash vector, convolutional neural network (Convolutional Neural Network, abbreviated as CNN) feature extraction and other ways, which can reflect the local or global features of the file. It is the core element of realizing content-based storage IO optimization and data management.

[0122] 3、Vector Index: A data structure for efficient querying of vector features, capable of supporting fast approximate similarity queries. Common vector index structures include Inverted File (IVF), Hierarchical Navigable Small World (HNSW), etc. In the optional embodiments of the present application, vector index is used to implement fast retrieval and management of files based on vector features.

[0123] 4、Incremental Update: Refers to a mechanism for recalculating and updating only the vector features corresponding to the modified part when part of the file content is modified, without the need for full processing of the vector features of the entire file. This mechanism can significantly reduce the maintenance cost during file update and improve the dynamic adaptability of the system.

[0124] 5、Vector Storage Pool: An independent storage area for storing file vector features. Vector features exist in the form of independent records in the vector pool and are bound to the original file through the association of file identifiers (IDs), realizing the association management of vector features and original files.

[0125] 6、Semantic Retrieval: A way of file retrieval based on content semantics using file vector features. By calculating the similarity between query vectors and file vector features, it can quickly locate files related to the query content, eliminating the dependence on surface information such as file names and directory structures in traditional retrieval.

[0126] 7、Single Instruction Multiple Data (SIMD): A technology that can process multiple data elements simultaneously in one instruction, such as Advanced Vector Extensions2 (AVX2) / Advanced Vector Extensions512 (AVX-512) instruction sets, which can improve the parallel efficiency of vector operations.

[0127] 8、Inverted File (IVF): A commonly used index structure for vector retrieval, which establishes an inverted table by clustering vectors to achieve efficient approximate similarity queries.

[0128] 9、Hierarchical Navigable Small World (HNSW): a graph-based vector index algorithm that realizes fast vector similarity search by constructing a multi-layer navigation graph.

[0129] 10、Convolutional Neural Network (CNN): a deep learning model, which is one of the vector generation algorithms in optional embodiments of the present application, and is used to extract deep content features of a file.

[0130] The distributed storage system in the related art has many deficiencies in IO processing and data management, and is difficult to meet the growing demand for efficient data processing. Specifically, the following problems may exist: in resource scheduling, the system in the related art usually divides data into hot data and cold data according to the access frequency of the file, and allocates storage media (for example, a Solid State Drive (SSD) is used for hot data, and a Hard Disk Drive (HDD) is used for cold data) accordingly, but this scheduling method does not consider the content value of the file, cannot perform differentiated resource allocation for key tasks, and may cause delay of core business; in data updating, when part of the content of the file is updated, the system in the related art needs to process the full amount of data again, for example, recalculate the hash value, and the update cost increases linearly with the file size. For a Terabyte (TB) level or other large files, the maintenance cost is extremely high.

[0131] The optimization scheme based on metadata and hash in the distributed storage system of the related technology has the following problems: the file retrieval efficiency is improved by improving the metadata index structure (for example, a distributed hash table, a B+ tree index), but its essence still depends on surface information such as a file name, a path, and a creation time, and cannot achieve accurate classification and semantic retrieval based on file content features, and a large number of irrelevant files still need to be traversed during retrieval, and the efficiency is improved limitedly; in the resource scheduling field, the related technology dynamically adjusts storage media (for example, migrates a high-frequency access file to an SSD) by analyzing file access frequency, size and other parameters, but cannot combine file content attributes (for example, real-time video and core code) to perform differentiated priority allocation, and it is difficult to guarantee the low-delay requirement of a key business; in the data updating aspect, the related technology uses a hash algorithm such as a Message-Digest Algorithm 5 (MD5) and a Secure Hash Algorithm (SHA) to achieve file accurate calculation, and needs to calculate the full amount of the file. When the file is partially updated, the hash value of the full amount of data needs to be recalculated and the index needs to be updated, and for a TB-level large file, the update time consumption increases linearly with the file size.

[0132] To solve the above problems, the optional embodiment of the present application proposes a vector-based storage IO and data association optimization method and system, which aims to solve the following problems: 1) the problem of extensive data management: the system in the related technology relies on surface information for file classification and retrieval, which is low in efficiency and easy to be disturbed. The optional embodiment of the present application realizes automatic clustering and semantic retrieval of files based on vector features of the files, gets rid of the dependence on surface information, and improves the accuracy and efficiency of data management; 2) the problem of unreasonable resource scheduling: the system in the related technology performs resource scheduling based on surface features, and cannot perform differentiated allocation according to file content attributes and values, which may affect the execution of key tasks. The optional embodiment of the present application identifies content attributes and values of files in combination with vector features of the files, realizes accurate scheduling of resources, and guarantees efficient operation of key tasks; 3) the problem of poor dynamic adaptability: the system in the related technology needs to perform full amount of data verification when a file is partially updated, and the update cost is high. The optional embodiment of the present application designs an incremental update mechanism, and when a file is partially modified, only the vector features corresponding to the modified part are updated, the maintenance cost is reduced, and the adaptability of the system to dynamic changes of files is improved.

[0133] Figure 3 It is a system architecture diagram of a vector-based storage IO and data association optimization system according to the optional embodiment of the present application, and on the basis of the distributed storage system of the related technology, modules and components related to vector features (vector operation API, distributed vector index service, distributed vector calculation framework, vector local calculation engine and vector pool) are newly added, and vector-based storage IO and data management optimization is realized, such asFigure 3 as shown:

[0134] 1) Data storage layer (i.e. Figure 3 of the storage layer): responsible for storing raw file data, vector feature data and metadata. Among them, the data storage layer includes: a data pool responsible for storing raw file data, the raw file is stored in the form of shards in the storage device of each node; a metadata pool responsible for storing file metadata; a vector pool responsible for storing shard-level vectors (i.e. multiple first shard vectors), file-level vectors (i.e. first file vectors, etc.). The vector feature data is stored in the vector pool, and is bound to the raw file through the association of the file ID, and an association index of the raw data and the vector is established, supporting bidirectional mapping and batch association.

[0135] 2) Computing layer: contains vector local computing engines (1, 2…, n) distributed in each storage node and a distributed vector computing framework (for example: Spark (a big data processing framework)). The vector computing engine is deployed in the storage node and has local vector feature computing capability, which can use the CPU (supporting SIMD instruction set) or lightweight graphics processing unit (Graphics Processing Unit, referred to as GPU) of the node for parallel computing; the distributed vector computing framework is responsible for distributing vector computing tasks to appropriate nodes according to the load of the nodes, and follows the principle of "data does not move and computing moves", and is preferentially scheduled to the node storing the file shard.

[0136] 3) Index layer: composed of distributed vector index service and distributed metadata service. The distributed vector index service uses IVF, HNSW and the like structure for efficient vector similarity query, and the index shards are stored in multiple nodes to realize load balancing; the distributed metadata service manages not only the traditional file metadata, but also cooperates with the vector index to support multi-condition query and real-time update of the index.

[0137] 4) Interface layer: extends the vector operation API on the basis of the storage interface in the related art, for example: "Get ObjectVector" (get object vector feature), "Search Similar Vectors" (search similar vectors), "BatchExtract Vectors" (batch extract vector features), etc., to facilitate the calling of the upper layer application.

[0138] The flow of the vector-based storage I / O and data management optimization method of the optional embodiment of the application includes:

[0139] 1, file writing and vector feature extraction:

[0140] (1) File Sharding Storage: After receiving a file write request, the system divides the file into N data blocks according to the file size, type, and preset sharding strategy (i.e., file information such as fixed-size sharding or content structure-based sharding), and distributes them to different storage nodes through a distributed storage algorithm (e.g., consistent hashing) to complete the original data storage.

[0141] 2) Local Sharding Vector Calculation: After detecting a new sharding write, each storage node triggers a local vector calculation engine to select an appropriate vector generation algorithm (e.g., Term Frequency-Inverse Document Frequency (TF-IDF) vector for text or CNN feature vector for images) based on the file type, calculate the vector features of the sharding, and generate a sharding-level vector (i.e., multiple first sharding vectors such as 256-dimensional float arrays).

[0142] 3) Master Calculation Engine Selection and Aggregation: A distributed vector calculation framework (e.g., Spark) selects a master calculation engine from all nodes through a coordination mechanism (e.g., election based on node load and computing capacity), and each node sends the local sharding vector to the master calculation engine; the master calculation engine (i.e., master calculation node) combines the sharding vectors into a file-level vector (i.e., first file vector) using a preset aggregation strategy (e.g., weighted average or time sequence fusion).

[0143] 4) Vector Storage and Associated Indexing: The master calculation engine stores the file-level vector into the vector pool, generates metadata containing vector dimensions, generation algorithm, and timestamp, and establishes a bidirectional mapping index with the original file through the associated file ID (i.e., first and second associated indexes, supporting file-to-vector and vector-to-file retrieval).

[0144] 5) Vector Index Update: The metadata service listens to vector pool write events and synchronously inserts the newly generated file-level vector into the distributed vector index (e.g., IVF, HNSW) to complete index construction, ensuring that the file can be retrieved by the vector later.

[0145] 2, File Retrieval Process:

[0146] 1) Retrieval Request Reception: Users initiate a request (i.e., retrieval request) by calling the Search Similar Vectors interface, input the target vector (e.g., file feature vector to be matched) and Top-K parameter (number of similar files to be returned), and the interface layer parses the request and forwards it to the coordination node.

[0147] 2) Query task distribution: The coordination node determines the index shard to which the target vector belongs according to the sharding rule of the vector index (such as sharding according to the vector dimension range), and distributes the query task to all index nodes managing the shard.

[0148] 3) Local vector query: After receiving the task, each index node calls the local vector index to perform approximate similarity calculation (for example, calculating the cosine distance between the target vector and the vectors in the index), and filters out the top K vectors with the highest similarity and the corresponding file ID, similarity score.

[0149] 4) Result aggregation and sorting: The coordination node collects all the candidate results returned by the index nodes, merges and removes duplicates, sorts them in descending order of similarity score, and extracts the top K results, and supplements the file metadata (such as file name, path).

[0150] 5) Retrieval result return: The coordination node returns the final Top-K similar file list to the user through the interface, completing the retrieval process.

[0151] 3, File update process:

[0152] 1) Update operation monitoring and shard positioning: The system monitors file partial update requests (such as appending and modifying content) through file operation logs, analyzes the update range, and locates the N data block shards and storage nodes affected;

[0153] 2) Recalculation of modified shard vectors: Notify the nodes storing the modified shard to trigger the vector calculation engine to recalculate the vector features of the shard (using the same algorithm as the initial calculation) and generate new shard-level vectors;

[0154] 3) File-level vector incremental correction: The main calculation engine (the same as the node during initial aggregation or a newly elected node) receives the modified shard vectors, compares the differences with the original shard vectors, and corrects the file-level vectors through incremental aggregation (such as replacing the weights of the corresponding shards and recalculating);

[0155] 4) Vector pool and associated index update: The main calculation engine updates the corrected file-level vectors to the vector pool and synchronizes the timestamp of the vector metadata; at the same time, updates the associated index of the file and the vector to ensure the accuracy of the mapping relationship.

[0156] 5) Vector index synchronization update: The metadata service monitors the update event of the vector pool and triggers incremental update of the vector index (such as deleting old vectors and inserting new vectors), avoiding dirty data during retrieval, and completing the file update closed loop.

[0157] 4, Similar clustering process:

[0158] 1) Cluster task trigger: The system triggers similar clustering tasks at a preset period (e.g., daily) or when the storage capacity reaches a threshold, and the computing task coordination module coordinates the execution;

[0159] 2) Vector traversal and similarity calculation: The computing task coordination module schedules each node to scan the locally stored file-level vectors, loads vector data in batches through a distributed computing framework (e.g., Spark), uses an efficient similarity algorithm (e.g., Locality Sensitive Hashing (LSH) hashing) to preliminarily screen potential similar vector pairs, and then confirms the similarity through accurate calculation (e.g., Euclidean distance);

[0160] 3) High-similarity file categorization: Set a similarity threshold (e.g., cosine similarity > 0.8), and divide the files that meet the conditions into the same cluster group, generating cluster labels (e.g., "similar picture group", "same theme document group").

[0161] 4) Application of clustering results: Perform optimization operations according to the clustering results, such as similar deduplication (retain reference files + difference data) for high-similarity files, adjust storage levels (store the same cluster file set to the same medium), and optimize backup strategies (cluster group shares backup resources).

[0162] 5) Storage and update of clustering results: Store the cluster labels and associated file IDs in the metadata service, and automatically match existing cluster groups when writing or updating files in the future, to achieve dynamic cluster maintenance.

[0163] In summary, the core technical solutions of the optional embodiments of the present application include: distributed data vector feature calculation: for the file fragment mechanism of distributed storage, the vector calculation engine of each storage node calculates the vector features of the local data block in parallel, and then aggregates the fragment vectors into file-level vectors through weighted averaging, time sequence fusion and other strategies. This way avoids the high cost of full data aggregation and improves the efficiency of vector feature calculation; Structured storage and association management of vector features: define the metadata of vector features, including vector dimensions (such as 128 dimensions, 256 dimensions), generation algorithms (such as hash vector, CNN feature extraction), timestamps (generation / update time), associated file ID (bound to the original file), etc.); Vector data is stored in the vector pool in the form of independent object storage (stored separately from the original file, associated through ID) or file attribute expansion (embedded metadata as file expansion attributes), and the bidirectional mapping and batch association with the original file are realized through the associated index, ensuring the effective association of vector features and the original file; Distributed vector index and collaborative query: metadata service integrates IVF, HNSW and other vector index structures, and the index fragments are stored in multiple nodes to achieve load balancing. Metadata and vector index work collaboratively to support real-time updating of the index and multi-condition query (combined with file metadata and vector features), improving file retrieval efficiency; Resource scheduling based on vector features: determine the similarity and content attributes (such as real-time video, picture data) of files through vector features, and classify and manage similar files; Differentiate IO priority and storage medium according to file content attributes to improve resource utilization efficiency; Incremental update mechanism: when a file is partially updated, listen to the file operation log, only recalculate the vector features of the modified fragments, then correct the file-level vector through incremental aggregation, and update the vector index synchronously, avoiding full data reprocessing and reducing maintenance cost; Task scheduling optimization: the task scheduling module performs load-aware scheduling according to the computing power, storage load and index load of the nodes, and divides the task priority to ensure the priority execution of real-time tasks.

[0164] Through the optional embodiments of the present application, the accuracy and efficiency of data management are improved. Automatic clustering and semantic retrieval of files are realized based on vector features, which breaks the dependence on surface information such as file names and directory structures, and the retrieval time is reduced from O(n) (linear time complexity) to O(log n) (logarithmic time complexity), reducing the cost of manual intervention. Users can quickly and accurately find the required files, improving the experience of data management; fine scheduling of resources is realized. Based on the vector features, the content attributes and values of the files are distinguished, higher IO priority and better storage media are allocated to real-time videos and other key tasks, the delay of key tasks is reduced, the utilization efficiency of resources is improved, and the efficient operation of the system is ensured; the dynamic adaptability of the system is enhanced. The incremental update mechanism reduces the maintenance cost of the modified files from being proportional to the file size to being proportional to the modification amount, efficiently supports the management of TB-level large files, and reduces the maintenance difficulty and cost of the system; the scalability requirement of the distributed architecture is adapted. Vector calculation and sharded storage are deeply integrated, parallel processing is performed using local computing power of nodes, the bottleneck of centralized calculation is avoided, the system can be horizontally expanded to a large-scale cluster, and can cope with the growing demand for massive data storage and management.

[0165] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platforms, of course, it can also be realized by hardware, but in many cases, the former is a better implementation.

[0166] The embodiments of the present application also provide a retrieval device, Figure 4 is a structural block diagram of a retrieval device according to an embodiment of the present application, as Figure 4 shown, the device comprises:

[0167] The division module 42 is configured to divide the first file into a plurality of data blocks, and distribute the plurality of data blocks to a plurality of storage nodes;

[0168] The calculation module 44 is configured to calculate a first shard vector corresponding to the data block in each storage node according to a vector generation algorithm, and aggregate a plurality of first shard vectors into a first file vector;

[0169] The establishment module 46 is configured to establish a first association index between each first shard vector and each data block, and establish a second association index between the first file vector and the first file;

[0170] The retrieval module 48 is configured to retrieve the first file based on the first association index and / or the second association index.

[0171] By the above device, by dividing the first file into a plurality of data blocks, and distributing the plurality of data blocks into a plurality of storage nodes; according to the vector generation algorithm, the first file vector is generated by calculating the corresponding first shard vector of the data block in each storage node, and the plurality of first shard vectors are aggregated into the first file vector; the first association index between the first shard vector and the data block is established, and the second association index between the first file vector and the first file is established; the first file is searched based on the first association index and / or the second association index. That is, the present application does not rely on the surface layer to search the file, but searches the first file at the vector level, so that the vector features of the first file can be quickly searched. The problem of low data search efficiency caused by relying on surface layer information for file search in the related art is solved, thereby allowing similarity search based on vectors without viewing the specific content of the first file, and improving the search efficiency.

[0172] In one example embodiment, the division module 42 is further configured to obtain file information of the first file, wherein the file information comprises at least one of the following: file size of the first file, file type of the first file; determine a shard strategy corresponding to the first file according to the file information; and divide the first file into the plurality of data blocks based on the shard strategy.

[0173] In one example embodiment, the calculation module 44 is further configured to determine load information of the each storage node, and determine a master calculation node among the plurality of storage nodes based on the load information; send the first shard vector in each storage node except the master calculation node to the master calculation node, so that the master calculation node aggregates the plurality of first shard vectors according to a preset aggregation algorithm to generate the first file vector.

[0174] In one example embodiment, the establishment module 46 is further configured to obtain a data block identifier corresponding to each data block, and establish a first mapping relationship from the each first shard vector to the each data block identifier; encapsulate the each first shard vector and the each data block identifier into a first entry based on the first mapping relationship, and insert the first entry into a first index data structure corresponding to a vector pool to generate a first index entry, wherein the vector pool is configured to store the each first shard vector; establish a second mapping relationship from the each data block identifier to the each first shard vector; encapsulate the each data block identifier and the each first shard vector into a second entry based on the second mapping relationship, and insert the second entry into a second index data structure corresponding to a data pool to generate a second index entry, wherein the data pool is configured to store the plurality of data blocks; and establish the first association index according to the first index entry and the second index entry.

[0175] In an example embodiment, the establishing module 46 is further configured to obtain a file identifier corresponding to the first file, and establish a third mapping relationship from the first file vector to the file identifier.

[0176] The first file vector and the file identifier are encapsulated into a third entry based on the third mapping relationship, and the third entry is inserted into a third index data structure corresponding to a vector pool to generate a third index entry, where the vector pool is configured to store the first file vector; a fourth mapping relationship from the file identifier to the first file vector is established; the file identifier and the first file vector are encapsulated into a fourth entry based on the fourth mapping relationship, and the fourth entry is inserted into a fourth index data structure corresponding to a data pool to generate a fourth index entry, where the data pool is configured to store the first file; and the second association index is established according to the third index entry and the fourth index entry.

[0177] In an example embodiment, the establishing module 46 is further configured to, in a case where a retrieval request sent by a target object is received, determine a first type of a first retrieval result to be retrieved corresponding to the retrieval request, and obtain a feature vector and a first parameter corresponding to the retrieval request, where the retrieval request is a request for retrieving the first file, the first type includes at least one of a data block type and a file type, and the first parameter is used to indicate a target number of the first retrieval result to be retrieved; determine a target association index according to the first type, and retrieve the first file according to the feature vector and the target association index to obtain the target number of first retrieval results, where the target association index includes at least one of the first association index and the second association index.

[0178] In an example embodiment, the establishing module 46 is further configured to, in a case where it is determined that the first type only includes the data block type, determine the first association index as the target association index; in a case where it is determined that the first type only includes the file type, determine the second association index as the target association index; and in a case where it is determined that the first type includes the data block type and the file type, determine the first association index and the second association index as the target association index.

[0179] In an example embodiment, the establishing module 46 is further configured to, in a case where it is determined that the first type of the first search result to be searched corresponding to the search request comprises a data block type, parse the feature vector to determine a vector dimension of the feature vector; determine a search range corresponding to the search request according to the vector dimension, wherein the search range comprises one or more storage nodes; send the feature vector to each storage node in the search range, so that each storage node in the search range calculates a similarity score between the feature vector and a second shard vector corresponding to each storage node in the search range according to a similarity algorithm, wherein the plurality of first shard vectors comprises the second shard vector; sort a plurality of similarity scores between the second shard vectors corresponding to one or more storage nodes in the search range in descending order, and determine the target number of target shard vectors according to an arrangement order of the plurality of similarity scores; determine a target number of target data blocks associated with the target shard vectors according to the target association index, and determine the target number of target data blocks as the target number of first search results.

[0180] In an example embodiment, the establishing module 46 is further configured to, in a case where it is determined that the first data block in the first file has been updated to a second data block, calculate a third shard vector corresponding to the second data block according to the vector generation algorithm, and aggregate the third shard vector and a fourth shard vector in the plurality of first shard vectors except the first shard vector corresponding to the first data block into a second file vector; establish a third association index between the third shard vector and the second data block, and update the first association index according to the second association index; establish a fourth association index between the second file vector and the updated first file, and update the second association index according to the fourth association index.

[0181] In an example embodiment, the establishing module 46 is further configured to, in a case where it is determined that a third data block is added to the first file, calculate a fifth shard vector corresponding to the third data block according to the vector generation algorithm, and aggregate the fifth shard vector and the plurality of first shard vectors into a third file vector; establish a fifth association index between the fifth shard vector and the third data block, and add the fifth association index to the first association index; establish a sixth association index between the third file vector and the first file after the third data block is added, and update the second association index according to the sixth association index.

[0182] In an example embodiment, the establishing module 46 is further configured to, in a case where it is determined that a fourth data block in the first file has been deleted, aggregate all of the first shard vectors except a sixth shard vector corresponding to the fourth data block into a fourth file vector; delete the association index between the sixth shard vector and the fourth data block in the first association index; establish a seventh association index between the fourth file vector and the first file after the fourth data block is deleted, and update the second association index according to the seventh association index.

[0183] In an example embodiment, the retrieving module 48 is further configured to detect a storage capacity in a vector pool based on a preset period, wherein the vector pool is configured to store target vectors stored in the plurality of storage nodes, and the target vectors at least include the first file vector and the plurality of first shard vectors; in a case where it is determined that the storage capacity is greater than or equal to a preset threshold, determine whether other file vectors except the first file vector are stored in the plurality of storage nodes; in a case where it is determined that the other file vectors are stored in the plurality of storage nodes, perform similar deduplication on a plurality of second files corresponding to a plurality of fifth file vectors, and update the storage capacity of the vector pool according to the plurality of second files after similar deduplication, wherein the plurality of fifth file vectors include the first file vector and the other file vectors stored in the plurality of storage nodes.

[0184] In an example embodiment, the retrieving module 48 is further configured to determine whether the updated storage capacity is greater than or equal to a preset threshold; in a case where it is determined that the updated storage capacity is greater than or equal to the preset threshold, obtain the plurality of first shard vectors stored in the plurality of storage nodes; perform similar deduplication on the plurality of data blocks corresponding to the plurality of first shard vectors, and update the updated storage capacity again according to the plurality of data blocks after similar deduplication.

[0185] In an example embodiment, the retrieving module 48 is further configured to determine a first similarity between each two files in the plurality of second files; in a case where it is determined that a second similarity greater than or equal to a similarity threshold exists in a plurality of first similarities, divide a plurality of third files corresponding to the second similarity into a same cluster group, wherein the plurality of second files include the plurality of third files; and perform similar deduplication on the plurality of third files in the same cluster group.

[0186] In an example embodiment, the retrieval module 48 is further configured to determine a reference file corresponding to the plurality of third files and a difference file corresponding to each third file, and replace the plurality of third files with the reference file and the difference file to perform the similarity deduplication on the plurality of third files, wherein the reference file is a file composed of fifth data blocks with a similarity greater than or equal to a target similarity in the plurality of third files, and the target similarity is greater than the similarity threshold, and the difference file is a file composed of sixth data blocks except the fifth data blocks in each third file.

[0187] In an example embodiment, the retrieval module 48 is further configured to calculate a sixth file vector of the plurality of second files after the similarity deduplication according to the vector generation algorithm, update the fifth file vector in the vector pool according to the sixth file vector, and determine a target storage capacity of the updated vector pool as the updated storage capacity.

[0188] The above-mentioned features of the embodiments of the retrieval device can refer to the related descriptions of the embodiments of the retrieval method, which will not be repeated here.

[0189] The embodiments of the present application also provide an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-mentioned embodiments of the retrieval method.

[0190] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is configured to perform the steps in any of the above-mentioned embodiments of the retrieval method when running.

[0191] In an example embodiment, the above-mentioned computer readable storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0192] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the retrieval method.

[0193] The embodiments of the present application also provide another computer program product, which includes a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the retrieval method.

[0194] Those skilled in the art will further appreciate that the units and algorithms described in connection with the examples disclosed herein can be implemented in electronic hardware, computer software, or both. As described above, the disclosure is capable of being implemented in hardware and software, with the specific implementation depending on the particular application and design constraints imposed on the overall system. Skilled persons can use various methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0195] The above provides a retrieval method, device, storage medium, electronic equipment and computer program product. The principles and implementation modes of the present application are described by applying specific examples. The above description of the examples is only applicable to help understand the method and core idea of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A retrieval method characterized by, The method comprises the following steps: dividing a first file into a plurality of data blocks and distributing the plurality of data blocks to a plurality of storage nodes; calculating a first shard vector corresponding to each data block in each storage node according to a vector generation algorithm, and aggregating a plurality of first shard vectors into a first file vector; establishing a first association index between each first shard vector and each data block, and establishing a second association index between the first file vector and the first file; retrieving the first file based on the first association index and / or the second association index; wherein retrieving the first file based on the first association index and / or the second association index comprises: in the case of receiving a retrieval request sent by a target object, determining a first type of a first retrieval result to be retrieved corresponding to the retrieval request, and obtaining a feature vector and a first parameter corresponding to the retrieval request, wherein the retrieval request is a request for retrieving the first file, the first type includes at least one of the following: data block type, file type, and the first parameter is used to indicate a target number of the first retrieval result to be retrieved; determining a target association index according to the first type, and retrieving the first file according to the feature vector and the target association index to obtain the target number of first retrieval results, wherein the target association index includes at least one of the following: the first association index, the second association index; wherein retrieving the first file according to the feature vector and the target association index to obtain the target number of first retrieval results comprises: in the case of determining that the first type of the first retrieval result to be retrieved corresponding to the retrieval request includes the data block type, parsing the feature vector to determine the vector dimension of the feature vector; determining a retrieval range corresponding to the retrieval request according to the vector dimension, wherein the retrieval range includes one or more storage nodes; sending the feature vector to each storage node in the retrieval range, so that each storage node in the retrieval range calculates a similarity score between the feature vector and a second shard vector corresponding to each storage node in the retrieval range according to a similarity algorithm, wherein the plurality of first shard vectors include the second shard vector; sorting a plurality of similarity scores between the second shard vectors corresponding to one or more storage nodes in the retrieval range in descending order, and determining a target number of target shard vectors according to the arrangement order of the plurality of similarity scores; determining a target number of target data blocks associated with the target shard vectors according to the target association index, and determining the target number of target data blocks as the target number of first retrieval results.

2. The search method of claim 1, wherein, dividing a first file into a plurality of data blocks comprises: obtaining file information of the first file, wherein the file information includes at least one of the following: file size of the first file, file type of the first file; determining a shard strategy corresponding to the first file according to the file information; divide the first file into the plurality of data blocks based on the sharding strategy.

3. The search method of claim 1, wherein, aggregating a plurality of first shard vectors into a first file vector, comprising: determining load information of each storage node, and determining a master computing node among the plurality of storage nodes based on the load information; sending first shard vectors in storage nodes other than the master computing node to the master computing node, so that the master computing node aggregates the plurality of first shard vectors according to a preset aggregation algorithm to generate the first file vector.

4. The search method of claim 1, wherein, establishing a first association index between each first shard vector and each data block, comprising: obtaining a data block identifier corresponding to each data block, and establishing a first mapping relationship from each first shard vector to each data block identifier; based on the first mapping relationship, encapsulating each first shard vector and each data block identifier into a first entry, and inserting the first entry into a first index data structure corresponding to a vector pool, to generate a first index entry, wherein the vector pool is used to store each first shard vector; establishing a second mapping relationship from each data block identifier to each first shard vector; based on the second mapping relationship, encapsulating each data block identifier and each first shard vector into a second entry, and inserting the second entry into a second index data structure corresponding to a data pool, to generate a second index entry, wherein the data pool is used to store the plurality of data blocks; establishing the first association index according to the first index entry and the second index entry.

5. The search method of claim 1, wherein, establishing a second association index between the first file vector and the first file, comprising: obtaining a file identifier corresponding to the first file, and establishing a third mapping relationship from the first file vector to the file identifier; based on the third mapping relationship, encapsulating the first file vector and the file identifier into a third entry, and inserting the third entry into a third index data structure corresponding to a vector pool, to generate a third index entry, wherein the vector pool is used to store the first file vector; establishing a fourth mapping relationship from the file identifier to the first file vector; based on the fourth mapping relationship, encapsulating the file identifier and the first file vector into a fourth entry, and inserting the fourth entry into a fourth index data structure corresponding to a data pool, to generate a fourth index entry, wherein the data pool is used to store the first file; establishing the second association index according to the third index entry and the fourth index entry.

6. The search method of claim 1, wherein, determining a target association index according to the first type, comprising: in a case where it is determined that the first type only includes a data block type, determining the first association index as the target association index; in a case where it is determined that the first type only includes a file type, determining the second association index as the target association index; in a case where it is determined that the first type includes a data block type and a file type, determining the first association index and the second association index as the target association index.

7. The search method of claim 1, wherein, After establishing the first association index between each first shard vector and each data block, and establishing the second association index between the first file vector and the first file, the method further comprises: In a case where it is determined that a first data block in the first file is updated to a second data block, a third shard vector corresponding to the second data block is calculated according to the vector generation algorithm, and the third shard vector and a fourth shard vector in the plurality of first shard vectors except the first shard vector corresponding to the first data block are aggregated into a second file vector; A third association index between the third shard vector and the second data block is established, and the first association index is updated according to the third association index; A fourth association index between the second file vector and the updated first file is established, and the second association index is updated according to the fourth association index.

8. The search method of claim 1, wherein, After establishing the first association index between each first shard vector and each data block, and establishing the second association index between the first file vector and the first file, the method further comprises: In a case where it is determined that a third data block is added to the first file, a fifth shard vector corresponding to the third data block is calculated according to the vector generation algorithm, and the fifth shard vector and the plurality of first shard vectors are aggregated into a third file vector; A fifth association index between the fifth shard vector and the third data block is established, and the fifth association index is added to the first association index; A sixth association index between the third file vector and the first file after adding the third data block is established, and the second association index is updated according to the sixth association index.

9. The search method of claim 1, wherein, After establishing the first association index between each first shard vector and each data block, and establishing the second association index between the first file vector and the first file, the method further comprises: In a case where it is determined that a fourth data block in the first file has been deleted, other shard vectors in the plurality of first shard vectors except a sixth shard vector corresponding to the fourth data block are aggregated into a fourth file vector; An association index between the sixth shard vector and the fourth data block is deleted in the first association index; A seventh association index between the fourth file vector and the first file after deleting the fourth data block is established, and the second association index is updated according to the seventh association index.

10. The search method of claim 1, wherein, After retrieving the first file based on the first association index and / or the second association index, the method further comprises: Detecting a storage capacity in a vector pool based on a preset period, wherein the vector pool is used to store target vectors stored in the plurality of storage nodes, and the target vectors at least include the first file vector and the plurality of first shard vectors; In a case where it is determined that the storage capacity is greater than or equal to a preset threshold, it is determined whether other file vectors except the first file vector are stored in the plurality of storage nodes; In a case where it is determined that the other file vectors have been stored in the plurality of storage nodes, similar deduplication is performed on a plurality of second files corresponding to a plurality of fifth file vectors, and the storage capacity of the vector pool is updated according to the plurality of second files after the similar deduplication, wherein the plurality of fifth file vectors include the first file vectors and the other file vectors stored in the plurality of storage nodes.

11. The search method of claim 10, wherein, After the similar deduplication is performed on the plurality of second files corresponding to the plurality of fifth file vectors, the method further includes: determining whether the updated storage capacity is greater than or equal to a preset threshold; in a case where it is determined that the updated storage capacity is greater than or equal to the preset threshold, obtaining the plurality of first shard vectors stored in the plurality of storage nodes; performing similar deduplication on the plurality of data blocks corresponding to the plurality of first shard vectors, and updating the updated storage capacity again according to the plurality of data blocks after the similar deduplication.

12. The search method of claim 10, wherein, The similar deduplication performed on the plurality of second files corresponding to the plurality of fifth file vectors includes: determining a first similarity between each two files in the plurality of second files; in a case where it is determined that there is a second similarity greater than or equal to a similarity threshold in the plurality of first similarities, dividing a plurality of third files corresponding to the second similarity into a same cluster group, wherein the plurality of second files include the plurality of third files; performing similar deduplication on the plurality of third files in the same cluster group.

13. The search method of claim 12, wherein, The similar deduplication performed on the plurality of third files in the same cluster group includes: determining a reference file corresponding to the plurality of third files and a difference file corresponding to each third file, and replacing the plurality of third files with the reference file and the difference file to perform similar deduplication on the plurality of third files, wherein the reference file is a file composed of fifth data blocks in the plurality of third files with a similarity greater than or equal to a target similarity, and the target similarity is greater than the similarity threshold, and the difference file is a file composed of sixth data blocks in each third file except the fifth data blocks.

14. The search method of claim 10, wherein, The updating of the storage capacity of the vector pool according to the plurality of second files after the similar deduplication includes: calculating a sixth file vector of the plurality of second files after the similar deduplication according to the vector generation algorithm; updating the fifth file vector in the vector pool according to the sixth file vector; determining a target storage capacity of the updated vector pool, and determining the target storage capacity as the updated storage capacity.

15. A retrieval device characterized by The method includes: a division module configured to divide a first file into a plurality of data blocks, and distribute the plurality of data blocks to a plurality of storage nodes; a calculation module configured to calculate a first shard vector corresponding to the data blocks in each storage node according to a vector generation algorithm, and aggregate a plurality of first shard vectors into a first file vector; an establishment module configured to establish a first association index between each first shard vector and each data block, and establish a second association index between the first file vector and the first file; a retrieval module configured to retrieve the first file based on the first association index and / or the second association index. The retrieving the first file based on the first association index and / or the second association index comprises: in a case where a retrieval request sent by a target object is received, determining a first type of a first retrieval result to be retrieved corresponding to the retrieval request, and acquiring a feature vector and a first parameter corresponding to the retrieval request, wherein the retrieval request is a request for retrieving the first file, the first type at least comprises one of a data block type and a file type, and the first parameter is used to indicate a target number of the first retrieval result to be retrieved; determining a target association index according to the first type, and retrieving the first file according to the feature vector and the target association index to acquire the target number of the first retrieval result, wherein the target association index at least comprises one of the first association index and the second association index; wherein the retrieving the first file according to the feature vector and the target association index to acquire the target number of the first retrieval result comprises: in a case where the first type of the first retrieval result to be retrieved corresponding to the retrieval request comprises the data block type, parsing the feature vector to determine a vector dimension of the feature vector; determining a retrieval range corresponding to the retrieval request according to the vector dimension, wherein the retrieval range comprises one or more storage nodes; sending the feature vector to each storage node in the retrieval range, so that each storage node in the retrieval range calculates a similarity score between the feature vector and a second shard vector corresponding to each storage node in the retrieval range according to a similarity algorithm, wherein the plurality of first shard vectors comprises the second shard vector; sorting a plurality of similarity scores between the second shard vectors corresponding to one or more storage nodes in the retrieval range in descending order, and determining a target number of target shard vectors according to an arrangement order of the plurality of similarity scores; determining a target number of target data blocks associated with the target shard vectors according to the target association index, and determining the target number of target data blocks as the target number of the first retrieval result.

16. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the retrieval method according to any one of claims 1 to 14. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the retrieval method according to any one of claims 1 to 14. The computer program is executed by the processor to implement the steps of the retrieval method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, ​ 18. A computer program product comprising a computer program, characterized in that, ​

Images