Retrieval method and apparatus, electronic device, and readable storage medium

Abstract

The present application relates to the technical field of data processing, and discloses a retrieval method and apparatus, an electronic device, and a readable storage medium. The method comprises: on the basis of first feature information of a first object, calculating the similarities between the first feature information and feature information of objects in an N-th layer structure, wherein N is a positive integer; determining a second object having the maximum similarity; on the basis of the N-th layer structure and a connecting line between the N-th layer structure and an (N-1)-th layer structure adjacent thereto, determining a third object in an (N-1)-th layer, wherein the second object and the third object are the same object, and the connecting line is used for connecting the same two objects in any two adjacent layer structures; and on the basis of the third object, determining at least one fourth object similar to the first object.

Description

Retrieval method, device, electronic device and readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 8, 2024, with application number 202410177272.2 and titled “Retrieval method, device, electronic device and readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of data processing technology, and specifically relates to a retrieval method, device, electronic device and readable storage medium. Background Art

[0003] Currently, search scenarios are widely used in electronic devices. For example, a user wants to search for photos of a person doing something in a large number of photos.

[0004] In the prior art, for a large number of objects, some feature tags are extracted for each object, and then used to match the key information input by the user in a retrieval scenario.

[0005] It can be seen that in the prior art, since there are many objects stored and many feature tags, the retrieval process has a large amount of calculation and consumes a lot of power. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a retrieval method that can solve the problem in the prior art that the retrieval process has a large amount of computation and power consumption due to the large number of stored objects and feature tags.

[0007] In a first aspect, an embodiment of the present application provides a retrieval method, the method comprising: calculating, based on first feature information of a first object, the similarity between the first feature information and the feature information of an object in an N-th layer structure; wherein N is a positive integer; determining a second object with the greatest similarity; determining a third object in the N-1 layer based on the N-th layer structure and a connecting line between an N-1 layer structure adjacent to the N-th layer structure; wherein the second object and the third object are the same object, and the connecting line is used to connect two identical objects in any two adjacent layer structures; and determining, based on the third object, at least one fourth object similar to the first object.

[0008] In a second aspect, an embodiment of the present application provides a retrieval device, which includes: a calculation module for calculating the similarity between the first feature information of the first object and the feature information of the object in the N-th layer structure based on the first feature information of the first object; wherein N is a positive integer; a first determination module for determining the second object with the greatest similarity; a second determination module for determining a third object in the N-1 layer based on the N-th layer structure and the connecting line between the N-1 layer structure adjacent to the N-th layer structure; wherein the second object and the third object are the same object, and the connecting line is used to connect two identical objects in any two adjacent layer structures; a third determination module for determining at least one fourth object similar to the first object based on the third object.

[0009] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps described in the first aspect are implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0012] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0013] In an embodiment of the present application, a retrieval method based on a multi-layer structure is proposed. When a user inputs a first object, the first feature information of the first object is used to query the object corresponding to the feature information most similar to the first feature information, starting from a higher-level structure. Then, based on the found object, the connection line connecting the same object with the lower-level structure is used. For example, during the query process, the second object and the third object at both ends of the connection line between the two-level structure are the same object, so that at least one fourth object similar to the first object can be determined in the lower-level structure. It can be seen that based on the embodiment of the present application, by searching layer by layer, it is helpful to quickly lock the target object, and then further search is performed based on the target object. The retrieval process is progressive, without the need for blind search of a large number of objects, effectively shortening the retrieval time and improving the retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a flowchart of a search method according to an embodiment of the present invention;

[0015] FIG2 is a schematic diagram illustrating a search method according to an embodiment of the present application;

[0016] FIG3 is a second schematic diagram illustrating the retrieval method according to an embodiment of the present application;

[0017] FIG4 is a third schematic diagram illustrating the retrieval method according to an embodiment of the present application;

[0018] FIG5 is a fourth schematic diagram illustrating the retrieval method according to an embodiment of the present application;

[0019] FIG6 is a second flowchart of the retrieval method according to an embodiment of the present application;

[0020] FIG7 is a third flowchart of the retrieval method according to an embodiment of the present application;

[0021] FIG8 is a block diagram of a retrieval device according to an embodiment of the present application;

[0022] FIG9 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application;

[0023] FIG10 is a second schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application. Specific embodiments

[0024] The following will be combined with the accompanying drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0025] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0026] The following describes the search method provided in the embodiment of the present application in detail through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0027] As shown in FIG1 , a flowchart of a search method according to an embodiment of the present application is shown. Taking the method applied to an electronic device as an example, the method includes:

[0028] Step 110: Calculate the similarity between the first feature information of the first object and the feature information of the objects in the Nth layer structure based on the first feature information of the first object, where N is a positive integer.

[0029] In this embodiment, before performing a search, a search library is first created based on a large number of objects. The large number of objects can be a large number of pictures, a large number of objects can also be a large number of texts, a large number of objects can also be a large number of videos, and a large number of objects can also be a large number of chemical structures.

[0030] For reference, a large number of objects are input into a vectorized model. After processing, the model generates a vector for each object, which is used to represent its feature information. For example, the model output is id1 = [0.01, 0.23, -0.399, 0.2203, ...], id2 = [-0.932, 1.023, 3.02, 0.002, ...]. Where id1 and id2 represent the identity information of the object and are integers, and [0.01, 0.23, -0.399, 0.2203, ...] represents the object vector. The length of each object vector is the same, meaning that the dimension is fixed. The dimension is determined by the model and is typically hundreds or thousands of dimensions, for example, 2048 dimensions can be used.

[0031] In one example, when creating a retrieval library based on all the pictures in an album, first label each picture in the album with a unique id, starting from the first picture, and label them as id1, id2, id3, id4..., and then obtain [id, vector] through the model, where vector represents a vector, and then store all [id, vector] in the retrieval library.

[0032] Among them, the vectorization model can be a multimodal model to perform vector conversion on objects of different forms such as pictures and texts; or different models can be used for pictures, texts, etc. to perform vector conversion on objects of different forms respectively. Different models need to be associated during the training process so that the output results can be used to create the retrieval library of this embodiment.

[0033] Optionally, the first object is content input by the user, which can be text or an image. Correspondingly, the first feature information can be a query vector obtained by processing the first object through the model, and the query vector has the same dimension as the vector in the search library.

[0034] In this embodiment, the constructed search library is a multi-layered structure, including at least the Nth layer and the N-1th layer, with the Nth layer adjacent to the N-1th layer and the Nth layer above the N-1th layer. In this step, the similarity between the query vector and the vectors in the Nth layer is calculated, taking the Nth layer as an example.

[0035] Step 110 is a repeated step, and each time it is repeated, the value of N is different. For example, N takes the values ​​5, 4, 3, 2, and 1 in sequence, so that similarity is calculated in the 5th structure, the 4th structure, the 3rd structure, the 2nd structure, and the 1st structure in sequence.

[0036] Step 120: Determine the second object with the greatest similarity.

[0037] In this step, in the Nth layer structure, the vector that is most similar to the query vector is found, and this vector is the vector of the second object.

[0038] Step 130: Determine a third object in the N-1 layer based on the N-1 layer structure and the connecting line between the N-1 layer structure and the N-1 layer structure adjacent to the N-1 layer structure. The second object and the third object are identical objects, and the connecting line is used to connect two identical objects in any two adjacent layers of the structure.

[0039] In this embodiment, the algorithm used is the Hierarchical Navigable Small World (HNSW) algorithm. Specifically, if the query vector is similar to vector A in the search database, the algorithm further matches other vectors close to vector A to find even more similar vectors. Therefore, the search must be spread outward from vector A to complete the search. This embodiment utilizes a multi-layered structure to quickly locate vector A. Starting with the highest level, the search continues using the connections between different levels to find the most similar vector in the lowest level, which serves as vector A.

[0040] Therefore, in this step, after finding the second object, we can find the third object in the next layer of the structure, and then repeat step 110 with the third object as the center to find the vector in the next layer of the structure that is most similar to the query vector. Correspondingly, steps 120 and 130 are also repeated steps.

[0041] Step 140: Determine at least one fourth object similar to the first object based on the third object.

[0042] In this step, the at least one fourth object similar to the first object may be an object in the lowest level structure.

[0043] The lowest level structure is level 0. That is, when N is 1, the vector A that is most similar to the query vector can be found in the level 0 structure. Then, the HNSW algorithm is used to find at least one vector similar to the query vector. The found vector is the vector of the fourth object.

[0044] An application scenario of this embodiment is, for example, that a user inputs the text "a lip balm placed on the table", and a picture closest to the user's description of "a lip balm placed on the table" is found in the search library.

[0045] Another example of an application scenario of this embodiment is that a user enters the text "a fountain pen placed on a table," and at least one image similar to the user's description of "a fountain pen placed on a table" is found in the search database. The found images may include an image of a fountain pen and an image of a fountain pen. Based on the similarity comparison, the fountain pen image is displayed first.

[0046] In an embodiment of the present application, a retrieval method based on a multi-layer structure is proposed. When a user inputs a first object, the first feature information of the first object is used to query the object corresponding to the feature information most similar to the first feature information, starting from a higher-level structure. Then, based on the found object, the connection line connecting the same object with the lower-level structure is used. For example, during the query process, the second object and the third object at both ends of the connection line between the two-level structure are the same object, so that at least one fourth object similar to the first object can be determined in the lower-level structure. It can be seen that based on the embodiment of the present application, by searching layer by layer, it is helpful to quickly lock the target object, and then further search is performed based on the target object. The retrieval process is progressive, without the need for blind search of a large number of objects, effectively shortening the retrieval time and improving the retrieval performance.

[0047] In the process of the search method of another embodiment of the present application, step 110 includes:

[0048] Sub-step A1: Calculating the similarity between the first feature information and the feature information of the fifth object in the N-th layer structure, and calculating the similarity between the first feature information and the feature information of the neighboring objects of the fifth object.

[0049] When N is equal to the first preset value, the fifth object is a preset object in the Nth layer structure; when N is less than the first preset value, the fifth object is the second object with the greatest similarity determined based on the N+1th layer structure.

[0050] The first preset value is used to limit the highest level structure of the search library. For example, if the first preset value is 5, the highest level structure is the 5th level structure, and the following levels are: the 4th level structure, the 3rd level structure, the 2nd level structure, the 1st level structure, and the 0th level structure.

[0051] In this embodiment, when calculating similarity in the highest-level structure, the fifth object is a specified object that can be set in advance, that is, the starting position of the retrieval is fixed; when calculating similarity in a non-highest-level structure, the fifth object is the object corresponding to the vector most similar to the query vector found in the adjacent higher-level structure.

[0052] For reference, taking the retrieval process of the highest-level structure as an example, a first-in-first-out queue is created to place the id, which is initially empty; a visited table is created to record whether the object pointed to by the id has been calculated, with an initial value of false, indicating that it has not been visited; a most similar marker is created, denoted as max_similarity, to record the maximum similarity, with the initial value set to a 32-bit float to ensure that it is large enough to accommodate the minimum similarity; a 32-bit integer space is created, denoted as max_id, to record the id of the most similar object. Next, the entry point is read, which is the starting position in the highest-level structure. Assuming the entry point id is i0, i0 is placed in the queue, and the i0 position in the visited table is marked as true, indicating that it has been visited. Among them, as long as it is placed in the queue, it means it has been visited. Then, the first object in the queue is dequeued. This time, object i0 is dequeued. The similarity between i0's vector and the query vector is calculated, resulting in a similarity score (score). If score is greater than max_similarity, max_similarity is updated to score, and i0 is recorded in max_id. Furthermore, neighboring objects of i0 are obtained and the similarity between the neighboring object's vector and the query vector is repeatedly calculated to obtain a similarity score (score). If score is greater than max_similarity, max_similarity is updated to score, and the neighboring object's ID is recorded in max_id. This continues until all neighboring objects are calculated, meaning the queue is empty.

[0053] The calculation of adjacent objects is completed until the queue is empty.

[0054] Optionally, a neighbors array is used to record adjacent objects. In the multi-layer structure of the present application, an object may appear in multiple layers, so its adjacent objects in different layers will all be recorded in the neighbors array.

[0055] As shown in Figure 2, for any object, its neighboring objects (represented as neighbors in the figure) at layer 0 (L0) include id_a1, id_a2, ..., id_an, and its neighboring objects at layer 1 (L1) include id_b1, id_b2, ..., id_bn. Therefore, the neighboring objects of any object are a continuous segment of data in the neighbors array.

[0056] Furthermore, as shown in Figure 3, the neighbors array includes all neighboring objects of all objects. For example, the neighbors array includes the L0-level neighbors (neighboring nodes in the diagram) of the first object (node ​​0 in the diagram), the L1-level neighbors of the first object, the L0-level neighbors of the second object (node ​​1 in the diagram), and the L1-level neighbors of the second object. Additional offset information is used to locate the position of neighboring objects in the entire array. The offset information is shown in Table 1, and the neighboring objects are represented by the two-dimensional array represented in Table 1. For example, the first neighbor of object id in the Ln-level structure is represented as p = offset[id][L(n)], the last neighbor is represented as q = offset[id][L(n+1)] – 1, and the remaining neighbors are: neighbors[p], neighbors[p+1], neighbors[p+2], …, neighbors[q]. Specifically, the neighbors of object i3 in the L1-level are represented starting from the 580th number and ending at the 592nd number.

[0057] Table 1

[0058] Optionally, based on Table 1, when loading vectors of adjacent objects for similarity calculation, the offset is pre-calculated, and then the file handle jumps directly to the corresponding offset for loading. Furthermore, this embodiment proposes a vector file storage rule that specifies what each byte stores. During retrieval, only bytes at the specified location are read. This significantly reduces file reading and writing, significantly reduces memory usage, and supports segmented parallel processing for retrieval, making it more suitable for running on electronic devices without significantly increasing their power consumption.

[0059] For example, when storing a vector, the header information occupies 16 bytes. Assuming the vector dimension is d, the offset of the entry point i0 is: 16+(4+4*d)*i0+4. After the pointer jumps to this position, loading 4*d bytes is the vector of i0. Among them, i0 not only represents the identity information of the object, but also the number of objects arranged before i0. One dimension occupies 4 bytes, the id occupies 4 bytes, 4+4*d represents the bytes occupied by the id and vector of an object, (4+4*d)*i0 represents the bytes occupied by the id and vector of all objects before i0, and the last +4 represents the bytes occupied by the id i0. Furthermore, for adjacent objects, after finding the corresponding number in Table 1, the adjacent object is found, and then the id of the adjacent object needs to be obtained. Then the offset is obtained using the above formula, and finally the vector of the adjacent object is obtained.

[0060] During the entire retrieval process, vectors are loaded into memory on demand and released after use. As can be seen from the above content, the process of loading a vector is very easy. You only need to calculate the offset according to the rule settings and directly obtain the corresponding byte string.

[0061] The above process is illustrated using the entry point as an example. In more cases, after calculating the maximum similarity in a non-lowest level structure, the corresponding object is determined, and then the same object is found in a lower level structure along the connecting line between the layer structures, and the calculation starts with this object as the fifth object.

[0062] In this embodiment, the multi-layer structure of the retrieval library is utilized to find objects corresponding to the feature information most similar to the first feature information in each layer in descending order, and the amount of calculation for each layer is much smaller than that for a large number of objects, thereby improving the calculation speed and reducing the power consumed by the retrieval process.

[0063] In the process of the search method of another embodiment of the present application, before step 110, the method further includes:

[0064] Step B1: In any layer structure, any two objects whose feature information similarity is greater than a first threshold are connected as two adjacent objects.

[0065] For all objects in a layer structure, the similarity between the feature information of every two objects is calculated separately.

[0066] Alternatively, the similarity calculation between the feature information of two objects is actually the similarity calculation between two vectors. The final similarity is represented by a score, and the size of the score directly determines whether the two vectors are similar. The score can be the cosine value or the Euclidean distance between the two vectors. Taking the cosine value as an example, the larger the cosine value, the more similar it is. The maximum score is 1, which means they are exactly the same. The cosine value reflects the size of the angle between the two vectors. Its formula (1) is as follows:

[0067] In formula (1), A and B represent two objects to be compared.

[0068] For example, id1 = [0.3, 0.22, 0.02], id2 = [0.34, 0.445, 0.04], id3 = [0.74, 0.44, 0.84], and the cosine values ​​between each pair are calculated: cos(id1, id2) = 0.959, cos(id1, d3) = 0.847, cos(id2, id3) = 0.888. Therefore, the similarity between id1 and id2 is the highest.

[0069] Furthermore, two objects whose similarity is greater than a first threshold are connected, wherein the first threshold is used to define that the similarity between the two objects is relatively high.

[0070] In this step, the line connecting any two objects is defined as a neighbor edge.

[0071] Optionally, this embodiment further includes:

[0072] Step B2: When any two objects whose feature information has a similarity greater than a first threshold value are connected, any two objects are randomly connected as two adjacent objects.

[0073] In this step, any two objects are connected randomly. Dissimilar objects are randomly connected to avoid falling into local optimality during retrieval. Each object will have at least one random edge connected to other objects. Therefore, the line connecting any two objects in this step is defined as a random edge.

[0074] Step B3: Connect any two identical objects in any two adjacent layers of the structure using connecting lines.

[0075] In this step, if an object in a certain layer structure is sampled into an adjacent higher layer structure, the object will appear in both layers simultaneously, thereby connecting the same two objects in the two layers. The line connecting the same objects is defined as a cross-layer edge.

[0076] For reference, as shown in Figure 4, when creating a search library, the lowest level (L0) is first created. L0 contains the vectors of all objects, and pairwise similarities are calculated. For any object, the objects with higher similarity are selected as adjacent objects, connected using neighbor edges. Then, objects are randomly sampled from L0 to create L1, and objects are randomly sampled from L1 to create L2. This process can be repeated until the highest level is created. The number of layers can be preset.

[0077] Among them, for any object, it has the following properties: it will definitely appear in the L0 layer; if it appears in the Ln layer, it will appear in any lower layer Li (0≤i≤n).

[0078] The objects in the N-1th layer structure include the objects in the Nth layer structure.

[0079] Optionally, the first thresholds in different layer structures may be different, so as to limit the number of similar objects connected to objects in different layer structures.

[0080] Optionally, in the retrieval library, nodes are used to represent objects, and the nodes indicate feature information and identity information of the objects.

[0081] In this embodiment, the search library is a multi-layered structure. For one layer, for example, layer N-1, all objects in layer N-1 are randomly sampled to obtain a subset of objects, which then form layer N. Objects in adjacent layers are connected by connecting lines, and similar objects in the same layer are connected as neighbors. Thus, the search process begins at the highest layer and moves down along the connecting lines until a specific object is located in the lowest layer, where the search continues. This eliminates the need to blindly search across a large number of objects, making the computation simpler and faster.

[0082] In the retrieval method of another embodiment of the present application, the number of objects whose similarity between any objects in the 0th layer structure and the connected feature information is greater than the first threshold is greater than the number of objects whose similarity between any objects in the Nth layer structure and the connected feature information is greater than the first threshold.

[0083] In this embodiment, in the 0th layer structure, the number of similar objects connected by the object is less than or equal to M1, and in other layer structures, the number of similar objects connected by the object is less than or equal to M2, M1 and M2 are positive integers, and M1>M2.

[0084] For example, M1=2S, M2=S, S=32.

[0085] For reference, when creating a search library, for any object in the 0th layer structure, connect to the M1 objects with the highest similarity. Correspondingly, the first threshold must be met for any object to be connected to the M1 objects with the highest similarity. For any object in a non-0th layer structure, connect to the M2 objects with the highest similarity. Correspondingly, the first threshold must be met for any object to be connected to the M2 objects with the highest similarity.

[0086] In this embodiment, the 0th level structure contains all object vectors. In this level, searches are performed based on a specific object vector, without traversing the object vectors of the entire level structure. Therefore, objects in this level structure need to be connected to a larger number of objects to ensure the number of object vectors to be calculated and avoid missing some highly similar object vectors. However, the other levels of the structure only contain a portion of the object vectors, and their more important function is to find the calculation location in the lower levels of the structure. Therefore, objects do not need to be connected to a larger number of objects, ensuring a smaller amount of calculation.

[0087] In the process of the search method of another embodiment of the present application, step 140 includes:

[0088] Sub-step C1: when the N-1th layer structure is the 0th layer structure, calculating the similarity between the first feature information and the feature information of the third object, and calculating the similarity between the first feature information and the feature information of the neighboring objects of the third object.

[0089] Sub-step C2: Determine the sixth object with the greatest similarity.

[0090] Sub-step C3: calculating the similarity between the first feature information and the feature information of the sixth object, and calculating the similarity between the first feature information and the feature information of an object adjacent to the sixth object.

[0091] Sub-step C4: Determine the seventh object with the greatest similarity.

[0092] Sub-step C5: Based on the seventh object, determine in the 0th level structure a fourth object whose similarity satisfies the first condition.

[0093] In this embodiment, after the third object is determined in the 0th layer structure, the starting position of the search is determined. Based on the starting position, it is necessary to expand outward several times to complete the search.

[0094] For example, M represents diffusion, and its initial value is 0. First, when calculating the similarity between the third object and the adjacent objects of the third object and the first feature information of the first object, the value of M is increased by 1 and recorded as 1; secondly, when obtaining the sixth object with the greatest similarity, the similarity between the sixth object and the adjacent objects of the sixth object and the first feature information of the first object is calculated, the value of M is increased by 1 and recorded as 2; then, when obtaining the seventh object with the greatest similarity, the similarity between the seventh object and the adjacent objects of the seventh object and the first feature information of the first object is calculated, the value of M is increased by 1 and recorded as 3. And so on. After each round of calculation, the value of M is increased by 1. When the value of M is the second preset value, the object whose similarity meets the first condition in the 0th layer structure is determined to be the fourth object.

[0095] The second preset value is represented as efSearch.

[0096] For example, if efSearch=4, when the value of M is increased by 1 and updated to 4, no further calculation is required after the current calculation is completed.

[0097] Optionally, objects corresponding to the first few largest similarities among all similarities calculated in the 0th layer structure are used as the fourth objects. Correspondingly, the first condition is used to limit the similarity to be high, so as to further limit the number of fourth objects.

[0098] In this embodiment, a layer-by-layer approach is used to ultimately reach the lowest level, which contains all object vectors. Because the higher-level structure has already quickly locked onto an object, the algorithm simply expands outward from that object to find similar objects as the final output object. This demonstrates that this embodiment employs multiple rounds of computation at the lowest level to ensure a wide coverage of comparison objects.

[0099] In another embodiment of the search method, as shown in Figure 5 , the layer structure is stored separately from the object data. Therefore, before searching, the first file loaded is the graph file for the upper half of the structure shown in Figure 5 , which does not include the object data and is therefore smaller. Specifically, it contains the entry point, total number of layers, S,efSearch, neighbors, and offset.

[0100] As shown in Figure 5, the lower half of the vector file contains object data, divided into header information and data information. The header information includes the vector dimensions, the total number of vectors, and an auto-incrementing ID. The main body mainly stores the vectors and IDs, where x_1, x_2, …, x_d represent each number in the vector. Vectors are loaded on demand during retrieval.

[0101] The application scenarios of the embodiments of this application include: image retrieval and text retrieval.

[0102] As shown in Figure 6, in the image retrieval scenario, the images in the album are named and converted into vectors and stored in the retrieval library in the form of nodes; the query text is then converted into vectors for retrieval in the retrieval library, and finally each image is output according to the image ID.

[0103] As shown in Figure 7, in the text retrieval scenario, the original article is cut into multiple fragments, and the multiple fragments are converted into vectors and stored in the retrieval library in the form of nodes; the query text is then converted into a vector to be searched in the retrieval library, and finally each fragment is output according to the fragment ID.

[0104] In summary, this application proposes a retrieval method based on a multi-layered graph constructed from a large number of objects. This method can reduce memory usage and power consumption during the retrieval process, thereby improving retrieval performance. A multi-layered structure is constructed for object data, and the object data and structure are processed separately. On the one hand, the object data is stored according to a rule, and on the other hand, the multi-layered structure is stored separately. During the retrieval process, only the structure and a small amount of object data within it need to be loaded separately, thereby achieving the goal of consuming less memory and CPU resources during the retrieval process.

[0105] The retrieval method provided in the embodiment of the present application can be executed by a retrieval device. In the embodiment of the present application, the retrieval device provided in the embodiment of the present application is described by taking the retrieval device executing the retrieval method as an example.

[0106] FIG8 shows a block diagram of a retrieval device according to an embodiment of the present application, the device comprising:

[0107] A calculation module 10 is configured to calculate, based on the first feature information of the first object, a similarity between the first feature information and feature information of objects in the Nth layer structure; wherein N is a positive integer;

[0108] A first determining module 20 is configured to determine a second object with the greatest similarity;

[0109] a second determining module 30 configured to determine a third object in the N-1 layer based on the N-1 layer structure and a connecting line between the N-1 layer structure adjacent to the N-1 layer structure; wherein the second object and the third object are identical objects, and the connecting line is used to connect two identical objects in any two adjacent layers of the structure;

[0110] The third determining module 40 is configured to determine, based on the third object, at least one fourth object similar to the first object.

[0111] In an embodiment of the present application, a retrieval method based on a multi-layer structure is proposed. When a user inputs a first object, the first feature information of the first object is used to query the object corresponding to the feature information most similar to the first feature information, starting from a higher-level structure. Then, based on the found object, the connection line connecting the same object with the lower-level structure is used. For example, during the query process, the second object and the third object at both ends of the connection line between the two-level structure are the same object, so that at least one fourth object similar to the first object can be determined in the lower-level structure. It can be seen that based on the embodiment of the present application, by searching layer by layer, it is helpful to quickly lock the target object, and then further search is performed based on the target object. The retrieval process is progressive, without the need for blind search of a large number of objects, effectively shortening the retrieval time and improving the retrieval performance.

[0112] Optionally, the calculation module 10 includes:

[0113] a first calculating unit, configured to calculate a similarity between the first feature information and feature information of a fifth object in the Nth layer structure, and to calculate a similarity between the first feature information and feature information of an object adjacent to the fifth object;

[0114] Wherein, when N is equal to a first preset value, the fifth object is a preset object in the Nth layer structure;

[0115] When N is less than a first preset value, the fifth object is the second object with the greatest similarity determined based on the N+1th layer structure.

[0116] Optionally, the device further comprises:

[0117] A first connection module is configured to connect any two objects whose feature information has a similarity greater than a first threshold value in any layer structure as two adjacent objects;

[0118] The second connection module is used to connect two identical objects in any two adjacent layers of the structure using a connection line;

[0119] The objects in the N-1th layer structure include the objects in the Nth layer structure.

[0120] Optionally, the number of objects whose similarity between any objects and connected feature information in the 0th layer structure is greater than the first threshold is greater than the number of objects whose similarity between any objects and connected feature information in the Nth layer structure is greater than the first threshold.

[0121] Optionally, the third determining module 40 includes:

[0122] a second calculating unit, configured to calculate, when the N-1th layer structure is the 0th layer structure, the similarity between the first feature information and the feature information of the third object, and the similarity between the first feature information and the feature information of an object adjacent to the third object;

[0123] A first determining unit, configured to determine a sixth object having the greatest similarity;

[0124] a third calculating unit, configured to calculate the similarity between the first feature information and the feature information of the sixth object, and to calculate the similarity between the first feature information and the feature information of an object adjacent to the sixth object;

[0125] a second determining unit, configured to determine a seventh object having the greatest similarity;

[0126] The third determining unit is configured to determine, based on the seventh object, a fourth object in the 0th layer structure whose similarity satisfies a first condition.

[0127] The device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0128] The device of the embodiment of the present application may be a device having an action system. The action system may be an Android action system, an iOS action system, or other possible action systems, which are not specifically limited in the embodiment of the present application.

[0129] The device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0130] Optionally, as shown in Figure 9, an embodiment of the present application also provides an electronic device 100, including a processor 101, a memory 102, and a program or instruction stored in the memory 102 and executable on the processor 101. When the program or instruction is executed by the processor 101, the various steps of any of the above-mentioned retrieval method embodiments are implemented and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0131] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0132] FIG10 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0133] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, a processor 1010, a camera 1011 and other components.

[0134] Those skilled in the art will appreciate that the electronic device 1000 may further include a power source (such as a battery) to power various components. The power source may be logically connected to the processor 1010 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG10 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or arrange the components differently, which will not be described in detail here.

[0135] Among them, the processor 1010 is used to calculate the similarity between the first feature information of the first object and the feature information of the object in the N-th layer structure based on the first feature information of the first object; wherein N is a positive integer; determine the second object with the greatest similarity; determine the third object in the N-1 layer based on the N-th layer structure and the connecting line between the N-1 layer structure adjacent to the N-th layer structure; wherein the second object and the third object are the same object, and the connecting line is used to connect the two same objects in any adjacent two-layer structures; and determine at least one fourth object similar to the first object based on the third object.

[0136] In an embodiment of the present application, a retrieval method based on a multi-layer structure is proposed. When a user inputs a first object, the first feature information of the first object is used to query the object corresponding to the feature information most similar to the first feature information, starting from a higher-level structure. Then, based on the found object, the connection line connecting the same object with the lower-level structure is used. For example, during the query process, the second object and the third object at both ends of the connection line between the two-level structure are the same object, so that at least one fourth object similar to the first object can be determined in the lower-level structure. It can be seen that based on the embodiment of the present application, by searching layer by layer, it is helpful to quickly lock the target object, and then further search is performed based on the target object. The retrieval process is progressive, without the need for blind search of a large number of objects, effectively shortening the retrieval time and improving the retrieval performance.

[0137] Optionally, the processor 1010 is further used to calculate the similarity between the first feature information and the feature information of the fifth object in the Nth layer structure, and to calculate the similarity between the first feature information and the feature information of the adjacent objects of the fifth object; wherein, when N is equal to a first preset value, the fifth object is a preset object in the Nth layer structure; when N is less than the first preset value, the fifth object is the second object with the greatest similarity determined based on the N+1th layer structure.

[0138] Optionally, the processor 1010 is further used to connect any two objects in any layer structure whose feature information similarity is greater than a first threshold as two adjacent objects; and to connect any two identical objects in any two adjacent layer structures using a connecting line; wherein, the objects in the N-1 layer structure include the objects in the N layer structure.

[0139] Optionally, the number of objects whose similarity between feature information of any objects or connections in the 0th layer structure is greater than the first threshold is greater than the number of objects whose similarity between feature information of any objects or connections in the Nth layer structure is greater than the first threshold.

[0140] Optionally, the processor 1010 is further used to, when the N-1th layer structure is the 0th layer structure, calculate the similarity between the first feature information and the feature information of the third object, and calculate the similarity between the first feature information and the feature information of the adjacent objects of the third object; determine the sixth object with the greatest similarity; calculate the similarity between the first feature information and the feature information of the sixth object, and calculate the similarity between the first feature information and the feature information of the adjacent objects of the sixth object; determine the seventh object with the greatest similarity; and based on the seventh object, determine a fourth object in the 0th layer structure whose similarity meets the first condition.

[0141] In summary, this application proposes a retrieval method based on a multi-layered graph constructed from a large number of objects. This method can reduce memory usage and power consumption during the retrieval process, thereby improving retrieval performance. A multi-layered structure is constructed for object data, and the object data and structure are processed separately. On the one hand, the object data is stored according to a rule, and on the other hand, the multi-layered structure is stored separately. During the retrieval process, only the structure and a small amount of object data within it need to be loaded separately, thereby achieving the goal of consuming less memory and CPU resources during the retrieval process.

[0142] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes image data of a static picture or video image obtained by an image capture device (such as a camera) in a video image capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of the other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an action stick, which will not be repeated here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and action systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the action system, user pages and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1010.

[0143] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0144] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0145] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned retrieval method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0146] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0147] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned retrieval method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0148] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0149] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned retrieval method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0150] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0152] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A retrieval method, comprising: Calculating, based on first feature information of a first object, a similarity between the first feature information and feature information of an object in an N-th layer of the structure; wherein N is a positive integer; determining a second object with the greatest similarity; Determining a third object in the N-1 layer based on the N-1 layer structure and a connecting line between the N-1 layer structure adjacent to the N-1 layer structure; wherein the second object and the third object are the same object, and the connecting line is used to connect two identical objects in any two adjacent layers of structures; At least one fourth object similar to the first object is determined based on the third object.

2. The method according to claim 1, wherein The calculating, based on the first feature information of the first object, the similarity between the first feature information and the feature information of the objects in the N-th layer structure includes: Calculating the similarity between the first feature information and feature information of a fifth object in the Nth layer structure, and calculating the similarity between the first feature information and feature information of an object adjacent to the fifth object; Wherein, when N is equal to a first preset value, the fifth object is a preset object in the Nth layer structure; When N is less than the first preset value, the fifth object is the second object with the greatest similarity determined based on the N+1th layer structure.

3. The method according to claim 1, wherein Before calculating, based on the first feature information of the first object, the similarity between the first feature information and the feature information of the objects in the N-th layer structure, the method further includes: In any layer structure, any two objects whose feature information similarity is greater than a first threshold are connected as two adjacent objects; Connect any two identical objects in any adjacent two-layer structure with a connecting line; The objects in the N-1th layer structure include the objects in the Nth layer structure.

4. The search method according to claim 3, wherein: The number of objects whose similarity between any objects and connected feature information in the 0th layer structure is greater than the first threshold is greater than the number of objects whose similarity between any objects and connected feature information in the Nth layer structure is greater than the first threshold.

5. The search method according to claim 1, wherein: The determining, based on the third object, at least one fourth object similar to the first object includes: When the N-1th layer structure is the 0th layer structure, calculating the similarity between the first feature information and the feature information of the third object, and calculating the similarity between the first feature information and the feature information of an object adjacent to the third object; Determine the sixth object with the greatest similarity; Calculating the similarity between the first feature information and the feature information of the sixth object, and calculating the similarity between the first feature information and the feature information of an object adjacent to the sixth object; Determine the seventh object with the greatest similarity; According to the seventh object, a fourth object whose similarity satisfies a first condition is determined in the 0th layer structure.

6. A retrieval device, comprising: a calculation module, configured to calculate, based on first feature information of a first object, a similarity between the first feature information and feature information of an object in an N-th layer of the structure; wherein N is a positive integer; A first determining module, configured to determine a second object with the greatest similarity; a second determining module, configured to determine a third object in the N-1 layer based on the N-1 layer structure and a connecting line between the N-1 layer structure adjacent to the N-1 layer structure; wherein the second object and the third object are the same object, and the connecting line is used to connect two identical objects in any two adjacent layers of structures; The third determining module is configured to determine, based on the third object, at least one fourth object similar to the first object.

7. The device according to claim 6, wherein The computing module includes: a first calculating unit, configured to calculate a similarity between the first feature information and feature information of a fifth object in the Nth layer structure, and calculate a similarity between the first feature information and feature information of an object adjacent to the fifth object; Wherein, when N is equal to a first preset value, the fifth object is a preset object in the Nth layer structure; When N is less than the first preset value, the fifth object is the second object with the greatest similarity determined based on the N+1th layer structure.

8. The device according to claim 6, wherein The device further comprises: A first connection module is configured to connect any two objects whose feature information has a similarity greater than a first threshold value in any layer structure as two adjacent objects; The second connection module is used to connect two identical objects in any two adjacent layers of the structure using a connection line; The objects in the N-1th layer structure include the objects in the Nth layer structure.

9. The search device according to claim 8, wherein: The number of objects whose similarity between any objects and connected feature information in the 0th layer structure is greater than the first threshold is greater than the number of objects whose similarity between any objects and connected feature information in the Nth layer structure is greater than the first threshold.

10. The search device according to claim 6, wherein: The third determining module includes: a second calculating unit, configured to calculate, when the N-1th layer structure is a 0th layer structure, a similarity between the first feature information and the feature information of the third object, and a similarity between the first feature information and the feature information of an object adjacent to the third object; A first determining unit, configured to determine a sixth object having the greatest similarity; a third calculating unit, configured to calculate a similarity between the first feature information and the feature information of the sixth object, and to calculate a similarity between the first feature information and feature information of an object adjacent to the sixth object; a second determining unit, configured to determine a seventh object having the greatest similarity; The third determining unit is configured to determine, based on the seventh object, a fourth object in the 0th layer structure whose similarity satisfies a first condition.

11. An electronic device comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the retrieval method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the retrieval method according to any one of claims 1 to 5.

13. A chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the retrieval method according to any one of claims 1 to 5.

14. A computer program product, wherein the program product is stored in a storage medium and is executed by at least one frame processor to implement the steps of the search method according to any one of claims 1 to 5.

15. An electronic device, characterized in that: The electronic device is configured to execute the steps of the search method according to any one of claims 1 to 5.

Images