Data retrieval method and apparatus, data processing method and apparatus, device, and medium

Abstract

Provided are a data retrieval method and device, a data processing method and device, equipment and a medium, which are related to the technical field of computers and can be used for data retrieval and data clustering. The implementation method comprises the following steps: in response to receiving a to-be-retrieved vector, determining at least one to-be-retrieved storage medium corresponding to the to-be-retrieved vector; for each to-be-retrieved storage medium in the at least one to-be-retrieved storage medium, using a storage controller corresponding to the to-be-retrieved storage medium to extract a first number of sample vectors from at least one sample vector stored in the to-be-retrieved storage medium, the similarity of the first number of sample vectors to the to-be-retrieved vector being higher than the similarity of other sample vectors in the at least one sample vector to the to-be-retrieved vector; and determining a retrieval result corresponding to the to-be-retrieved vector based on the first number of sample vectors from each of the at least one to-be-retrieved storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data retrieval method, a data processing method, a data retrieval device, a data processing device, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of internet technology, search engines have become increasingly powerful, and the content they retrieve has become richer. For example, voice, images, videos, audio-text, and user behavior data can all be provided to users as sample data for retrieval. In this era of rapid information development, users' demand for information is constantly increasing. Therefore, how to provide search services more quickly, efficiently, and accurately is a challenge currently facing search engine technology. Summary of the Invention

[0003] This disclosure provides a data retrieval method, a data processing method, a data retrieval device, a data processing device, a computer device, a computer-readable storage medium, and a computer program product.

[0004] According to one aspect of this disclosure, a data retrieval method is provided, comprising: in response to receiving a vector to be retrieved, determining at least one storage medium to be retrieved corresponding to the vector to be retrieved, wherein each of the at least one storage medium to be retrieved stores at least one sample vector, and each storage medium to be retrieved has a corresponding storage controller; for each of the at least one storage medium to be retrieved, using the storage controller corresponding to the storage medium to be retrieved, extracting a first number of sample vectors from the at least one sample vector stored in the storage medium to be retrieved, wherein the similarity between the first number of sample vectors and the vector to be retrieved is higher than the similarity between the other sample vectors in the at least one sample vector and the vector to be retrieved; and determining a retrieval result corresponding to the vector to be retrieved based on the first number of sample vectors from each of the at least one storage medium to be retrieved.

[0005] According to another aspect of this disclosure, a data processing method is provided, comprising: transmitting a second number of centroid vectors to a storage controller corresponding to each of a plurality of storage media, wherein the second number of centroid vectors are used to divide a plurality of sample vectors, and each storage medium stores a portion of the sample vectors among the plurality of sample vectors; for each of the plurality of storage media, using the storage controller corresponding to the storage medium, calculating the similarity between each of the second number of centroid vectors and each sample vector in the storage medium; and performing a correction operation for each of the second number of centroid vectors, wherein the correction operation comprises: determining at least one similar sample vector corresponding to the centroid vector, wherein the similarity between each similar sample vector and the centroid vector is not less than the similarity between the similar sample vector and other centroid vectors among the second number of centroid vectors; and correcting the centroid vector based on the at least one similar sample vector.

[0006] According to another aspect of this disclosure, a data retrieval apparatus is provided, comprising: a first determining unit configured to, in response to receiving a retrieval vector, determine at least one retrieval storage medium corresponding to the retrieval vector, wherein each of the at least one retrieval storage medium stores at least one sample vector, and each retrieval storage medium has a corresponding storage controller; an extraction unit configured to, for each of the at least one retrieval storage mediums, extract a first number of sample vectors from the at least one sample vector stored in the retrieval storage medium using the storage controller corresponding to the retrieval storage medium, wherein the similarity between the first number of sample vectors and the retrieval vector is higher than the similarity between the other sample vectors in the at least one sample vector and the retrieval vector; and a second determining unit configured to, based on the first number of sample vectors from each of the at least one retrieval storage mediums, determine a retrieval result corresponding to the retrieval vector.

[0007] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: a transmission unit configured to transmit a second number of centroid vectors to a storage controller corresponding to each of a plurality of storage media, wherein the second number of centroid vectors are used to divide a plurality of sample vectors, and each storage medium stores a portion of the plurality of sample vectors; a control unit configured to, for each of the plurality of storage media, calculate, using the storage controller corresponding to that storage medium, a similarity between each of the second number of centroid vectors and each sample vector in that storage medium; and a correction unit configured to perform a correction operation on each of the second number of centroid vectors, wherein the correction unit comprises: a determining subunit configured to determine at least one similar sample vector corresponding to the centroid vector, wherein the similarity between each similar sample vector and the centroid vector is not less than the similarity between the similar sample vector and other centroid vectors in the second number of centroid vectors; and a correction subunit configured to correct the centroid vector based on the at least one similar sample vector.

[0008] According to another aspect of this disclosure, a computer device is provided, comprising: a processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the processor, causes the processor to perform any of the methods described above.

[0009] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform any of the methods described above.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform any of the methods described above.

[0011] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0012] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a schematic diagram illustrating an example system in which various methods described herein may be implemented according to exemplary embodiments;

[0014] Figure 2 This is a flowchart illustrating a data retrieval method according to an exemplary embodiment;

[0015] Figure 3This is a schematic diagram illustrating a storage controller according to an exemplary embodiment;

[0016] Figure 4 This is a schematic diagram illustrating a processing system based on multiple storage controllers according to an exemplary embodiment;

[0017] Figure 5 This is a flowchart illustrating a data processing method according to another exemplary embodiment;

[0018] Figure 6 This is a schematic block diagram illustrating a data retrieval apparatus according to an exemplary embodiment;

[0019] Figure 7 This is a schematic block diagram illustrating a data processing apparatus according to an exemplary embodiment; and

[0020] Figure 8 This is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. Detailed Implementation

[0021] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0022] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context expressly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0023] Vector retrieval is an important method in information retrieval. Every sample data point in a database—such as speech, images, videos, audio-text, or user behavior—can be represented by a vector, called a sample vector. The number of sample vectors is vast, and in practical applications, they are often stored across multiple different storage media. During the retrieval process, the information input by the user can also be represented by a vector, called the retrieval vector. The retrieval vector needs to have its similarity calculated with every sample vector in the database, and based on the similarity value corresponding to each sample vector, the retrieval results corresponding to the retrieval vector are determined.

[0024] In related technologies, the processor first needs to transfer sample vectors stored in one or more different storage media to memory, and then the processor performs similarity calculations between the vector to be retrieved and each sample vector to determine the retrieval result. During this process, the transfer of sample data requires a large amount of bandwidth resources, and the huge amount of similarity calculation also puts significant pressure on the processor's data throughput and processing capabilities.

[0025] Based on this, this disclosure proposes a data retrieval method, in which a corresponding storage controller is configured for each storage medium to be retrieved. For each storage medium to be retrieved, the storage controller corresponding to the storage medium to be retrieved extracts a first number of sample vectors similar to the vector to be retrieved from at least one sample vector stored in the storage medium to be retrieved. Then, the processor integrates the first number of sample vectors from each storage medium to be retrieved and determines the retrieval result corresponding to the vector to be retrieved.

[0026] This disclosure assigns a corresponding storage controller to each storage medium to be retrieved, enabling the similarity calculation between the target vector and sample vectors to be performed separately at the storage medium and by the corresponding storage controller. In other words, based on the above-described scheme provided by this disclosure, the storage controller can transmit a first number of sample vectors, selected through similarity calculation and most similar to the target vector, to the processor. This significantly reduces the amount of data transfer between the storage medium and the processor, as well as the amount of data processing by the processor, thereby reducing the bandwidth consumption of vector retrieval, lowering the processor load, and enabling larger-scale vector retrieval with limited processing resources.

[0027] Exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0028] Figure 1 This is a schematic diagram illustrating an example system 100 in which various methods described herein may be implemented according to exemplary embodiments.

[0029] refer to Figure 1 The system 100 includes a client device 110, a server 120, and a network 130 that communicatively couples the client device 110 and the server 120.

[0030] Client device 110 includes a display 114 and a client application (APP) 112 that can be displayed on the display 114. Client application 112 can be an application that needs to be downloaded and installed before running, or a lightweight application (liteapp). If client application 112 is an application that needs to be downloaded and installed before running, client application 112 can be pre-installed on client device 110 and activated. If client application 112 is a mini-app, user 102 can directly run client application 112 on client device 110 without installing it, by searching for client application 112 in the host application (e.g., by the name of client application 112) or scanning the graphic code of client application 112 (e.g., barcode, QR code, etc.). In some embodiments, client device 110 can be any type of mobile computing device, including mobile computers, mobile phones, wearable computing devices (e.g., smartwatches, head-mounted devices including smart glasses, etc.), or other types of mobile devices. In some embodiments, the client device 110 may alternatively be a fixed computer device, such as a desktop computer, server computer, or other type of fixed computer device.

[0031] Server 120 is typically a server deployed by an Internet Service Provider (ISP) or Internet Content Provider (ICP). Server 120 can represent a single server, a cluster of multiple servers, a distributed system, or a cloud server providing basic cloud services (such as cloud databases, cloud computing, cloud storage, and cloud communications). It will be understood that, although... Figure 1 The diagram shows that server 120 communicates with only one client device 110, but server 120 can provide background services to multiple client devices simultaneously.

[0032] Examples of network 130 include combinations of local area networks (LANs), wide area networks (WANs), personal area networks (PANs), and / or communication networks such as the Internet. Network 130 can be wired or wireless. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to process data exchanged through network 130. Furthermore, encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some of the links. In some embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0033] For the purposes of this disclosure's embodiments, Figure 1In the example, client application 112 can be a search application that provides a user with an interface for inputting data to be retrieved and provides the user with search results corresponding to the data. Correspondingly, server 120 can be a server used in conjunction with the search application. Server 120 can provide data retrieval services to client application 112 running on client device 110 based on road network data. Alternatively, server 120 can also provide road network data to client device 110, and client application 112 running on client device 110 can provide local retrieval services based on this road network data.

[0034] Figure 2 This is a flowchart illustrating a data retrieval method 200 according to an exemplary embodiment. Method 200 can be implemented on a server (e.g., Figure 1 The method 200 is executed at server 120 (as shown in the diagram). In some embodiments, method 200 may be executed in combination by a client device (e.g., client device 110) and a server (e.g., server 120). Hereinafter, the various steps of method 200 will be described in detail with server 120 as the executing entity.

[0035] like Figure 2 As shown, the data retrieval method 200 includes: step S210, in response to receiving a retrieval vector, determining at least one retrieval storage medium corresponding to the retrieval vector, wherein each of the at least one retrieval storage medium stores at least one sample vector, and each retrieval storage medium has a corresponding storage controller; step S220, for each of the at least one retrieval storage mediums, using the storage controller corresponding to the retrieval storage medium, extracting a first number of sample vectors from the at least one sample vector stored in the retrieval storage medium, wherein the similarity between the first number of sample vectors and the retrieval vector is higher than the similarity between the other sample vectors in the at least one sample vector (excluding the first number of sample vectors) and the retrieval vector; and step S230, determining the retrieval result corresponding to the retrieval vector based on the first number of sample vectors from each of the at least one retrieval storage mediums.

[0036] Therefore, by assigning a corresponding storage controller to each storage medium to be retrieved, the similarity calculation between the target vector and the sample vector can be performed separately on each storage medium by its corresponding storage controller. This allows the first number of sample vectors, selected through similarity calculation and most similar to the target vector, to be transmitted to the processor from each storage medium. Consequently, the amount of data transferred between the storage medium and the processor, as well as the amount of data processed by the processor, is significantly reduced, decreasing the bandwidth consumption of vector retrieval, lowering the processor load, and enabling larger-scale vector retrieval with limited processing resources.

[0037] The data retrieval method 200 described above can be executed by a processor.

[0038] In step S210, the storage medium to be retrieved can be one or more, and there is no limitation here.

[0039] According to some embodiments, determining at least one storage medium to be retrieved corresponding to a vector to be retrieved may include: determining at least one matching centroid vector that matches the vector to be retrieved from a plurality of pre-stored candidate centroid vectors, wherein each matching centroid vector corresponds to a plurality of sample vectors; and determining the storage medium storing at least one sample vector from the plurality of sample vectors as the storage medium to be retrieved.

[0040] The processor matches the vector to be retrieved against multiple candidate centroid vectors, thereby identifying at least one matching centroid vector that matches the vector to be retrieved. The search range is then defined as the multiple sample vectors corresponding to each of these matching centroid vectors. This significantly reduces the range of sample vectors to be retrieved, effectively decreasing bandwidth usage and lowering the processor's processing load.

[0041] According to some embodiments, clustering can be performed on the sample vectors used for retrieval to divide the sample vectors used for retrieval into multiple vector groups, each vector group including multiple sample vectors; for each vector group in the multiple vector groups, a candidate centroid vector is determined to indicate the vector group, wherein the multiple candidate centroid vectors corresponding to the multiple vector groups can be pre-stored in memory.

[0042] In one implementation, multiple sample vectors in each vector group are stored in a separate storage medium, i.e., each matching centroid vector points to a storage medium to be retrieved.

[0043] In another implementation, the multiple sample vectors in each vector group are stored in multiple storage media. Specifically, the multiple sample vectors in each vector group are stored evenly in each storage medium. In this way, each matching centroid vector can point to multiple storage media to be searched, allowing the transfer and computation of sample vectors during the retrieval process to be performed in parallel on multiple storage media, effectively improving retrieval efficiency.

[0044] According to some embodiments, each of the at least one matching centroid vector can be the mean of the multiple sample vectors corresponding to that matching centroid vector. In this way, the matching centroid vector can characterize the overall features of the multiple sample vectors corresponding to it, ensuring that the multiple sample vectors determined based on the matching centroid vector have a high similarity to the vector to be retrieved, thereby guaranteeing the accuracy of the retrieval results.

[0045] According to some embodiments, a plurality of pre-stored candidate centroid vectors have corresponding index data structures. Determining at least one matching centroid vector from the pre-stored candidate centroid vectors that matches the vector to be retrieved may include: retrieving at least one matching centroid vector from the plurality of candidate centroid vectors by searching the index data structure. The similarity between the at least one matching centroid vector and the vector to be retrieved is higher than the similarity between the other candidate centroid vectors (excluding the at least one matching centroid vector) and the vector to be retrieved. This improves the efficiency of retrieving matching centroid vectors.

[0046] The index data structure can be constructed based on various methods such as IVF, KDtree, annoy, or SPTAG, and no specific method is specified here.

[0047] In step S220, according to some embodiments, extracting a first number of sample vectors from at least one sample vector stored in the storage medium to be retrieved using the storage controller corresponding to the storage medium to be retrieved may include: calculating the similarity value between each of the at least one sample vector and the vector to be retrieved; and extracting a first number of sample vectors from the storage medium to be retrieved by sorting the similarity values ​​corresponding to the at least one sample vector respectively.

[0048] Therefore, the similarity calculation of at least one sample vector in the storage medium to be retrieved can be performed through the storage controller corresponding to each storage medium to be retrieved. At least one sample vector can be sorted and filtered based on similarity value at the storage medium to be retrieved, saving the bandwidth occupied by sample vector transmission and reducing the processor load.

[0049] In this case, a separate storage controller can be set for each storage medium to be retrieved, or a common storage controller can be set for multiple storage media to be retrieved; there is no limitation on this.

[0050] According to some embodiments, the storage controller may include a computation unit and a sorting unit, the computation unit further including a multiplier and an adder. For each sample vector in at least one sample vector, the sample vector and the vector to be retrieved are input into the multiplier, the multiplier multiplies the corresponding elements of the sample vector and the vector to be retrieved to obtain a similarity vector, and the adder adds each element in the similarity vector to obtain a similarity value. The sorting unit arranges the similarity values ​​corresponding to the at least one sample vector in descending order to determine a first number of sample vectors.

[0051] In one implementation, a similarity threshold is preset, and sample vectors with similarity values ​​greater than the threshold are selected as the first number of sample vectors to be extracted. In other words, the value of the first number can vary depending on the different vectors to be retrieved and the different storage media to be retrieved.

[0052] In another implementation, a first number of values ​​is preset so that the first number of sample vectors with the highest similarity values ​​among at least one sample vector are used as the extracted sample vectors. In other words, the first number of values ​​can be a constant.

[0053] According to some embodiments, the storage controller may include a cache for caching the similarity values ​​of each of at least one sample vector.

[0054] When the similarity value of each sample vector calculated by the computing unit cannot be processed by the sorting unit in a timely manner, the similarity value of each sample vector can be cached in the cache to ensure that the sorting unit can obtain complete and reliable processing data and ensure the reliability of the sorting results.

[0055] Figure 3 This is a schematic diagram illustrating a storage controller according to an exemplary embodiment. (As shown) Figure 3 As shown, the storage controller 300 includes a multiplier 310, an adder 320, a cache 330, and a sorting unit 340.

[0056] The search vector 301 and any sample vector 302 stored in the search storage medium are input into the multiplier 310. The multiplier 310 multiplies the corresponding elements of the search vector 301 and the sample vector 302 to obtain a similarity vector, which is then input into the adder 320. The adder 320 adds each element in the similarity vector to obtain a similarity value, which is then input into the buffer 330. The sorting unit 340, through interaction with the buffer 330, obtains the similarity value of each sample vector in the search storage medium and determines a first number of sample vectors 303 by sorting the similarity values ​​of each sample vector. The first number of sample vectors 303 can be cached in the buffer 330 and output from the storage controller 300 at an appropriate time.

[0057] In step S230, a first number of sample vectors from each of the at least one storage medium to be retrieved are integrated at the processor. It is understood that each sample vector from the at least one storage medium to be retrieved undergoes similarity calculation by the corresponding storage controller at the storage medium where it is stored. The similarity value of each sample vector can be transmitted to the processor along with the sample vector, eliminating the need for the processor to perform further similarity calculations on the sample vectors.

[0058] According to some embodiments, determining the retrieval result corresponding to the retrieval vector based on a first number of sample vectors from each of at least one storage medium to be retrieved may include: determining the retrieval result corresponding to the retrieval vector by sorting the similarity values ​​of each sample vector from each of the at least one storage medium to be retrieved. Thus, the processor can integrate the sample vectors from each storage medium to be retrieved, extract one or more sample vectors with the highest similarity to the retrieval vector, and use these as the retrieval result corresponding to the retrieval vector.

[0059] Figure 4 This is a schematic diagram illustrating a processing system based on multiple storage controllers according to an exemplary embodiment. Figure 4 As shown, the system includes a processor 410, a memory 420, n storage media 431 to 43n, and n storage controllers 441 to 44n corresponding to the n storage media 431 to 43n respectively. Each storage media stores a sample vector, and each storage controller can perform similarity calculation between vectors.

[0060] When applying this processing system to a data retrieval scenario, multiple candidate centroid vectors can be pre-stored in memory 420. Input data 401 is the vector to be retrieved, and processor 410 determines at least one matching centroid vector from the multiple candidate centroid vectors in memory 420 that matches the vector to be retrieved in input data 401.

[0061] The following example illustrates how a matching centroid vector is determined, and the multiple sample vectors corresponding to the matching centroid vector are uniformly stored in each of the storage media 431 to 43n.

[0062] The processor 410 sends the identifiers of the target vector and the matching centroid vector in the input data 401 to the storage controllers 441-44n, respectively. Based on the identifier of the matching centroid vector, each storage controller retrieves at least one sample vector corresponding to the matching centroid vector from its corresponding storage medium, calculates the similarity value between each of the at least one sample vector and the target vector in the input data 401, and extracts a first number of sample vectors that are most similar to the target vector by sorting the similarity values ​​corresponding to the at least one sample vector.

[0063] Each of the storage controllers 441 to 44n sends the extracted first number of sample vectors and their similarity values ​​to the memory 420. The processor 410 sorts the similarity values ​​of each sample vector from the storage medium 431 to 43n to obtain output data 402, which is the retrieval result corresponding to the vector to be retrieved. The output data 402 can be a preset number of sample vectors with the highest similarity values ​​among the sample vectors from the storage medium 431 to 43n.

[0064] Figure 5 This is a flowchart illustrating a data processing method 500 according to an exemplary embodiment. Method 500 can be implemented on a server (e.g., Figure 1 The execution is performed at server 120 shown.

[0065] like Figure 5 As shown, the data processing method 500 includes: step S510, transmitting a second number of centroid vectors to the storage controller corresponding to each of the plurality of storage media, wherein the second number of centroid vectors is used to divide a plurality of sample vectors, and each storage medium stores a portion of the sample vectors in the plurality of sample vectors; step S520, for each of the plurality of storage media, using the storage controller corresponding to the storage medium, calculating the similarity between each of the second number of centroid vectors and each sample vector in the storage medium; and step S530, performing a correction operation for each of the second number of centroid vectors, wherein the correction operation includes: step S531, determining at least one similar sample vector corresponding to the centroid vector, wherein the similarity between each similar sample vector and the centroid vector is not less than the similarity between the similar sample vector and other centroid vectors in the second number of centroid vectors; and step S532, correcting the centroid vector based on the similarity between each of the at least one similar sample vector and the centroid vector.

[0066] Therefore, by assigning a corresponding storage controller to each storage medium, the similarity calculation between the second number of centroid vectors and sample vectors can be performed separately on each storage medium by the storage controller corresponding to that medium. In other words, instead of transmitting all sample vectors from each storage medium to the processor for similarity calculation, the already calculated similarity value of each sample vector is transmitted to the processor to correct each centroid vector. This significantly reduces the amount of data transferred between the storage medium and the processor, as well as the amount of data processed by the processor, reducing bandwidth usage and lowering the processor load, enabling larger-scale data processing with limited processing resources.

[0067] Steps S510 to S530 can be repeated until the centroid vector before and after correction are the same, and the at least one similar sample vector corresponding to the centroid vector no longer changes. In this way, multiple sample vectors can be divided into a second number of vector groups, each vector group can be represented by a centroid vector, and the sample vectors in each vector group are at least one similar sample vector corresponding to the centroid vector.

[0068] In step S510, in the initial state, the second number of centroid vectors can be the second number of random vectors; during the iteration process, the second number of centroid vectors are the second number of centroid vectors that have been corrected after the previous round of data processing.

[0069] In step S520, a separate storage controller can be set for each storage medium, or a common storage controller can be set for multiple storage media; this is not limited here.

[0070] According to some embodiments, the storage controller may include a multiplier and an adder. For each centroid vector in the second number of centroid vectors, each sample vector in the storage medium and the centroid vector are input to the multiplier, which multiplies the corresponding elements of the sample vector and the centroid vector to obtain a similarity vector. The adder adds each element of the similarity vector to obtain a similarity value between the centroid vector and the sample vector.

[0071] According to some embodiments, the correction operation is performed by a processor, and wherein calculating the similarity between each of the second number of centroid vectors and each sample vector in the storage medium using a storage controller corresponding to the storage medium may include: determining a first subset of the sample vectors stored in the storage medium; and calculating the similarity result of the first subset using the storage controller corresponding to the storage medium, the similarity result of the first subset being transmitted to the processor, wherein the similarity result of the first subset includes the similarity between each of the second number of centroid vectors and each sample vector in the first subset.

[0072] Therefore, after calculating the similarity result of the first subset, the storage controller can transmit the similarity result of the first subset to the processor. During the transmission of the similarity result of the first subset, the storage controller can continue to perform similarity calculations for other sample vectors stored in the storage medium. This effectively improves the efficiency of data processing.

[0073] After the processor receives the similarity results of each sample vector from multiple storage media, step S530 can be executed to correct each centroid vector in the second number of centroid vectors, wherein the similarity result of each sample vector includes the similarity value between the sample vector and each centroid vector in the second number of centroid vectors.

[0074] In step S531, for each sample vector, the sample vector is divided into similar sample vectors among the centroid vectors with the largest similarity value to the sample vector in the second number of centroid vectors. Thus, multiple sample vectors can be divided into a second number of vector groups based on the second number of centroid vectors.

[0075] In step S532, according to some embodiments, correcting the centroid vector based on at least one similar sample vector may include: correcting the centroid vector to the mean of at least one similar sample vector. This allows the corrected centroid vector to more accurately represent the features of at least one similar sample vector, making the corrected centroid vector closer to the final cluster center.

[0076] Still with Figure 4 For example, when this processing system is applied to a data processing scenario, storage media 431 to 43n all store ungrouped sample vectors. Input data 401 can be a second number of randomly generated centroid vectors, and processor 410 transmits the second number of centroid vectors to storage controllers 441 to 44n respectively. The following explanation uses the processing procedure of storage controller 441 as an example.

[0077] For each centroid vector in the second number of centroid vectors, the storage controller 441 calculates the similarity between the centroid vector and each sample vector stored in the storage medium 431, and transmits the similarity result of each sample vector to the memory 420. The similarity result includes the similarity value between the sample vector and each centroid vector in the second number of centroid vectors. The processing procedures of storage controllers 442 to 44n are similar to those of storage controller 441 and will not be described in detail here.

[0078] Processor 410 retrieves the similarity result of each sample vector from memory 420 among multiple sample vectors. Based on the similarity result of each sample vector, it determines at least one similar sample vector corresponding to each centroid vector. It can be understood that the at least one similar sample vector corresponding to each centroid vector can come from multiple different storage media 431 to 43n. Processor 410 corrects the centroid vector based on the similarity value of the at least one similar sample vector corresponding to each centroid vector.

[0079] The corrected second number of centroid vectors can be used as input data 410 again. The above processing is performed in the input processor 410 until the second number of centroid vectors no longer change, and the final second number of centroid vectors are output as output data 402.

[0080] According to some embodiments, after performing a correction operation on each centroid vector in a second number of centroid vectors, the corrected second number of centroid vectors are determined as a plurality of candidate centroid vectors for data retrieval. The corrected second number of centroid vectors can characterize the overall features of at least one similar sample vector corresponding to them. Therefore, they can be stored in memory as a plurality of candidate centroid vectors in data retrieval for the processor's initial retrieval of sample vectors, that is, determining the retrieved sample vector as at least one similar sample vector corresponding to the matching centroid vector among the plurality of candidate centroid vectors.

[0081] Although the operations are depicted in the accompanying drawings in a specific order, this should not be construed as requiring that these operations be performed in the specific order shown or in chronological order, nor should it be construed as requiring that all the operations shown be performed to obtain the desired result.

[0082] Figure 6 This is a schematic block diagram illustrating a data retrieval apparatus 600 according to an exemplary embodiment. Figure 6As shown, the data retrieval device 600 includes: a first determining unit 610 configured to determine at least one retrieval storage medium corresponding to the retrieval vector in response to receiving a retrieval vector, wherein each of the at least one retrieval storage medium stores at least one sample vector, and each retrieval storage medium has a corresponding storage controller; an extraction unit 620 configured to extract a first number of sample vectors from the at least one sample vector stored in each of the at least one retrieval storage medium using the storage controller corresponding to the retrieval storage medium, wherein the similarity between the first number of sample vectors and the retrieval vector is higher than the similarity between the other sample vectors in the at least one sample vector and the retrieval vector; and a second determining unit 630 configured to determine a retrieval result corresponding to the retrieval vector based on the first number of sample vectors from each of the at least one retrieval storage medium.

[0083] Figure 7 This is a schematic block diagram illustrating a data processing apparatus 700 according to an exemplary embodiment. Figure 7 As shown, the data processing apparatus 700 includes: a transmission unit 710 configured to transmit a second number of centroid vectors to a storage controller corresponding to each of a plurality of storage media, wherein the second number of centroid vectors is used to divide a plurality of sample vectors, and each storage medium stores a portion of the sample vectors among the plurality of sample vectors; a control unit 720 configured to, for each of the plurality of storage media, calculate, using the storage controller corresponding to that storage medium, the similarity between each of the second number of centroid vectors and each sample vector in that storage medium; and a correction unit 730 configured to perform a correction operation on each of the second number of centroid vectors, wherein the correction unit includes: a determination subunit 731 configured to determine at least one similar sample vector corresponding to the centroid vector, wherein the similarity between each similar sample vector and the centroid vector is not less than the similarity between the similar sample vector and other centroid vectors among the second number of centroid vectors; and a correction subunit 732 configured to correct the centroid vector based on the similarity between each of the at least one similar sample vector and the centroid vector.

[0084] It should be understood that Figure 6 Each unit of the data retrieval device 600 shown can be connected to a reference. Figure 2 The steps in the described method 200 correspond to each other, and Figure 7 Each unit of the data processing apparatus 700 shown can be connected to a reference. Figure 5The steps in method 500 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to device 600 and its included units, and the operations, features, and advantages described above for method 500 also apply to device 700 and its included units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0085] While specific functions have been discussed above with reference to specific units, it should be noted that the functions of the units discussed herein may be divided into multiple units, and / or at least some functions of multiple units may be combined into a single unit. The specific unit performing an action discussed herein includes the specific unit itself performing the action, or alternatively, the specific unit calling or otherwise accessing another component or unit that performs the action (or performs the action in conjunction with the specific unit). Therefore, a specific unit performing an action may include the specific unit performing the action itself and / or another unit that performs the action, called or otherwise accessed by the specific unit.

[0086] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program units. The above regarding... Figure 6 and 7 The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in computer-readable storage. Alternatively, these units can be implemented as hardware logic / circuit. For example, in some embodiments, the first determining unit 610, the extracting unit 620, the second determining unit 630, the transmitting unit 710, the controlling unit 720, and the correcting unit 730 can be implemented together in a system-on-chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0087] According to one aspect of this disclosure, a computer device is also provided, comprising: a processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the processor, causes the processor to perform any of the methods described above.

[0088] According to one aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform any of the methods described above.

[0089] According to one aspect of this disclosure, a computer program product is also provided, comprising a computer program that, when executed by a processor, causes the processor to perform any of the methods described above.

[0090] In the following text, combined with Figure 8 Illustrative examples describing such computer devices, non-transitory computer-readable storage devices, and computer program products.

[0091] Figure 8 An example configuration of a computer device 800 that can be used to implement the methods described herein is shown. For example, Figure 1 The server 120 and / or client device 110 shown may include an architecture similar to computer device 800. The aforementioned data retrieval and data processing devices may also be implemented wholly or at least partially by computer device 800 or similar devices or systems.

[0092] Computer device 800 can be a variety of different types of devices. Examples of computer device 800 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0093] Computer device 800 may include at least one processor 802, memory 804, multiple communication interfaces 806, display device 808, other input / output (I / O) devices 810, and one or more mass storage devices 812 capable of communicating with each other, such as via system bus 814 or other suitable connections.

[0094] Processor 802 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 802 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 802 may be configured to acquire and execute computer-readable instructions stored in memory 804, mass storage device 812, or other computer-readable media, such as program code of operating system 816, program code of application program 818, program code of other program 820, etc.

[0095] Memory 804 and mass storage device 812 are examples of computer-readable storage devices used to store instructions that are executed by processor 802 to perform the various functions described above. For example, memory 804 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 812 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 804 and mass storage device 812 may be collectively referred to herein as memory or computer-readable storage, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 802 as a specific machine configured to perform the operations and functions described in the examples herein.

[0096] Multiple programs may be stored on mass storage device 812. These programs include operating system 816, one or more application programs 818, other programs 820, and program data 822, and they may be loaded into memory 804 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing components / functions such as method 200 and / or method 500, and / or other embodiments described herein.

[0097] Although Figure 8 The modules 816, 818, 820, and 822, or portions thereof, are illustrated as being stored in memory 804 of computer device 800; however, modules 816, 818, 820, and 822 may be implemented using any form of computer-readable medium accessible by computer device 800. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage devices and communication media.

[0098] Computer-readable storage includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage as defined herein does not include communication media.

[0099] One or more communication interfaces 806 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces include near-field communication (NFC) interfaces, etc. Communication interface 806 facilitates communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 806 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0100] In some examples, a display device 808, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 810 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0101] The techniques described herein can be supported by these various configurations of computer device 800, and are not limited to specific examples of the techniques described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from computer device 800. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 800 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality may be implemented partly on computer device 800 and partly through a platform that abstracts the functionality of the cloud.

[0102] Although this disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and suggestive, not restrictive; this disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments in practice with respect to the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps not listed, the indefinite article "a" or "an" does not exclude a plurality, the term "a plurality" means two or more, and the term "based on" should be interpreted as "at least partially based on". The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

Claims

1. A data retrieval method based on a storage controller, comprising: In response to receiving a retrieval vector, the processor determines at least one retrieval storage medium corresponding to the retrieval vector, wherein each of the at least one retrieval storage medium stores at least one sample vector, and each retrieval storage medium has a corresponding storage controller. For each of the at least one storage medium to be retrieved, using the storage controller corresponding to that storage medium, a first number of sample vectors are extracted from at least one sample vector stored in that storage medium. The similarity between the first number of sample vectors and the target vector is higher than the similarity between the target vector and other sample vectors in the at least one sample vector, excluding the first number of sample vectors. The storage controller includes: A multiplier that multiplies the corresponding elements of the vector to be retrieved and the at least one sample vector to obtain a similarity vector; An adder that adds each element in the similarity vector to obtain a similarity value; A cache, which receives and caches the similarity values ​​and is capable of caching the first number of sample vectors; and A sorting unit, which interacts with the buffer, determines the first number of sample vectors by sorting the similarity values ​​corresponding to the at least one sample vector; and The processor determines the retrieval result corresponding to the retrieval vector based on a first number of sample vectors from each of the at least one storage medium to be retrieved.

2. The method according to claim 1, wherein, The step of determining at least one storage medium to be retrieved corresponding to the vector to be retrieved includes: From a plurality of pre-stored candidate centroid vectors, at least one matching centroid vector is determined that matches the vector to be retrieved, wherein each matching centroid vector corresponds to a plurality of sample vectors; and The storage medium containing at least one of the plurality of sample vectors is determined as the storage medium to be retrieved.

3. The method according to claim 2, wherein, Each of the at least one matching centroid vector is the mean of the plurality of sample vectors corresponding to that matching centroid vector.

4. The method according to claim 2, wherein, The pre-stored multiple candidate centroid vectors have corresponding index data structures, and the step of determining at least one matching centroid vector that matches the vector to be retrieved from the pre-stored multiple candidate centroid vectors includes: By searching the index data structure, at least one matching centroid vector is determined from the plurality of candidate centroid vectors. The similarity between the at least one matching centroid vector and the vector to be searched is higher than the similarity between the other candidate centroid vectors (excluding the at least one matching centroid vector) and the vector to be searched.

5. The method according to claim 1, wherein, The step of determining the retrieval result corresponding to the retrieval vector based on a first number of sample vectors from each of the at least one storage medium to be retrieved includes: The retrieval result corresponding to the retrieval vector is determined by sorting the similarity values ​​of each sample vector from each of the at least one storage medium to be retrieved.

6. The method according to claim 2, wherein the plurality of candidate centroid vectors are determined by the following process, including: A second number of centroid vectors are transmitted to the storage controller corresponding to each of the plurality of storage media, wherein the second number of centroid vectors are used to divide the plurality of sample vectors, and each storage medium stores a portion of the plurality of sample vectors. For each of the plurality of storage media, using the storage controller corresponding to that storage media, the similarity between each of the second number of centroid vectors and each sample vector in that storage media is calculated, including: Determine the first subset of sample vectors stored in the storage medium; and The similarity result of the first subset is calculated using the storage controller corresponding to the storage medium. This similarity result is then transmitted to the processor. The first subset's similarity result includes the similarity between each of the second number of centroid vectors and each sample vector in the first subset. For each centroid vector in the second number of centroid vectors, the processor performs a correction operation, wherein the correction operation includes: Determine at least one similar sample vector corresponding to the centroid vector, wherein the similarity between each similar sample vector and the centroid vector is not less than the similarity between the similar sample vector and other centroid vectors in the second number of centroid vectors; Based on the similarity between each of the at least one similar sample vector and the centroid vector, the centroid vector is corrected; and After performing a correction operation on each of the second number of centroid vectors, the corrected second number of centroid vectors are determined as the plurality of candidate centroid vectors.

7. The method according to claim 6, wherein, The step of correcting the centroid vector based on the at least one similar sample vector includes: The centroid vector is corrected to the mean of the at least one similar sample vector.

8. A data retrieval device based on a storage controller, comprising: The first determining unit is configured to cause the processor to determine at least one storage medium corresponding to the vector to be retrieved in response to receiving the vector to be retrieved, wherein each of the at least one storage medium to be retrieved stores at least one sample vector, and each storage medium to be retrieved has a corresponding storage controller. An extraction unit is configured to, for each of the at least one searchable storage media, extract a first number of sample vectors from at least one sample vector stored in the searchable storage media using a storage controller corresponding to that searchable storage media. The first number of sample vectors has a higher similarity to the searchable vector than the similarity between the searchable vector and other sample vectors (excluding the first number of sample vectors) among the at least one sample vectors. The storage controller includes: A multiplier that multiplies the corresponding elements of the vector to be retrieved and the at least one sample vector to obtain a similarity vector; An adder that adds each element in the similarity vector to obtain a similarity value; A cache, which receives and caches the similarity values ​​and is capable of caching the first number of sample vectors; and A sorting unit, which interacts with the buffer, determines the first number of sample vectors by sorting the similarity values ​​corresponding to the at least one sample vector; and The second determining unit is configured to enable the processor to determine the retrieval result corresponding to the retrieval vector based on a first number of sample vectors from each of the at least one retrieval storage medium.

9. A computer device, comprising: processor; as well as The memory, on which computer programs are stored, When the computer program is executed by the processor, it causes the processor to perform the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

Images