Data processing method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a test vector set and a base library vector set; determining the recall rate of the test vector set; determining a first compensation parameter set and a second compensation parameter set under the condition that the recall rate of the test vector set is smaller than or equal to a first preset threshold value; and compensating a plurality of first results output by a storage array in the vector retrieval system by utilizing a plurality of first compensation parameters in the first compensation parameter set, and compensating a plurality of second results output by a processor in the vector retrieval system by utilizing a plurality of second compensation parameters in the second compensation parameter set.

Description

Technical Field

[0001] The present application relates to the field of data retrieval technology, and in particular to a data processing method, device, equipment, storage medium and computer program product. Background Art

[0002] At present, in the current vector retrieval system based on the integrated storage and computing processor, the main solution is to calibrate the storage value of the base database feature by rewriting the base database data, so as to ensure the recall rate of the retrieval. However, rewriting the base database data will bring energy consumption overhead. The more times the base database data is rewritten and the larger the amount of base database data is, the higher the energy consumption overhead generated to ensure the recall rate of the vector retrieval. The longer the time to rewrite the base database data is, the interruption of the retrieval application service will be caused, and the computing efficiency will be reduced, so that the retrieval accuracy of the vector retrieval system cannot be guaranteed. Summary of the invention

[0003] In view of this, embodiments of the present application hope to provide a data processing method, apparatus, device, storage medium and computer program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a data processing method, the method comprising:

[0006] Obtain a test vector set and a base library vector set;

[0007] Determining a recall rate of the test vector set;

[0008] When the recall rate of the test vector set is less than or equal to a first preset threshold, determining a first compensation parameter set and a second compensation parameter set;

[0009] Using a plurality of first compensation parameters in the first compensation parameter set, compensating a plurality of first results output by a storage and calculation array in a vector retrieval system, and using a plurality of second compensation parameters in the second compensation parameter set, compensating a plurality of second results output by a processor in the vector retrieval system;

[0010] Among them, the multiple first results output by the storage and calculation array are obtained by performing dot product operations on each base vector in the base vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to a first result of the dot product operation between a base vector and the multiple test vectors; the multiple second results output by the processor are obtained by performing a modulo correlation operation on each base vector in the base vector set; and one second compensation parameter corresponds to one second result.

[0011] In addition, according to at least one embodiment of the present application, determining the recall rate of the test vector set includes:

[0012] According to a preset period, a recall rate of the test vector set is determined.

[0013] In addition, according to at least one embodiment of the present application, determining the recall rate of the test vector set includes:

[0014] Determine the similarity between each test vector in the test vector set and each base vector in the base vector set to obtain a similarity result;

[0015] Determine, according to the similarity result, a first number of measurement vectors in the test vector set whose similarity to the base library vector is greater than or equal to a second preset threshold;

[0016] The recall rate of the test vector set is determined according to the first number and a second number of all test vectors in the test vector set; the recall rate represents a ratio of the first number to the second number.

[0017] In addition, according to at least one embodiment of the present application, determining the first compensation parameter set and the second compensation parameter set includes:

[0018] Performing dot product operations on each test vector in the test vector set and each base vector in the base vector set to obtain multiple operation results; and using the multiple operation results as multiple theoretical values;

[0019] Using the test vector set and the base vector set as inputs of a storage and calculation array in the vector retrieval system, obtaining a plurality of first results output by the storage and calculation array; using the plurality of first results as a plurality of actual values;

[0020] Performing a fitting operation on the plurality of theoretical values ​​and the plurality of actual values ​​to obtain a plurality of fitting functions; the fitting function includes a first fitting parameter and a second fitting parameter; the fitting function corresponds to a fitting operation of a base library vector;

[0021] The first compensation parameter set and the second compensation parameter set are determined according to the multiple fitting functions.

[0022] In addition, according to at least one embodiment of the present application, determining the first compensation parameter set and the second compensation parameter set according to the multiple fitting functions includes:

[0023] For each of the plurality of fitting functions, determining an expression of a compensation value of the actual value according to the corresponding fitting function; and determining a first compensation parameter and a second compensation parameter corresponding to the corresponding fitting function according to the expression of the compensation value;

[0024] The first compensation parameter set is obtained according to the first compensation parameters respectively corresponding to the multiple fitting functions, and the second compensation parameter set is obtained according to the second compensation parameters respectively corresponding to the multiple fitting functions.

[0025] In addition, according to at least one embodiment of the present application, using the multiple first compensation parameters in the first compensation parameter set to compensate the multiple first results output by the storage and calculation array in the vector retrieval system includes:

[0026] For each first compensation parameter of the plurality of first compensation parameters, the following operations are performed:

[0027] The first compensation parameter is multiplied by the corresponding first result to compensate the first result.

[0028] In addition, according to at least one embodiment of the present application, using the plurality of second compensation parameters in the second compensation parameter set to compensate the plurality of second results output by the processor in the vector search system includes:

[0029] For each second compensation parameter of the plurality of second compensation parameters, the following operations are performed:

[0030] The second compensation parameter is added to the corresponding second result to compensate the second result.

[0031] In addition, according to at least one embodiment of the present application, after using multiple first compensation parameters in the first compensation parameter set to compensate multiple first results output by the storage and calculation array in the vector retrieval system, and using multiple second compensation parameters in the second compensation parameter set to compensate multiple second results output by the processor in the vector retrieval system, the method further includes:

[0032] Re-determining the recall rate of the test vector set;

[0033] Determining whether the recall rate of the test vector set is less than or equal to the first preset threshold;

[0034] If the recall rate of the test vector set is less than or equal to the first preset threshold, the base database data is rewritten; the base database data is used to be converted into a base database vector in the base database vector set.

[0035] At least one embodiment of the present application provides a data processing device, including:

[0036] An acquisition module, used to acquire a test vector set and a base library vector set;

[0037] A processing module, used to determine the recall rate of the test vector set; determine a first compensation parameter set and a second compensation parameter set when the recall rate of the test vector set is less than or equal to a first preset threshold; use multiple first compensation parameters in the first compensation parameter set to compensate multiple first results output by a storage array in a vector retrieval system, and use multiple second compensation parameters in the second compensation parameter set to compensate multiple second results output by a processor in the vector retrieval system; wherein the multiple first results output by the storage array are obtained by performing dot product operations on each base vector in the base vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to a first result of dot product operations on a base vector and the multiple test vectors; the multiple second results output by the processor are obtained by performing modulo correlation operations on each base vector in the base vector set; and one second compensation parameter corresponds to one second result.

[0038] At least one embodiment of the present application provides a server, including a processor and a memory for storing a computer program that can be run on the processor.

[0039] Wherein, when the processor is used to run the computer program, it executes the steps of any one of the above methods.

[0040] At least one embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0041] At least one embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0042] The data processing method, apparatus, device, storage medium and computer program product provided by the embodiments of the present application include: obtaining a test vector set and a base library vector set; determining the recall rate of the test vector set; determining a first compensation parameter set and a second compensation parameter set when the recall rate of the test vector set is less than or equal to a first preset threshold; using multiple first compensation parameters in the first compensation parameter set to compensate multiple first results output by a storage array in a vector retrieval system, and using multiple second compensation parameters in the second compensation parameter set to compensate multiple second results output by a processor in the vector retrieval system; wherein the multiple first results output by the storage array are obtained by performing dot product operations on each base library vector in the base library vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to a first result of dot product operations on one base library vector and the multiple test vectors; the multiple second results output by the processor are obtained by performing modulo correlation operations on each base library vector in the base library vector set; and one second compensation parameter corresponds to one second result.

[0043] By adopting the technical solution provided in the embodiment of the present application, multiple first compensation parameters in the first compensation parameter set are used to compensate for multiple first results output by the storage and computing array in the vector retrieval system, and multiple second compensation parameters in the second compensation parameter set are used to compensate for multiple second results output by the processor in the vector retrieval system. Compared with the method of rewriting the base database data when the recall rate is lower than a threshold in the related art, there is no need to spend a lot of time and energy to rewrite the base database data, which improves the efficiency of error calibration of the vector retrieval system and ensures the retrieval accuracy of the vector retrieval system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the calculation principle of realizing storage and computing integration based on memristor in the related technology;

[0045] Figure 2 It is a schematic diagram of realizing calculation based on storage and computing array and processor in the related technology;

[0046] Figure 3 It is a schematic diagram of the implementation flow of the data processing method of the embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of a specific implementation flow of the data processing method of an embodiment of the present application;

[0048] Figure 5 is a schematic diagram of fitting results obtained by fitting multiple theoretical values ​​and multiple actual values ​​in an embodiment of the present application;

[0049] Figure 6is a schematic diagram of compensating for multiple first results and multiple second results in an embodiment of the present application;

[0050] Figure 7 This is a schematic diagram of the structure of the data processing device of the embodiment of the present application.

[0051] Figure 8 It is a schematic diagram of the composition structure of the server of the embodiment of the present application. DETAILED DESCRIPTION

[0052] Before introducing the technical solutions of the embodiments of the present application, the related technologies are first introduced.

[0053] In the related art, a vector retrieval system is an information retrieval method based on a vector space model, which refers to retrieving vectors similar to a given query vector in a given set of base vectors. A vector retrieval system represents data such as documents, images, and audio as base vectors in a high-dimensional space, and measures the similarity between the comparison objects by calculating the similarity between the base vectors and the query vectors. This model has the advantages of being simple, flexible, and easy to expand, and has been widely used in search engines, recommendation systems, image recognition, and other fields. With the popularization of the Internet and the widespread application of IoT devices, a huge amount of data has been generated. The amount of data for vector retrieval is experiencing an unprecedented surge. The size of the data set in a retrieval system can reach tens of millions or even hundreds of millions of base vectors. For a vector retrieval system built with general-purpose processors such as a central processing unit (CPU) and a graphics processing unit (GPU), since each retrieval request needs to be compared and calculated with all base vectors, a large amount of data needs to be moved between the storage unit and the computing unit during this calculation process, which brings huge energy consumption and computing delay.

[0054] At present, storage-computing integration, as a new computing technology with comprehensive innovation from devices to architecture, can realize parallel multiplication and addition calculations directly on the storage array, which can greatly reduce the time overhead caused by data movement in the calculation process and greatly improve computing energy efficiency. It is necessary to consider using storage-computing integration to solve the computing and energy consumption problems faced by vector retrieval systems.

[0055] Figure 1 This is a schematic diagram of the computing principle of realizing storage and computing integration based on memristors in related technologies, such as Figure 1 As shown in the figure, the calculation principle of the integrated storage and computing is shown in Figure 1, taking the memristor as an example. A storage and computing device is equivalent to a multiplier, an adder and a memory. However, since the similarity calculation method in the vector retrieval system is not entirely a vector multiplication and addition calculation, the current vector retrieval system that uses the integrated storage and computing as the computing chip usually uses a method that combines the processor (CPU) and the integrated storage and computing.

[0056] Figure 2 It is a schematic diagram of implementing calculation based on storage and computing array and processor in related technology, such as Figure 2 As shown, for the base vector set All element values ​​of all base vectors in the base vector set are stored in the storage array of the vector retrieval system, and each column stores a base vector data. The vector to be searched is passed as input into the storage array of the vector search system.

[0057] Calculate the vectors in the base vector set X that match the vector to be searched The calculation method of the most similar K base vectors is shown in formula (1), where K is a positive integer:

[0058]

[0059] in, Indicates that the vector to be retrieved Perform dot product operation with each base vector in the base vector set X; 2 Indicates that each base vector in the base vector set X is subjected to a square operation; 0.5X 2 ={c 1 ,c 2 ,...,c n}=C, that is, C represents an offset constant set, and the offset constant set includes n offset constants, where n is an integer greater than 1. It is the dot product (or dot multiplication) operation of the vector, which can be efficiently implemented based on the storage and computing array (or storage and computing integrated chip) of the vector retrieval system. The offset constant set is 0.5X 2 It can be calculated offline in advance and stored in the CPU of the vector retrieval system.

[0060] At present, fast and efficient vector retrieval applications can be achieved through the method of combining storage and computing with CPU. However, due to the non-ideal characteristics of storage and computing devices, electrons will have a certain degree of free movement and instability. Therefore, the resistance state of the storage and computing device will drift in an undirected manner over time, that is, the element value of the saved base library vector will change, which will affect the calculation of the similarity between the retrieval vector and the base library vector, resulting in a decrease in the recall rate of the retrieval system and affecting the user experience. When this happens, the current processing method is to rewrite the base library data after the recall rate drops. However, since the base library data set is too large and the storage and computing device writes data slowly, the time for rewriting and maintaining the base library data will be very long, which will lead to problems such as low computing efficiency, high energy consumption, and application interruption.

[0061] That is, in response to the problem of decreased retrieval recall rate due to the non-ideal characteristics of the integrated storage and computing chip devices in the current vector retrieval system based on the integrated storage and computing processor, the current main solution is to calibrate the storage value of the base database features by rewriting the base database data, thereby ensuring the retrieval recall rate. There are mainly two problems: First, the integrated storage and computing chip will bring energy consumption overhead due to rewriting the base database data. The more rewrite times and the larger the amount of base database data, the higher the energy consumption overhead generated to ensure the retrieval recall rate. Second, because it takes a long time to rewrite the base database data in the integrated storage and computing chip, it will cause the retrieval application service to be interrupted and the computing efficiency will decrease.

[0062] Based on this, in an embodiment of the present application, a test vector set and a base library vector set are obtained; the recall rate of the test vector set is determined; when the recall rate of the test vector set is less than or equal to a first preset threshold, a first compensation parameter set and a second compensation parameter set are determined; multiple first compensation parameters in the first compensation parameter set are used to compensate multiple first results output by a storage and calculation array in a vector retrieval system, and multiple second compensation parameters in the second compensation parameter set are used to compensate multiple second results output by a processor in the vector retrieval system; wherein the multiple first results output by the storage and calculation array are obtained by performing dot product operations on each base library vector in the base library vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to the first result of dot product operations on one base library vector and the multiple test vectors; the multiple second results output by the processor are obtained by performing modulo correlation operations on each base library vector in the base library vector set; and one second compensation parameter corresponds to one second result.

[0063] See also Figure 3 , Figure 3 Schematic diagram of the implementation flow of the data processing method of the embodiment of the present application. Figure 3 As shown, it includes steps 301 to 304:

[0064] Step 301: Acquire a test vector set and a base library vector set.

[0065] It can be understood that the base library vector set may refer to a set consisting of multiple base library vectors.

[0066] It can be understood that the base database vector refers to converting base database data such as documents, images, audio, etc. into vectors in a high-dimensional space and storing them in a vector retrieval system.

[0067] It can be understood that the vector retrieval system is used to retrieve at least one vector closest to the query vector from the multiple base vectors included in the base vector set.

[0068] It can be understood that the vector retrieval system may include a processor (CPU) and a storage array.

[0069] It is understandable that the test vector set may refer to a set consisting of multiple test vectors; wherein the test vector may refer to a constructed vector whose similarity with the base library vector is greater than a preset threshold. The test vector may also be described as a query vector, a target vector, etc.

[0070] It can be understood that the base-base vectors in the base-base vector set can be written into the storage array of the vector retrieval system.

[0071] It is understandable that the offset constant can be calculated based on the base library vector set, and the offset constant can be stored in the processor (CPU) of the vector retrieval system.

[0072] Here, the offset constant can be calculated according to formula (2), that is, a modular correlation operation is performed on each base vector in the base vector set to obtain each offset constant, as follows:

[0073] C=0.5X 2 ={c 1 ,c 2 ,...,c n} (2)

[0074] Among them, C represents the offset constant set, which consists of n offset constants, X represents the base library vector set, and It means that the base vector set consists of n base vectors, where n is an integer greater than 1.

[0075] Step 302: Determine the recall rate of the test vector set.

[0076] Here, recall can refer to an important indicator for measuring the relevance of vectors returned by a vector retrieval system.

[0077] In some embodiments, determining the recall rate of the test vector set includes:

[0078] According to a preset period, a recall rate of the test vector set is determined.

[0079] Here, since the non-ideal characteristics of the storage and computing array (which can also be described as a storage and computing integrated device) will cause the element values ​​in the written base vector set (i.e., the base vector) to change, thereby affecting the application recall rate of the vector retrieval system, the recall rate of the vector retrieval system can be tested periodically.

[0080] In some embodiments, determining the recall rate of the test vector set includes:

[0081] Determine the similarity between each test vector in the test vector set and each base vector in the base vector set to obtain a similarity result;

[0082] Determine, according to the similarity result, a first number of measurement vectors in the test vector set whose similarity to the base library vector is greater than or equal to a second preset threshold;

[0083] The recall rate of the test vector set is determined according to the first number and a second number of all test vectors in the test vector set; the recall rate represents a ratio of the first number to the second number.

[0084] Here, the test vector set can be used Indicates that the test vector set consists of m test vectors, where m is an integer greater than 1. The base vector set is represented by It indicates that the base vector set consists of n base vectors, where n is an integer greater than 1.

[0085] For example, assuming that the test vector set includes m=3 test vectors, represented by test vector 1, test vector 2, and test vector 3 respectively, and the base vector set includes n=2 base vectors, represented by base vector 1 and base vector 2 respectively, then according to the preset period, the similarity between each test vector in the test vector set and each base vector in the base vector set is determined, that is, the similarity between test vector 1 and base vector 1 is calculated to obtain similarity result 1; the similarity between test vector 1 and base vector 2 is calculated to obtain similarity result 2; the similarity between test vector 2 and base vector 1 is calculated Calculate the similarity to obtain similarity result 3; calculate the similarity between test vector 2 and base library vector 2 to obtain similarity result 4; calculate the similarity between test vector 3 and base library vector 1 to obtain similarity result 5; calculate the similarity between test vector 3 and base library vector 2 to obtain similarity result 6; if among these 6 similarity results, the similarity between test vector 1 and base library vector 1 is greater than the second preset threshold, and the similarity between test vector 2 and base library vector 2 is greater than the second preset threshold, then determine that the first number is 2; calculate the ratio of the first number, i.e. 2, to the second number, i.e. 3, to obtain the recall rate.

[0086] Step 303: When the recall rate of the test vector set is less than or equal to a first preset threshold, determine a first compensation parameter set and a second compensation parameter set.

[0087] Here, if the recall rate is less than or equal to the preset first threshold, it indicates that the element value of the base vector stored in the storage array in the vector retrieval system has changed, and error compensation is required.

[0088] Here, the multiple first compensation parameters in the first compensation parameter set are used to compensate for the multiple first results output by the storage and calculation array in the vector retrieval system; the input of the storage and calculation array is the test vector set and the base library vector set; the multiple first results output by the storage and calculation array are obtained by performing dot product operations on each base library vector in the base library vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to a first result of dot product operations on a base library vector and the multiple test vectors.

[0089] Here, the multiple second compensation parameters in the second compensation parameter set are used to compensate for the multiple second results output by the processor in the vector retrieval system; the input of the processor is the base vector set; the multiple second results output by the processor are obtained by performing a modular correlation operation on each base vector in the base vector set; one second compensation parameter corresponds to one second result.

[0090] In some embodiments, determining the first compensation parameter set and the second compensation parameter set includes:

[0091] Performing dot product operations on each test vector in the test vector set and each base vector in the base vector set to obtain multiple operation results; and using the multiple operation results as multiple theoretical values;

[0092] Using the test vector set and the base vector set as inputs of a storage and calculation array in the vector retrieval system, obtaining a plurality of first results output by the storage and calculation array; using the plurality of first results as a plurality of actual values;

[0093] Performing a fitting operation on the plurality of theoretical values ​​and the plurality of actual values ​​to obtain a plurality of fitting functions; the fitting function comprises a first fitting parameter and a second fitting parameter; the fitting parameter corresponds to a fitting operation of a base library vector;

[0094] The first compensation parameter set and the second compensation parameter set are determined according to the multiple fitting functions.

[0095] It can be understood that for any base vector in the base vector set, multiple first theoretical values ​​and multiple first actual values ​​can be obtained together with multiple test vectors in the test vector set. In this way, linear fitting is performed using the multiple first actual values ​​as vertical coordinates and the multiple first theoretical values ​​as horizontal coordinates to obtain a linear fitting result, i.e., a fitting function. The fitting function includes a first fitting parameter and a second fitting parameter. By analogy, the fitting functions of all base vectors can be obtained.

[0096] In some embodiments, determining the first compensation parameter set and the second compensation parameter set according to the multiple fitting functions includes:

[0097] For each of the plurality of fitting functions, determining an expression of a compensation value of the actual value according to the corresponding fitting function; and determining a first compensation parameter and a second compensation parameter corresponding to the corresponding fitting parameter according to the expression of the compensation value;

[0098] The first compensation parameter set is obtained according to the first compensation parameters respectively corresponding to the multiple fitting functions, and the second compensation parameter set is obtained according to the second compensation parameters respectively corresponding to the multiple fitting functions.

[0099] Here, the nth base vector is Taking the example of FIG. 1 as an example, the process of obtaining the first compensation parameter and the second compensation parameter is described as follows:

[0100] First, for the nth base vector The base library vector and each test vector in the test vector set are respectively subjected to dot product operations to obtain a plurality of first theoretical values ​​(which can also be described as theoretical true values). The base library vector and each test vector in the test vector set are used as inputs of a storage and calculation array in the vector retrieval system to obtain a plurality of first actual values ​​output by the storage and calculation array.

[0101] Then, a linear fit is performed on the multiple first actual values ​​on the storage and computing array (which can also be described as a storage and computing integrated chip) and the multiple first theoretical values ​​to obtain a fitting function, which includes a first fitting parameter and a second fitting parameter.

[0102] Here, the fitting function can be: R m,n =k n ×R' m,n +b n , R' m,n Indicates the actual value, R m,n represents the theoretical value, k n represents the first fitting parameter, b n Indicates that the second fitting parameter is.

[0103] Next, based on the fitting function, an expression for the compensation value of the actual value is determined.

[0104] Here, by fitting the function, replacing the actual value in the fitting function with the compensation value of the actual value, and replacing the theoretical value in the fitting function with the actual value, the expression of the compensation value of the actual value can be obtained as follows:

[0105]

[0106] in, Indicates the compensation value of the actual value, R' m,n represents the actual value, k n and b n is the nth base vector in the corresponding base vector set The first fitting parameter and the second fitting parameter are , and n is an integer greater than 1.

[0107] Here, the actual value compensation value R' m ' ,n and the theoretical value R m,n The error between them will be smaller, that is, satisfying R' m ' ,n -R m,n <R' m,n -R m,n , that is, the recall rate of the compensated vector retrieval system will be closer to the actual situation.

[0108] Here, after obtaining the expression of the compensation value of the actual value, the first compensation parameter and the second compensation parameter can be determined according to the expression of the compensation value of the actual value, that is, Denoted as the first compensation parameter (can also be described as the error coefficient), It is recorded as the second compensation parameter (it can also be described as the error offset).

[0109] Step 304: Use multiple first compensation parameters in the first compensation parameter set to compensate for multiple first results output by the storage and calculation array in the vector retrieval system, and use multiple second compensation parameters in the second compensation parameter set to compensate for multiple second results output by the processor in the vector retrieval system.

[0110] Among them, the multiple first results output by the storage and calculation array are obtained by performing dot product operations on each base vector in the base vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to a first result of the dot product operation between a base vector and the multiple test vectors; the multiple second results output by the processor are obtained by performing a modulo correlation operation on each base vector in the base vector set; and one second compensation parameter corresponds to one second result.

[0111] In some embodiments, using the plurality of first compensation parameters in the first compensation parameter set to compensate the plurality of first results output by the storage and calculation array in the vector search system includes:

[0112] For each first compensation parameter of the plurality of first compensation parameters, the following operations are performed: multiplying the first compensation parameter by the corresponding first result to compensate the first result.

[0113] For example, the first compensation parameters are respectively It indicates that, Corresponding to the first base vector in the base vector set The first compensation parameter of Corresponding to the nth base vector in the base vector set The first compensation parameter. Take the first base vector As an example, the first results of the dot product operation between the base vector and multiple test vectors in the test vector set are respectively used It indicates that, Represents the first base vector With the first test vector Perform dot product operations, and so on. Represents the first base vector With the nth test vector Perform dot product operation, where n is an integer greater than 1; Corresponding to the first base vector The first result of dot product operation with multiple test vectors is obtained by and Multiply, and Multiply, and so on, and Multiply by to achieve compensation for the first result. Other first compensation parameters and The compensation process is similar.

[0114] In some embodiments, using the plurality of second compensation parameters in the second compensation parameter set to compensate the plurality of second results output by the processor in the vector search system includes:

[0115] For each second compensation parameter of the plurality of second compensation parameters, the following operations are performed: the second compensation parameter is added to the corresponding second result to compensate for the second result.

[0116] For example, the second compensation parameters are respectively express, Corresponding to the first base vector in the base vector set The second compensation parameter, and so on, Corresponding to the nth base vector in the base vector set The second compensation parameter of the first base vector The corresponding second result is c 1 Indicates that, by analogy, the nth base vector The corresponding second result is cn Indicates that a second compensation parameter corresponds to a second result, so that by With c 1 Add, With c 2 Add, and so on, c n and Adding together realizes compensation of multiple second results.

[0117] In some embodiments, after using a plurality of first compensation parameters in the first compensation parameter set to compensate a plurality of first results output by a storage and calculation array in a vector search system, and using a plurality of second compensation parameters in the second compensation parameter set to compensate a plurality of second results output by a processor in the vector search system, the method further comprises:

[0118] Re-determining the recall rate of the test vector set;

[0119] Determining whether the recall rate of the test vector set is less than or equal to the first preset threshold;

[0120] If the recall rate of the test vector set is less than or equal to the first preset threshold, the base database data is rewritten; the base database data is used to be converted into a base database vector in the base database vector set.

[0121] Here, rewriting the base database data may refer to converting the base database data such as documents, images, audio, etc. back into vectors in a high-dimensional space, writing the base database vectors into the storage array of the vector retrieval system, and calculating the offset constants and writing the offset constants into the processor (CPU) of the vector retrieval system.

[0122] In the embodiment of the present application, the following advantages are possessed:

[0123] (1) determining a recall rate of the test vector set; and determining a first compensation parameter set and a second compensation parameter set when the recall rate of the test vector set is less than or equal to a first preset threshold.

[0124] In an embodiment of the present application, multiple first compensation parameters in the first compensation parameter set are used to compensate for multiple first results output by the storage and computing array in the vector retrieval system, and multiple second compensation parameters in the second compensation parameter set are used to compensate for multiple second results output by the processor in the vector retrieval system. Compared with the method of rewriting the base database data when the recall rate is lower than a threshold in the related art, there is no need to spend a lot of time and energy to rewrite the base database data, which improves the efficiency of error calibration of the vector retrieval system and ensures the retrieval accuracy of the vector retrieval system.

[0125] (2) A calibration method for a vector retrieval system based on a storage and computing array + processor (CPU) is proposed. First, the recall rate of the vector retrieval system is tested periodically. When its recall rate does not meet the application requirements, the actual calculation results and theoretical true results of the test vector set and the base library vector set on the storage and computing array are fitted to finally obtain a first compensation parameter (which can also be described as an error coefficient) and a second compensation parameter (which can also be described as an error offset). The second compensation parameter is superimposed on the offset constant in the CPU to update and complete the offset error compensation, and the first compensation parameter is used to compensate the error coefficient of the first result output by the storage and computing array. There is no need to spend a lot of time and energy to rewrite the base library data, and the recall rate calibration of the vector retrieval system is achieved without the need for additional storage and computing units.

[0126] See also Figure 4 , Figure 4 This is a schematic diagram of a specific implementation flow of the data processing method of the present application embodiment, as shown in FIG. Figure 4 As shown, the method includes steps 401 to 409:

[0127] Step 401: Acquire a test vector set and a base library vector set.

[0128] Here, the base library vector set may refer to a set consisting of multiple base library vectors.

[0129] Here, the base database vector refers to converting base database data such as documents, images, audio, etc. into vectors in a high-dimensional space and storing them in a vector retrieval system.

[0130] Here, the vector retrieval system is used to retrieve at least one vector closest to the query vector from the multiple base vectors included in the base vector set.

[0131] Here, the vector retrieval system may include a processor (CPU) and a storage and computing array.

[0132] Here, the test vector set may refer to a set consisting of multiple test vectors; wherein the test vector may refer to a constructed vector whose similarity with the base library vector is greater than a preset threshold. The test vector may also be described as a query vector, a target vector, etc.

[0133] Step 402: Calculate an offset constant according to the base library vector set, and store the offset constant in a processor (CPU) of the vector retrieval system.

[0134] Here, the base data such as documents, images, audios, etc. are converted into vectors in a high-dimensional space to obtain a plurality of base vectors; and the plurality of base vectors are combined to form the base vector set.

[0135] Here, the offset constant can be calculated according to the following formula, that is, each base vector in the base vector set is subjected to a modulo correlation operation to obtain each offset constant, as follows:

[0136]

[0137] Among them, C represents the offset constant set, which consists of n offset constants, X represents the base library vector set, and It means that the base vector set consists of n base vectors, where n is an integer greater than 1.

[0138] Step 403: Write the base vectors in the base vector set into the storage array of the vector retrieval system.

[0139] Here, after the vector search system completes writing the base library vector and the offset constant into the storage array and the processor (CPU) respectively and the vector search system is put into use, steps 404 to 409 are executed.

[0140] Step 404: Determine the recall rate of the test vector set according to a preset period.

[0141] Here, recall can refer to an important indicator for measuring the relevance of vectors returned by a vector retrieval system.

[0142] Here, the recall rate of the vector retrieval system is tested periodically.

[0143] Here, due to the non-ideal characteristics of the storage and computing array (which can also be described as a storage and computing integrated device), the element values ​​(i.e., the base library vectors) in the written base library vector set will change, thereby affecting the application recall rate of the vector retrieval system. Therefore, according to some characteristics of the application scenario, such as fewer application requests at night, the actual application recall rate of the vector retrieval system can be periodically tested using a test vector set according to a preset cycle (for example, starting at 0:00 p.m. and lasting for 24 hours as a time period), so as to monitor the recall rate of the retrieval system without affecting user use as much as possible. When the test recall rate is lower than the application threshold, the subsequent error compensation process is carried out, otherwise it continues to run and use.

[0144] Here, the recall of the test vector set is determined, including:

[0145] Determine the similarity between each test vector in the test vector set and each base vector in the base vector set to obtain a similarity result;

[0146] Determine, according to the similarity result, a first number of measurement vectors in the test vector set whose similarity to the base library vector is greater than or equal to a second preset threshold;

[0147] The recall rate of the test vector set is determined according to the first number and a second number of all test vectors in the test vector set; the recall rate represents a ratio of the first number to the second number.

[0148] Here, the test vector set can be used It indicates that the test vector set consists of m test vectors, where m is an integer greater than 1.

[0149] Here, the base vector set is It indicates that the base vector set consists of n base vectors, where n is an integer greater than 1.

[0150] For example, assuming that the test vector set includes m=3 test vectors, represented by test vector 1, test vector 2, and test vector 3 respectively, and the base vector set includes n=2 base vectors, represented by base vector 1 and base vector 2 respectively, then according to the preset period, the similarity between each test vector in the test vector set and each base vector in the base vector set is determined, that is, the similarity between test vector 1 and base vector 1 is calculated to obtain similarity result 1; the similarity between test vector 1 and base vector 2 is calculated to obtain similarity result 2; the similarity between test vector 2 and base vector 1 is calculated Calculate the similarity to obtain similarity result 3; calculate the similarity between test vector 2 and base library vector 2 to obtain similarity result 4; calculate the similarity between test vector 3 and base library vector 1 to obtain similarity result 5; calculate the similarity between test vector 3 and base library vector 2 to obtain similarity result 6; if among these 6 similarity results, the similarity between test vector 1 and base library vector 1 is greater than the second preset threshold, and the similarity between test vector 2 and base library vector 2 is greater than the second preset threshold, then determine that the first number is 2; calculate the ratio of the first number, i.e. 2, to the second number, i.e. 3, to obtain the recall rate.

[0151] Step 405: determine whether the recall rate is lower than a first preset threshold; if the recall rate is lower than the first preset threshold, execute step 406; otherwise, the vector retrieval system is put into use.

[0152] Here, if the recall rate is lower than the preset first threshold, it indicates that the element value of the base vector stored in the storage array in the vector retrieval system has changed, and error compensation is required.

[0153] Here, the recall rate test of the vector retrieval system is completed based on the test vector set. If the recall rate is lower than the application threshold, it means that the current base database data is not written accurately in the system and needs to be rewritten and fine-tuned until the recall rate meets the application requirements and is put into use.

[0154] Here, in the prior art, for the current vector retrieval system, due to the non-ideal characteristics of the storage and computing array, the numerical values ​​stored therein are offset, resulting in a decrease in the system recall rate. Users provide feedback on the system application problems, and then the system recall rate is verified based on the test vector set. If there is indeed a problem of decreased recall rate, the base database data is rewritten for calibration.

[0155] Step 406: Determine a first compensation parameter set and a second compensation parameter set.

[0156] Here, the multiple first compensation parameters in the first compensation parameter set are used to compensate for the multiple first results output by the storage and calculation array in the vector retrieval system; the input of the storage and calculation array is the test vector set and the base library vector set; the multiple first results output by the storage and calculation array are obtained by performing dot product operations on each base library vector in the base library vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to a first result of dot product operations on a base library vector and the multiple test vectors.

[0157] Here, the multiple second compensation parameters in the second compensation parameter set are used to compensate for the multiple second results output by the processor in the vector retrieval system; the input of the processor is the base vector set; the multiple second results output by the processor are obtained by performing a modular correlation operation on each base vector in the base vector set; one second compensation parameter corresponds to one second result.

[0158] Here, the error of the base vector in the storage array can be calibrated by fitting the data.

[0159] In this step 406, a test vector set is used Vector collection of backgrounds The first compensation parameter set and the second compensation parameter set are determined.

[0160] Specifically, it includes:

[0161] First, use each test vector in the test vector set as input, and perform vector dot product (or described as dot multiplication) operation with each base vector in the base vector set to obtain multiple operation results; the multiple operation results are used as multiple theoretical values ​​(which can also be described as theoretical true values).

[0162] Secondly, the test vector set and the base library vector set are used as actual inputs of the storage and computing array in the vector retrieval system to obtain multiple first results on the storage and computing array (which can also be described as a storage and computing chip); and the multiple first results are used as multiple actual values ​​(which can also be described as actual calculation results).

[0163] Next, a fitting operation is performed on the multiple theoretical values ​​and the multiple actual values ​​to obtain multiple fitting functions; the fitting function includes a first fitting parameter and a second fitting parameter; and the fitting function corresponds to a fitting operation of a base library vector.

[0164] Finally, the first compensation parameter set and the second compensation parameter set are determined according to the multiple fitting functions.

[0165] See also Figure 5 , Figure 5 is a schematic diagram of obtaining the first fitting parameter and the second fitting parameter in the embodiment of the present application, such as Figure 5 As shown, for any base vector in the base vector set, multiple first theoretical values ​​and multiple first actual values ​​can be obtained together with multiple test vectors in the test vector set. In this way, the multiple first actual values ​​are used as the horizontal coordinates of the points before fitting, and the multiple first theoretical values ​​are used as the vertical coordinates of the points before fitting, and linear fitting is performed to obtain a linear fitting result, that is, a fitting function. The fitting function includes a first fitting parameter and a second fitting parameter. This can be done by analogy to obtain the fitting functions of all base vectors.

[0166] Here, multiple first fitting parameters are represented by K = {k1, k2, ..., k n}, multiple second fitting parameters are represented by B = {b1, b2, ..., b n}, where k1 and b1 are the two fitting parameters of the first base vector in the base vector set, and so on. n and b n are two fitting parameters corresponding to the nth base vector in the base vector set, where n is an integer greater than 1.

[0167] Here, the nth base vector is Taking the example of FIG. 1 as an example, the process of obtaining the first compensation parameter and the second compensation parameter is described as follows:

[0168] First, for the nth base vector The base library vector and each test vector in the test vector set are respectively subjected to dot product operations to obtain a plurality of first theoretical values ​​(which can also be described as theoretical true values). The base library vector and each test vector in the test vector set are used as inputs of a storage and calculation array in the vector retrieval system to obtain a plurality of first actual values ​​output by the storage and calculation array.

[0169] Then, taking the multiple first actual values ​​as the vertical coordinates and the multiple first theoretical values ​​as the horizontal coordinates, linear fitting is performed on the multiple first actual values ​​and the multiple first theoretical values ​​on the storage and computing array (which can also be described as a storage and computing integrated chip) to obtain a fitting function, which includes a first fitting parameter and a second fitting parameter.

[0170] Here, the fitting function can be: R m,n =k n ×R' m,n +b n , R' m,n Indicates the actual value, R m,n represents the theoretical value, k n represents the first fitting parameter, b n Indicates that the second fitting parameter is.

[0171] Next, based on the fitting function, an expression for the compensation value of the actual value is determined.

[0172] Here, by fitting the function, replacing the actual value in the fitting function with the compensation value of the actual value, and replacing the theoretical value in the fitting function with the actual value, the expression of the compensation value of the actual value can be obtained as follows:

[0173]

[0174] Among them, R' m ' ,n Indicates the compensation value of the actual value, R' m,n represents the actual value, k n and b n is the nth base vector in the corresponding base vector set The first fitting parameter and the second fitting parameter are , and n is an integer greater than 1.

[0175] Here, the actual value compensation value R' m ' ,n and the theoretical value R m,n The error between them will be smaller, that is, satisfying R' m ' ,n -R m,n <R' m,n -R m,n , that is, the recall rate of the compensated vector retrieval system will be closer to the actual situation.

[0176] Here, after obtaining the expression of the compensation value of the actual value, the first compensation parameter and the second compensation parameter can be determined according to the expression of the compensation value of the actual value, that is, Denoted as the first compensation parameter (can also be described as the error coefficient), It is recorded as the second compensation parameter (which can also be described as the error offset). At this point, the error coefficient and error offset on the storage and computing array (which can also be described as a storage and computing integrated chip) are obtained through linear fitting.

[0177] Step 407: using the plurality of first compensation parameters in the first compensation parameter set, to compensate the plurality of first results output by the storage and calculation array in the vector search system.

[0178] Here, in order to compensate for the error of the storage and computing array (which can also be described as a storage and computing integrated chip) in the vector retrieval system, a plurality of first compensation parameters (which can also be described as linear error coefficients) in the first compensation parameter set, namely The first result obtained by the original multiplication and addition operation is converted into a digital signal by analog-to-digital conversion (ADC) and then directly output to the CPU. In the embodiment of the present application, multiple first compensation parameters (which can also be described as linear error compensation coefficients) are stored in the storage and calculation array device. The first result is deployed on the corresponding storage and computing array device before the analog-to-digital conversion is performed, so that the first result output by the storage and computing array is converted The error coefficient compensation of linear error is achieved.

[0179] Step 408: Use the multiple second compensation parameters in the second compensation parameter set to compensate the multiple second results output by the processor in the vector search system.

[0180] Here, in order to compensate for the error offset of the storage and computing array (also described as a storage and computing integrated chip) in the vector retrieval system, multiple second compensation parameters (also described as error offsets) in the second compensation parameter set are converted into Add the original offset constant (i.e. c 1 ,c 2 ,...,c n ) is stored in the CPU storage unit. In this way, the digital signal that has completed the linear error compensation will complete the offset constant superposition calculation process in the CPU calculation unit to complete the compensation of the error offset.

[0181] See also Figure 6 , Figure 6 is a schematic diagram of compensating for multiple first results and multiple second results in an embodiment of the present application, such as Figure 6 As shown, a plurality of first compensation parameters are respectively used It indicates that, Corresponding to the first base vector in the base vector set The first compensation parameter of Corresponding to the nth base vector in the base vector set The first compensation parameter. Take the first base vector As an example, the first results of the dot product operation between the base vector and multiple test vectors in the test vector set are respectively used It indicates that, Represents the first base vector With the first test vector Perform dot product operations, and so on. Represents the first base vector With the nth test vector Perform dot product operation, where n is an integer greater than 1; Corresponding to the first base vector The first result of dot product operation with multiple test vectors is obtained by and Multiply, and Multiply, and so on, and Multiply by to achieve compensation for the first result. Other first compensation parameters and The compensation process is similar.

[0182] like Figure 6 As shown, the second compensation parameters are respectively used express, Corresponding to the first base vector in the base vector set The second compensation parameter, and so on, Corresponding to the nth base vector in the base vector set The second compensation parameter of the first base vector The corresponding second result (offset constant) is expressed in c 1 Indicates that, by analogy, the nth base vector The corresponding second result is c n Indicates that a second compensation parameter corresponds to a second result, so that by With c 1 Add, With c 2 Add, and so on, c n and Adding together realizes compensation of multiple second results.

[0183] Step 409: Re-determine the recall rate of the test vector set.

[0184] Here, after using multiple first compensation parameters in the first compensation parameter set to compensate for multiple first results output by the storage and computing array in the vector retrieval system, and using multiple second compensation parameters in the second compensation parameter set to compensate for multiple second results output by the processor in the vector retrieval system, the recall rate of the test vector set is re-determined to determine whether the recall rate of the test vector set is less than or equal to the first preset threshold; if the recall rate of the test vector set is less than or equal to the first preset threshold, the base database data is rewritten; the base database data is used to convert into base database vectors in the base database vector set.

[0185] Here, rewriting the base database data may refer to reconverting the base database data such as documents, images, audios, etc. into vectors in a high-dimensional space, and writing the base database vectors into the storage array of the vector retrieval system.

[0186] Here, if the recall rate of the test vector set is greater than the first preset threshold, the vector retrieval system is put into use. After the deployment or calibration time of the base library vector is greater than the preset period T, the recall rate of the test vector set can be calculated again.

[0187] Here, after the error compensation is completed in the vector retrieval system implemented by the integrated storage and computing chip and the processor (CPU), it may not be directly put into use. The compensation value R' after the error compensation is completed m ' ,n is the actual value R' compared to the calculation output of the storage array chip m,n The actual calculation result is the theoretical value R m,n The error is smaller, that is, R' m ' ,n -R m,n <R' m,n -R m,n However, it cannot guarantee that the recall rate of the vector retrieval system will meet the application requirements after error compensation. Therefore, after completing the error compensation, it is necessary to use a test vector set to test the recall rate of the vector retrieval system. It will only be put into use when the recall rate meets the application requirements. Otherwise, it means that the recall rate requirements of the application cannot be met by introducing the error compensation value. In this case, the base library is rewritten to ensure the recall rate of the application system so that the vector retrieval system meets the application requirements.

[0188] In this example, the following advantages are achieved:

[0189] (1) A calibration method for a vector retrieval system based on a storage and computing array + processor (CPU) is proposed. The method can compensate for the reduced recall rate of the vector retrieval system caused by the non-ideal characteristics of the storage and computing array. By fitting the linear errors between the actual calculation results of the test vector set and the base vector set on the storage and computing array and the theoretical true results, a first compensation parameter (which can also be described as an error coefficient) and a second compensation parameter (which can also be described as an error offset) are finally obtained. This method can realize the compensation of the error coefficient and error offset of the linear error without the need for additional storage and computing units, and will not increase the computing delay.

[0190] (2) Compared with the existing method of calibrating the recall rate by rewriting the underlying database data set after users report that the application recall rate is poor, in the embodiment of the present application, the recall rate monitoring and calibration of the vector retrieval system can be achieved online through steps such as periodic judgment, error fitting, and error compensation, thereby ensuring the long-term availability of the application service and preventing uncontrollable application interruptions.

[0191] (3) The error compensation method in the embodiment of the present application can realize the compensation of the error parameter and error offset of the linear error without the need for additional storage and computing units, and will not increase the computing delay. It avoids the energy consumption overhead caused by frequently rewriting the base database data for recall rate calibration, and ensures the computing efficiency of the vector retrieval system.

[0192] In order to implement the data processing method of the embodiment of the present application, the embodiment of the present application also provides a data processing device, which is arranged on a server. Figure 7 Schematic diagram of the structure of the data processing device according to the embodiment of the present application. Figure 7 As shown, the device comprises:

[0193] An acquisition module 71 is used to acquire a test vector set and a base vector set;

[0194] The processing module 72 is used to determine the recall rate of the test vector set; when the recall rate of the test vector set is less than or equal to the first preset threshold, determine the first compensation parameter set and the second compensation parameter set; use the multiple first compensation parameters in the first compensation parameter set to compensate the multiple first results output by the storage array in the vector retrieval system, and use the multiple second compensation parameters in the second compensation parameter set to compensate the multiple second results output by the processor in the vector retrieval system; wherein the multiple first results output by the storage array are obtained by performing dot product operations on each base vector in the base vector set and the multiple test vectors in the test vector set; one first compensation parameter corresponds to the first result of the dot product operation of one base vector and the multiple test vectors; the multiple second results output by the processor are obtained by performing modulo correlation operations on each base vector in the base vector set; and one second compensation parameter corresponds to one second result.

[0195] In some embodiments, the processing module 72 is specifically configured to:

[0196] According to a preset period, a recall rate of the test vector set is determined.

[0197] In addition, according to at least one embodiment of the present application, determining the recall rate of the test vector set includes:

[0198] Determine the similarity between each test vector in the test vector set and each base vector in the base vector set to obtain a similarity result;

[0199] Determine, according to the similarity result, a first number of measurement vectors in the test vector set whose similarity to the base library vector is greater than or equal to a second preset threshold;

[0200] The recall rate of the test vector set is determined according to the first number and a second number of all test vectors in the test vector set; the recall rate represents a ratio of the first number to the second number.

[0201] In some embodiments, the processing module 72 is specifically configured to:

[0202] Performing dot product operations on each test vector in the test vector set and each base vector in the base vector set to obtain multiple operation results; and using the multiple operation results as multiple theoretical values;

[0203] Using the test vector set and the base vector set as inputs of a storage and calculation array in the vector retrieval system, obtaining a plurality of first results output by the storage and calculation array; using the plurality of first results as a plurality of actual values;

[0204] Performing a fitting operation on the plurality of theoretical values ​​and the plurality of actual values ​​to obtain a plurality of fitting functions; the fitting function includes a first fitting parameter and a second fitting parameter; the fitting function corresponds to a fitting operation of a base library vector;

[0205] The first compensation parameter set and the second compensation parameter set are determined according to the multiple fitting functions.

[0206] In some embodiments, the processing module 72 is specifically configured to:

[0207] For each of the plurality of fitting functions, determining an expression of a compensation value of the actual value according to the corresponding fitting function; and determining a first compensation parameter and a second compensation parameter corresponding to the corresponding fitting function according to the expression of the compensation value;

[0208] The first compensation parameter set is obtained according to the first compensation parameters respectively corresponding to the multiple fitting functions, and the second compensation parameter set is obtained according to the second compensation parameters respectively corresponding to the multiple fitting functions.

[0209] In some embodiments, the processing module 72 is specifically configured to:

[0210] For each first compensation parameter of the plurality of first compensation parameters, the following operations are performed:

[0211] The first compensation parameter is multiplied by the corresponding first result to compensate the first result.

[0212] In some embodiments, the processing module 72 is specifically configured to:

[0213] For each second compensation parameter of the plurality of second compensation parameters, the following operations are performed:

[0214] The second compensation parameter is added to the corresponding second result to compensate the second result.

[0215] In some embodiments, the processing module 72 is further configured to:

[0216] Re-determining the recall rate of the test vector set;

[0217] Determining whether the recall rate of the test vector set is less than or equal to the first preset threshold;

[0218] If the recall rate of the test vector set is less than or equal to the first preset threshold, the base database data is rewritten; the base database data is used to be converted into a base database vector in the base database vector set.

[0219] In actual application, the acquisition module 71 can be implemented by a communication interface in a data processing device; and the processing module 72 can be implemented by a processor in the data processing device.

[0220] It should be noted that: when the data processing device provided in the above embodiment performs data processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0221] The present application also provides a server, such as Figure 8 As shown, including:

[0222] Communication interface 81, capable of exchanging information with other devices;

[0223] The processor 82 is connected to the communication interface 81 and is used to execute the method provided by one or more technical solutions on the server side when running the computer program. The computer program is stored in the memory 83.

[0224] It should be noted that the specific processing process of the processor 82 and the communication interface 81 is detailed in the method embodiment and will not be repeated here.

[0225] Of course, in actual application, the various components in the server 80 are coupled together through the bus system 88. It is understood that the bus system 88 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 88 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as bus system 88 .

[0226] The memory 83 in the embodiment of the present application is used to store various types of data to support the operation of the server 80. Examples of such data include: any computer program used to operate on the server 80.

[0227] The method disclosed in the above embodiment of the present application can be applied to the processor 82, or implemented by the processor 82. The processor 82 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 82. The above-mentioned processor 82 may be a general-purpose processor, a digital data processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 82 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 83, and the processor 82 reads the information in the memory 83 and completes the steps of the above method in combination with its hardware.

[0228] In an exemplary embodiment, the server 80 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0229] It can be understood that the memory (memory 83) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0230] In an exemplary embodiment, the present invention also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory storing a computer program, and the computer program can be executed by the processor 82 of the server 80 to complete the steps of the aforementioned server-side method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0231] Illustratively, an embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 82 of the server 80 to complete the steps of any of the aforementioned methods.

[0232] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0233] In addition, the technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

[0234] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain a test vector set and a base library vector set; Determining a recall rate of the test vector set; When the recall rate of the test vector set is less than or equal to a first preset threshold, determining a first compensation parameter set and a second compensation parameter set; Using a plurality of first compensation parameters in the first compensation parameter set, compensating a plurality of first results output by a storage and calculation array in a vector retrieval system, and using a plurality of second compensation parameters in the second compensation parameter set, compensating a plurality of second results output by a processor in the vector retrieval system; Among them, the multiple first results output by the storage and calculation array are obtained by performing dot product operations on each base vector in the base vector set and multiple test vectors in the test vector set; one first compensation parameter corresponds to a first result of the dot product operation between a base vector and the multiple test vectors; the multiple second results output by the processor are obtained by performing a modulo correlation operation on each base vector in the base vector set; and one second compensation parameter corresponds to one second result.

2. The method according to claim 1, characterized in that The determining the recall rate of the test vector set includes: According to a preset period, a recall rate of the test vector set is determined.

3. The method according to claim 1 or 2, characterized in that: The determining the recall rate of the test vector set includes: Determine the similarity between each test vector in the test vector set and each base vector in the base vector set to obtain a similarity result; Determine, according to the similarity result, a first number of measurement vectors in the test vector set whose similarity to the base library vector is greater than or equal to a second preset threshold; The recall rate of the test vector set is determined according to the first number and a second number of all test vectors in the test vector set; the recall rate represents a ratio of the first number to the second number.

4. The method according to claim 1, characterized in that: The determining of the first compensation parameter set and the second compensation parameter set comprises: Performing dot product operations on each test vector in the test vector set and each base vector in the base vector set to obtain multiple operation results; and using the multiple operation results as multiple theoretical values; Using the test vector set and the base vector set as inputs of a storage and calculation array in the vector retrieval system, obtaining a plurality of first results output by the storage and calculation array; using the plurality of first results as a plurality of actual values; Performing a fitting operation on the plurality of theoretical values ​​and the plurality of actual values ​​to obtain a plurality of fitting functions; the fitting function includes a first fitting parameter and a second fitting parameter; the fitting function corresponds to a fitting operation of a base library vector; The first compensation parameter set and the second compensation parameter set are determined according to the multiple fitting functions.

5. The method according to claim 4, characterized in that The determining, according to the multiple fitting functions, the first compensation parameter set and the second compensation parameter set comprises: For each of the plurality of fitting functions, determining an expression of a compensation value of the actual value according to the corresponding fitting function; and determining a first compensation parameter and a second compensation parameter corresponding to the corresponding fitting function according to the expression of the compensation value; The first compensation parameter set is obtained according to the first compensation parameters respectively corresponding to the multiple fitting functions, and the second compensation parameter set is obtained according to the second compensation parameters respectively corresponding to the multiple fitting functions.

6. The method according to claim 1, characterized in that The method of using the plurality of first compensation parameters in the first compensation parameter set to compensate the plurality of first results output by the storage and calculation array in the vector retrieval system includes: For each first compensation parameter of the plurality of first compensation parameters, the following operations are performed: The first compensation parameter is multiplied by the corresponding first result to compensate the first result.

7. The method according to claim 1, characterized in that The compensating the plurality of second results output by the processor in the vector search system by using the plurality of second compensation parameters in the second compensation parameter set includes: For each second compensation parameter of the plurality of second compensation parameters, the following operations are performed: The second compensation parameter is added to the corresponding second result to compensate the second result.

8. The method according to claim 1, characterized in that After compensating a plurality of first results output by a storage and calculation array in a vector retrieval system using a plurality of first compensation parameters in the first compensation parameter set, and compensating a plurality of second results output by a processor in the vector retrieval system using a plurality of second compensation parameters in the second compensation parameter set, the method further includes: Re-determining the recall rate of the test vector set; Determining whether the recall rate of the test vector set is less than or equal to the first preset threshold; If the recall rate of the test vector set is less than or equal to the first preset threshold, the base database data is rewritten; the base database data is used to be converted into a base database vector in the base database vector set.

9. A data processing device, characterized in that: include: An acquisition module, used to acquire a test vector set and a base library vector set; A processing module, configured to determine a recall rate of the test vector set; In the case where the recall rate of the test vector set is less than or equal to the first preset threshold, determine a first compensation parameter set and a second compensation parameter set; use multiple first compensation parameters in the first compensation parameter set to compensate multiple first results output by the storage and calculation array in the vector retrieval system, and use multiple second compensation parameters in the second compensation parameter set to compensate multiple second results output by the processor in the vector retrieval system; wherein the multiple first results output by the storage and calculation array are obtained by performing dot product operations on each base vector in the base vector set and multiple test vectors in the test vector set; one of the first compensation parameters corresponds to a first result of dot product operations on a base vector and the multiple test vectors; and the multiple second results output by the processor are obtained by performing modulo correlation operations on each base vector in the base vector set; One second compensation parameter corresponds to one second result.

10. A server, characterized in that: comprising a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images