Data processing method and electronic equipment

Abstract

The invention discloses a data processing method and electronic equipment. The method comprises the steps of obtaining a first vector; the first vector is a high-dimensional vector; performing spatial distribution transformation on the first vector to obtain a second vector; based on the second vector, searching a third vector meeting a vector similarity condition in a target vector set; wherein the target vector set comprises a plurality of reconstruction vectors, the reconstruction vectors are obtained through spatial distribution transformation and vector reconstruction on the basis of original vectors in a database, and sub-vectors in the reconstruction vectors are low-dimensional vectors; and performing spatial distribution inverse transformation on the third vector to obtain a fourth vector.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method and an electronic device. Background Art

[0002] In the retrieval scheme for unstructured data, these unstructured data are usually converted into high-dimensional vectors, and data retrieval is achieved by searching for similar vectors. Summary of the Invention

[0003] In view of this, this application provides a data processing method and an electronic device, as follows:

[0004] A data processing method includes:

[0005] Obtain a first vector; the first vector is a high-dimensional vector;

[0006] Perform a spatial distribution transformation on the first vector to obtain a second vector;

[0007] Based on the second vector, search for a third vector in the target vector set that satisfies the vector similarity condition;

[0008] Wherein, the target vector set includes multiple reconstructed vectors, the reconstructed vectors are obtained through the spatial distribution transformation and vector reconstruction based on the original vectors in the database, and the sub-vectors in the reconstructed vectors are low-dimensional vectors;

[0009] Perform a spatial distribution inverse transformation on the third vector to obtain a fourth vector.

[0010] In the above method, preferably, searching for a third vector in the target vector set that satisfies the vector similarity condition based on the second vector includes:

[0011] Use a reconstruction code table to perform vector reconstruction on the second vector to obtain a fifth vector; the sub-vectors in the fifth vector are low-dimensional vectors;

[0012] Wherein, the reconstruction code table includes multiple sub-code tables, the sub-code tables include multiple center vectors; there is a one-to-one mapping between the sub-code tables and the sub-vectors in the fifth vector; the sub-vectors in the fifth vector are one of the center vectors in the corresponding sub-code table;

[0013] Based on the center distance between any two center vectors in the sub-code table, obtain a first distance between the fifth vector and the reconstructed vectors in the target vector set;

[0014] Determine the reconstructed vector in the target vector set whose first distance satisfies the vector similarity condition as the third vector.

[0015] Preferably, the above method for performing a spatial distribution transformation on the first vector to obtain a second vector includes:

[0016] Obtaining a first transfer vector for performing a single-step spatial distribution transformation on the first vector;

[0017] Using the first transfer vector to process the first vector to obtain a second vector.

[0018] Preferably, the above method for performing a spatial distribution transformation on the first vector to obtain a second vector includes:

[0019] Obtaining a second transfer vector for performing a one-step spatial distribution transformation on the first vector;

[0020] Using the second transfer vector to process the first vector to obtain an intermediate vector;

[0021] Taking the intermediate vector as the new first vector, and performing the step of obtaining the second transfer vector for performing a one-step spatial distribution transformation on the first vector until the number of times of the spatial distribution transformation reaches a target threshold to obtain a second vector.

[0022] Preferably, after performing a spatial distribution inverse transformation on the third vector to obtain a fourth vector, the above method further includes:

[0023] Obtaining a second distance between the first vector and the fourth vector;

[0024] Determining the fourth vector whose second distance satisfies the vector similarity condition as the new fourth vector.

[0025] A data processing method includes:

[0026] Obtaining an original vector in an original set, where the original vector is a high-dimensional vector;

[0027] Performing a spatial distribution transformation on the original vector to obtain a uniformly transformed vector;

[0028] Using a reconstruction code table to perform vector reconstruction on the uniformly transformed vector to obtain a reconstructed vector; sub-vectors in the reconstructed vector are low-dimensional vectors;

[0029] Wherein, the reconstruction code table includes a plurality of sub-code tables, each sub-code table includes a plurality of center vectors; there is a one-to-one mapping between the sub-code tables and the sub-vectors in the reconstructed vector; the sub-vectors in the reconstructed vector are one of the center vectors in the corresponding sub-code table;

[0030] Adding the reconstructed vector to a target vector set, and the reconstructed vectors in the target vector set are used for vector search.

[0031] Preferably, in the above method, the spatial distribution transformation and the vector reconstruction are implemented by an intelligent model;

[0032] wherein, the model parameters of the intelligent model are optimized based on at least a uniform loss value; the uniform loss value characterizes the degree of uniformity of the distribution of the vectors obtained by the intelligent model for spatial distribution transformation.

[0033] Preferably, in the above method, the intelligent model performs the vector reconstruction on the training samples, including:

[0034] Splitting the homogenized vector to obtain a plurality of sub-sample vectors; each of the sub-sample vectors corresponds to a sub-codebook in the reconstruction codebook; the homogenized vector is the vector obtained by subjecting the sample input vector in the training sample to spatial distribution change;

[0035] Obtaining the target similarity between the sub-sample vector and each central vector in the corresponding sub-codebook;

[0036] Multiplying the target similarity by the central vector in the corresponding sub-codebook to obtain a product vector;

[0037] Summing up all the product vectors corresponding to the sub-sample vector to obtain an integrated vector corresponding to the sub-sample vector;

[0038] Taking the integrated vector corresponding to each sub-sample vector as a sub-vector to obtain a reconstructed sample vector.

[0039] Preferably, in the above method, the method further includes:

[0040] Outputting a two-dimensional scatter plot; the arrangement shape between data points in the two-dimensional scatter plot characterizes the degree of uniform distribution from the original vector to the uniformly transformed vector.

[0041] An electronic device, comprising:

[0042] A memory, configured to store a computer program and data generated by running the computer program;

[0043] A processor, configured to execute the computer program to implement: obtaining a first vector; the first vector is a high-dimensional vector; performing a spatial distribution transformation on the first vector to obtain a second vector; based on the second vector, searching for a third vector that satisfies a vector similarity condition in a target vector set; wherein, the target vector set includes a plurality of reconstructed vectors, the reconstructed vectors are obtained by performing the spatial distribution transformation and vector reconstruction on the original vectors in a database, and the sub-vectors in the reconstructed vectors are low-dimensional vectors; performing a spatial distribution inverse transformation on the third vector to obtain a fourth vector.

[0044] As can be seen from the above technical solution, in a data processing method and an electronic device disclosed in this application, the high-dimensional vector to be searched is first subjected to a spatial distribution transformation, so that the obtained high-dimensional vector can be at least approximately uniformly distributed. Then, the vector that meets the vector similarity condition is searched from the set of reconstructed low-dimensional vectors using the at least approximately uniformly distributed high-dimensional vector. Finally, the searched vector is subjected to an inverse spatial distribution transformation, so that the searched high-dimensional vector can be obtained. It can be seen that in this application, through the spatial distribution transformation of the high-dimensional vector, the error existing in vector search due to non-uniform vector distribution is avoided, thereby improving the accuracy of vector search. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a flowchart of a data processing method provided by an embodiment of this application;

[0047] Figure 2 It is an example diagram of the spatial distribution transformation of vectors in an embodiment of this application;

[0048] Figure 3 It is an example diagram of the reconstructed vectors in an embodiment of this application;

[0049] Figure 4 It is a partial flowchart of a data processing method provided by an embodiment of this application;

[0050] Figure 5 It is an example diagram of the reconstruction of vectors in an embodiment of this application;

[0051] Figure 6 It is an example diagram of obtaining the vector distance based on the reconstructed codebook in an embodiment of this application;

[0052] Figure 7 It is another partial flowchart of a data processing method provided by an embodiment of this application;

[0053] Figure 8 It is a schematic diagram of the single-step spatial distribution transformation in an embodiment of this application;

[0054] Figure 9 It is an example diagram of a single non-linear module in an embodiment of this application;

[0055] Figure 10Another part of the flowchart of a data processing method provided by an embodiment of this application;

[0056] Figure 11 Schematic diagram of multi-step spatial distribution transformation in an embodiment of this application;

[0057] Figure 12 Example diagram of multiple non-linear modules in an embodiment of this application;

[0058] Figure 13 Another flowchart of a data processing method provided by an embodiment of this application;

[0059] Figure 14 Flowchart of another data processing method provided by an embodiment of this application;

[0060] Figure 15 Example diagram of vector reconstruction in an embodiment of this application;

[0061] Figure 16 Partial flowchart of another data processing method provided by an embodiment of this application;

[0062] Figure 17 Another partial flowchart of another data processing method provided by an embodiment of this application;

[0063] Figure 18 Another partial flowchart of another data processing method provided by an embodiment of this application;

[0064] Figure 19 Another partial flowchart of another data processing method provided by an embodiment of this application;

[0065] Figure 20 Example diagram of a scatter plot in an embodiment of this application;

[0066] Figure 21 Structural schematic diagram of a data processing device provided by an embodiment of this application;

[0067] Figure 22 Structural schematic diagram of another data processing device provided by an embodiment of this application;

[0068] Figure 23 Structural schematic diagram of an electronic device provided by an embodiment of this application;

[0069] Figure 24 Structural schematic diagram of another electronic device provided by an embodiment of this application;

[0070] Figure 25 Schematic diagram of an end-to-end learnable product quantization framework based on vector distribution transformation implemented by this application for AIPC;

[0071] Figure 26 This is a schematic diagram of the process for performing spatial distribution transformation and vector reconstruction on the reference vector and the process of vector query in this application. Detailed implementation manners

[0072] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0073] Refer to Figure 1 As shown, it is a flowchart of the implementation of a data processing method provided by an embodiment of the present application. This method can be applied to an electronic device capable of performing data processing, such as a computer or a server, etc. The technical solution in this embodiment is mainly used to improve the accuracy of vector search.

[0074] Specifically, the method in this embodiment may include the following steps:

[0075] Step 101: Obtain a first vector.

[0076] Among them, the first vector is a high-dimensional vector. The first vector can be vector data converted from unstructured data.

[0077] For example, taking the user's use of a large model for data search as an example, when the user inputs "What should I do if I have a headache" in the interaction interface of the large model, in this embodiment, the problem data "What should I do if I have a headache" input by the user is converted into vector data to obtain a high-dimensional first vector.

[0078] Step 102: Perform a spatial distribution transformation on the first vector to obtain a second vector.

[0079] Among them, in this embodiment, a spatial distribution transformation can be performed on the first vector based on distribution transformation parameters to obtain a second vector. The distribution change parameters here can make the distribution state between the vector subjected to the spatial distribution transformation and other vectors meet the uniform condition.

[0080] Specifically, the uniform condition can be: the parameter characterizing the uniformity between vectors is greater than or equal to the target threshold. Based on this, in this embodiment, the first vector can be converted into a second vector that can be evenly distributed with other vectors.

[0081] Step 103: Based on the second vector, search for a third vector that meets the vector similarity condition in the target vector set.

[0082] Among them, the target vector set includes multiple reconstruction vectors. The reconstruction vectors are obtained through spatial distribution transformation and vector reconstruction based on the original vectors in the database. The sub-vectors in the reconstruction vectors are low-dimensional vectors.

[0083] In a specific implementation, the original vectors in the database are high-dimensional vectors. In this embodiment, the original vectors in the database are pre-transformed in spatial distribution to obtain uniformly transformed vectors. For example, as Figure 2 shown, each dot represents a vector. The spatial distribution between the original vectors is non-uniform. After the spatial distribution transformation, the spatial distribution between the obtained uniformly transformed vectors is uniform. Then, in this embodiment, the uniformly transformed vectors are used to perform vector reconstruction using a reconstruction codebook to obtain reconstruction vectors, and these reconstruction vectors form the target vector set.

[0084] Among them, the reconstruction vectors are composed of multiple low-dimensional sub-vectors. And the reconstruction codebook includes multiple sub-codebooks, each sub-codebook contains multiple central vectors, and there is a one-to-one mapping between the sub-codebooks and the sub-vectors in the reconstruction vectors. As Figure 3 shown, the sub-vector in the reconstruction vector is a central vector in the corresponding sub-codebook.

[0085] Based on this, in this embodiment, the distance between the second vector and each reconstruction vector in the target vector set can be calculated to screen out the third vectors that meet the vector similarity condition. There can be one or more third vectors.

[0086] Step 104: Perform an inverse spatial distribution transformation on the third vector to obtain a fourth vector.

[0087] Among them, in this embodiment, the third vector can be inversely transformed in spatial distribution in the same way as the first vector is transformed in spatial distribution, so as to restore the third vector to the spatial distribution state before the spatial distribution transformation.

[0088] It can be seen from the above technical solutions that in a data processing method provided by an embodiment of the present application, first, the high-dimensional vector to be searched is transformed in spatial distribution, so that the obtained high-dimensional vector can at least be approximately uniformly distributed. Then, the high-dimensional vector that is at least approximately uniformly distributed is used to search for vectors that meet the vector similarity condition from the set composed of the reconstructed low-dimensional vectors. Finally, the searched vectors are inversely transformed in spatial distribution, so that the searched high-dimensional vectors can be obtained. It can be seen that in this embodiment, through the spatial distribution transformation of the high-dimensional vector, the error caused by the non-uniform distribution of the vectors during vector search is avoided, thereby improving the accuracy of vector search.

[0089] In one implementation manner, in step 103 during vector search, it can be implemented in the following manner, as Figure 4 shown:

[0090] Step 401: Use the reconstructed codebook to perform vector reconstruction on the second vector to obtain a fifth vector.

[0091] Among them, the sub-vectors in the fifth vector are low-dimensional vectors.

[0092] In a specific implementation, the reconstructed codebook includes multiple sub-codebooks, and each sub-codebook includes multiple central vectors; there is a one-to-one mapping between the sub-codebooks and the sub-vectors in the fifth vector; the sub-vectors in the fifth vector are a central vector in the corresponding sub-codebook.

[0093] In one implementation, step 401 can be implemented in the following manner:

[0094] First, split the second vector to obtain multiple initial vectors, and each initial vector corresponds to a sub-codebook in the reconstructed codebook. As Figure 5 shown, split the second vector into M initial vectors, and the initial vectors are low-dimensional vectors. M is a positive integer greater than or equal to 2.

[0095] Then, obtain the first similarity between the initial vector and each central vector in the corresponding sub-codebook. Here, the first similarity can be the cosine similarity or L2 similarity between the initial vector and the central vector.

[0096] Finally, according to the first similarity, in the sub-codebook corresponding to the initial vector, determine the corresponding central vector for the initial vector. For example, determine the central vector with the largest first similarity in the sub-codebook as the central vector corresponding to the initial vector. Based on this, use the central vector corresponding to each initial vector as the sub-vector to form the fifth vector.

[0097] Step 402: Based on the central distance between any two central vectors in the sub-codebook, obtain the first distance between the fifth vector and the reconstructed vectors in the target vector set.

[0098] It should be noted that the sub-vectors of each reconstructed vector in the target vector set are a central vector in the corresponding sub-codebook, and the sub-vectors in the fifth vector are a central vector in the corresponding sub-codebook. In this way, the reconstructed vector and the fifth vector each have a corresponding central vector in each sub-codebook, and the central distance between any two central vectors in each sub-codebook can be obtained in advance. In this embodiment, first, based on the central distance between any two central vectors in the sub-codebook, obtain the initial distance between the sub-vectors of the reconstructed vector and the fifth vector corresponding to the same sub-codebook. This initial distance is the distance between the central vectors corresponding to the sub-vectors of the reconstructed vector and the fifth vector corresponding to the same sub-codebook in the sub-codebook. Then, add up the initial distances corresponding to each sub-vector to obtain the first distance between the reconstructed vector and the fifth vector.

[0099] For example, as shown in Figure 3 and Figure 5 , there are M sub-code tables in the reconstructed code table, and the reconstructed vector and the fifth vector each have M sub-vectors. Each sub-vector of the reconstructed vector and the fifth vector corresponds to a central vector in one of the sub-code tables. Thus, for each sub-code table, there are central vector 1 and central vector 2. Central vector 1 corresponds to a sub-vector in the reconstructed vector, and central vector 2 corresponds to a sub-vector in the fifth vector. Central vector 1 and central vector 2 can be different central vectors, or central vector 1 and central vector 2 can be the same central vector. Based on this, as shown in Figure 6 , in this embodiment, for each sub-code table, the central distance between central vector 1 and central vector 2 is obtained as the initial distance between the sub-vectors of the reconstructed vector and the fifth vector corresponding to this sub-code table. Finally, these M initial distances are summed to obtain the first distance between the reconstructed vector and the fifth vector.

[0100] Step 403: Determine the reconstructed vector in the target vector set whose first distance satisfies the vector similarity condition as the third vector.

[0101] Among them, in this embodiment, the reconstructed vector in the target vector set whose vector distance from the second vector satisfies the vector similarity condition is used as the third vector.

[0102] Specifically, the vector similarity condition may include: a preliminary screening condition, and the preliminary screening condition may be: the first distances corresponding to the reconstructed vectors are sorted from small to large and are among the top L. L is a positive integer greater than or equal to 1. For example, the reconstructed vectors in the target vector set whose vector distances from the second vector are sorted among the top L are used as the third vectors.

[0103] In one implementation manner, when performing the spatial distribution transformation on the first vector in step 102, it can be implemented in the following manner, as shown in Figure 7 :

[0104] Step 701: Obtain the first transfer vector for the single-step spatial distribution transformation of the first vector.

[0105] For example, as shown in Figure 8 , in this embodiment, the first vector is converted into the second vector through a single-step spatial distribution transformation, and the first transfer vector is the spatial transfer vector for converting the first vector into the second vector. <>< /

[0106] Step 702: Use the first transfer vector to process the first vector to obtain the second vector.

[0107] Specifically, the single-step spatial distribution transformation can be implemented through a non-linear module, and the non-linear module is composed of a linear layer and an activation function.

[0108] For example, as Figure 9 shown, the non-linear module consists of linear layer 1, normalization layer 1, activation function GELU, linear layer 2, and normalization layer 2. Through the non-linear module, the first transfer vector for converting the first vector into the second vector can be obtained. The non-linear module can be a module for implementing spatial distribution transformation in the intelligent model. The non-linear module has model parameters such as a weight matrix and an activation function. The first transfer vector can be represented by formula (1):

[0109] v = W2σ(Norm(W1x)) (1)

[0110] where W1 and W2 are weight matrices, σ is an activation function such as GELU, Norm represents the normalization operation of the normalization layer such as LayerNorm, v is the first transfer vector, and x is the first vector.

[0111] Based on this, in this embodiment, the first vector and the first conversion vector can be added vectorially through formula (2) to obtain the second vector:

[0112]

[0113] where is the second vector.

[0114] In another implementation, when performing spatial distribution transformation on the first vector in step 102, it can be achieved in the following manner, as Figure 10 shown:

[0115] Step 1001: Obtain the second transfer vector for performing one-step spatial distribution transformation on the first vector.

[0116] Step 1002: Use the second transfer vector to process the first vector to obtain an intermediate vector.

[0117] Specifically, the second transfer vector at the t-th step can be represented by v t and v t can be obtained through formula (3):

[0118]

[0119] [[ID=4F]]Correspondingly, the intermediate vector obtained at the t-th step can be represented by formula (4):

[0120] [[ID=4F]]

[0121] Step 1003: Determine whether the number of spatial distribution transformations has reached the target threshold. If the number of spatial distribution transformations has not reached the target threshold, execute Step 1004; if the number of spatial distribution transformations has reached the target threshold, execute Step 1005.

[0122] Step 1004: Use the intermediate vector as the new first vector, return to execute Step 1001, and thus obtain the second transfer vector for one-step spatial distribution transformation of the new first vector;

[0123] Step 1005: Use the intermediate vector as the second vector.

[0124] It can be seen that in this embodiment, the first vector is sequentially subjected to multiple steps of spatial distribution transformation in an iterative manner until the number of spatial distribution transformations reaches the target threshold to obtain the second vector.

[0125] For example, as Figure 11 shown, in this embodiment, the first vector is converted into the second vector through T steps of spatial distribution transformation, and each step of spatial distribution transformation corresponds to a second transfer vector. For example, as Figure 12 shown, multiple steps of spatial distribution transformation can be implemented through multiple non-linear modules, and the structure of each non-linear module can be as Figure 9 shown, and the second vector after T steps of spatial distribution transformation can be expressed by formula (5):

[0126]

[0127] It can be seen that compared with the implementation method of obtaining the second vector through a single-step spatial distribution transformation, in this embodiment, considering that it is challenging to directly transfer the first vector to the second vector with a large offset in one step, the spatial distribution transformation is performed in multiple steps, which can reduce the implementation complexity and thus improve the efficiency of implementing the spatial distribution transformation.

[0128] In one implementation manner, after Step 104 in this embodiment, the following steps may further be included, as Figure 13 shown:

[0129] Step 105: Obtain the second distance between the first vector and the fourth vector.

[0130] In a specific implementation, in this embodiment, the cosine similarity or L2 similarity between the first vector and the fourth vector may be calculated to obtain the second distance between the first vector and the fourth vector.

[0131] Step 106: Determine the fourth vector whose second distance meets the vector similarity condition as the new fourth vector.

[0132] Among them, the vector similarity condition may include: a re-screening condition, and the re-screening condition may be: the second distances corresponding to the fourth vectors are sorted from small to large and are among the top K. K is a positive integer greater than or equal to 1. For example, in this embodiment, the fourth vectors with the second distances from small to large among the top K with respect to the first vector are used as the new fourth vectors.

[0133] It can be seen that in this embodiment, the calculation and rearrangement of vector distances are limited to a finite number of L vectors, which can effectively reduce the computational complexity of vector distance calculation, thereby significantly improving the vector screening performance.

[0134] Reference Figure 14 , is a flowchart of the implementation of a data processing method provided by an embodiment of the present application. This method can be applied to electronic devices capable of data processing, such as computers or servers. The technical solution in this embodiment is mainly used to improve the accuracy of vector search.

[0135] Specifically, the method in this embodiment may include the following steps:

[0136] Step 1401: Obtain the original vectors in the original set.

[0137] Among them, the original vectors are high-dimensional vectors. The original vectors in the original set are the original vectors in the database that serve as the basis for data search.

[0138] Step 1402: Perform a spatial distribution transformation on the original vectors to obtain uniformly transformed vectors.

[0139] Among them, in this embodiment, a spatial distribution transformation may be performed on the original vectors based on distribution transformation parameters to obtain uniformly transformed vectors. The distribution change parameters here can make the distribution states between the vectors subjected to the spatial distribution transformation and other vectors satisfy the uniform condition.

[0140] Specifically, the uniform condition may be: the parameter characterizing the uniformity between vectors is greater than or equal to the target threshold. Based on this, in this embodiment, the original vectors can be converted into uniformly transformed vectors that can be uniformly distributed with other vectors.

[0141] In one implementation, in step 1402, the third transfer vectors obtained by performing a single-step spatial distribution transformation on the original vectors may be obtained first, and then the third transfer vectors may be used to process the original vectors to obtain uniformly transformed vectors.

[0142] In another implementation, in step 1402, a fourth transfer vector for one-step spatial distribution transformation of the original vector can be obtained first; then, the fourth transfer vector is used to process the original vector to obtain an intermediate vector; taking the intermediate vector as the new original vector, a fourth transfer vector for one-step spatial distribution transformation of the original vector is obtained again until the number of times of spatial distribution transformation reaches the target threshold to obtain a uniform transformation vector.

[0143] Step 1403: Use the reconstruction codebook to perform vector reconstruction on the uniform transformation vector to obtain a reconstructed vector.

[0144] Among them, the sub-vectors in the reconstructed vector are low-dimensional vectors. The reconstruction codebook includes multiple sub-codebooks, and each sub-codebook includes multiple central vectors. There is a one-to-one mapping between the sub-codebooks and the sub-vectors in the reconstructed vector. The sub-vector in the reconstructed vector is a central vector in the corresponding sub-codebook.

[0145] In one implementation, in step 1403, when performing vector reconstruction on the uniform transformation vector, it can be implemented in the following manner:

[0146] First, split the uniform transformation vector to obtain multiple sub-initial vectors, and each sub-initial vector corresponds to a sub-codebook in the reconstruction codebook. As Figure 15 shown in, the uniform transformation vector is split into M sub-initial vectors, and the sub-initial vectors are low-dimensional vectors.

[0147] Then, obtain the second similarity between the sub-initial vector and each central vector in the corresponding sub-codebook. Here, the second similarity can be the cosine similarity or the L2 similarity between the sub-initial vector and the central vector.

[0148] For example, in this embodiment, the second similarity can be obtained through formula (6):

[0149]

[0150] Among them, is the m-th sub-initial vector in the uniform transformation vector corresponding to the i-th original vector, and c m is the central vector in the m-th sub-codebook, K is the number of clustering centers corresponding to the reconstruction codebook, that is, the number of central vectors in each sub-codebook; is the second similarity between the m-th sub-initial vector in the uniform transformation vector corresponding to the i-th original vector and the central vector.

[0151] Finally, according to the second similarity, in the sub-codebook corresponding to the sub-initial vector, determine the corresponding central vector for the sub-initial vector. For example, determine the central vector with the largest second similarity in the sub-codebook as the central vector corresponding to the sub-initial vector. Based on this, use the central vector corresponding to each sub-initial vector as a sub-vector to form a reconstructed vector.

[0152] For example, in this embodiment, the central vector corresponding to the sub-initial vector can be determined by formula (7):

[0153]

[0154] Wherein, is the second similarity after normalizing the second similarity; i m is the encoding of the central vector corresponding to the m-th sub-initial vector in the uniform transformation vector corresponding to the i-th original vector in the sub-codebook to which it belongs. Each central vector in the sub-codebook has a central encoding, such as 1 to M, which respectively represent the corresponding central vectors.

[0155] Step 1404: Add the reconstructed vector to the target vector set, and the reconstructed vectors in the target vector set are used for vector search.

[0156] As can be seen from the above solution, in a data processing method provided by an embodiment of the present application, before reconstructing the original vector, the original vector is first subjected to a spatial distribution transformation, so that the obtained vector can be at least approximately uniformly distributed. In this way, the at least approximately uniformly distributed vector is used to perform vector reconstruction through the reconstruction codebook. Thus, the vectors stored in the target vector set are low-dimensional vectors and are uniformly distributed vectors. In this way, when vector search is required, first perform a spatial distribution transformation, and then search based on these uniformly distributed and low-dimensional reconstructed vectors in the target vector set. Thus, in this embodiment, through the spatial distribution transformation of high-dimensional vectors, the error caused by uneven vector distribution during vector search can be avoided, thereby improving the accuracy of vector search.

[0157] In one implementation, the spatial distribution transformation and vector reconstruction in this embodiment can be implemented by an intelligent model. The intelligent model performs a spatial distribution transformation on the original vector based on model parameters such as distribution transformation parameters, and then performs vector reconstruction on the uniform transformation vector based on model parameters such as the reconstruction codebook to obtain a reconstructed vector and add it to the target vector set.

[0158] Specifically, the intelligent model is trained through training samples. During the training of the intelligent model, the model parameters of the intelligent model are at least optimized based on the uniform loss value. Specifically, the model parameters of the intelligent model are optimized based on the uniform loss value, the reconstruction loss value, and the contrast loss value.

[0159] Among them, the uniform loss value characterizes the degree of uniformity of the distribution of the vectors obtained by the intelligent model through spatial distribution transformation, that is, the degree of uniformity of the homogenized vectors obtained by the intelligent model through spatial distribution transformation of the sample input vectors in the sample transformation set. The sample transformation set includes: the homogenized vectors obtained by all sample input vectors through spatial distribution transformation by the intelligent model. The reconstruction loss value characterizes the vector distance between the sample input vector and the restored sample vector obtained by inverse spatial distribution transformation of the reconstructed output vector in the training samples of the intelligent model. The contrast loss value characterizes the degree of change of the reconstructed neighbor relationship of the restored sample vector with other vectors in the sample reconstruction set relative to the original neighbor relationship. The sample reconstruction set includes: the restored sample vectors corresponding to the reconstructed output vectors obtained by all sample input vectors processed by the intelligent model. The original neighbor relationship is the neighbor relationship between the sample input vector and other vectors in the training samples.

[0160] The following is a detailed description of training the intelligent model using the sample input vectors in the training samples, as Figure 16 shown in

[0161] Step 1601: Input any sample input vector in the training samples into the intelligent model.

[0162] Step 1602: Obtain the reconstructed output vector output by the intelligent model based on the sample input vector.

[0163] Among them, the intelligent model performs spatial distribution transformation on the sample input vector based on model parameters such as distribution transformation parameters to obtain a homogenized vector, and performs vector reconstruction on the homogenized vector based on model parameters such as a reconstruction codebook to obtain a reconstructed output vector.

[0164] Step 1603: Perform inverse spatial distribution transformation on the reconstructed output vector according to the model parameters to obtain a restored sample vector.

[0165] Step 1604: Obtain a loss value according to the restored sample vector and the sample input vector.

[0166] Among them, the loss value includes: a uniform loss value, a reconstruction loss value, and a contrast loss value.

[0167] In one implementation, the reconstruction loss value L in this embodiment can be obtained through formula (8) recon :

[0168]

[0169] Among them, n is the total number of sample input vectors in the training samples; x i is the i-th original vector, The restored sample vector obtained by the inverse spatial distribution transformation of the reconstructed output vector obtained by processing the i-th original vector through the intelligent model.

[0170] In one implementation, in this embodiment, the contrast loss value L can be obtained through formula (9) contra :

[0171]

[0172] where is a vector randomly selected from the restored sample vectors corresponding to the k + nearest neighbor vectors that satisfy the nearest neighbor condition in the training samples for the i-th sample input vector, is a vector randomly selected from the k - nearest neighbor vectors that satisfy the nearest neighbor condition in the sample reconstruction set for the restored sample vectors corresponding to the i-th sample input vector, removing the vectors that are repeated with . represents the original nearest neighbor relationship, represents the reconstructed nearest neighbor relationship. By constraining k + <k - , the negative samples selected from the training samples are made to be as far away from the positive samples as possible, and the samples are not too easy to distinguish from each other.

[0173] In one implementation, in this embodiment, the uniform loss value L can be obtained through formula (10) uniform :

[0174]

[0175] where [[ID=4�]]is the uniform vector obtained by the spatial distribution transformation of the i-th sample input vector through the intelligent model, is the vector that satisfies the nearest neighbor condition with the uniform vector corresponding to the i-th sample input vector in the sample transformation set.

[0176] Step 1605: Optimize at least the distribution transformation parameters and the reconstruction code table according to the loss value.

[0177] Specifically, in this embodiment, the uniform loss value, the reconstruction loss value, and the contrast loss value can be weighted and summed to obtain an integrated loss value, and then, according to the magnitude of the integrated loss value, the distribution transformation parameters and the reconstruction code table are optimized.

[0178] For example, in this embodiment, the integrated loss value L is obtained through the loss function of formula (11):

[0179] L = L recon + λ1L contra + λ2Luniform (11)

[0180] Among them, λ1 and λ2 are hyperparameters (which can also be called weights or coefficients) used to control the ratio between each term in the loss function, such as the uniform loss value, the reconstruction loss value, and the contrast loss value.

[0181] Based on the above implementation solution, the intelligent model reconstructs the vector of the training sample in the manner as Figure 17 shown below:

[0182] Step 1701: Split the uniform vector to obtain multiple sub-sample vectors.

[0183] Among them, each sub-sample vector corresponds to a sub-code table in the reconstruction code table; the uniform vector is the vector obtained by changing the spatial distribution of the sample input vector in the training sample.

[0184] Step 1702: Obtain the target similarity between the sub-sample vector and each central vector in the corresponding sub-code table.

[0185] Among them, the target similarity obtained in step 1702 can be the cosine similarity or the L2 similarity between the sub-sample vector and the central vector.

[0186] Step 1703: Multiply the target similarity by the central vector in its corresponding sub-code table to obtain a product vector.

[0187] Among them, in step 1703, the target similarity can be used as a weight to multiply the central vector in its corresponding sub-code table, thereby obtaining the product vector corresponding to each central vector in the sub-code table corresponding to the sub-sample vector.

[0188] Step 1704: Sum up all the product vectors corresponding to the sub-sample vector to obtain the integrated vector corresponding to the sub-sample vector.

[0189] Among them, in this embodiment, for each sub-sample vector, the product vectors corresponding to each central vector in its corresponding sub-code table are summed up, so that each sub-sample vector can correspond to an integrated vector.

[0190] Step 1705: Use the integrated vector corresponding to each sub-sample vector as a sub-vector to obtain the reconstructed sample vector.

[0191] Among them, the reconstructed sample vector is composed of the integrated vectors corresponding to each sub-sample vector.

[0192] It can be seen that in this embodiment, the target similarity between the sub-sample vector and each central vector in the corresponding sub-codebook is used as the weight, and the central vectors in the sub-codebook are weighted and summed to obtain the sub-vector in the reconstructed sample vector. Different from the method of obtaining the reconstructed sample vector by argmax in formula (7), in the training process of the intelligent model in this embodiment, since the argmax operation is non-differentiable, directly by multiplying with the sub-codebook to obtain an approximate reconstructed sample vector, so as to calculate the reconstruction loss for optimizing the intelligent model.

[0193] In one implementation, a two-dimensional scatter plot can also be output in this embodiment. In the two-dimensional scatter plot, the arrangement shape between data points represents the degree of uniformity of the transformation from the original vector to the uniform transformation vector.

[0194] Specifically, when the intelligent model performs a spatial distribution transformation on the sample input vector to obtain a uniform vector, it can be implemented in the following manner, as Figure 18 shown in:

[0195] Step 1801: Obtain the fifth transfer vector for one-step spatial distribution transformation of the sample input vector.

[0196] Step 1802: Use the fifth transfer vector to process the sample input vector to obtain an intermediate vector.

[0197] Step 1803: Determine whether the number of spatial distribution transformations has reached the target threshold. If the number of spatial distribution transformations has not reached the target threshold, execute Step 1804. If the number of spatial distribution transformations has reached the target threshold, execute Step 1805.

[0198] Step 1804: Use the intermediate vector as the new sample input vector, execute Step 1801, and obtain the fifth transfer vector for one-step spatial distribution transformation of the sample input vector again until the number of spatial distribution transformations reaches the target threshold to obtain a uniform vector.

[0199] Step 1805: Use the intermediate vector as the uniform vector.

[0200] And after each execution of Step 1802, a two-dimensional scatter plot can also be output in this embodiment. Specifically, the following processing can be performed, as Figure 19 shown in:

[0201] Step 1805: Cluster the intermediate vector to obtain multiple clustering sets.

[0202] Among them, each clustering set includes a clustering center, and the clustering center corresponds to multiple intermediate vectors.

[0203] Step 1806: Obtain the quantization error and clustering distance of the clustering set.

[0204] Among them, the quantization error is represented by the average distance between each intermediate vector in the clustering set and its corresponding clustering center. The clustering distance is represented by the distance between the clustering center in the clustering set and the vector mean, and the vector mean is the mean of all intermediate vectors.

[0205] Step 1807: Present the corresponding data points in a scatter plot with the quantization error as the ordinate and the clustering distance as the abscissa.

[0206] Among them, the scatter plot also presents the data points corresponding to the historical intermediate vectors previously obtained by the intelligent model. Thus, the arrangement shape between the data points in the scatter plot characterizes the degree of uniform distribution from the sample input vector to the homogenized vector.

[0207] For example, under the implementation method of multi-step spatial distribution transformation, the set of intermediate variables obtained after the n original vectors undergo the spatial distribution transformation in the t-th step can be represented by represented as is the intermediate variable obtained after the i-th original vector undergoes the spatial distribution transformation in the t-th step.

[0208] Based on this, in this embodiment, k-means clustering is performed on these intermediate vectors to simulate the quantization process, obtaining K clustering centers, and these K clustering centers form a set is the k-th clustering center obtained by clustering the intermediate vectors obtained by the spatial distribution transformation in the t-th step.

[0209] After that, calculate the average value of the third distance between each intermediate vector in each clustering set and its corresponding clustering center, that is, the average distance, and use this average distance as the quantization error, as shown in formula (12):

[0210]

[0211] Among them, is the average distance between each intermediate vector in the k-th clustering set obtained by clustering the intermediate vectors obtained by the spatial distribution transformation in the t-th step and its corresponding clustering center. Ω k represents the set of all intermediate vectors in the class with the clustering center i.e., the clustering set.

[0212] Then, calculate the vector mean of all intermediate vectors through formula (13), and then calculate the distance between each clustering center and this vector mean through formula (14), that is, the clustering distance

[0213]

[0214]

[0215] Finally, taking the quantization error as the ordinate and the clustering distance as the abscissa, data points are formed. Each is taken as a data point in the two-dimensional coordinate points, and a scatter plot is drawn. As shown in Figure 20 , if the intermediate vector conforms to the Gaussian distribution, the quantization error increases with the increase of the clustering distance . If it is uniformly distributed, the quantization error does not change with the change of the clustering distance . Thus, through the scatter plot, the process of the original vector gradually changing to a uniform distribution can be visualized, and the results of multi-step spatial distribution transformation can be detected.

[0216] Referring to Figure 21 , it is a schematic structural diagram of a data processing device provided in an embodiment of the present application. This device can be applied to electronic devices capable of data processing, such as computers or servers. The technical solution in this embodiment is mainly used to improve the accuracy of vector search.

[0217] Specifically, the device in this embodiment may include the following units:

[0218] A vector obtaining unit 2101, configured to obtain a first vector; the first vector is a high-dimensional vector;

[0219] A distribution transformation unit 2102, configured to perform a spatial distribution transformation on the first vector to obtain a second vector;

[0220] A vector search unit 2103, configured to search for a third vector that satisfies the vector similarity condition in a target vector set based on the second vector;

[0221] Wherein, the target vector set includes multiple reconstructed vectors, and the reconstructed vectors are obtained through the spatial distribution transformation and vector reconstruction based on the original vectors in the database. The sub-vectors in the reconstructed vectors are low-dimensional vectors;

[0222] A distribution inverse transformation unit 2104, configured to perform a spatial distribution inverse transformation on the third vector to obtain a fourth vector.

[0223] As can be seen from the above technical solution, in a data processing device provided in an embodiment of the present application, the high-dimensional vector to be searched is first subjected to a spatial distribution transformation, so that the obtained high-dimensional vector can be at least approximately uniformly distributed. Then, the at least approximately uniformly distributed high-dimensional vector is used to search for a vector that satisfies the vector similarity condition from a set of reconstructed low-dimensional vectors. Finally, the searched vector is subjected to an inverse spatial distribution transformation to obtain the searched high-dimensional vector. It can be seen that in this embodiment, through the spatial distribution transformation of the high-dimensional vector, the error caused by non-uniform vector distribution during vector search is avoided, thereby improving the accuracy of vector search.

[0224] In one implementation, the vector search unit 2103 is specifically configured to: use the reconstruction code table to perform vector reconstruction on the second vector to obtain a fifth vector; the sub-vectors in the fifth vector are low-dimensional vectors; wherein, the reconstruction code table includes a plurality of sub-code tables, and each sub-code table includes a plurality of central vectors; there is a one-to-one mapping between the sub-code tables and the sub-vectors in the fifth vector; the sub-vectors in the fifth vector are a central vector in the corresponding sub-code table; based on the central distance between any two central vectors in the sub-code table, obtain a first distance between the fifth vector and the reconstructed vectors in the target vector set; and determine the reconstructed vectors in the target vector set whose first distance satisfies the vector similarity condition as the third vector.

[0225] In one implementation, the distribution transformation unit 2102 is specifically configured to: obtain a first transfer vector for performing a single-step spatial distribution transformation on the first vector; and use the first transfer vector to process the first vector to obtain a second vector.

[0226] In one implementation, the distribution transformation unit 2102 is specifically configured to: obtain a second transfer vector for performing one-step spatial distribution transformation on the first vector; use the second transfer vector to process the first vector to obtain an intermediate vector; take the intermediate vector as the new first vector, and execute the step of obtaining the second transfer vector for performing one-step spatial distribution transformation on the first vector until the number of times of the spatial distribution transformation reaches the target threshold to obtain a second vector.

[0227] In one implementation, after the distribution inverse transformation unit performs an inverse spatial distribution transformation on the third vector to obtain a fourth vector, it is further configured to: obtain a second distance between the first vector and the fourth vector; and determine the fourth vector whose second distance satisfies the vector similarity condition as the new fourth vector.

[0228] It should be noted that the specific implementation manners of the units in this embodiment can refer to the corresponding content in the foregoing, and will not be elaborated here.

[0229] ReferenceFigure 22 , which is a schematic structural diagram of a data processing device provided by an embodiment of the present application. This device can be applied to electronic devices capable of data processing, such as computers or servers. The technical solution in this embodiment is mainly used to improve the accuracy of vector search.

[0230] Specifically, the device in this embodiment may include the following units:

[0231] A vector obtaining unit 2201, configured to obtain original vectors in an original set, where the original vectors are high-dimensional vectors;

[0232] A vector transformation unit 2202, configured to perform a spatial distribution transformation on the original vectors to obtain uniformly transformed vectors;

[0233] A vector reconstruction unit 2203, configured to use a reconstruction codebook to perform vector reconstruction on the uniformly transformed vectors to obtain reconstructed vectors; sub-vectors in the reconstructed vectors are low-dimensional vectors;

[0234] Wherein, the reconstruction codebook includes a plurality of sub-codebooks, the sub-codebooks include a plurality of center vectors; there is a one-to-one mapping between the sub-codebooks and the sub-vectors in the reconstructed vectors; the sub-vectors in the reconstructed vectors are one center vector in the corresponding sub-codebook;

[0235] A vector addition unit 2204, configured to add the reconstructed vectors to a target vector set, and the reconstructed vectors in the target vector set are used for vector search.

[0236] As can be seen from the above solution, in a data processing device provided by an embodiment of the present application, before reconstructing the original vectors, the original vectors are first subjected to a spatial distribution transformation, so that the obtained vectors can be at least approximately uniformly distributed. In this way, vectors that are at least approximately uniformly distributed are used for vector reconstruction through a reconstruction codebook. Thus, the vectors stored in the target vector set are low-dimensional vectors and are uniformly distributed vectors. In this way, when vector search is required, first perform a spatial distribution transformation, and then search from the target vector set based on these uniformly distributed and low-dimensional reconstructed vectors. Therefore, in this embodiment, through the spatial distribution transformation of high-dimensional vectors, the error existing in vector search due to non-uniform vector distribution can be avoided, and thus the accuracy of vector search can be improved.

[0237] In one implementation, the spatial distribution transformation and the vector reconstruction are implemented through an intelligent model; wherein, the model parameters of the intelligent model are at least optimized based on a uniformity loss value; the uniformity loss value characterizes the degree of uniform distribution of the vectors obtained by the intelligent model through the spatial distribution transformation.

[0238] In one implementation, the intelligent model performs the vector reconstruction on the training samples, including:

[0239] Split the homogenized vector to obtain multiple sub-sample vectors; each of the sub-sample vectors corresponds to a sub-code table in the reconstruction code table; the homogenized vector is a vector obtained by changing the spatial distribution of the sample input vector in the training sample;

[0240] Obtain the target similarity between the sub-sample vector and each center vector in the corresponding sub-code table;

[0241] Multiply the target similarity by the center vector in the corresponding sub-code table to obtain a product vector;

[0242] Sum all the product vectors corresponding to the sub-sample vector to obtain an integrated vector corresponding to the sub-sample vector;

[0243] Use the integrated vector corresponding to each sub-sample vector as a sub-vector to obtain a reconstructed sample vector.

[0244] In one implementation, the vector transformation unit 2202 is further configured to: output a two-dimensional scatter plot; the arrangement shape between data points in the two-dimensional scatter plot represents the degree of uniform distribution from the original vector to the uniformly transformed vector.

[0245] It should be noted that the specific implementation manners of the units in this embodiment can refer to the corresponding content above, and will not be elaborated here.

[0246] Refer to Figure 23 , which is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device may include the following structures:

[0247] A memory 2301 for storing a computer program and data generated by running the computer program;

[0248] A processor 2302 for executing the computer program to implement: obtaining a first vector; the first vector is a high-dimensional vector; performing a spatial distribution transformation on the first vector to obtain a second vector; based on the second vector, searching for a third vector that satisfies the vector similarity condition in a target vector set; wherein, the target vector set includes multiple reconstructed vectors, and the reconstructed vectors are obtained by performing the spatial distribution transformation and vector reconstruction on the original vectors in the database, and the sub-vectors in the reconstructed vectors are low-dimensional vectors; performing a spatial distribution inverse transformation on the third vector to obtain a fourth vector.

[0249] It should be noted that the reconstructed vectors in the target vector set can be stored in the memory in the form of key-value pairs (kvcache). The vector data processed by the processor 2302 is also stored in the cache area in the form of key-value pairs.

[0250] In a specific implementation, the processor 2302 may include three parts: a distribution transformation module (also referred to as a distribution transformer), a differentiable quantization module (which can also be a differentiable quantizer), and an optimization module (also referred to as an optimizer).

[0251] Among them, the distribution transformer is mainly used to perform a spatial distribution transformation on the original vector or the first vector. The differentiable quantizer is mainly used to perform vector reconstruction on the uniformly transformed vector (obtained by performing a spatial distribution transformation on the original vector) and to perform vector reconstruction on the second vector (obtained by performing a spatial distribution transformation on the first vector). The optimizer is mainly used to optimize the distribution transformer and the differentiable quantizer.

[0252] From the above technical solution, it can be seen that in an electronic device provided in an embodiment of the present application, first, a high-dimensional vector to be searched is subjected to a spatial distribution transformation, so that the obtained high-dimensional vector can be at least approximately uniformly distributed. Then, using the at least approximately uniformly distributed high-dimensional vector to search for a vector that satisfies the vector similarity condition from a set of reconstructed low-dimensional vectors, and finally, performing an inverse spatial distribution transformation on the found vector, so that the found high-dimensional vector can be obtained. It can be seen that in this embodiment, through the spatial distribution transformation of the high-dimensional vector, the error caused by non-uniform vector distribution during vector search is avoided, thereby improving the accuracy of vector search.

[0253] Reference Figure 24 , which is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device may include the following structures:

[0254] A memory 2401, configured to store a computer program and data generated by the running of the computer program;

[0255] A processor 2402, configured to execute the computer program to implement: obtaining an original vector in the original set, the original vector being a high-dimensional vector; performing a spatial distribution transformation on the original vector to obtain a uniformly transformed vector; using a reconstruction codebook to perform vector reconstruction on the uniformly transformed vector to obtain a reconstructed vector; a sub-vector in the reconstructed vector being a low-dimensional vector; wherein, the reconstruction codebook includes a plurality of sub-codebooks, the sub-codebook includes a plurality of center vectors; there is a one-to-one mapping between the sub-codebook and the sub-vector in the reconstructed vector; the sub-vector in the reconstructed vector is a center vector in the corresponding sub-codebook; adding the reconstructed vector to the target vector set, and the reconstructed vectors in the target vector set are used for vector search.

[0256] It should be noted that the reconstructed vectors in the target vector set can be stored in the memory in the form of key-value pairs (kvcache). The vector data processed by the processor 2402 is also stored in the cache area in the form of key-value pairs.

[0257] In a specific implementation, the processor 2402 may include three parts: a distribution transformation module, a differentiable quantization module, and an optimization module.

[0258] Among them, the distribution transformer is mainly used to perform a spatial distribution transformation on the original vector. The differentiable quantizer is mainly used to perform vector reconstruction on the uniformly transformed vector (obtained by the original vector through spatial distribution transformation). The optimizer is mainly used to optimize the distribution transformer and the differentiable quantizer.

[0259] From the above solution, it can be seen that in an electronic device provided by an embodiment of the present application, before reconstructing the original vector, the original vector is first subjected to a spatial distribution transformation, so that the obtained vector can be at least approximately uniformly distributed. Then, the vector that is at least approximately uniformly distributed is used to perform vector reconstruction through a reconstruction code table. Therefore, the vectors stored in the target vector set are low-dimensional vectors and are uniformly distributed vectors. When vector search is required, first perform a spatial distribution transformation, and then search from the target vector set based on these uniformly distributed and low-dimensional reconstructed vectors. Therefore, in this embodiment, through the spatial distribution transformation of high-dimensional vectors, the error existing in vector search due to non-uniform vector distribution can be avoided, thereby improving the accuracy of vector search.

[0260] Taking the electronic device as an Artificial Intelligence Personal Computer (AIPC) as an example, the technical solution of the present application will be described in detail below:

[0261] The mainstream unstructured data retrieval method is to convert this data into high-dimensional vectors and retrieve the required data by finding similar or approximate vectors. However, due to the limited resources of the AIPC, in order to reduce the computing and storage resources required for vector search, a dedicated vector compression technology becomes particularly important.

[0262] Product Quantization (PQ) is a technique for compressing high-dimensional vector data into a smaller space. It decomposes the original vector into several low-dimensional vectors, independently quantizes the decomposed sub-vectors, generates a codebook through the K-means clustering algorithm, and finally combines the quantization results of all sub-vectors into the quantization result of the original vector. Product quantization effectively reduces the computational complexity and storage requirements of vector search. Many mainstream vector retrieval algorithm libraries or vector databases such as FAISS, NMSLIB, Milvus, and Qdrant utilize product quantization to improve the efficiency of large-scale vector approximate nearest neighbor search.

[0263] Although product quantization greatly improves the efficiency of vector search and reduces the requirements for memory resources, it has a negative impact on the accuracy of vector search. One important reason is that the distribution of vector data is often uneven, and product quantization ignores this factor, resulting in small quantization errors in areas with high data density and large quantization errors in areas with low data density.

[0264] In view of this, the technical solution of this application designs an end-to-end learnable product quantization framework based on vector distribution transformation, which includes:

[0265] (1) A transformer that can change the distribution of the vector dataset. The key point is to design a multi-step data distribution transformation method that gradually transforms the unevenly distributed vector data into a uniform distribution while maintaining the vector neighbor relationship unchanged.

[0266] (2) A differentiable quantizer. The key point is to use a two-step decoding strategy of coarse ranking + re-ranking in the differentiable decoding process, effectively improving the accuracy of vector search after decoding.

[0267] (3) An optimizer that fuses multiple features to constrain the maintenance of neighbor relationships, distribution uniformity, and the magnitude of quantization errors.

[0268] The basic working process of the technical solution of this application is as Figure 25 shown. This application mainly includes three parts: a distribution transformer, a differentiable quantizer, and an optimizer. For the n original vectors X = {x1,..., x n} stored in the vector database, where x i ∈ R d represents a single vector data, which satisfies the Gaussian distribution in the d-dimensional space. Since the uneven distribution of the original vectors in space is not conducive to product quantization coding. Therefore, first, the original vectors are passed through a distribution transformer to convert the original vectors into uniformly transformed vectors that approximately conform to a uniform distribution Secondly, the uniformly transformed vectors are input into the differentiable quantizer, and by learning the quantization code table, that is, the reconstruction code table C, each vector Reconstructed as {c 1 (i1), …, c M (i M )}, where M is the number of bits for encoding each vector, and i m represents the value of the m-th encoding, and c m (i m ) represents the m-th sub-vector reconstructed by looking up the table. Based on this, the reconstructed uniform transformation vector can also be restored to the original vector through a distribution transformer to obtain the final reconstructed vector Finally, the reconstructed vector is optimized by an optimizer to achieve end-to-end training of the entire framework. Specifically as follows:

[0269] 1. Distribution Transformer:

[0270] In the distribution transformer of the framework, a distribution transformation module is used to process the input vector. Specifically, for the input original vector x i , in order to perform a distribution transformation on it, it needs to be moved in space. Therefore, the main idea of the distribution transformation module is to learn a position transfer vector of the original vector through a non-linear module and perform a position transformation on the original vector based on this transfer vector.

[0271] Figure 8 and Figure 9 As shown, it is a schematic diagram of a single-step distribution transformation module. Non-linear learning is achieved through a combination of a linear layer + activation function, and the generation process of the transfer vector can be expressed by formula (15):

[0272] v i = W2σ(Norm(W1x i )) (15) ...

[0273] where v i is the transfer vector obtained by processing the i-th original vector through the distribution transformation module; x i is the i-th original vector.

[0274] where W1 and W2 are learnable weight matrices, σ is the activation function (here it is GELU), and Norm represents the normalization operation (here LayerNorm is used). Therefore, the final vector after distribution transformation can be obtained through the following formula (16):

[0275]

[0276] where is the uniform transformation vector obtained by performing distribution transformation on the i-th original vector.

[0277] Based on this, by learning the transfer vector v iThe advantages of not directly inputting the original vector into the non-linear layer for distribution transformation are mainly twofold: 1) It is easier to learn a small offset on the basis of the original vector compared to directly learning a distribution transformation weight. 2) The introduction of residual connections can make the training process converge more easily.

[0278] As Figure 11 and Figure 12 shown, considering that it is challenging to transfer the vector to a new position in one step with a large offset, in this embodiment, performing distribution transformation in multiple steps is a feasible optimization method. By iteratively inputting the vector into the non-linear module and obtaining the transfer vector multiple times, multi-step distribution transformation is achieved. Let the vector after the t-th step of distribution transformation be After T steps, the final homogenized vector is obtained, and this process is expressed as formula (17):

[0279]

[0280] where is the homogenized transformation vector obtained by the i-th original vector after T steps of distribution transformation. α ∈ (0, 1] is a hyperparameter that controls the distance advanced (transferred) in each step.

[0281] During the process of multi-step distribution transformation, in order to improve the detectability of this module, all corresponding vectors are output after each step of transformation and it is detected whether their distribution has changed. In order to detect the degree to which the distribution of vector data conforms to the uniform distribution, k-means clustering can be performed on it to simulate the quantization process, and K clustering centers are obtained Then, the average distance from all vectors in each cluster to the cluster center is calculated as the quantization error of this cluster, which can be expressed as formula (12).

[0282] In addition, the mean value of all vectors is calculated as in formula (13), and the distance from each cluster center to this mean value is obtained, which can be expressed as formula (14).

[0283] Finally, each is used as a point in the two-dimensional coordinate system to draw a scatter plot. If the vector conforms to the Gaussian distribution, the quantization error will increase as increases. If it is a uniform distribution, the quantization error should not change with Therefore, through this scatter plot, the process of the vector gradually changing into a uniform distribution can be visualized, thereby detecting the result of the multi-step transformation module.

[0284] 2. Differentiable quantizer:

[0285] In the differentiable quantizer of the framework, by designing a differentiable quantization function, the discrete quantization process is transformed into a continuous process to achieve end-to-end training of the framework. The structure of the differentiable quantization module is as shown in Figure 26 , which includes the processes of spatial distribution transformation and vector reconstruction of the original vector (i.e., the reference vector), and also includes the process of vector query.

[0286] Specifically, for the input vector after distribution transformation , similar to the traditional product quantization method, it is first evenly divided into M parts in dimension, that is , where each is a sub-vector. The differentiable product quantization module contains a learnable codebook C = {c 1 , …, c M}, where each c m ∈ R d / M×K is the sub-codebook corresponding to the corresponding sub-vector.

[0287] In the process of vector encoding (i.e., vector reconstruction, representing the reconstructed vector with the identifier of the center vector), first calculate the similarity between each sub-vector and each center vector in the corresponding codebook to obtain the similarity score. This process can be expressed by formula (6). Among them, <·,·> represents the similarity calculation operation, such as cosine similarity, L2 similarity, etc. Then, normalize the similarity score through a Softmax function. Then the encoding of the final sub-vector can be expressed by formula (7).

[0288] In the inference process (i.e., vector search process), the corresponding encoding can be directly obtained through this function, and the corresponding sub-vector in the codebook can be indexed to reconstruct the original sub-vector. And in the training process, since the argmax operation is not differentiable, directly multiply with the sub-codebook to obtain an approximate reconstructed sub-vector, and then calculate the reconstruction loss with the original sub-vector.

[0289] In the decoding process, the encoded vector is represented as a combination of M codewords {i1, …, i M}. First, directly find the vector corresponding to each codeword through table lookup, so as to roughly reconstruct the corresponding vector {c 1 (i1), …, c M (i M ). In the process of constructing and searching the index for vector search, directly use this roughly reconstructed vector and calculate the distance between vectors through table lookup.

[0290] Since the search results obtained using the roughly reconstructed vector may not be accurate, in the search process, the search results are decoded and rearranged.

[0291] Specifically, for the query vector x q , first, it is also passed through the distribution transformer to obtain the transformed query . By using this query to calculate the distance from the existing n roughly reconstructed vectors in the vector library through table lookup, the L nearest neighbor vectors closest to it are obtained . After that, these L vectors are passed through the inverse distribution transformation D to obtain the final reconstructed vectors . Finally, the distance between these L reconstructed vectors and the query vector x q is calculated, such as cosine or L2 similarity, and the k nearest neighbors of the query vector are reordered.

[0292] Since the distribution transformation and reordering of vectors only occur in a limited number of L vectors, the computational complexity is limited, and the search performance can be significantly improved. In this process, two sets of nearest neighbor results of the query vector are output, and the reordered result is significantly better than the non-reordered search result in terms of accuracy.

[0293] 3 Optimizer

[0294] In the optimizer part of the framework, the model is trained by jointly optimizing multiple losses. Specifically, the loss function for model training includes three main parts: reconstruction loss, contrastive loss, and uniformity loss. The reconstruction loss is mainly used to optimize the reconstructed vectors after the distribution transformation at the decoding end to make them as close as possible to the original input vectors. The loss function of this reconstruction loss can be expressed as formula (8).

[0295] In addition to using the reconstruction loss, in order to ensure that the reconstructed vectors maintain the same nearest neighbor relationship as the original vectors, a contrastive learning method is also used to constrain them. The loss function of this contrastive loss can be expressed as formula (9).

[0296] Among them, is randomly selected from the k i nearest neighbors of the vector x in the original vector dataset + , and is randomly selected from the k nearest neighbors of the vector in the reconstructed vectors, after removing the aforementioned k - nearest neighbors. By constraining k + <k + <k - , the selected negative samples are made to be as far away from the positive samples as possible, and the samples are not too easy to distinguish from each other.

[0297] Finally, in order to optimize the vectors after the distribution transformation to conform to the uniform distribution as much as possible, a uniformity loss is used for optimization. The loss function of this uniformity loss can be expressed as formula (10).

[0298] Among them, Indicates the vector in the transformed vector data that is closest to The vector with the closest distance. By constraining the distances between the vectors with the closest distances to be as large as possible, the vector distribution can be implicitly optimized so that the distances between all vectors are kept relatively far apart, thereby achieving a uniform filling of the entire space. Combining the three parts of the loss function, the final loss function can be expressed as formula (11).

[0299] It can be seen that the main problem addressed in this application is the low accuracy of vector search based on product quantization caused by the non-uniform distribution of vector data. By designing a product quantization optimization method based on data distribution homogenization, the data is evenly distributed in space, and the vectors in each cluster are as close to the center as possible, thereby reducing the quantization error and improving the accuracy of vector search.

[0300] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts between the various embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0301] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0302] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0303] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, comprising: Obtaining a first vector; The first vector is a high-dimensional vector; Performing a spatial distribution transformation on the first vector to obtain a second vector; Based on the second vector, searching for a third vector in a target vector set that satisfies a vector similarity condition; Wherein, the target vector set includes a plurality of reconstructed vectors, and the reconstructed vectors are obtained through the spatial distribution transformation and vector reconstruction based on the original vectors in the database, and the sub-vectors in the reconstructed vectors are low-dimensional vectors; Performing a spatial distribution inverse transformation on the third vector to obtain a fourth vector.

2. The method according to claim 1, wherein searching for a third vector in a target vector set that satisfies a vector similarity condition based on the second vector comprises: Using a reconstruction code table to perform vector reconstruction on the second vector to obtain a fifth vector; the sub-vectors in the fifth vector are low-dimensional vectors; Wherein, the reconstruction code table includes a plurality of sub-code tables, and each sub-code table includes a plurality of central vectors; there is a one-to-one mapping between the sub-code tables and the sub-vectors in the fifth vector; the sub-vectors in the fifth vector are one of the central vectors in the corresponding sub-code table; Based on the central distance between any two central vectors in the sub-code table, obtaining a first distance between the fifth vector and the reconstructed vectors in the target vector set; Determining the reconstructed vector in the target vector set whose first distance satisfies the vector similarity condition as the third vector.

3. The method according to claim 1, wherein performing a spatial distribution transformation on the first vector to obtain a second vector comprises: Obtaining a first transfer vector for a single-step spatial distribution transformation of the first vector; Using the first transfer vector to process the first vector to obtain a second vector. [[ID=I15]]4. The method according to claim 1, wherein performing a spatial distribution transformation on the first vector to obtain a second vector comprises: Obtaining a second transfer vector for a one-step spatial distribution transformation of the first vector; Using the second transfer vector to process the first vector to obtain an intermediate vector; [[ID=I18]]Taking the intermediate vector as the new first vector, and performing the operation of: obtaining the second transfer vector for a one-step spatial distribution transformation of the first vector until the number of times of the spatial distribution transformation reaches a target threshold to obtain a second vector.

5. The method according to claim 1, after performing a spatial distribution inverse transformation on the third vector to obtain a fourth vector, the method further comprises: Obtaining a second distance between the first vector and the fourth vector; Determining the fourth vector whose second distance satisfies the vector similarity condition as the new fourth vector.

6. A data processing method, comprising: Obtaining an original vector in an original set, the original vector being a high-dimensional vector; Performing a spatial distribution transformation on the original vector to obtain a uniformly transformed vector; Using a reconstruction code table to perform vector reconstruction on the uniformly transformed vector to obtain a reconstructed vector; the sub-vectors in the reconstructed vector are low-dimensional vectors; Among them, the reconstructed codebook includes a plurality of sub-codebooks, and each sub-codebook includes a plurality of central vectors; there is a one-to-one mapping between the sub-codebooks and the sub-vectors in the reconstructed vector; the sub-vectors in the reconstructed vector are one central vector in the corresponding sub-codebook. Add the reconstructed vector to the target vector set, and the reconstructed vectors in the target vector set are used for vector search.

7. The method according to claim 6, wherein the spatial distribution transformation and the vector reconstruction are implemented by an intelligent model. Among them, The model parameters of the intelligent model are optimized at least based on a uniformity loss value; the uniformity loss value characterizes the degree of distribution uniformity of the vectors obtained by the intelligent model through spatial distribution transformation.

8. The method according to claim 7, wherein the intelligent model performs the vector reconstruction on the training samples, including: Splitting the homogenized vector to obtain a plurality of sub-sample vectors; each sub-sample vector corresponds to one sub-codebook in the reconstructed codebook. The homogenized vector is the vector obtained by performing spatial distribution transformation on the sample input vector in the training sample. Obtaining the target similarity between the sub-sample vector and each central vector in the corresponding sub-codebook. Multiplying the target similarity by the central vector in the corresponding sub-codebook to obtain a product vector. Adding up all the product vectors corresponding to the sub-sample vector to obtain the integrated vector corresponding to the sub-sample vector. Using the integrated vector corresponding to each sub-sample vector as a sub-vector to obtain a reconstructed sample vector.

9. The method according to claim 6, the method further includes: Outputting a two-dimensional scatter plot. In the two-dimensional scatter plot, the arrangement shape between data points characterizes the degree of uniform distribution from the original vector to the uniformly transformed vector.

10. An electronic device, including: A memory for storing a computer program and data generated by running the computer program. A processor for executing the computer program to achieve: obtaining a first vector. The first vector is a high-dimensional vector; performing a spatial distribution transformation on the first vector to obtain a second vector; based on the second vector, searching in the target vector set for a third vector that satisfies the vector similarity condition; wherein, the target vector set includes a plurality of reconstructed vectors, and the reconstructed vectors are obtained through the spatial distribution transformation and vector reconstruction based on the original vectors in the database, and the sub-vectors in the reconstructed vectors are low-dimensional vectors; performing a spatial distribution inverse transformation on the third vector to obtain a fourth vector.

Images