Data Retrieval Method, Device, Electronic Device and Storage Medium

Abstract

The present application provides a data retrieval method, apparatus, electronic device, and storage medium. The method includes: determining the distances from each original vector to each of the target quantization centers according to the target vector coefficient, where the target vector coefficient is determined based on the dimension of the original vector and a preset optimization level; determining the target vector corresponding to each original vector according to the distances from each original vector to each of the target quantization centers, and using the target vector as the vector corresponding to the original data. By determining the distances from each original vector to each of the target quantization centers according to the target vector coefficient, a general vector coefficient can be used when calculating the distances from the original vectors to each target quantization center, making the calculation process simpler and easier to use, reducing the computational complexity, and avoiding the need to calculate the parallel direction loss coefficient and the perpendicular direction loss coefficient of the original vector every time.

Description

Technical Field

[0001] This application relates to the field of vector quantization, and more particularly, to a data retrieval method, apparatus, electronic device, and storage medium. Background Art

[0002] Maximum inner product search is a common vector search method in machine learning and deep learning. The quantization error between vectors can reflect the degree of association between vectors. Therefore, maximum inner product search is often used for association queries.

[0003] In the prior art, the quantization error between vectors is calculated by the method of anisotropic vector quantization. However, each vector calculated by this method has an independent vector coefficient. During the calculation process, it is necessary to calculate the vector coefficient corresponding to each vector according to the modulus length of each vector, and then calculate the quantization error between vectors according to the vector coefficient.

[0004] Therefore, the prior art has the problem of high computational complexity. Summary of the Invention

[0005] The purpose of this application is to provide a data retrieval method, apparatus, electronic device, and storage medium to reduce the computational complexity for the deficiencies in the above prior art.

[0006] To achieve the above purpose, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, an embodiment of this application provides a data retrieval method, and the method includes:

[0008] Vectorize multiple pieces of original data respectively to obtain multiple original vectors, where the original data is data available for users to retrieve;

[0009] Determine multiple target quantization centers according to the multiple original vectors, and each target quantization center is a central vector corresponding to at least one original vector among the multiple original vectors;

[0010] Determine the distance from each original vector to each target quantization center according to the target vector coefficient, where the target vector coefficient is determined based on the dimension of the original vector and a preset optimization level;

[0011] Determine the target vector corresponding to each original vector according to the distance from each original vector to each target quantization center, and use the target vector as the vector corresponding to the original data;

[0012] After obtaining a data retrieval request input by a user, retrieve the target vectors according to the parameter information in the data retrieval request to obtain at least one retrieval vector, and use the original data corresponding to each retrieval vector as the retrieval result to be output.

[0013] Optionally, determining a plurality of target quantization centers according to the plurality of original vectors includes:

[0014] Randomly screening a plurality of vectors from the plurality of original vectors as a plurality of initial quantization centers;

[0015] Determining the distances from each original vector to each of the initial quantization centers according to the target vector coefficients;

[0016] Clustering each original vector according to the distances from each original vector to each of the initial quantization centers to obtain a plurality of clusters, each of the clusters including each initial quantization center and at least one original vector associated with each initial quantization center;

[0017] Updating the quantization centers of each of the clusters to obtain a plurality of new initial quantization centers, and re-executing the step of determining the distances from each original vector to each of the initial quantization centers until a convergence condition is satisfied, and taking each new initial quantization center when the convergence condition is satisfied as the target quantization center.

[0018] Optionally, the determining the distances from each original vector to each of the target quantization centers according to the target vector coefficients includes:

[0019] Taking the target vector coefficients as input parameters of a preset weighted distance calculation formula, and determining the distances from each original vector to each of the target quantization centers through the weighted distance calculation formula.

[0020] Optionally, the weighted distance calculation formula is:

[0021]

[0022] where x i is each original vector, c i is each target quantization center, and ω N is the target vector coefficient.

[0023] Optionally, before the determining the distances from each original vector to each of the target quantization centers according to the target vector coefficients, further includes:

[0024] Determining the target vector coefficients according to the dimension of the original vector and the optimization level.

[0025] Optionally, the determining the target vector coefficients according to the dimension of the original vector and the optimization level includes:

[0026] Taking the dimension of the original vector and the optimization level as input parameters of a preset weighted coefficient calculation formula, and determining the target vector coefficients through the weighted coefficient calculation formula.

[0027] Optionally, the calculation formula of the weighting coefficient is:

[0028]

[0029] where ω N is the target vector coefficient, d is the dimension of the original vector, and N is the optimization level.

[0030] In a second aspect, an embodiment of the present application further provides a data retrieval device, and the device includes:

[0031] A quantization module, configured to vectorize multiple pieces of original data respectively to obtain multiple original vectors, where the original data is data available for users to retrieve;

[0032] A determination module, configured to determine multiple target quantization centers according to the multiple original vectors, and each target quantization center is a central vector corresponding to at least one original vector among the multiple original vectors;

[0033] A determination module, configured to determine the distance from each original vector to each target quantization center according to the target vector coefficient, where the target vector coefficient is determined based on the dimension of the original vector and a preset optimization level;

[0034] A determination module, configured to determine a target vector corresponding to each original vector according to the distance from each original vector to each target quantization center, and use the target vector as the vector corresponding to the original data;

[0035] A retrieval module, configured to, after receiving a data retrieval request input by a user, retrieve the target vector according to parameter information in the data retrieval request to obtain at least one retrieval vector, and use the original data corresponding to each retrieval vector as a retrieval result to be output.

[0036] Optionally, the determination module is specifically configured to:

[0037] Randomly select multiple vectors from the multiple original vectors as multiple initial quantization centers;

[0038] Determine the distance from each original vector to each initial quantization center according to the target vector coefficient;

[0039] Cluster each original vector according to the distance from each original vector to each initial quantization center to obtain multiple clusters, and each cluster includes each initial quantization center and at least one original vector associated with each initial quantization center;

[0040] Update the quantization centers of each of the clusters to obtain a plurality of new initial quantization centers, and re - execute the step of determining the distances from each original vector to each of the initial quantization centers according to the target vector coefficients until the convergence condition is satisfied. Take each new initial quantization center when the convergence condition is satisfied as the target quantization center.

[0041] Optionally, the determining module is specifically configured to:

[0042] Use the target vector coefficients as input parameters of a preset weighted distance calculation formula, and determine the distances from each original vector to each of the target quantization centers through the weighted distance calculation formula.

[0043] Optionally, the weighted distance calculation formula is:

[0044]

[0045] where x i is each original vector, c i is each target quantization center, and ω N is the target vector coefficient.

[0046] Optionally, the determining module is specifically configured to:

[0047] Determine the target vector coefficients according to the dimension of the original vector and the optimization level.

[0048] Optionally, the determining module is specifically configured to:

[0049] Use the dimension of the original vector and the optimization level as input parameters of a preset weighted coefficient calculation formula, and determine the target vector coefficients through the weighted coefficient calculation formula.

[0050] Optionally, the weighted coefficient calculation formula is:

[0051]

[0052] where ω N is the target vector coefficient, d is the dimension of the original vector, and N is the optimization level.

[0053] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the application program runs, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to perform the steps of the data retrieval method described in the first aspect above.

[0054] Fourthly, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is read and executed to perform the steps of the data retrieval method described in the first aspect above.

[0055] The beneficial effects of the present application are as follows:

[0056] A data retrieval method, device, electronic device and storage medium provided by the present application respectively vectorize a plurality of original data to obtain a plurality of original vectors, where the original data is data available for user retrieval; determine a plurality of target quantization centers according to the plurality of original vectors, and each target quantization center is a central vector corresponding to at least one original vector among the plurality of original vectors; determine the distances from each original vector to each target quantization center according to the target vector coefficients, where the target vector coefficients are determined based on the dimension of the original vector and a preset optimization level; determine the target vector corresponding to each original vector according to the distances from each original vector to each target quantization center, and use the target vector as the vector corresponding to the original data; after obtaining a data retrieval request input by the user, retrieve the target vector according to the parameter information in the data retrieval request to obtain at least one retrieval vector, and use the original data corresponding to each retrieval vector as the retrieval result to be output. By determining the distances from each original vector to each target quantization center according to the target vector coefficients, a general vector coefficient can be used when calculating the distances from the original vectors to each target quantization center, making the calculation process simpler and easier to use, reducing the calculation complexity, and avoiding the need to calculate the parallel direction loss coefficient and the vertical direction loss coefficient of the original vector every time. Description of the Drawings

[0057] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0058] Figure 1 It is a schematic flowchart of a data retrieval method provided by an embodiment of the present application;

[0059] Figure 2 It is a schematic flowchart of another data retrieval method provided by an embodiment of the present application;

[0060] Figure 3 It is a schematic diagram of a device for a data retrieval method provided by an embodiment of the present application;

[0061] Figure 4Block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0062] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purposes of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present application illustrate operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical context may be reversed or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0063] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Usually, the components of the embodiments of the present application described and illustrated in the accompanying drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0064] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the features stated hereinafter, but does not exclude the addition of other features.

[0065] In the prior art, the Euclidean distance clustering quantization method is usually used to calculate the quantization error between vectors. However, the commonly used clustering quantization method only calculates the quantization error through the Euclidean distance and does not consider the influence of the angle between vectors on the quantization error. Therefore, the accuracy of the quantization error calculated in the prior art is not accurate enough.

[0066] As a possible design, the anisotropic vector quantization method can be used. In this method, the influence of the angle between vectors on the quantization error is taken into account. However, when applying the formula in the anisotropic vector quantization method to clustering quantization calculation, since each vector in the formula has an independent parallel direction loss coefficient and a perpendicular direction loss coefficient, if both the parallel direction loss coefficient and the perpendicular direction loss coefficient of each vector are saved, it will increase the computing memory cost; if the parallel direction loss coefficient and the perpendicular direction loss coefficient of each vector are not saved, when calculating the quantization error between the vector and the quantization center, it is necessary to recalculate the parallel direction loss coefficient and the perpendicular direction loss coefficient of the vector according to the modulus of the vector. At this time, the computing complexity and cost will increase.

[0067] Therefore, compared with the Euclidean distance quantization method in the embodiments of the present application, considering the influence of the angle between vectors, the accuracy of vector quantization can be improved; compared with the above anisotropic vector quantization method, a simple calculation method for a general coefficient is provided, which can greatly reduce the computing complexity on the premise of ensuring the quantization accuracy.

[0068] Figure 1 The flowchart of a data retrieval method provided by an embodiment of the present application is shown as Figure 1 shown, and the method includes:

[0069] S101. Vectorize multiple original data respectively to obtain multiple original vectors.

[0070] Optionally, the original data is data available for user retrieval. This data can be, for example, region data, person data, disease data, or symptom data that the user needs to retrieve. Among them, the region data can be, for example, Shaanxi or Shanghai, etc.; the person data can be, for example, Zhang or Zhao, etc.; the disease data can be, for example, cold or hypertension, etc.; the symptom data can be, for example, fever or cough, etc.

[0071] Optionally, when vectorizing each original data, specifically, each original data can be represented in the form of a vector to generate the original vector of each original data. The original vector can be a region data vector, a person data vector, a disease data vector, or a symptom data vector. Each original vector can include one or more dimensions, and the elements of each dimension can be set according to the original data.

[0072] S102. Determine multiple target quantization centers according to the multiple original vectors.

[0073] Optionally, each target quantization center is a central vector corresponding to at least one original vector among the multiple original vectors.

[0074] Optionally, multiple original vectors can be distributed in one or more different regions. For each region, a central vector can be determined as the clustering center, and this clustering center is the target quantization center. Among them, the original vectors in the same region as this target quantization center can be one or more original vectors.

[0075] S103. Determine the distances from each original vector to each target quantization center according to the target vector coefficients.

[0076] Optionally, the target vector coefficients are determined based on the dimension of the original vector and a preset optimization level. Among them, the original vector can include multiple dimensions, such as the vector being 2D, 3D, 4D, 5D, 6D, etc.

[0077] Optionally, the preset optimization level can indicate that different optimization levels can use different optimization range angles to quantize the original vector. Among them, the optimization range angle is the optimization range of the angle between each original vector and each target quantization center. The original vectors within this range angle are effective optimization vectors. That is to say, the quantization error precision calculated by the original vectors within this range is higher.

[0078] Optionally, the target vector coefficients can be combined with a preset formula to obtain a formula for calculating the distances from each original vector to each target quantization center. According to this formula, the distances from each original vector to each target quantization center can be calculated.

[0079] S104. Determine the target vector corresponding to each original vector according to the distances from each original vector to each target quantization center, and use the target vector as the vector corresponding to the original data.

[0080] Optionally, by calculating the distances from each original vector to each target quantization center in S103 above, multiple distance values can be obtained. The minimum distance value can be selected from the multiple calculated distance values. Then, the target quantization center with the smallest distance from the original vector is used as the target vector of this original vector, and the original vector can be quantized into this target vector. That is to say, this target vector can be used to represent this original vector.

[0081] S105. After receiving the data retrieval request input by the user, retrieve the target vector according to the parameter information in the data retrieval request to obtain at least one retrieval vector, and use the original data corresponding to each retrieval vector as the retrieval result to be output.

[0082] Among them, the parameter information in the data retrieval request can be, for example, the number of the target vector. Each target vector can have a unique number, and this number can indicate the association relationship between this target vector, the original data, and the original vector.

[0083] Optionally, a target vector can be retrieved according to the input data retrieval request, and one or more original vectors can also be retrieved according to the target vector. The retrieved one or more original vectors are used as retrieval vectors, and each retrieval vector is represented in the form of a vector of each original data. Then, the obtained retrieval vectors are converted into each original data, and the converted original data is used as the retrieval result to be output.

[0084] In this embodiment, multiple original vectors are obtained by vectorizing multiple original data respectively, where the original data is data available for user retrieval; according to the multiple original vectors, multiple target quantization centers are determined, and each target quantization center is a central vector corresponding to at least one original vector among the multiple original vectors; according to the target vector coefficients, the distances from each original vector to each target quantization center are determined, where the target vector coefficients are determined based on the dimension of the original vector and a preset optimization level; according to the distances from each original vector to each target quantization center, the target vector corresponding to each original vector is determined, and the target vector is used as the vector corresponding to the original data; after receiving a data retrieval request input by the user, according to the parameter information in the data retrieval request, the target vector is retrieved to obtain at least one retrieval vector, and the original data corresponding to each retrieval vector is used as the retrieval result to be output. By determining the distances from each original vector to each target quantization center according to the target vector coefficients, a general vector coefficient can be used when calculating the distances from the original vectors to each target quantization center, making the calculation process simpler and easier to use, reducing the calculation complexity, and avoiding the need to calculate the parallel direction loss coefficient and the vertical direction loss coefficient of the original vector every time.

[0085] Figure 2 It is a schematic flowchart of another data retrieval method provided by an embodiment of the present application. As Figure 2 shown, in the above step S102, determining multiple target quantization centers according to multiple original vectors may include:

[0086] S201. Randomly select multiple vectors from the multiple original vectors as multiple initial quantization centers.

[0087] Optionally, some original vectors can be selected from the multiple original vectors as vectors for training the target quantization centers. The selected part of the original vectors is used as a training vector set, and multiple vectors, such as k vectors, can be randomly selected from this training vector set, and the selected k vectors are used as k initial quantization centers.

[0088] S202. Cluster each original vector according to the distances from each original vector to each initial quantization center to obtain multiple clusters.

[0089] Among them, each cluster may include each initial quantization center and at least one original vector associated with each initial quantization center.

[0090] Optionally, each original vector refers to each vector in the training vector set in S201 above. Specifically, the distance from each vector in the training vector set to each initial quantization center can be calculated according to the target vector coefficient. According to the magnitudes of the calculated distances from each vector to each initial quantization center, each vector can be assigned to the region of the initial quantization center with the smallest distance to it. After calculating the distances from all vectors in the training vector set to each initial quantization center and completing the assignment of each vector, multiple clusters can be obtained.

[0091] Optionally, each cluster may include the initial quantization center of the cluster and at least one original vector associated with the initial quantization center. At least one vector center associated with the initial quantization center can be an original vector whose distance from the initial quantization center meets a preset condition. The preset condition is, for example, the smallest distance.

[0092] Exemplarily, if there are 3 initial quantization centers in the training vector set, for example, vector A, vector B, and vector C, calculate the distances from a certain vector H in the training vector set to these 3 initial quantization centers, and obtain 3 distances l1, l2, and l3 respectively, where l1 > l2 > l3. Then the distance from vector H to vector C is the closest, so vector H is assigned to the region of vector C, and vector H and vector C form a cluster.

[0093] S203. Update the quantization centers of each cluster to obtain multiple new initial quantization centers, and re - execute the above S202 - S203 until the convergence condition is met. Take each new initial quantization center when the convergence condition is met as the target quantization center.

[0094] Optionally, to update the quantization centers of each cluster, specifically, the central vector of all vectors in each cluster can be calculated. For example, the average value of the coordinates of all vectors included in each cluster can be calculated, and the average value of the calculated coordinates is used as the coordinates of the new initial quantization center to obtain multiple new initial quantization centers.

[0095] Optionally, repeat the above S202 - S203 steps, that is, calculate the distances from each vector in the training vector set to the newly updated initial quantization centers, obtain multiple new clusters, and update the quantization centers of the multiple new clusters obtained until the convergence condition is met.

[0096] Optionally, the convergence condition may be, for example, that the newly obtained initial quantization center from the last calculation is the same as the newly obtained initial quantization center from the previous calculation of the last time, and then the newly obtained initial quantization center is used as the target quantization center; it may also be the preset number of times to update the quantization center of each cluster. If the preset number of times is reached, the loop calculation is stopped, and the newly obtained initial quantization center is used as the target quantization center.

[0097] Optionally, in the above step S103, determining the distances from the respective original vectors to the respective target quantization centers according to the target vector coefficients may include:

[0098] Optionally, using the target vector coefficients as the input parameters of a preset weighted distance calculation formula, and determining the distances from the respective original vectors to the respective target quantization centers through the weighted distance calculation formula.

[0099] Optionally, the weighted distance calculation formula is:

[0100]

[0101] where x i are the respective original vectors, c i are the respective target quantization centers, and ω N are the target vector coefficients.

[0102] Optionally, the above weighted distance calculation formula is a quantization error formula.

[0103] Optionally, the target vector coefficients may be the ratio of the error loss coefficients of the respective original vectors and the target quantization centers in the parallel direction and the error loss coefficients of the respective original vectors and the target quantization centers in the vertical direction. Among them, the error loss coefficients of the respective original vectors and the target quantization centers in the parallel direction may be represented by h || for example, and the error loss coefficients of the respective original vectors and the target quantization centers in the vertical direction may be h ⊥ for example.

[0104] Optionally, before the above step S103 determines the distances from the respective original vectors to the respective target quantization centers according to the target vector coefficients, it may further include:

[0105] Optionally, determining the target vector coefficients according to the dimension of the original vector and the optimization level.

[0106] Optionally, the determining the target vector coefficients according to the dimension of the original vector and the optimization level may include:

[0107] Optionally, using the dimension of the original vector and the optimization level as the input parameters of a preset weighted coefficient calculation formula, and determining the target vector coefficients through the weighted coefficient calculation formula.

[0108] Optionally, the weighted coefficient calculation formula is:

[0109]

[0110] where ω N is the target vector coefficient, d is the dimension of the original vector, and N is the optimization level.

[0111] The above weighted distance calculation formula can be derived through the following process. Specifically:

[0112] Based on the quantization error formula of the anisotropic vector quantization method, the quantization error formula of the anisotropic vector quantization method is further simplified to obtain the weighted distance calculation formula in this application.

[0113] Among them, the quantization error formula of the anisotropic vector quantization method is as follows:

[0114] l(x i , c i ) = h ‖ (‖x i ‖, T, d)·‖r ‖ (x i - c i )‖ 2 + h ⊥ ‖x i ‖, T, d)·‖r ⊥ (x i - c i )‖ 2 Formula (2)

[0115] where x i is each original vector, c i is each target quantization center, r ‖ (x i - c i ) is the sub-vector parallel to x i - c i and x i , r ⊥ (x i - c i ) is the sub-vector perpendicular to x i - c i and x i , h ‖ is the loss function of the parallel direction error, h ⊥ is the loss function of the perpendicular direction error, and this loss function is related to the modulus length of x i and the included angle between the original vector and the target vector center. d is the dimension of the original vector, and T is the included angle between x i and c iThe inner product value is a pre-set range value, and

[0116]

[0117]

[0118] Optionally, based on the quantization error formula (2) of the above anisotropic vector quantization method, simplify the h ‖ coefficient and the h ⊥ coefficient, and represent them using the first vector coefficient. Specifically, set the original vector x i and the target vector center c i When the included angle between them is within the range of Φ, the calculation result of the inner product value of x i and c i can be the candidate result for the maximum inner product retrieval, that is, Φ is the included angle range of the effective retrieval result, where the inner product value is the result calculated by the quantization error formula. For other cases outside the Φ included angle, it can be considered that x i and c i belong to vectors with extremely weak correlation and are no longer within the candidate range of the maximum inner product retrieval result. If the included angle Φ is greater than 90°, the inner product value of the two vectors x i and c i is negative, and its result is not a candidate result for the maximum inner product retrieval; if the included angle Φ is equal to 90°, the inner product value of the two vectors x i and c i is 0, which is also not a candidate result for the maximum inner product retrieval. Therefore, the value range of Φ is 0 ≤ Φ < 90°.

[0119] Optionally, by setting the value range of Φ, the h ‖ coefficient in formula (3) and the h ⊥ coefficient in formula (4) can be simplified. The simplified h ‖ coefficient and h ⊥ coefficient are calculated through the vector dimension d and the included angle range. The specific formula is as follows:

[0120]

[0121]

[0122] Then the simplified quantization error formula is as follows:

[0123] l(x i ,c i )=h ‖ (Φ,d)·‖r ‖ (x i ,c i )‖ 2 +h⊥ (Φ, d)·‖r ⊥ (x i , c i )‖ 2 Formula (VII)

[0124] Optionally, the first vector coefficient can be The above simplified quantization error formula (VII) can be a weighted distance calculation formula of the first vector coefficient calculated according to the vector dimension and the angle range. The simplified h ‖ coefficient and h ⊥ coefficient are both calculated through the vector dimension d and the angle range Φ. Therefore, the value of the first vector coefficient only depends on the vector dimension d and the angle range Φ for calculation, and does not depend on the original vector x i , so the first vector coefficient is a preset constant.

[0125] The first weighted distance calculation formula obtained by taking the first vector coefficient as an input parameter is:[[]]

[0126] l(x i , c i ) = ‖r ‖ (x i , c i )‖ 2 + ω·‖r ⊥ (x i , c i )‖ 2 Formula (VIII)

[0127] Among them, the first vector coefficient in the above first weighted distance calculation formula (VIII) is only calculated depending on the vector dimension d and the angle range Φ. By setting the optimization level, the optimization level and the vector dimension are used as parameters of the first vector coefficient. The specific derivation and calculation process is as follows:

[0128] Optionally, the maximum optimization range angle can be determined based on the above first vector coefficient formula, formula (V), formula (VI) and the existing Euclidean distance formula. The maximum optimization range angle is only related to the vector dimension, so vectors of different dimensions can calculate different maximum optimization range angles. According to the maximum optimization range angle, the angle threshold can be determined, and then the target vector coefficient can be determined according to the relationship between the priority level, the vector dimension and the angle threshold.

[0129] Optionally, the process of determining the maximum optimization range angle: According to the existing Euclidean distance formula γ(x i , c i ) = ‖r ‖ (x i , c i )‖ 2 + ‖r⊥ (x i , c i )‖ 2 It can be obtained that this formula can be the weighted distance calculation formula when ω = 1 in the first weighted distance calculation formula (VIII). Among them, in the simplified quantization error formula (VII), the smaller the included angle Φ of the effective range, the smaller the quantization error within the effective range. Therefore, to further reduce the quantization error on the basis of the Euclidean distance formula, the selection of the threshold value of the included angle Φ of the effective range needs to be smaller than that of γ(x i , c i ), that is, the included angle corresponding to ω = 1 is smaller.

[0130] Optionally, if the included angle of the effective range corresponding to ω = 1 is set as then when , the quantization error can be smaller, then is the maximum optimization range angle.

[0131] Through the value of can be calculated. Specifically, By calculation, is obtained, where d is the vector dimension. Among them, the maximum optimization range angle can be, in the d-dimensional space, a quadrant is divided with the vector center c i as the central axis. The vector x i within this quadrant is the effective optimization vector, and the maximum included angle between the vectors within this quadrant and the vector center is

[0132] Optionally, when the selected angle threshold Φ is less than , that is, , the quantization accuracy of the vectors whose included angle with the vector center is within this angle threshold range is improved.

[0133] Optionally, different maximum optimization range angles can be calculated for vectors of different dimensions, which can be specifically represented by Table 1 as follows:

[0134]

[0135] It can be obtained from Table 1 that the maximum optimization range angle increases with the increase of the dimension, and the maximum optimization range angle can be used as the selected value of the angle threshold Φ. For example, if the dimension of the vector is d, the selectable angle threshold is where N is the optimization level, and 0 ≤ N ≤ d - 2.

[0136] Exemplarily, if the optimization level is 0, the threshold Φ is If the optimization level is 1, the threshold Φ is If the optimization level is 2, the threshold Φ is When the optimization level is d-2, the threshold Φ is

[0137] In this embodiment, by changing the selection of the threshold into the selection of the priority level, the selection of the threshold becomes more general.

[0138] Optionally, the target vector coefficient is determined according to the relationship between the above priority level, the vector dimension, and the angle threshold.

[0139] The threshold can be obtained through the above description The corresponding coefficient ω N is calculated as follows: The calculated where Substituting it into, we get

[0140] In this embodiment, only a general weighted coefficient can be calculated according to the priority level and the vector dimension, and the calculation of this weighted coefficient is simple, so that each original vector only needs to calculate the weighted coefficient once when calculating the distance to the center of each target vector, greatly reducing the calculation complexity.

[0141] Optionally, substituting the weighted coefficient ω N into the above formula (VIII), the above formula (I) can be obtained through calculation. The specific implementation process is as follows:

[0142]

[0143] Figure 3 is a schematic diagram of a device for a data retrieval method provided by an embodiment of the present application. As Figure 3 shown, the device includes:

[0144] A quantization module 301, configured to vectorize multiple pieces of original data respectively to obtain multiple original vectors, where the original data is data available for users to retrieve;

[0145] A determination module 302, configured to determine multiple target quantization centers according to the multiple original vectors, and each target quantization center is a central vector corresponding to at least one original vector among the multiple original vectors;

[0146] The determination module 302 is configured to determine the distance from each original vector to each target quantization center according to the target vector coefficient, where the target vector coefficient is determined based on the dimension of the original vector and a preset optimization level;

[0147] A determination module 302, configured to determine a target vector corresponding to each original vector according to the distances from the original vectors to the target quantization centers, and use the target vector as the vector corresponding to the original data;

[0148] A retrieval module 303, configured to, after receiving a data retrieval request input by a user, retrieve the target vectors according to the parameter information in the data retrieval request, obtain at least one retrieved vector, and use the original data corresponding to each retrieved vector as a retrieval result to be output.

[0149] Optionally, the determination module 302 is specifically configured to:

[0150] Randomly select multiple vectors from multiple original vectors as multiple initial quantization centers;

[0151] Determine the distances from the original vectors to the initial quantization centers according to the target vector coefficients;

[0152] Cluster the original vectors according to the distances from the original vectors to the initial quantization centers, to obtain multiple clusters, where each cluster includes each initial quantization center and at least one original vector associated with each initial quantization center;

[0153] Update the quantization centers of the clusters to obtain multiple new initial quantization centers, and re-execute the step of determining the distances from the original vectors to the initial quantization centers according to the target vector coefficients until a convergence condition is satisfied, and use the new initial quantization centers when the convergence condition is satisfied as the target quantization centers.

[0154] Optionally, the determination module 302 is specifically configured to:

[0155] Use the target vector coefficients as input parameters of a preset weighted distance calculation formula, and determine the distances from the original vectors to the target quantization centers through the weighted distance calculation formula.

[0156] Optionally, the weighted distance calculation formula is:

[0157]

[0158] where x i are the original vectors, c i are the target quantization centers, and ω N are the target vector coefficients.

[0159] Optionally, the determination module 302 is specifically configured to:

[0160] Determine the target vector coefficients according to the dimension of the original vectors and the optimization level.

[0161] Optionally, the determination module 302 is specifically configured to:

[0162] Use the dimension of the original vector and the optimization level as input parameters of a preset weighted coefficient calculation formula, and determine the target vector coefficient through the weighted coefficient calculation formula.

[0163] Optionally, the weighted coefficient calculation formula is:

[0164]

[0165] where ω N is the target vector coefficient, d is the dimension of the original vector, and N is the optimization level.

[0166] Figure 4 FIG. 400 is a structural block diagram of an electronic device 400 provided by an embodiment of the present application. As Figure 4 shown, the electronic device may include: a processor 401 and a memory 402.

[0167] Optionally, it may further include a bus 403. Among them, the memory 402 is used to store machine-readable instructions executable by the processor 401. When the electronic device 400 runs, the processor 401 communicates with the memory 402 through the bus 403. When the machine-readable instructions are executed by the processor 401, the method steps in the above method embodiments are executed.

[0168] An embodiment of the present application further provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the method steps in the above data retrieval method embodiments are executed.

[0169] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the method embodiments, which will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces. The indirect coupling or communication connection of the devices or modules can be in an electrical, mechanical, or other form.

[0170] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0171] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application.

Claims

1. A data retrieval method, characterized in that, The method includes: Vectorize multiple pieces of original data respectively to obtain multiple original vectors, where the original data is data available for user retrieval; Determine multiple target quantization centers according to the multiple original vectors, and each target quantization center is a central vector corresponding to at least one of the multiple original vectors; Determine the distances from each original vector to each target quantization center according to the target vector coefficients, where the target vector coefficients are determined based on the dimension of the original vector and a preset optimization level, and the preset optimization level is used to represent that different optimization levels use different optimization range angles to quantize the original vector, the optimization range angle is the optimization range of the included angle between each original vector and each target quantization center, and each original vector within the optimization range angle is an effective optimization vector; Determine the target vector corresponding to each original vector according to the distances from each original vector to each target quantization center, and use the target vector as the vector corresponding to the original data, where the target vector is the target quantization center with the smallest distance from the original vector; After receiving a data retrieval request input by the user, retrieve the target vectors according to the parameter information in the data retrieval request to obtain at least one retrieval vector, and use the original data corresponding to each retrieval vector as the retrieval result to be output; The step of determining the distances from each original vector to each target quantization center according to the target vector coefficients includes: Use the target vector coefficients as the input parameters of a preset weighted distance calculation formula, and determine the distances from each original vector to each target quantization center through the weighted distance calculation formula; The weighted distance calculation formula is: Among them, are the respective original vectors, are the respective target quantization centers, are the target vector coefficients.

2. The data retrieval method according to claim 1, wherein The step of determining multiple target quantization centers according to the multiple original vectors includes: Randomly select multiple vectors from the multiple original vectors as multiple initial quantization centers; Determine the distances from each original vector to each initial quantization center according to the target vector coefficients; Cluster each original vector according to the distances from each original vector to each initial quantization center to obtain multiple clusters, and each cluster includes each initial quantization center and at least one original vector associated with each initial quantization center; Update the quantization centers of each cluster to obtain multiple new initial quantization centers, and re-execute the step of determining the distances from each original vector to each initial quantization center according to the target vector coefficients until the convergence condition is satisfied, and use each new initial quantization center when the convergence condition is satisfied as the target quantization center.

3. The data retrieval method according to claim 1, wherein Before the step of determining the distances from each original vector to each target quantization center according to the target vector coefficients, it further includes: Determine the target vector coefficients according to the dimension of the original vector and the optimization level.

4. The data retrieval method according to claim 3, characterized in that, Determining the target vector coefficients according to the dimension of the original vector and the optimization level includes: Use the dimension of the original vector and the optimization level as the input parameters of a preset weighted coefficient calculation formula, and determine the target vector coefficients through the weighted coefficient calculation formula.

5. The data retrieval method according to claim 4, characterized in that, The weighted coefficient calculation formula is: Among them, is the target vector coefficient, d is the dimension of the original vector, and N is the optimization level.

6. A data retrieval device, characterized in that, Includes: A quantization module for vectorizing multiple pieces of original data respectively to obtain multiple original vectors, where the original data is data available for user retrieval; A determination module for determining multiple target quantization centers according to the multiple original vectors, each of the target quantization centers being a central vector corresponding to at least one of the multiple original vectors; A determination module for determining the distances from each original vector to each of the target quantization centers according to target vector coefficients, where the target vector coefficients are determined based on the dimension of the original vector and a preset optimization level, and the preset optimization level is used to represent that different optimization levels use different optimization range angles to quantize the original vector, the optimization range angle being the optimization range of the included angle between each original vector and each target quantization center, and each original vector within the optimization range angle being an effective optimization vector; A determination module for determining a target vector corresponding to each original vector according to the distances from each original vector to each of the target quantization centers, and using the target vector as the vector corresponding to the original data, the target vector being the target quantization center with the smallest distance from the original vector; A retrieval module for, after receiving a data retrieval request input by a user, retrieving the target vector according to parameter information in the data retrieval request to obtain at least one retrieval vector, and using the original data corresponding to each retrieval vector as a retrieval result to be output; The determination module is specifically configured to: Use the target vector coefficients as input parameters of a preset weighted distance calculation formula, and determine the distances from each original vector to each of the target quantization centers through the weighted distance calculation formula; The weighted distance calculation formula is: Among them, are the respective original vectors, are the respective target quantization centers, are the target vector coefficients.

7. An electronic device, characterized in that, Comprising a memory and a processor, the memory stores a computer program executable by the processor, and when the processor executes the computer program, the steps of the data retrieval method according to any one of claims 1-5 above are implemented.

8. A computer-readable storage medium, characterized in that, A computer program is stored on this computer-readable storage medium, and when the computer program is run by a processor, the steps of the data retrieval method according to any one of claims 1-5 above are executed.

Images