Vectorization-based database query processing method and system

Abstract

The invention discloses a vectorization-based database query processing method, which comprises the following steps of: vectorizing all data, converting each piece of data into a data vector, and representing one attribute of the data by each dimension in the data vector; the query instruction is converted into a query vector, the query vector is a mathematical representation of the query instruction, and the query vector and the data vector have the same dimension and structure; and calculating the similarity between the query vector and each data vector in sequence, and taking the data corresponding to the data vector meeting a similarity threshold as the data needing to be queried.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a database query processing method and system based on vectorization. Background Art

[0002] When processing queries, traditional database systems usually store data in tables and execute queries through executors. However, as the amount of data continues to grow and the complexity of queries increases, the efficiency of this method gradually decreases. Traditional database systems usually store data in a table format, and this structured storage method is difficult to effectively represent and process complex multidimensional data. In particular, when processing unstructured or semi-structured data, existing technologies often seem to be unable to cope with it. Summary of the invention

[0003] To achieve the above-mentioned purpose and other related purposes, the present invention discloses a database query processing method based on vectorization, comprising:

[0004] Vectorize all data and convert each data into a data vector, where each dimension in the data vector represents an attribute of the data;

[0005] Convert the query instruction into a query vector, where the query vector is a mathematical representation of the query instruction and has the same dimension and structure as the data vector;

[0006] The similarity between the query vector and each data vector is calculated in turn, and the data corresponding to the data vector that meets the similarity threshold is used as the data to be queried.

[0007] Furthermore, vectorizing all data includes:

[0008] Suppose the data set D contains n data items, and each data item d_i contains m attributes, then each data item can be represented as an m-dimensional vector, V(d_i) = [v_1, v_2, ..., v_m], where v_j is the numerical representation of attribute j.

[0009] Furthermore, converting the query instruction into a query vector includes:

[0010] Suppose the query Q contains m query conditions, each condition corresponds to a dimension of the data vector, then the query vector QVec = [q_1, q_2, ..., q_m], where q_j is the numerical representation of query condition j.

[0011] Furthermore, the method also includes: if there are multiple data vectors that meet the similarity threshold condition, the data vectors are sorted according to the similarity value, and the data corresponding to the data vector that best meets the requirements is used as the data to be queried by the query instruction.

[0012] Further, calculating the similarity between the query vector and each data vector includes:

[0013] Sim(QVec,V(di))=QVec·V(d_i) / (∥QVec∥∥V(d_i)∥);

[0014] Among them, "·" represents the dot product of the vectors, and "∥∥" represents the modulus of the vectors.

[0015] Further, calculating the similarity between the query vector and each data vector includes:

[0016] Dist(QVec,V(di))=(∑ j＝1 m (q_j-v_ij) 2 ) 0.5 ;

[0017] Among them, vi_ij is the j-th dimension of the data vector V(d_i).

[0018] On the other hand, the present invention provides a vectorized database query processing system, comprising:

[0019] A data conversion module is used to vectorize all data and convert each data into a data vector, wherein each dimension in the data vector represents an attribute of the data;

[0020] A query conversion module, used for converting a query instruction into a query vector, where the query vector is a mathematical representation of the query instruction, and the query vector has the same dimension and structure as the data vector;

[0021] The vector operation module is used to calculate the similarity between the query vector and each data vector in turn, and take the data corresponding to the data vector that meets the similarity threshold as the data to be queried.

[0022] By adopting the above technical solution, by converting data and query instructions into vector form, and using the similarity calculation between vectors to achieve efficient database query. The technical solution of the present invention can not only significantly improve the query efficiency, but also has good scalability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0024] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] Embodiment 1:

[0027] Reference Figure 1 The embodiment of the present invention provides a database query processing method based on vectorization, which is characterized by comprising:

[0028] S1: Vectorize all data and convert each data into a data vector, where each dimension in the data vector represents an attribute of the data.

[0029] Specifically, vectorizing all data includes:

[0030] Suppose the data set D contains n data items, and each data item d_i contains m attributes, then each data item can be represented as an m-dimensional vector, V(d_i) = [v_1, v_2, ..., v_m], where v_j is the numerical representation of attribute j.

[0031] For example, for a personnel database containing four attributes: "name", "age", "height", and "weight", each record can be represented as a 4-dimensional vector. For example:

[0032] Zhang San: [1001,25,175,70]

[0033] Li Si: [1002,30,180,75]

[0034] Wang Wu: [1003,35,170,65]

[0035] In this example, the name is encoded as a unique numeric identifier, and the other attributes use their numeric values ​​directly.

[0036] S2: Convert the query instruction into a query vector. The query vector is a mathematical representation of the query instruction. The query vector has the same dimension and structure as the data vector.

[0037] Specifically, for example, the query "people older than 30 and taller than 170cm" can be represented as a vector: QVec = [*, 30, 170, *]

[0038] Here, * indicates that the dimension is not used as a query condition. In actual implementation, special values ​​(such as -1) can be used to indicate dimensions that are not involved in the query.

[0039] S3: Calculate the similarity between the query vector and each data vector in turn, and use the data corresponding to the data vector that meets the similarity threshold as the data to be queried.

[0040] Specifically, calculating the similarity between the query vector and each data vector includes:

[0041] Sim(QVec,V(di))=QVec·V(d_i) / (∥QVec∥∥V(d_i)∥);

[0042] Among them, "·" represents the dot product of the vectors, and "∥∥" represents the modulus of the vectors.

[0043] The value range of cosine similarity is [-1, 1], and the larger the value, the more similar it is. In practical applications, the similarity can be mapped to the [0, 1] interval as needed.

[0044] Step S4: Result screening and sorting

[0045] In this step, we first set a similarity threshold (such as 0.9) and take the data corresponding to the data vector that meets the threshold condition as the preliminary query result. Then, the system will sort these results in descending order of similarity and return the data that best meets the requirements.

[0046] For example, if the query result has three records with similarities of 0.95, 0.92, and 0.91, the system will return the three records in this order.

[0047] Embodiment 2:

[0048] This embodiment uses Euclidean distance as the similarity calculation method based on Embodiment 1:

[0049] The calculating the similarity between the query vector and each data vector comprises:

[0050] Dist(QVec,V(di))=(∑ j＝1 m (q_j-v_ij) 2 ) 0.5 ;

[0051] Among them, vi_ij is the jth dimension of the data vector V(d_i). The smaller the distance, the higher the similarity.

[0052] The advantage of using Euclidean distance is that it can better reflect the actual distance between vectors in space, which is particularly suitable for scenarios where the absolute difference between attributes needs to be considered. For example, in a product recommendation system, Euclidean distance can more accurately measure the match between user preferences and product features.

[0053] The present invention also provides a vectorized database query processing system, which is characterized by comprising:

[0054] A data conversion module is used to vectorize all data and convert each data into a data vector, wherein each dimension in the data vector represents an attribute of the data;

[0055] A query conversion module, used to convert the query instruction into a query vector, where the query vector is a mathematical representation of the query instruction, and the query vector has the same dimension and structure as the data vector;

[0056] The vector operation module is used to calculate the similarity between the query vector and each data vector in turn, and take the data corresponding to the data vector that meets the similarity threshold as the data to be queried.

[0057] For the convenience of description, the above devices are described in terms of functions and are divided into various units and modules. Of course, when implementing the present application, the functions of each unit and module can be implemented in the same or multiple software and / or hardware.

[0058] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0059] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0060] It can be known from the description of the above implementation modes that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various implementation modes of the present application or certain parts of the implementation modes.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A database query processing method based on vectorization, characterized in that: include: Vectorize all data and convert each data into a data vector, where each dimension in the data vector represents an attribute of the data; Convert the query instruction into a query vector, where the query vector is a mathematical representation of the query instruction and has the same dimension and structure as the data vector; The similarity between the query vector and each data vector is calculated in turn, and the data corresponding to the data vector that meets the similarity threshold is used as the data to be queried.

2. The method according to claim 1, characterized in that The vectorization of all data includes: Suppose the data set D contains n data items, and each data item d_i contains m attributes, then each data item can be represented as an m-dimensional vector, V(d_i) = [v_1, v_2, ..., v_m], where v_j is the numerical representation of attribute j.

3. The method according to claim 2, characterized in that The converting the query instruction into a query vector comprises: Suppose the query Q contains m query conditions, each condition corresponds to a dimension of the data vector, then the query vector QVec = [q_1, q_2, ..., q_m], where q_j is the numerical representation of query condition j.

4. The method according to claim 1, characterized in that: The method further includes: if there are multiple data vectors that meet the similarity threshold condition, the data vectors are sorted according to the similarity values, and the data corresponding to the data vector that best meets the requirements is used as the data to be queried by the query instruction.

5. The method according to claim 3, characterized in that: The calculating the similarity between the query vector and each data vector comprises: Sim(QVec,V(di))=QVec·V(d_i) / (∥QVec∥∥V(d_i)∥); Among them, "·" represents the dot product of the vectors, and "∥∥" represents the modulus of the vectors.

6. The method according to claim 3, characterized in that The calculating the similarity between the query vector and each data vector comprises: Dist(QVec,V(di))=(∑ j＝1 m (q_j-v_j) 2 ) 0.5 4 Among them, vi_ij is the j-th dimension of the data vector V(d_i).

7. A vectorized database query processing system, characterized in that: include: A data conversion module is used to vectorize all data and convert each data into a data vector, wherein each dimension in the data vector represents an attribute of the data; A query conversion module, used for converting a query instruction into a query vector, where the query vector is a mathematical representation of the query instruction, and the query vector has the same dimension and structure as the data vector; The vector operation module is used to calculate the similarity between the query vector and each data vector in turn, and take the data corresponding to the data vector that meets the similarity threshold as the data to be queried.

Images