Data storage, retrieval method and device, electronic device, and computer-readable storage medium

Abstract

The embodiments of the present invention provide a data storage and retrieval method and device, an electronic device, and a computer-readable storage medium. The data storage method includes: creating a data storage table; writing first type data in at least one first data column in the data storage table; writing second type data in at least one second data column in the data storage table; and writing a row identifier of the first type data associated with the stored second type data at a corresponding position in the second data column. The embodiments of the present invention facilitate expansion of the storage structure by storing structured data and vector data by columns; associating structured data and vector data by row identifiers, so that structured data and vector data can be truly associated in storage, so that when retrieving data, the retrieval range of vector data can be narrowed, and the efficiency of hybrid retrieval can be improved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data storage and retrieval method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of big data technology, there are more and more requirements for retrieval and analysis of vector data such as images, audio, and video files, especially for mixed retrieval of vector data and structured data. For example, in some products or services, it is necessary to retrieve portraits within a certain time range, or vehicles within a certain location range.

[0003] To this end, the prior art has proposed solving the problem of mixed retrieval of vector data and structured data by means of a fixed mixed structure.

[0004] However, such a fixed hybrid structure in the prior art has poor scalability due to its fixedness, and its structured data and vector data are separated from each other during actual storage, so that the effect of accelerating vector data retrieval cannot be achieved in practical applications. Summary of the invention

[0005] The embodiments of the present invention provide a data storage and retrieval method and device, an electronic device, and a computer-readable storage medium to solve the defects of poor scalability and low hybrid retrieval efficiency of the fixed hybrid structure in the prior art.

[0006] To achieve the above object, an embodiment of the present invention provides a data storage method, including:

[0007] Create a data storage table;

[0008] Writing first type data in at least one first data column in the data storage table;

[0009] Writing second type data into at least one second data column in the data storage table, wherein one of the first type data and the second type data is vector data and the other is structured data;

[0010] A row identifier of the first type of data associated with the stored second type of data is written at a corresponding position in the second data column.

[0011] The embodiment of the present invention further provides a data retrieval method for the data stored by the above data storage method, comprising:

[0012] Acquiring data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions;

[0013] In the data column where the structured data is stored, obtaining a first row identifier corresponding to the structured data that meets the structured search condition;

[0014] According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the data column where the vector data is stored.

[0015] An embodiment of the present invention further provides a data storage device, comprising:

[0016] The first creation module is used to create a data storage table;

[0017] A first processing module, configured to write first type data into at least one first data column in the data storage table;

[0018] A second processing module, configured to write second type data into at least one second data column in the data storage table, wherein one of the first type data and the second type data is vector data and the other is structured data;

[0019] The third processing module is configured to write a row identifier of the first type of data associated with the stored second type of data at a corresponding position in the second data column.

[0020] The embodiment of the present invention further provides a data retrieval device for the data stored in the above storage device, comprising:

[0021] A first acquisition module, used to acquire data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions;

[0022] A second acquisition module, configured to acquire, from the data column where the structured data is stored, a first row identifier corresponding to the structured data that meets the structured search condition;

[0023] The third acquisition module is used to acquire, according to the acquired first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition in the data column where the vector data is stored.

[0024] An embodiment of the present invention further provides an electronic device, including:

[0025] Memory, used to store programs;

[0026] A processor, configured to run the program stored in the memory, to:

[0027] Create a data storage table;

[0028] Writing first type data in at least one first data column in the data storage table;

[0029] Writing second type data into at least one second data column in the data storage table, wherein one of the first type data and the second type data is vector data and the other is structured data;

[0030] A row identifier of the first type of data associated with the stored second type of data is written at a corresponding position in the second data column.

[0031] An embodiment of the present invention further provides an electronic device, including:

[0032] Memory, used to store programs;

[0033] A processor, configured to run the program stored in the memory, to:

[0034] Acquiring data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions;

[0035] In the data column where the structured data is stored, obtaining a first row identifier corresponding to the structured data that meets the structured search condition;

[0036] According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the data column where the vector data is stored.

[0037] An embodiment of the present invention further provides a computer-readable storage medium, on which instructions are stored, the instructions comprising:

[0038] Create a data storage table;

[0039] Writing first type data in at least one first data column in the data storage table;

[0040] Writing second type data into at least one second data column in the data storage table, wherein one of the first type data and the second type data is vector data and the other is structured data;

[0041] A row identifier of the first type of data associated with the stored second type of data is written at a corresponding position in the second data column.

[0042] An embodiment of the present invention further provides a computer-readable storage medium, on which instructions are stored, the instructions comprising:

[0043] Acquiring data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions;

[0044] In the data column where the structured data is stored, obtaining a first row identifier corresponding to the structured data that meets the structured search condition;

[0045] According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the data column where the vector data is stored.

[0046] The data storage and retrieval method and device, electronic device, and computer-readable storage medium provided by the embodiments of the present invention facilitate expansion of the storage structure by storing structured data and vector data in columns; and associating structured data and vector data through row identifiers, so that structured data and vector data can be truly associated in storage, so that when retrieving data, the retrieval scope of vector data can be narrowed, thereby improving the efficiency of hybrid retrieval.

[0047] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0049] Figure 1 A schematic diagram of an application scenario of a business system provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of an embodiment of a data storage method provided by the present invention;

[0051] Figure 3 A flowchart of another embodiment of the data storage method provided by the present invention;

[0052] Figure 4 A flow chart of an embodiment of a data retrieval method provided by the present invention;

[0053] Figure 5 A flow chart of another embodiment of the data retrieval method provided by the present invention;

[0054] Figure 6 A schematic structural diagram of an embodiment of a data storage device provided by the present invention;

[0055] Figure 7A schematic structural diagram of another embodiment of the data storage device provided by the present invention;

[0056] Figure 8 A schematic diagram of the structure of an embodiment of a data retrieval device provided by the present invention;

[0057] Fig. 9 A schematic diagram of the structure of another embodiment of the data retrieval device provided by the present invention;

[0058] Fig.10 A schematic diagram of the structure of an embodiment of an electronic device provided by the present invention;

[0059] Fig.11 A schematic structural diagram of another embodiment of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0060] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0061] The fixed hybrid structure in the prior art leads to poor scalability, and its structured data and vector data are separated from each other during actual storage, so that the effect of accelerating vector data retrieval cannot be achieved in practical applications.

[0062] Therefore, this application proposes a data storage solution, the main principle of which is to store structured data and vector data by column, and to associate structured data and vector data in different columns through row identifiers. On the one hand, storing data by column facilitates the expansion of the storage structure; on the other hand, it can truly associate structured data and vector data in storage, and no longer be separated from each other as in the prior art, which can narrow the vector search range and improve the search efficiency of vector data.

[0063] The method provided by the embodiment of the present invention can be applied to any business system with a data retrieval function. Figure 1 A schematic diagram of an application scenario of a business system provided by an embodiment of the present invention, Figure 1 The structure shown is only one example of a business system to which the technical solution of the present invention can be applied. Figure 1 As shown, the business system includes a storage device, which can be used to perform the following Figure 2 and Figure 3 The business system also includes a retrieval device, which can be used to perform the following Figure 4 and Figure 5 The processing flow shown. When the business system performs data storage, first, a data storage table is created; then, first type data is written in at least one first data column in the data storage table, and second type data is written in at least one second data column in the data storage table, wherein one of the first type data and the second type data is vector data and the other is structured data; and, the row identifier of the first type data associated with the stored second type data is written in the corresponding position in the second data column. During the storage process, the storage device may store vector data first and then structured data; or may store structured data first and then vector data. The row identifier of the associated first stored data, i.e., the first type data, is carried in the later stored data, i.e., the second type data, so as to realize the association between the structured data and the vector data.

[0064] When the business system performs data retrieval, first, the data retrieval condition is obtained, and the data retrieval condition includes a structured retrieval condition and a vector retrieval condition; then, in the data column where the structured data is stored, the first row identifier corresponding to the structured data that meets the structured retrieval condition is obtained; and according to the obtained first row identifier, in the data column where the vector data is stored, the target vector data corresponding to the first row identifier and meeting the vector retrieval condition is obtained. By storing the structured data and the vector data by column, it is convenient to expand the storage structure; by associating the structured data and the vector data by row identifier, when retrieving data, the retrieval range of the vector data can be narrowed, and the efficiency of the hybrid retrieval can be improved.

[0065] The above embodiments are illustrations of the technical principles and exemplary application frameworks of the embodiments of the present invention. The specific technical solutions of the embodiments of the present invention are further described in detail below through multiple embodiments.

[0066] Embodiment 1

[0067] Figure 2 This is a flow chart of an embodiment of the data storage method provided by the present invention. The execution subject of the method can be a storage device in the above-mentioned business system, or various server devices with data processing functions, or a device or chip integrated in these server devices. Figure 2 As shown, the data storage method includes the following steps:

[0068] S201, creating a data storage table.

[0069] In the embodiment of the present invention, when storing mixed data including both vector data and structured data, first, it is necessary to create a data storage table including multiple data columns.

[0070] S202: Write first type data into at least one first data column in a data storage table.

[0071] S203: Write second type data into at least one second data column in the data storage table.

[0072] In an embodiment of the present invention, after a data storage table is created, first type data is written into a first data column in the data storage table, and second type data is written into a second data column in the data storage table. Among them, one of the first type data and the second type data is vector data, and the other is structured data. When storing vector data or structured data, data with different attributes are stored in different data columns according to the attributes of the data. For example, when storing vector data, the vector data of a picture can be stored in one data column, the vector data of a video can be stored in another data column, and so on; when storing structured data, the time data can be stored in one data column, the location data can be stored in another data column, and so on.

[0073] S204: Write a row identifier of the first type of data associated with the stored second type of data at a corresponding position in the second data column.

[0074] In an embodiment of the present invention, on the basis of storing the first type of data and the second type of data, the row identifier of the first type of data associated with the second type of data is also written into the corresponding position of the second data column. In the data storage process, vector data can be stored first, and then structured data; or structured data can be stored first, and then vector data. What needs to be done is to carry the row identifier of the associated first stored data, that is, the first type of data, in the later stored data, that is, the second type of data, to realize the association between structured data and vector data.

[0075] For example, when the first type of data is vector data and the second type of data is structured data, it is assumed that the data to be stored includes picture vector data and time data. Then, the picture vector data can be written in the first data column of the data storage table, and each picture vector data occupies one row, and then the time data can be written in the second data column of the data storage table, and each time data occupies one row. Then, according to the correspondence between the picture vector data and the time data, the row identifier of the picture vector data is written into the row where the corresponding time data is located in the second data column. For example, as shown in Table 1, the shooting time of picture A is a, and the shooting time of picture B is b. The picture vector data A' corresponding to the picture A is written into the first column and the first row of the data storage table, and the picture vector data B' corresponding to the picture B is written into the first column and the second row of the data storage table; and the time data a is written into the second column and the third row of the data storage table, and the time data b is written into the second column and the fourth row of the data storage table. The picture vector data A' corresponds to the time data a, and the picture vector data B' corresponds to the time data b. Therefore, in the second column, in addition to writing the time data, the row identifier of the corresponding vector data is also written.

[0076] Table 1

[0077] Column 1 Column 2 Row 1 A' ** Row 2 B' ** Row 3 a / 1 Row 4 b / 2

[0078] The data storage method provided by the embodiment of the present invention facilitates the expansion of the storage structure by storing structured data and vector data in columns; and by associating structured data and vector data through row identifiers, the structured data and vector data can be truly associated in storage, so that when searching for data, the search scope of vector data can be narrowed, thereby improving the efficiency of hybrid search.

[0079] Embodiment 2

[0080] Figure 3 This is a flow chart of another embodiment of the data storage method provided by the present invention. Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, the data storage method provided in this embodiment may further include the following steps:

[0081] S301, creating a data storage table.

[0082] S302: Write structured data into at least one first data column in a data storage table.

[0083] S303: Write vector data into at least one second data column in the data storage table.

[0084] S304: Write a row identifier of structured data associated with the stored vector data at a corresponding position in the second data column.

[0085] In the embodiment of the present invention, the first type of data is structured data, and the second type of data is vector data, that is, the structured data is stored first, and then the vector data is stored. Then, according to the corresponding relationship between the structured data and the vector data, the row identifier of the structured data is written into the row where the corresponding vector data is located in the second data column.

[0086] S305: Divide the data stored in the second data column into a plurality of data groups.

[0087] S306, generating a group mapping table according to the row identifier and index code in the data group.

[0088] In an embodiment of the present invention. After the structured data and the vector data are stored in columns successively, the multiple rows of vector data stored in one column can be further grouped into multiple data groups. Specifically, the data stored in the second data column can be grouped according to the similarity of the vector data based on the clustering algorithm, and similar vector data can be saved in the same data group. Then, based on the row identifier and index code in each data group, a group mapping table is generated for each data group, and in each group mapping table, a group code for identifying the data group and an offset of the row identifier and index code corresponding to the data in the data group are stored.

[0089] The data storage method provided by the embodiment of the present invention facilitates the expansion of the storage structure by storing structured data and vector data in columns; and by associating structured data and vector data through row identifiers, the structured data and vector data can be truly associated in storage, so that when retrieving data, the retrieval scope of vector data can be narrowed, thereby improving the efficiency of hybrid retrieval; by grouping vector data according to their similarity, when retrieving vector data, group retrieval can be performed first, and after finding the target group, vector data retrieval can be performed, thereby accelerating the retrieval process and further improving the retrieval efficiency.

[0090] Embodiment 3

[0091] Figure 4 This is a flow chart of an embodiment of the data retrieval method provided by the present invention. The execution subject of this method can be the retrieval device in the above business system, or various server devices with data retrieval functions, or devices or chips integrated in these server devices. Figure 4 As shown, for the above Figure 2 The data stored in the data storage method provided in the embodiment shown, the embodiment of the present invention provides a data retrieval method, comprising the following steps:

[0092] S401, obtaining data retrieval conditions.

[0093] In the embodiment of the present invention, a hybrid search is performed on the data, and therefore, the data search conditions include structured search conditions and vector search conditions.

[0094] S402: Obtain, from the data column where the structured data is stored, a first row identifier corresponding to the structured data that meets the structured search condition.

[0095] S403: According to the obtained first row identifier, in the data column where the vector data is stored, obtain the target vector data corresponding to the first row identifier and meeting the vector search condition.

[0096] In the embodiment of the present invention, since the retrieval process of vector data is more complicated, more computationally intensive, and more time-consuming than the retrieval of structured data, the row identifier corresponding to the structured data that meets the conditions, i.e., the first row identifier, can be first obtained through the structured data retrieval, thereby narrowing the retrieval scope of the vector data retrieval according to the first row identifier.

[0097] against Figure 2 The data stored in the data storage method provided by the illustrated embodiment, when the first type of data is vector data and the second type of data is structured data, stores the vector data first and then the structured data. Then, based on the correspondence between the structured data and the vector data, the row identifier of the vector data is written into the row where the corresponding structured data is located in the second data column. For this storage method, when performing data retrieval, first perform data retrieval on the structured data to obtain the first row identifier corresponding to the structured data that meets the structured retrieval conditions. The first row identifier is the row identifier of the vector data corresponding to the structured data, and therefore, the storage location of the corresponding vector data can be directly located according to the first row identifier. Then, vector data retrieval is performed on these corresponding vector data to obtain the target vector data that meets the data retrieval conditions.

[0098] On the other hand, when the first type of data is structured data and the second type of data is vector data, the structured data is stored first and then the vector data. Then, based on the correspondence between the structured data and the vector data, the row identifier of the structured data is written into the row where the corresponding vector data is located in the second data column. For this storage method, when performing data retrieval, the structured data is still retrieved first to obtain the first row identifier corresponding to the structured data that meets the structured retrieval conditions. The first row identifier is the row identifier of the row where the structured data is stored, and then, the vector data corresponding to the first row identifier is searched in the vector data according to the first row identifier. Finally, vector data retrieval is performed on these corresponding vector data to obtain the target vector data that meets the data retrieval conditions.

[0099] The data retrieval method provided by the embodiment of the present invention narrows the retrieval scope of vector data by using the correlation between structured data and vector data, thereby improving the retrieval efficiency of hybrid retrieval.

[0100] Embodiment 4

[0101] Figure 5 FIG. 1 is a flow chart of another embodiment of the data retrieval method provided by the present invention. Figure 5 As shown, for the above Figure 3 The data stored in the data storage method provided by the embodiment shown, the data retrieval method provided by the embodiment may further include the following steps:

[0102] S501, obtaining data retrieval conditions.

[0103] S502: Obtain, from the data column where the structured data is stored, a first row identifier corresponding to the structured data that meets the structured search condition.

[0104] In the embodiment of the present invention, structured data supports various indexes, such as BTree index, inverted index, bitmap index, etc. Structured data can be searched according to the index, thereby obtaining structured data that meets the search conditions, and further obtaining the first row identifier corresponding to the structured data.

[0105] S503: Obtain cluster centers of multiple data groups in the data column stored in the vector data.

[0106] S504: Calculate the distance between the vector search condition and the cluster centers of the plurality of data groups.

[0107] S505: Acquire at least one target data group whose distance is smaller than a preset threshold.

[0108] In the embodiment of the present invention, group retrieval can be performed for the vector data that is stored in groups. Specifically, the cluster center of each data group can be obtained, and then similar target data groups can be obtained by calculating the vector retrieval condition, that is, the distance between the vector to be retrieved and each cluster center, thereby avoiding searching in all data groups, and thus further improving the retrieval efficiency.

[0109] S506: According to the obtained first row identifier, in at least one target data group, obtain target vector data corresponding to the first row identifier and meeting the vector search condition.

[0110] In the embodiment of the present invention, a bitmap can be used to perform vector data retrieval in each target data group. Specifically, according to the group code of the target data group, the group mapping table can be read to obtain the offset of the second row identifier corresponding to the data in the target data group; then, according to the offset of the second row identifier and the first row identifier, a bitmap for the target data group is generated; then, vector retrieval is performed according to the bitmap to obtain the target vector data that meets the vector retrieval condition.

[0111] Specifically, for each target data group, a bitmap can be constructed according to the offset of its second row identifier, and the initial value of each bit in the established bitmap is zero; then, the value of the bit corresponding to the first row identifier in the bitmap is set to 1. When performing vector data retrieval, the above bitmap can be read; then, a vector search is performed for the vector data corresponding to the first row identifier corresponding to the bit with a value of 1 in the bitmap, so as to obtain the target vector data that meets the vector retrieval conditions.

[0112] The data retrieval method provided by the embodiment of the present invention narrows the retrieval scope of vector data by using the correlation between structured data and vector data, and further narrows the retrieval scope of vector data by performing group retrieval, thereby greatly improving the retrieval efficiency of hybrid retrieval.

[0113] Embodiment 5

[0114] Figure 6 A schematic diagram of a data storage device according to an embodiment of the present invention can be used to perform the following steps: Figure 2 The method steps shown. Figure 6 As shown, the device may include: a first creation module 61 , a first processing module 62 , a second processing module 63 and a third processing module 64 .

[0115] Among them, the first creation module 61 is used to create a data storage table; the first processing module 62 is used to write first type data in at least one first data column in the data storage table; the second processing module 63 is used to write second type data in at least one second data column in the data storage table, wherein one of the first type data and the second type data is vector data and the other is structured data; the third processing module 64 is used to write a row identifier of the first type data associated with the stored second type data at a corresponding position in the second data column.

[0116] In an embodiment of the present invention, when storing data for mixed data including both vector data and structured data, first, a data storage table needs to be created by the first creation module 61, and the data storage table includes multiple data columns. After the first creation module 61 creates the data storage table, the first processing module 62 writes the first type of data into the first data column in the data storage table, and the second processing module 63 writes the second type of data into the second data column in the data storage table. When storing vector data or structured data, data with different attributes are stored in different data columns according to the attributes of the data. For example, when storing vector data, the vector data of the picture can be stored in one data column, and the vector data of the video can be stored in another data column; when storing structured data, the time data can be stored in one data column, and the location data can be stored in another data column. Then, on the basis of storing the first type of data and the second type of data, the third processing module 64 also writes the row identifier of the first type of data associated with the second type of data into the corresponding position of the second data column. In the data storage process, the vector data can be stored first, and then the structured data; or the structured data can be stored first, and then the vector data. What needs to be done is to carry the row identifier of the associated previously stored data, that is, the first type of data, in the later stored data, that is, the second type of data, so as to realize the association between the structured data and the vector data.

[0117] The data storage device provided by the embodiment of the present invention facilitates the expansion of the storage structure by storing structured data and vector data in columns; and by associating structured data and vector data through row identifiers, the structured data and vector data can be truly associated in storage, so that when searching for data, the search scope of vector data can be narrowed, thereby improving the efficiency of hybrid search.

[0118] Embodiment 6

[0119] Figure 7 A schematic diagram of another embodiment of the data storage device provided by the present invention can be used to perform the following steps: Figure 3 The method steps shown. Figure 7 As shown in the above Figure 6 Based on the illustrated embodiment, the data storage device provided by the embodiment of the present invention may further include: a grouping module 71, which can be used to divide the data stored in the second data column into multiple data groups; and generate a group mapping table based on the row identifier and index code in the data group, wherein the group mapping table stores the group code for identifying the data group, and the offset of the row identifier and index code corresponding to the data in the data group.

[0120] In an embodiment of the present invention, the first type of data is structured data, and the second type of data is vector data, that is, the first processing module 62 stores the structured data first, and the second processing module 63 stores the vector data later. Then, the third processing module 64 writes the row identifier of the structured data into the row where the corresponding vector data is located in the second data column according to the correspondence between the structured data and the vector data. The grouping module 71 can further group the multiple rows of vector data stored in a column into multiple data groups. Specifically, the grouping module 71 can group the data stored in the second data column according to the similarity of the vector data based on the clustering algorithm, and save similar vector data in the same data group. Then, the grouping module 71 generates a group mapping table for each data group based on the row identifier and index code in each data group.

[0121] The data storage device provided by the embodiment of the present invention facilitates the expansion of the storage structure by storing structured data and vector data in columns; and by associating structured data and vector data through row identifiers, the structured data and vector data can be truly associated in storage, so that when retrieving data, the retrieval range of vector data can be narrowed, thereby improving the efficiency of hybrid retrieval; by grouping vector data according to their similarity, when retrieving vector data, group retrieval can be performed first, and after finding the target group, vector data retrieval can be performed, thereby speeding up the retrieval process and further improving the retrieval efficiency.

[0122] Embodiment 7

[0123] Figure 8 A schematic diagram of the structure of an embodiment of the data retrieval device provided by the present invention can be used to perform the following steps: Figure 4 The method steps shown. Figure 8 As shown, for the above Figure 6 The data stored in the data storage device provided in the illustrated embodiment, the embodiment of the present invention provides a data retrieval device including: a first acquisition module 81, a second acquisition module 82 and a third acquisition module 83.

[0124] Among them, the first acquisition module 81 is used to obtain data retrieval conditions, which include structured retrieval conditions and vector retrieval conditions; the second acquisition module 82 is used to obtain the first row identifier corresponding to the structured data that meets the structured retrieval conditions in the data column where the structured data is stored; the third acquisition module 83 is used to obtain the target vector data that corresponds to the first row identifier and meets the vector retrieval conditions in the data column where the vector data is stored based on the acquired first row identifier.

[0125] In the embodiment of the present invention, a hybrid search is performed on the data, so the first acquisition module 81 first acquires the data search conditions, that is, the structured search conditions and the vector search conditions. Compared with the search of structured data, the search process of vector data is more complicated, the amount of calculation is larger, and it is more time-consuming. Therefore, the second acquisition module 82 can first acquire the row identifier corresponding to the structured data that meets the conditions through the structured data search, that is, the first row identifier. Thereby, the search scope of the vector data search is narrowed according to the first row identifier.

[0126] against Figure 6 The data stored in the data storage device provided by the illustrated embodiment, when the first type of data is vector data and the second type of data is structured data, stores the vector data first and then the structured data. Then, based on the correspondence between the structured data and the vector data, the row identifier of the vector data is written into the row where the corresponding structured data is located in the second data column. For this storage method, when performing data retrieval, the second acquisition module 82 first performs data retrieval on the structured data to obtain the first row identifier corresponding to the structured data that meets the structured retrieval conditions. The first row identifier is the row identifier of the vector data corresponding to the structured data, and therefore, the storage location of the corresponding vector data can be directly located according to the first row identifier. Then, the third acquisition module 83 performs vector data retrieval on these corresponding vector data, thereby obtaining the target vector data that meets the data retrieval conditions.

[0127] On the other hand, when the first type of data is structured data and the second type of data is vector data, the structured data is stored first and the vector data is stored later. Then, based on the correspondence between the structured data and the vector data, the row identifier of the structured data is written into the row where the corresponding vector data is located in the second data column. For this storage method, when performing data retrieval, the second acquisition module 82 still first performs data retrieval on the structured data to obtain the first row identifier corresponding to the structured data that meets the structured retrieval conditions. The first row identifier is the row identifier of the row where the structured data is stored, and then the third acquisition module 83 searches the vector data for the vector data corresponding to the first row identifier based on the first row identifier. Finally, the third acquisition module 83 performs vector data retrieval on these corresponding vector data to obtain the target vector data that meets the data retrieval conditions.

[0128] The data retrieval device provided by the embodiment of the present invention reduces the retrieval scope of vector data by using the correlation between structured data and vector data, thereby improving the retrieval efficiency of hybrid retrieval.

[0129] Embodiment 8

[0130] Fig. 9 A schematic diagram of another embodiment of the data retrieval device provided by the present invention can be used to perform the following steps: Figure 5 The method steps shown. Fig. 9 As shown, for the above Figure 7 The data stored in the data storage device provided by the illustrated embodiment, in the data retrieval device provided by this embodiment, the third acquisition module 83 may include: a first acquisition unit 831, a calculation unit 832, a second acquisition unit 833 and a third acquisition unit 834.

[0131] Among them, the first acquisition unit 831 can be used to obtain the cluster centers of multiple data groups in the data column where the vector data is stored; the calculation unit 832 can be used to calculate the distance between the vector retrieval condition and the cluster centers of multiple data groups; the second acquisition unit 833 can be used to obtain at least one target data group whose distance is less than a preset threshold; the third acquisition unit 834 can be used to obtain, based on the acquired first row identifier, target vector data corresponding to the above-mentioned first row identifier and meeting the vector retrieval condition in at least one target data group.

[0132] In the embodiment of the present invention, group retrieval can be performed for the vector data that is stored in groups. Specifically, the first acquisition unit 831 can acquire the cluster center of each data group, and then the calculation unit 832 can calculate the vector retrieval condition, that is, the distance between the vector to be retrieved and each cluster center, and the second acquisition unit 833 can acquire similar target data groups, and the third acquisition unit 834 can acquire the target vector data corresponding to the first row identifier and meeting the vector retrieval condition in at least one target data group acquired by the second acquisition unit 833 based on the acquired first row identifier, thereby avoiding retrieval in all data groups, and thus can further improve the retrieval efficiency.

[0133] In the embodiment of the present invention, a bitmap can be used to perform vector data retrieval in each target data group. Specifically, the third acquisition unit 834 can read the group mapping table according to the group coding of the target data group to obtain the offset of the second row identifier corresponding to the data in the target data group; then, the third acquisition unit 834 generates a bitmap for the target data group according to the offset of the second row identifier and the first row identifier; then, the third acquisition unit 834 performs vector retrieval according to the bitmap to obtain the target vector data that meets the vector retrieval condition.

[0134] Specifically, the third acquisition unit 834 can construct a bitmap for each target data group according to the offset of the second row identifier, and the initial value of each bit in the established bitmap is zero; then, the value of the bit corresponding to the first row identifier in the bitmap is set to 1. When performing vector data retrieval, the above bitmap can be read; then, a vector search is performed for the vector data corresponding to the first row identifier corresponding to the bit with a value of 1 in the bitmap, so as to obtain the target vector data that meets the vector retrieval condition.

[0135] The data retrieval device provided by the embodiment of the present invention narrows the retrieval scope of vector data by using the correlation between structured data and vector data, and further narrows the retrieval scope of vector data by performing group retrieval, thereby greatly improving the retrieval efficiency of hybrid retrieval.

[0136] Embodiment 9

[0137] The internal functions and structure of the data storage device are described above, and the device can be implemented as an electronic device. Fig.10 The structure diagram of an embodiment of the electronic device provided by the present invention is shown in FIG. Fig.10 As shown, the electronic device includes a memory 101 and a processor 102 .

[0138] The memory 101 is used to store programs. In addition to the above programs, the memory 101 can also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0139] The memory 101 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0140] The processor 102 is not limited to a central processing unit (CPU), but may also be a graphics processing unit (GPU), a field programmable gate array (FPGA) or an artificial intelligence (AI) chip. The processor 102 is coupled to the memory 101 and executes the program stored in the memory 101 to:

[0141] Create a data storage table;

[0142] Writing first type data in at least one first data column in the data storage table;

[0143] Writing second type data into at least one second data column in the data storage table, wherein one of the first type data and the second type data is vector data and the other is structured data;

[0144] A row identifier of the first type of data associated with the stored second type of data is written at a corresponding position in the second data column.

[0145] Further, if Fig.10 As shown, the electronic device may further include: a communication component 103, a power component 104, an audio component 105, a display 106 and other components. Fig.10 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.10 Components shown.

[0146] The communication component 103 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 103 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 103 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0147] The power supply assembly 104 provides power to various components of the electronic device. The power supply assembly 104 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0148] The audio component 105 is configured to output and / or input audio signals. For example, the audio component 105 includes a microphone (MIC), and when the electronic device is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 101 or sent via the communication component 103. In some embodiments, the audio component 105 also includes a speaker for outputting audio signals.

[0149] The display 106 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0150] Embodiment 10

[0151] The internal functions and structure of the data retrieval device are described above. The device can be implemented as an electronic device. Fig.11 This is a schematic diagram of the structure of another embodiment of the electronic device provided by the present invention. Fig.11 As shown, the electronic device includes a memory 111 and a processor 112 .

[0152] The memory 111 is used to store programs. In addition to the above programs, the memory 111 can also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0153] The memory 111 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0154] The processor 112 is not limited to a central processing unit (CPU), but may also be a graphics processing unit (GPU), a field programmable gate array (FPGA) or an artificial intelligence (AI) chip. The processor 112 is coupled to the memory 111 and executes the program stored in the memory 111 to:

[0155] Acquiring data retrieval conditions, the data retrieval conditions including structured retrieval conditions and vector retrieval conditions;

[0156] In the data column where the structured data is stored, obtaining the first row identifier corresponding to the structured data that meets the structured search condition;

[0157] According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the data column where the vector data is stored.

[0158] Further, if Fig.11 As shown, the electronic device may also include: a communication component 113, a power component 114, an audio component 115, a display 116 and other components. Fig.11 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.11 Components shown.

[0159] The communication component 113 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 113 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 113 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0160] The power supply component 114 provides power to various components of the electronic device. The power supply component 114 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0161] The audio component 115 is configured to output and / or input audio signals. For example, the audio component 115 includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 111 or sent via the communication component 113. In some embodiments, the audio component 115 also includes a speaker for outputting audio signals.

[0162] The display 116 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0163] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data storage method, characterized in that: include: Create a data storage table; Writing first type data in at least one first data column in the data storage table; Writing second type data into at least one second data column in the data storage table, wherein the first type data is structured data and the second type data is vector data; Writing a row identifier of first type data associated with the stored second type data at a corresponding position in the second data column; Dividing the data stored in the second data column into a plurality of data groups; A group mapping table is generated according to the row identifier and index code in the data group, wherein the group mapping table stores the group code for identifying the data group and the offset of the row identifier and index code corresponding to the data in the data group.

2. The data storage method according to claim 1, characterized in that: The step of dividing the data stored in the second data column into a plurality of data groups comprises: According to the clustering algorithm, the data stored in the second data column is grouped according to the similarity of the vector data.

3. A data retrieval method for data stored in the data storage method according to any one of claims 1 to 2, characterized in that: include: Acquiring data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions; In the data column where the structured data is stored, obtaining a first row identifier corresponding to the structured data that meets the structured search condition; According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the data column where the vector data is stored.

4. The data retrieval method according to claim 3, characterized in that: The step of acquiring, according to the acquired first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition in the data column where the vector data is stored, includes: Obtaining cluster centers of multiple data groups in a data column stored in the vector data; Calculating the distance between the vector search condition and the cluster centers of the plurality of data groups; Acquire at least one target data group whose distance is less than a preset threshold; According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the at least one target data group.

5. The data retrieval method according to claim 4, characterized in that: The step of acquiring, in the at least one target data group, target vector data corresponding to the first row identifier and meeting the vector search condition according to the first row identifier acquired, includes: Reading a group mapping table according to the group code of the target data group, and acquiring an offset of a second row identifier corresponding to the data in the target data group; generating a bitmap for the target data group according to the offset of the second row identifier and the first row identifier; Vector search is performed according to the bitmap to obtain target vector data that meets the vector search conditions.

6. The data retrieval method according to claim 5, characterized in that: The step of constructing a bitmap for the target data group according to the offset of the second row identifier and the first row identifier includes: For the target data group, construct a bitmap according to the offset identified by the second row, wherein an initial value of each bit in the bitmap is zero; The value of the bit corresponding to the first row identifier in the bitmap is set to one.

7. The data retrieval method according to claim 6, characterized in that: The performing vector search according to the bitmap to obtain target vector data that meets the vector search condition includes: Reading the bitmap; A vector search is performed on the vector data corresponding to the first row identifier corresponding to the bit with a value of one in the bitmap to obtain target vector data that meets the vector search condition.

8. A data storage device, characterized in that: include: The first creation module is used to create a data storage table; A first processing module, configured to write first type data into at least one first data column in the data storage table; A second processing module, configured to write second type data into at least one second data column in the data storage table, wherein the first type data is structured data and the second type data is vector data; A third processing module, configured to write a row identifier of first type data associated with the stored second type data at a corresponding position in the second data column; A grouping module is used to divide the data stored in the second data column into multiple data groups; and generate a group mapping table based on the row identifier and index code in the data group, wherein the group mapping table stores the group code for identifying the data group, and the offset of the row identifier and index code corresponding to the data in the data group.

9. The data storage device according to claim 8, characterized in that: The grouping module is specifically used to group the data stored in the second data column according to the similarity of vector data based on a clustering algorithm.

10. A data retrieval device for data stored in the storage device according to any one of claims 8 to 9, characterized in that: include: A first acquisition module, used to acquire data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions; A second acquisition module, configured to acquire, from the data column where the structured data is stored, a first row identifier corresponding to the structured data that meets the structured search condition; The third acquisition module is used to acquire, according to the acquired first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition in the data column where the vector data is stored.

11. The data retrieval device according to claim 10, characterized in that: The third acquisition module includes: A first acquisition unit, used for acquiring cluster centers of a plurality of data groups in a data column stored in the vector data; A calculation unit, used for calculating the distance between the vector search condition and the cluster centers of the plurality of data groups; A second acquisition unit, configured to acquire at least one target data group whose distance is less than a preset threshold; The third acquisition unit is used to acquire, according to the acquired first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition in the at least one target data group.

12. The data retrieval device according to claim 11, characterized in that: The third acquisition unit is specifically configured to read the group mapping table according to the group code of the target data group, and acquire the offset of the second row identifier corresponding to the data in the target data group; and generate a bitmap for the target data group according to the offset of the second row identifier and the first row identifier; Vector search is performed according to the bitmap to obtain target vector data that meets the vector search conditions.

13. An electronic device, characterized in that: include: Memory, used to store programs; A processor, configured to run the program stored in the memory, to: Create a data storage table; Writing first type data in at least one first data column in the data storage table; Writing second type data into at least one second data column in the data storage table, wherein the first type data is structured data and the second type data is vector data; Writing a row identifier of first type data associated with the stored second type data at a corresponding position in the second data column; Dividing the data stored in the second data column into a plurality of data groups; A group mapping table is generated according to the row identifier and index code in the data group, wherein the group mapping table stores the group code for identifying the data group and the offset of the row identifier and index code corresponding to the data in the data group.

14. An electronic device, characterized in that: include: Memory, used to store programs; A processor, configured to run the program stored in the memory, to: Acquiring data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions; In the data column where the structured data is stored, obtaining a first row identifier corresponding to the structured data that meets the structured search condition; According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the data column where the vector data is stored.

15. A computer-readable storage medium having instructions stored thereon, the instructions comprising: Create a data storage table; Writing first type data in at least one first data column in the data storage table; Writing second type data into at least one second data column in the data storage table, wherein the first type data is structured data and the second type data is vector data; Writing a row identifier of first type data associated with the stored second type data at a corresponding position in the second data column; Dividing the data stored in the second data column into a plurality of data groups; A group mapping table is generated according to the row identifier and index code in the data group, wherein the group mapping table stores the group code for identifying the data group and the offset of the row identifier and index code corresponding to the data in the data group.

16. A computer-readable storage medium having instructions stored thereon, the instructions comprising: Acquiring data retrieval conditions, wherein the data retrieval conditions include structured retrieval conditions and vector retrieval conditions; In the data column where the structured data is stored, obtaining a first row identifier corresponding to the structured data that meets the structured search condition; According to the obtained first row identifier, target vector data corresponding to the first row identifier and meeting the vector search condition is obtained in the data column where the vector data is stored.

Images