Data recommendation method, device, computing device and computer-readable storage medium

Abstract

The embodiment of the present invention relates to the field of data processing technology and discloses a data recommendation method, which includes: obtaining a query request, wherein the query request includes data to be matched; processing the data to be matched into a target embedding vector; generating a target embedding vector hash key-value pair based on the target embedding vector; querying a pre-built embedding vector database based on the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched; the pre-built embedding vector database includes multiple pre-stored embedding vector hash key-value pairs; the embedding vector hash key-value pair includes a local sensitive hash value of the embedding vector and the embedding vector itself; and generating a recommendation result based on the embedding vector hash key-value pair list. Through the above method, the embodiment of the present invention achieves the beneficial effect of improving the efficiency of data recommendation.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and specifically to a data recommendation method, apparatus, computing device, and computer-readable storage medium. Background Art

[0002] Currently, mainstream recommendation systems have three core and critical stages: feature engineering, recall, and ranking. Nearest neighbor recommendation can occur in the recall or ranking stage, or both, depending on the application and scenario, and also relies on the output of the feature engineering stage as its input. Nearest neighbor recommendation technology often uses the concept of collaborative filtering. It first constructs a vector or relationship matrix, then calculates the similarity between vectors pairwise using similarity calculation methods (such as cosine similarity) or weighted quantization. Based on the similarity and the relationships between vectors and matrices, it forms an item similarity matrix or similar user-item relationships, or builds a model to predict scores for missing items in the relationship matrix. The final recommendation results are then sorted based on the scores and similarities, resulting in low data processing efficiency. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention provide a data recommendation method, apparatus, computing device, and computer-readable storage medium to solve the technical problem in the prior art of being unable to perform data recommendation efficiently.

[0004] According to one aspect of an embodiment of the present invention, a data recommendation method is provided, the method comprising:

[0005] Obtaining a query request, wherein the query request includes data to be matched;

[0006] Processing the data to be matched into a target embedding vector;

[0007] Generate a target embedding vector hash key-value pair according to the target embedding vector;

[0008] A pre-built embedding vector database is searched according to the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched; the pre-built embedding vector database includes a plurality of pre-stored embedding vector hash key-value pairs, each embedding vector hash key-value pair including a locality-sensitive hash value of the embedding vector and the embedding vector itself;

[0009] Generate recommendation results based on the embedding vector hash key-value pair list.

[0010] In an optional manner, the method for constructing the embedding vector database includes: acquiring user behavior data in streaming or batch manner; processing each acquired user behavior data into an embedding vector; generating an embedding vector hash key-value pair based on the embedding vector; and storing the generated embedding vector hash key-value pair in the embedding vector database.

[0011] In an optional manner, generating a target embedding vector hash key-value pair based on the target embedding vector includes: generating a hash tree of the same dimension as the embedding vector; the hash tree includes multiple random vectors; respectively calculating the angle between the embedding vector and each random vector in the hash tree; determining the local sensitive hash value based on the angle; and generating the target embedding vector hash key-value pair based on the target local sensitive hash value and the target embedding vector.

[0012] In an optional manner, the embedding vector database includes basic index data, neighbor set index data, and embedding vector index data; the method for constructing the embedding vector database includes: establishing neighbor set index data in the embedding vector database; the neighbor set index data includes multiple neighbor sets stored corresponding to keywords of embedding vector hash key-value pairs, and the neighbor sets include similar embedding vector hash key values; establishing basic index data in the embedding vector database; the basic index data includes multiple topic sets stored corresponding to topics, and the topic sets include multiple embedding vector hash key-value pairs with the same topics; establishing embedding vector index data in the embedding vector database; the embedding vector index data includes multiple sets of embedding vectors stored according to topics and user identifiers or content identifiers.

[0013] In an optional manner, the embedding vector database uses an embedding index data structure for data storage, and the embedding index data structure includes a row key column and a data storage column cluster; the neighbor set index data includes a first row key column and a first data storage column cluster, the first row key column includes multiple first row key information, the first data storage column cluster includes at least one neighbor set corresponding to the first row key information, and the neighbor set stores an embedding vector hash key-value pair; the first row key information is generated according to the keyword key of the embedding vector hash key-value pair; the basic index data includes a second row key column and a second data storage column cluster, the second row key column includes multiple second row key information, the second data storage column cluster includes the embedding vector hash key-value pair corresponding to the second row key information; the second row key information is generated according to the subject; the embedding vector index data includes a third row key column and a third data storage column cluster, the third row key column includes multiple third row key information, the third data storage column cluster includes the embedding vector corresponding to the third row key information, and the third row key information is generated according to the subject and user identifier or content identifier.

[0014] In an optional embodiment, the method further includes: obtaining user behavior data to be input; determining one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier of an embedding vector hash key-value pair of the user behavior data to be input; determining one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier of the embedding vector hash key-value pair of the user behavior data to be input, searching the index database for a matching target row key column, wherein the target row key column is any one or more of the first row key column, the second row key column, and the third row key column; determining whether the user behavior data to be input is duplicate data based on the target row key column and a corresponding target data storage column cluster; the target row key column stores one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier, and the target data storage column cluster stores an embedding vector corresponding to the target row key column; when the user behavior data to be input is duplicate data, deleting the user behavior data to be input; and when the user behavior data to be input is not duplicate data, adding the user behavior data to be input to the embedding vector database.

[0015] In an optional manner, after establishing the neighbor set index data in the embedding vector database, the method includes: constructing an index structure based on the neighbor set index data; the index structure includes a primary index, a primary index object, a secondary index, and a secondary index object; the primary index includes an index truncation of the keyword key of the embedding vector hash key-value pair, and the primary index object is a local sensitive hash value in the embedding vector hash key-value pair containing the index truncation; the secondary index is a local sensitive hash value in the embedding vector hash key-value pair, and the secondary index object is the embedding vector hash key-value pair.

[0016] In an optional manner, establishing basic index data in the embedding vector database includes: sorting all embedding vector hash key-value pairs according to the size of the local sensitive hash value; storing the sorted embedding vector hash key-value pairs in multiple topic sets according to a preset storage threshold order; the preset storage threshold is a threshold for the number of embedding vector hash key-value pairs stored in each topic set.

[0017] In an optional manner, a pre-built embedding vector database is queried according to the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched, including: indexing the target local sensitive hash value in the target embedding vector hash key-value pair in the neighbor set index data to determine a target neighbor set that matches the target local sensitive hash value; calculating the similarity between the target local sensitive hash value and the local sensitive hash value in each embedding vector hash key-value pair in the target neighbor set; and determining the embedding vector hash key-value pair list based on the similarity.

[0018] According to another aspect of an embodiment of the present invention, a data recommendation device is provided, comprising:

[0019] An acquisition module, configured to acquire a query request, wherein the query request includes data to be matched;

[0020] A processing module, configured to process the to-be-matched data into a target embedding vector;

[0021] A generation module, configured to generate a target embedding vector hash key-value pair according to the target embedding vector;

[0022] a matching module configured to query a pre-built embedding vector database based on the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched; the pre-built embedding vector database includes a plurality of pre-stored embedding vector hash key-value pairs; the embedding vector hash key-value pairs include a locality-sensitive hash value of the embedding vector and the embedding vector itself;

[0023] The recommendation module is used to generate recommendation results based on the embedding vector hash key-value pair list.

[0024] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a computing device, the computing device executes the operation of the data recommendation method.

[0025] The embodiment of the present invention maps the embedding vector into a local hash value to ensure local sensitivity, and establishes a deep conversion and combination with the local sensitive hashing technology. It is not limited to static text or one-dimensional space, and can further reduce the dimensionality of high-dimensional objects or things. Therefore, after obtaining the query request, the data to be matched in the query request is processed into a target embedding vector, and a target embedding vector hash key-value pair is generated according to the target embedding vector. According to the target embedding vector hash key-value pair, a query is performed in a pre-constructed embedding vector database to obtain a list of embedding vector hash key-value pairs matching the data to be matched, wherein the pre-constructed embedding vector database includes multiple pre-stored embedding vector hash key-value pairs, and the embedding vector hash key-value pairs include the local sensitive hash value of the embedding vector and the embedding vector. Finally, a recommendation result is generated according to the embedding vector hash key-value pair list, thereby effectively improving the efficiency of neighbor data recommendation.

[0026] Furthermore, by establishing index data and performing special splitting and sorting, when the data to be deduplicated is put into the warehouse, the construction of multi-level indexes allows the global data to be moved dynamically and regularly, so that the incremental calculation of accurate deduplication is completed synchronously when the data is put into the warehouse. This process is incremental accumulation rather than full calculation each time, making the index construction process efficient, and the accurate deduplication value can be queried in real time. There is no calculation process during the query, and it is completely unrelated to the amount of data, further improving the data processing efficiency.

[0027] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to more clearly understand the technical means of the embodiments of the present invention, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0029] Figure 1 A schematic diagram showing a flow chart of a data recommendation method provided by an embodiment of the present invention is shown;

[0030] Figure 2 A diagram showing an application environment of the data recommendation method provided by an embodiment of the present invention;

[0031] Figure 3 A schematic diagram showing the structure of neighbor set index data in the data recommendation method provided by an embodiment of the present invention is shown;

[0032] Figure 4 A schematic diagram showing the structure of basic index data in the data recommendation method provided by an embodiment of the present invention;

[0033] Figure 5 A schematic diagram showing the structure of embedding vector index data in the data recommendation method provided by an embodiment of the present invention is shown;

[0034] Figure 6 A schematic diagram of the process of data synchronization and index construction calculation in the data recommendation method provided by an embodiment of the present invention is shown;

[0035] Figure 7 A schematic diagram of the process of nearest neighbor search and accurate deduplication in the data recommendation method provided by an embodiment of the present invention is shown;

[0036] Figure 8 A schematic structural diagram of a data recommendation device provided by an embodiment of the present invention is shown;

[0037] Figure 9 A schematic structural diagram of a computing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0039] The inventors of this application discovered during the implementation of the embodiments of the present invention that, in the prior art, neighbor recommendation technology often uses the idea of ​​collaborative filtering, which consumes a lot of model computing resources, especially in the recall stage, which usually increases resource consumption and burden when faced with a large amount of data or even massive data. At the same time, calculating the similarity between vectors one by one (such as calculating cosine similarity) consumes a lot of resources in engineering and is usually inefficient. It usually needs to be completed offline in advance and cannot achieve quasi-real-time results to meet scenarios that are sensitive to effectiveness. Some neighbor search or recommendation schemes with higher timeliness usually use clustering algorithms for the most critical similarity calculations, and avoid calculating similarities one by one by comparing with the calculation of cluster centers to improve efficiency. However, compared with one-by-one comparison, this usually lacks accuracy. At the same time, the prior art cannot simultaneously support both efficient neighbor recommendation and accurate deduplication for massive data. In addition, similarity can be calculated using local sensitive hashing. However, existing local sensitive hashing techniques are primarily used for text content (webpages), which are textual and have real meaning. However, the same approach cannot be used for embedding vectors generated by machine learning, as the spatial nature of the vectors cannot be preserved. Furthermore, when dealing with large amounts of data, comparing each embedding individually can be inefficient.

[0040] The embodiments of the present invention achieve neighbor recommendation and improve the effectiveness of recommendations in massive data scenarios by designing and implementing a specific embedding index data structure and performing key processes such as index construction, split sorting, neighbor aggregation, and neighbor search. This effectiveness is no longer constrained by accuracy. The key problem it solves is to support efficient real-time (up to millisecond level) neighbor recommendation for massive data (billions of data) and accurate data deduplication (the amount of accurate deduplication data is theoretically unlimited and can be obtained in real time). Based on multi-level / classified index construction and the dynamic and regular movement of global data, the same system device can simultaneously support neighbor recommendation and accurate deduplication for massive data.

[0041] Figure 1 The flowchart of the data recommendation method provided by the embodiment of the present invention is shown, and the method is executed by a computing device. The computing device can be a computer device, a terminal device, a cloud processing device, a server, etc., and the embodiment of the present invention does not make specific restrictions. Figure 1 As shown, the method includes the following steps:

[0042] Step 110: Obtain a query request, which includes data to be matched.

[0043] The query request may be a request triggered when a user click or browse behavior is obtained. The data to be matched may be data generated by the user click or browse behavior, including user information data such as user ID and user gender, and content data such as browsed or clicked content.

[0044] Step 120: Process the data to be matched into a target embedding vector.

[0045] Among them, embedding vector, also known as embedding vector, is a general term for a set of language modeling and feature learning techniques in word embedded natural language processing (NLP), in which words or phrases from a vocabulary are mapped to vectors of real numbers, which involves mathematical embedding from a one-dimensional space for each word to a continuous vector space with a lower dimension.

[0046] In an embodiment of the present invention, after obtaining the data to be matched, embedding vector processing is performed on the data to be matched to obtain a target embedding vector. Specifically, based on unified label classification, the target embedding vector is obtained through unsupervised modeling (using word2vec or item2vec for sequence data; deepwalk or node2vec for graph data), matrix decomposition (such as collaborative filtering matrix decomposition method), DNN deep learning method (for example, in a DNN model, the weights of the embedding layer), etc.

[0047] The data to be matched includes user information data and content data. For the user information data and content data, embedding vector processing can be performed respectively to obtain user embedding and content embedding.

[0048] Step 130: Generate a target embedding vector hash key-value pair according to the target embedding vector.

[0049] In the embodiment of the present invention, the specific process of generating a target embedding vector hash key-value pair according to the target embedding vector includes:

[0050] Generate a hash tree with the same dimensions as the embedding vector. In this embodiment of the present invention, 64 random vectors X form the hash tree. For two embedding vectors S and V, their similarity (distance) is defined as: P = SV / |S||V|. Generate an n-dimensional random vector X from an n-dimensional vector space with the same dimensions as the embedding vector. Use this random vector X as the normal vector, and its corresponding random plane is Px.

[0051] Calculate the angle between the target embedding vector Y and the random vector X.

[0052] The target embedding vector hash key-value pair is determined based on the angle. The target embedding vector Y is hashed with the random vector X, where the space on the side indicated by the random vector X represents 1, and the opposite side represents 0. The angle between the target embedding vector Y and the random vector X can determine the spatial direction to which the target embedding vector belongs, that is, if Y·X>0, the hash value is 1, otherwise the hash value is 0. Each random vector X is separately calculated with the target embedding vector Y, thereby mapping the target embedding vector Y into a 64-bit hash value consisting of 0 and 1, thereby obtaining the target local sensitive hash value.

[0053] The embodiments of the present invention utilize a specific local hashing algorithm. This key difference from traditional local sensitive hashing algorithms (such as minhash or simhash) is that minhash and simhash primarily target text (webpage) objects, whose content is textual and has actual meaning or semantics. This mapping transformation allows any technology involving spatial vectors (such as embedding) to be deeply integrated and transformed with local sensitive hashing. This approach is not limited to static text or one-dimensional space, but can also provide further dimensionality reduction for high-dimensional objects or entities.

[0054] In an embodiment of the present invention, after obtaining the target local sensitive hash value corresponding to the target embedding vector, a target embedding vector hash key-value pair is constructed. Specifically, the target local sensitive hash value is used as the key and the target embedding vector is used as the value, thereby obtaining the target embedding vector hash key-value pair. The value of the target embedding vector hash key-value pair may also include the ID value of the target embedding vector and the corresponding object.

[0055] Step 140: Query the pre-built embedding vector database according to the target local sensitive hash value to obtain a list of embedding vector hash key-value pairs that match the data to be matched.

[0056] The pre-built embedding vector database includes multiple pre-stored embedding vector hash key-value pairs; the embedding vector hash key-value pairs include the local sensitive hash value of the embedding vector and the embedding vector itself.

[0057] In an embodiment of the present invention, before querying the pre-constructed embedding vector database according to the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched, it also includes: establishing index data in the embedding vector database. In an embodiment of the present invention, functionally speaking, neighbor set index data, basic index data, and embedding vector index data are respectively constructed in the embedding vector database. Specifically, neighbor set index data is established in the embedding vector database; the neighbor set index data includes multiple neighbor sets stored corresponding to the keyword key of the embedding vector hash key-value pair, and the neighbor sets include similar embedding vector hash key-value pairs and embedding vector hash key-value pair information. Basic index data is established in the embedding vector database; the basic index data includes multiple topic sets stored corresponding to the topic, and the topic set includes multiple embedding vector hash key-value pairs and embedding vector hash key-value pair information with the same topic. Embedding vector index data is established in an embedding vector database; the embedding vector index data includes a set of pairs of multiple embedding vectors stored according to the subject and the user identifier or the content identifier.

[0058] Among them, in terms of storage form, the embedding vector database uses embedding index data for data storage, and the storage form of the embedding index data includes a row key column and a data storage column cluster. The row key column includes the keyword key, subject, and one or more of the subject and user identifier or content identifier of the embedding vector hash key-value pair; the data storage column cluster includes the embedding vector hash key-value pair set corresponding to the row key column. This step is implemented based on a preset data recommendation device, which includes an embedding vector database. Figure 2As shown, in an embodiment of the present invention, the data recommendation device is pre-constructed, and the data recommendation device can construct a data index according to the topic, construct a data index according to the local sensitive hash value, and construct a data index according to the topic and user identifier or content identifier. Specifically, the data recommendation device includes several modules: a feature engineer, an index builder, an embedding vector database, and a neighbor recommender. Among them, the feature engineer processes the data acquired in real time or periodically to obtain the corresponding embedding vector, and generates the corresponding local sensitive hash value through the above-mentioned local sensitive hash value generation algorithm. According to the local sensitive hash value and the corresponding embedding vector, a target embedding vector hash key-value pair is constructed. Specifically, the local sensitive hash value is used as the keyword key and the embedding vector is used as the value value, so that an embedding vector hash key-value pair can be obtained. Among them, the value value of the embedding vector hash key-value pair can also include the embedding vector and the ID value of the corresponding object, such as user ID, user group ID, content ID, identification ID, etc.

[0059] After obtaining the data's embedding vector hash key-value pairs, these embedding vector hash key-value pairs are stored as index data in the embedding vector database. Specifically, when storing the embedding vector database, each index data item consists of two components: the row key column [ROW_KEY] and the data storage column clusters, which are organized into column clusters. The row key column [ROW_KEY] is structured as [Row_Hash_Key]l[Dim_Type][Key_Name].[DateTime]. [Row_Hash_Key] is a hash function to prevent data hotspots. The hash value is calculated by hashing the value of [Dim_Type][Key_Name].[DateTime] modulo the pre-partition number. The hash value is obtained by taking the absolute value of the hashed value of [Dim_Type][Key_Name].[DateTime]. The data storage column cluster is used to store embedding vector hash key-value pairs and related data corresponding to the row key column [ROW_KEY]. The following details the specific construction process of neighbor set index data, basic index data, and embedding vector index data from the perspective of storage format.

[0060] Create neighbor set index data in the embedding vector database: Figure 3As shown, the structure of the neighbor set index data includes a first row key column and a first data storage column cluster. The first row key column includes multiple first row key information. The first data storage column cluster includes at least one neighbor set corresponding to the first row key information. The neighbor set stores embedding vector hash key-value pairs and embedding vector hash key-value pair information. The first row key information is generated based on the keyword key of the embedding vector hash key-value pair. That is, the structure of the first row key column [ROW_KEY] is as described above, where the encoding of [Dim_Type] is used to distinguish whether the data column corresponding to the current first row key value [ROW_KEY] is the set index position or the set itself; [Key_Name] is the keyword key of the embedding vector hash key-value pair. Among them, the structure of the first data storage column cluster includes [NI][NV][NT][NS1][NS2][...][NSi], [NI] in the first data storage column cluster is the position (index) of the [NSi] set corresponding to [ROW_KEY], [NV] is the newly inserted embedding vector hash key-value pair, [NT] is the sum of the number of all set columns ([NSi]) under the current [ROW_KEY], and [NSi] is a set of similar vectors, the elements of which include multiple similar embedding vector hash key-value pairs, wherein the value in the embedding vector hash key-value pair is also added with a timestamp.

[0061] Establish basic index data in the embedding vector database: the basic index data is a set of embedding vector hash key-value pairs corresponding to each indicator under a certain topic. Figure 4As shown, the structure of the basic index data includes a second row key column [ROW_KEY] and a second data storage column cluster, wherein the second row key column includes multiple second row key information, and the second data storage column cluster includes an embedding vector hash key-value pair corresponding to the second row key information; the second row key information is generated according to the subject. Specifically, the structure of the second row key column [ROW_KEY] is as described above, wherein [Dim_Type] is a dimension or classification code, which is used to distinguish different data subjects in the basic index data, and the subject can be set accordingly according to the specific scenario; [Key_Name] is an indicator code, which is an n-bit binary string consisting of 0 and 1 or a code corresponding to a certain indicator in the basic index data, and the code of the indicator can be used for accurate deduplication calculation of the indicator; [DateTime] is a date and time code. The structure of the second data storage column cluster includes [QV][ST][SD][S1][S2][...][Si]. The [QV] column in the second data storage column cluster represents the total number of objects in all sets [Si] under the current [ROW_KEY]. [ST] represents a queue of pre-stored object values ​​under the current [ROW_KEY]. [SD] represents the elements stored in all sets [Si], deduplicated by key. [Si] represents a specific value storage set. Its stored elements are an ordered map-type data structure, storing object data in the form of key-value pairs, i.e., embedding vector hash key-value pairs. It should be noted that in embodiments of the present invention, a certain topic may correspond to multiple indicators, each of which may correspond to different dimensions. By comparing the dimensions of the same indicator, the status of the indicator under a certain topic can be reflected. Indicators are units or methods used to measure the development level of things, also known as metrics. Examples include population, GDP, revenue, number of users, profit margin, retention rate, and coverage rate. Dimensions are characteristics of things or phenomena, such as gender, region, and time.

[0062] Establish embedding vector index data in the embedding vector database: the embedding vector index data is an embedding direct mapping index, and the embedding vector index data includes the third row key column and the third data storage column cluster. Figure 5As shown, the third row key column includes multiple third row key information, and the third data storage column cluster includes the embedding vector corresponding to the third row key information, and the third row key information is generated according to the subject and user identifier or content identifier. Specifically, the structure of the third row key column [ROW_KEY] is the same as the above structure, where [Key_Name] is the user or user group ID, or content ID or identification ID, and is classified according to [Dim_Type]. The structure of the third data storage column cluster is [EV][VERSION], where [EV] is the corresponding specific embedding vector value, and [VERSION] is the version number corresponding to [EV].

[0063] After constructing the neighbor set index data, basic index data and embedding vector index data, the embedding vector index data in the embedding vector database are indexed by the index builder. Specifically, an index structure is constructed for the neighbor set index data, and the index structure corresponding to the neighbor set index data includes a primary index, a primary index object, a secondary index, and a secondary index object. Among them, the primary index includes the index truncation of the keyword key of the embedding vector hash key-value pair, and the primary index object is the local sensitive hash value in the embedding vector hash key-value pair containing the index truncation; the secondary index is the local sensitive hash value in the embedding vector hash key-value pair, and the index object corresponding to the secondary index is the embedding vector hash key-value pair. Specifically, the value key of the embedding vector hash key-value pair (that is, the binary local sensitive hash value of N bits) is truncated and segmented in bit order from low to high as the primary index. The object of its index (that is, and the index object) is the secondary index composed of the full hash value of the index truncation. The secondary index key is a complete locality-sensitive hash value, and the secondary index object is a specific embedding vector hash key-value pair, avoiding the need for bit-by-bit comparison and calculation during the subsequent similarity calculation. The index structure corresponding to the basic index data is consistent with the index structure corresponding to the neighbor set index data described above and is not further described here.

[0064] After the index is constructed, a neighbor search is performed through the index. Specifically, the process of obtaining a list of embedding vector hash key-value pairs that match the data to be matched is as follows: indexing the target local sensitive hash value in the target embedding vector hash key-value pair in the neighbor set index data, and determining a target neighbor set that matches the target local sensitive hash value; calculating the similarity between the target local sensitive hash value and the local sensitive hash values ​​in each embedding vector hash key-value pair in the target neighbor set; and determining the embedding vector hash key-value pair list based on the similarity. Among them, since the first row key information of the neighbor set index data is generated according to the keyword key of the embedding vector hash key-value pair, the target local sensitive hash value in the target embedding vector hash key-value pair of the data to be matched is determined, and the first row key column [ROW_KEY] in the row key format of the neighbor set index data corresponding to the target local sensitive hash value is obtained. According to the first row key column [ROW_KEY], indexing is performed in the above index structure, first obtaining the [NI] value in the data storage cluster to obtain the neighbor set position, and then obtaining the set [NSi] from the memory cache according to the position, and then according to the key of the embedding vector hash key-value pair, the neighbor set position is obtained. The key is directly checked to see if it exists in the set [NSi]. If so, the [NV] value is checked to see if it matches the current embedding vector hash key-value pair. If so, the set consisting of the values ​​of the top N elements in the corresponding set [NSi] is returned as the recommended embedding vector hash key-value pair list. If not, the Hamming distance is calculated between the keys of the elements in the [NSi] set and the key of the current embedding vector hash key-value pair. The values ​​of the top N elements are sorted by Hamming distance and returned as the recommended embedding vector hash key-value pair list. When querying, the value is first retrieved from the memory cache. If the corresponding value is not found, it is retrieved from the neighbor set index in the underlying index data structure. If the current embedding vector hash key-value pair key does not exist in [NSi], a similarity calculation and neighbor aggregation operation are performed to obtain the specified top N sorted set and return it. This is then flushed to the corresponding [NSi] set in the neighbor set index and loaded into memory. If no neighbor value is still found, the hot content column is returned by default.For index truncation, the local sensitive hash value to be matched is truncated bit by bit (e.g. 16 bits as a truncation segment) and a chain or set is retrieved level by level according to the first-level index. The hash value to be matched is compared with the hash value in the set. Assume that it is divided into 4 truncations (each truncation is 16 bits) and the corresponding 4 ROW_KEYs are generated. The neighboring set index data is traversed with these 4 ROW_KEYs. According to the key value of the embedding vector hash key-value pair, the data set column corresponding to the ROW_KEY is found in the basic index according to the third-level index [Si] , and directly determine whether the key value exists in the set. If it exists, similarity calculations are performed with the keys of the elements in the set in turn. This can improve efficiency. For example, there are 2^32 pieces of data to be matched and calculated. Each hash value data is 64 bits, and there is a truncation for every 16 bits. Then there are 2^16 combinations for each 16 bits. The candidate results of each truncation index are 2^(32-16). If there are 4 truncations, the total number is 4*2^(32-16). Compared with the original 2^32 comparison calculations one by one, the efficiency of preliminary retrieval and construction is improved.

[0065] In an embodiment of the present invention, the method further includes data deduplication, specifically including the following steps: obtaining user behavior data to be input; determining the keyword key, subject, and one or more of the subject and user identifier or content identifier of the embedding vector hash key-value pair of the user behavior data to be input; determining the keyword key, subject, and one or more of the subject and user identifier or content identifier of the embedding vector hash key-value pair of the user behavior data to be input; searching for a matching target row key column in the embedding vector database; the target row key column is any one of the first row key column, the second row key column, and the third row key column or more; determine whether the user behavior data to be incoming is duplicate data based on the target row key column and the corresponding target data storage column cluster; the target row key column stores one or more of the keyword key, subject, and subject and user identifier or content identifier, and the target data storage column cluster stores an embedding vector hash key-value pair or embedding vector corresponding to the target row key column; when the user behavior data to be incoming is duplicate data, delete the user behavior data to be incoming; when the user behavior data to be incoming is not duplicate data, add the user behavior data to be incoming to the embedding vector database.

[0066] Only by constructing the neighbor set index data, basic index data and embedding vector index data in the above manner, it is possible to compare and calculate whether a certain hash value has similar objects in the library to be matched, but it is not enough to achieve efficient neighbor recommendation and accurate deduplication. At the same time, the retrieval of objects in each truncated index (primary index) is still not efficient. Therefore, it is necessary to first use a special split sort to globally sort and move the object elements of the primary index (secondary index structure) according to the key of the secondary index. The chain heads of each set are ordered, and the chain head element is used as the third-level index. It can be indexed to a certain chain (set) in the serial chain with a maximum time complexity of O(logN). This global sorting and movement is a real-time and dynamic splitting process. The multiple sets after the split are logically presented as a serial chain structure connected end to end. Therefore, when establishing index data, the embodiment of the present invention also sorts the index data through the index builder, including the following steps:

[0067] Sort the embedding vector hash key-value pairs in the embedding vector hash key-value pair set; wherein, the sorting can be performed according to the size of the embedding vector hash key value. Specifically, taking the basic index data as an example:

[0068] Sort all embedding vector hash key-value pairs according to the size of the local sensitive hash value;

[0069] The sorted embedding vector hash key-value pairs are stored in a plurality of topic sets in an order according to a preset storage threshold; the preset storage threshold is a threshold value for the number of embedding vector hash key-value pairs stored in each topic set.

[0070] In an embodiment of the present invention, the neighbor set index data, the basic index data and the embedding vector index data can be sorted respectively. For example, the process of sorting the basic index data includes: splitting and sorting when writing data, and concurrently obtaining the values ​​of the objects to be written in the write queues of different ROW_KEYs, that is, the values ​​in [QV]. For a certain [ROW_KEY], if the initial data set [Si] is empty, it is directly written into the first set [S1]. If there is already a value in [S1], it is written in ascending order based on the keyword key of the embedding vector hash key-value pair. If the number of elements in the S1 set reaches the preset threshold, the second set [S2] is split out, and the embedding vector hash key-value pair with the last one in the [S1] set, that is, the largest keyword key, is moved into the [S2] set. If the [S2] set is also full, the last element is also moved into the next set, that is, [S3], and so on. This ensures that the elements in the [Si] (i = 1, 2, ...) set are sorted in ascending order and the data is constantly moving and changing. These sets logically form a serial chain structure connected at the first position, that is, the key of the first element in each chain (set) is the smallest, the key of the last element is the largest, and the chain heads of each set are in order. Subsequent writes first determine the size of the keyword key value of the first element of all [Si] sets and the keyword key value to be inserted. The position to be inserted is found using the chain head element (that is, the first element of [SI] under [QV] corresponding to a certain [ROW_KEY]) as the third-level index. It can index to a chain (set) in the serial chain with a maximum time complexity of O(logN). Then insert it into the corresponding set in order and move the data (if necessary). If the key of the element to be inserted matches the same value in the set [Si], the value of the element with the corresponding key is updated. After the sorting module completes the sorting, it atomically updates the number of elements in the current set to the [ST] column and atomically increases the [SD] value by one as the deduplication value of the number of elements in the current set.

[0071] Among them, when achieving accurate deduplication of indicator data, since it does not involve similarity calculation and comparison of local sensitive hashing, the index construction process does not involve index construction of local sensitive hashing, which can be more efficient. That is, for basic index data, the keyword key of its first-level index is the indicator code, the keyword key of the second-level index is the identification ID of the indicator data (such as user ID, etc.), and the object of the index is the data corresponding to the indicator data identification. These data undergo the above-mentioned splitting and sorting process and are added to a certain chain. When added, the above-mentioned first-level, second-level, and third-level indexes will be used to directly search whether the indicator exists according to the keyword key. If it already exists, the value will be discarded or updated; if it does not exist, the data element will be added and the dynamic deduplication value will be accumulated and updated as the global indicator deduplication value of the current time node. For different time ranges, the deduplication value of the corresponding range will be synchronously accumulated and updated according to the data timestamp and data location classification information. At this point, the accurate deduplication incremental calculation of the data is completed. This process is continuously iteratively executed with the increasing indicator data, and is a dynamic process. Therefore, by pre-sorting processing, real-time dynamic accurate deduplication and inventory update are achieved.

[0072] Combine 2 and Figure 6 In one embodiment of the present invention, the pre-built embedding vector database can update the neighbor set index data (neighbor recommendation index construction) and basic index data (precise deduplication index construction) according to the original data in a preset period or real-time streaming or batch processing, and update them in the pre-built embedding vector database. Therefore, when incrementally storing, it is necessary to determine whether the data is repeated indicator data and to be able to quickly update it in the neighbor set index. Specifically, if Figure 6As shown, for neighbor recommendation index construction: After receiving raw data in streaming or batch processing, the neighbor recommendation index construction module obtains the embedding vector hash key-value pairs corresponding to the raw data and inputs them into the basic index module. The key value of the embedding vector hash key-value pair is segmented according to the above truncation method, which serves as the first-level index. Its index object is a second-level index structure composed of the fully local sensitive hash value containing the truncated index. The key of the second-level index structure is the fully local sensitive hash value, and the index object is a specific embedding vector hash key-value pair. The sorting module is connected to the sorting module for classification and sorting. Because the sorting module uses a special split sorting method to globally sort and move the object elements of the first-level index (secondary index structure) according to the second-level index key, this global sorting and movement is a real-time and dynamic splitting process. The multiple sets after the split are logically presented as a serial chain structure connected end to end. At the same time, the chain heads of each set are ordered, and the chain head element serves as the third-level index. Therefore, based on the input embedding vector hash key-value pair, it can determine the index to a certain chain in the serial chain, namely the neighbor set. After accessing the neighbor index module, the data will be updated, which will trigger the neighbor index module to perform similarity calculations and neighbor aggregation operations, thereby obtaining the neighbor data corresponding to the embedding vector hash key-value pair corresponding to the original data. The generated neighbor calculation results will be deduplicated and updated to the corresponding neighbor set in the neighbor set index, thereby realizing the construction or update of the neighbor set index data. For basic index data construction (precise deduplication index construction): After the original data is processed, precise deduplication indicator key-value pair data is obtained, where the key is the identification ID and the value is the detailed data. After accessing the basic index module according to the indicator classification, since the second row key value ROW_KEY in the basic index data is used as the first-level index, its index object is a second-level index structure, the key is the identification ID, and the value is the corresponding detailed data. After searching in the sorting module, it can be determined whether the precise deduplication indicator key-value pair data is duplicate data. Since the sorting module globally sorts and moves the object elements of the primary index (secondary index structure) according to the key of the secondary index, this global sorting and movement is a real-time and dynamic splitting process. The multiple sets after the split are logically presented as a serial chain structure connected end to end. At the same time, the chain heads of each set are ordered, and the chain head elements are used as the third-level index. Therefore, it is possible to quickly determine whether there is data in the third-level index that is repeated with the key-value pair data of the precise deduplication index. If so, the key-value pair data of the precise deduplication index is deleted. If not, the corresponding increments are accumulated to the corresponding set. This process combines the construction of multi-level / classified indexes to continuously move the dynamic rules of global data, so that the incremental calculation of precise deduplication is completed synchronously when the data is stored. This process is incremental accumulation rather than full calculation each time.This approach enables fast and accurate deduplication when storing massive amounts of data. The above steps complete an iterative data synchronization and index building calculation. The pre-built embedding vector database continuously updates the index data in the pre-built embedding vector database by continuously processing the raw data in a preset periodic or real-time streaming or batch manner.

[0073] Step 150: Generate recommendation results based on the embedding vector hash key-value pair list.

[0074] After obtaining the embedding vector hash key-value pair list, obtain the target data list corresponding to the embedding vector hash key-value pair and recommend the target data list to the user.

[0075] like Figure 7 As shown, the embodiment of the present invention can realize neighbor recommendation and accurate deduplication of massive data through the above-mentioned method. For neighbor search, after the neighbor set index data is constructed, the neighbor search module can directly retrieve and obtain the neighbor set of a certain embedding vector from the neighbor set according to the above-mentioned neighbor search method, and finally output it as a recall / recommendation list. At the same time, in the process of accurate deduplication, the deduplication retrieval can directly query the dynamically updated global index deduplication or range index deduplication value through the index rule to directly obtain the current accurate deduplication of the index data in real time. At this time, there is no calculation process, and it is independent of the data volume, achieving O(1) complexity to obtain accurate deduplication values ​​of massive data.

[0076] The embodiment of the present invention maps the embedding vector into a local hash value to ensure local sensitivity, and establishes a deep conversion and combination with the local sensitive hashing technology. It is not limited to static text or one-dimensional space, and can further reduce the dimensionality of high-dimensional objects or things. Therefore, after obtaining the query request, the data to be matched in the query request is processed into a target embedding vector, and a target local sensitive hash value is generated according to the target embedding vector. The target local sensitive hash value is used to query in a pre-built embedding vector database to obtain a list of embedding vector hash key-value pairs matching the data to be matched, wherein the pre-built embedding vector database includes multiple pre-stored embedding vector hash key-value pairs, the target embedding vector hash key-value pair is any one of the multiple embedding vector hash key-value pairs, and the embedding vector hash key-value pair includes the local sensitive hash value of the embedding vector and the embedding vector. Finally, a recommendation result is generated according to the embedding vector hash key-value pair list, thereby effectively improving the efficiency of neighbor data recommendation.

[0077] Furthermore, by establishing index data and performing special splitting and sorting, when the data to be deduplicated is put into the warehouse, the construction of multi-level indexes allows the global data to be moved dynamically and regularly, so that the incremental calculation of accurate deduplication is completed synchronously when the data is put into the warehouse. This process is incremental accumulation rather than full calculation each time, making the index construction process efficient, and the accurate deduplication value can be queried in real time. There is no calculation process during the query, and it is completely unrelated to the amount of data, further improving the data processing efficiency.

[0078] Figure 8 FIG. 1 shows a schematic diagram of the structure of a data recommendation device provided by an embodiment of the present invention. Figure 8 As shown, the device 300 includes:

[0079] The acquisition module 310 is configured to acquire a query request, wherein the query request includes data to be matched.

[0080] The processing module 320 is configured to process the to-be-matched data into a target embedding vector.

[0081] The generating module 330 is configured to generate a target embedding vector hash key-value pair according to the target embedding vector.

[0082] A matching module 340 is configured to query a pre-built embedding vector database based on the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched; the pre-built embedding vector database includes a plurality of pre-stored embedding vector hash key-value pairs; the embedding vector hash key-value pairs include a local sensitive hash value of the embedding vector and the embedding vector itself.

[0083] The recommendation module 350 is configured to generate a recommendation result based on the embedding vector hash key-value pair list.

[0084] The working process of the data recommendation device 300 according to the embodiment of the present invention is substantially consistent with the specific implementation steps of the above-mentioned data recommendation method, and will not be described in detail here.

[0085] The embodiment of the present invention maps the embedding vector into a local hash value to ensure local sensitivity, and establishes a deep conversion and combination with the local sensitive hashing technology. It is not limited to static text or one-dimensional space, and can further reduce the dimensionality of high-dimensional objects or things. Therefore, after obtaining the query request, the data to be matched in the query request is processed into a target embedding vector, and a target local sensitive hash value is generated according to the target embedding vector. The target local sensitive hash value is used to query in a pre-built embedding vector database to obtain a list of embedding vector hash key-value pairs matching the data to be matched, wherein the pre-built embedding vector database includes multiple pre-stored embedding vector hash key-value pairs, the target embedding vector hash key-value pair is any one of the multiple embedding vector hash key-value pairs, and the embedding vector hash key-value pair includes the local sensitive hash value of the embedding vector and the embedding vector. Finally, a recommendation result is generated according to the embedding vector hash key-value pair list, thereby effectively improving the efficiency of neighbor data recommendation.

[0086] Furthermore, by establishing index data and performing special splitting and sorting, when the data to be deduplicated is put into the warehouse, the construction of multi-level indexes allows the global data to be moved dynamically and regularly, so that the incremental calculation of accurate deduplication is completed synchronously when the data is put into the warehouse. This process is incremental accumulation rather than full calculation each time, making the index construction process efficient, and the accurate deduplication value can be queried in real time. There is no calculation process during the query, and it is completely unrelated to the amount of data, further improving the data processing efficiency.

[0087] Figure 9The schematic diagram of the structure of the computing device provided by the embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0088] like Figure 9 As shown, the computing device may include: a processor 402 , a communications interface 404 , a memory 406 , and a communication bus 408 .

[0089] Processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other devices, such as clients or other server network elements. Processor 402 is used to execute program 410, which may specifically perform the steps described in the aforementioned embodiment of the data recommendation method.

[0090] Specifically, the program 410 may include program code including computer-executable instructions.

[0091] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a computing device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0092] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0093] The program 410 may be specifically invoked by the processor 402 to cause the computing device to perform the following operations:

[0094] Obtaining a query request, wherein the query request includes data to be matched;

[0095] Processing the data to be matched into a target embedding vector;

[0096] Generate a target embedding vector hash key-value pair according to the target embedding vector;

[0097] A list of embedding vector hash key-value pairs matching the target embedding vector hash key-value pair is obtained by querying a pre-built embedding vector database according to the target embedding vector hash key-value pair; the pre-built embedding vector database includes a plurality of pre-stored embedding vector hash key-value pairs; the embedding vector hash key-value pairs include a locality-sensitive hash value of the embedding vector and the embedding vector itself;

[0098] Generate recommendation results based on the embedding vector hash key-value pair list.

[0099] In an optional manner, the method for constructing the embedding vector database includes: acquiring user behavior data in streaming or batch manner; processing each acquired user behavior data into an embedding vector; generating an embedding vector hash key-value pair based on the embedding vector; and storing the generated embedding vector hash key-value pair in the embedding vector database.

[0100] In an optional manner, generating a target embedding vector hash key-value pair based on the target embedding vector includes: generating a hash tree of the same dimension as the embedding vector; the hash tree includes multiple random vectors; respectively calculating the angle between the embedding vector and each random vector in the hash tree; determining the local sensitive hash value based on the angle; and generating the target embedding vector hash key-value pair based on the target local sensitive hash value and the target embedding vector.

[0101] In an optional manner, the embedding vector database includes basic index data, neighbor set index data, and embedding vector index data; the method for constructing the embedding vector database includes: establishing neighbor set index data in the embedding vector database; the neighbor set index data includes multiple neighbor sets stored corresponding to keywords of embedding vector hash key-value pairs, and the neighbor sets include similar embedding vector hash key values; establishing basic index data in the embedding vector database; the basic index data includes multiple topic sets stored corresponding to topics, and the topic sets include multiple embedding vector hash key-value pairs with the same topics; establishing embedding vector index data in the embedding vector database; the embedding vector index data includes multiple sets of embedding vectors stored according to topics and user identifiers or content identifiers.

[0102] In an optional manner, the embedding vector database uses an embedding index data structure for data storage, and the embedding index data structure includes a row key column and a data storage column cluster; the neighbor set index data includes a first row key column and a first data storage column cluster, the first row key column includes multiple first row key information, the first data storage column cluster includes at least one neighbor set corresponding to the first row key information, and the neighbor set stores an embedding vector hash key-value pair; the first row key information is generated according to the keyword key of the embedding vector hash key-value pair; the basic index data includes a second row key column and a second data storage column cluster, the second row key column includes multiple second row key information, the second data storage column cluster includes the embedding vector hash key-value pair corresponding to the second row key information; the second row key information is generated according to the subject; the embedding vector index data includes a third row key column and a third data storage column cluster, the third row key column includes multiple third row key information, the third data storage column cluster includes the embedding vector corresponding to the third row key information, and the third row key information is generated according to the subject and user identifier or content identifier.

[0103] In an optional embodiment, the method further includes: obtaining user behavior data to be input; determining one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier of an embedding vector hash key-value pair of the user behavior data to be input; determining one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier of the embedding vector hash key-value pair of the user behavior data to be input, searching the index database for a matching target row key column, wherein the target row key column is any one or more of the first row key column, the second row key column, and the third row key column; determining whether the user behavior data to be input is duplicate data based on the target row key column and a corresponding target data storage column cluster; the target row key column stores one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier, and the target data storage column cluster stores an embedding vector corresponding to the target row key column; when the user behavior data to be input is duplicate data, deleting the user behavior data to be input; and when the user behavior data to be input is not duplicate data, adding the user behavior data to be input to the embedding vector database.

[0104] In an optional manner, after establishing the neighbor set index data in the embedding vector database, the method includes: constructing an index structure based on the neighbor set index data; the index structure includes a primary index, a primary index object, a secondary index, and a secondary index object; the primary index includes an index truncation of the keyword key of the embedding vector hash key-value pair, and the primary index object is a local sensitive hash value in the embedding vector hash key-value pair containing the index truncation; the secondary index is a local sensitive hash value in the embedding vector hash key-value pair, and the secondary index object is the embedding vector hash key-value pair.

[0105] In an optional manner, establishing basic index data in the embedding vector database includes: sorting all embedding vector hash key-value pairs according to the size of the local sensitive hash value; storing the sorted embedding vector hash key-value pairs in multiple topic sets according to a preset storage threshold order; the preset storage threshold is a threshold for the number of embedding vector hash key-value pairs stored in each topic set.

[0106] In an optional manner, a pre-built embedding vector database is queried according to the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched, including: indexing the target local sensitive hash value in the target embedding vector hash key-value pair in the neighbor set index data to determine a target neighbor set that matches the target local sensitive hash value; calculating the similarity between the target local sensitive hash value and the local sensitive hash value in each embedding vector hash key-value pair in the target neighbor set; and determining the embedding vector hash key-value pair list based on the similarity.

[0107] The embodiment of the present invention maps the embedding vector into a local hash value to ensure local sensitivity, and establishes a deep conversion and combination with the local sensitive hashing technology. It is not limited to static text or one-dimensional space, and can further reduce the dimensionality of high-dimensional objects or things. Therefore, after obtaining the query request, the data to be matched in the query request is processed into a target embedding vector, and a target local sensitive hash value is generated according to the target embedding vector. The target local sensitive hash value is used to query in a pre-built embedding vector database to obtain a list of embedding vector hash key-value pairs matching the data to be matched, wherein the pre-built embedding vector database includes multiple pre-stored embedding vector hash key-value pairs, the target embedding vector hash key-value pair is any one of the multiple embedding vector hash key-value pairs, and the embedding vector hash key-value pair includes the local sensitive hash value of the embedding vector and the embedding vector. Finally, a recommendation result is generated according to the embedding vector hash key-value pair list, thereby effectively improving the efficiency of neighbor data recommendation.

[0108] Furthermore, by establishing index data and performing special splitting and sorting, when the data to be deduplicated is put into the warehouse, the construction of multi-level indexes allows the global data to be moved dynamically and regularly, so that the incremental calculation of accurate deduplication is completed synchronously when the data is put into the warehouse. This process is incremental accumulation rather than full calculation each time, making the index construction process efficient, and the accurate deduplication value can be queried in real time. There is no calculation process during the query, and it is completely unrelated to the amount of data, further improving the data processing efficiency.

[0109] An embodiment of the present invention provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on a computing device, the computing device executes the data recommendation method in any of the above method embodiments.

[0110] The executable instructions may be used to cause the computing device to perform the following operations:

[0111] Obtaining a query request, wherein the query request includes data to be matched;

[0112] Processing the data to be matched into a target embedding vector;

[0113] Generate a target embedding vector hash key-value pair according to the target embedding vector;

[0114] A list of embedding vector hash key-value pairs matching the target embedding vector hash key-value pair is obtained by querying a pre-built embedding vector database according to the target embedding vector hash key-value pair; the pre-built embedding vector database includes a plurality of pre-stored embedding vector hash key-value pairs; the embedding vector hash key-value pairs include a locality-sensitive hash value of the embedding vector and the embedding vector itself;

[0115] Generate recommendation results based on the embedding vector hash key-value pair list.

[0116] In an optional manner, the method for constructing the embedding vector database includes: acquiring user behavior data in streaming or batch manner; processing each acquired user behavior data into an embedding vector; generating an embedding vector hash key-value pair based on the embedding vector; and storing the generated embedding vector hash key-value pair in the embedding vector database.

[0117] In an optional manner, generating a target embedding vector hash key-value pair based on the target embedding vector includes: generating a hash tree of the same dimension as the embedding vector; the hash tree includes multiple random vectors; respectively calculating the angle between the embedding vector and each random vector in the hash tree; determining the local sensitive hash value based on the angle; and generating the target embedding vector hash key-value pair based on the target local sensitive hash value and the target embedding vector.

[0118] In an optional manner, the embedding vector database includes basic index data, neighbor set index data, and embedding vector index data; the method for constructing the embedding vector database includes: establishing neighbor set index data in the embedding vector database; the neighbor set index data includes multiple neighbor sets stored corresponding to keywords of embedding vector hash key-value pairs, and the neighbor sets include similar embedding vector hash key values; establishing basic index data in the embedding vector database; the basic index data includes multiple topic sets stored corresponding to topics, and the topic sets include multiple embedding vector hash key-value pairs with the same topics; establishing embedding vector index data in the embedding vector database; the embedding vector index data includes multiple sets of embedding vectors stored according to topics and user identifiers or content identifiers.

[0119] In an optional manner, the embedding vector database uses an embedding index data structure for data storage, and the embedding index data structure includes a row key column and a data storage column cluster; the neighbor set index data includes a first row key column and a first data storage column cluster, the first row key column includes multiple first row key information, the first data storage column cluster includes at least one neighbor set corresponding to the first row key information, and the neighbor set stores an embedding vector hash key-value pair; the first row key information is generated according to the keyword key of the embedding vector hash key-value pair; the basic index data includes a second row key column and a second data storage column cluster, the second row key column includes multiple second row key information, the second data storage column cluster includes the embedding vector hash key-value pair corresponding to the second row key information; the second row key information is generated according to the subject; the embedding vector index data includes a third row key column and a third data storage column cluster, the third row key column includes multiple third row key information, the third data storage column cluster includes the embedding vector corresponding to the third row key information, and the third row key information is generated according to the subject and user identifier or content identifier.

[0120] In an optional embodiment, the method further includes: obtaining user behavior data to be input; determining one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier of an embedding vector hash key-value pair of the user behavior data to be input; determining one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier of the embedding vector hash key-value pair of the user behavior data to be input, searching the index database for a matching target row key column, wherein the target row key column is any one or more of the first row key column, the second row key column, and the third row key column; determining whether the user behavior data to be input is duplicate data based on the target row key column and a corresponding target data storage column cluster; the target row key column stores one or more of a keyword key, a topic, and a topic and a user identifier or a content identifier, and the target data storage column cluster stores an embedding vector corresponding to the target row key column; when the user behavior data to be input is duplicate data, deleting the user behavior data to be input; and when the user behavior data to be input is not duplicate data, adding the user behavior data to be input to the embedding vector database.

[0121] In an optional manner, after establishing the neighbor set index data in the embedding vector database, the method includes: constructing an index structure based on the neighbor set index data; the index structure includes a primary index, a primary index object, a secondary index, and a secondary index object; the primary index includes an index truncation of the keyword key of the embedding vector hash key-value pair, and the primary index object is a local sensitive hash value in the embedding vector hash key-value pair containing the index truncation; the secondary index is a local sensitive hash value in the embedding vector hash key-value pair, and the secondary index object is the embedding vector hash key-value pair.

[0122] In an optional manner, establishing basic index data in the embedding vector database includes: sorting all embedding vector hash key-value pairs according to the size of the local sensitive hash value; storing the sorted embedding vector hash key-value pairs in multiple topic sets according to a preset storage threshold order; the preset storage threshold is a threshold for the number of embedding vector hash key-value pairs stored in each topic set.

[0123] In an optional manner, a pre-built embedding vector database is queried according to the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched, including: indexing the target local sensitive hash value in the target embedding vector hash key-value pair in the neighbor set index data to determine a target neighbor set that matches the target local sensitive hash value; calculating the similarity between the target local sensitive hash value and the local sensitive hash value in each embedding vector hash key-value pair in the target neighbor set; and determining the embedding vector hash key-value pair list based on the similarity.

[0124] The embodiment of the present invention maps the embedding vector into a local hash value to ensure local sensitivity, and establishes a deep conversion and combination with the local sensitive hashing technology. It is not limited to static text or one-dimensional space, and can further reduce the dimensionality of high-dimensional objects or things. Therefore, after obtaining the query request, the data to be matched in the query request is processed into a target embedding vector, and a target local sensitive hash value is generated according to the target embedding vector. The target local sensitive hash value is used to query in a pre-built embedding vector database to obtain a list of embedding vector hash key-value pairs matching the data to be matched, wherein the pre-built embedding vector database includes multiple pre-stored embedding vector hash key-value pairs, the target embedding vector hash key-value pair is any one of the multiple embedding vector hash key-value pairs, and the embedding vector hash key-value pair includes the local sensitive hash value of the embedding vector and the embedding vector. Finally, a recommendation result is generated according to the embedding vector hash key-value pair list, thereby effectively improving the efficiency of neighbor data recommendation.

[0125] Furthermore, by establishing index data and performing special splitting and sorting, when the data to be deduplicated is put into the warehouse, the construction of multi-level indexes allows the global data to be moved dynamically and regularly, so that the incremental calculation of accurate deduplication is completed synchronously when the data is put into the warehouse. This process is incremental accumulation rather than full calculation each time, making the index construction process efficient, and the accurate deduplication value can be queried in real time. There is no calculation process during the query, and it is completely unrelated to the amount of data, further improving the data processing efficiency.

[0126] An embodiment of the present invention provides a data recommendation device for executing the above-mentioned data recommendation method.

[0127] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computing device to execute the data recommendation method in any of the above method embodiments.

[0128] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are run on a computer, the computer executes the data recommendation method in any of the above method embodiments.

[0129] The algorithm or demonstration provided herein are not inherently relevant to any particular computer, virtual system or other equipment. Various general-purpose systems may also be used together with the teachings based on this. According to the above description, it is apparent that the structure required for constructing this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the above description of specific languages ​​is for the purpose of disclosing the best mode of the present invention.

[0130] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0131] Similarly, it should be understood that in order to streamline the present invention and facilitate understanding of one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0132] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed so far can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0133] It should be noted that the above embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by a person skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

Claims

1. A data recommendation method, characterized in that: The method comprises: Obtaining a query request, wherein the query request includes data to be matched; Processing the data to be matched into a target embedding vector; Generate a target embedding vector hash key-value pair based on the target embedding vector; wherein, generate a hash tree including multiple random vectors of the same dimension as the target embedding vector; calculate the angle between the target embedding vector and each of the random vectors in the hash tree; determine a target local sensitive hash value based on the angle; generate the target embedding vector hash key-value pair based on the target local sensitive hash value and the target embedding vector; According to the target embedding vector hash key-value pair, a query is performed in a pre-built embedding vector database to obtain an embedding vector hash key-value pair list that matches the data to be matched; wherein, the target local sensitive hash value in the target embedding vector hash key-value pair is indexed in the neighbor set index data to determine the target neighbor set that matches the target local sensitive hash value; the similarity between the target local sensitive hash value and the local sensitive hash value in each embedding vector hash key-value pair in the target neighbor set is calculated; according to the similarity, the embedding vector hash key-value pair list is determined; the neighbor set index data includes the embedding vector hash key-value pairs according to the embedding vector hash key-value pair. The keyword key of the g vector hash key-value pair corresponds to a plurality of neighbor sets stored, wherein the neighbor sets include similar embedding vector hash key values; the pre-constructed embedding vector database includes a plurality of pre-stored embedding vector hash key-value pairs, wherein the embedding vector hash key-value pairs include a locality-sensitive hash value of the embedding vector and the embedding vector itself; the embedding vector database includes basic index data, neighbor set index data, and embedding vector index data; the embedding vector database construction method includes: establishing neighbor set index data in the embedding vector database; Generate recommendation results based on the embedding vector hash key-value pair list.

2. The method according to claim 1, characterized in that The method for constructing the embedding vector database includes: Obtain user behavior data in streaming or batch mode; For each acquired user behavior data, process the user behavior data into an embedding vector; Generate an embedding vector hash key-value pair according to the embedding vector; Store the generated embedding vector hash key-value pairs into the embedding vector database.

3. The method according to claim 2, characterized in that The method for constructing the embedding vector database further includes: Establishing basic index data in an embedding vector database; the basic index data includes multiple topic sets stored according to topic correspondence, and the topic sets include multiple embedding vector hash key-value pairs with the same topic; Embedding vector index data is established in an embedding vector database; the embedding vector index data includes a set of multiple embedding vectors stored according to the subject and user identifier or content identifier.

4. The method according to claim 3, characterized in that The embedding vector database uses an embedding index data structure for data storage, wherein the embedding index data structure includes a row key column and a data storage column cluster; the neighbor set index data includes a first row key column and a first data storage column cluster, wherein the first row key column includes a plurality of first row key information, and the first data storage column cluster includes at least one neighbor set corresponding to the first row key information, wherein the neighbor set stores an embedding vector hash key-value pair; The first row of key information is generated based on the keyword key of the embedding vector hash key-value pair; The basic index data includes a second row key column and a second data storage column cluster, the second row key column includes a plurality of second row key information, the second data storage column cluster includes an embedding vector hash key-value pair corresponding to the second row key information; the second row key information is generated according to the subject; The embedding vector index data includes a third row key column and a third data storage column cluster, the third row key column includes multiple third row key information, the third data storage column cluster includes an embedding vector corresponding to the third row key information, and the third row key information is generated according to the subject and user identifier or content identifier.

5. The method according to claim 4, characterized in that The method further comprises: Get the user behavior data to be passed in; Determine one or more of the keyword key, the subject, and the subject and user identifier or content identifier of the embedding vector hash key-value pair of the user behavior data to be input; Determine a target row key column for searching for a match between a keyword key, a topic, and one or more of a topic and a user identifier or a content identifier of an embedding vector hash key-value pair of the to-be-input user behavior data in the embedding vector database; the target row key column is any one or more of the first row key column, the second row key column, and the third row key column; Determining whether the user behavior data to be imported is duplicate data based on the target row key column and the corresponding target data storage column cluster; the target row key column stores one or more of a keyword, a topic, a topic and a user identifier, or a content identifier; and the target data storage column cluster stores an embedding vector hash key-value pair or an embedding vector corresponding to the target row key column; When the user behavior data to be input is duplicate data, the user behavior data to be input is deleted; when the user behavior data to be input is not duplicate data, the user behavior data to be input is added to the embedding vector database.

6. The method according to claim 3, characterized in that The step of establishing basic index data in the embedding vector database includes: Sort all embedding vector hash key-value pairs according to the size of the local sensitive hash value; The sorted embedding vector hash key-value pairs are stored in a plurality of topic sets in an order according to a preset storage threshold; the preset storage threshold is a threshold value for the number of embedding vector hash key-value pairs stored in each topic set.

7. The method according to claim 4, characterized in that After establishing the neighbor set index data in the embedding vector database, the following steps are included: Constructing an index structure based on the neighbor set index data; the index structure includes a primary index, a primary index object, a secondary index, and a secondary index object; The primary index includes an index truncation of a keyword key of the embedding vector hash key-value pair, and the primary index object is a locality-sensitive hash value in the embedding vector hash key-value pair including the index truncation; The secondary index is a local sensitive hash value in the embedding vector hash key-value pair, and the secondary index object is the embedding vector hash key-value pair.

8. A data recommendation device, characterized in that: The device comprises: An acquisition module, configured to acquire a query request, wherein the query request includes data to be matched; A processing module, configured to process the to-be-matched data into a target embedding vector; A generation module, configured to generate a target embedding vector hash key-value pair based on the target embedding vector; wherein, a hash tree including multiple random vectors of the same dimension as the target embedding vector is generated; an angle between the target embedding vector and each of the random vectors in the hash tree is calculated; a target local sensitive hash value is determined based on the angle; and a target embedding vector hash key-value pair is generated based on the target local sensitive hash value and the target embedding vector; A matching module is configured to query a pre-built embedding vector database according to the target embedding vector hash key-value pair to obtain a list of embedding vector hash key-value pairs that match the data to be matched; wherein the target local sensitive hash value in the target embedding vector hash key-value pair is indexed in the neighbor set index data to determine a target neighbor set that matches the target local sensitive hash value; calculate the similarity between the target local sensitive hash value and the local sensitive hash value in each embedding vector hash key-value pair in the target neighbor set; determine the embedding vector hash key-value pair list based on the similarity; the neighbor set index data includes the embedding vector hash key-value pairs according to the embedding vector hash key-value pair. The keyword key of the embedding vector hash key-value pair corresponds to multiple neighbor sets stored, and the neighbor sets include similar embedding vector hash key values; the pre-built embedding vector database includes multiple pre-stored embedding vector hash key-value pairs; the embedding vector hash key-value pairs include the local sensitive hash value of the embedding vector and the embedding vector itself; the embedding vector database includes basic index data, neighbor set index data, and embedding vector index data; the embedding vector database construction method includes: establishing neighbor set index data in the embedding vector database; The recommendation module is used to generate recommendation results based on the embedding vector hash key-value pair list.

9. A computing device, characterized in that include: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the data recommendation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one executable instruction, and when the executable instruction is executed on the computing device, the computing device executes the operation of the data recommendation method according to any one of claims 1 to 7.

Images