Book recommendation method and system based on vector database

Abstract

The invention provides a book recommendation method and system based on a vector database, and the method comprises the steps: inputting book feature vectors and behavior feature vectors on each dimension in the vector database into a BERT pre-training language model for training and learning, generating a corresponding BERT model, and deploying the BERT model in a background server for book recommendation. The method can recommend the books consistent with the interests and hobbies to the user, and can combine the book reading behaviors and hobbies of the user for comprehensive recommendation, thereby improving the recommendation accuracy. By adopting similarity recommendation, books with the same dimension can be recommended for the user on the interest dimension according to the interest label of the user, recommendation can be performed according to the sequence of vector similarity, and the most interested book of the user is preferentially recommended to the front-end user, so that the user generates relatively strong book recommendation experience, and the user experience is improved. The corresponding books are quickly recommended, and the book recommendation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent big data processing, and in particular to a book recommendation method, system and electronic device based on a vector database. Background Art

[0002] There are many ways to recommend books. Here are some common methods:

[0003] 1. Understand readers' needs: When recommending books, you must first understand readers' needs and interests, including their areas of interest, reading level, personal preferences, etc. Only by understanding readers' needs can you better recommend suitable books for them.

[0004] 2. Refer to professional book reviews: Professional book reviews are usually written by experts or senior readers. They objectively evaluate the advantages and disadvantages of books and can provide valuable reference for readers. When recommending books, you can refer to professional book reviews to understand the content, quality, value and other aspects of the book.

[0005] 3. Based on recommendation lists: Many bookstores, libraries, media and other institutions will publish book recommendation lists. These lists are usually selected by professionals or senior readers and have certain authority and reference value. When recommending books, you can recommend high-quality books to readers based on the recommendation lists.

[0006] 4. Combine personal experience: Personal experience is one of the important bases for recommending books. You can recommend books suitable for readers based on your own reading experience, professional background, knowledge reserves, etc. At the same time, you can also share your reading experience with readers to arouse their interest in reading.

[0007] 5. Use social media: Social media is one of the important channels for modern people to obtain information. You can use social media platforms such as WeChat and Weibo to recommend books to readers. You can show your reading experience to readers by publishing book reviews, reading notes, and experience, etc., to attract their interest in reading.

[0008] Although there are many methods for recommending books, we can choose the appropriate method according to the specific situation. However, no matter which method is used, it requires the recommender or the platform to combine the knowledge of a third party to recommend books to users, and it has subjective factors. The recommended books are not accurate enough, and most of the time users will directly reject the recommended books, which is inefficient. Summary of the invention

[0009] In order to solve the above technical problems existing in the prior art, the embodiments of the present invention provide a book recommendation method, system and electronic device based on a vector database. The technical solution is as follows:

[0010] On the one hand, a book recommendation method based on a vector database is provided, comprising the following steps:

[0011] S1. Prepare book big data P and user reading behavior big data L based on a distributed storage system, wherein the distributed storage system is configured as a database that can recognize a Bag of Words model or a Word2Vec model;

[0012] S2. Pre-deploy the Bag of Words model and Word2Vec model in the model library of the backend server;

[0013] S3, in a random polling manner of the preset model, randomly call the Bag of Words model or the word vector model Word2Vec in the model library to extract book vectors from the book big data P, and determine whether both models are involved. If both are involved, generate book feature vectors under different dimensions, and build a corresponding book vector database based on the book feature vectors; otherwise, reuse the two models to poll and extract vectors, wherein the book feature vectors include book category, spatial storage address, name, number and book attributes;

[0014] S4, performing vectorization processing (embedding) on ​​the user reading behavior big data L to generate behavior feature vectors corresponding to user reading behaviors in different dimensions;

[0015] S5, clustering based on vector similarity, clustering and binding the behavior feature vector and the book feature vector with similar vectors with maximum similarity;

[0016] S6. Input the book feature vector and the behavior feature vector on each dimension in the vector database into the BERT pre-trained language model for training and learning, generate a corresponding BERT model, and deploy the BERT model on a backend server for book recommendation.

[0017] Optionally, it also includes:

[0018] S7, the front-end user logs in to the back-end server and enters the book preference tag;

[0019] The backend server identifies the behavior and preference dimensions of the front-end user for books according to the book preference tags, and sends the behavior and preference dimensions to the BERT model;

[0020] The BERT model identifies the behavior feature vector that matches the behavior hobby dimension, and outputs all the book feature vectors that are bound to the behavior feature vector;

[0021] Calculating the similarity between the behavior feature vector and the corresponding book feature vector, sorting them according to the similarity value, and writing each of the book feature vectors into a preset personalized book recommendation list in sequence;

[0022] The personalized book recommendation list is sent to the front end for users to browse and select books.

[0023] Optionally, S1, based on a distributed storage system, prepare book big data P and user reading behavior big data L, wherein the distributed storage system is configured as a database that can recognize a bag of words model Bag of Words or a word vector model Word2Vec, including:

[0024] Preset an HDFS distributed file system on the background server, enter the configuration center of the HDFS distributed file system, and configure the data storage rules for storing the book big data P and the user reading behavior big data L, wherein the data storage rules are used to make the HDFS distributed file system automatically conform to the data format of the bag of words model Bagof Words or the word vector model Word2Vec;

[0025] The HDFS distributed file system sends a data retrieval request containing the data storage rules to the backend database, requesting to retrieve and collect the book big data P and the user reading behavior big data L that comply with the data storage rules, and perform data preprocessing respectively;

[0026] Import the pre-processed book big data P and the user reading behavior big data L into the HDFS distributed file system, and send the book big data P and the user reading behavior big data L to the management node --- master node;

[0027] The management node --- master node counts the storage capacity status of each storage node --- data node, and allocates the corresponding storage node --- data node to each storage node according to the data size of the book big data P and the user reading behavior big data L;

[0028] The book big data P and the user reading behavior big data L are respectively stored in the allocated storage node---data node.

[0029] Optionally, before step S2, the method further includes:

[0030] In the model library of the backend server, the Bag of Words model and the Word2Vec model are pre-deployed;

[0031] Establish a communication connection between the model library and the HDFS distributed file system.

[0032] Optionally, S2, in a random polling manner of a preset model, randomly calls the bag of words model Bag of Words or the word vector model Word2Vec in the model library to extract book vectors from the book big data P, and determines whether both models are involved. If both are involved, book feature vectors under different dimensions are generated, and a corresponding book vector database is constructed based on the book feature vectors; otherwise, the two models are reused to poll and extract vectors, wherein the book feature vectors include book categories, spatial storage addresses, names, numbers and book attributes, including:

[0033] The administrator initiates a request to the HDFS distributed file system to extract the book big data P;

[0034] The HDFS distributed file system responds to the request, and the management node --- master node extracts the book big data P from the corresponding storage node --- data node, and forwards it to the model library by the HDFS distributed file system;

[0035] The bag of words model or the word vector model Word2Vec is selected by a random round-robin model, and image vectors are extracted from the book big data P to obtain a book feature vector data set composed of book feature vectors on various dimensions in the book big data P, wherein the book feature vector includes a book category, a spatial storage address, a name, a number, and a book attribute;

[0036] According to the above-mentioned model random polling method, after multiple rounds of extraction, the book feature vectors on the same dimension are merged to obtain the book feature vector data set composed of the book feature vectors on each dimension; in the process of model random polling, the bag of words model Bag of Words and the word vector model Word2Vec are required to participate in the feature vector extraction process, and the background records the log of the model being called. After the polling is completed, it is determined whether there is a model that has not been called or a model whose number of calls accounts for less than 1 / 2:

[0037] If it exists, then generate book feature vectors in different dimensions; otherwise, re-polling to extract feature vectors or adding the model that has not been called or the model whose number of calls accounts for less than 1 / 2 to extract feature vectors;

[0038] The book feature vector data set is saved in the index data storage of the preset vector database, and the book vector database used for book recommendation is constructed.

[0039] Optionally, S3, performing vectorization processing (embedding) on ​​the user reading behavior big data L to generate behavior feature vectors corresponding to user reading behaviors in different dimensions, including:

[0040] The administrator initiates a request to the HDFS distributed file system to extract the user reading behavior big data L;

[0041] The HDFS distributed file system responds to the request, and the management node --- master node extracts the user reading behavior big data L from the corresponding storage node --- data node based on the data storage rule, and forwards it to the embedding model by the HDFS distributed file system;

[0042] In the embedding model, the user reading behavior big data L is processed by data vectorization to extract and generate the behavior feature vectors of the user reading behavior in different dimensions;

[0043] The behavior feature vector is independently saved in the storage data storage of the vector database.

[0044] Optionally, S4, clustering based on vector similarity, clustering and binding the behavior feature vectors and the book feature vectors with similar vectors with maximum similarity, including:

[0045] Using a clustering algorithm, the behavior feature vector and the book feature vector on the same dimension are clustered together, and the similarity between the behavior feature vector and the book feature vector on the same dimension is calculated using a vector cosine distance formula;

[0046] According to the similarity between the behavior feature vector and the book feature vector, find all book feature vectors B similar to the behavior feature vector A: B={b1,b2,...}, where b1,b2,... are the book feature vectors similar to the behavior feature vector A respectively, and bind the top n book feature vectors in the book feature vector B with the behavior feature vector A in similarity ranking to the book feature vector A to the behavior feature vector A;

[0047] The binding relationship between the behavior feature vector A and the corresponding book feature vector B in each dimension is saved in the vector database.

[0048] Optionally, S5, inputting the book feature vector and the behavior feature vector on each dimension in the vector database into a BERT pre-trained language model for training and learning, generating a corresponding BERT model, and deploying the BERT model on a backend server for book recommendation, including:

[0049] Deploy the BERT pre-trained language model on the backend server;

[0050] The book feature vector B on each dimension in the vector database and the behavior feature vector A bound thereto are used as training data features on different dimensions and are imported into the BERT pre-trained language model, and feature learning training is performed according to preset iterative training conditions, so that the BERT pre-trained language model learns to recognize the binding relationship between the book feature vector B on each dimension and the behavior feature vector A bound thereto;

[0051] After the training is completed, the BERT model is obtained; after parameter configuration and adjustment, the BERT model is deployed on the background server for book recommendation.

[0052] In another aspect, the present invention provides a system for implementing the book recommendation method based on the vector database, comprising:

[0053] The front end is used for users to input book preference tags to the back end server; and receives and displays the personalized book recommendation list fed back by the back end for users to browse and select books.

[0054] The backend identifies the behavior and hobby dimensions of the frontend user through the backend server, and sends the behavior and hobby dimensions to the BERT model;

[0055] as well as,

[0056] Identify the behavior feature vector through the BERT model deployed on the backend server, and output all the book feature vectors bound to the behavior feature vector;

[0057] as well as,

[0058] Calculating the similarity between the behavior feature vector and the corresponding book feature vector, sorting them according to the similarity value, and writing each of the book feature vectors into a preset personalized book recommendation list in sequence;

[0059] as well as,

[0060] Sending the personalized book recommendation list to the front end;

[0061] The front end is connected to the back end for communication.

[0062] In another aspect, the present invention further provides an electronic device, comprising:

[0063] processor;

[0064] a memory for storing processor-executable instructions;

[0065] Wherein, the processor is configured to implement the book recommendation method based on a vector database when executing the executable instructions.

[0066] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0067] In the above scheme, a corresponding vector database is constructed through book feature vectors, and the user reading behavior big data L is vectorized and embedded to generate behavioral feature vectors corresponding to user reading behaviors in different dimensions. The book feature vectors and the behavioral feature vectors in each dimension of the vector database are input into the BERT pre-trained language model for training and learning to generate a corresponding BERT model, and the BERT model is deployed on the backend server for book recommendation. In this way, the BERT model can recommend books that are consistent with the user's interests and hobbies based on the binding relationship between the behavioral feature vector and the book feature vector in the vector database, and can make comprehensive recommendations based on the user's reading behavior and hobbies, thereby improving the accuracy of recommendations. The use of intelligent big data processing and model recommendation can quickly recommend corresponding books and improve the efficiency of book recommendations.

[0068] By adopting similarity recommendation, we can recommend books with the same dimension to users based on their interest tags, and make recommendations based on the order of vector similarity, giving priority to recommending books that users are most interested in to front-end users, so that users can have a stronger book recommendation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 It is shown as a schematic diagram of the implementation process of the present invention;

[0071] Figure 2 Shown is a schematic diagram of the front-end and back-end application system of the present invention;

[0072] Figure 3 The figure shows a schematic diagram of the architecture of the HDFS distributed file system of the present invention;

[0073] Figure 4 Shown is a schematic diagram of the application of the model library of the present invention;

[0074] Figure 5 Shown is a schematic diagram of the structure of the vector database of the present invention;

[0075] Figure 6 It is a schematic diagram showing the application of the electronic device of the present invention. DETAILED DESCRIPTION

[0076] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0077] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0078] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0079] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0080] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0081] like Figure 1 As shown, on the one hand, an embodiment of the present invention provides a book recommendation method based on a vector database, comprising the following steps:

[0082] S1. Prepare book big data P and user reading behavior big data L based on a distributed storage system, wherein the distributed storage system is configured as a database that can recognize a Bag of Words model or a Word2Vec model;

[0083] S2. Pre-deploy the Bag of Words model and Word2Vec model in the model library of the backend server;

[0084] S3, in a random polling manner of the preset model, randomly call the Bag of Words model or the word vector model Word2Vec in the model library to extract book vectors from the book big data P, and determine whether both models are involved. If both are involved, generate book feature vectors under different dimensions, and build a corresponding book vector database based on the book feature vectors; otherwise, reuse the two models to poll and extract vectors, wherein the book feature vectors include book category, spatial storage address, name, number and book attributes;

[0085] S4, performing vectorization processing (embedding) on ​​the user reading behavior big data L to generate behavior feature vectors corresponding to user reading behaviors in different dimensions;

[0086] S5, clustering based on vector similarity, clustering and binding the behavior feature vector and the book feature vector with similar vectors with maximum similarity;

[0087] S6. Input the book feature vector and the behavior feature vector on each dimension in the vector database into the BERT pre-trained language model for training and learning, generate a corresponding BERT model, and deploy the BERT model on a backend server for book recommendation.

[0088] This solution needs to combine the behavioral feature vectors of user reading behavior and the book feature vectors to build a vector database, and learn the binding relationship between the behavioral feature vectors and the book feature vectors in the vector database through intelligent model training, so as to deploy the model and use it for book recommendations.

[0089] The corresponding vector database is constructed through the book feature vector, and the user reading behavior big data L is vectorized and processed - embedding, to generate behavioral feature vectors corresponding to the user reading behavior in different dimensions, and the book feature vectors and the behavioral feature vectors in each dimension of the vector database are input into the BERT pre-trained language model for training and learning to generate the corresponding BERT model, and the BERT model is deployed on the backend server for book recommendation. In this way, the BERT model can recommend books that are consistent with the user's interests and hobbies based on the binding relationship between the behavioral feature vector and the book feature vector in the vector database, and can make comprehensive recommendations based on the user's reading behavior and hobbies, thereby improving the accuracy of the recommendation. The use of intelligent big data processing and model recommendation can quickly recommend corresponding books and improve the efficiency of book recommendations.

[0090] As an optional embodiment of the present invention, optionally, it also includes:

[0091] S7, the front-end user logs in to the back-end server and enters the book preference tag;

[0092] The backend server identifies the behavior and preference dimensions of the front-end user for books according to the book preference tags, and sends the behavior and preference dimensions to the BERT model;

[0093] The BERT model identifies the behavior feature vector that matches the behavior hobby dimension, and outputs all the book feature vectors that are bound to the behavior feature vector;

[0094] Calculating the similarity between the behavior feature vector and the corresponding book feature vector, sorting them according to the similarity value, and writing each of the book feature vectors into a preset personalized book recommendation list in sequence;

[0095] The personalized book recommendation list is sent to the front end for users to browse and select books.

[0096] Combined with Figure 2 As shown, the book recommendation system of this scheme.

[0097] The recommendation system can provide an operation page on the front end, through which the user can input the user's book hobby tags to the back end where the back end server is located. For example, if the user is interested in military books, the corresponding military tags can be input. The back end will determine the current user's behavioral hobbies based on the tags input by the current user, and match the corresponding behavioral feature vectors through the hobby dimension.

[0098] Because in the early stage of extracting behavioral feature vectors, the user's book reading behavior is divided according to the behavioral dimension and the corresponding behavioral feature vector is extracted. Each behavioral feature vector corresponds to a dimension, and one dimension corresponds to a user's behavioral hobby. Therefore, the matching of the behavioral feature vector of the hobby dimension can be completed by the background. The subsequent BERT model can identify and output the book feature vector that matches and binds the current user's behavioral feature vector from the vector database based on the behavioral feature vector matched for the current user's hobby.

[0099] The BERT model can output several book feature vectors that match and bind the features of the current user behavior. Each book feature vector contains the corresponding book data and its storage address, book title, etc. Each book feature vector can be used to represent the basic information of the book, such as the category and name of the book, which can be viewed by users later, allowing front-end users to browse and determine whether the recommended books match their interests and hobbies.

[0100] Therefore, when the BERT model recommends, it can recommend several book feature vectors for the current user. In order to recommend the most matching books to the user first, the similarity between each book feature vector and the behavior feature vector is calculated according to the similarity calculation method. Sort by the size of the similarity, put the book feature vector with the greatest similarity in the first column, and the others in order. In this sorting order, write each book feature vector into the personalized book recommendation list preset in the recommendation system.

[0101] By adopting similarity recommendation, we can recommend books with the same dimension to users based on their interest tags, and make recommendations based on the order of vector similarity, giving priority to recommending books that users are most interested in to front-end users, so that users can have a stronger book recommendation experience.

[0102] The background will then send the personalized book recommendation list to the front end, which will be displayed on the front end operation page for the front end users to browse and select the corresponding books.

[0103] As an optional implementation scheme of the present invention, optionally, S1, based on a distributed storage system, prepare book big data P and user reading behavior big data L, wherein the distributed storage system is configured as a database that can recognize bag of words model Bagof Words or word vector model Word2Vec, including:

[0104] Preset an HDFS distributed file system on the background server, enter the configuration center of the HDFS distributed file system, and configure the data storage rules for storing the book big data P and the user reading behavior big data L, wherein the data storage rules are used to make the HDFS distributed file system automatically conform to the data format of the bag of words model Bagof Words or the word vector model Word2Vec;

[0105] The HDFS distributed file system sends a data retrieval request containing the data storage rules to the backend database, requesting to retrieve and collect the book big data P and the user reading behavior big data L that comply with the data storage rules, and perform data preprocessing respectively;

[0106] Import the pre-processed book big data P and the user reading behavior big data L into the HDFS distributed file system, and send the book big data P and the user reading behavior big data L to the management node --- master node;

[0107] The management node --- master node counts the storage capacity status of each storage node --- data node, and allocates the corresponding storage node --- data node to each of them according to the data size of the book big data P and the user reading behavior big data L;

[0108] The book big data P and the user reading behavior big data L are respectively stored in the allocated storage node---data node.

[0109] As attached Figure 3 The figure shows an architecture of a HDFS distributed file system.

[0110] In the HDFS distributed file system, HDFS includes a unique NameNode (management node, also called masternode), which provides metadata services within HDFS. It mainly includes executable operations: creating, deleting, moving or renaming files, etc. The architecture is similar to that of a traditional hierarchical file system.

[0111] DataNode provides storage blocks for HDES. DataNode is a storage node---data node.

[0112] For HDFS distributed file system, please refer to the description of the prior art, which will not be repeated in this embodiment.

[0113] This solution uses the HDFS distributed file system to store the book big data P and the user reading behavior big data L in a distributed manner. The distributed storage system can store the two types of big data, which can reduce the storage and operation pressure of the backend server. In the big data processing and calculation, the corresponding big data can be quickly retrieved from the corresponding storage node according to the scheduling instructions and used for calculation, thereby improving the calculation speed.

[0114] S2. Pre-deploy the Bag of Words model and the Word2Vec model in the model library of the background server.

[0115] In the model library of the backend server, the Bag of Words model and the Word2Vec model are pre-deployed.

[0116] And establish a communication connection between the model library and the HDFS distributed file system.

[0117] The model files pre-deployed in the model library can be prepared in advance by the administrator.

[0118] As an optional implementation scheme of the present invention, optionally, S3, in a random polling manner of a preset model, randomly calls the bag of words model Bag of Words or the word vector model Word2Vec in the model library to extract book vectors from the book big data P, and determines whether both models are involved. If both are involved, book feature vectors under different dimensions are generated, and a corresponding book vector database is constructed based on the book feature vectors; otherwise, the two models are reused to poll and extract vectors, wherein the book feature vectors include book categories, spatial storage addresses, names, numbers and book attributes, including:

[0119] The administrator initiates a request to the HDFS distributed file system to extract the book big data P;

[0120] The HDFS distributed file system responds to the request, and the management node --- master node extracts the book big data P from the corresponding storage node --- data node, and forwards it to the model library by the HDFS distributed file system;

[0121] The bag of words model or the word vector model Word2Vec is selected by a random round-robin model, and image vectors are extracted from the book big data P to obtain a book feature vector data set composed of book feature vectors on various dimensions in the book big data P, wherein the book feature vector includes a book category, a spatial storage address, a name, a number, and a book attribute;

[0122] According to the above-mentioned model random polling method, after multiple rounds of extraction, the book feature vectors on the same dimension are merged to obtain the book feature vector data set composed of the book feature vectors on each dimension; in the process of model random polling, the bag of words model Bag of Words and the word vector model Word2Vec are required to participate in the feature vector extraction process, and the background records the log of the model being called. After the polling is completed, it is determined whether there is a model that has not been called or a model whose number of calls accounts for less than 1 / 2:

[0123] If it exists, then generate book feature vectors in different dimensions; otherwise, re-polling to extract feature vectors or adding the model that has not been called or the model whose number of calls accounts for less than 1 / 2 to extract feature vectors;

[0124] The book feature vector data set is saved in the index data storage of the preset vector database, and the book vector database used for book recommendation is constructed.

[0125] like Figure 4As shown, this solution pre-deploys two models in the background model library: the bag-of-words model Bag ofWords and the word vector model Word2Vec, and uses the random extraction method of the two models to randomly use one of the models to extract and generate the corresponding book vector for the book big data P to generate the corresponding book feature vector.

[0126] By using the random polling method of multiple cycles of models, the model is randomly polled, and the image vector is extracted from the book big data P through the called model, so as to obtain the image vector features extracted by the random model. The method of extracting image vectors by randomly polling the model multiple times can avoid the singleness of image vector features extracted by a single model, and avoid the disadvantages of insufficient comprehensiveness of the vector or insufficient dimensionality.

[0127] In order to avoid a certain model not being used multiple times in the feature vector extraction stage, the model log identification rule is used: whether there is a model that has not been called or a model that has been called less than 1 / 2 times, the process of random model polling is monitored. If a certain model is not sufficiently involved in the feature vector identification process, it is determined that there is a certain error in the feature vector extraction. Therefore, it is necessary to re-poll the feature vector extraction or increase the number of feature vector extractions for the model that has not been called or the model that has been called less than 1 / 2 times. This ensures the stability of the extraction and the rationality of the model comparison extraction.

[0128] Therefore, a dual model is adopted and a random model polling method is used to extract and generate book feature vectors in each dimension to form a book feature vector data set. By calling and extracting the random model, the image feature vectors in each dimension can be extracted in a larger range, making the feature vectors in the vector data set more comprehensive and detailed, and avoiding missing the book feature vectors in each dimension.

[0129] The book feature vectors for different book types may be, for example, book feature vectors in the military dimension, geographical dimension, historical dimension, mathematical dimension, and so on.

[0130] For the construction method and structure of the vector database, you can refer to the structure of the existing vector database (see the attached Figure 5 The architecture of the vector database).

[0131] As an optional implementation scheme of the present invention, optionally, S4, vectorization processing-embedding is performed on the user reading behavior big data L to generate behavior feature vectors corresponding to user reading behaviors in different dimensions, including:

[0132] The administrator initiates a request to the HDFS distributed file system to extract the user reading behavior big data L;

[0133] The HDFS distributed file system responds to the request, and the management node --- master node extracts the user reading behavior big data L from the corresponding storage node --- data node based on the data storage rule, and forwards it to the embedding model by the HDFS distributed file system;

[0134] In the embedding model, the user reading behavior big data L is processed by data vectorization to extract and generate the behavior feature vectors of the user reading behavior in different dimensions;

[0135] The behavior feature vector is independently saved in the storage data storage of the vector database.

[0136] In this solution, the method of requesting the HDFS distributed file system to obtain the corresponding storage data can be understood in conjunction with the working principle of the HDFS distributed file system. The management node, the master node, mainly issues a scheduling instruction to the corresponding storage node, the data node, to retrieve the data.

[0137] As attached Figure 5 The figure shows a schematic diagram of the working architecture of a vector database.

[0138] Vector Database, also called vector database, is mainly used to store and process vector data. In mathematics, a vector is a quantity with size and direction, which can be represented by a line segment with an arrow. The direction of the arrow is the direction of the vector, and the length of the line segment represents the size of the vector. The distance or similarity between two vectors can be obtained by Hamming distance, Euclidean distance or cosine distance.

[0139] Unstructured data such as images, texts, and audio and video can be converted into vector data through some transformation or embedding learning and stored in the vector database, thereby realizing similarity search and retrieval of images, texts, and audio and video. This means that you can use the vector database to find the most similar or related data based on semantics or contextual meaning, rather than using the traditional method of querying the database Q based on exact matching or predefined criteria.

[0140] The main feature of vector databases is efficient storage and retrieval. Using indexing technology and vector retrieval algorithms can achieve fast response under high-dimensional big data. Vector databases are also a type of database. In addition to managing vector data, they also support the management of traditional structured data. In actual use, there are many scenarios where vector fields and structured fields are filtered and retrieved at the same time, which is also a challenge for vector databases.

[0141] This solution requires vectorization processing (embedding) of the user reading behavior big data L to generate behavior feature vectors corresponding to user reading behaviors in different dimensions. Therefore, the Word2Vec or Glove model can be used to extract word vectors.

[0142] Word2Vec is a classic word vector model that learns the distributed representation of words by training a neural network model. Word2Vec has two training methods, one is based on the Skip-gram model, and the other is based on the CBOW (Continuous Bag-of-Words) model. By training the model, we can get the vector representation of each word, which can be used to measure the semantic similarity between words or as feature input for other NLP tasks.

[0143] GloVe (Global Vectors for Word Representation) is a word vector model that uses global word co-occurrence statistics. Unlike Word2Vec, GloVe regards the vector representation of words as the ratio of co-occurrence probabilities between reconstructed words, and learns word vectors by minimizing the error of this ratio. GloVe performs well in many NLP tasks, and compared to Word2Vec, it has a faster training speed when processing large-scale corpora.

[0144] As an optional implementation scheme of the present invention, optionally, S5, clustering based on vector similarity, clustering and binding the behavior feature vector and the book feature vector with similar vectors with maximum similarity, includes:

[0145] Using a clustering algorithm, the behavior feature vector and the book feature vector on the same dimension are clustered together, and the similarity between the behavior feature vector and the book feature vector on the same dimension is calculated using a vector cosine distance formula;

[0146] According to the similarity between the behavior feature vector and the book feature vector, find all book feature vectors B similar to the behavior feature vector A: B={b1,b2,...}, where b1,b2,... are the book feature vectors similar to the behavior feature vector A respectively, and bind the top n book feature vectors in the book feature vector B with the behavior feature vector A in similarity ranking to the book feature vector A to the behavior feature vector A;

[0147] The binding relationship between the behavior feature vector A and the corresponding book feature vector B in each dimension is saved in the vector database.

[0148] With respect to the user's behavior feature vector, at least one book feature vector may be matched for the behavior feature vector.

[0149] The behavior feature vector A and the book feature vector B here are just an exemplary description, indicating that there are multiple book feature vectors B with vector similarity to the behavior feature vector A. If there are multiple book feature vectors b1, b2, ... similar to the behavior feature vector A, then b1, b2, ... are respectively bound to the behavior feature vector A, and a mapping relationship between the behavior feature vector A and b1, b2, ... is established, indicating that the books corresponding to b1, b2, ... all conform to the current behavior feature vector A. For example, multiple books conform to the behavior feature vector (behavior vector) of the current user, so as to provide recommended books for the front-end user association.

[0150] The book feature vector B is a book feature vector set consisting of several book feature vectors B on the same dimension.

[0151] Among them, in order to quickly find the books that the user likes, it is necessary to compare the similarity of each book feature vector in the book feature vector B with the behavior feature vector A, so as to find several book feature vectors closest to the behavior feature vector A (such as book feature vectors similar to the user's military hobbies). Therefore, the similarity traversal and comparison method is adopted to compare the similarity of the behavior feature vector A with each vector in the book feature vector B one by one, so as to find the book feature vector closest to the behavior feature vector A.

[0152] Here, we can take the book feature vectors with the top 25% similarity ranking (for example, if there are 100 book feature vectors in book feature vector B, we will take the top 25 with the closest similarity to behavior feature vector A as the feature data set for this training) to improve the recognition level of the model and accurately recommend books to users.

[0153] For example, for the behavior of users reading military books, the book feature vectors of multiple military books can be matched and bound for the dimension of the military behavior. Therefore, in the military dimension, the behavior feature vector of the user behavior who likes to read military books can be matched and bound to at least one book feature vector of military books. A distance algorithm or a similarity algorithm can be used to aggregate and bind all the book feature vectors similar to the behavior feature vectors on the same dimension, so that the binding relationship between each behavior feature vector on each dimension can be generated, and the behavior feature vector on each dimension will be bound to at least one book feature vector of the corresponding book.

[0154] When the subsequent model identifies and matches the behavior feature vector and outputs the corresponding book feature vector, it can output several matching book feature vectors for the current behavior vector according to the binding relationship, and further recommend the most similar book feature vectors in order based on the similarity algorithm, thereby improving the accuracy of book recommendations.

[0155] As an optional implementation scheme of the present invention, optionally, S6, inputting the book feature vector and the behavior feature vector on each dimension in the vector database into a BERT pre-trained language model for training and learning, generating a corresponding BERT model, and deploying the BERT model on a backend server for book recommendation, including:

[0156] Deploy the BERT pre-trained language model on the backend server;

[0157] The book feature vector B on each dimension in the vector database and the behavior feature vector A bound thereto are used as training data features on different dimensions and are imported into the BERT pre-trained language model, and feature learning training is performed according to preset iterative training conditions, so that the BERT pre-trained language model learns to recognize the binding relationship between the book feature vector B on each dimension and the behavior feature vector A bound thereto;

[0158] After the training is completed, the BERT model is obtained; after parameter configuration and adjustment, the BERT model is deployed on the background server for book recommendation.

[0159] Combined with Figure 2 and 4 As shown, this solution uses the BERT model to learn the behavioral feature vectors in each dimension in the vector database and the book feature vectors bound to them, so that the BERT model can learn and identify the book feature vectors corresponding to various behavioral feature vectors.

[0160] By learning to identify the binding relationship between the two vectors, the BERT model recommends corresponding books to front-end users. A personalized book recommendation list is preset in the recommendation system.

[0161] After the BERT model learns the vector features in the vector database, the generated BERT model is deployed on the backend server for book recommendations. After the front-end user enters the corresponding hobby tag, the backend identifies the behavior dimension and further matches it to obtain the behavior feature vector. The BERT model identifies it based on the behavior feature vector and outputs the corresponding book feature vector to recommend books to the user.

[0162] Therefore, this solution can recommend a list of books to users, provide better leg and foot results for users to choose, and improve leg and foot response speed.

[0163] Obviously, those skilled in the art should understand that the implementation of all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0164] On the other hand, an embodiment of the present invention provides a system for implementing the book recommendation method based on a vector database, comprising:

[0165] The front end is used for users to input book preference tags to the back end server; and receives and displays the personalized book recommendation list fed back by the back end for users to browse and select books.

[0166] The backend identifies the behavior and hobby dimensions of the frontend user through the backend server, and sends the behavior and hobby dimensions to the BERT model;

[0167] as well as,

[0168] Identify the behavior feature vector through the BERT model deployed on the backend server, and output all the book feature vectors bound to the behavior feature vector;

[0169] as well as,

[0170] Calculating the similarity between the behavior feature vector and the corresponding book feature vector, sorting them according to the similarity value, and writing each of the book feature vectors into a preset personalized book recommendation list in sequence;

[0171] as well as,

[0172] Sending the personalized book recommendation list to the front end;

[0173] The front end is connected to the back end for communication.

[0174] Please understand the above interaction between the front-end and back-end in conjunction with Example 1.

[0175] The modules or steps of the present invention described above can be implemented by a general-purpose computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0176] like Figure 6 As shown, the present invention also provides an electronic device, including:

[0177] processor;

[0178] a memory for storing processor-executable instructions;

[0179] Wherein, the processor is configured to implement the book recommendation method based on a vector database when executing the executable instructions.

[0180] The electronic device according to the embodiment of the present disclosure includes a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement any of the above-mentioned book recommendation methods based on a vector database when executing the executable instructions.

[0181] Here, it should be noted that the number of processors can be one or more. At the same time, the electronic device of the embodiment of the present disclosure may also include an input system and an output system. Among them, the processor, memory, input system and output system may be connected through a bus or in other ways, which are not specifically limited here.

[0182] As a computer-readable storage medium, the memory can be used to store software programs, computer executable programs and various modules, such as the program or module corresponding to the book recommendation method based on a vector database in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the electronic device by running the software programs or modules stored in the memory.

[0183] The input system can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output system can include display devices such as display screens.

[0184] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A book recommendation method based on vector database, characterized in that: The steps include: S1. Prepare book big data P and user reading behavior big data L based on a distributed storage system, wherein the distributed storage system is configured as a database that can recognize a Bag of Words model or a Word2Vec model; S2. Pre-deploy the Bag of Words model and Word2Vec model in the model library of the backend server; S3, in a random polling manner of the preset model, randomly call the Bag of Words model or the word vector model Word2Vec in the model library to extract book vectors from the book big data P, and determine whether both models are involved. If both are involved, generate book feature vectors under different dimensions, and build a corresponding book vector database based on the book feature vectors; otherwise, reuse the two models to poll and extract vectors, wherein the book feature vectors include book category, spatial storage address, name, number and book attributes; S4, performing vectorization processing (embedding) on ​​the user reading behavior big data L to generate behavior feature vectors corresponding to user reading behaviors in different dimensions; S5, clustering based on vector similarity, clustering and binding the behavior feature vector and the book feature vector with similar vectors with maximum similarity; S6. Input the book feature vector and the behavior feature vector on each dimension in the vector database into the BERT pre-trained language model for training and learning, generate a corresponding BERT model, and deploy the BERT model on a backend server for book recommendation.

2. The book recommendation method based on vector database according to claim 1, characterized in that: Also includes: S7, the front-end user logs in to the back-end server and enters the book preference tag; The backend server identifies the behavior and preference dimensions of the front-end user for books according to the book preference tags, and sends the behavior and preference dimensions to the BERT model; The BERT model identifies the behavior feature vector that matches the behavior hobby dimension, and outputs all the book feature vectors that are bound to the behavior feature vector; Calculating the similarity between the behavior feature vector and the corresponding book feature vector, sorting them according to the similarity value, and writing each of the book feature vectors into a preset personalized book recommendation list in sequence; The personalized book recommendation list is sent to the front end for users to browse and select books.

3. The book recommendation method based on vector database according to claim 1, characterized in that: S1. Based on a distributed storage system, prepare book big data P and user reading behavior big data L, wherein the distributed storage system is configured as a database that can recognize a bag of words model Bag of Words or a word vector model Word2Vec, including: Preset an HDFS distributed file system on the background server, enter the configuration center of the HDFS distributed file system, and configure the data storage rules for storing the book big data P and the user reading behavior big data L, wherein the data storage rules are used to make the HDFS distributed file system automatically conform to the data format of the bag of words model Bag of Words or the word vector model Word2Vec; The HDFS distributed file system sends a data retrieval request containing the data storage rules to the backend database, requesting to retrieve and collect the book big data P and the user reading behavior big data L that comply with the data storage rules, and perform data preprocessing respectively; Import the pre-processed book big data P and the user reading behavior big data L into the HDFS distributed file system, and send the book big data P and the user reading behavior big data L to the management node --- masternode; The management node --- master node counts the storage capacity status of each storage node --- data node, and allocates the corresponding storage node --- data node to each storage node according to the data size of the book big data P and the user reading behavior big data L; The book big data P and the user reading behavior big data L are respectively stored in the allocated storage node---data node.

4. The book recommendation method based on vector database according to claim 3 is characterized in that: Before step S2, the method further includes: Establish a communication connection between the model library and the HDFS distributed file system.

5. The book recommendation method based on vector database according to claim 4 is characterized in that: S3, in a random polling manner of a preset model, randomly call the Bag of Words model or the Word2Vec model in the model library to extract book vectors from the book big data P, and determine whether both models are involved. If both are involved, generate book feature vectors under different dimensions, and build a corresponding book vector database based on the book feature vectors; otherwise, reuse the two models to poll and extract vectors, including: The administrator initiates a request to the HDFS distributed file system to extract the book big data P; The HDFS distributed file system responds to the request, and the management node --- master node extracts the book big data P from the corresponding storage node --- data node, and forwards it to the model library by the HDFS distributed file system; The bag of words model or the word vector model Word2Vec is selected by a random round-robin model, and image vectors are extracted from the book big data P to obtain a book feature vector data set composed of book feature vectors on various dimensions in the book big data P, wherein the book feature vector includes a book category, a spatial storage address, a name, a number, and a book attribute; According to the above-mentioned model random polling method, after multiple rounds of extraction, the book feature vectors on the same dimension are merged to obtain the book feature vector data set composed of the book feature vectors on each dimension; in the process of model random polling, the bag of words model Bag of Words and the word vector model Word2Vec are required to participate in the feature vector extraction process, and the background records the log of the model being called. After the polling is completed, it is determined whether there is a model that has not been called or a model whose number of calls accounts for less than 1 / 2: If it exists, then generate book feature vectors in different dimensions; otherwise, re-polling to extract feature vectors or adding the model that has not been called or the model whose number of calls accounts for less than 1 / 2 to extract feature vectors; The book feature vector data set is saved in the index data storage of the preset vector database, and the book vector database used for book recommendation is constructed.

6. The book recommendation method based on vector database according to claim 5, characterized in that: S3, performing vectorization processing (embedding) on ​​the user reading behavior big data L to generate behavior feature vectors corresponding to user reading behaviors in different dimensions, including: The administrator initiates a request to the HDFS distributed file system to extract the user reading behavior big data L; The HDFS distributed file system responds to the request, and the management node --- master node extracts the user reading behavior big data L from the corresponding storage node --- data node based on the data storage rule, and forwards it to the embedding model by the HDFS distributed file system; In the embedding model, the user reading behavior big data L is processed by data vectorization to extract and generate the behavior feature vectors of the user reading behavior in different dimensions; The behavior feature vector is independently saved in the storage data storage of the vector database.

7. The book recommendation method based on vector database according to claim 6, characterized in that: S4, clustering based on vector similarity, clustering and binding the behavior feature vector and the book feature vector with similar vectors to each other with maximum similarity, including: Using a clustering algorithm, the behavior feature vector and the book feature vector on the same dimension are clustered together, and the similarity between the behavior feature vector and the book feature vector on the same dimension is calculated using a vector cosine distance formula; According to the similarity between the behavior feature vector and the book feature vector, find all book feature vectors B similar to the behavior feature vector A: B={b1,b2,...}, where b1,b2,... are the book feature vectors similar to the behavior feature vector A respectively, and bind the top n book feature vectors in the book feature vector B with the behavior feature vector A in similarity ranking to the book feature vector A to the behavior feature vector A; The binding relationship between the behavior feature vector A and the corresponding book feature vector B in each dimension is saved in the vector database.

8. The book recommendation method based on vector database according to claim 7, characterized in that: S5, inputting the book feature vectors and the behavior feature vectors on each dimension in the vector database into the BERT pre-trained language model for training and learning, generating a corresponding BERT model, and deploying the BERT model on a backend server for book recommendation, including: Deploy the BERT pre-trained language model on the backend server; The book feature vector B on each dimension in the vector database and the behavior feature vector A bound thereto are used as training data features on different dimensions and are imported into the BERT pre-trained language model, and feature learning training is performed according to preset iterative training conditions, so that the BERT pre-trained language model learns to recognize the binding relationship between the book feature vector B on each dimension and the behavior feature vector A bound thereto; After the training is completed, the BERT model is obtained; after parameter configuration and adjustment, the BERT model is deployed on the background server for book recommendation.

9. A system for implementing the book recommendation method based on vector database as described in any one of claims 1 to 8, characterized in that: include: The front end is used for users to input book preference tags to the back end server; and receives and displays the personalized book recommendation list fed back by the back end for users to browse and select books. The backend identifies the behavior and hobby dimensions of the frontend user through the backend server, and sends the behavior and hobby dimensions to the BERT model; as well as, Identify the behavior feature vector through the BERT model deployed on the backend server, and output all the book feature vectors bound to the behavior feature vector; as well as, Calculating the similarity between the behavior feature vector and the corresponding book feature vector, sorting them according to the similarity value, and writing each of the book feature vectors into a preset personalized book recommendation list in sequence; as well as, Sending the personalized book recommendation list to the front end; The front end is connected to the back end for communication.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement a book recommendation method based on a vector database as described in any one of claims 1 to 8 when executing the executable instructions.

Images