Deep learning-based quick retrieval method for literatures in smart library

Abstract

The invention discloses an intelligent library literature quick retrieval method based on deep learning, and particularly relates to the technical field of natural language processing, and the method comprises the steps: converting text data in a data set into vectors in a high-dimensional space through a pre-trained Transform model, and converting the text data into dense vectors for feature extraction; a SimCSE model is adopted to generate positive sample pairs for codes of the same text content in different Dropout environments, unsupervised training is carried out on the model by utilizing the positive sample pairs, so that deep semantic representation of the text is learned, and finally, a model fusion strategy is carried out on the two models; then, an In-batch Networks method is used, other samples in the same batch are used as negative samples in the model training process, a dynamic learning rate adjustment mechanism based on performance feedback is adopted, the learning rate is automatically adjusted according to the expression of the model on a verification set, and the training process is optimized, so that the model is accelerated.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for rapid document retrieval in a smart library based on deep learning. Background Art

[0002] With the changing social environment and the continuous development of information technology, information resource sharing is becoming increasingly important in the digital age, and people's demand for electronic reading is increasing. Traditional libraries are gradually failing to keep up with people's demand for electronic reading. They are unable to accurately identify the needs of target customers and push relevant information, and they are unable to fully utilize library resources. The establishment of smart libraries can effectively combine resources, users, and services to realize a "resource + platform + service" intelligent reading promotion solution. This can not only improve the efficiency of reading promotion, but also ensure the rationality, timeliness, and adaptability of reading promotion, effectively avoiding the inefficiency caused by previous promotion based on subjective experience.

[0003] Traditional information retrieval technology can be divided into two categories: keyword-based retrieval and conceptual information retrieval.

[0004] Keyword-based search technology utilizes keyword matching to perform searches. When users are unable to accurately express the information they need using keywords, keyword-based search mechanisms cannot accurately retrieve the information they need. Keyword matching also fails to address the problems of multiple synonyms and multiple meanings of a single word, resulting in an inability to fully and accurately analyze user needs and missing a lot of useful information. Keyword matching is essentially a character-based search that doesn't consider semantics, resulting in a lot of irrelevant information being retrieved.

[0005] Conceptual information retrieval processes information semantically to form a concept base, then retrieves relevant information from the concept base to generate search results. While it overcomes some of the limitations of keyword retrieval, it lacks inter-concept relationships and therefore lacks robust reasoning. Furthermore, traditional retrieval methods are inadequate for searching similar documents within the same research area. Traditional retrieval technologies lack the necessary intelligence to meet users' growing search needs. Summary of the Invention

[0006] To this end, the present invention aims to address the shortcomings of the existing technology and design a method for rapid document retrieval in smart libraries based on deep learning. By using semantic retrieval (also known as vector-based retrieval), the retrieval system is no longer restricted to the literal meaning of the keywords searched by the user, but can accurately capture the true intention behind the user's search and use this to search, thereby more accurately returning the most appropriate results to the user. By utilizing advanced deep learning models and efficient vector search engines, this method can accurately capture the intention behind user queries, significantly improving the accuracy and response speed of the retrieval system, thereby solving the problems of low relevance of retrieval results and slow retrieval speed in the existing technology.

[0007] In order to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a method for rapid retrieval of smart library documents based on deep learning, which selects a document data set provided by the library, including titles, abstracts and keywords of documents such as papers, books and journals. The data set is preprocessed, and the steps include removing noise data, text segmentation, removing stop words, and standardization. These steps can clean the data, reduce the computational complexity of model training, and ensure the consistency of input data, and are characterized by: using a pre-trained Transformer model to convert the text data in the data set into vectors in a high-dimensional space to capture the deep semantic relationships in the text data, and converting the text data into dense vectors for feature extraction; for specific application scenarios, metadata (such as author, publication year, number of citations, etc.) are further extracted from the document as auxiliary features to input into the model to enhance the model's ability to understand the importance and relevance of the document.

[0008] The SimCSE model is used to generate positive sample pairs for encoding the same text content under different Dropout environments. The SimCSE model is then trained unsupervised using these positive samples to learn the deep semantic representation of the text. In the unsupervised learning process, no externally labeled data is required, significantly reducing the cost of manual labeling. Finally, the two models are combined using a model fusion strategy.

[0009] Then, the In-batch Negatives method is used to utilize non-identical text content in the same batch as negative samples during model training, thereby enhancing the model's ability to distinguish between positive and negative samples. A dynamic learning rate adjustment mechanism based on performance feedback is adopted (if the model's loss on the validation set does not improve significantly within a certain number of epochs, the learning rate will automatically decrease. This prevents the model from still using an excessively large learning rate in the later stages of training, preventing learning from stagnating at a local optimal solution. Assuming that the model's loss on the validation set does not improve within several epochs, the learning rate will be reduced by half (usually attenuated at a set ratio, such as 0.1 times) until the loss begins to recover). The learning rate is automatically adjusted based on the model's performance on the validation set to optimize the training process and accelerate the model.

[0010] In contrastive learning often uses inflection loss:

[0011]

[0012] where (h,h + ) is a positive sample pair, h - is a negative sample, r is the temperature parameter, sim() represents the similarity between two samples, l i represents the loss value of the i-th sample, h represents the i-th sample vector, Represents the negative sample vector of the jth sample, i refers to the current loss sample, and j refers to all samples in a batch;

[0013] The present invention introduces the local representation of the context window on the original basis and As a local representation of the sentence, the local contrast loss function is:

[0014]

[0015] in represents the local context window representation of the i-th sample, is the local context representation of its positive sample, is the local context window representation of the negative sample of the j-th sample, and the final loss for:

[0016]

[0017] Among them, λ global and λ local It controls the weights of global loss and local loss. By comparing the local context of each sentence, it generates more fine-grained negative samples, further improving the efficiency of contrastive learning, enhancing the model's ability to distinguish between positive and negative samples, and better learning the important information in the sentence.

[0018] Preferably, the preprocessing is to further implement data standardization processing after data cleaning, including unifying text format, standardizing date and reference format, to ensure the consistency of data input and the effectiveness of model processing.

[0019] Preferably, the feature extraction is specifically as follows: in addition to basic text vectorization, further extracting metadata such as author, publication year, citation count, etc. from the document. This information will be input into the model as auxiliary features to enhance the model's ability to understand the importance and relevance of the document; improving the Multi-Head Attention module in the Transformer model and using the (Dual-Normalization Multi-Head Attention) DNMHA module (such as Figure 1 As shown), the input is represented by a matrix H with a shape of (n×d), where n is the sequence length and d is the feature dimension; the matrix H is column-normalized and row-normalized, and the output is the matrix H v and H u , so that the features are balanced between different feature dimensions and different time steps, and then H v and H u They are passed to h different attention layers as input to generate the Q, K, and V matrices in the multi-head self-attention mechanism:

[0020]

[0021] Where W Q , W K , W V It is a learnable linear transformation matrix. After obtaining the matrix Q, K, V, the output is calculated:

[0022]

[0023] d k It is the number of columns of the Q and K matrices, that is, the vector dimension, which is used for scaling to avoid the influence of large values. Finally, the attention weights of multiple different layers are calculated in parallel. After the outputs of each attention head in the multi-head self-attention mechanism are spliced ​​together, they are passed into a Linear layer to obtain the final output Z of the multi-head self-attention mechanism, which contains the information of all heads. After linear transformation, it forms an overall representation of the input sequence. The context information is integrated into the attention calculation through the DNMHA module to enhance the context perception ability of the model.

[0024] Preferably, model fusion is specifically as follows: based on a single SimCSE and In-batchNegatives model, model fusion is introduced to combine the advantages of multiple models, such as integrating the characteristics of BERT and RoBERTa, to improve the robustness and accuracy of the model.

[0025] The present invention has the following advantages:

[0026] 1. This deep learning-based smart library document retrieval system significantly improves the accuracy and speed of document retrieval by integrating advanced Transformer pre-training models with a dual-mode training mechanism (unsupervised and supervised learning). The system utilizes vector search engines such as Milvus for efficient retrieval, achieving rapid recall, and continuously optimizes the search algorithm based on user feedback.

[0027] 2. By implementing load balancing and caching strategies, the system ensures stability and rapid response under high concurrency conditions. A user-friendly interface and personalized search results further optimize the user experience. This system not only improves the efficiency of library services but also significantly enhances service quality by accurately matching user query intent with document content. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the DNMHA module structure provided by the present invention;

[0029] Figure 2 This is the unsupervised training result diagram based on SimCSE for Comparative Example 1 provided by the present invention;

[0030] Figure 3 This is a graph of the supervised training results of Comparative Example 2 provided by the present invention based on In-batch Negatives. DETAILED DESCRIPTION

[0031] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0032] Comparative Example 1: Unsupervised training implementation method based on SimCSE:

[0033] In this comparative example, a large amount of unlabeled literature data, including titles, abstracts, and full texts, is first collected from the library's database. After the data is cleaned to remove invalid characters and inconsistent formats, the text is segmented and vectorized using a preprocessing script to prepare data for model training. The SimCSE model is used for unsupervised training. The model generates two semantically close but slightly different representations in the vector space for the same text input by applying different dropout configurations to construct positive sample pairs. In this way, the model learns how to extract meaningful semantic features from the text without the guidance of external labels. Finally, the trained model is deployed to the production environment and provided through a RESTful API. Users can send query requests through the API and receive corresponding literature recommendations. The unsupervised training results are shown below. Figure 2 shown.

[0034] Furthermore, to ensure system stability and responsiveness in high-concurrency environments, the present invention optimizes system performance through load balancing and caching strategies. Load balancing technology evenly distributes user requests across multiple servers to avoid single-point bottlenecks and server overload, thereby improving system availability and concurrent processing capabilities. Caching strategies utilize in-memory databases such as Redis to cache popular document search results and frequently used user queries, thereby reducing the number of database queries and improving system responsiveness. These optimization measures further ensure the stability and user experience of the smart library document retrieval system during large-scale access.

[0035] After unsupervised model training, the similarity between the two sets of data is calculated, which is 0.53 for the first set and 0.15 for the second set. It can be seen that the similarity between the first set is higher than that between the second set, and the similarity between the second set is extremely low.

[0036] Comparative Example 2: Supervised training implementation method based on In-batch Negatives:

[0037] This comparative example follows the data preprocessing steps of comparative example 1 to ensure data quality and consistency. The difference is that the model training uses the In-batch Negatives method for supervised learning. This method uses non-positive samples as negative samples within a batch to train the model, thereby strengthening the model's ability to distinguish between relevant and irrelevant documents. Specifically, the document vectors in each batch are calculated similarity with other document vectors. Except for the high similarity with their true matching documents, the rest are regarded as negative samples, thereby improving the model's ability to identify irrelevant documents. Through continuous iterative training, the model shows higher precision and recall when processing professional field document queries. After the model is deployed, it can quickly respond to user queries through specially optimized query processing paths and provide accurate document retrieval services. The supervised training results are as follows: Figure 3 。

[0038] After being trained by a supervised model, the similarity between these two sets of data is calculated. The first set is 0.76 and the second set is 0.14. The similarity of the first set is higher than that of the second set, and compared with unsupervised training, its test results are more accurate.

[0039] Example: A hybrid training method integrating SimCSE and In-batch Negatives:

[0040] This example combines the advantages of the previous two examples and implements a hybrid training strategy. This strategy first uses the SimCSE model for unsupervised pre-training to extract deep text semantic features, and then uses the In-batch Negatives technology for supervised training to optimize the model's response ability to specific queries. In addition, in order to further improve the retrieval performance, a model fusion technology is implemented to integrate the advantages of multiple pre-trained models. Taking the retrieval of Zhu Yunwen as an example, sorted by relevance, the recall results of Nanjing University of Aeronautics and Astronautics Library are in order as "Volume of Zhu Yunwen in the Great Ming Empire, Mian Chuan, The Emperor Jianwen Settled in Xiangtan, Those Things of the Ming Dynasty, The Great Ming National Master of the Jinyiwei, Six Cases of Ancient History, The Ming Chengzu, Anqiao Moon, Chatting about the Three Emperors of the Ming Dynasty, Hormones of History.3, The Romance and Loneliness of the Ancients". The results of this system are "Volume of Zhu Yunwen in the Great Ming Empire, The Emperor Jianwen Settled in Xiangtan, Those Things of the Ming Dynasty. Part 2, The Jinyiwei. The Great Ming National Master, The Ming Chengzu, Chatting about the Three Emperors of the Ming Dynasty, Anqiao Moon, Six Cases of Ancient History, Mian Chuan, Hormones of History.3, The Romance and Loneliness of the Ancients". The comparison results are shown in Table 1:

[0041] Table 1 Comparison of Retrieval Results for "Zhu Yunwen"

[0042]

[0043] Comparing the library retrieval results with the results of this system, it can be found that there are mainly three books with different rankings. Among them, the difference of "Mian Chuan" is the largest. It ranks second in the library and ninth in this system, dropping seven places. "Six Cases of Ancient History" drops two places, and "Chatting about the Three Emperors of the Ming Dynasty" rises three places.

[0044] By carefully reading the introductions, it can be found that among these four books, the relevance of "Chatting about the Three Emperors of the Ming Dynasty" should be the highest and it should be ranked at the top. "Anqiao Moon" mainly tells about the hidden front that was loyal to Zhu Yunwen. Zhu Yunwen is the main character in one of the cases in "Six Cases of Ancient History", while in "Mian Chuan", Zhu Yunwen's role is not much and the relevance should be the lowest. From this, it can be seen that the ranking of this system is reasonable and more accurate than the existing retrieval system in the library.

[0045] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A fast retrieval method for smart library documents based on deep learning, which selects a document dataset provided by the library and preprocesses the dataset. The method is characterized by: Use the pre-trained Transformer model to convert the text data in the dataset into vectors in a high-dimensional space to capture the deep semantic relationships in the text data and convert the text data into dense vectors for feature extraction; The SimCSE model is used to encode the same text content in different Dropout environments to generate positive sample pairs. The SimCSE model is trained unsupervised using the positive samples to learn the deep semantic representation of the text. Finally, the two models are fused. Then, the In-batch Negatives method is used to use non-identical text content in the same batch as negative samples during model training. A dynamic learning rate adjustment mechanism based on performance feedback is adopted to automatically adjust the learning rate according to the performance of the model on the validation set to optimize the training process and accelerate the model. InfoNCE loss is commonly used in contrastive learning: where (h,h + ) is a positive sample pair, h - is a negative sample, r is the temperature parameter, sim() represents the similarity between two samples, l i represents the loss value of the i-th sample, h i represents the i-th sample vector, Represents the negative sample vector of the jth sample, i refers to the current lost sample, and j refers to all samples in a batch; The present invention introduces the local representation of the context window on the original basis and As a local representation of the sentence, the local contrast loss function is: in represents the local context window representation of the i-th sample, is the local context representation of its positive sample, is the local context window representation of the negative sample of the jth sample, and the final loss for: Among them, λ global and λ local It is the weight that controls the global loss and local loss. By comparing the local context of each sentence, more fine-grained negative samples are generated.

2. The method for rapid document retrieval in a smart library based on deep learning according to claim 1 is characterized by: The preprocessing is to further implement data standardization after data cleaning, including unified text format, standardized date and reference format, to ensure the consistency of data input and the effectiveness of model processing.

3. The method for rapid document retrieval in a smart library based on deep learning according to claim 1 is characterized in that: The feature extraction is specifically as follows: the Multi-Head Attention module in the Transformer model is improved, and the DNMHA module is used. The input is represented by a matrix H with a shape of (n×d), where n is the sequence length and d is the feature dimension; the matrix H is column-normalized and row-normalized respectively, and the output is the matrix H v and H u , so that the features are balanced between different feature dimensions and different time steps, and then H v and H u They are passed to h different attention layers as input to generate the Q, K, and V matrices in the multi-head attention mechanism: Where W Q , W K , W V It is a learnable linear transformation matrix. After obtaining the matrices Q, K, and V, the output is calculated: d k It is the number of columns of the Q, K matrix, that is, the vector dimension, which is used for scaling to avoid the influence of large values. Finally, the attention weights of multiple different layers are calculated in parallel. After the outputs of each attention head in the multi-head self-attention mechanism are spliced ​​together, they are passed into a Linear layer to obtain the final output Z of the multi-head self-attention mechanism. After the context information of the DNMHA module is integrated into the attention calculation, it can improve the context perception ability of the model.

4. The method for rapid document retrieval in a smart library based on deep learning according to claim 1 is characterized in that: The specific model fusion is as follows: Based on the single SimCSE and In-batch Negatives models, model fusion is introduced to combine the advantages of multiple models, integrating the characteristics of BERT and RoBERTa to improve the robustness and accuracy of the model.

Images