Correlation calculation method based on Bert and ordered regression
Through the combination of Bert and ordered regression, the shortcomings of multi-stage classification and subtle differences evaluation in Query and Doc correlation calculation are solved, high-quality text representation and precise sorting are achieved, and user experience and information acquisition efficiency are improved.
Patent Information
- Application Number
- CN202411215707.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-20
Smart Images
Figure CN119166810A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of Bert correlation calculation methods, and specifically is a correlation calculation method based on Bert and ordered regression. Background Art
[0002] Search is one of the core entrances of each app or website. Users use search to meet their information needs in different scenarios. The long-term goal of search is to continuously optimize the search experience and improve user search satisfaction. This requires us to understand user search intent, accurately measure the relevance between the query and the Docs recalled by the system, display relevant Docs as much as possible, and rank more relevant Docs higher. Therefore, the calculation of the relevance between the query and the Doc is an important and basic part of search;
[0003] At present, the main methods for calculating the semantic similarity between Query and Doc include word2vec and the twin tower model. This matching method mainly includes two steps: first, obtaining the semantic vector representation of the text; second, calculating the similarity;
[0004] Word2Vec is a shallow neural network model proposed by Google for training word vectors. Its core includes two structures: CBOW and Skip-gram. The CBOW model takes the words around the central word (one-hot encoded) as input, sums them through the hidden layer, and then calculates the generation probability of the central word through the softmax layer. During the training process, the probability of occurrence of words in the corpus is maximized, and the final weight matrix is the word vector. These word vectors can be used to calculate the similarity between sentences, providing a basis for natural language processing tasks;
[0005] The twin-tower model is a neural network architecture for calculating the semantic similarity of texts. It is characterized by containing two parallel networks, each tower independently processing an input sentence and outputting its own text vector. The two towers can share network weights to save computing resources. Each tower converts the input sentence into a high-dimensional vector representation through its internal network structure. Subsequently, these vectors undergo interactive operations such as crossover and matching to calculate the semantic similarity between the two sentences. DSSM is one of the representatives of this type of model, and its representation layer can be further refined into forms such as CNN-DSSM and LSTM-DSSM to meet different text representation requirements. The twin-tower model uses supervised training to optimize the model using labeled data, so that the semantic similarity between texts can be calculated more accurately;
[0006] Word2Vec is an unsupervised training method. You only need to input the corresponding corpus to train the model parameters. Then, when calculating the query-doc correlation, you only need to input the text to obtain the text vector representation, and use the vector representation to calculate the text similarity.
[0007] The twin tower model structure is a supervised training method. First, manually labeled data is needed to train the model. The training process is divided into two steps: first, input text to obtain text representation and calculate text relevance. Second, input the obtained relevance and the manually labeled relevance label into the loss function to calculate the model loss. Minimizing the loss can get the optimal model. For classification problems, the cross entropy loss function is generally used. Then, in the inference stage, the text to be calculated is input into the trained model to get the corresponding text relevance. Summary of the invention
[0008] The purpose of the present invention is to provide a correlation calculation method based on Bert and ordered regression to solve the problems raised in the above background technology.
[0009] In order to achieve the above object, the present invention provides the following technical solution: a correlation calculation method based on Bert and ordered regression, the method is:
[0010] S1: Divide the Query-Doc relevance into four levels and formulate the relevance level division criteria;
[0011] S2: Get data. Get the Query-Doc relevance level dataset, which contains 100,000 pairs of Query and Doc data. Each data pair has a manually labeled relevance level.
[0012] S3: Get related words. Use Autohome related tools to get the car series words, car model words and error correction words on the query side text. Get the car model words, car series words and high-click query words on the doc side, and add them to the training text. The final text format for the input model is [CLS]query[SEP]query side car series words[SEP]query side car model words[SEP]query error correction words[SEP]doc[SEP]doc side car series words[SEP]doc side car model words[SEP]doc high-click query words;
[0013] S4: Input related words, input the text in S3 into the Bert model, use [SEP] as the interval between query and doc and combine it with the ordered regression method.
[0014] Preferably, the Query in S1 is a query word input by a user in a search system, and the Doc is a document. In a search system, the document retrieved by the search system when a user inputs a Query is referred to as Doc.
[0015] Preferably, the four correlation levels are irrelevant, relatively irrelevant, relatively relevant, and very relevant.
[0016] Preferably, the training methods of the Bert model in S4 are Masked LM and Next Sentence Prediction, and the Masked LM and Next Sentence Prediction methods capture the representations at the word and sentence levels respectively.
[0017] Preferably, the correlation level classification standard in S1 is from high to low, and in the classification model, it is replaced by 0, 1, 2, and 3 respectively.
[0018] Preferably, semantic matching refers to a text match in which two texts do not overlap literally but are similar in semantics.
[0019] Preferably, the main features of the BERT are:
[0020] Bidirectionality: It can capture the contextual information of each word in the text at the same time;
[0021] Powerful pre-training: Learn deep representations of language by pre-training on large-scale text data;
[0022] Flexible fine-tuning: After pre-training, it can be fine-tuned for specific tasks and quickly adapt to different NLP tasks.
[0023] Preferably, the combination of BERT and the ordered regression model is mainly to use BERT as a feature extractor and input the extracted features into the ordered regression model to achieve prediction of the ordered classification task.
[0024] The beneficial effects of the present invention are as follows:
[0025] 1. The present invention obtains the representation of the input text by using the BERT model. The BERT model effectively improves the quality and accuracy of text representation compared with traditional methods through its advantages such as bidirectionality, context awareness, pre-training and fine-tuning, and context word embedding.
[0026] 2. The present invention reduces the classification cross-level problem by about 30% by introducing ordered regression into the model. In addition, the ordered regression method can output the similarity score at the same time in addition to the correlation level. The score can effectively distinguish the closeness of docs at the same level and sort the docs at the same level. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a schematic diagram of the process of the present invention;
[0028] Figure 2 Input representation graph for Bert. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] like Figure 1 As shown, the embodiment of the present invention provides 1. a correlation calculation method based on Bert and ordered regression, the method is:
[0031] S1: Divide the Query-Doc relevance into four levels and formulate the relevance level division criteria;
[0032] S2: Get data. Get the Query-Doc relevance level dataset, which contains 100,000 pairs of Query and Doc data. Each data pair has a manually labeled relevance level.
[0033] S3: Get related words. Use Autohome related tools to get the car series words, car model words and error correction words on the query side text. Get the car model words, car series words and high-click query words on the doc side, and add them to the training text. The final text format for the input model is [CLS]query[SEP]query side car series words[SEP]query side car model words[SEP]query error correction words[SEP]doc[SEP]doc side car series words[SEP]doc side car model words[SEP]doc high-click query words;
[0034] S4: Input related words and input the text in S3 into the Bert model (such as Figure 2 As shown), [SEP] is used as the interval between query and doc and combined with the ordered regression method.
[0035] By using the BERT model to obtain the representation of the input text, the BERT model has the advantages of bidirectionality, context awareness, pre-training and fine-tuning, and context word embedding. Compared with traditional methods, it effectively improves the quality and accuracy of text representation. By introducing ordered regression into the model, the classification cross-level problem is reduced by about 30%. In addition to outputting the relevance level, the ordered regression method can also output a similarity score, which can effectively distinguish the closeness of docs in the same level and sort the docs in the same level.
[0036] Among them, Query in S1 is the query word entered by the user in the search system, and Doc is document. In the search system, the document recalled by the search system when the user enters Query is referred to as Doc.
[0037] The deep combination of the ordered regression model and BERT not only significantly reduces the cross-level problem in the classification process, greatly improves the accuracy and reliability of classification, but also pioneered the introduction of similarity scores as output dimensions. This innovative measure not only consolidates the model's advantages in distinguishing different levels of relevance, but also further refines the difference evaluation between documents at the same level. The introduction of similarity scores provides a quantitative basis for the sorting of documents at the same level, making the sorting results more scientific and reasonable, meeting the refined needs for document sorting in practical applications, and demonstrating strong practical value and application potential.
[0038] Among them, there are four levels of correlation, namely irrelevant, relatively irrelevant, relatively relevant, and very relevant.
[0039] The core role of defining four relevance levels is not limited to clarifying the classification standards, but also to build a rigorous and detailed grading system for text processing and analysis. Through detailed level division, we can significantly improve the accuracy of classification and ensure that text information is accurately classified into its most appropriate semantic category. At the same time, this multi-level design flexibly meets the diverse needs of different application scenarios. Whether it is precise retrieval, content recommendation or sentiment analysis, the most matching classification standards can be found. Ultimately, these levels optimize the user experience, allowing users to find the most relevant content more quickly, improving the efficiency and satisfaction of information acquisition.
[0040] Among them, the training methods of the Bert model in S4 are Masked LM and Next Sentence Pred ict ion. The Masked LM and Next Sentence Pred ict ion methods capture the representations at the word and sentence levels respectively.
[0041] Masked Language Model (MLM) and Next Sentence Prediction (NSP) are like two wings in the BERT model, complementing each other and jointly building a solid representation foundation for the model. MLM deeply explores the dependencies and contextual information between words by randomly masking some input words and predicting their original identities, giving BERT excellent word-level representation capabilities. NSP focuses on sentence-level coherence and relational reasoning. By predicting whether two sentences appear naturally in succession, it enhances the model's understanding of sentence structure, logic, and semantics, thereby achieving powerful sentence-level representation. This multi-level representation system has enabled BERT to shine in the field of natural language processing and has promoted significant performance improvements and technological innovations in multiple tasks.
[0042] Among them, the correlation level classification standard in S1 is from high to low, and in the classification model, it is replaced by 0, 1, 2, and 3 respectively.
[0043] The classification standards of relevance levels and their digital alternatives play a pivotal role. These standards not only provide clear guidance for model training, ensuring that the model can learn the inherent laws of text relevance at different levels, but also greatly simplify the model evaluation process, making performance evaluation more intuitive and quantifiable. By optimizing the level division, the model can more accurately capture the subtle differences between texts, improve classification accuracy and robustness, and at the same time, the application of digital alternatives enhances the flexibility and scalability of the model, enabling it to easily adapt to a variety of application scenarios and user needs, ultimately bringing users a more accurate and efficient text processing experience.
[0044] Among them, semantic matching refers to the matching of two texts that do not overlap literally but are similar at the semantic level.
[0045] As one of the core technologies of natural language processing, semantic matching has a far-reaching and wide-ranging impact. In the field of information retrieval, semantic matching significantly improves the accuracy and relevance of retrieval results by accurately capturing the deep connection between queries and documents, allowing users to find the information they need faster. At the same time, it deepens the ability to understand natural language, enabling machines to be closer to human thinking, understand and respond to complex and diverse language needs. From diverse application scenarios to bridges for multilingual interaction, semantic matching has demonstrated its unique value and injected strong impetus into the comprehensive development of artificial intelligence technology.
[0046] Among them, the main features of BERT are:
[0047] Bidirectionality: It can capture the contextual information of each word in the text at the same time;
[0048] Powerful pre-training: Learn deep representations of language by pre-training on large-scale text data;
[0049] Flexible fine-tuning: After pre-training, it can be fine-tuned for specific tasks and quickly adapt to different NLP tasks.
[0050] Together, they build a powerful and flexible framework for text understanding. These features enable BERT to surpass traditional methods, analyze text content from a more comprehensive and in-depth perspective, and capture the subtle meanings and complex relationships between the lines. At the same time, BERT's flexibility and scalability also allow it to quickly adapt to the needs of various new tasks, and it can achieve excellent performance through simple fine-tuning, bringing unprecedented changes and breakthroughs to the field of natural language processing.
[0051] Among them, the combination of BERT and the ordered regression model is mainly to use BERT as a feature extractor and input the extracted features into the ordered regression model to achieve the prediction of ordered classification tasks.
[0052] It has brought new breakthroughs and progress in the field of natural language processing by efficient feature extraction, improving classification performance, adapting to complex task requirements, accelerating model training, and promoting the development of the NLP field.
[0053] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0054] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A correlation calculation method based on Bert and ordered regression, characterized in that: The method is: S1: Divide the Query-Doc relevance into four levels and formulate the relevance level division criteria; S2: Get data. Get the Query-Doc relevance level dataset, which contains 100,000 pairs of Query and Doc data. Each data pair has a manually labeled relevance level. S3: Get related words. Use Autohome related tools to get the car series words, car model words and error correction words on the query side text. Get the car model words, car series words and high-click query words on the doc side, and add them to the training text. The final text format for the input model is [CLS]query[SEP]query side car series words[SEP]query side car model words[SEP]query error correction words[SEP]doc[SEP]doc side car series words[SEP]doc side car model words[SEP]doc high-click query words; S4: Input related words, input the text in S3 into the Bert model, use [SEP] as the interval between query and doc and combine it with the ordered regression method.
2. The method for calculating correlation based on Bert and ordered regression according to claim 1, characterized in that: The Query in S1 is the query word entered by the user in the search system, and the Doc is a document. In the search system, the document retrieved by the search system when the user enters the Query is referred to as Doc.
3. The method for calculating correlation based on Bert and ordered regression according to claim 1, characterized in that: The four correlation levels are irrelevant, relatively irrelevant, relatively relevant, and very relevant.
4. The method for calculating correlation based on Bert and ordered regression according to claim 1, characterized in that: The training methods of the Bert model in S4 are Masked LM and Next Sentence Prediction, which capture the representations at the word and sentence levels respectively.
5. The method for calculating correlation based on Bert and ordered regression according to claim 1, characterized in that: The correlation level classification standard in S1 is from high to low, and in the classification model, it is replaced by 0, 1, 2, and 3 respectively.
6. The method for calculating correlation based on Bert and ordered regression according to claim 1, characterized in that: Semantic matching refers to the matching of two texts that do not overlap literally but are similar at the semantic level.
7. The method for calculating correlation based on Bert and ordered regression according to claim 1, characterized in that: The main features of BERT are: Bidirectionality: It can capture the contextual information of each word in the text at the same time; Powerful pre-training: Learn deep representations of language by pre-training on large-scale text data; Flexible fine-tuning: After pre-training, fine-tune for specific tasks to quickly adapt to different NLP tasks.
8. The method for calculating correlation based on Bert and ordered regression according to claim 1, characterized in that: The combination of BERT and the ordered regression model is mainly to use BERT as a feature extractor and input the extracted features into the ordered regression model to achieve prediction of ordered classification tasks.