Word semantic similarity calculation method based on xLSTM and Albert

Abstract

The invention provides a word semantic similarity calculation method based on xLSTM and Albert. The method comprises the following specific steps: step 1, constructing a hybrid model architecture; step 2, dynamic weight adjustment; 3, training and optimizing the model; and step 4, semantic similarity calculation. According to the method, the improved sequence modeling capability of xLSTM and deep semantic representation of Albert are fused, and a dynamic weight adjustment strategy is combined, so that the accuracy and efficiency of similarity calculation are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and particularly relates to a method for calculating word semantic similarity based on an extended long short-term memory network (xLSTM) and a lightweight bidirectional encoder representation from Transformers (Albert). Background Art

[0002] Calculating word semantic similarity is one of the core tasks in natural language processing and is widely applied in scenarios such as machine translation, information retrieval, and intelligent question answering. Traditional methods such as those based on term frequency-inverse document frequency (TF-IDF), latent semantic analysis (LSA), etc. rely on shallow features and are difficult to capture the context semantic information of words. With the development of deep learning, models such as Word2Vec and GloVe have improved semantic modeling capabilities through word vector representations, but still have limitations in static representations. In recent years, pre-trained language models such as BERT have significantly improved semantic understanding capabilities through bidirectional attention mechanisms, but their large number of parameters and high computational costs limit their practical applications.

[0003] The deficiencies of the prior art include:

[0004] 1. Limitations of static word vectors: Word vectors generated by models such as Word2Vec cannot dynamically adapt to different context contexts, resulting in limited accuracy in similarity calculation.

[0005] 2. High model complexity: Large models such as BERT require a large amount of computing resources and are difficult to deploy in resource-constrained scenarios.

[0006] 3. Insufficient handling of long-distance dependencies: Traditional LSTM has a problem of gradient disappearance when processing long sequences, affecting the coherence of semantic modeling.

[0007] To address the above problems, the present invention proposes a hybrid model combining xLSTM and Albert, enhancing the ability to capture long-distance dependencies in sequences through xLSTM and achieving efficient and accurate calculation of word semantic similarity by combining the lightweight design of Albert. Summary of the Invention

[0008] The purpose of the present invention is to solve the problems in the prior art and proposes a method for calculating word semantic similarity based on xLSTM and Albert. The method improves the accuracy and efficiency of similarity calculation by integrating the improved sequence modeling ability of xLSTM and the deep semantic representation of Albert, combined with a dynamic weight adjustment strategy.

[0009] The present invention is implemented through the following technical solutions. The present invention proposes a method for calculating word semantic similarity based on xLSTM and Albert. The specific steps of the method are as follows:

[0010] Step 1, construct a hybrid model architecture;

[0011] Step 1.1, input layer: used to receive the words to be calculated and their context texts;

[0012] Step 1.2, Albert encoding layer: used to extract the deep semantic features of words using the Albert model and generate dynamic context vectors;

[0013] Step 1.3, xLSTM enhancement layer: used to input the output of the Albert encoding layer into the xLSTM network, capture long-distance dependencies through an enhanced gating mechanism, and generate semantic vectors that fuse temporal features;

[0014] Step 1.4, similarity calculation layer: used to measure the similarity score between two word vectors based on cosine similarity;

[0015] Step 2, dynamic weight adjustment;

[0016] Step 3, model training and optimization;

[0017] Step 4, semantic similarity calculation.

[0018] Furthermore, in Step 2, define a loss function and jointly optimize the model using contrastive loss and cross-entropy loss;

[0019] L = λL contrastive +(1 - λ)L cross-entropy

[0020] where λ is a dynamic weight coefficient and is adaptively adjusted according to the training stage.

[0021] Furthermore, Step 3 is specifically as follows:

[0022] Step 3.1, data preprocessing, tokenize, denoise the input text, and construct a semantic similarity dataset containing positive and negative samples;

[0023] Step 3.2, parameter initialization, load the pre-trained Albert weights, and use He initialization for the xLSTM part;

[0024] Step 3.3, use the AdamW optimizer, combined with a learning rate decay strategy, to iteratively update the model parameters.

[0025] Furthermore, Step 4 is specifically as follows:

[0026] Step 4.1, input the word pairs to be compared and generate semantic vectors through the hybrid model;

[0027] Step 4.2: Calculate the similarity score between semantic vectors and normalize it to a probability value within the range of 0 - 1.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. Efficiency improvement: Through the parameter compression technology of Albert (such as parameter sharing) and the optimized structure of xLSTM, the model calculation efficiency is significantly improved compared with the traditional BERT.

[0030] 2. Accuracy improvement: By fusing context dynamic features and long sequence dependencies, the accuracy of word similarity calculation is significantly improved, which is better than a single model.

[0031] 3. Scalability: It supports multi - language scenarios and can quickly adapt to the needs of vertical fields through transfer learning. Description of the Drawings

[0032] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required to be used in the description of the specific embodiments or the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0033] Figure 1 It is a structural diagram of the hybrid model;

[0034] Figure 2 It is a flowchart of the dynamic weight adjustment strategy;

[0035] Figure 3 It is a flowchart of the method for calculating word semantic similarity based on xLSTM and Albert. Detailed Embodiments

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0037] Refer to Figures 1 - 3 , the present invention proposes a method for calculating word semantic similarity based on xLSTM and Albert. The specific steps of the method are as follows:

[0038] Step 1: Construct a hybrid model architecture; as Figure 1 shown;

[0039] Step 1.1, Input layer: Used to receive the words to be calculated and their context texts.

[0040] Step 1.2, Albert Encoding Layer: Used to extract the deep semantic features of words using the Albert model and generate dynamic context vectors;

[0041] Step 1.3, xLSTM Enhancement Layer: Used to input the output of the Albert Encoding Layer into the xLSTM network, capture long-distance dependencies through an enhanced gating mechanism, and generate semantic vectors that fuse temporal features;

[0042] Step 1.4, Similarity Calculation Layer: Used to measure the similarity score between two word vectors based on cosine similarity;

[0043] Step 2, Dynamic Weight Adjustment; as Figure 2 shown;

[0044] In Step 2, define the loss function and jointly optimize the model using Contrastive Loss and Cross-Entropy Loss;

[0045] L = λL contrastive +(1 - λ)L cross-entropy

[0046] where λ is the dynamic weight coefficient, which is adaptively adjusted according to the training stage.

[0047] Step 3, Model Training and Optimization;

[0048] The specific steps of Step 3 are as follows:

[0049] Step 3.1, Data Preprocessing: Tokenize, denoise the input text, and construct a semantic similarity dataset containing positive and negative samples; as shown in Table 1;

[0050] Table 1 is an example of a semantic similarity dataset for positive and negative samples

[0051]

[0052]

[0053] Step 3.2, Parameter Initialization: Load the pre-trained Albert weights, and use He initialization for the xLSTM part;

[0054] Step 3.3, Use the AdamW optimizer, combined with the learning rate decay strategy, to iteratively update the model parameters.

[0055] Step 4, Semantic Similarity Calculation.

[0056] The specific steps of Step 4 are as follows:

[0057] Step 4.1: Input the word pairs to be compared and generate semantic vectors through the hybrid model;

[0058] Step 4.2: Calculate the similarity score between the semantic vectors and normalize it to a probability value within the range of 0 - 1.

[0059] Embodiment

[0060] In this embodiment of the present invention, taking the calculation of Chinese word similarity as an example, the method of the present invention will be described in detail.

[0061] The method of the present invention proposes a method for calculating word semantic similarity based on xLSTM and Albert. As Figure 3 shown, the specific steps of the method are as follows:

[0062] Step 1: Data preparation, using a Chinese semantic similarity dataset;

[0063] Step 2: Construct a hybrid model architecture;

[0064] Input layer: Used to receive the words to be calculated and their context texts;

[0065] Albert encoding layer: Used to extract the deep semantic features of words using the Albert model and generate dynamic context vectors;

[0066] xLSTM enhancement layer: Used to input the output of the Albert encoding layer into the xLSTM network, capture long-distance dependencies through an enhanced gating mechanism, and generate semantic vectors that fuse temporal features;

[0067] Similarity calculation layer: Used to measure the similarity score between two word vectors based on cosine similarity;

[0068] Step 3: Dynamic weight adjustment;

[0069] Define the loss function and jointly optimize the model using contrastive loss and cross-entropy loss;

[0070] L = λL contrastive +(1 - λ)L cross-entropy

[0071] where λ is the dynamic weight coefficient, which is adaptively adjusted according to the training stage.

[0072] Step 4: Model training and optimization;

[0073] Model configuration: For the Albert model, the pre-trained weights of albert-base-chinese are selected. The number of hidden units in the xLSTM layer is 512. An improved parallel gating mechanism is adopted. The training parameters are set as follows: the initial learning rate is 2e-5, the batch size is 32, and the number of training epochs is 10.

[0074] Step 5: Calculate semantic similarity.

[0075] Step 6: Model evaluation. The performance of the model is measured using accuracy and F1 score.

[0076] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Although the present invention has been disclosed above with the preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to the above-disclosed technical content to obtain equivalent embodiments with equivalent changes within the scope of the technical solution of the present invention. However, as long as it does not depart from the technical solution content of the present invention and is based on the technical essence of the present invention, any simple modification, equivalent replacement, and improvement of the above embodiments still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for calculating word semantic similarity based on xLSTM and Albert, characterized in that The specific steps of the method are as follows: Step 1, construct a hybrid model architecture; Step 1.1, input layer: used to receive the word to be calculated and its context text; Step 1.2, Albert encoding layer: used to extract the deep semantic features of the word using the Albert model and generate a dynamic context vector; Step 1.3, xLSTM enhancement layer: used to input the output of the Albert encoding layer into the xLSTM network, capture long-distance dependencies through an enhanced gating mechanism, and generate a semantic vector that fuses temporal features; Step 1.4, similarity calculation layer: used to measure the similarity score of two word vectors based on cosine similarity; Step 2, dynamic weight adjustment; Step 3, model training and optimization; Step 4, semantic similarity calculation.

2. The method according to claim 1, wherein: In Step 2, define a loss function and jointly optimize the model using Contrastive Loss and Cross-Entropy Loss; L = λL contrastive +(1 - λ)L cross-entropy Among them, λ is the dynamic weight coefficient, which is adaptively adjusted according to the training stage.

3. The method according to claim 1, characterized in that: The specific content of Step 3 is as follows: Step 3.1, data preprocessing, perform word segmentation, denoising on the input text, and construct a semantic similarity dataset containing positive and negative samples; Step 3.2, parameter initialization, load the pre-trained Albert weights, and use He initialization for the xLSTM part; Step 3.3, use the AdamW optimizer, combined with a learning rate decay strategy, to iteratively update the model parameters.

4. The method according to claim 1, characterized in that: The specific content of Step 4 is as follows: Step 4.1, input the word pair to be compared and generate a semantic vector through the hybrid model; Step 4.2, calculate the similarity score between the semantic vectors and normalize it to a probability value within the range of 0-1.

Images