Word similarity calculation method based on DeepSeek and CNN fusion

Abstract

According to the word similarity calculation method based on the DeepSeek and CNN fusion, the word semantic similarity is calculated, the semantic vector extracted by the DeepSeek and the semantic vector extracted by the CNN are effectively fused through a full connection layer, the cosine similarity calculation is carried out on the fused vectors, and the accuracy of word similarity calculation is improved. The method utilizes DeepSeek to capture semantic association capability of words in context, combines CNN to extract character-level local features, and fuses the two features through a full connection layer to realize efficient and accurate word semantic similarity calculation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a word similarity calculation method based on DeepSeek and CNN fusion. In particular, it relates to a word vector that fuses a large language model DeepSeek and a convolutional neural network model CNN, and is committed to improving the accuracy of word similarity calculation and realizing a method of word semantic similarity calculation combining global features and local features. BACKGROUND

[0002] Word semantic similarity calculation is a technology in natural language processing for quantifying the closeness of two words in meaning. Its goal is to output a value reflecting similarity (usually between 0 and 1) by analyzing the contextual association, concept hierarchy or distribution characteristics of words. Traditional methods rely on semantic relationship paths in artificially constructed semantic networks (such as WordNet) or statistical co-occurrence patterns (such as LSA), while modern methods more often use word vector models (such as Word2Vec, GloVe) to map words into dense vector spaces, capturing semantic associations through cosine similarity and other measures. With the popularity of pre-trained language models (such as BERT, RoBERTa), dynamic context representation can more accurately handle polysemous words and complex contexts. It is currently widely used in query expansion for information retrieval, word alignment for machine translation, and intent matching for question and answer systems, but also faces challenges such as polysemy, domain adaptability, and cross-language differences.

[0003] As a core basic task of natural language processing, word semantic similarity calculation has evolved from dictionary-based and statistical models (such as WordNet, LSA) to deep learning models (CNN, LSTM) and then to pre-trained language models (BERT, Albert) as artificial intelligence technology has developed. Although pre-trained models have significantly improved semantic understanding through context awareness, existing technologies still have shortcomings: 1. Single model representation is insufficient: a single model is difficult to capture both global semantics and local structural features. BERT can effectively capture rich semantic associations of words in context through a multi-layer Transformer architecture, but it lacks character-level local feature extraction. While CNN can accurately capture character combination patterns through multi-scale convolution kernels, it lacks semantic depth.

[0004] 2. Insufficient feature fusion: traditional methods use simple vector concatenation or weighted averaging, which cannot fully realize the effective combination of deep semantic features and local structural features. Although the context vectors generated by DeepSeek can reflect the associativity between words, the character features extracted by CNN can identify the morphological differences between words, but mechanical fusion often causes feature conflicts and loss of semantic information. SUMMARY

[0005] The purpose of the present application is to solve the problems in the prior art, and a word similarity calculation method based on DeepSeek and CNN fusion is proposed. The method uses the ability of DeepSeek to capture the semantic association of words in context, combines CNN to extract character-level local features, and realizes efficient and accurate word semantic similarity calculation by fusing the two features through a fully connected layer.

[0006] The present application is realized by the following technical solutions, and the present application proposes a word similarity calculation method based on DeepSeek and CNN fusion, and the specific steps of the method are: Step 1, double-channel vector extraction; the double-channel includes a DeepSeek channel and a CNN channel; Step 1.1, DeepSeek channel vector extraction; Step 1.1.1, input the word and context into the DeepSeek model; Step 1.1.2, extract the semantic vector of the word V d ; Step 1.2, CNN channel vector extraction; Step 1.2.1, convert the word into a vector as a basic vector by using a character-level embedding model Char-Level Embedding; Step 1.2.2, extract n-gram features through multi-scale convolution kernels; Step 1.2.3, obtain a structure vector after max-pooling V c ; Step 2, splice the vectors extracted by the double channels; Step 3, fuse the vectors through a fully connected layer to obtain the final vector representation of the word; Step 4, semantic similarity calculation, calculate the cosine similarity of the vector.

[0007] Further, the DeepSeek channel uses a DeepSeek-LLM-7B-Base model, and in the DeepSeek-LLM-7B-Base model, the relationship between different positions in the sequence is captured through a self-attention mechanism, (1) Wherein, Q, K, V are matrices obtained from the input sequence through linear transformation, is the dimension of K, which is used to prevent the gradient from disappearing due to the large dot product.

[0008] Further, in step 1.1.2, the words are converted into subwords Token by a tokenizer, each Token is mapped into a vector by an embedding matrix W embed mapping into a vector e i ; (2) where, e i is the embedding vector of a word, W embed is the embedding matrix, with shape V × d , V is the size of the vocabulary, d is the embedding dimension, i is the index of the word in the vocabulary; for each position, the e i applying RoPE generates a vector with fused position information ; (3) (4) where, x m is the word embedding vector of position m , is a preset angle parameter, controlling the rotation speed, is the rotation matrix, acting on each pair of dimensions of the word vector; the final input vector to the model is , which is directly used for the self-attention calculation of the Transformer.

[0009] Further, in step 1.2.2, n-gram features are extracted by multi-scale convolution kernels: (5) where, is the convolution kernel parameter, d emb is the word vector dimension, f () is the activation function, is the j th feature vector after convolution, k is the size of the convolution kernel.

[0010] Further, in step 1.2.3, in order to compress the features and retain important information, the convolution output is max-pooled, (6) where, is the global feature vector after pooling, representing the key information of the input sequence.

[0011] Further, in step 2, two vectors and are concatenated into a long vector , (7).

[0012] Further, in step 3, the target dimension is mapped by a fully connected layer, (8) where, f is the activation function, is the weight matrix, b is the bias, d out is the output dimension.

[0013] Further, in step 4, the cosine similarity of the vectors is calculated: (9) where, V 1 and V 2 are the final vector representations of word 1 and word 2, respectively.

[0014] Compared with the prior art, the present application has the following advantages: 1. Accuracy is improved, DeepSeek captures the context semantics of words, CNN captures the character structure of words, and the global semantic and local structural features are fused, which significantly improves the accuracy of word similarity calculation, which is better than a single model; 2. Fully integrate features, fully integrate the vectors obtained by DeepSeek and CNN through a fully connected layer, which can retain all information of the two vectors and automatically learn the interaction mode of the two vectors. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual proportions.

[0016] Figure 1 is the flow chart of the word similarity calculation method based on DeepSeek and CNN fusion; Figure 2 is the flow chart of the word similarity calculation method in the embodiment. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0018] To solve the problems in the prior art, the present application provides a word similarity calculation method based on DeepSeek and CNN fusion to calculate word semantic similarity. The semantic vector extracted by DeepSeek and the semantic vector extracted by CNN are effectively fused through a full connection layer, and the cosine similarity of the fused vector is calculated to improve the accuracy of word similarity calculation. The hybrid model structure is as shown in Figure 1 .

[0019] Specifically, referring to Figures 1-2 , the present application provides a word similarity calculation method based on DeepSeek and CNN fusion, and the specific steps of the method are as follows: Step 1, double-channel vector extraction; the double-channel includes a DeepSeek channel and a CNN channel; Step 1.1, DeepSeek channel vector extraction; Step 1.1.1, input the word and context into the DeepSeek model; Step 1.1.2, extract the semantic vector of the word V d ; Step 1.2, CNN channel vector extraction; Step 1.2.1, convert the word into a vector as a basic vector by using a character-level embedding model Char-Level Embedding; Step 1.2.2, extract n-gram features through a multi-scale convolution kernel; Step 1.2.3, obtain a structure vector after maximum pooling V c ; Step 2, splice the vectors extracted by the double channels; Step 3, fuse the vectors through a full connection layer to obtain the final vector representation of the word; Step 4, semantic similarity calculation, calculate the cosine similarity of the vector.

[0020] In step 1, the DeepSeek channel uses the DeepSeek-LLM-7B-Base model. DeepSeek-LLM-7B-Base is a large-scale open-source language model based on the Transformer Decoder architecture, boasting 7 billion parameters. It contains a high-quality corpus of 2 trillion multilingual (primarily Chinese-English) tokens, supports 16K context lengths, and utilizes techniques such as FlashAttention to optimize long sequence processing during training. The DeepSeek-LLM-7B-Base model captures the relationships between different positions in the sequence through a self-attention mechanism. (1) Where Q, K, and V are matrices obtained from the input sequence through linear transformation. It is the dimension of K, used to prevent the gradient from vanishing due to an excessively large dot product.

[0021] In step 1.1.2, words are converted into sub-word tokens by the tokenizer, and each token is embedded through an embedding matrix. W embed Mapped to vector e i ; (2) in, e i The embedding vector representing the word. W embed Represents an embedding matrix with shape . V × d , V It refers to the size of the vocabulary list. d It is the embedded dimension. i It is the index of the word in the vocabulary list; For each position e i Apply RoPE to generate vectors with fused location information ; (3) (4) in, x m Indicates position m Word embedding vectors, This indicates the preset angle parameter, which controls the rotation speed. This represents a rotation matrix that operates on each pair of dimensions of the word vectors; The final input vector to the model is It is directly used for the self-attention calculation of Transformer.

[0022] In step 1.2.2, n-gram features are extracted using multi-scale convolutional kernels: (5) in, These are the convolution kernel parameters. d emb It is the word vector dimension. f () is the activation function. It is the first convolutional step. j 1 eigenvector k It is the kernel size.

[0023] In step 1.2.3, in order to compress features and retain important information, max pooling is performed on the convolution output. (6) in, It is the global feature vector after pooling, which represents the key information of the input sequence.

[0024] In step 2, the two vectors and Concatenate into a long vector , (7).

[0025] In step 3, the target dimension is mapped through a fully connected layer. (8) in, f It is an activation function. This is the weight matrix. b For bias, d out It is the output dimension.

[0026] In step 4, the cosine similarity of the vectors is calculated: (9) in, V 1 and V 2 represents the final vector representation of word 1 and word 2, respectively.

[0027] Example This invention takes Chinese word similarity calculation as an example for implementation. Figure 2 The process is shown below, with the following steps: Step 1, Data Extraction: Extract Chinese words from the text that need to be used to calculate similarity; Step 2: Manually label 1 / 3 of the extracted Chinese words with similarity values ​​as the training set; Step 3, Model Training: The DeepSeek model selected is the deepseek-11m-7b-base pre-trained model. The CNN character embedding dimension is 64, the convolutional kernels are [3,4,5] with 128 kernels each, the learning rate is set to 3e-5, the batch size is set to 48, and the number of training epochs is set to 50. Step 4: Save the trained model; Step 5: Input the test set into the trained model and perform vector concatenation and fusion on the words; Step 6: Calculate the cosine similarity of the two fused vectors to obtain the similarity value.

[0028] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A word similarity calculation method based on the fusion of DeepSeek and CNN, characterized in that, The specific steps of the method are as follows: Step 1, dual-channel vector extraction; the dual channels include a DeepSeek channel and a CNN channel; Step 1.1, DeepSeek channel vector extraction; Step 1.1.1: Input the words and context into the DeepSeek model; Step 1.1.2: Extract the semantic vectors of words. V d ; Step 1.2, CNN channel vector extraction; Step 1.2.1: Use the character-level embedding model (Char-Level Embedding) to convert words into vectors as base vectors; Step 1.2.2: Extract n-gram features using multi-scale convolutional kernels; Step 1.2.3: Obtain the structure vector after max pooling. V c ; Step 2: Concatenate the vectors extracted from the two channels; Step 3: The fully connected layer fuses the vectors to obtain the final vector representation of the words; Step 4: Semantic similarity calculation, calculate the cosine similarity of the vectors.

2. The method according to claim 1, characterized in that: The DeepSeek channel uses the DeepSeek-LLM-7B-Base model, which employs a self-attention mechanism to capture the relationships between different positions in the sequence. （1） Where Q, K, and V are matrices obtained from the input sequence through linear transformation. It is the dimension of K, used to prevent the gradient from vanishing due to an excessively large dot product.

3. The method according to claim 1, characterized in that: In step 1.1.2, words are converted into sub-word tokens by the tokenizer, and each token is embedded through an embedding matrix. W embed Mapped to vector e i ; （2） in, e i The embedding vector representing the word. W embed Represents an embedding matrix with shape . V × d , V It refers to the size of the vocabulary list. d It is the embedded dimension. i It is the index of the word in the vocabulary list; For each position e i Apply RoPE to generate vectors with fused location information ; （3） （4） in, x m Indicates position m Word embedding vectors, This indicates the preset angle parameter, which controls the rotation speed. This represents a rotation matrix that operates on each pair of dimensions of the word vectors; The final input vector to the model is It is directly used for the self-attention calculation of Transformer.

4. The method according to claim 1, characterized in that: In step 1.2.2, n-gram features are extracted using multi-scale convolutional kernels: （5） in, These are the convolution kernel parameters. d emb It is the word vector dimension. f () is the activation function. It is the first convolutional step. j 1 eigenvector k It is the kernel size.

5. The method according to claim 1, characterized in that: In step 1.2.3, in order to compress features and retain important information, max pooling is performed on the convolution output. （6） in, It is the global feature vector after pooling, which represents the key information of the input sequence.

6. The method according to claim 1, characterized in that: In step 2, the two vectors and Concatenate into a long vector , （7）。 7. The method according to claim 1, characterized in that: In step 3, the target dimension is mapped through a fully connected layer. （8） in, f It is an activation function. This is the weight matrix. b For bias, d out It is the output dimension.

8. The method according to claim 1, characterized in that: In step 4, the cosine similarity of the vectors is calculated: （9） in, V 1 and V 2 represents the final vector representation of word 1 and word 2, respectively.

Images