Patent recommendation method based on transformer encoder and regularization strategy

Abstract

The present application belongs to the technical field of patent recommendation based on specific calculation model, and more particularly relates to a patent recommendation method based on a Transformer encoder and a regularization strategy. The method comprises: establishing a certain number of patent data sets according to a plurality of search texts, and classifying according to search statements; performing data preprocessing on the search texts and the patent texts in the data sets; establishing a word and sentence double-processing model based on the Transformer encoder and the regularization strategy, and optimizing the model structure and output results using a regularization strategy based on Dropout and symmetric JS divergence; processing the word granularity level at the sentence granularity level; linearly weighting the sentence granularity level results and the word granularity level results, and outputting classification results. The present application solves the problems of inaccurate information matching caused by inconsistent lengths of patent texts and search texts in the prior art and low accuracy of traditional patent recommendation.

Description

Technical Field

[0001] This invention belongs to the field of patent recommendation technology based on specific computational models, and more specifically, relates to a patent recommendation method based on Transformer encoders and regularization strategies. Background Technology

[0002] With the rapid development of my country's science and technology service industry, intellectual property rights are receiving increasing attention. Patents, as a major category of intellectual property, are one of the most important carriers of various scientific and technological achievements, characterized by high information value density, wide coverage, and close connection with production and daily life. Therefore, patent recommendation services have become a significant demand across all sectors of society. Patent data collection and analysis not only help enterprises understand the current state of industry development but also provide decision-making support for their scientific and technological service research and development processes.

[0003] Chinese patent document CN115481324A discloses a recommendation method based on a Transformer encoder and positional encoding. By incorporating positional encoding, this method can make sequential recommendations based on all of a user's purchase history. Considering the diversity of user interests, a Transformer encoder is introduced. The encoder generates better context alignment vectors for FM and DNN. The encoder aligns feature embeddings with cues from other features. The embeddings generated by the encoder facilitate further feature interactions, thereby uncovering the correlation between features and providing users with more accurate and practical recommendation results.

[0004] In summary, the application of artificial intelligence in various fields continues to expand, leading to the emergence of many methods for patent recommendation using machine learning. Traditional content-based recommendation uses the BERT model to directly extract features from the search text and patent text (patent title and patent abstract). Due to the inconsistent lengths of the two, the extracted information is too redundant, resulting in inaccurate recommendation results. Furthermore, existing models suffer from low text matching accuracy after a single feature vector extraction. This paper aims to improve the model's accuracy. Summary of the Invention

[0005] The present invention aims to overcome at least one of the defects of the prior art and provide a patent recommendation method based on Transformer encoder and regularization strategy to solve the problems of low accuracy of content-based patent recommendation and information redundancy caused by the inconsistency between the length of demand text and patent information in the prior art. In this invention, the dataset is divided into three categories, "1", "2", and "3", representing different similarities, and a similarity classification method is used to complete the patent recommendation task. During data preprocessing, a method of segmenting patent text into sentences and filling in titles is used to address the problem of low model accuracy caused by inconsistent input lengths. The SBER model is used to process the patent text and search text, while the BERT model is used to process the patent titles and search text, obtaining sentence-level and word-level feature representations as input data. A word-sentence dual-processing model based on a Transformer encoder and regularization strategy is created to extract features from the input data at both sentence-level and word-level. Sentence-level feature extraction uses a Transformer encoder with a multi-head attention mechanism to capture the relationships and information between vector groups. Word-level feature extraction uses a Siamese network model with a Transformer encoder and a multi-head attention mechanism to extract word-level feature information. Finally, a linear weighting method is used to merge the results of the two sub-models. Furthermore, a regularization strategy based on Dropout and symmetric JS divergence is proposed to improve the model's generalization ability and accuracy.

[0006] The detailed technical solution of this invention is as follows:

[0007] To address the aforementioned technical problems, this invention provides a patent recommendation method based on a Transformer encoder and regularization strategy. This method solves the problems of low accuracy and information redundancy caused by inconsistencies in the length of demand text and patent information in existing content-based patent recommendations. The process includes: dataset preparation; data preprocessing; establishing a word-sentence dual-processing model based on a Transformer encoder and regularization strategy, optimizing the model structure and output using a regularization strategy based on Dropout and symmetric JS divergence; extracting and calculating patent similarity classification at the sentence level; extracting and calculating patent similarity classification at the word level; and performing a linear weighted operation on the two similarity classification results to output the patent similarity classification result.

[0008] A patent recommendation method based on Transformer encoders and regularization strategies is characterized by the following steps:

[0009] S1. Establish a certain number of patent datasets based on multiple search texts, and classify them according to the search statements;

[0010] S1 specifically includes:

[0011] S11. Divide the samples in the patent dataset into three categories: label "1", label "2", and label "3". Samples labeled "1" indicate that the patent and the search text are basically the same, i.e., the highest similarity. Samples labeled "2" indicate that the patent text and the search text are basically similar, i.e., the similarity is moderate. Samples labeled "3" indicate that the patent text and the search text are unrelated, i.e., the similarity is low.

[0012] S12. In the application, the content of tag "3" is excluded first, the content of tag "1" is displayed first, and then tag "2" is displayed. The content of tag "2" is used when the amount of data is insufficient or the search text is obscure.

[0013] S2. Perform data preprocessing on the search text and the patent text in the dataset to obtain the search text sentence vector, search text word vector, patent text sentence vector group and patent title word vector;

[0014] S2 specifically includes:

[0015] S21. Preprocess the search text by sentence segmentation: Process the input search text using the SBERT pre-trained model to obtain the sentence vector of the search text;

[0016] S22. Preprocess the patent text by segmenting it into sentences: Process the patent text by filling in the title after segmenting the patent abstract into sentences to obtain a group of sentence vectors for the patent text;

[0017] S22 specifically includes:

[0018] S221. Statistical analysis of the number of sentences in the patent abstract information in the patent dataset;

[0019] S222. Set a segmentation threshold and segment the summary into sentences. If the number of segments is insufficient to meet the segmentation threshold, use patent titles to pad the number of patent text sentence groups to meet the segmentation threshold. Finally, use the pre-trained SBERT model to convert the patent text sentence groups into patent text sentence vector groups.

[0020] If the number of segments exceeds the segmentation threshold, a similarity selection is performed, and the sentence least similar to the patent title is selected and converted into a patent text sentence vector group.

[0021] Specifically, the step of selecting the sentence least similar to the patent title includes: using the SBERT model to convert the segmented sentences and patent titles into vectors, calculating the cosine similarity between the sentence vector of each summary and the sentence vector of the patent title, and sorting them in reverse order according to the cosine similarity to obtain the patent text sentence group that meets the segmentation threshold.

[0022] S23. Perform word segmentation on the patent title and search text of the patent text: Obtain the patent title of the search text and the patent title of the patent text, and perform preprocessing on them respectively. The preprocessing includes jieba word segmentation, stop word removal, and punctuation mark removal. After processing, use the pre-trained BERT model to perform vector processing on the word segmented text representation to obtain the patent title word vector and the search text word vector respectively.

[0023] S3. Establish a word and sentence dual processing model based on Transformer encoder and regularization strategy;

[0024] The word and sentence dual processing model based on Transformer encoder and regularization strategy includes a sentence-level feature vector extraction module, a sentence-level classification module, a word-level feature vector extraction module, and a word-level classification module.

[0025] The specific model structure is as follows:

[0026] The sentence-level feature vector extraction module includes a sentence-level feature extractor and a short text feature extractor; the sentence-level feature extractor includes a Transformer encoder and a multi-head attention mechanism to extract patent text vector group information; the short text feature extractor includes a fully connected layer;

[0027] The word-level feature vector extraction module includes two twin word-level feature extractors with shared parameters. The word-level feature extractor includes a Transformer encoder and a multi-head attention mechanism.

[0028] The similarity classification process in the sentence-level classification module uses fully connected layers and activation functions to calculate the patent similarity classification results.

[0029] The similarity classification process in the word-granularity level classification module uses a multi-head attention mechanism, a fully connected layer, and an activation function to calculate the patent similarity classification results.

[0030] Furthermore, a regularization strategy based on Dropout and symmetric JS divergence is used to optimize the model structure and output results;

[0031] This paper employs a regularization strategy based on Dropout and symmetric JS divergence. By minimizing the bidirectional JS divergence between two distributions, it ensures that the two distributions output by the two sub-models of the same data remain consistent. Specifically, this includes:

[0032] The training dataset is as follows: Where n is the number of training samples, (x i y i () represents a data pair;

[0033] Input data x i Inputting the data into a word-sentence dual-processing model based on a Transformer encoder and regularization strategy twice yields two different results for data x. i The output distributions are respectively and

[0034] The regularization strategy based on Dropout and symmetric JS divergence is implemented by adding the symmetric JS divergence of the same sample with two different output distributions to the cross-entropy loss function.

[0035] The KL divergence representation of the output distributions of the two sub-models: D KL (P1||P2), where P1 and P2 refer to the two output distributions obtained after inputting the model twice;

[0036] The formula for calculating the JS divergence of the model is as follows:

[0037]

[0038] Cross-entropy loss function for the two sub-models:

[0039]

[0040] The loss function of the final model is:

[0041]

[0042] Among them, α1 and α2 are used for control. The weighting coefficients.

[0043] S4. Input the search text sentence vector and the patent text sentence vector group into the sentence-granularity level feature vector extraction module and the sentence-granularity level classification module to obtain the sentence-granularity level patent similarity classification;

[0044] The processing flow of the sentence-level feature vector extraction module is as follows:

[0045] The search text sentence vectors are processed by a short text feature extractor to extract feature information, and the patent text sentence vector groups are processed by a sentence-granular feature extractor to extract feature information. The feature information extracted by the short text feature extractor and the feature information extracted by the sentence-granular feature extractor from the patent text sentence vector groups are concatenated, and finally, the sentence-granular level classification module performs similarity classification to obtain the sentence-granular level patent similarity classification.

[0046] Specifically as follows, such as Figure 2 As shown:

[0047] Search text sentence vector S emb The processing procedure is as follows:

[0048] First, use the Linear function to apply S emb Perform a linear transformation to obtain S emb_L Next, regarding S emb_L The activated search text sentence vector S is obtained using the nonlinear transformation activation function GELU. emb_L_G .

[0049] Patent text sentence vector group P emb The processing procedure is as follows:

[0050] First, the patent text sentence vector group P emb After the location encoding part, a location encoding vector P with the same dimension as the batch of patent text vector groups is generated. pos , will P pos Directly superimposed with Pemb, we obtain a vector group of patent text sentences with location information: P emb_P =P emb +P pos .

[0051] Next, P emb_P Entering the multi-head attention mechanism layer built into the Transformer encoder, the definition of the multi-head attention mechanism is as follows: QK T Represents the attention matrix, where ... The operation aims to transform the attention matrix into a standard normal distribution. The number of self-attention heads, i, is set to 4. The outputs O1, O2, O3, and O4 of each self-attention mechanism are concatenated, and a linear transformation is performed to obtain the output P of the multi-head self-attention mechanism. emb_P_MH ;

[0052] P emb_P_MH Entering the summation and normalization layer of the Transformer encoder, residual connection operations are performed; the purpose of layer normalization is to normalize the hidden layers in the neural network to a standard normal distribution, thereby accelerating convergence;

[0053] Then, the data undergoes linear mapping and activation through the feedforward neural network layer of the Transformer encoder, before entering the encoder's summation and normalization layer. This layer operates in the same manner as described above, ultimately leading to the encoder obtaining the sentence vector group P. emb_T .

[0054] The batch normalization layer processes the sentence vector group P after extracting feature vectors. emb_T Normalization is performed to obtain P emb_T_B Batch normalization is used to transform the sentence vector distribution into a smooth, isotropic standard Gaussian distribution to solve the problem of uneven and non-smooth distribution of sentence vector groups leading to unsatisfactory recommendation results.

[0055] The sentence vector group P is processed again using a multi-head attention mechanism. emb_T_B To extract deeper information, the feature extraction process is consistent with the multi-head attention mechanism in the encoder described above, resulting in P. emb_T_B_A .

[0056] Finally, P emb_T_B_A After the final batch normalization layer, Pe is obtained. mb_T_B_A_B .

[0057] For the patent text sentence group P emb_T_B_A_B An averaging operation is performed to average the six sentence vectors in the patent sentence vector group to obtain an average sentence vector P representing the overall characteristics. emb_ave .

[0058] The sentence vector S of the search text emb_L_G And the average sentence vector P of the patent text emb_ave The concatenation function is used to concatenate two vectors to obtain S. Con Next, S Con The variable undergoes a linear transformation using the Linear function, followed by activation using the GELU function. Then, the variable is mapped to the range of 0 and 1 using the sigmoid activation function, resulting in output_1.

[0059] S5. Input the word vectors of the search text and the word vectors of the patent title into the word-granularity level feature vector extraction module and the word-granularity level classification module to obtain the word-granularity level patent similarity classification;

[0060] The processing flow of the word-level feature vector extraction module is as follows:

[0061] The search text word vector and the patent title word vector are respectively input into the word-granular feature extractor. The feature information extracted by the Transformer encoder of the word-granular feature extractor is concatenated into a first concatenated vector. Then, the feature information extracted by the search text word vector and the patent title word vector through the multi-head attention mechanism is concatenated into a second concatenated vector. The first concatenated vector and the second concatenated vector are then processed by the word-granular level classification module using the multi-head attention mechanism to extract feature information for the second time, and then concatenated again. Finally, similarity classification is performed to obtain the word-granular level patent similarity classification.

[0062] First, here P emb Each includes P emb_1 (Patent title word vector), P emb_2 (Search text word vectors), the input vector is processed by a positional encoding part to generate a positional encoding vector P with the same dimensions as the input vector. pos , will P pos Directly with Pemb The superposition yields a sentence embedding with location information: P emb_P =P emb +P pos ;

[0063] The P emb_P Including P emb_P1 P emb_P2 P emb_P1 P emb_P2 P emb_P2 Input the multi-head attention mechanism layer built into the Transformer encoder respectively. The definition of the multi-head attention mechanism is as follows: in QK T Represents the attention matrix, where ... The operation is to transform the attention matrix into a standard normal distribution. The number of self-attention heads i is set to 4. The outputs O1, O2, O3, and O4 of each self-attention mechanism are concatenated and linearly transformed to obtain the output of multi-head self-attention.

[0064] Enter the Transformer encoder's built-in summation and normalization layer to perform residual connection operations, as shown in the equation: SubLayer_s(P emb_P ) = P emb_P +Attention_P(Q, K, V) is shown. Layer normalization normalizes the hidden layers in the neural network to a standard normal distribution, accelerating convergence. The calculation process is shown in the equation: O sublayer =LayerNorm(P emb_P +SubLayer(P emb_P As shown in the figure.

[0065] Then, the signal is linearly mapped and activated by the feedforward neural network layer built into the Transformer encoder, and then enters the summation and normalization layer of the Transformer encoder. The operation of this layer is the same as the process described above.

[0066] After the two input vectors are processed in the Transformer encoder, P is obtained. emb_1_T P emb_2_T After external batch normalization, P is obtained. emb_1_TB P emb_2_TB ;

[0067] Next, P emb_1_TB P emb_2_TB The input is an external multi-head attention mechanism that extracts a deeper vector representation. The number of heads is set to i = 8. The outputs O1, O2, O3, O4, O5, O6, O7, and O8 of each self-attention mechanism are concatenated and linearly transformed to obtain the outputs of the multi-head self-attention mechanism, which are P.fc_1 P fc_2 ;

[0068] Finally, P fc_1 P fc_2 P is obtained by external batch normalization of the input. fc_1_B P fc_2_B This makes the distribution of input data in each layer of the network relatively stable, thus accelerating the learning speed of the model.

[0069] P emb_1_TB P emb_2_TB Perform vector concatenation to obtain P emb_con , will P fc_1_B P fc_2_B P is obtained by concatenating vectors. fc_con .

[0070] The specific implementation process of the classification network, specifically the word-level classification module, in the twin network will be P emb_con P fc_con In the multi-head attention mechanism of the input word granularity level classification module, Q and K of this multi-head attention mechanism come from P. emb_con V comes from P fc_con The calculation process is as follows:

[0071] head i =Attention(P) emb_con_i W q-i P emb_con_i W k_i P fc_con_i W v_i (4);

[0072] MA(P emb_con_i P fc_con_i )=Concat(head1, head2,..., head h W 0 (5);

[0073] Among them, W q_i W k_i W v_i ∈R C×d , It is a linear projection matrix; then, after a batch normalization process, P is obtained. fc ;

[0074] Next, for P fc A fully connected operation is performed; finally, the variable is mapped to the range of 0 and 1 by the sigmoid activation function to obtain the patent similarity classification result output_2 at the word granular level.

[0075] S6. Perform linear weighting on the patent similarity classification results at the sentence granularity level and the patent similarity classification results at the word granularity level, and output the patent similarity classification results.

[0076] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0077] (1) The patented recommendation method based on Transformer encoder and regularization strategy provided by this invention establishes a word and sentence dual processing model based on Transformer encoder and regularization strategy, and extracts features from the input data at the sentence granularity and word granularity levels respectively. This not only meets the user's requirements for the similarity of search results and makes the recommendation more flexible, but also makes the overall recommendation accuracy of the model higher.

[0078] (2) The patent recommendation method based on Transformer encoder and regularization strategy provided by this invention uses Transformer encoder superimposed with multi-head attention mechanism to extract features of sentence vector group information during the feature vector extraction process at the sentence granular level. It integrates the deep meaning of each sentence and extracts more features, so that the final fused vector representation can more accurately and comprehensively summarize the patent text information. This method solves the problem of inaccurate recommendation results caused by the inconsistency between the length of patent text and search text, and improves the accuracy of model recommendation results.

[0079] (3) The patented recommendation method based on Transformer encoder and regularization strategy provided by this invention uses a dual feature vector extraction method based on Siamese network using Transformer encoder and multi-head attention mechanism in the feature vector extraction process at the word granular level, which solves the problem of low text matching accuracy after single feature vector extraction and improves the model accuracy.

[0080] (4) The patented recommendation method based on Transformer encoder and regularization strategy provided by this invention uses a regularization strategy based on Dropout and symmetric JS divergence. The regularization strategy based on symmetric JS divergence solves the problem of inaccurate similarity judgment caused by the asymmetry of traditional KL divergence. It not only solves the problem of model overfitting and enhances the text expression ability of the model, but also improves the accuracy of the model. Attached Figure Description

[0081] Figure 1 This is a flowchart of the patent recommendation process based on the Transformer encoder and regularization strategy described in this invention.

[0082] Figure 2 This is a detailed structural flowchart of the model in Embodiment 1 of the present invention.

[0083] Figure 3This is a diagram of the overall structure of the model in Embodiment 1 of the present invention.

[0084] Figure 4 This is a structural diagram of the Transformer encoder superimposed with a multi-head attention mechanism in Embodiment 1 of the present invention.

[0085] Figure 5 This is a schematic diagram of the regularization strategy in Embodiment 1 of the present invention. Detailed Implementation

[0086] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0087] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0088] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0089] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0090] Example 1

[0091] This embodiment provides a patent recommendation method based on Transformer encoders and regularization strategies, such as... Figure 1 As shown:

[0092] S1. Establish a certain number of patent datasets based on multiple search texts, and classify them according to the search statements;

[0093] The patent dataset is shown in Table 1. S1 specifically includes:

[0094] S11. Considering the applicability of the patent recommendation function and the accuracy of the recommendation results, the patent data samples are divided into three categories (i.e., label "1", label "2", and label "3"). The samples of label "1" indicate that the patent and the search text are basically the same, the samples of label "2" indicate that the patent text and the search text are basically similar, and the samples of label "3" indicate that the patent text and the search text are unrelated.

[0095] S12. In practical application, the content of tag "3" is excluded first, and the content of tag "1" is displayed first, followed by tag "2". The content of tag "2" is used when the amount of data is insufficient or the search text is obscure. The setting of tag "2" largely solves the problem of zero patent recommendation. Zero recommendation means that there are no patents in the database that are basically consistent with the search text.

[0096] Specifically, the patent dataset selected in the database contains a total of 32,136 samples, of which 11,070 samples are in the category of similarity "1", 10,644 samples are in the category of similarity "2", and 10,424 samples are in the category of similarity "3". The dataset is divided into training set and validation set in an 8:2 ratio.

[0097] Table 1: Patent Dataset

[0098] S2. Perform data preprocessing on the search text and the patent text in the dataset to obtain the search text sentence vector, search text word vector, patent text sentence vector group and patent title word vector;

[0099] S2 specifically includes:

[0100] S21. Preprocess the search text by sentence segmentation: Process the input search text using the SBERT pre-trained model to obtain the search text sentence vector; the SBERT model refers to the Sentence Transformer, which completes the transformation from sentence to sentence vector.

[0101] S22. Preprocessing the patent text by sentence segmentation: The patent text is processed by using the method of segmenting the patent abstract into sentences and filling in the title. That is, the input patent text information is divided into patent title and patent abstract, and finally the patent text sentence vector group is obtained.

[0102] S22 specifically includes:

[0103] S221. Statistical analysis of the number of sentences in the patent abstract information in the patent dataset;

[0104] S222. The number of sentences in most patent abstracts is between 0 and 5, with only a small portion of patent abstracts having more than 5 sentences. Therefore, a segmentation threshold is set, usually 5, and the abstract is segmented. Then, if a patent abstract has fewer than 5 sentences, the patent title is used to complete the patent text sentence group to 5 sentences (6 sentences in total including the patent title). Finally, the pre-trained SBERT model is used to convert the patent text sentence group into a patent text sentence vector group.

[0105] If the patent abstract is divided into more than 5 sentences, then a similarity selection is performed, and the top 5 sentences that are least similar to the patent title are selected. The sentences that are least similar to the patent title are then converted into a patent text sentence vector group (5 patent abstract sentences and 1 patent title sentence).

[0106] Specifically, the selection of the least similar sentence to the patent title includes: using the SBERT model to convert the segmented sentences and patent title into vectors, calculating the cosine similarity between the sentence vector of each abstract and the sentence vector of the patent title, and sorting them in reverse according to the cosine similarity to obtain a group of patent text sentences that meet the segmentation threshold (5 patent abstracts and 1 patent title).

[0107] S23. Preprocessing of the patent text title and search text by word segmentation: After obtaining the patent text title and search text, preprocessing is performed on both, including jieba word segmentation, stop word removal, and punctuation removal. For example, "The machine learning model is configured to include information from the benchmark source in the computer-generated text and focus the computer-generated text based on control signals." is processed into "The machine learning model is configured to include computer-generated text based on benchmark source information and focus computer-generated text based on control signals." After processing, the pre-trained BERT model is used to perform vector processing on the segmented text representation to obtain the patent title word vector and the search text word vector, respectively.

[0108] S3. Establish a word and sentence dual processing model based on Transformer encoder and regularization strategy;

[0109] The word and sentence dual processing model based on Transformer encoder and regularization strategy includes a sentence-level feature vector extraction module, a sentence-level classification module, a word-level feature vector extraction module, and a word-level classification module.

[0110] The specific model structure is as follows Figure 3 As shown, it specifically includes:

[0111] The sentence-level feature vector extraction module includes a sentence-level feature extractor and a short text feature extractor; the sentence-level feature extractor includes a Transformer encoder and a multi-head attention mechanism to extract patent text vector group information; the short text feature extractor includes a fully connected layer;

[0112] The word-level feature vector extraction module includes two twin word-level feature extractors with shared parameters. The word-level feature extractor includes a Transformer encoder and a multi-head attention mechanism.

[0113] The similarity classification process in the sentence-level classification module uses fully connected layers and activation functions to calculate the patent similarity classification results.

[0114] The similarity classification process in the word-granularity level classification module uses a multi-head attention mechanism, a fully connected layer, and an activation function to calculate the patent similarity classification results.

[0115] Specifically, the Transformer encoder includes: a built-in multi-head attention mechanism, summation and normalization, a feedforward neural network, and a second summation and normalization, such as... Figure 4 As shown;

[0116] Furthermore, a regularization strategy based on Dropout and symmetric JS divergence is used to optimize the model structure and output results;

[0117] This invention uses the Dropout strategy to solve the overfitting problem of the model and improve the generalization ability of the model. The randomness of the Dropout strategy leads to inconsistency between the actual application model during the training process. That is, the method of randomly deleting units is used in the training phase, while the complete model without deleting any units is used in the actual application.

[0118] This invention employs a regularization strategy based on Dropout and symmetric JS divergence. By minimizing the bidirectional JS divergence between two distributions, it ensures that the two distributions output by the two sub-models of the same data remain consistent. Compared to the traditional Dropout strategy, this paper adds a symmetric JS divergence loss. Traditional KL divergence is asymmetric, and training the neural network can result in different training results depending on the order of the divergence. Using symmetric JS divergence can solve this problem. Figure 5 As shown;

[0119] The training dataset is as follows: Where n is the number of training samples, (x i y i () represents a data pair;

[0120] Input data x i Inputting the data into a word-sentence dual-processing model based on a Transformer encoder and regularization strategy twice yields two different results for data x. i The output distributions are respectively and A neuron is a vector that receives input from the previous layer, performs some operation on the input using its own parameters, and then produces an output. Because Dropout randomly removes a certain percentage of neurons, although the same data is input into the same model twice, it is actually input into two sub-models of the main model (the main model refers to the model that processes the input data x). iThe input is a word and sentence dual-processing model based on a Transformer encoder and regularization strategy, which is input twice. These two input models constitute the main model; the sub-model is any one of the two input models. and They are different, but the output distributions of sub-models input to the same data and input to the same model should be as similar as possible. Therefore, the regularization strategy based on Dropout and symmetric JS divergence is implemented by adding the symmetric JS divergence of the two different output distributions of the same sample to the cross-entropy loss function. The KL divergence of the output distributions of the two sub-models is represented as: D KL (P1||P2), where P1 and P2 refer to the two output distributions obtained after two inputs to the model. The formula for calculating the JS divergence of the model is:

[0121]

[0122] Cross-entropy loss function for the two sub-models:

[0123]

[0124] The loss function of the final model is:

[0125]

[0126]

[0127] Among them, α1 and α2 are used for control. The weighting coefficients are assigned in a specific way. Since the overall loss requirement is relatively small, the first two terms should have as large an α as possible, while the last term should have as small an α as possible. This method of assigning weights optimizes the overall loss. This regularization strategy improves both the model's generalization ability and its text representation ability, thereby enhancing the model's accuracy.

[0128] S4. Sentence-level processing: Input the sentence vectors of the search text and the sentence vectors of the patent text into the sentence-level feature vector extraction module and the sentence-level classification module to obtain the sentence-level patent similarity classification;

[0129] The processing flow of the sentence-level feature vector extraction module is as follows:

[0130] The search text sentence vectors are processed by a short text feature extractor to extract feature information, and the patent text sentence vector groups are processed by a sentence-granular feature extractor to extract feature information. The feature information extracted by the short text feature extractor and the feature information extracted by the sentence-granular feature extractor from the patent text sentence vector groups are concatenated, and finally, the sentence-granular level classification module performs similarity classification to obtain the sentence-granular level patent similarity classification.

[0131] Specifically as follows, such as Figure 2 As shown:

[0132] Search text sentence vector S emb The processing procedure is as follows:

[0133] First, use the Linear function to apply S emb Perform a linear transformation to obtain S emb_L Next, regarding S emb_L The activated search text sentence vector S is obtained using the nonlinear transformation activation function GELU. emb_L_G .

[0134] Patent text sentence vector group P emb The processing procedure is as follows:

[0135] like Figure 4 As shown, firstly, the patent text sentence vector group P emb After the location encoding part, a location encoding vector P with the same dimension as the batch of patent text vector groups is generated. pos , will P pos Directly with P emb The resulting vector group of patent text sentences with location information is obtained by superposition: P emb_P =P emb +P pos ;

[0136] Next, P emb_P Entering the multi-head attention mechanism layer built into the Transformer encoder, the definition of the multi-head attention mechanism is as follows: in QK T Represents the attention matrix, where ... The operation aims to transform the attention matrix into a standard normal distribution. The number of self-attention heads, i, is set to 4. The outputs O1, O2, O3, and O4 of each self-attention mechanism are concatenated, and a linear transformation is performed to obtain the output P of the multi-head self-attention mechanism. emb_P_MH ;

[0137] P emb_P_MH Entering the summation and normalization layer of the Transformer encoder, residual connection operations are performed; the purpose of layer normalization is to normalize the hidden layers in the neural network to a standard normal distribution, thereby accelerating convergence;

[0138] Then, the data undergoes linear mapping and activation through the feedforward neural network layer of the Transformer encoder, before entering the encoder's summation and normalization layer. This layer operates in the same manner as described above, ultimately leading to the encoder obtaining the sentence vector group P. emb_T .

[0139] The batch normalization layer processes the sentence vector group P after extracting feature vectors. emb_T Normalization is performed to obtain P emb_T_B Batch normalization is used to transform the sentence vector distribution into a smooth, isotropic standard Gaussian distribution to solve the problem of uneven and non-smooth distribution of sentence vector groups leading to unsatisfactory recommendation results.

[0140] The sentence vector group P is processed again using a multi-head attention mechanism. emb_T_B To extract deeper information, the feature extraction process is consistent with the multi-head attention mechanism in the encoder described above, resulting in P. emb_T_B_A ;

[0141] Finally, P emb_T_B_A After the final batch normalization layer, P is obtained. emb_T_B_A_B .

[0142] For the patent text sentence group P emb_T_B_A_B An averaging operation is performed to average the six sentence vectors in the patent sentence vector group to obtain an average sentence vector P representing the overall characteristics. emb_ave .

[0143] The sentence vector S of the search text emb_L_G And the average sentence vector P of the patent text emb_ave The concatenation function is used to concatenate two vectors to obtain S. Con Next, S Con After undergoing a linear transformation using the Linear function, the variable is activated by the GELU function. Then, the variable is activated by the sigmoid function, mapping it to a value between 0 and 1, resulting in the sentence-level patent similarity classification result output_1.

[0144] S5. Word-level processing: Input the word vectors of the search text and the word vectors of the patent title into the word-level feature vector extraction module and the word-level classification module to obtain the word-level patent similarity classification;

[0145] The processing flow of the word-level feature vector extraction module is as follows:

[0146] The search text word vector and the patent title word vector are respectively input into the word-granular feature extractor. The feature information extracted by the Transformer encoder of the word-granular feature extractor is concatenated into a first concatenated vector. Then, the feature information extracted by the search text word vector and the patent title word vector through the multi-head attention mechanism is concatenated into a second concatenated vector. The first concatenated vector and the second concatenated vector are then processed by the word-granular level classification module using the multi-head attention mechanism to extract feature information for the second time, and then concatenated again. Finally, similarity classification is performed to obtain the word-granular level patent similarity classification.

[0147] Specifically as follows, such as Figure 2 As shown:

[0148] This invention utilizes a Siamese network to extract feature vectors from segmented patent titles and search text using a Transformer encoder and multi-head attention mechanism, then calculates their similarity for classification. Traditional Siamese network models typically measure the similarity between two input feature vectors by comparing their distance. To adapt to the overall network model and improve performance, this invention employs a similarity classification method to measure the similarity between two input vectors. Applying the Siamese network model to extract feature vectors from segmented patent titles and search text not only allows for learning with small samples but also reduces susceptibility to errors from missamples, thus improving overall model performance.

[0149] The branch network in the twin network consists of two branches. Each branch passes through a Transformer encoder, batch normalization, multi-head attention mechanism, and batch normalization sequentially from input to output. The specific calculation process is as follows:

[0150] The data input to this module includes patent title word vectors and search text word vectors. Both vectors are fed into the twin's word feature extractor and undergo the same processing:

[0151] like Figure 4 As shown, firstly, here P emb Each includes P emb_1 (Patent title word vector), P emb_2 (Search text word vectors), the input vector is processed by a positional encoding part to generate a positional encoding vector P with the same dimensions as the input vector. pos , will P pos Directly with P emb The superposition yields a sentence embedding with location information: P emb_P =P emb +P pos ;

[0152] The P emb_P Including P emb_P1 P emb_P2 P emb_P1 P emb_P2 Input the multi-head attention mechanism layer built into the Transformer encoder respectively. The definition of the multi-head attention mechanism is as follows: in QK T Represents the attention matrix, where ... The operation is to transform the attention matrix into a standard normal distribution. The number of self-attention heads i is set to 4. The outputs O1, O2, O3, and O4 of each self-attention mechanism are concatenated and linearly transformed to obtain the output of multi-head self-attention.

[0153] Enter the Transformer encoder's built-in summation and normalization layer to perform residual connection operations, as shown in the equation: SubLayer_s(P emb_P ) = P emb_P +Attention_P(Q, K, V) is shown. Layer normalization normalizes the hidden layers in the neural network to a standard normal distribution, accelerating convergence. The calculation process is shown in the equation: O sublayer =LayerNorm(P emb_P +SubLayer(P emb_P As shown in the figure.

[0154] Then, the signal is linearly mapped and activated by the feedforward neural network layer built into the Transformer encoder, and then enters the summation and normalization layer built into the Transformer encoder. The operation of this layer is the same as the process described above.

[0155] After the two input vectors are processed in the Transformer encoder, P is obtained. emb_1_T P emb_2_T After external batch normalization, P is obtained. emb_1_TB P emb_2_TB ;

[0156] Next, P emb_1_TB P emb_2_TB The input is an external multi-head attention mechanism that extracts a deeper vector representation. The number of heads is set to i = 8. The outputs O1, O2, O3, O4, O5, O6, O7, and O8 of each self-attention mechanism are concatenated and linearly transformed to obtain the outputs of the multi-head self-attention mechanism, which are P. fc_1 P fc_2 ;

[0157] Finally, P fc_1 P fc_2 P is obtained by external batch normalization of the input. fc_1_B P fc_2_B This makes the distribution of input data in each layer of the network relatively stable, thus accelerating the learning speed of the model.

[0158] P emb_1_TB P emb_2_TB Perform vector concatenation to obtain P emb_con , will P fc_1_B P fc_2_B P is obtained by concatenating vectors. fc_con .

[0159] The specific implementation process of the classification network, specifically the word-level classification module, in the twin network will be P emb_con P fc_con In the multi-head attention mechanism of the input word granularity level classification module, Q and K of this multi-head attention mechanism come from P. emb_con V comes from P fc_con The calculation process is as follows:

[0160] head i =Attention(P) emb_con_i W q_i P emb_con_i W k_i P fc_con_i W v-i (4);

[0161] MA(P emb_con_i P fc_con_i )=Concat(head1, head2,..., head h W 0 (5);

[0162] Among them, W q_i W k_i W v_i ∈R C×d , It is a linear projection matrix; then, after a batch normalization process, P is obtained. fc ;

[0163] Next, for P fc A fully connected operation is performed; finally, the variable is mapped to the range of 0 and 1 by the sigmoid activation function to obtain the patent similarity classification result output_2 at the word granular level.

[0164] S6. Perform linear weighting on the sentence-level and word-level results to output the classification results: Perform linear weighting on the sentence-level and word-level patent similarity classification results to output the patent similarity classification results.

[0165] After matching the patent text with the search text and the patent title with the search text, the similarity classification results of the two sub-models are obtained, and the results of the two sub-models are linearly weighted.

[0166] : outputProba=W1×output_1+W2×output_2, where W1 and W2 are adjustable parameters, and output_1 and output_2 are the results of the two sub-models after the sigmoid activation function operation.

[0167] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A patent recommendation method based on Transformer encoder and regularization strategy, characterized in that, include; S1. Establish a certain number of patent datasets based on multiple search texts, and classify them according to the search statements; S2. Perform data preprocessing on the search text and the patent text in the dataset to obtain the search text sentence vector, search text word vector, patent text sentence vector group and patent title word vector; S3. Establish a word and sentence dual processing model based on Transformer encoder and regularization strategy; The word and sentence dual processing model based on Transformer encoder and regularization strategy includes a sentence-level feature vector extraction module, a sentence-level classification module, a word-level feature vector extraction module, and a word-level classification module. The sentence-level feature vector extraction module includes a sentence-level feature extractor and a short text feature extractor; the sentence-level feature extractor includes a Transformer encoder and a multi-head attention mechanism to extract patent text vector group information; the short text feature extractor includes a fully connected layer; The word-level feature vector extraction module includes two twin word-level feature extractors with shared parameters. The word-level feature extractor includes a Transformer encoder and a multi-head attention mechanism. The similarity classification process in the sentence-level classification module uses fully connected layers and activation functions to calculate the patent similarity classification results. The similarity classification process in the word-granularity level classification module uses a multi-head attention mechanism, a fully connected layer, and an activation function to calculate the patent similarity classification result; S4. Input the search text sentence vector and the patent text sentence vector group into the sentence-granularity level feature vector extraction module and the sentence-granularity level classification module to obtain the sentence-granularity level patent similarity classification; S5. Input the word vectors of the search text and the word vectors of the patent title into the word-granularity level feature vector extraction module and the word-granularity level classification module to obtain the word-granularity level patent similarity classification; S6. Perform linear weighting on the patent similarity classification results at the sentence granularity level and the patent similarity classification results at the word granularity level, and output the patent similarity classification results.

2. The patent recommendation method based on Transformer encoder and regularization strategy according to claim 1, characterized in that, S1 specifically includes: S11. Divide the samples in the patent dataset into three categories: label "1", label "2", and label "3". Samples with label "1" indicate that the patent and search text have the highest similarity. Samples with label "2" indicate that the patent and search text have a moderate similarity. Samples with label "3" indicate that the patent and search text have a low similarity. S12. In the application, the content of tag "3" is excluded first, the content of tag "1" is displayed first, and the content of tag "2" is displayed second.

3. The patent recommendation method based on Transformer encoder and regularization strategy according to claim 1, characterized in that, S2 specifically includes: S21. Preprocess the search text by sentence segmentation: Process the input search text using the SBERT pre-trained model to obtain the sentence vector of the search text; S22. Preprocess the patent text by segmenting it into sentences: Process the patent text by filling in the title after segmenting the patent abstract into sentences to obtain a group of sentence vectors for the patent text; S22 specifically includes: S221. Statistical analysis of the number of sentences in the patent abstract information in the patent dataset; S222. Set a segmentation threshold and segment the summary into sentences. If the number of segments is insufficient to meet the segmentation threshold, use patent titles to pad the number of patent text sentence groups to meet the segmentation threshold. Finally, use the pre-trained SBERT model to convert the patent text sentence groups into patent text sentence vector groups. If the number of segments exceeds the segmentation threshold, a similarity selection is performed, and the sentence least similar to the patent title is selected and converted into a patent text sentence vector group. The specific steps of selecting the sentence least similar to the patent title include: using the SBERT model to convert the segmented sentences and patent titles into vectors, calculating the cosine similarity between the sentence vector of each summary and the sentence vector of the patent title, and sorting them in reverse according to the cosine similarity to obtain the patent text sentence group that meets the segmentation threshold. S23. Perform word segmentation on the patent title and search text of the patent text: The patent titles of the search text and patent text are obtained and preprocessed. The preprocessing includes jieba word segmentation, stop word removal, and punctuation removal. After processing, the pre-trained BERT model is used to perform vector processing on the segmented text representation to obtain the word vectors of the patent title and the word vectors of the search text.

4. The patent recommendation method based on Transformer encoder and regularization strategy according to claim 3, characterized in that, S4 specifically includes: The search text sentence vectors are processed by a short text feature extractor to extract feature information, and the patent text sentence vector groups are processed by a sentence-granular feature extractor to extract feature information. The feature information extracted by the short text feature extractor and the feature information extracted by the sentence-granular feature extractor from the patent text sentence vector groups are concatenated, and finally, the sentence-granular level classification module performs similarity classification to obtain the sentence-granular level patent similarity classification.

5. The patent recommendation method based on Transformer encoder and regularization strategy according to claim 1, characterized in that, S5 specifically includes: The search text word vectors and patent title word vectors are respectively input into the word-granular feature extractor. The feature information extracted by the Transformer encoder of the word-granular feature extractor is concatenated into a first concatenated vector. Then, the feature information extracted by the search text word vectors and patent title word vectors through the multi-head attention mechanism is concatenated into a second concatenated vector. The first concatenated vector and the second concatenated vector are then processed by the word-granular level classification module using the multi-head attention mechanism to extract feature information for the second time, and then concatenated again. Finally, similarity classification is performed to obtain the word-granular level patent similarity classification.

6. The patent recommendation method based on Transformer encoder and regularization strategy according to claim 1, characterized in that, It also includes optimizing the model structure and output using regularization strategies based on Dropout and symmetric JS divergence: By minimizing the bidirectional JS divergence between two distributions, the two distributions output by the two sub-models of the same data are kept consistent. Specifically, this includes: The training dataset is as follows: Where n is the number of training samples, (x i y i () represents a data pair; Input data x i Input a word-sentence dual-processing model based on a Transformer encoder and regularization strategy twice, with each iteration serving as a sub-model, to obtain two different results for data x. i The output distributions are respectively and The regularization strategy based on Dropout and symmetric JS divergence is implemented by adding the symmetric JS divergence of the same sample with two different output distributions to the cross-entropy loss function. The KL divergence representation of the output distributions of the two sub-models: D KL (P1||P2), where P1 and P2 refer to the two output distributions obtained after inputting the model twice; The formula for calculating the JS divergence of the model is as follows: Cross-entropy loss function for the two sub-models: The loss function of the final model is: Among them, α1 and α2 are used for control. The weighting coefficients.

Images