Enterprise name vectorization method based on SimCSE

Abstract

The invention relates to the technical field of big data processing, in particular to an enterprise name vectorization method based on SimCSE. An enterprise training set is constructed through a large model and an instruction mode, a part of supervised data sets are also constructed in a regular mode, meanwhile, open-source short text supervised data sets are widely collected for data semantic transfer learning, and through the mode that the three data sets are mixed together for training, in the aspect of data processing, three kinds of data are mixed together, so that the data semantic transfer learning efficiency is improved. Only supervised data exist during data loading, and unsupervised data are only a loading mode under data modification, so that training in supervised and unsupervised modes can be realized, and the method is very suitable for long-term model training.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method for vectorizing enterprise names based on SimCSE. Background Art

[0002] In the process of analyzing the text content of digital government livelihood appeals, it is usually necessary to extract the responsible entities, especially the company names, based on information extraction technologies such as entity recognition and relationship extraction. This is of great significance for the government to quickly solve the potential risk problems in the company. However, the company names in the text content of livelihood appeals usually appear in abbreviated form, or even non-standard or irregular situations. That is, the entity names usually identified from the text are not standardized, which leads to the failure of big data analysis. Therefore, the normalization of company names in the text content of livelihood appeals is a very complex and important task. This patent mainly solves the problem of company name normalization. If a certain number (5 million) of normalized standard company name libraries have been built, the task of this patent is to extract the company names in the content of livelihood appeals through BiGRU-CRF and large language model technology, and then map the company names to the standard company name library, which is conducive to subsequent big data analysis tasks. Another technical background of this patent is that the era of large language models has arrived. This patent combines SimCSE comparative learning technology with large language model technology to effectively improve the effect of company name vectorization, so that the extracted company names can be mapped to the standard company name library. Summary of the invention

[0003] 1. The normalization and standardization of enterprise names are very beneficial to the subsequent big data analysis in the content of people's livelihood demands. However, enterprise names are neither words nor sentences, that is, using word vectors will cause some enterprise names to not be in the word vectors. Using sentence vectors to represent enterprise names, because enterprise names are generally short, the use of open source semantic vector models will waste computing resources and storage space. This patent proposes a method for generating supervised data based on large language model technology, and then obtaining a semantic model based on SimCSE training. At the same time, in order not to lose the semantic information of specific words in the enterprise name, many open source short text matching data sets are added to the supervised data set. This patent aims to propose a method for vectorizing enterprise names to effectively improve the effect of mapping an enterprise name to a standard enterprise name.

[0004] 2. SimCSE is a very good technology. The author of SimCSE innovatively regards the dropout operation as a method of data enhancement. For the same sentence, two different dropouts are used in training, and the sentence pairs after the two dropouts are regarded as a set of sample pairs that are mutually positive examples, that is, similar sentence pairs. Dissimilar sentence pairs are the remaining sentences in the same batch of samples. SimCSE has two training methods: unsupervised and supervised. This patent proposes a method of training SimCSE in a supervised and unsupervised manner at the same time, that is, a method of training supervised data and unsupervised data together can be achieved through simple data processing. At the same time, this patent proposes a data set construction method: one part of the supervised data is a widely collected open source short text similar matching pair, and one part is input into the big model through [company name + rewrite name instruction], that is, "XXX please rewrite the above unit name?" The output of the big model is YYY, XXX is the standard company name, YYY is the company name rewritten by the big model, and another part is constructed through the standard company name through rules, such as replacing XXX Technology Co., Ltd. with XXX Company, thereby forming a supervised data set of the same or similar company names, thereby completing the construction of the supervised data set; the unsupervised data set is the widely collected company names.

[0005] 3. The era of big language models has arrived. This patent uses a big language model at the data level to annotate massive data. It only requires simple cleaning to annotate massive data. At the technical level, the technology related to the big language model is introduced into SimCSE.

[0006] The present invention solves the existing technical problems by adopting the following technical solutions: 1. A method for vectorizing enterprise names based on SimCSE, characterized by comprising the following steps: S1. Construct training data set: (1) For the standard company name set, SimCSE needs to ensure that the same company name does not appear twice in a batch as much as possible. For some regular company names, corresponding training data sets are formed. For some company names without any obvious rules, supervised data sets are formed. Some company name data are not processed and used for unsupervised data set training, thus forming unsupervised and supervised data sets of company names. (2) Collect short text matching datasets from open source projects, filter out two semantically similar short texts, and build training sets through supervised and unsupervised datasets using large model instructions and regular matching technology; S2. Labeled data and model adaptation: If a text sample in the training data set obtained in step S1 is unsupervised data, that is, it is just a company name, then the text is tokenized. If the length after tokenization is less than 36, it will be padded to the model maximum length of 36 through [PAD]. If it exceeds 36, it will be truncated to 36. At this time, the text is processed into X, and X and X are directly spliced ​​together; if a text sample is supervised data, that is, two short texts with similar semantics or different expressions of a company name, the two texts are recorded as x1 and x2, and the two texts are tokenized separately. If the length is less than 36, it will be padded to 36 through [PAD], otherwise it will be truncated. At this time, the two texts are processed to obtain X1 and X2 respectively, and X1 and X2 are directly spliced ​​together. X and X are concatenated together, which is equivalent to forming a supervised data set through dropout during SimCSE training. In the supervised data, X1 and X2 are equivalent to the output of X1 after dropout and X2 after another dropout, so that SimCSE can be trained in both supervised and unsupervised ways. Then, through the dictionary Dict_token, it is mapped into a list of index numbers, which then becomes the Tensor tensor input to the SimCSE model. S3. Build the text feature extraction model of SimCSE: In SimCSE, the text feature extractor is recorded as Encoder. Encoder represents a text feature as a vector, which is used for subsequent enterprise name normalization. During the training process, the whole is first loaded into train.txt and trained in sequence. After training 1000 batches, train.txt is shuffled and reloaded, and the model is trained in sequence again. At the same time, the dropout value is selected from [0.1, 0.2, 0.3, 0.4] and continuously updated; S4. Training model: If batch_size=128, then the text content of [128, 72] will be input into the model, 72 is the length of 36*2 after the concatenation of two different texts but with the same semantics or the same company name, or a text, the length of 36*2 after concatenation with itself, and then transformed into [256, 36]. That is, at this time, the first 36-dimensional vector of 256 is the vectorized representation of the first text of the two concatenated texts of the first sample in the batch, and the second 36-dimensional vector of 256 is the vectorized representation of the second text of the two concatenated texts of the second sample in the batch, and so on. Then [256, 36] is input into the text encoder to obtain a semantic vectorized representation of a batch of [256, 128], 256=batch_size*2, the second dimension 128 is the embedding layer representation dimension, and then 1 is inserted in the first dimension and the second dimension respectively, becoming a tensor pred1 of size [256, 1, 128], and a tensor pred2 of size [1, 256, 128] tensor pred2, and then calculate the cosine similarity of pred1 and pred2. At this time, the tensor size is a tensor of [256, 256]. According to the set network structure and cross entropy loss function, the training time is 24 hours to complete the training. During the training process, the model encoder part is saved, and the enterprise name vectorization model of this patent is obtained.

[0007] The beneficial effect of the present invention is the first technical point: this patent proposes a technical solution for enterprise name vectorization based on SimCSE, constructs an enterprise training set through a large model + instruction method, and also uses regularization to construct a part of the supervised data set. At the same time, a wide range of open source short text supervised data sets are collected for data semantic transfer learning. By mixing the above three data sets together for training, in terms of data processing, this patent mixes the three types of data together, but when loading the data, the supervised data and the unsupervised data are only loaded under data modification, so that supervised and unsupervised methods can be trained together, and it is very suitable for long-term model training.

[0008] Technical Point 2: At the model level, this patent mainly proposes a technical solution for changing the dropout in the model training process, that is, each time the model is saved, the dropout value changes once, which is conducive to improving the generalization of the model. DETAILED DESCRIPTION

[0009] S1. Constructing training data set: This patent implements a data construction scheme suitable for SimCSE model training, as follows: (1) For the 5 million standard company names that have been built in the project, SimCSE needs to ensure that the same company name does not appear twice in a batch as much as possible. If a company name contains "Technology Co., Ltd.", 1 / 4 of the company names are replaced with "Technology Co., Ltd.", and 1 / 4 of the company names are replaced with "Limited Liability Company". 1 / 4 of these company names are unsupervised data sets, and 1 / 4 of these company names are generated into supervised data sets through a large model. The specific generation method is as follows: "XXX \nPlease rewrite the above company name? "The output of the big model is YYY, XXX is the standard company name, YYY will contain multiple expressions of the company name XXX generated by the big model, and then the company name in YYY will be parsed through regular matching. Each expression and XXX form a supervised data pair. The data generated by the big model does not need to be cleaned; then other company naming rules are discovered in turn. For example, "limited liability company" can replace "limited company", then 1 / 3 of the company names replace "limited liability company" with "limited company", 1 / 3 of such company names are unsupervised data sets, and 1 / 3 of such company names are supervised data sets generated by the big model. For the above-mentioned regular enterprise names, corresponding training data sets (including unsupervised data sets and supervised data sets) are formed. For enterprise names without any obvious rules, 1 / 3 of them are formed into supervised data sets through the above-mentioned large model generation method, and 2 / 3 of the enterprise name data are not processed in any way and used for unsupervised data set training. Through the above-mentioned method, 5 million unsupervised and supervised data sets about enterprise names are formed; (2) In order to fully understand the semantic content in enterprise names, short text matching data sets in open source projects are widely collected, and two short texts with similar semantics are filtered out, totaling 300,000 data sets. These 300,000 data sets are mainly used to reflect the ability to understand semantic vocabulary in some of the enterprise names. The above-mentioned 300,000 data sets and 5 million data sets are mixed together (no distinction is made between unsupervised data sets and supervised data sets). The above-mentioned training set construction method is one of the protection points of this patent, which can effectively improve the effect of SimCSE model on enterprise name vectorization. The operation of replacing "Limited Liability Company" with "Co., Ltd." is not the innovation of this patent. It is mainly for replacing different expressions of commonly used corporate names, such as replacing "Electronic Technology Co., Ltd." with "Electronic Technology Co., Ltd.". This patent mainly proposes a general solution for constructing a training set suitable for SimCSE, that is, a solution for constructing a training set by integrating supervised data sets, unsupervised data sets, large model instruction engineering technology, and regular matching technology.

[0010] S2. Labeled data and model adaptation: If a text sample in the training data set obtained in step S1 is unsupervised data, that is, it is just a company name, then the text is tokenized. If the length after tokenization is less than 36, it will be padded to the maximum length of the model 36 through [PAD]. If it exceeds 36, it will be truncated to 36. At this time, the text is processed into X, and X and X are directly spliced ​​together; if a text sample is supervised data, that is, two short texts with similar semantics or different expressions of a company name, the two texts are recorded as x1 and x2, and the two texts are tokenized separately. If the length is less than 36, it will be padded to 36 through [PAD], otherwise it will be truncated. At this time, the two texts After processing, X1 and X2 are obtained respectively, and X1 and X2 are directly concatenated together. This is one of the innovations of this patent. Concatenating X and X together is equivalent to forming a supervised data set through dropout during SimCSE training; in the supervised data, X1 and X2 can be equivalent to the output of X2 obtained by a very severe dropout of X1 and then another dropout. This becomes the theoretical basis for the two data forms of supervised data and unsupervised data in this patent to be compatible for training. This training method can maintain the highest possible vectorization representation for two companies that are the same company but whose names vary greatly at the character level. This is one of the innovations of this patent. Through the above simple data processing, SimCSE can be trained in supervised and unsupervised ways together, which is very helpful for developers to gradually iterate and update the model effect. Then, through the dictionary Dict_token, it is mapped into a list of index numbers, and then becomes a Tensor tensor input to the SimCSE model; the purpose of model training is to make the vectorized representation of any unsupervised sample in a batch as similar as possible to its vectorized representation obtained after dropout, and the vectorized representation of an input of any supervised sample as similar as possible to another vectorized representation of the supervised sample (another text with the same semantics or another expression of a company) after dropout, while the similarity with the vectorized representation of other texts in the batch is as low as possible.

[0011] S3. Constructing the text feature extraction model of SimCSE: In the era of large models, the technologies and working methods related to deep learning have undergone great changes. The model of text feature extraction also needs to undergo technical iteration. The text feature extractor in SimCSE is recorded as Encoder. Encoder represents a text feature as a vector for subsequent normalization of the company name. The Encoder of this patent adopts the same model architecture as BERT. The difference from the BERT structure is: the training method. If the training set is train.txt, then during the SimCSE training process, it is necessary to continuously minimize the similarity between a company name and all irrelevant company names, that is, a company name needs to be trained with more negative samples. During the training process of this patent, the entire train.txt is first loaded and trained in sequence. After training 1000 batches, train.txt is shuffled and reloaded, and the model is trained sequentially again. At the same time, the dropout value is selected from [0.1, 0.2, 0.3, 0.4] and continuously updated, which further improves the generalization of the model, that is, dropout is constantly changing during the training process. The code structure of each layer is: residual = enc_inputs enc_inputs = RMSNorm(enc_inputs) enc_inputs = residual+enc_self_attn(enc_inputs) residual = enc_inputs enc_inputs = RMSNorm2(enc_inputs) enc_outputs = residual+feed_forward(enc_inputs) Notes: enc_inputs: input of the previous layer RMSNorm: mean square error normalization enc_self_attn: multi-head attention layer, same as BERT feed_forward: feed-forward layer, same as BERT The shape and size of a text after being output by the Encoder is [1, 128], that is, a text (that is, a company name) is a 128-dimensional vector after being output by the Encoder.

[0012] S4. Training model: If batch_size=128, then the text content of [128, 72] will be input into the model, 72 is the length of a text pair after concatenation of two different texts but with the same semantics or the same company name (36*2), or the length of a text after concatenation with itself (36*2), and then transformed into [256, 36]. That is, at this time, the first 36-dimensional vector of 256 is the vectorized representation of the first text of the two concatenated texts of the first sample in the batch, and the second 36-dimensional vector of 256 is the vectorized representation of the second text of the two concatenated texts of the second sample in the batch, and so on. Then [256, 36] is input into the text encoder to obtain a semantic vectorized representation of a batch of [256, 128]. 256=batch_size*2, the second dimension 128 is the embedding layer representation dimension, and then 1 is inserted in the first dimension and the second dimension respectively, becoming a tensor pred1 of size [256, 1, 128], and a tensor pred2 of size [1, 256, 128] tensor pred2, and then calculate the cosine similarity of pred1 and pred2. At this time, the tensor size is a tensor of [256, 256]. The loss function of SimCSE training makes the values ​​of semantic similarity and textual similarity as large as possible, and the values ​​of other texts in a batch are the same, that is, the value of the index position corresponding to [256, 256] in the 256 dimension [1, 0, 3, 2, 5, 4, 7, 6, 9, 8, 11, 10, 13, 12......] is the largest. In order to avoid meaningless calculations between the diagonal and itself, the influence of the diagonal itself needs to be eliminated, that is, the diagonal part of the cosine similarity of [256, 256] is a very small value. According to the set network structure and cross entropy loss function, the training time is 24 hours to complete the training. During the training process, the model encoder part is saved, that is, the enterprise name vectorization model of this patent is obtained.

[0013] Glossary SimCSE: SimCSE is a contrastive learning technique. The semantic embedding of a text obtained through dropout is used as a positive example, and the embeddings obtained from other sentences are used as negative examples. SimCSE is a method that can obtain the vectorization of corporate names through unsupervised training.

[0014] Supervised learning: Supervised learning is a method of machine learning that learns from manually labeled training data and builds a model, which is then used to predict the output of unknown data.

[0015] Unsupervised learning: Unsupervised learning is another approach to machine learning that, unlike supervised learning, does not rely on labeled training data to train the model. Instead, unsupervised learning attempts to discover the inherent structure or patterns in the data from unlabeled data.

[0016] Token: A token is a basic unit in a text. It can be a Chinese character, a Chinese word, an English word, a set of numbers, punctuation marks, a single number, or other language elements. When processing text through deep learning, the text is generally tokenized and then entered into the embedding layer to train the model.

Claims

1. A method for vectorizing enterprise names based on SimCSE, characterized in that The steps include: S1. Construct training data set: (1) For the standard company name set, SimCSE needs to ensure that the same company name does not appear twice in a batch as much as possible. For some regular company names, corresponding training data sets are formed. For some company names without any obvious rules, supervised data sets are formed. Some company name data are not processed and used for unsupervised data set training, thus forming unsupervised and supervised data sets of company names. (2) Collect short text matching datasets from open source projects, filter out two semantically similar short texts, and build training sets through supervised and unsupervised datasets using large model instructions and regular matching technology; S2. Labeled data and model adaptation: If a text sample in the training data set obtained in step S1 is unsupervised data, that is, it is just a company name, then the text is tokenized. If the length after tokenization is less than 36, it will be padded to the model maximum length of 36 through [PAD]. If it exceeds 36, it will be truncated to 36. At this time, the text is processed into X, and X and X are directly spliced ​​together; if a text sample is supervised data, that is, two short texts with similar semantics or different expressions of a company name, the two texts are recorded as x1 and x2, and the two texts are tokenized separately. If the length is less than 36, it will be padded to 36 through [PAD], otherwise it will be truncated. At this time, the two texts are processed to obtain X1 and X2 respectively, and X1 and X2 are directly spliced ​​together. X and X are concatenated together, which is equivalent to forming a supervised data set through dropout during SimCSE training. In the supervised data, X1 and X2 are equivalent to the output of X1 after dropout and X2 after another dropout, so that SimCSE can be trained in both supervised and unsupervised ways. Then, through the dictionary Dict_token, it is mapped into a list of index numbers, which then becomes the Tensor tensor input to the SimCSE model. S3. Build the text feature extraction model of SimCSE: In SimCSE, the text feature extractor is recorded as Encoder. Encoder represents a text feature as a vector, which is used for subsequent enterprise name normalization. During the training process, the whole is first loaded into train.txt and trained in sequence. After training 1000 batches, train.txt is shuffled and reloaded, and the model is trained in sequence again. At the same time, the dropout value is selected from [0.1, 0.2, 0.3, 0.4] and continuously updated; S4. Training model: If batch_size=128, then the text content of [128, 72] will be input into the model, 72 is the length of 36*2 after the concatenation of two different texts but with the same semantics or the same company name, or a text, the length of 36*2 after concatenation with itself, and then transformed into [256, 36]. That is, at this time, the first 36-dimensional vector of 256 is the vectorized representation of the first text of the two concatenated texts of the first sample in the batch, and the second 36-dimensional vector of 256 is the vectorized representation of the second text of the two concatenated texts of the second sample in the batch, and so on. Then [256, 36] is input into the text encoder to obtain a semantic vectorized representation of a batch of [256, 128], 256=batch_size*2, the second dimension 128 is the embedding layer representation dimension, and then 1 is inserted in the first dimension and the second dimension respectively, becoming a tensor pred1 of size [256, 1, 128], and a tensor pred2 of size [1, 256, 128] tensor pred2, and then calculate the cosine similarity of pred1 and pred2. At this time, the tensor size is a tensor of [256, 256]. According to the set network structure and cross entropy loss function, the training time is 24 hours to complete the training. During the training process, the model encoder part is saved, and the enterprise name vectorization model of this patent is obtained.