A model construction method and device for commodity name classification

Abstract

The present invention relates to the field of network models, and specifically provides a model construction method for commodity name classification, which has the following steps: S1, open source Wikipedia and Books Corpus corpus collection, tax commodity name corpus collection; S2, data set operation on open source Wikipedia and BooksCorpus corpus; S3, introduction of BERT network and modular programming; S4, combination of various programming modules, construction of a complete End-to-End pipeline; S5, pre-training of the adjusted network using pre-processed open source data; S6, fine-tuning of the network to obtain the final tax field commodity name classification transfer learning model. Compared with the prior art, the model construction method for commodity name classification of the present invention can realize tax commodity name classification and has good promotion value.

Description

Technical Field

[0001] The present invention relates to the field of network models, and specifically provides a model building method and device for commodity name classification. Background Art

[0002] How to better represent natural language text (especially simplified Chinese) has become an unavoidable problem in the nearly 20 years of development of NLP (Natural Language Processing) technology. In recent years, the development of NLP technology can be roughly summarized as some forms of language modeling. Following statistical language models such as BOW (Bag of Words), TextRank (PageRank web search random algorithm idea), BM25 (TF-IDF evolved multi-word and text association degree measurement algorithm), since Bengio proposed the NNLM (Neural Network Language Models) neural language model in 2001, it has experienced Multi-mask Learning (multi-task learning), Word Embeddings (word embeddings), Neural networksfor NLP (NLP neural network), Sequence-to-sequence models for generation tasks, Attention (egScaled dot-product attention, self-attention, encoder-decoder attention) attention mechanism, Memory-based networks (memory-based networks), Pretrained language models (pre-trained language models) technology development, derived a large number of models of various ideas, the representation ability of natural language text has been continuously enhanced.

[0003] In the underlying NLP modeling framework, CWS (Chinese Word Segmentation) Chinese word segmentation is used as a pre-operation for natural language modeling (except for the model based on Transformer Chinese natural language processing task word piece as token). Its technical development has also evolved from traditional machine learning [BMES] annotation (HMM counting and CRF iteration) to DL (Deep Learning) deep learning based on NN (Neural Networks) neural network neural pre-trained language model (eg Transformer encoder network with Attention mechanism) representation, and the structured prediction direction of the post-GLUE task, which can form an ecological closed loop. In terms of specific language modeling tasks, dense word embedding has replaced sparse one-hot encoding (or Dummy encoding). Looking at the overall situation, for machine understanding of natural language text, the representation of the granularity of pre-words (or characters) (NNLM, Word2vec), the use of seq2seq encoder encoder in processing, and the introduction of attentin mechanism to replace RNN, so as to achieve real semantic understanding has become an inevitable choice.

[0004] In the formulation of the technical solution for commodity name classification modeling in the tax field, we continued the mainstream language model construction idea, took the Transformer Encoder as the model construction basis, and then fine-tuned the transfer learning model in the tax vertical field. However, how to design a reasonable classifier to achieve tax commodity name classification is a technical difficulty in this field. Summary of the invention

[0005] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a method for constructing a model for classifying commodity names with strong practicality.

[0006] A further technical task of the present invention is to provide a model building device for commodity name classification that is reasonably designed, safe and applicable.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A model building method for commodity name classification has the following steps:

[0009] S1, open source Wikipedia and Books Corpus corpus collection, tax commodity name corpus collection;

[0010] S2, perform dataset operations on open source Wikipedia and Books Corpus corpus;

[0011] S3. Introduce the BERT network and conduct modular programming;

[0012] S4. Combine various programming modules to build a complete End-to-End pipeline;

[0013] S5. Use the preprocessed open source data to pre-train the adjusted network;

[0014] S6. Fine-tune the network to obtain the final model of transfer learning for commodity name classification in the tax field.

[0015] Furthermore, in step S2, based on the pre-trained single sentence prediction, sentence pair prediction, SQuAD, Cloze, and structured prediction sequence annotation CWS, the open source Wikipedia and Books Corpus corpus are segmented, embedded, and labeled to formulate a dataset.

[0016] According to the Fine-tun product name classification Specific-task, information extraction, word segmentation and word embedding operations are performed on the tax product name data.

[0017] Furthermore, in step S3, BERT is introduced as the backbone network, and three major types of optimization modules are designed for the network, including the N-Gram structure display and implicit introduction of coarse-grained and fine-grained knowledge, the construction of Retrospective Feed Mechanism and Enhanced Recurrence Mechanism algorithms, and the hyperparameter improvement of Multi-headed and Scale dot-product Self-attention, and modular programming is implemented.

[0018] Furthermore, word granularity and word pair granularity knowledge are introduced, and the coarse-grained semantic knowledge ignored in BERT's MLM is enhanced through an explicit n-gram masking method. The components include continuous MLM, explicit n-gram masked language modeling, comprehensive n-gram prediction, enhanced n-gram relation modeling and n-gram extraction.

[0019] Furthermore, in the Multi-headed attention, BERT introduces an 8-head attention mechanism, and the multi-head representation is concatenated and then matrix product is performed. IS-gram uses horovod parallel technology and 12-head attention. At the same time, batch normalization is performed after dot-product to enhance the robustness of the model.

[0020] Furthermore, in the Scaled dot-product attention and the IS-gram model self-attention, when penalizing the dot product value before the softmax, the square root of the QKV vector dimension is no longer specified and is set as a learnable variable.

[0021] Furthermore, in step S5, the adjusted network is pre-trained using the pre-processed open source data until the network converges to generate a serialized file of the Pre-train network structure, weights, attributes, and optimizer status.

[0022] Furthermore, in step S6, two strategies are adopted respectively, namely, shielding the text representation module and shielding without module, and the preprocessed tax commodity name data is used to perform transfer learning fine-tuning on the network generated in step S5, and a serialized file of the fine-tune network weights and optimizer status is generated to obtain the final model of transfer learning of commodity name classification in the tax field.

[0023] A model building device for commodity name classification includes: at least one memory and at least one processor;

[0024] The at least one memory is used to store a machine-readable program;

[0025] The at least one processor is used to call the machine-readable program to execute a model building method for commodity name classification.

[0026] Compared with the prior art, the method and device for building a model for commodity name classification of the present invention have the following outstanding beneficial effects:

[0027] (1) By introducing the coarse-grained knowledge of explicit n-grams and combining it with fine-grained semantic knowledge, we can effectively mine the mutual / dependent relationships within the coarse-grained language information to alleviate the problem of insufficient semantic understanding ability of the original model, and effectively improve the accuracy and reliability of text semantic understanding.

[0028] (2) By introducing the Enhanced Recurrence Mechanism and Retrospective Feed Mechanism algorithms, the problem of the model being unable to process long texts due to the exponential growth of memory consumption and computational complexity as the length of the text increases has been effectively improved.

[0029] (3) By drawing on the continuous learning framework built by the Cooper Learning Circle, incremental data can be effectively utilized in real time. At the same time, as the data set continues to expand, the model's semantic learning ability is improved, prompting the model to learn more, remember more, and know more.

[0030] (4) Through tricks such as optimizing and improving multi-headed attention and scaled dot-product attention, the subjective problem of modeling is improved by fine-tuning the subtle structure of the network. At the same time, through the introduction of BN (batch normnalization) and LN (layer normalization), the problem of excessive propagation of deviations in the data feed forward process leading to the disappearance or explosion of backward gradients is effectively avoided. Relying on this type of optimization, the semantic understanding ability of the model is improved. In post-processing tasks, vectors with better semantic representation capabilities are used to significantly improve the effect of specific tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Attached Figure 1 It is a flowchart of a model building method for commodity name classification;

[0033] Attached Figure 2 It is a schematic diagram of a multi-granular input layer of a model building method for commodity name classification;

[0034] Attached Figure 3 It is a schematic diagram of Retrospective Feed Mechanism in a model building method for commodity name classification;

[0035] Attached Figure 4 It is a schematic diagram of the Enhanced Recurrence Mechanism in the model building method of commodity name classification;

[0036] Attached Figure 5 This is a schematic diagram of the attention mechanism of Multi-headed attention in the model construction method of commodity name classification;

[0037] Attached Figure 6 This is a schematic diagram of an MLP network in a model building method for commodity name classification. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] A best embodiment is given below:

[0040] In this embodiment, relying on the large-scale data sets of Wikipedia and Books Corpus, a neural language model with stronger semantic representation ability is designed and pre-trained, and transfer learning is performed on a small amount of tax commodity name data sets to obtain an encoding network that can effectively represent the tax commodity name, so as to achieve the purpose of digging out the real meaning behind the text from the representational symbols and expressing it in the form of vectors. After such conversion, the practical problem of commodity name classification is transformed into a pure mathematical classification problem, and the classification of tax commodity names is finally achieved by designing a reasonable classifier.

[0041] For single-granularity semantic modeling, the introduction of multi-granularity knowledge is ignored, resulting in the problem of insufficient semantic understanding ability of the model.

[0042] The word granularity and word pair granularity knowledge are introduced. Through the explicit n-gram masking method (n-gram mask), N consecutive token sequences are no longer used. The coarse (multi) granularity semantic knowledge ignored in BERT's MLM is enhanced. The components include: continuous MLM, explicit n-gram mask language modeling, comprehensive n-gram prediction, enhanced n-gram relation modeling, and n-gram extraction. The input layer is as follows: Figure 2 shown.

[0043] Transformer modeling length Bottleneck, resulting in the inability to handle longer input texts.

[0044] Drawing on the human habit of reading long texts, we redesigned the model framework, used the strategy of first rough reading and second intensive reading, and built a Retrospective Feed Mechanism. Figure 3 ) and Enhanced RecurrenceMechanism (Enhanced Recurrence Mechanism, Figure 4 )’s model structure (algorithm).

[0045] In vertical fields, new data is generated every moment, and the problem arises as to how to make timely and effective use of real-time incremental data.

[0046] Considering the incremental real-time online learning of data set expansion, a continuous learning framework is constructed, and the advanced MAML (La-MAML) is compared and analyzed. The online continuous learning Meta Learning algorithm based on fast optimization can provide a small amount of experience memory ability. By modulating the learning rate of each parameter, the current task is connected with the previous work. Compared with the traditional method based on prior, this algorithm provides a more flexible and effective method to solve the problem of catastrophic forgetting. Compared with the two-layer optimization framework based on cardinality constraints to construct core sets, it changes the shortcomings of only applying core sets to simple models, and can construct online maintainable core sets for large deep neural networks, so that they have sufficient experience advantages in continuous learning. Compared with meta-continuous learning, pre-training models are expected to reduce the problem of catastrophic forgetting as much as possible when fine-tuning a series of tasks, and based on continuous meta-learning, when measuring the performance of agents, the focus is on the speed of their recovery performance, rather than measuring without any application. The limitations of two types of methods that may promote the development of continuous learning are proposed. The agent must achieve fast processing and memory for newly distributed tasks, and a strong baseline for Continuous-MAML based on online expansion of the MAML algorithm is given to ensure the stability and plasticity of continuous learning.

[0047] In the Pre-train stage of Transfer Learning, how to improve the model structure, enhance the effectiveness of text representation, improve model overfitting and enhance model generalization ability.

[0048] In terms of the specific details of the network design, Multi-headed attention and Scaled dot-product attention are optimized and improved to enhance the representation ability of word / text vectors.

[0049] 1) Multi-headed attention

[0050] BERT introduces an 8-head attention mechanism, where multi-head representations are concatenated and then matrix product is performed. IS-gram uses the horovod parallel technology and 12-head attention. At the same time, BN (batch normalization) is performed after dot-product to improve the sensitivity of the gradient and enhance the robustness of the model. Figure 5 shown.

[0051] 2)Scaled dot-product attention

[0052] In the IS-gram model self-attention, when penalizing the dot product value before softmax, the square root of the QKV vector dimension is no longer specified. Instead, it is set as a learnable variable and learned by the model itself, eliminating human subjectivity.

[0053] In the Fine-tune stage of Transfer Learning, we will design a high-performance classifier for the post-processing specific-task of product name classification.

[0054] Through Benchmark verification and comparison experiments, it is known that the post-classifier with higher model capacity (network complexity) combined with Adam's lower learning rate can effectively improve the learning ability of the model gradient. Redesign the 8-layer MLP (Multiple Layers Perceptron) classifier, including 7 layers of linear segmentation and 8 layers of activation function nonlinear distortion. The model structure is as follows Figure 6 .

[0055] like Figure 1 As shown, a model building method for commodity name classification has the following steps:

[0056] S1, open source Wikipedia and Books Corpus corpus collection, tax commodity name corpus collection;

[0057] S2. Based on the pre-trained single sentence prediction, sentence pair prediction, SQuAD, Cloze, and structured prediction sequence annotation CWS, a total of 5 types of tasks, the open source Wikipedia and Books Corpus corpus is segmented, embedded, and labeled to formulate data set operations; based on the Fine-tun product name classification Specific-task, the tax product name data is subjected to information extraction, segmentation, and word embedding operations;

[0058] S3. Introduce BERT (Transformer Encoder) as the backbone network, design three major types of optimization modules for the network, including the introduction of N-Gram structure display and implicit coarse-grained and fine-grained knowledge, the construction of Retrospective Feed Mechanism and Enhanced Recurrence Mechanism algorithms, and the hyperparameter improvement of Multi-headed and Scale dot-product Self-attention, and implement modular programming.

[0059] S4. Combine various programming modules to build a complete End-to-End pipeline;

[0060] S5. Use the preprocessed open source data to pre-train the adjusted network until the network converges to generate a serialized file of the Pre-train network structure, weights, attributes, and optimizer status;

[0061] S6. Adopt two strategies, shielding the text representation module and shielding without module, respectively. Use the preprocessed tax commodity name data to perform transfer learning fine-tuning on the network generated by S5, and generate a serialized file of the network weights and optimizer status of Fine-tune to obtain the final model of transfer learning for tax commodity name classification.

[0062] A model building device for commodity name classification includes: at least one memory and at least one processor;

[0063] The at least one memory is used to store a machine-readable program;

[0064] The at least one processor is used to call the machine-readable program to execute a model building method for commodity name classification.

[0065] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any model building method and device claims that conform to a commodity name classification of the present invention and any appropriate changes or substitutions made by ordinary technicians in the technical field shall fall within the patent protection scope of the present invention.

[0066] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A model building method for commodity name classification, characterized in that: The steps are as follows: S1, open source Wikipedia and Books Corpus corpus collection, tax commodity name corpus collection; S2, perform dataset operations on open source Wikipedia and Books Corpus corpus; Based on the pre-trained single sentence prediction, sentence pair prediction, SQuAD, Cloze, and structured prediction sequence annotation CWS, we segmented, embedded, and labeled the open source Wikipedia and Books Corpus corpus to create a dataset. According to the Fine-tun commodity name classification Specific-task, information extraction, word segmentation, and word embedding operations are performed on the tax commodity name data; S3. Introduce the BERT network and conduct modular programming; BERT is introduced as the backbone network, and three types of optimization modules are designed for the network, including the introduction of N-Gram structure display and implicit coarse-grained and fine-grained knowledge, the construction of Retrospective Feed Mechanism and Enhanced Recurrence Mechanism algorithms, and the hyperparameter improvement of Multi-headed and Scale dot-product Self-attention, and modular programming implementation; Introducing word granularity and word pair granularity knowledge, and enhancing the coarse-grained semantic knowledge ignored in BERT's MLM through an explicit n-gram masking method. The components include continuous MLM, explicit n-gram masked language modeling, comprehensive n-gram prediction, enhanced n-gram relation modeling, and n-gram extraction. In the Multi-headed attention, BERT introduces an 8-head attention mechanism, performs concatenation operation on multi-head representations, and then performs matrix product. IS-gram uses horovod parallel technology and 12-head attention. At the same time, batch normalization is performed after dot-product to enhance the robustness of the model. In the Scaled dot-product attention, in the IS-gram model self-attention, when penalizing the dot product value before softmax, the square root of the QKV vector dimension is no longer specified and is set as a learnable variable; S4. Combine various programming modules to build a complete End-to-End pipeline; S5. Use the preprocessed open source data to pre-train the adjusted network; Use the preprocessed open source data to pre-train the adjusted network until the network converges to generate a serialized file of the Pre-train network structure, weights, attributes, and optimizer status; S6. Fine-tune the network to obtain the final tax-related commodity name classification transfer learning model; Two strategies, shielding the text representation module and shielding without module, are adopted respectively. The preprocessed tax commodity name data is used to perform transfer learning fine-tuning on the network generated in step S5, and a serialized file of the fine-tune network weights and optimizer status is generated to obtain the final model of transfer learning for tax commodity name classification.

2. A model building device for commodity name classification, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method of claim 1.

Images