Method and device for classifying achievement texts based on BERT model and multi-scale convolutional network

Abstract

An achievement text classification method based on a BERT model and a multi-scale convolutional network comprises the steps that firstly, an achievement text is expressed as a sub-word sequence through the BERT model, then the sub-word sequence is split to obtain a word segmentation result, and then the achievement text is converted into a word vector matrix through the word segmentation result and the achievement text of a corpus; the method comprises the following steps: firstly, carrying out convolution on a word vector matrix by using a multi-scale convolution network to obtain an explicit multi-scale semantic information matrix, and processing explicit multi-scale semantic information to obtain implicit multi-scale semantic information; performing weight convolution operation on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features and explicit-implicit interaction features, and fusing the explicit features, the implicit features and the explicit-implicit interaction features to obtain a semantic feature model; and classifying the result texts through a semantic feature model. According to the method, the achievement texts can be accurately classified into predefined categories, and support is provided for later achievement analysis.

Description

Technical Field

[0001] The present invention relates to the field of text management technology, and in particular to a method and device for classifying achievement text based on a BERT model and a multi-scale convolutional network. Background Art

[0002] The innovation studio is composed of well-known model workers, highly skilled talents, technical and skill leaders, and management leaders who have a certain theoretical level, work experience, and innovation ability in technology research and development, professional skills, and management personnel. The main members are to improve the innovation ability and employee quality of their respective units. They carry out technical innovation and management innovation activities around production and operation, skill talent training, and key and difficult issues of the enterprise. It is also an inspiration room for management innovation, work innovation, and institutional innovation. In the innovation studio, a large number of innovative achievement texts will be collected and classified online. However, the existing innovation studios mostly classify texts through manual computer operation. When the scale of the achievement text is large and complex, manual text classification is used, which is labor-intensive and makes the text classification efficiency low. At the same time, during manual classification, the classification criteria are relatively subjective and the key content of the text cannot be accurately obtained, which makes the text classification accuracy low. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects and problems of low efficiency and low classification accuracy of achievement text classification in the prior art, and to provide an achievement text classification method and device based on the BERT model and a multi-scale convolutional network with high classification efficiency and high classification accuracy.

[0004] To achieve the above objectives, the technical solution of the present invention is: a method for classifying achievement text based on the BERT model and a multi-scale convolutional network, comprising:

[0005] Collect the resulting texts and compile them into a corpus;

[0006] Using the BERT model, the result text is first represented as a subword sequence, and then the subword sequence is split to obtain the word segmentation result. Then, the result text is converted into a word vector matrix based on the word segmentation result and the result text of the corpus.

[0007] First, a multi-scale convolutional network is used to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and the explicit multi-scale semantic information is processed through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information. Then, the explicit multi-scale semantic information and the implicit multi-scale semantic information are weighted convolutionally operated to obtain explicit features, implicit features, and explicit-implicit interactive features. Then, the explicit features, implicit features, and explicit-implicit interactive features are fused to obtain a semantic feature model.

[0008] The collected results text is classified through the semantic feature model.

[0009] The result text is represented as a subword sequence, and then the subword sequence is split to obtain a word segmentation result, including:

[0010] Extract words from the corpus and generate corresponding frequencies to obtain a word frequency table in which each word corresponds to its frequency of occurrence;

[0011] Input the result text into the BERT model, and the calculation formula of the language model likelihood value of the input sentence S is:

[0012]

[0013] In the formula, S is the input sentence, which represents a subword sequence (x 1 、x 2 ,...x n ); P(x i ) is the probability of subword occurrence;

[0014] The three most frequent subwords in the input sentence S are merged, and the merged new word is used as the new subword. The calculation formula for the variation range of the language model likelihood value of the input sentence S is as follows:

[0015] ΔlogP(S)=logP(S N )-logP(S ABC );

[0016] In the formula, P(S N ) is the probability of occurrence of a new subword combination N in the input sentence S; P(S ABC ) is the probability of subwords A, B, and C appearing simultaneously in the input sentence S;

[0017] When the language model likelihood values ​​of all new sub-word combinations are smaller than a preset value, the combination is selected as the sub-word merging item of the input sentence S;

[0018] The tokenize module in the spacy natural language processing library is used to split the subword merge items, and the longest recognizable word is used as the word segmentation result.

[0019] The method of converting the result text into a word vector matrix through the word segmentation result and the result text of the corpus includes:

[0020] At the Token Embedding level, each subword in the word segmentation result is converted into an n-dimensional vector, and each subword generates a corresponding vector T;

[0021] At the Segment Embedding level, each sentence in the text is marked as 0 or 1, and the above classification is converted into the corresponding vector S e ;

[0022] At the Position Embedding level, the position information of each word in the text is encoded as a vector P, and the vector value is:

[0023]

[0024]

[0025] In the formula, i is the position of the word; j is the dimension of Position Embedding; d is the dimension of word vector;

[0026] The vectors T and S e , P are added to obtain the multi-eigenvector V;

[0027] The multi-layer Transformer structure in the BERT model is used to process the multiple feature vectors V to obtain the word vector matrix.

[0028] The method uses a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and processes the explicit multi-scale semantic information through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information, including:

[0029] Use a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix in, It represents the semantic information captured by each word itself; Represents the semantic information of each word and its two adjacent words captured; It represents the semantic information of each word and its surrounding four words captured;

[0030] The explicit multi-scale semantic information matrix is ​​concatenated in dimension to obtain a matrix of size 4n×d;

[0031] The matrix of size 4n×d is input into the bidirectional long short-term memory network for preprocessing, including:

[0032] Forward LSTM processing:

[0033]

[0034] In the formula, h l represents the LSTM hidden state at time step l; W l is the weight matrix of the LSTM layer;

[0035] Reverse LSTM processing:

[0036]

[0037] The results of forward processing and reverse processing are concatenated to obtain implicit multi-scale semantic information

[0038]

[0039] The implicit multi-scale semantic information obtained As the input of GRU, after the update gate and reset gate, the candidate hidden state and the final hidden state are obtained:

[0040]

[0041]

[0042]

[0043]

[0044] Where σ is the sigmoid activation function; W z , W r , W h is the weight matrix of the GRU layer; b z , b r , b h is the bias term of the GRU layer; z t is the update gate; r t To reset the gate; is the candidate hidden state; h′ t is the final hidden state, which represents the output of the time step in the current state.

[0045] The step of performing weighted convolution operation on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features, and explicit-implicit interactive features includes:

[0046] The explicit multi-scale semantic information of the corresponding scale With implicit multi-scale semantic information Multiply to get the weight interaction matrix at this scale:

[0047]

[0048] In the formula, is the weight interaction matrix; ⊙ represents the element-by-element multiplication of the matrix;

[0049] By adding the row and column weight interaction matrices, we can obtain explicit multi-scale semantic information and implicit multi-scale semantic information of length n:

[0050]

[0051]

[0052] In the formula, Represents the weight interaction matrix The jth row of Represents the weight interaction matrix The jth column of

[0053] Use the softmax function to calculate the weight coefficient of each element in the sequence:

[0054]

[0055] In the formula, α i and β i is the weight coefficient; x is the value of explicit or implicit semantic information;

[0056] Multiply the weight coefficient with the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain the explicit multi-scale weighted semantic information and the implicit multi-scale weighted semantic information:

[0057]

[0058]

[0059] In the formula, To provide explicit multi-scale weighted semantic information; It is implicit multi-scale weighted semantic information;

[0060] Extract the high-order features of the explicit multi-scale weighted semantic information matrix, the implicit multi-scale weighted semantic information matrix, and the weight interaction matrix to obtain the explicit feature ob vec , implicit feature hd vec And the interactive features of visibility and concealment vec :

[0061]

[0062] Where ReLU is the activation function; W con v is the weight of the convolution kernel; b con v is a one-dimensional array whose length is equal to the number of convolution kernels; * indicates the convolution operation;

[0063] Adopt and explicit features ob vec The same calculation method is used to obtain the implicit feature hd vec And the interactive features of visibility and concealment vec .

[0064] The method of fusing explicit features, implicit features and explicit-implicit interactive features to obtain a semantic feature model includes:

[0065] The explicit feature ob vec , implicit feature hd vec and the explicit and implicit interactive features ob ve c is fused to obtain three sets of matrices:

[0066] obhd vec =[ob vec ;hd vec ;ob vec ⊙hd vec ];

[0067] oboh vec =[ob vec ;oh vec ;ob vec Oh vec ];

[0068] hdoh vec =[hd vec ;oh vec ;hd vec Oh vec ];

[0069] Add the three sets of matrices to get the semantic feature model Svec:

[0070] S vec =obhd vec +oboh vec +hdoh vec .

[0071] A text classification device based on a BERT model and a multi-scale convolutional network, which is applied to the above-mentioned method, comprises:

[0072] The corpus building module is used to collect the output texts and aggregate them into a corpus;

[0073] The word vector matrix construction module is used to use the BERT model to first represent the result text as a subword sequence, then split the subword sequence to obtain the word segmentation result, and then convert the result text into a word vector matrix through the word segmentation result and the result text of the corpus;

[0074] The semantic feature model construction module is used to first use a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and then process the explicit multi-scale semantic information through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information, and then perform weighted convolution operations on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features, and explicit-implicit interactive features, and then fuse the explicit features, implicit features, and explicit-implicit interactive features to obtain a semantic feature model;

[0075] The achievement text classification module is used to classify the collected achievement texts through a semantic feature model.

[0076] A text classification device based on a BERT model and a multi-scale convolutional network, including a memory and a processor;

[0077] The memory is used to store computer program code and transmit the computer program code to the processor;

[0078] The processor is used to execute the method as described above according to the instructions in the computer program code.

[0079] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] The present invention provides a method and device for classifying achievement text based on a BERT model and a multi-scale convolutional network. The method can capture a wide range of semantic differences, including explicit and implicit ones, and can effectively identify and classify key information related to the innovation studio, thereby improving the accuracy of achievement text classification; at the same time, the model is suitable for classifying large-scale and complex text collections. The present invention can accurately classify achievement texts into predefined categories, providing strong technical support for the analysis of innovation studio achievements. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a flowchart of a text classification method based on a BERT model and a multi-scale convolutional network of the present invention.

[0083] Figure 2 It is a schematic diagram of obtaining explicit semantic information in an embodiment of the present invention.

[0084] Figure 3 It is a schematic diagram of obtaining implicit semantic information in an embodiment of the present invention.

[0085] Figure 4It is a schematic diagram of obtaining multi-scale weight semantic information in an embodiment of the present invention.

[0086] Figure 5 It is a structural block diagram of a text classification device based on a BERT model and a multi-scale convolutional network of the present invention.

[0087] Figure 6 It is a structural block diagram of a text classification device based on a BERT model and a multi-scale convolutional network of the present invention. DETAILED DESCRIPTION

[0088] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0089] See also Figure 1 , a text classification method based on the BERT model and multi-scale convolutional network, including:

[0090] S1. Collect the output texts and summarize them into a corpus. The output texts include innovation studio news, journals, project planning reports, technical and economic analysis reports, meeting minutes, carbon reduction technology research reports, market development trend analysis reports, energy market analysis reports, etc.

[0091] S2. Using the BERT model, the result text is first represented as a subword sequence, and then the subword sequence is split to obtain the word segmentation result. Then, the result text is converted into a word vector matrix through the word segmentation result and the result text of the corpus.

[0092] The result text is represented as a subword sequence, and then the subword sequence is split to obtain a word segmentation result, including:

[0093] Using Python programming, extract words from the corpus and generate corresponding frequencies to obtain a word frequency table in which each word corresponds to its frequency of occurrence;

[0094] Input the result text into the BERT model, and the calculation formula of the language model likelihood value of the input sentence S is:

[0095]

[0096] In the formula, S is the input sentence, which is represented as a subword sequence (x 1 、x 2 ,...x n ); P(x i ) is the probability of subword occurrence, and the probabilities between words are independent of each other;

[0097] According to the WordPiece algorithm, the three most frequent subwords in the input sentence S are merged, and the merged new word is used as the new subword, that is, a new subword combination N is selected to replace the original subwords A, B, and C. The calculation formula for the change range of the language model likelihood value of the input sentence S is as follows:

[0098] ΔlogP(S)=logP(S N )-logP(S ABC );

[0099] In the formula, P(S N ) is the probability of occurrence of a new subword combination N in the input sentence S; P(S ABC ) is the probability of subwords A, B, and C appearing simultaneously in the input sentence S;

[0100] When the language model likelihood values ​​of all new sub-word combinations are smaller than a preset value (the preset value is determined through experimental evaluation), the combination is selected as the sub-word merging item of the input sentence S;

[0101] The tokenize module in the spacy natural language processing library is used to split the sub-word merge items, and the longest recognizable word among them is used as the segmentation result, and the segmentation result is stored in the corresponding document in the form of a text sequence. For example, the sub-word merge item is "Innovation Studio Intelligent Analysis". The tokenize module is used to segment the sub-word merge item and obtain the following segmentation results: ["Innovation", "Studio", "Intelligence", "Analysis"].

[0102] The method of converting the result text into a word vector matrix through the word segmentation result and the result text of the corpus includes:

[0103] At the Token Embedding level, each subword in the word segmentation result is converted into an n-dimensional vector, and each subword generates a corresponding vector T;

[0104] At the Segment Embedding level, each sentence in the text is marked as 0 or 1, and the above classification is converted into the corresponding vector S e ; For example, we have two sentences: "I like the Innovation Studio" and "You published a journal", the entire input sequence is ["I", "like", "innovation", "studio", "you", "publish", "published", "journal"], then the corresponding Segment Embedding tag sequence is [0, 0, 0, 0, 1, 1, 1, 1].

[0105] At the Position Embedding level, the position information of each word in the text is encoded as a vector P, and the vector value is calculated using the existing sinusoidal encoding:

[0106]

[0107]

[0108] In the formula, i is the position of the word; j is the dimension of Position Embedding; d is the dimension of word vector;

[0109] The vectors T and S e , P are added to obtain the multi-eigenvector V, that is, V = T + S e +P;

[0110] The multi-layer Transformer structure in the BERT model is used to process the multi-feature vector V, that is, each word vector in the multi-feature vector V interacts with other word vectors, and the existing dot product attention method is used to calculate the similarity between each word and other words, so as to find the important words and key information of the text, and represent and encode this information through multi-dimensional vectors, and finally output the word vector matrix W∈R n×d , where n is the number of words and d is the dimension of the BERT model embedding vector.

[0111] S3. First, use a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and then process the explicit multi-scale semantic information through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information. Then, perform weighted convolution operations on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features, and explicit-implicit interactive features. Then, the explicit features, implicit features, and explicit-implicit interactive features are fused to obtain a semantic feature model.

[0112] The method uses a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and processes the explicit multi-scale semantic information through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information, including:

[0113] See also Figure 2 , use a multi-scale convolutional network to convolve the word vector matrix, and keep the convolution kernel size as 1, 3 and 5 to obtain an explicit multi-scale semantic information matrix in, It represents the semantic information captured by each word itself; Represents the semantic information of each word and its two adjacent words captured; It represents the semantic information of each word and its surrounding four words captured;

[0114] Compared to common models such as Skip-gram, which generate static word embeddings, that is, the representation of a word is independent of the context in which it appears, the BERT model is able to generate context-dependent word embeddings. The same word can have different representations in different sentences. BERT takes into account the context of the entire sentence.

[0115] The explicit multi-scale semantic information matrix is ​​concatenated in dimension to fuse the original word vector matrix and the explicit multi-scale semantic information, which is expressed as: It can be seen that W The matrix size is n×d, and after splicing, the resulting matrix size is 4n×d;

[0116] See also Figure 3 , a matrix of size 4n×d is input into the bidirectional long short-term memory network for preprocessing, including:

[0117] Forward LSTM processing:

[0118]

[0119] In the formula, h l W represents the LSTM hidden state at time step l (corresponding to the lth row of the Concatenated Matrix). LSTM captures and retains the information in the sequence so far through this hidden state; l is the weight matrix of the LSTM layer, which is used to convert the input data into a suitable space so that the LSTM unit can process it more efficiently. Its matrix size is p×d, and p is the hidden state dimension of the LSTM unit. Through this step, each row in the Concatenated Matrix is ​​processed separately, and the LSTM unit updates its hidden state at each time step, encoding the information in the sequence in this way.

[0120] Reverse LSTM processing:

[0121]

[0122] The results of forward processing and reverse processing are concatenated to obtain implicit multi-scale semantic information

[0123]

[0124] The implicit multi-scale semantic information obtained As the input of GRU, after the update gate and reset gate, the candidate hidden state and the final hidden state are obtained:

[0125]

[0126]

[0127]

[0128]

[0129] Where σ is the sigmoid activation function; W z , W r , W h is the weight matrix of the GRU layer; b z , b r , b h is the bias term of the GRU layer; z t is the update gate; r t To reset the gate; is the candidate hidden state; h′ t is the final hidden state, which represents the output of the time step in the current state.

[0130] GRU and its gating mechanism are used to extract more refined sequence features. This combination can effectively capture the implicit multi-scale semantic information in the text, while taking into account that GRU is particularly useful for the complex text structures involved in the innovation studio's results.

[0131] The step of performing weighted convolution operation on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features, and explicit-implicit interactive features includes:

[0132] The explicit multi-scale semantic information of the corresponding scale With implicit multi-scale semantic information Multiply to get the weight interaction matrix at this scale:

[0133]

[0134] In the formula, is the weight interaction matrix; ⊙ represents the element-by-element multiplication of the matrix;

[0135] By adding the row and column weight interaction matrices, we can obtain explicit multi-scale semantic information and implicit multi-scale semantic information of length n:

[0136]

[0137]

[0138] In the formula, Represents the weight interaction matrix The jth row of Represents the weight interaction matrix The jth column of

[0139] Use the softmax function to calculate the weight coefficient of each element in the sequence:

[0140]

[0141] In the formula, α i and β i is the weight coefficient; x is the value of explicit or implicit semantic information, representing a dimension of the feature vector;

[0142] Multiply the weight coefficient with the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain the explicit multi-scale weighted semantic information and the implicit multi-scale weighted semantic information:

[0143]

[0144]

[0145] In the formula, To provide explicit multi-scale weighted semantic information; It is implicit multi-scale weighted semantic information;

[0146] See also Figure 4 , extract the high-order features of the explicit multi-scale weighted semantic information matrix, the implicit multi-scale weighted semantic information matrix, and the weight interaction matrix, and obtain the explicit feature ob vec , implicit feature hd vec And the interactive features of visibility and concealment vec :

[0147]

[0148] Where ReLU is an activation function used to increase the nonlinear characteristics of the network. Its formula is f(x)=max(0,x); W conh is the weight of the convolution kernel, which is a four-dimensional tensor; b conv The bias term is a one-dimensional array with a length equal to the number of convolution kernels; * indicates a convolution operation;

[0149] Adopt and explicit features ob vec The same calculation method is used to obtain the implicit feature hd vec And the interactive features of visibility and concealment vec .

[0150] The method of fusing explicit features, implicit features and explicit-implicit interactive features to obtain a semantic feature model includes:

[0151] The explicit feature ob vec , implicit feature hd vec And the interactive features of visibility and concealment vec The fusion results in three sets of matrices:

[0152] ovhd vec =[ob vec ;hd vec ;ob vec ⊙hd vec ];

[0153] oboh vec =[ob vec ;oh vec ;ob vec Oh vec ];

[0154] hdoh vec =[hd vec ;oh vec ;hd vec Oh vec ];

[0155] Add the three sets of matrices to get the semantic feature model S vec :

[0156] S vec =pbjd vec +oboh vec +hdoh vec .

[0157] S4. Classify the collected results text through semantic feature model.

[0158] In the present invention, first, a corpus is constructed by manually summarizing the output results of each studio, and Python is used to extract word frequency to generate a word frequency table. Secondly, the text is processed using the preset dictionary method and the WordPiece algorithm to obtain the optimal subword merge item; the subwords are split and stored by the tokenize module in the spacy library, and then the segmentation results are further processed in the BERT model. Thirdly, the BERT model vectorizes the text through the Token, Segment and Position Embedding layers, and then applies a multi-layer Transformer structure to extract text key information in combination with the dot product attention method. Next, a multi-scale convolutional network is used to extract explicit multi-scale semantic information. In addition, high-order semantic information is mined in combination with the Bi-LSTM and GRU networks to further extract explicit and implicit interactive information features. Finally, explicit features, implicit features and interactive features are merged through a convolutional neural network (CNN) to form a comprehensive semantic feature vector. This method is particularly suitable for processing and analyzing large-scale and complex text collections, such as papers and patents produced by innovative studios, and can effectively identify and classify key information related to innovative studios. Through the present invention, the actual documents of the power industry are processed and classified through the above process. According to the final feature representation, the model can accurately classify the text into predefined categories, providing strong technical support for the analysis of the results of the innovation studio.

[0159] See also Figure 5 The present invention also provides a device for classifying achievement text based on a BERT model and a multi-scale convolutional network, which is applied to the above-mentioned method for classifying achievement text based on a BERT model and a multi-scale convolutional network, and the device comprises:

[0160] The corpus building module is used to collect the output texts and aggregate them into a corpus;

[0161] The word vector matrix construction module is used to use the BERT model to first represent the result text as a subword sequence, then split the subword sequence to obtain the word segmentation result, and then convert the result text into a word vector matrix through the word segmentation result and the result text of the corpus;

[0162] The semantic feature model construction module is used to first use a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and then process the explicit multi-scale semantic information through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information, and then perform weighted convolution operations on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features, and explicit-implicit interactive features, and then fuse the explicit features, implicit features, and explicit-implicit interactive features to obtain a semantic feature model;

[0163] The achievement text classification module is used to classify the collected achievement texts through a semantic feature model.

[0164] See also Figure 6 , the present invention also provides a text classification device based on the BERT model and the multi-scale convolutional network, including a memory and a processor;

[0165] The memory is used to store computer program code and transmit the computer program code to the processor;

[0166] The processor is used to execute a text classification method based on a BERT model and a multi-scale convolutional network according to the instructions in the computer program code.

[0167] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, a text classification method based on a BERT model and a multi-scale convolutional network is implemented.

[0168] Generally speaking, the computer instructions for implementing the method of the present invention may be carried in any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media may include any computer-readable media, except for the signal itself that is temporarily propagating.

[0169] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EKROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0170] Computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer or to an external computer (for example, using an Internet service provider to connect via the Internet) through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0171] The above-mentioned device and non-temporary computer-readable storage medium can be found in the specific description of a method for classifying results based on a BERT model and a multi-scale convolutional network and its beneficial effects, which will not be repeated here.

[0172] Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for text classification based on BERT model and multi-scale convolutional network, characterized in that: include: Collect the resulting texts and compile them into a corpus; Using the BERT model, the result text is first represented as a subword sequence, and then the subword sequence is split to obtain the word segmentation result. Then, the result text is converted into a word vector matrix based on the word segmentation result and the result text of the corpus. First, a multi-scale convolutional network is used to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and the explicit multi-scale semantic information is processed through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information. Then, the explicit multi-scale semantic information and the implicit multi-scale semantic information are weighted convolutionally operated to obtain explicit features, implicit features, and explicit-implicit interactive features. Then, the explicit features, implicit features, and explicit-implicit interactive features are fused to obtain a semantic feature model. The collected results text is classified through the semantic feature model.

2. The method for classifying achievement text based on the BERT model and multi-scale convolutional network according to claim 1, characterized in that: The result text is represented as a subword sequence, and then the subword sequence is split to obtain a word segmentation result, including: Extract words from the corpus and generate corresponding frequencies to obtain a word frequency table in which each word corresponds to its frequency of occurrence; Input the result text into the BERT model, and the calculation formula of the language model likelihood value of the input sentence S is: Where S is the input sentence, which represents a subword sequence (x1, x2, ...x n ); P(x i ) is the probability of subword occurrence; The three most frequent subwords in the input sentence S are merged, and the merged new word is used as the new subword. The calculation formula for the variation range of the language model likelihood value of the input sentence S is as follows: ΔlogP(S)=logP(S N )-logP(S ABC ); In the formula, P(S N ) is the probability of occurrence of a new subword combination N in the input sentence S; P(S ABC ) is the probability of subwords A, B, and C appearing simultaneously in the input sentence S; When the language model likelihood values ​​of all new sub-word combinations are smaller than a preset value, the combination is selected as the sub-word merging item of the input sentence S; The tokenize module in the spacy natural language processing library is used to split the subword merge items, and the longest recognizable word is used as the word segmentation result.

3. The method for classifying achievement texts based on the BERT model and multi-scale convolutional network according to claim 1, characterized in that: The method of converting the result text into a word vector matrix through the word segmentation result and the result text of the corpus includes: At the Token Embedding level, each subword in the word segmentation result is converted into an n-dimensional vector, and each subword generates a corresponding vector T; At the Segment Embedding level, each sentence in the text is marked as 0 or 1, and the above classification is converted into the corresponding vector S e ; At the Position Embedding level, the position information of each word in the text is encoded as a vector P, and the vector value is: In the formula, i is the position of the word; j is the dimension of Position Embedding; d is the dimension of word vector; The vectors T and S e , P are added to obtain the multi-eigenvector V; The multi-layer Transformer structure in the BERT model is used to process the multiple feature vectors V to obtain the word vector matrix.

4. The method for classifying achievement text based on the BERT model and multi-scale convolutional network according to claim 1, characterized in that: The method uses a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and processes the explicit multi-scale semantic information through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information, including: Use a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix in, It represents the semantic information captured by each word itself; Represents the semantic information of each word and its two adjacent words captured; It represents the semantic information of each word and its surrounding four words captured; The explicit multi-scale semantic information matrix is ​​concatenated in dimension to obtain a matrix of size 4n×d; The matrix of size 4n×d is input into the bidirectional long short-term memory network for preprocessing, including: Forward LSTM processing: In the formula, h l represents the LSTM hidden state at time step l; W l is the weight matrix of the LSTM layer; Reverse LSTM processing: The results of forward processing and reverse processing are concatenated to obtain implicit multi-scale semantic information 5. The method for classifying achievement text based on the BERT model and multi-scale convolutional network according to claim 4, characterized in that: The implicit multi-scale semantic information obtained As the input of GRU, after the update gate and reset gate, the candidate hidden state and the final hidden state are obtained: Where σ is the sigmoid activation function; W z , W r , W h is the weight matrix of the GRU layer; b z , b r , b h is the bias term of the GRU layer; z t is the update gate; r t To reset the gate; is the candidate hidden state; h′ t is the final hidden state, which represents the output of the time step in the current state.

6. The method for classifying achievement texts based on the BERT model and multi-scale convolutional network according to claim 4, characterized in that: The step of performing weighted convolution operation on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features, and explicit-implicit interactive features includes: The explicit multi-scale semantic information of the corresponding scale With implicit multi-scale semantic information Multiply to get the weight interaction matrix at this scale: In the formula, is the weight interaction matrix; ⊙ represents the element-by-element multiplication of the matrix; By adding the row and column weight interaction matrices, we can obtain explicit multi-scale semantic information and implicit multi-scale semantic information of length n: In the formula, Represents the weight interaction matrix The jth row of Represents the weight interaction matrix The jth column of Use the softmax function to calculate the weight coefficient of each element in the sequence: In the formula, α i and β i is the weight coefficient; x is the value of explicit or implicit semantic information; Multiply the weight coefficient with the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain the explicit multi-scale weighted semantic information and the implicit multi-scale weighted semantic information: In the formula, To provide explicit multi-scale weighted semantic information; It is implicit multi-scale weighted semantic information; Extract the high-order features of the explicit multi-scale weighted semantic information matrix, the implicit multi-scale weighted semantic information matrix, and the weight interaction matrix to obtain the explicit feature ob vec , implicit feature hd vec And the interactive features of visibility and concealment vec : Where ReLU is the activation function; W conv is the weight of the convolution kernel; b conv The bias term is a one-dimensional array with a length equal to the number of convolution kernels; * indicates a convolution operation; Adopt and explicit features ob vec The same calculation method is used to obtain the implicit feature hd vec And the interactive features of visibility and concealment vec .

7. The method for classifying achievement text based on the BERT model and multi-scale convolutional network according to claim 6, characterized in that: The method of fusing explicit features, implicit features and explicit-implicit interactive features to obtain a semantic feature model includes: The explicit feature ob vec , implicit feature hd vec And the interactive features of visibility and concealment vec The fusion results in three sets of matrices: obhd vec =[ob vec ;hd vec ob vec ⊙hd vec ]; oh my god vec =[about vec oh vec ;about vec ⊙oh vec ]; hdoh vec =[hd vec ;oh vec ;hd vec ⊙oh vec ]; Add the three sets of matrices to get the semantic feature model S vec : S vec =obhd vec +oboh vec +hdoh vec 。 8. A text classification device based on the BERT model and a multi-scale convolutional network, characterized in that: The device is applied to the method described in any one of claims 1 to 7, and the device comprises: The corpus building module is used to collect the output texts and aggregate them into a corpus; The word vector matrix construction module is used to use the BERT model to first represent the result text as a subword sequence, then split the subword sequence to obtain the word segmentation result, and then convert the result text into a word vector matrix through the word segmentation result and the result text of the corpus; The semantic feature model construction module is used to first use a multi-scale convolutional network to convolve the word vector matrix to obtain an explicit multi-scale semantic information matrix, and then process the explicit multi-scale semantic information through a bidirectional long short-term memory network to obtain implicit multi-scale semantic information, and then perform weighted convolution operations on the explicit multi-scale semantic information and the implicit multi-scale semantic information to obtain explicit features, implicit features, and explicit-implicit interactive features, and then fuse the explicit features, implicit features, and explicit-implicit interactive features to obtain a semantic feature model; The achievement text classification module is used to classify the collected achievement texts through a semantic feature model.

9. A text classification device based on the BERT model and a multi-scale convolutional network, characterized in that: including memory and processor; The memory is used to store computer program code and transmit the computer program code to the processor; The processor is configured to execute the method according to any one of claims 1 to 7 according to instructions in the computer program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images