A patent text classification method and system based on a self-growing model
By extending the growth dimension and structure of the neural network model through a self-growing model, the problems of insufficient model scalability and knowledge inheritance in text classification are solved, thereby improving the accuracy and adaptability of patent text classification.
Patent Information
- Application Number
- CN202410230585.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-29
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-02-29
AI Technical Summary
Existing technologies for text classification suffer from problems with model scalability and knowledge inheritance, especially when data is scarce, and transfer learning of pre-trained models suffers from insufficient knowledge preservation.
The self-growing model method is adopted to expand the growth dimension of the neural network model by using the self-growing operator, increase the number of neurons, and optimize the model structure by using the self-attention mechanism and skip connections to generate a grown neural network model for classification.
It enables automated expansion of the text classification model, improves knowledge inheritance capability and classification accuracy, and adapts to the classification of larger-scale patent text data.
Smart Images

Figure CN121301573B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of text classification technology, and more specifically, relates to a patented text classification method and system based on a self-growing model. Background Technology
[0002] Significant progress has been made in text classification technology, primarily due to advancements in deep learning and the availability of large-scale labeled datasets. Below are some key aspects and current status of text classification technology:
[0003] Deep learning models, especially neural network-based models, have achieved great success in text classification tasks. Among them, recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and gated recurrent units (GRUs) are widely used to process sequential data, such as text. Furthermore, fine-tuning pre-trained language models (such as BERT and GPT) has become a common practice in text classification tasks.
[0004] Transfer learning of pre-trained models: Pre-trained models (such as BERT, GPT, etc.) learn rich language representations by pre-training on large-scale corpora. These models can be fine-tuned or used for transfer learning to solve specific text classification tasks, greatly improving classification performance, especially when data is scarce.
[0005] In terms of accelerating model pre-training, existing technologies mainly adopt the approach of optimizing underlying hardware and algorithms. This approach will result in a loss of scalability and knowledge capacity, and there are also significant problems with knowledge inheritance and preservation. Summary of the Invention
[0006] To address the above technical problems, this invention proposes a patent text classification method based on a self-growing model, comprising:
[0007] The patent text data is acquired and preprocessed to remove stop words and punctuation marks. The preprocessed patent text data is then converted into a text tag sequence using a word segmenter.
[0008] The text tag sequence is used as input to the neural network model, and a model self-growth operation is performed on the neural network model to generate a grown neural network model.
[0009] The grown neural network model is used as a classification model to classify patent text data.
[0010] Furthermore, the self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator.
[0011] Furthermore, based on the determined growth dimension, a corresponding number of neurons are added to the neural network model.
[0012] Furthermore, the growth operator for the number of layers in the neural network model includes:
[0013] x n =c′⊙σ(W n ·x n-1 +b n )+(1-c′)⊙x n-1 +α·Layer att (x n-1 )
[0014] Where, x n The output of the nth layer of the neural network is given by c′, where c′ is the growth factor and W is the output of the nth layer. n Let x be the weight matrix of the nth layer of the neural network, σ be the non-linear loss function, and x be the weight matrix of the nth layer. n-1 b is the output of the (n-1)th layer of the neural network. n Let α be the bias term of the nth layer of the neural network, and let α be the weight of the attention mechanism in each layer of the neural network. att (x n-1 ) represents the attention weights of the output of the (n-1)th layer of the neural network.
[0015] Furthermore, the growth operators for hidden layers include:
[0016]
[0017] Where y′ is the output of the grown neural network layer, c is the mask, W′ is the connection weight of the grown neural network layer, and x T This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. Here, σ is the second weight matrix for the skip connections, σ is the non-linear activation function, and b is the bias term. Here, α is the skip connection bias term, α′ is a hyperparameter used to control the influence of attention weights on skip connections, and y is the output of a single neural network layer before growth. Let d'2 be an all-zero vector, where d'2 is the number of output neurons after growth and d2 is the number of output neurons before growth.
[0018] Furthermore, during model training, the growth factor c′ is used as an adjustable hyperparameter, and the optimal value of the growth factor c′ is determined through grid search, random search, or Bayesian optimization.
[0019] Furthermore, the attention weights of the outputs of the (n-1)th neural network layer are calculated through a self-attention mechanism, wherein the correlation score between the outputs of the (n-1)th neural network layer is calculated through dot product attention, additive attention, or scaled dot product attention.
[0020] Furthermore, the first weight matrix for skip connections and the second weight matrix for skip connections are set in each layer of the neural network to allow skip connections between different layers of the neural network, so that the output of the lower layer of the neural network can be directly passed to the higher layer of the neural network.
[0021] This invention also proposes a patent text classification system based on a self-growing model, comprising:
[0022] The data processing module is used to acquire patent text data and preprocess it, removing stop words and punctuation marks from the patent text data, and converting the preprocessed patent text data into a text tag sequence through a word segmenter;
[0023] The growth module is used to take the text tag sequence as input, input it into the neural network model, perform a model self-growth operation on the neural network model, and generate a grown neural network model.
[0024] The classification module is used to classify the patent text data by using the grown neural network model as a classification model.
[0025] Furthermore, the self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator.
[0026] Furthermore, based on the determined growth dimension, a corresponding number of neurons are added to the neural network model.
[0027] Furthermore, the growth operator for the number of layers in the neural network model includes:
[0028] x n =c′⊙σ(W n ·x n-1 +b n )+(1-c′)⊙x n-1 +α·Layer att (x n-1 )
[0029] Where, xn The output of the nth layer of the neural network is given by c′, where c′ is the growth factor and W is the output of the nth layer. n Let x be the weight matrix of the nth layer of the neural network, σ be the non-linear loss function, and x be the weight matrix of the nth layer. n-1 b is the output of the (n-1)th layer of the neural network. n Let α be the bias term of the nth layer of the neural network, and let α be the weight of the attention mechanism in each layer of the neural network. att (x n-1 ) represents the attention weights of the output of the (n-1)th layer of the neural network.
[0030] Furthermore, the growth operators for hidden layers include:
[0031]
[0032] Where y′ is the output of the grown neural network layer, c is the mask, W′ is the connection weight of the grown neural network layer, and x T This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. Here, σ is the second weight matrix for the skip connections, σ is the non-linear activation function, and b is the bias term. Here, α is the skip connection bias term, α′ is a hyperparameter used to control the influence of attention weights on skip connections, and y is the output of a single neural network layer before growth. Let d'2 be an all-zero vector, where d'2 is the number of output neurons after growth and d2 is the number of output neurons before growth.
[0033] Furthermore, during model training, the growth factor c′ is used as an adjustable hyperparameter, and the optimal value of the growth factor c′ is determined through grid search, random search, or Bayesian optimization.
[0034] Furthermore, the attention weights of the outputs of the (n-1)th neural network layer are calculated through a self-attention mechanism, wherein the correlation score between the outputs of the (n-1)th neural network layer is calculated through dot product attention, additive attention, or scaled dot product attention.
[0035] Furthermore, the first weight matrix for skip connections and the second weight matrix for skip connections are set in each layer of the neural network to allow skip connections between different layers of the neural network, so that the output of the lower layer of the neural network can be directly passed to the higher layer of the neural network.
[0036] The present invention also proposes an electronic device, including a processor and a memory, wherein the memory stores multiple instructions, and the processor is used to read the instructions and execute the patent text classification method based on a self-growing model.
[0037] The present invention also proposes a storage medium that stores multiple instructions, which can be read and executed by a processor to provide a patent text classification method based on a self-growing model.
[0038] Compared with the prior art, the above-described technical solutions conceived in this invention have the following beneficial effects:
[0039] This invention acquires patent text data and preprocesses it to remove stop words and punctuation marks. A word segmenter then converts the preprocessed patent text data into a text tag sequence. This text tag sequence is input into a neural network model, where a self-growing operation is performed to generate a grown neural network model. This grown neural network model is then used as a classification model to classify the patent text data. Through this technical solution, this invention can automatically extend the text classification model, enabling it to grow and achieve knowledge preservation based on knowledge inheritance. This allows it to adapt to larger-scale patent text data classification. Furthermore, due to the knowledge inheritance capability, the classification accuracy of the patent text is also improved. Attached Figure Description
[0040] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;
[0041] Figure 2 This is a system structure diagram of Embodiment 2 of the present invention. Detailed Implementation
[0042] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.
[0043] The method provided by this invention can be implemented in a terminal environment that may include one or more of the following components: a processor, a storage medium, and a display screen. The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.
[0044] A processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the terminal, and performs various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in the storage medium, and by calling data stored in the storage medium.
[0045] Storage media can include random access memory (RAM) or read-only memory (ROM). Storage media can be used to store instructions, programs, code, code sets, or instructions.
[0046] The display screen is used to show the interactive sections of various applications.
[0047] In the formula of this invention, all subscripts are only used to distinguish parameters and have no actual meaning.
[0048] In addition, those skilled in the art will understand that the structure of the terminal described above does not constitute a limitation on the terminal. The terminal may include more or fewer components, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, input units, sensors, audio circuits, power supplies, and other components, which will not be described in detail here.
[0049] Example 1
[0050] like Figure 1 As shown, this embodiment of the invention provides a patent text classification method based on a self-growing model, including:
[0051] Step 101: Obtain patent text data and preprocess it to remove stop words and punctuation marks from the patent text data. Then, use a word segmenter to convert the preprocessed patent text data into a text tag sequence.
[0052] Step 102: The text tag sequence is used as input and fed into a neural network model (for example, the neural network model can be BERT, GPT, etc.). The neural network model is subjected to a model self-growth operation to generate a grown neural network model.
[0053] Specifically, the self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator.
[0054] Specifically, based on the determined growth dimension, a corresponding number of neurons are added to the neural network model.
[0055] Specifically, the growth operator for the number of layers in the neural network model includes:
[0056] x n =c′⊙σ(W n ·x n-1 +b n )+(1-c′)⊙x n-1 +α·Layer att(x n-1 )
[0057] Where, x n The output of the nth layer of the neural network is given by c′, where c′ is the growth factor and W is the output of the nth layer. n Let x be the weight matrix of the nth layer of the neural network, σ be the non-linear loss function, and x be the weight matrix of the nth layer. n-1 b is the output of the (n-1)th layer of the neural network. n Let α be the bias term of the nth layer of the neural network, and let α be the weight of the attention mechanism in each layer of the neural network. att (x n-1 ) represents the attention weights of the output of the (n-1)th layer of the neural network.
[0058] Specifically, the growth operators for hidden layers include:
[0059]
[0060] Where y′ is the output of the grown neural network layer, c is the mask, W′ is the connection weight of the grown neural network layer, and x T This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. Here, σ is the second weight matrix for the skip connections, σ is the non-linear activation function, and b is the bias term. Here, α is the skip connection bias term, α′ is a hyperparameter used to control the influence of attention weights on skip connections, and y is the output of a single neural network layer before growth. Let d'2 be an all-zero vector, where d'2 is the number of output neurons after growth and d2 is the number of output neurons before growth.
[0061] Specifically, when training the model, the growth factor c′ is used as an adjustable hyperparameter, and the optimal value of the growth factor c′ is determined through grid search, random search, or Bayesian optimization.
[0062] Specifically, the attention weights of the outputs of the (n-1)th layer of the neural network are calculated through a self-attention mechanism, wherein the correlation score between the outputs of the (n-1)th layer of the neural network is calculated through dot product attention, additive attention, or scaled dot product attention.
[0063] Specifically, a first weight matrix for skip connections and a second weight matrix for skip connections are set in each layer of the neural network to allow skip connections between different layers, so that the output of the lower layer of the neural network can be directly passed to the higher layer of the neural network.
[0064] Step 103: Use the grown neural network model as a classification model to classify the patent text data.
[0065] Example 2
[0066] like Figure 2 As shown, this embodiment of the invention also proposes a patent text classification system based on a self-growing model, comprising:
[0067] The data processing module is used to acquire patent text data and preprocess it, removing stop words and punctuation marks from the patent text data, and converting the preprocessed patent text data into a text tag sequence through a word segmenter;
[0068] The growth module is used to take the text tag sequence as input and input it into a neural network model (for example, the neural network model can be BERT, GPT, etc.), perform a model self-growth operation on the neural network model, and generate a grown neural network model.
[0069] Specifically, the self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator.
[0070] Specifically, based on the determined growth dimension, a corresponding number of neurons are added to the neural network model.
[0071] Specifically, the growth operator for the number of layers in the neural network model includes:
[0072] x n =c′⊙σ(W n ·x n-1 +b n )+(1-c′)⊙x n-1 +α·Layer att (x n-1 )
[0073] Where, x n The output of the nth layer of the neural network is given by c′, where c′ is the growth factor and W is the output of the nth layer. n Let x be the weight matrix of the nth layer of the neural network, σ be the non-linear loss function, and x be the weight matrix of the nth layer. n-1 b is the output of the (n-1)th layer of the neural network. n Let α be the bias term of the nth layer of the neural network, and let α be the weight of the attention mechanism in each layer of the neural network. att (x n-1 ) represents the attention weights of the output of the (n-1)th layer of the neural network.
[0074] Specifically, the growth operators for hidden layers include:
[0075]
[0076] Where y′ is the output of the grown neural network layer, c is the mask, W′ is the connection weight of the grown neural network layer, and x T This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. Here, σ is the second weight matrix for the skip connections, σ is the non-linear activation function, and b is the bias term. Here, α is the skip connection bias term, α′ is a hyperparameter used to control the influence of attention weights on skip connections, and y is the output of a single neural network layer before growth. Let d'2 be an all-zero vector, where d'2 is the number of output neurons after growth and d2 is the number of output neurons before growth.
[0077] Specifically, when training the model, the growth factor c′ is used as an adjustable hyperparameter, and the optimal value of the growth factor c′ is determined through grid search, random search, or Bayesian optimization.
[0078] Specifically, the attention weights of the outputs of the (n-1)th layer of the neural network are calculated through a self-attention mechanism, wherein the correlation score between the outputs of the (n-1)th layer of the neural network is calculated through dot product attention, additive attention, or scaled dot product attention.
[0079] Specifically, a first weight matrix for skip connections and a second weight matrix for skip connections are set in each layer of the neural network to allow skip connections between different layers, so that the output of the lower layer of the neural network can be directly passed to the higher layer of the neural network.
[0080] The classification module is used to classify the patent text data by using the grown neural network model as a classification model.
[0081] Example 3
[0082] This invention also proposes a storage medium storing multiple instructions for implementing the patent text classification method based on a self-growing model.
[0083] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.
[0084] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: Step 101, acquire patent text data and preprocess it to remove stop words and punctuation marks from the patent text data, and convert the preprocessed patent text data into a text tag sequence through a word segmenter;
[0085] Step 102: The text tag sequence is used as input and fed into a neural network model (for example, the neural network model can be BERT, GPT, etc.). The neural network model is subjected to a model self-growth operation to generate a grown neural network model.
[0086] Specifically, the self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator.
[0087] Specifically, based on the determined growth dimension, a corresponding number of neurons are added to the neural network model.
[0088] Specifically, the growth operator for the number of layers in the neural network model includes:
[0089] x n =c′⊙σ(W n ·x n-1 +b n )+(1-c′)⊙x n-1 +α·Layer att (x n-1 )
[0090] Where, x n The output of the nth layer of the neural network is given by c′, where c′ is the growth factor and W is the output of the nth layer. n Let x be the weight matrix of the nth layer of the neural network, σ be the non-linear loss function, and x be the weight matrix of the nth layer. n-1 b is the output of the (n-1)th layer of the neural network. n Let α be the bias term of the nth layer of the neural network, and let α be the weight of the attention mechanism in each layer of the neural network. att (x n-1 ) represents the attention weights of the output of the (n-1)th layer of the neural network.
[0091] Specifically, the growth operators for hidden layers include:
[0092]
[0093] Where y′ is the output of the grown neural network layer, c is the mask, W′ is the connection weight of the grown neural network layer, and x T This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. Here, σ is the second weight matrix for the skip connections, σ is the non-linear activation function, and b is the bias term. Here, α is the skip connection bias term, α′ is a hyperparameter used to control the influence of attention weights on skip connections, and y is the output of a single neural network layer before growth. Let d'2 be an all-zero vector, where d'2 is the number of output neurons after growth and d2 is the number of output neurons before growth.
[0094] Specifically, when training the model, the growth factor c′ is used as an adjustable hyperparameter, and the optimal value of the growth factor c′ is determined through grid search, random search, or Bayesian optimization.
[0095] Specifically, the attention weights of the outputs of the (n-1)th layer of the neural network are calculated through a self-attention mechanism, wherein the correlation score between the outputs of the (n-1)th layer of the neural network is calculated through dot product attention, additive attention, or scaled dot product attention.
[0096] Specifically, a first weight matrix for skip connections and a second weight matrix for skip connections are set in each layer of the neural network to allow skip connections between different layers, so that the output of the lower layer of the neural network can be directly passed to the higher layer of the neural network.
[0097] Step 103: Use the grown neural network model as a classification model to classify the patent text data.
[0098] Example 4
[0099] This invention also proposes an electronic device, including a processor and a storage medium connected to the processor. The storage medium stores multiple instructions, which can be loaded and executed by the processor to enable the processor to execute a patent text classification method based on a self-growing model.
[0100] Specifically, the electronic device in this embodiment can be a computer terminal, which may include one or more processors and a storage medium.
[0101] The storage medium can be used to store software programs and modules, such as the patent text classification method based on a self-growing model in this embodiment of the invention. The processor executes the software programs and modules stored in the storage medium to perform various functional applications and data processing, thus realizing the aforementioned patent text classification method based on a self-growing model. The storage medium may include high-speed random access storage media, and may also include non-volatile storage media, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state storage media. In some instances, the storage medium may further include storage media remotely configured relative to the processor, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0102] The processor can call the information and application stored in the storage medium through the transmission system to execute the following steps: Step 101, acquire patent text data and perform preprocessing, remove stop words and punctuation marks from the patent text data, and convert the preprocessed patent text data into a text tag sequence through a word segmenter;
[0103] Step 102: The text tag sequence is used as input and fed into a neural network model (for example, the neural network model can be BERT, GPT, etc.). The neural network model is subjected to a model self-growth operation to generate a grown neural network model.
[0104] Specifically, the self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator.
[0105] Specifically, based on the determined growth dimension, a corresponding number of neurons are added to the neural network model.
[0106] Specifically, the growth operator for the number of layers in the neural network model includes:
[0107] x n =c′⊙σ(W n ·x n-1 +b n )+(1-c′)⊙x n-1 +α·Layer att (x n-1 )
[0108] Where, x n The output of the nth layer of the neural network is given by c′, where c′ is the growth factor and W is the output of the nth layer. n Let x be the weight matrix of the nth layer of the neural network, σ be the non-linear loss function, and x be the weight matrix of the nth layer.n-1 b is the output of the (n-1)th layer of the neural network. n Let α be the bias term of the nth layer of the neural network, and let α be the weight of the attention mechanism in each layer of the neural network. att (x n-1 ) represents the attention weights of the output of the (n-1)th layer of the neural network.
[0109] Specifically, the growth operators for hidden layers include:
[0110]
[0111] Where y′ is the output of the grown neural network layer, c is the mask, W′ is the connection weight of the grown neural network layer, and x T This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. Here, σ is the second weight matrix for the skip connections, σ is the non-linear activation function, and b is the bias term. Here, α is the skip connection bias term, α′ is a hyperparameter used to control the influence of attention weights on skip connections, and y is the output of a single neural network layer before growth. Let d'2 be an all-zero vector, where d'2 is the number of output neurons after growth and d2 is the number of output neurons before growth.
[0112] Specifically, when training the model, the growth factor c′ is used as an adjustable hyperparameter, and the optimal value of the growth factor c′ is determined through grid search, random search, or Bayesian optimization.
[0113] Specifically, the attention weights of the outputs of the (n-1)th layer of the neural network are calculated through a self-attention mechanism, wherein the correlation score between the outputs of the (n-1)th layer of the neural network is calculated through dot product attention, additive attention, or scaled dot product attention.
[0114] Specifically, a first weight matrix for skip connections and a second weight matrix for skip connections are set in each layer of the neural network to allow skip connections between different layers, so that the output of the lower layer of the neural network can be directly passed to the higher layer of the neural network.
[0115] Step 103: Use the grown neural network model as a classification model to classify the patent text data.
[0116] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0117] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0118] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The system embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between units or modules, and may be electrical or other forms.
[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0120] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0121] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, read-only storage media (ROM), random access storage media (RAM), portable hard drives, magnetic disks, optical disks, and other media capable of storing program code.
[0122] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A patent text classification method based on a self-growing model, characterized in that, include: The patent text data is acquired and preprocessed to remove stop words and punctuation marks. The preprocessed patent text data is then converted into a text tag sequence using a word segmenter. The text tag sequence is used as input to the neural network model, and a model self-growth operation is performed on the neural network model to generate a grown neural network model. The self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator; Based on the determined growth dimension, a corresponding number of neurons are added to the neural network model; The growth operators for the number of layers in the neural network model include: wherein, is the output of the layer neural network layer, is a growth factor, is the output of the layer neural network layer, is a non-linear loss function, is the output of the layer neural network layer, is the output of the layer neural network layer, is a weight of the attention mechanism for each layer neural network layer, is the output of the layer neural network layer, Growth operators for hidden layers include: in, This is the output of one layer of the grown neural network. For the mask, These are the connection weights of the next layer in the grown neural network. This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. The second weight matrix is used for jump connections. It is a non-linear activation function. For bias terms, For the jump connection bias term, These are hyperparameters used to control the degree to which attention weights affect skip connections. This is the output of a neural network layer before growth. It is a vector of all zeros. This represents the number of output neurons after growth. This represents the number of output neurons before growth; The grown neural network model is used as a classification model to classify patent text data.
2. The method of claim 1, wherein the self-growing model is based on a patent text classification model. In training the model, the growth factor is adjusted as a hyperparameter and determined by grid search, random search or Bayesian optimization to find the optimal value of the growth factor .
3. The method of claim 1, wherein the self-growing model is based on a patent text classification. computing attention weights for the output of the first layer of neural network layers by a self-attention mechanism, wherein the attention weights are computed by a dot-product attention, an additive attention, or a scaled dot-product attention a score of a correlation between the outputs of the first layer of neural network layers.
4. The method of claim 1, wherein the self-growing model is based on a patent text classification. The first weight matrix and the second weight matrix of the skip connection are set in each layer of the neural network to allow skip connections between different layers of the neural network, so that the output of the lower layer of the neural network can be directly passed to the higher layer of the neural network.
5. A patent text classification system based on a self-growing model, characterized by, include: The data processing module is used to acquire patent text data and preprocess it, removing stop words and punctuation marks from the patent text data, and converting the preprocessed patent text data into a text tag sequence through a word segmenter; The growth module is used to take the text tag sequence as input, input it into the neural network model, perform a model self-growth operation on the neural network model, and generate a grown neural network model. The self-growth operation on the neural network model includes: defining a growth operator associated with each growth dimension of the neural network model, and expanding the growth dimension of the neural network model through the growth operator; Based on the determined growth dimension, a corresponding number of neurons are added to the neural network model; The growth operators for the number of layers in the neural network model include: in, For the first The output of a layer in a neural network. As a growth factor, For the first The weight matrix of a layer in a neural network. It is a non-linear loss function. For the first The output of a layer in a neural network. For the first Bias terms of the layers in a neural network The weights of the attention mechanism for each layer of the neural network. For the first Attention weights in the output of a layer in a neural network; Growth operators for hidden layers include: in, This is the output of one layer of the grown neural network. For the mask, These are the connection weights of the next layer in the grown neural network. This is the input matrix of the first layer of the neural network before growth. The first weight matrix is used for jump connections. The second weight matrix is used for jump connections. It is a non-linear activation function. For bias terms, For the jump connection bias term, These are hyperparameters used to control the degree to which attention weights affect skip connections. This is the output of a neural network layer before growth. It is a vector of all zeros. This represents the number of output neurons after growth. This represents the number of output neurons before growth; The classification module is used to classify the patent text data by using the grown neural network model as a classification model.
6. An electronic device, comprising: It includes a processor and a memory, the memory storing multiple instructions, and the processor being used to read the instructions and execute a patent text classification method based on a self-growing model as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Text classification model construction method and device, terminal and storage medium
CN109960726A
Model self-growth training acceleration method and device, electronic equipment and storage medium
CN116306796A
