Patent label information generation method and system based on transfer learning

Abstract

The invention discloses a patent label information generation method and system based on transfer learning, the method comprises five steps of task configuration, model design, corpus construction, model training and execution prediction, the task configuration selects a patent prediction task, division is carried out based on all patent texts according to the technical field in an IPC classification system, and the patent prediction task is obtained; selecting a proper source domain and a target domain according to the patent data; according to the model design, a feature coding model and a prediction model are respectively designed according to task configuration information; corpus construction: respectively preparing training corpus data on a source domain and a target domain; model training is carried out on the selected source domain and target domain by using corresponding training corpus data; and executing prediction, applying a model interface after model training to predict unlabeled patents on the target domain, generating prediction label information and storing the prediction label information. According to the method, the economic cost and the time cost of a large amount of annotated data are effectively reduced under the condition that the accuracy of an existing model is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of natural language processing and deep learning, and in particular to a method and system for generating patent label information based on transfer learning. Background Art

[0002] The technology of natural language processing in patent text processing is becoming more and more mature, which has greatly promoted the realization of intelligent and automated patent review, application and analysis. There are similarities between various problems in different fields recorded in patent documents. Patents in different fields have both basic features common to the fields and specific features in the patent fields. Existing methods for generating patent label information are generally implemented based on patent features combined with rules or models. Rule-based methods have the problems of poor flexibility and scalability, and model-based methods are generally implemented in a supervised manner, requiring manual annotation of a large amount of patent data. Therefore, how to target the specific text characteristics of multiple patent fields, integrate natural language algorithms based on deep learning with transfer learning capabilities, and face the contradiction between the poor unsupervised prediction effect of patent value assessment problems and the demand for labeled data of supervised models. The application of semi-supervised multi-target field migration technology to achieve accurate and efficient patent prediction has become a field problem that needs to be solved urgently.

[0003] Existing methods for generating patent label information are generally based on patent features combined with rules or models. Rule-based methods have the problems of poor flexibility and scalability, and model-based methods are generally implemented in a supervised manner, which requires manual annotation of a large amount of patent data. Since patent texts contain rich detailed descriptions of legal, technical, and economic dimensions that are missing from the bibliographic items, supervised models require a large amount of labeled data from various fields for training. Patents cover many fields, and patent text annotation requires a lot of time for field experts to complete. Manual annotation is time-consuming and laborious, resulting in high economic and time costs.

[0004] At present, common prediction models are generally designed for general fields. When applied to the patent field, due to the differences in professional terms, application fields, and pragmatic scenarios, professional texts represented by patents contain a large amount of literal, grammatical and semantic information, such as professional vocabulary, semantic similarity, and text structure relationships. At the same time, the diversity of the patent field causes large differences in the distribution of text features, resulting in poor multi-field adaptability of general prediction models. At present, there is still a lack of effective patent prediction models. Based on the IPC classification system, patents are divided into multi-level and multi-category professional fields. Technological development is accompanied by the evolution of IPC standards, which has led to the disappearance of some original IPC classification numbers, the emergence of new IPC classifications, and changes in standard definitions and judgment rules of the same IPC classification in different periods. Data will migrate in the time dimension, resulting in the original fixed classification test algorithms and models being unable to adapt to this multi-target field migration application scenario. Summary of the invention

[0005] The present application provides a patent label information generation method and system based on transfer learning, constructs source domain and target domain training corpus based on patent text, designs a patent prediction model system based on deep learning, effectively solves the problems of low accuracy of existing unsupervised models and lack of labeled data for supervised models, and realizes accurate and efficient multi-field patent prediction.

[0006] In the first aspect, a method for generating patent label information based on transfer learning is provided, the method comprising five steps: task configuration, model design, corpus construction, model training, and execution prediction, which specifically include:

[0007] Task configuration: select the patent prediction task, based on the entire patent text, divide it into technical fields according to the IPC classification system, and select appropriate source and target domains based on patent data;

[0008] Model design: design a feature encoding model and a prediction model according to the task configuration information; the feature encoding model includes a baseline encoding model and a momentum encoding model. The baseline encoding model is used to encode labeled patents and corresponding citation patents in the source domain, and the momentum encoding model is used to keep synchronization with the baseline encoding model through periodic updates; the prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding model are input into the classification prediction model, the prototype prediction model, and the covariance prediction model respectively. The classification prediction model is used to realize the classification prediction of patents, the prototype prediction model is used to predict the representative type of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain;

[0009] Corpus construction: prepare training corpus data in the source domain and target domain respectively; the source domain corpus is constructed by labeled patents and corresponding patent citation data, and the target domain corpus is constructed by a small number of labeled patents and a large number of unlabeled original patent texts;

[0010] Model training: For the selected source domain and target domain, use the corresponding training corpus data to perform model training;

[0011] Execute prediction, apply the model interface after model training, predict the unlabeled patents in the target domain, generate and store the predicted label information; wherein the predicted label information at least includes the IPC classification number, technical field and legal status of the patent.

[0012] Optionally, in the task configuration, appropriate source domain and target domain are selected according to the patent data, including:

[0013] Based on all patent texts, patents are divided according to the technical fields in the IPC classification system. Fields with high-value patent label data are selected as source domains, and fields with little or no label data to which the knowledge to be extracted belongs are selected as target domains. Among them, the fields are based on the IPC classification and can take the third-level subclasses. Patent labels select high-value, high-disruptive, core-critical, green and low-carbon, key digital, and emerging industry task goals. Fields are limited to tobacco, communications, chips, or new energy fields according to the task situation. Patent data selects the abstracts or claims in the five patent books.

[0014] Optionally, in the model design, the encoding model designs the encoding model structure according to the configuration information of the source domain and the target domain, and selects a BERT model based on a multi-layer Transformer Encoder structure for implementation. The BERT model is used as an encoding layer to implement patent text encoding, and the CLS position vector of the output layer is taken as a feature encoding vector;

[0015] The benchmark encoding model is built based on the encoding model structure and is used for encoding the embedded feature representation of the citation patents corresponding to the patents in the source domain and the target domain. During initialization, the benchmark encoder is trained by a labeled dataset on the source domain.

[0016] The momentum coding model is built based on the coding model structure and is used for encoding the embedded feature representation of patents in the source domain and the target domain. When the model training is initialized, the momentum encoder is initialized using the benchmark encoder. During the model training process, the momentum coding model parameters are updated using the momentum coding model parameter update algorithm.

[0017] Optionally, in the model design, the prediction model is constructed using a two-layer fully connected neural network structure, the input is the feature vector output by the encoding model, and the output is the binary high-value prediction probability; the feature vector receiving the encoding model input is n-dimensional, and the first hidden layer contains h 1 neurons, using ReLU as the activation function, and the second hidden layer contains h 2 The neurons also use ReLU as the activation function, mainly to increase nonlinearity and accelerate training. The output layer contains 1 neuron, using the sigmoid function to limit the output to between 0 and 1 to represent probability; among them, when the input dimension is 100, h 1 and h 2 The distribution is 128 and 64.

[0018] Optionally, in the model design, the classification prediction model adopts a prediction model structure, the input is a feature vector output by the encoding model, and the output is a binary classification high-value prediction probability;

[0019] The representative type prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the embedded feature vector of the representative type of high-value patents in this domain;

[0020] The covariance prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the representative covariance matrix.

[0021] Optionally, the model training, for the selected source domain and target domain, uses corresponding training corpus data to perform model training, and further includes:

[0022] Select the corresponding optimizer and configure the relevant training hyperparameters. Then train the encoder model and the predictor model, use the back propagation algorithm to optimize the model parameters, and save the trained model parameters after the training is completed.

[0023] In the second aspect, a patent label information generation system based on transfer learning, the system includes a task configuration module, a model design module, a corpus construction module, a model training module, and an execution prediction module, which specifically includes:

[0024] The task configuration module is used to select patent prediction tasks, based on the entire patent text, divided according to the technical fields in the IPC classification system, and select appropriate source and target domains based on patent data;

[0025] A model design module is used to design a feature encoding model and a prediction model according to the task configuration information; wherein the feature encoding model includes a baseline encoding model and a momentum encoding model, the baseline encoding model is used to encode labeled patents and corresponding citation patents in the source domain, and the momentum encoding model is used to keep synchronization with the baseline encoding model through periodic updates; the prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model, and the feature vectors generated by the feature encoding model are respectively input into the classification prediction model, the prototype prediction model, and the covariance prediction model, the classification prediction model is used to realize the classification prediction of patents, the prototype prediction model is used to predict the representative type of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain;

[0026] The corpus construction module is used to prepare training corpus data in the source domain and the target domain respectively; the source domain corpus is constructed by labeled patents and corresponding patent citation data, and the target domain corpus is constructed by a small number of labeled patents and a large number of unlabeled original patent texts;

[0027] The model training module is used to train the model using the corresponding training corpus data for the selected source domain and target domain;

[0028] The execution prediction module is used to apply the model interface after model training to predict the unlabeled patents in the target domain, generate and store the predicted label information; wherein the predicted label information at least includes the IPC classification number, technical field and legal status of the patent.

[0029] In a third aspect, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for generating patent label information based on transfer learning as described in any one of the first aspects above is implemented.

[0030] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for generating patent label information based on transfer learning as described in any one of the first aspects above is implemented.

[0031] In a fifth aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the patent label information generation method based on transfer learning as described in any one of the first aspects above.

[0032] The present invention provides a patent label information generation method based on semi-supervised domain transfer learning. Based on the labeled patent data in the source domain and a small amount of labeled and a large amount of unlabeled patent data in the target domain, a coding model including a general benchmark coding model and a momentum coding model is designed, and a classification prediction model, a representative prediction model, and a covariance prediction model are combined to realize patent prediction of cross-domain migration. The coding model is trained based on contrastive learning in the embedding space shared by the source and target domains, and the method of generating pseudo labels for the unlabeled data in the target domain supplements the deficiency of a large amount of unlabeled data in the target domain. The model gives full play to the feature characterization ability of the representative type for the IPC domain, combines the domain dependence measurement of the feature vector characterized by the covariance model, corrects the difference in the distribution of patent data in different IPC fields, and causes the deviation of the prediction results, effectively solves the problem of low accuracy of the existing unsupervised model and lack of labeled data of the supervised model, and realizes accurate and efficient multi-domain patent prediction. The method only requires labeled data in the source domain combined with a small amount of labeled data in the target domain, and does not require labeled data in many target domains. While ensuring the accuracy of the existing model, it effectively reduces the economic cost and time cost of a large amount of labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0034] Figure 1 The main flow chart provided for the embodiment of the present application;

[0035] Figure 2 A domain migration framework provided for embodiments of the present application;

[0036] Figure 3 The overall structure diagram of the model provided for the embodiment of the present application;

[0037] Figure 4 A schematic diagram of the labeled and unlabeled sample space provided in an embodiment of the present application;

[0038] Figure 5 Input and output flow chart provided for the embodiment of the present application;

[0039] Figure 6 The overall structure diagram provided for the embodiment of the present application;

[0040] Figure 7 A block diagram of the module architecture of a patent label information generation system based on transfer learning provided in one embodiment of the present application;

[0041] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] In the description of the present application: the terms "including", "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units that have been explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.

[0044] In view of the problem of accurate and efficient prediction of patents in multiple fields, the present invention provides a method for generating patent label information based on semi-supervised domain transfer learning, constructs source domain and target domain training corpus based on patent text, and designs a patent prediction model system based on deep learning. Among them, the source domain is labeled domain patent text data, and the target domain is a small amount of labeled and a large amount of unlabeled domain patent text data. The model system includes a feature encoding model and a prediction model. The encoding model is to embed feature vector encoding of the text, including a general benchmark encoding model and a momentum encoding model. Among them, the benchmark encoding model is used for feature encoding of target domain patent text, and the momentum encoding model is used for feature encoding of source domain patent text. The encoding model is trained based on contrastive learning in the embedding space shared by the source and target domains, and then generates pseudo labels for unlabeled data in the target domain. The prediction model includes a classification prediction model, a representative prediction model, and a covariance prediction model. The label prediction model realizes patent label prediction based on the encoding feature. The representative prediction model realizes the prediction of the domain representative type, and the covariance prediction model realizes the prediction of the covariance matrix, which is used to effectively alleviate the deviation of the prediction results caused by the difference in the distribution of patent data in different IPC fields. Among them, the labeled patent prediction model in the source domain is trained in a supervised manner, and the unlabeled patent prediction model in the target domain is trained in a semi-supervised manner by generating pseudo labels. All the above models are integrated to form a complete model system for end-to-end training and reasoning on the data sets of the source domain and the target domain.

[0045] In one embodiment, Figure 1As shown, a patent label information generation method based on transfer learning is provided, which can be applied to the server. The method includes five steps: task configuration, model design, corpus construction, model training, and execution prediction, which specifically include:

[0046] Task configuration requires first selecting the patent prediction task, then dividing the entire patent text into fields corresponding to the IPC classification, and selecting appropriate source and target domains based on the patent data. The target domain includes a small number of labeled patents and a large number of unlabeled original patent texts; save and generate the above-mentioned field and task configuration information as input for subsequent model training;

[0047] The model design is based on the above-mentioned domain and task configuration information, and the model architecture of the source task and the target task are designed respectively, including the feature encoding model and the prediction model. Among them, the feature encoding model includes the baseline encoding model and the momentum encoding model, which are used to encode the labeled high-value patents and the corresponding citation patents in the source domain respectively. The momentum encoding model is synchronized with the baseline encoding model through periodic updates. The prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the baseline feature encoding model are input into the classification prediction model, the prototype prediction model, and the covariance prediction model respectively. The classification prediction model realizes the classification prediction of patents, the prototype prediction model is used to predict the representative type of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain; the unlabeled patents in the target domain generate pseudo labels through the representative type kNN method. The overall loss function of the model is designed, including patent prediction loss, representative type prediction loss, feature contrast loss, and covariance prediction loss.

[0048] Corpus construction is to prepare training data in the source domain and target domain respectively. The source domain corpus is constructed by labeled patents and corresponding patent citation data, and the target domain corpus is constructed by a small number of labeled patents and a large number of unlabeled original patent texts.

[0049] Model training is to select the optimizer, set the training hyperparameters, train the encoding model and prediction model for the training corpus data in the selected source domain and target domain, update the model parameters by reverse gradient propagation, and save the model parameters after training is completed;

[0050] Model application is to apply the above-mentioned trained model interface to predict the unlabeled patents in the target domain, generate and store the predicted label information.

[0051] Among them, the predicted label information may include IPC classification number, technical field: the application field of the patent can also be described more specifically, such as specific industrial applications, product types, etc.

[0052] Patent legal status: The prediction label may include the legal status of the patent, such as whether it is valid, under review, or expired.

[0053] Patent technology effects: Prediction labels may describe the actual effects of patented technologies, such as improved efficiency, reduced costs, enhanced performance, etc.

[0054] The task configuration determines and selects appropriate source and target domains based on the existing high-value patent data.

[0055] The source domain and target domain selection are based on the entire patent text, and the patents are divided according to the technical fields in the IPC classification system. The fields with high-value patent label data are selected as the source domain, and the fields with little or no labeled data to which the knowledge to be extracted belongs are selected as the target domain.

[0056] The fields here are based on IPC classification, and can be subcategories (third level). Patent labels can be selected as high value, high disruptive, core key, green and low carbon, key digital, emerging industries and other task targets, and fields can be limited to tobacco, communications, chips or new energy fields according to the task situation. Patent data can be selected from the abstract or claims in the five patent books.

[0057] An embodiment:

[0058] The present invention selects subcategory A24F (smokers' supplies) in the IPC system as the source domain, and subcategory C11B (production, such as by squeezing raw materials or extracting from waste materials, refining or preserving fats, fatty substances such as lanolin, fats or waxes; essential oils; spices) as the target domain. The patents in the source domain are data with high-value labels, a small number of patents in the target domain have high-value labels, and a large number of patents are unlabeled. If all patents in the target domain are unlabeled data, some small number of high-value labels can be supplemented through manual assistance. Figure 4 An example diagram shows the target and source domains of the labeled and unlabeled sample spaces.

[0059] The model design is based on the above-mentioned domain and task configuration information, and designs encoding models and prediction models respectively, including feature encoding models and classification models. Among them, the feature encoding model includes a benchmark encoding model and a momentum encoding model, which are respectively used to encode labeled patents and corresponding cited patents in the source domain. The momentum encoding model keeps synchronization with the benchmark encoding model through periodic updates. The classification model includes a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding model are input into the classification prediction model, the prototype prediction model, and the covariance prediction model respectively. The classification prediction model realizes the classification prediction of patents. The prototype prediction model is used to predict the representative type of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain; the unlabeled patents in the target domain generate pseudo labels through the representative kNN method. The overall structure of the model is designed and the loss function of the optimization target is optimized, including classification prediction loss, representative type prediction loss, contrast loss, covariance prediction loss, etc. Figure 5 It represents the feature vector obtained by inputting the citation patents corresponding to the patents in the source domain and the target domain, which is the input of the classification prediction model, the representative prediction model and the covariance prediction model to obtain the output of each model. Figure 6 The overall structure diagram of the algorithm is shown.

[0060] Encoding model: The encoding model structure is designed according to the configuration information of the source domain and the target domain. The BERT model based on the multi-layer Transformer Encoder structure is selected for implementation. The BERT model is used as the encoding layer to implement patent text encoding, and the [CLS] position vector of the output layer is taken as the feature encoding vector.

[0061] Baseline Encoding Model: Based on the encoding model structure, it is used to encode the embedding features of the citation patents corresponding to the patents in the source and target domains. During initialization, the benchmark encoder F is trained on a labeled dataset on the source domain.

[0062] Momentum encoding model: Based on the encoding model structure, it is used for encoding the embedded feature representation of patents in the source domain and the target domain. When the model training is initialized, the momentum encoder F' is initialized using the reference encoder F. During the model training process, the momentum encoding model parameters are updated using the following algorithm:

[0063] Algorithm 1: Momentum coding model parameter update algorithm:

[0064] Input: Initialize model parameters , iterative steps K, step length ,momentum , stability parameter , weight decay , restart condition, parameters of the reference encoder F at the kth step and the k-1th step , the momentum encoder of the k-1th step in the previous step Parameters Objective loss function .

[0065] Output: The current step of the momentum encoder F` on the kth step .

[0066] Initialization: Initialization is performed when it is first executed .

[0067] calculate:

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] return , Note: Here, β 1 =0.9,β 2 =0.999, β 3 =0.9999, =0.001, =10 -8 , K=1000.

[0075] Prediction model: It is constructed with a two-layer fully connected neural network structure. The input is the feature vector output by the encoding model, and the output is the binary high-value prediction probability. The feature vector receiving the encoding model input is n-dimensional, and the first hidden layer contains h 1 neurons, using ReLU as the activation function, and the second hidden layer contains h 2 The neurons also use ReLU as the activation function, mainly to increase nonlinearity and accelerate training. The output layer contains 1 neuron, using the sigmoid function, and the output is limited to 0-1 to represent probability; when the input dimension is 100, h 1 and h 2 The distribution can be 128 and 64, specifically z 1 =W 1 x+b 1 , a1 =ReLU(z 1 ), z 2 =W 2 x+b 2 , a 2 =ReLU(z 2 ), z 3 =W 3 x+b 3 , y=σ(z 3 ), where W 1 ,W 2 ,W 3 is the weight matrix, b 1 ,b 2 ,b 3 is the bias vector, σ is the sigmoid function , x is the input feature vector.

[0076] The classification prediction model (Classification) adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the binary high-value prediction probability.

[0077] The representative prediction model (Prototype) adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the embedded feature vector of the representative type of high-value patents in this domain.

[0078] The representative type is a P-dimensional feature vector , calculated by the average value of the eigenvectors in the class set .

[0079] in, is the embedding feature encoding function, which is implemented as an encoding model, where are the learnable parameters of the model.

[0080] The working principle of the representative model is that given a distance function , the representative prediction model outputs the softmax distribution of the query point x in this class at the distance d in the embedded space:

[0081]

[0082] The model parameters are learned by gradient descent on class k to minimize the negative log probability:

[0083] accomplish.

[0084] The covariance prediction model (Covariance) adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the representative covariance matrix.

[0085] The metric embedded in the feature space vector is calculated using the Bregman divergence:

[0086]

[0087] in, For a differentiable strictly Legendre-type convex function, the Bregman divergence of the squared Mahalanobis distance is used here:

[0088] when hour, .

[0089] In order to simplify the calculation and improve the execution speed, Q can be implemented as a unit matrix, and the Euclidean square distance is obtained:

[0090] at this time , .

[0091] The covariance matrix here is a Gaussian covariance matrix, which automatically learns to construct a representative type based on the distance metric and direction in the category-specific feature vector space. Among them, the cluster center is the center point of the representative type, and the Gaussian covariance matrix represents the confidence (uncertainty) in a specific direction, and has anisotropic scaling of the feature space. Here, Q==S -1 .

[0092] Prediction output covariance matrix of the representative network , where D is the dimension of the embedding space feature vector.

[0093] The distance between feature vectors is calculated as:

[0094]

[0095] in, is the center point, i.e., the representative type of class c. M is the covariance matrix. The representative type model combined with the embedding feature vector space distance represented by the covariance model can learn the class and direction-dependent distance measure of the embedding space. In practical applications, in order to improve the model training and inference speed, the covariance matrix can be further simplified to a diagonal matrix.

[0096] Model system: The dynamic mechanism of the model system includes the synchronous update mode of the momentum coding model and the benchmark coding model, the representative feature vector calculation and update mode, the covariance matrix calculation and update mode, the loss function calculation, and the model parameter update.

[0097] The dynamic mechanism of the semi-supervised embedding space domain adaptation model is shown in Algorithm 2:

[0098] Algorithm 2: Dynamic mechanism of semi-supervised embedding space domain adaptation model:

[0099] Input: Source domain dataset D s , there is a labeled target dataset D t ,,Unlabeled target dataset D u .

[0100] Initialization: Initialize the momentum encoder F` using the baseline encoder F trained on the labeled dataset and initialize the database Q = {}.

[0101] Repeat the following steps until the model training is completed:

[0102] 1. From the source domain dataset D s Samples with corresponding citation patents (x s , y s )∈D s , from the labeled dataset D of the target domain t Samples with corresponding citation patents (x t , y t )∈D t , from the unlabeled dataset D of the target domain u Sample x of patents with corresponding citations u ∈D u .

[0103] 2.Search x from the patent database s , x t , x u Corresponding citation patent data .

[0104] 3. Apply momentum encoder F` to x s , x t , x u Encode and get the embedded feature vector , using the reference encoder F Encode and get the embedded feature vector , apply the representative prediction model P to obtain the representative embedding feature vector , apply the covariance prediction model M to obtain the covariance matrix of the current instance . Update all the above generated eigenvectors, representative vectors, and covariance matrices into the database Q.

[0105] 4. For unlabeled instances Generate pseudo labels: If , then generate pseudo labels Among them, f is respectively for Pick and , the distance in kNN uses the covariance corrected distance formula calculate, is the representative vector of class c, M is the covariance matrix, Embed the feature vector for the candidates in Q. Update all the pseudo labels generated above into the database Q.

[0106] 5. Calculate model loss and update model parameters through back propagation.

[0107] 5.1. Calculating patent data with labeled citations Cross entropy loss of labels :Apply high-value prediction model C and input label citation patent data Embedding vector features Get predicted label information .

[0108] Through the cross entropy loss function Calculating cross entropy loss .in, are the true label and predicted label information respectively, The citation patent data in the source domain and the target domain are .

[0109] 5.2. Calculating unlabeled citation patent data Cross entropy loss of generated pseudo labels , apply high-value prediction model C, input unlabeled citation patent data Embedding vector features Get predicted label information , through the cross entropy loss function Calculating cross entropy loss .in, are predicted label information and pseudo labels respectively. Unlabeled citation patent data of the target domain Generated pseudo labels.

[0110] 5.3. Application formula in, = 0.9 and Formula 6 Calculate the unlabeled samples x on the target domain in the vector library Q u And the corresponding citation data Eigenvector and Contrastive loss:

[0111] .

[0112] 5.4 Use a gradient reversal layer before the classifier layer to calculate the gradient and use Update the parameters of the reference encoder F and initialize the momentum encoder F', where represents the regularization coefficient, its value determines the weight of the regularization term in the total loss function, and H indicates that the prototype is updated based on the entropy of the unlabeled target prediction.

[0113] 5.5 Use a gradient reversal layer before the classifier layer to calculate the gradient and use the high value to predict the parameters of model C , among which represents the regularization coefficient, its value determines the weight of the regularization term in the total loss function, and H indicates that the prototype is updated based on the entropy of the unlabeled target prediction.

[0114] 5.6. Application formula Update the representative vectors of all categories c in all Q , apply the formula Update the representative vector of category c predicted by the representative model ,in, is the set of all embedded feature vectors belonging to category c in Q, is the set of representative vectors of all categories c in Q, for The number of representative morphological vectors in M ​​is the covariance matrix. Calculate the Euclidean distance loss of representative morphological prediction: .

[0115] 5.7. Applying the covariance formula Compute the covariance matrix of all eigenvectors in Q ,in represents the i-th eigenvector in Q, is the expected mean of all eigenvectors in Q, and N is the total number of eigenvectors in Q. Apply the formula Update the covariance matrix predicted by the covariance model, where m is the set of all covariance matrices in Q The covariance matrix predicted by the covariance model in , for The total number of covariance matrices in . Calculate the covariance matrix prediction loss: ,in Represent the elements in the i-th row and j-th column of the covariance matrix respectively.

[0116] Sum all the loss functions in steps 5.1 to 5.7 above: , and update them respectively using the back-propagation gradient descent algorithm:

[0117] Parameters of the reference encoder , parameters of high value prediction model , parameters of representative prediction models , parameters of the covariance prediction model .

[0118] 6. Apply Algorithm 1 to update the parameters of the momentum encoder model .

[0119] 7. Update all the above generated feature vectors, labels, representative vectors, and covariance matrices to the database Q. Return all model parameters. Corpus construction is to prepare training data in the source domain and target domain respectively. The source domain corpus is constructed by labeled patents and corresponding patent citation data, and the target domain corpus is constructed by a small number of labeled patents and a large number of unlabeled original patent texts;

[0120] The model training process involves the following steps: First, for the selected source domain and target domain, use the corresponding training corpus data to train the model. Next, select a suitable optimizer and configure the relevant training hyperparameters. Then, train the encoder model and predictor model, and use the backpropagation algorithm to optimize the model parameters. After the training is completed, save the trained model parameters.

[0121] Model application is to apply the above-mentioned trained model interface to predict the unlabeled patents in the target domain, generate and store the predicted label information.

[0122] From the above, it can be seen that the present invention provides a patent multi-domain knowledge extraction method based on feature alignment, divides the fields based on the patent IPC classification, selects the source field, the target field and its corresponding labeled and unlabeled data combination in a targeted manner, designs a coding model including a general benchmark coding model and a momentum coding model, and combines the classification prediction model, the representative prediction model and the covariance prediction model to realize the patent prediction of cross-domain migration. The coding model and the prediction model comprehensively adopt cross entropy, contrast loss, representative type, and covariance loss functions in the embedding space shared by the source and target domains, and automatically generates pseudo labels for a large amount of unlabeled data in the target domain, converting the original unsupervised training method into a supervised training mode, and forming a complete model system after integration, which realizes efficient end-to-end training and reasoning on the data sets of the source and target domains.

[0123] In one embodiment, Figure 7 As shown, a patent label information generation system based on transfer learning is provided, the system includes a task configuration module, a model design module, a corpus construction module, a model training module, and an execution prediction module, which specifically include:

[0124] The task configuration module is used to select patent prediction tasks, based on the entire patent text, divided according to the technical fields in the IPC classification system, and select appropriate source and target domains based on patent data;

[0125] A model design module is used to design a feature encoding model and a prediction model according to the task configuration information; wherein the feature encoding model includes a baseline encoding model and a momentum encoding model, the baseline encoding model is used to encode labeled patents and corresponding citation patents in the source domain, and the momentum encoding model is used to keep synchronization with the baseline encoding model through periodic updates; the prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model, and the feature vectors generated by the feature encoding model are respectively input into the classification prediction model, the prototype prediction model, and the covariance prediction model, the classification prediction model is used to realize the classification prediction of patents, the prototype prediction model is used to predict the representative type of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain;

[0126] The corpus construction module is used to prepare training corpus data in the source domain and the target domain respectively; the source domain corpus is constructed by labeled patents and corresponding patent citation data, and the target domain corpus is constructed by a small number of labeled patents and a large number of unlabeled original patent texts;

[0127] The model training module is used to train the model using the corresponding training corpus data for the selected source domain and target domain;

[0128] The execution prediction module is used to apply the model interface after model training, predict the unlabeled patents on the target domain, generate and store the predicted label information.

[0129] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities, and the network interface is used to communicate with an external terminal through a network connection. The computer device implements the above-mentioned patent label information generation method by loading and running a computer program.

[0130] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which involves all or part of the processes in the above-mentioned embodiment method.

[0132] In one embodiment, a computer program product is also provided, including a computer program / instruction, which involves all or part of the process in the above embodiment method.

[0133] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A patent label information generation method based on transfer learning, characterized in that: The method includes five steps: task configuration, model design, corpus construction, model training, and execution prediction, which specifically include: Task configuration: select the patent prediction task, based on the entire patent text, divide it into technical fields according to the IPC classification system, and select appropriate source and target domains based on patent data; Model design: design a feature encoding model and a prediction model according to the task configuration information; the feature encoding model includes a baseline encoding model and a momentum encoding model. The baseline encoding model is used to encode labeled patents and corresponding citation patents in the source domain, and the momentum encoding model is used to keep synchronization with the baseline encoding model through periodic updates; the prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding model are input into the classification prediction model, the prototype prediction model, and the covariance prediction model respectively. The classification prediction model is used to realize the classification prediction of patents, the prototype prediction model is used to predict the representative type of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain; Corpus construction: prepare training corpus data in the source domain and target domain respectively; the source domain corpus is constructed by labeled patents and corresponding patent citation data, and the target domain corpus is constructed by a small number of labeled patents and a large number of unlabeled original patent texts; Model training: For the selected source domain and target domain, use the corresponding training corpus data to perform model training; Execute prediction, apply the model interface after model training, predict the unlabeled patents in the target domain, generate and store the predicted label information; wherein the predicted label information at least includes the IPC classification number, technical field and legal status of the patent.

2. The method according to claim 1, characterized in that In task configuration, appropriate source and target domains are selected based on patent data, including: Based on all patent texts, patents are divided according to the technical fields in the IPC classification system. Fields with high-value patent label data are selected as source domains, and fields with little or no label data to which the knowledge to be extracted belongs are selected as target domains. Among them, the fields are based on the IPC classification and can take the third-level subclasses. Patent labels select high-value, high-disruptive, core-critical, green and low-carbon, key digital, and emerging industry task goals. Fields are limited to tobacco, communications, chips, or new energy fields according to the task situation. Patent data selects the abstracts or claims in the five patent books.

3. The method according to claim 1, characterized in that In the model design, the encoding model is designed according to the configuration information of the source domain and the target domain. The BERT model based on the multi-layer Transformer Encoder structure is selected for implementation. The BERT model is used as the encoding layer to implement patent text encoding, and the CLS position vector of the output layer is taken as the feature encoding vector. The baseline encoding model is built based on the encoding model structure and is used for encoding the embedded feature representation of the citation patents corresponding to the patents in the source and target domains; During initialization, the baseline encoder is trained by training a labeled dataset on the source domain; The momentum coding model is built based on the coding model structure and is used for encoding the embedded feature representation of patents in the source and target domains; When the model training is initialized, the benchmark encoder is used to initialize the momentum encoder. During the model training process, the momentum encoding model parameters are updated using the momentum encoding model parameter update algorithm.

4. The method according to claim 1, characterized in that: In the model design, the prediction model is constructed with a two-layer fully connected neural network structure. The input is the feature vector output by the encoding model, and the output is the binary high-value prediction probability. The feature vector receiving the encoding model input is n-dimensional. The first hidden layer contains h1 neurons and uses ReLU as the activation function. The second hidden layer contains h2 neurons and also uses ReLU as the activation function, mainly to increase nonlinearity and accelerate training. The output layer contains 1 neuron and uses the sigmoid function to limit the output to between 0 and 1 to represent probability. Among them, when the input dimension is 100, h1 and h2 are distributed as 128 and 64.

5. The method according to claim 1, characterized in that In the model design, the classification prediction model adopts the prediction model structure, the input is the feature vector of the encoding model output, and the output is the binary classification high-value prediction probability; The representative type prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the embedded feature vector of the representative type of high-value patents in this domain; The covariance prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the representative covariance matrix.

6. The method according to claim 1, characterized in that Model training, for the selected source domain and target domain, uses the corresponding training corpus data to perform model training, and also includes: Select the corresponding optimizer and configure the relevant training hyperparameters. Then train the encoder model and the predictor model, use the back propagation algorithm to optimize the model parameters, and save the trained model parameters after the training is completed.

7. A patent label information generation system based on transfer learning, characterized in that: The system includes a task configuration module, a model design module, a corpus construction module, a model training module, and an execution prediction module, which specifically include: The task configuration module is used to select patent prediction tasks, based on the entire patent text, divided according to the technical fields in the IPC classification system, and select appropriate source and target domains based on patent data; A model design module is used to design a feature encoding model and a prediction model according to the task configuration information; wherein the feature encoding model includes a baseline encoding model and a momentum encoding model, the baseline encoding model is used to encode labeled patents and corresponding citation patents in the source domain, and the momentum encoding model is used to keep synchronization with the baseline encoding model through periodic updates; the prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model, and the feature vectors generated by the feature encoding model are respectively input into the classification prediction model, the prototype prediction model, and the covariance prediction model, the classification prediction model is used to realize the classification prediction of patents, the prototype prediction model is used to predict the representative type of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain; The corpus construction module is used to prepare training corpus data in the source domain and the target domain respectively; the source domain corpus is constructed by labeled patents and corresponding patent citation data, and the target domain corpus is constructed by a small number of labeled patents and a large number of unlabeled original patent texts; The model training module is used to train the model using the corresponding training corpus data for the selected source domain and target domain; The execution prediction module is used to apply the model interface after model training to predict the unlabeled patents in the target domain, generate and store the predicted label information; wherein the predicted label information at least includes the IPC classification number, technical field and legal status of the patent.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images