A method and system for generating patent tag information based on transfer learning

Abstract

This application discloses a method and system for generating patent tag information based on transfer learning. The method includes five steps: task configuration, model design, corpus construction, model training, and prediction. Task configuration involves selecting a patent prediction task, using all patent texts as a basis, dividing them according to the technical fields in the IPC classification system, and selecting appropriate source and target domains based on the patent data. Model design involves designing a feature encoding model and a prediction model based on the task configuration information. Corpus construction involves preparing training corpus data for both the source and target domains. Model training involves training the selected source and target domains using the corresponding training corpus data. Prediction involves applying the trained model interface to predict unlabeled patents in the target domain, generating and storing predicted tag information. This method effectively reduces the economic and time costs of generating large amounts of labeled data while maintaining the accuracy of existing models.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the fields of natural language processing and deep learning, and in particular to a method and system for generating patent tag information based on transfer learning. Background Technology

[0002] Natural language processing (NLP) technology has matured significantly in patent text processing, greatly promoting intelligent and automated patent examination, application, and analysis. Patent documents record similarities across various fields, and patents in different fields possess both common fundamental features and field-specific characteristics. Existing methods for generating patent tag information generally combine patent features with rules or models. Rule-based methods suffer from poor flexibility and scalability, while model-based methods are typically implemented in a supervised manner, requiring manual annotation of large amounts of patent data. Therefore, how to address the contradiction between the poor performance of unsupervised prediction in patent valuation and the data requirements of supervised models by integrating deep learning-based NLP algorithms with transfer learning capabilities, and applying semi-supervised multi-objective domain transfer techniques to achieve accurate and efficient patent prediction, has become a pressing challenge.

[0003] Existing methods for generating patent label information generally rely on a combination of patent features and rules or models. Rule-based methods suffer from poor flexibility and scalability, while model-based methods are typically implemented in a supervised manner, requiring extensive manual annotation of patent data. Since patent texts contain rich, detailed legal, technical, and economic information missing from bibliographical entries, supervised models require training with large amounts of labeled data across various fields. Given the wide range of fields covered by patents, patent text annotation necessitates significant time investment from domain experts. Manual annotation is both time-consuming and labor-intensive, resulting in high economic and time costs.

[0004] Currently, common prediction models are generally designed for general domains. When applied to the patent domain, due to differences in professional terminology, application areas, and pragmatic scenarios, specialized texts, represented by patents, contain a large amount of literal, grammatical, and semantic information, such as professional vocabulary, semantic similarity, and textual structure relationships. Furthermore, the diversity of patent domains leads to significant differences in the distribution of textual features, resulting in poor multi-domain adaptability of general prediction models. Currently, there is a lack of effective prediction models specifically for patents. Using the IPC classification system as a benchmark, patents are divided into multi-level and multi-category professional domains. Technological development, along with the evolution of IPC standards, has led to the disappearance of some original IPC classification numbers, the emergence of new IPC classifications, and changes in the standard definitions and judgment rules of the same IPC classification at different times. All of these factors result in data migration across domains over time, making existing fixed-classification testing algorithms and models unable to adapt to this multi-target domain migration application scenario. Summary of the Invention

[0005] This application provides a patent tag information generation method and system based on transfer learning. It constructs source and target domain training corpora based on patent texts and designs a patent prediction model system based on deep learning. It effectively solves the problems of low accuracy of existing unsupervised models and lack of labeled data in supervised models, and achieves accurate and efficient multi-domain patent prediction.

[0006] Firstly, a method for generating patent tag information based on transfer learning, the method comprising five steps: task configuration, model design, corpus construction, model training, and prediction execution, specifically including:

[0007] Task configuration: Select the patent prediction task, based on all patent texts, and divide them according to the technical fields in the IPC classification system. Select the appropriate source domain and target domain based on the patent data.

[0008] The model design involves designing a feature encoding model and a prediction model based on the task configuration information. The feature encoding model includes a baseline encoding model and a momentum encoding model. The baseline encoding model encodes tagged patents and their corresponding citation patents in the source domain, while the momentum encoding model maintains synchronization with the baseline encoding model through periodic updates. The prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding model are input into these three models, respectively. The classification prediction model is used to predict the classification of patents, the prototype prediction model is used to predict the representativeness of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain.

[0009] Corpus construction involves preparing training corpus data for the source and target domains respectively. The source domain corpus is constructed using labeled patents and their corresponding patent citation data, while the target domain corpus is constructed using a small number of labeled patents and a large number of unlabeled original patent texts.

[0010] Model training involves training the model using the corresponding training corpus data for the selected source and target domains.

[0011] The prediction is performed by applying the model interface after model training to predict unlabeled patents in the target domain, generating and storing predicted label information; the predicted label information includes at least the patent's IPC classification number, technical field, and legal status.

[0012] Optionally, the task configuration selects appropriate source and target domains based on the patent data, specifically including:

[0013] Based on all patent texts, the patents are divided into technical fields according to the IPC classification system. Fields with high-value patent tag data are selected as source fields, and fields with little or no tag data to which the knowledge to be extracted belongs are selected as target fields. The fields are based on the IPC classification, and can be subcategories at the third level. Patent tags are selected based on high value, high disruptiveness, core and key, green and low-carbon, key data, and emerging industry task objectives. Fields are limited to tobacco, communications, chips, or new energy fields according to the task situation. Patent data is selected from the abstracts or claims in the five patent books.

[0014] Optionally, in the model design, the encoding model structure is designed according to the configuration information of the source domain and the target domain. The BERT model based on the multi-layer Transformer Encoder structure is selected for implementation. The BERT model is used as the encoding layer to implement the patent text encoding, and the CLS position vector of the output layer is taken as the feature encoding vector.

[0015] The benchmark encoding model is built on the encoding model structure and is used to encode the embedded feature representation of the citation patents corresponding to the patents in the source and target domains. During initialization, the benchmark encoder is trained on a labeled dataset in the source domain.

[0016] The momentum coding model is built on the coding model structure and is used to encode the embedded feature representation of patents in the source and target domains. During model training initialization, the momentum encoder is initialized using the baseline encoder. During model training, the momentum coding model parameters are updated using the momentum coding model parameter update algorithm.

[0017] Optionally, the momentum encoding model parameter update algorithm is implemented through iterative updates, specifically by using the parameters m of the momentum encoder F′ at the (k-1)th step. k-1 vk-1 , n k-1 , θ′ k-1 Update the parameters θ at the k-th and (k-1)-th steps of the baseline encoder F. k θ k-1 :

[0018] m k =β1m k-1 +(1-β1)θ k

[0019] v k =β2v k-1 +(1-β2)(θ k -θ k-1 )

[0020]

[0021] n k =β3n k-1 +(1-β3)[θ k +(1-β2)(θ k -θ k-1 )] 2

[0022]

[0023] Optionally, in the model design, the classification prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the binary classification high-value prediction probability;

[0024] The representative prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the embedded feature vector of the high-value patent representative in this domain.

[0025] The covariance prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being a representative covariance matrix;

[0026] Optionally, the representativeness of the representative prediction model is a P-dimensional feature vector. Calculate by the average of the feature vectors in this class set. The measure embedded in the feature space vector by the covariance prediction model is calculated using Bregman divergence.

[0027]

[0028] The distance between feature vectors is calculated as follows:

[0029]

[0030] Optionally, the semi-supervised embedding spatial domain adaptation model dynamic mechanism applies a momentum encoder and a benchmark encoder to encode patent and its cited patent data on the source domain dataset, labeled target dataset, and unlabeled target dataset, respectively. It then applies a representative prediction model to obtain a representative embedding feature vector, applies a covariance prediction model M to obtain the covariance matrix of the current instance, generates pseudo-labels for unlabeled instances, calculates the model loss, and updates the model parameters through backpropagation.

[0031] Optionally, the model training step further includes regularization of the model, specifically including:

[0032] Add an L2 regularization term to the model's loss function;

[0033] Set the regularization parameter and determine the range of values ​​for the regularization parameter;

[0034] During training, the model weights are constrained by the L2 norm.

[0035] In addition, the forecasting process also includes interpreting the forecast results, specifically including:

[0036] Extract feature importance scores that are relevant to the prediction results;

[0037] Generate a visual chart of feature importance scores;

[0038] Generate an explanatory report that includes feature importance scores and prediction results.

[0039] Secondly, a patent tag information generation system based on transfer learning, comprising a task configuration module, a model design module, a corpus construction module, a model training module, and an execution prediction module, specifically including:

[0040] The task configuration module is used to select patent prediction tasks. Based on all patent texts, the tasks are divided according to the technical fields in the IPC classification system, and appropriate source and target domains are selected based on the patent data.

[0041] The model design module is used to design feature encoding models and prediction models based on task configuration information. The feature encoding models include a baseline encoding model and a momentum encoding model. The baseline encoding model encodes tagged patents and their corresponding citation patents in the source domain, while the momentum encoding model keeps synchronized with the baseline encoding model through periodic updates. The prediction models include a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding models are input into these models, respectively. The classification prediction model is used to predict the classification of patents, the prototype prediction model is used to predict the representativeness of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain.

[0042] The corpus construction module is used to prepare training corpus data for the source domain and the target domain, respectively. The source domain corpus is constructed using labeled patents and corresponding patent citation data, while the target domain corpus is constructed using a small number of labeled patents and a large number of unlabeled original patent texts.

[0043] The model training module is used to train the model using the corresponding training corpus data for the selected source and target domains.

[0044] The execution prediction module is used to apply the model interface after model training to predict unlabeled patents in the target domain, generate and store predicted label information; wherein, the predicted label information includes at least the patent's IPC classification number, technical field and legal status.

[0045] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the patent tag information generation method based on transfer learning as described in any of the first aspects above.

[0046] This invention provides a patent labeling information generation method based on semi-supervised domain transfer learning. Based on labeled patent data in the source domain and a small amount of labeled and a large amount of unlabeled patent data in the target domain, an encoding model including a general baseline encoding model and a momentum encoding model is designed. This model, combined with a classification prediction model, a representativeness prediction model, and a covariance prediction model, achieves cross-domain transfer patent prediction. The encoding model is trained in a shared embedding space between the source and target domains using contrastive learning, generating pseudo-labels for the unlabeled data in the target domain to compensate for the large amount of unlabeled data. The model fully leverages the representativeness model's ability to represent domain-specific features of IPCs, and combines this with the domain dependency measure of feature vectors represented by the covariance model to correct for prediction biases caused by differences in the distribution of patent data across different IPC domains. This effectively solves the problems of low accuracy in existing unsupervised models and lack of labeled data in supervised models, achieving accurate and efficient multi-domain patent prediction. This method only requires labeled data from the source domain combined with a small amount of labeled data from the target domain, eliminating the need for numerous labeled data points in the target domain. While maintaining the accuracy of existing models, it effectively reduces the economic and time costs associated with large amounts of labeled data. Attached Figure Description

[0047] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0048] Figure 1 The main flowchart provided for the embodiments of this application;

[0049] Figure 2 The domain migration framework provided for embodiments of this application;

[0050] Figure 3 This is a diagram illustrating the overall structure of the model provided in the embodiments of this application.

[0051] Figure 4 This is a schematic diagram of the labeled and unlabeled sample space provided in the embodiments of this application;

[0052] Figure 5 The input / output flowchart provided for the embodiments of this application;

[0053] Figure 6 This is an overall structural diagram provided for an embodiment of this application;

[0054] Figure 7 A block diagram of the module architecture of a patent tag information generation system based on transfer learning provided in one embodiment of this application;

[0055] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] In the description of this application, the terms “comprising,” “having,” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or devices, or steps or units added based on further optimizations of the inventive concept.

[0058] To address the challenge of accurate and efficient prediction of patents across multiple domains, this invention provides a patent labeling method based on semi-supervised domain transfer learning. It constructs source and target domain training corpora based on patent texts and designs a deep learning-based patent prediction model system. The source domain consists of labeled domain patent text data, while the target domain comprises a small amount of labeled and a large amount of unlabeled domain patent text data. The model system includes a feature encoding model and a prediction model. The encoding model embeds feature vectors into the text, including a general baseline encoding model and a momentum encoding model. The baseline encoding model is used for feature encoding of the target domain patent text, while the momentum encoding model is used for feature encoding of the source domain patent text. The encoding model is trained using contrastive learning in an embedding space shared by the source and target domains, thereby generating pseudo-labels for the unlabeled data in the target domain. The prediction model includes a classification prediction model, a representativeness prediction model, and a covariance prediction model. The label prediction model predicts patent labels based on the encoded features. The representativeness prediction model predicts domain representativeness, and the covariance prediction model predicts the covariance matrix, effectively mitigating the bias in prediction results caused by differences in the distribution of patent data across different IPC domains. The labeled patent prediction model in the source domain is trained in a supervised manner, while the unlabeled patent prediction model in the target domain is trained in a semi-supervised manner by generating pseudo-labels. All these models are then integrated to form a complete model system for end-to-end training and inference on both the source and target domain datasets.

[0059] In one embodiment, such as Figure 1As shown, a patent tag information generation method based on transfer learning is provided. This method can be applied to a server and includes five steps: task configuration, model design, corpus construction, model training, and prediction execution. Specifically, it includes:

[0060] The task configuration requires first selecting the patent prediction task, then dividing the patent texts into domains corresponding to IPC classifications, and selecting appropriate source and target domains based on the patent data. The target domain includes a small number of labeled patents and a large number of unlabeled original patent texts. The above domain and task configuration information is saved and generated as input for subsequent model training.

[0061] The model design is based on the aforementioned domain and task configuration information, designing model architectures for both the source and target tasks, including feature encoding models and prediction models. The feature encoding models include a baseline encoding model and a momentum encoding model, used to encode high-value patents with labels and their corresponding cited patents in the source domain, respectively. The momentum encoding model is synchronized with the baseline encoding model through periodic updates. The prediction models include a classification prediction model, a prototype prediction model, and a covariance prediction model. Feature vectors generated by the baseline feature encoding model are input into the classification prediction model, the prototype prediction model, and the covariance prediction model, respectively. The classification prediction model performs patent classification prediction, the prototype prediction model predicts the representativeness of patents in the source domain, and the covariance prediction model predicts the covariance matrix of patents in the source domain. Unlabeled patents in the target domain are given pseudo-labels through a k-NN method based on their representativeness. The overall model loss function is designed, including patent prediction loss, representativeness prediction loss, feature comparison loss, and covariance prediction loss.

[0062] Corpus construction involves preparing training data for the source domain and the target domain respectively. The source domain corpus is constructed using labeled patents and corresponding patent citation data, while the target domain corpus is constructed using a small number of labeled patents and a large number of unlabeled original patent texts.

[0063] Model training involves selecting an optimizer and setting training hyperparameters for training corpora in the chosen source and target domains, respectively, and then training the encoding and prediction models. The model parameters are updated using backpropagation, and the model parameters are saved after training is complete.

[0064] Model application involves using the trained model interface to predict unlabeled patents in the target domain, generate predicted label information, and store it.

[0065] The predicted label information may include IPC classification number, technical field: the application field of the patent can also be described more specifically, such as specific industrial applications, product types, etc.

[0066] Patent legal status: Predictive tags may include the patent's legal status, such as whether it is valid, under examination, or expired.

[0067] Patent technology effect: Predictive tags may describe the actual effects of patent technology, such as improved efficiency, reduced costs, enhanced performance, etc.

[0068] The task configuration determines and selects appropriate source and target domains based on the existing high-value patent data.

[0069] The selection of source and target domains is based on all patent texts. The patents are divided into technical fields according to the IPC classification system. Fields with high-value patent tag data are selected as source domains, and fields with little or no tag data to which the knowledge to be extracted belongs are selected as target domains.

[0070] The field here is based on the IPC classification, and can be a subclass (level 3). Patent tags can be selected based on mission objectives such as high value, highly disruptive, core and critical, green and low-carbon, key digital technologies, and emerging industries. The field can be limited to tobacco, communications, chips, or new energy, depending on the mission requirements. Patent data can be selected from the abstracts or claims in the five patent documents.

[0071] An example:

[0072] This invention selects subclass A24F (smokers' products) in the IPC system as the source domain and subclass C11B (production, such as by pressing raw materials or extracting from waste, refining or preserving lipids, fatty substances such as lanolin, fats or waxes; essential oils; fragrances) as the target domain. The patents in the source domain are data with high-value labels, a small number of patents in the target domain have high-value labels, and a large number of patents have no labels. If all patents in the target domain are unlabeled data, a small number of high-value labels can be added manually. Figure 4 An example diagram illustrates the target and source domains of labeled and unlabeled sample spaces.

[0073] The model design is based on the aforementioned domain and task configuration information, and includes an encoding model and a prediction model, comprising a feature encoding model and a classification model. The feature encoding model includes a baseline encoding model and a momentum encoding model, used to encode tagged patents and their corresponding cited patents in the source domain, respectively. The momentum encoding model is synchronized with the baseline encoding model through periodic updates. The classification model includes a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding model are input into the classification prediction model, the prototype prediction model, and the covariance prediction model, respectively. The classification prediction model performs patent classification prediction, the prototype prediction model predicts the representativeness of patents in the source domain, and the covariance prediction model predicts the covariance matrix of patents in the source domain. Unlabeled patents in the target domain are given pseudo-labels through a kNN method based on the representativeness. The overall model structure and optimization target loss functions are designed, including classification prediction loss, representativeness prediction loss, contrast loss, and covariance prediction loss. Figure 5 This means that the feature vector is obtained by inputting the citation patents corresponding to the patents in the source and target domains, and then used as the input for the classification prediction model, the representative prediction model, and the covariance prediction model to obtain the output of each model. Figure 6 The diagram shows the overall structure of the algorithm.

[0074] Encoding Model: The encoding model structure is designed based on the configuration information of the source and target domains. The BERT model based on the multi-layer TransformerEncoder structure is selected for implementation. The BERT model is used as the encoding layer to implement the patent text encoding, and the [CLS] position vector of the output layer is taken as the feature encoding vector.

[0075] Benchmark Encoding Model: Constructed based on the encoding model structure, it is used for encoding the embedding feature representation of citation patents corresponding to patents in the source and target domains. During initialization, the benchmark encoder F is trained using a labeled dataset in the source domain.

[0076] Momentum encoding model: Based on the encoding model structure, it is used for encoding the embedded feature representation of patents in the source and target domains. During model training initialization, the momentum encoder F' is initialized using the baseline encoder F. During model training, the momentum encoding model parameters are updated using the following algorithm:

[0077] Algorithm 1: Parameter update algorithm for momentum coding model:

[0078] Input: Initialize model parameters θ0, number of iterations K, step size η, momentum (β1, β2, β3) ∈ [0, 1] 3 Stability parameter ε > 0, weight decay λ k >0, reset condition, parameters θ at the k-th and (k-1)-th steps of the reference encoder F. kθ k-1 The parameters m of the momentum encoder F′ in the (k-1)th step of the previous step k-1 v k-1 , n k-1 , θ′ k-1 Target loss function f θ ,

[0079] Output: m at the current step k-th step of the momentum encoder F' k v k n k , θ′ k .

[0080] Initialization: On the first execution, initialization is performed: m0 = 0, v0 = 0, k = 0.

[0081] calculate:

[0082] m k =β1m k-1 +(1-β1)θ k

[0083] v k =β2v k-1 +(1-β2)(θ k -θ k-1 )

[0084]

[0085] n k =β3n k-1 +(1-β3)[θ k +(1-β2)(θ k -θ k-1 )] 2

[0086] Return m k v k , n k , θ′ k , Note: Here, β1=09, β2=0999, β3=09999, η=0001ε=10 -8 K = 1000.

[0087] The prediction model employs a two-layer fully connected neural network structure. The input is the feature vector output by the encoding model, and the output is the binary classification high-value prediction probability. The feature vector receiving the input from the encoding model is n-dimensional. The first hidden layer contains h1 neurons, using ReLU as the activation function. The second hidden layer contains h2 neurons, also using ReLU as the activation function, primarily to increase non-linearity and accelerate training. The output layer contains one neuron, using the sigmoid function. The output is restricted to a probability between 0 and 1. Specifically, z1 = W1x + b1, a1 = ReLU(z1), z2 = W2x + b2, a2 ​​= ReLU(z2), z3 = W3x + b3, y = σ(z3), where W1, W2, and W3 are weight matrices, b1, b2, and b3 are bias vectors, and σ is the sigmoid function. x is the input feature vector.

[0088] The classification prediction model adopts a prediction model structure. The input is the feature vector output by the encoding model, and the output is the binary high-value prediction probability.

[0089] The representative prediction model (Prototype) adopts a prediction model structure. The input is the feature vector output by the encoding model, and the output is the embedded feature vector of the high-value patent representative in this domain.

[0090] The representative type is a P-dimensional feature vector. Calculate by the average of the feature vectors in this class set.

[0091] in, The embedding feature encoding function is implemented as an encoding model, where φ is a learnable parameter of the model.

[0092] The representative model works by giving a distance function. The representative prediction model outputs the softmax distribution of the query point x in this class along the distance d in the embedded space:

[0093]

[0094] The model parameters are learned by minimizing the negative log probability through gradient descent on class k:

[0095] accomplish.

[0096] The covariance prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being a representative covariance matrix.

[0097] The metric embedded in the feature space vector is calculated using the Bregman divergence:

[0098]

[0099] in, For a differentiable strictly Legendre convex function, the Bregman divergence of the square Mahalanobis distance is used here:

[0100] when hour,

[0101] To simplify calculations and improve execution speed, Q can be implemented as an identity matrix, resulting in the Euclidean squared distance:

[0102] at this time

[0103] The covariance matrix here is a Gaussian covariance matrix, which is automatically learned to construct representative patterns based on distance metrics and orientations in the category-specific feature vector space. The cluster centers are the center points of the representative patterns, and the Gaussian covariance matrix represents the confidence (uncertainty) in a specific direction, exhibiting anisotropic scaling in the feature space. Here, Q == S -1 .

[0104] The covariance matrix of the predicted output of the representative network Where D is the dimension of the feature vector in the embedding space.

[0105] The distance between feature vectors is calculated as follows:

[0106]

[0107] in, The centroid is the representative point of class c. M is the covariance matrix. The representative model, combined with the distance between the embedded feature vectors represented by the covariance model, can learn the distance metric of class and orientation dependence in the embedding space. In practical applications, to improve the speed of model training and inference, the covariance matrix can be further simplified to a diagonal matrix.

[0108] Model System: The dynamic mechanism of the model system includes the synchronous update mode of the momentum encoding model and the baseline encoding model, the calculation and update mode of representative feature vectors, the calculation and update mode of covariance matrix, the calculation of loss function, and the update of model parameters.

[0109] The dynamic mechanism of the semi-supervised embedding space domain adaptation model is shown in Algorithm 2:

[0110] Algorithm 2: Dynamic mechanism of semi-supervised embedding space domain adaptation model:

[0111] Input: Source domain dataset Ds The labeled target dataset D t Unlabeled target dataset D u .

[0112] Initialization: Initialize the momentum encoder F` using the baseline encoder F trained on the labeled dataset, and initialize the database Q = {}.

[0113] Repeat the following steps until the model training is complete:

[0114] 1. From the source domain dataset D s Samples with corresponding citation patents in the middle sampling (x) s ,y s )∈D s From the labeled dataset D of the target domain t Samples with corresponding citation patents in the middle sampling (x) t ,y t )∈D t From the unlabeled dataset D of the target domain u Sample x of the sampled patents with corresponding citations u ∈D u .

[0115] 2. Search x from the patent database respectively. s ,x t ,x u Corresponding citation patent data

[0116] 3. Applying a momentum encoder F' to x s ,x t ,x u Encode the feature vector f′ to obtain the embedded feature vector. s f′ t f′ u The application of the reference encoder F pair Encode to obtain embedded feature vectors The representative prediction model P is applied to obtain the representative embedded feature vector p′. s , p′ t , p′ u , The covariance matrix M′ of the current instance is obtained by applying the covariance prediction model M. s M′ t M′ u , Update all generated eigenvectors, representative vectors, and covariance matrices to database Q.

[0117] 4. For unlabeled instance x u , Generate pseudo-tags: if Then pseudo-tags are generated. Where f is for x u , Take f′ u and In kNN, the distance is corrected using the covariance-modified distance formula. calculate, Let be the representative vector of class c, and M be the covariance matrix. Let Q be the candidate embedding feature vectors. Update the database Q with all the generated pseudo-labels.

[0118] 5. Calculate the model loss and update the model parameters through backpropagation.

[0119] 5.1. Calculate tagged citation patent data Tag cross-entropy loss L1: Applying a high-value prediction model C, inputting tag citation patent data. Embedded vector features Obtain predicted label information Through cross-entropy loss function Calculate cross-entropy loss Where, y′ These are the real label and predicted label information, y′ s y′ t Citation patent data in the source and target domains respectively The tag.

[0120] 5.2. Calculating Unlabeled Citation Patent Data The generated pseudo-label cross-entropy loss L u Applying the high-value prediction model C, inputting unlabeled citation patent data Embedded vector features Obtain predicted label information Through cross-entropy loss function Calculate cross-entropy loss in, These are predicted label information and pseudo-labels, respectively. Unlabeled citation patent data of the target domain Generated pseudo-tags.

[0121] 5.3. Application of Formulas Where τ is 0.9 and Equation 6 Calculate the unlabeled sample x in the target domain using the vector library Q. u and the corresponding citation data eigenvector f u and f′ u Contrast loss: L c =L u(f u f′ u ).

[0122] 5.4 Use a gradient inversion layer before the classifier layer to compute the gradient, and use... Update the parameters of the reference encoder F and the initialization momentum encoder F', where λ is... u , λ c represents the regularization coefficient, whose value determines the weight of the regularization term in the total loss function; H represents the update of the prototype based on the entropy predicted from the unlabeled target.

[0123] 5.5 A gradient inversion layer is used before the classifier layer to compute the gradient, and the parameters of the high-value prediction model C are used. Where λ u , λ c represents the regularization coefficient, whose value determines the weight of the regularization term in the total loss function, and H indicates that the prototype was updated based on the entropy predicted from the unlabeled target.

[0124] 5.6. Applying the formula Update the representative vectors of all classes c in all Q. Application formula Update the representative vector of class c predicted by the representative model. Among them, Q fc Let Q be the set of all embedded feature vectors belonging to class c. pc Let Q be the set of all representative vectors belonging to class c, and N be the set of all vectors in Q. pc For Q pc The number of representative vectors, M is the covariance matrix. Calculate the Euclidean distance loss for representative prediction:

[0125] 5.7. Applying the covariance formula Calculate the covariance matrix of all eigenvectors in Q. Where X i ,Y i This represents the i-th eigenvector in Q. Let N be the expected mean of all eigenvectors in Q, and N be the total number of eigenvectors in Q. Apply the formula... Update the covariance matrix predicted by the covariance model, where m is the set of all covariance matrices in Q. M The covariance matrix predicted by the covariance model in the model, N M For Q M The total number of covariance matrices. Calculate the prediction loss based on the covariance matrix: in Let represent the elements in the i-th row and j-th column of the covariance matrix, respectively.

[0126] Sum all the loss functions from steps 5.1 to 5.7 above:

[0127] L = L l +L u +L c +L p +L m Update them separately using the gradient descent algorithm based on backpropagation:

[0128] Parameters of the reference encoder Parameters of high-value prediction models Parameters of a representative predictive model Parameters of the covariance prediction model

[0129] 6. Apply Algorithm 1 to update the parameters of the momentum encoder model.

[0130] 7. Update all generated feature vectors, labels, representative vectors, and covariance matrices to the database Q. Return all model parameters. Corpus construction involves preparing training data for the source and target domains respectively. The source domain corpus is constructed using labeled patents and corresponding patent citation data, while the target domain corpus is constructed using a small number of labeled patents and a large number of unlabeled original patent texts.

[0131] The model training process involves the following steps: first, training the model using the corresponding training corpus data; then, training both the encoder and predictor models, and using the backpropagation algorithm to optimize the model parameters; finally, saving the trained model parameters after training is complete.

[0132] In optional embodiments of this application, a semi-supervised embedding space domain adaptation model dynamic mechanism encodes patent and its cited patent data by applying momentum encoders and benchmark encoders to the source domain dataset, labeled target dataset, and unlabeled target dataset, respectively, generating embedding feature vectors. Subsequently, a representative embedding feature vector is extracted using a representative prediction model, and a covariance matrix is ​​calculated using a covariance prediction model to characterize the data distribution characteristics. For unlabeled instances, pseudo-labels are generated based on the representative embedding feature vectors and the covariance matrix to further enrich the training data. By calculating the model loss and updating the model parameters using backpropagation, model performance is optimized, achieving knowledge transfer from the source domain to the target domain, effectively improving the model's adaptability to target domain data and prediction accuracy.

[0133] In an optional embodiment of this application, a regularization step is added to the model training process to prevent overfitting. Specific implementation details are as follows:

[0134] Add an L2 regularization term to the model's loss function. L2 regularization makes the model's weight vector smoother by adding the sum of squares of the weights to the loss function, thus reducing the model's complexity.

[0135] Set a regularization parameter with a value ranging from [0.001, 0.1]. The regularization parameter controls the weight of the regularization term in the loss function. By adjusting the value of the regularization parameter, a balance can be found between model complexity and goodness of fit. Generally, a smaller regularization parameter value makes the model more inclined to fit the training data, while a larger regularization parameter value makes the model smoother and reduces the risk of overfitting. During training, L2 norm constraints are applied to the model weights. Specifically, by normalizing the weights in each iteration, it is ensured that the L2 norm of the weights does not exceed a certain preset threshold.

[0136] In an optional embodiment of this application, to help users better understand the basis and logic of the prediction results, an interpretive analysis step of the prediction results is added during the prediction process. Specific implementation details are as follows:

[0137] In predictive models, the importance score of each feature to the prediction result is calculated by analyzing the model's weights and input features. Specifically, methods such as SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) can be used to calculate feature importance scores. These methods quantify the contribution of each feature to the model's prediction, thereby helping users understand which features have a significant impact on the prediction results.

[0138] Generate visual charts, such as bar charts or heatmaps, to represent feature importance scores, visually demonstrating the impact of each feature on the prediction results. For example, bar charts clearly show the importance score of each feature, while heatmaps illustrate the interactions between features and their combined impact on the prediction results. These charts help users quickly identify key features and their contribution to the prediction results.

[0139] Generates an explanatory report containing feature importance scores and prediction results. The report details the importance score of each feature and its specific impact on the prediction results. Furthermore, the report can include information such as the confidence level of the prediction results and the source of the predictions (e.g., similar cases in the training data) to help users fully understand the basis and logic behind the predictions.

[0140] Model application involves using the trained model interface to predict unlabeled patents in the target domain, generate predicted label information, and store it.

[0141] In summary, this invention provides a feature-aligned method for extracting multi-domain patent knowledge. It divides the domain based on patent IPC classification, selectively choosing source and target domains and their corresponding labeled and unlabeled data combinations. An encoding model, including a general baseline encoding model and a momentum encoding model, is designed. Combined with classification prediction, representativeness prediction, and covariance prediction models, it achieves cross-domain transfer patent prediction. The encoding and prediction models comprehensively employ multiple loss functions—cross-entropy, contrastive loss, representativeness, and covariance—in the shared embedding space of the source and target domains. Pseudo-labels are automatically generated from a large amount of unlabeled data in the target domain, transforming the original unsupervised training method into a supervised training mode. After integration, a complete model system is formed, achieving efficient end-to-end training and inference on datasets from both the source and target domains.

[0142] In one embodiment, such as Figure 7 As shown, a patent tag information generation system based on transfer learning is provided. The system includes a task configuration module, a model design module, a corpus construction module, a model training module, and an execution prediction module, specifically including:

[0143] The task configuration module is used to select patent prediction tasks. Based on all patent texts, the tasks are divided according to the technical fields in the IPC classification system, and appropriate source and target domains are selected based on the patent data.

[0144] The model design module is used to design feature encoding models and prediction models based on task configuration information. The feature encoding models include a baseline encoding model and a momentum encoding model. The baseline encoding model encodes tagged patents and their corresponding citation patents in the source domain, while the momentum encoding model keeps synchronized with the baseline encoding model through periodic updates. The prediction models include a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding models are input into these models, respectively. The classification prediction model is used to predict the classification of patents, the prototype prediction model is used to predict the representativeness of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain.

[0145] The corpus construction module is used to prepare training corpus data for the source domain and the target domain, respectively. The source domain corpus is constructed using labeled patents and corresponding patent citation data, while the target domain corpus is constructed using a small number of labeled patents and a large number of unlabeled original patent texts.

[0146] The model training module is used to train the model using the corresponding training corpus data for the selected source and target domains.

[0147] The prediction module is used to apply the model interface after model training to predict unlabeled patents in the target domain, generate predicted label information, and store it.

[0148] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computing and control capabilities, and the network interface enables communication with external terminals via a network connection. The computer device loads and runs a computer program to implement the aforementioned patent label information generation method.

[0149] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0150] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored relating to all or part of the processes in the methods of the above embodiments.

[0151] In one embodiment, a computer program product is also provided, including a computer program / instruction relating to all or part of the processes in the methods of the above embodiments.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for generating patent tag information based on transfer learning, characterized in that, The method comprises five steps: task configuration, model design, corpus construction, model training, and prediction execution. Specifically, it includes: Task configuration: Select the patent prediction task, based on all patent texts, and divide them according to the technical fields in the IPC classification system. Select the appropriate source domain and target domain based on the patent data. The model design involves designing a feature encoding model and a prediction model based on the task configuration information. The feature encoding model includes a baseline encoding model and a momentum encoding model. The baseline encoding model encodes tagged patents and their corresponding citation patents in the source domain, while the momentum encoding model maintains synchronization with the baseline encoding model through periodic updates. The prediction model includes a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding model are input into these three models, respectively. The classification prediction model is used to predict the classification of patents, the prototype prediction model is used to predict the representativeness of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain. Corpus construction involves preparing training corpus data for the source and target domains respectively. The source domain corpus is constructed using labeled patents and their corresponding patent citation data, while the target domain corpus is constructed using a small number of labeled patents and a large number of unlabeled original patent texts. Model training involves training the model using the corresponding training corpus data for the selected source and target domains. The prediction is performed by applying the model interface after model training to predict unlabeled patents in the target domain, generating and storing predicted label information; the predicted label information includes at least the patent's IPC classification number, technical field, and legal status. The momentum encoding model parameter update algorithm is implemented through iterative updates. Specifically, the calculation process uses the parameters m of the momentum encoder F′ at the (k-1)th step. k-1 v k-1 , n k-1 , θ′ k-1 Update the parameters θ at the k-th and (k-1)-th steps of the baseline encoder F. k θ k-1 : m k =β1m k-1 +(1-β1)θ k v k =β2v k-1 +(1-β2)(θ k -θ k-1 ) n k =β3n k-1 +(1-β3)[θ k +(1-β2)(θ k -θ k-1 )] 2 m k v k , n k , θ′ k Let θk be the parameters of the momentum encoder F′ at step k, θ0 be the initialization model parameters, k be the number of iterations, η be the step size, and (β1, β2, β3) ∈ [0, 1]. 3 Represents momentum, stability parameter ε > 0, and weight decay λ. k >0.

2. The method according to claim 1, characterized in that, The task configuration involves selecting appropriate source and target domains based on patent data, specifically including: Based on all patent texts, the patents are divided into technical fields according to the IPC classification system. Fields with high-value patent tag data are selected as source fields, and fields with little or no tag data to which the knowledge to be extracted belongs are selected as target fields. The fields are based on the IPC classification, and can be subcategories at the third level. Patent tags are selected based on high value, high disruptiveness, core and key, green and low-carbon, key data, and emerging industry task objectives. Fields are limited to tobacco, communications, chips, or new energy fields according to the task situation. Patent data is selected from the abstracts or claims in the five patent books.

3. The method according to claim 1, characterized in that, In the model design, the encoding model structure is designed according to the configuration information of the source domain and the target domain. The BERT model based on the multi-layer Transformer Encoder structure is selected for implementation. The BERT model is used as the encoding layer to implement the patent text encoding, and the CLS position vector of the output layer is taken as the feature encoding vector. The benchmark coding model is constructed based on the coding model structure and is used to encode the embedding feature representation of the citation patents corresponding to the patents in the source and target domains. During initialization, a baseline encoder is obtained by training on a labeled dataset in the source domain; The momentum coding model is built on the coding model structure and is used for encoding the embedded feature representation of patents in the source and target domains; During model training initialization, the momentum encoder is initialized using the baseline encoder. During model training, the momentum encoder model parameters are updated using the momentum encoder model parameter update algorithm.

4. The method according to claim 1, characterized in that, In the model design, the classification prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the binary high-value prediction probability. The representative prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being the embedded feature vector of the high-value patent representative in this domain. The covariance prediction model adopts a prediction model structure, with the input being the feature vector output by the encoding model and the output being a representative covariance matrix.

5. The method according to claim 4, characterized in that, The representative predictive model is represented by a P-dimensional feature vector. Calculate by the average of the feature vectors in this class set. The embedding feature encoding function is implemented using an encoding model, where φ represents the learnable parameters of the model, and x... i ,y i S represents the embedded features. k Indicates the total number of embedded features; The measure embedded in the feature space vector by the covariance prediction model is calculated using Bregman divergence. in, Let z and z′ be differentiable strictly Legendre-type convex functions, and z′ be the feature vectors output by the encoding model. The distance between feature vectors is calculated as follows: in, It is the center point, that is, the representative type of class c, and M is the covariance matrix. The distance in the embedded feature vector space.

6. The method according to claim 1, characterized in that, The semi-supervised embedding spatial domain adaptation model dynamic mechanism applies momentum encoders and benchmark encoders to the source domain dataset, labeled target dataset, and unlabeled target dataset to encode patent and its cited patent data, respectively. It applies a representative prediction model to obtain representative embedding feature vectors, applies a covariance prediction model M to obtain the covariance matrix of the current instance, generates pseudo-labels for unlabeled instances, calculates the model loss, and updates the model parameters through backpropagation.

7. The method according to claim 1, characterized in that, The model training step also includes regularization processing of the model, specifically including: Add an L2 regularization term to the model's loss function; Set the regularization parameter and determine the range of values ​​for the regularization parameter; During training, the model weights are constrained by the L2 norm. In addition, the forecasting process also includes interpreting the forecast results, specifically including: Extract feature importance scores that are relevant to the prediction results; Generate a visual chart of feature importance scores; Generate an explanatory report that includes feature importance scores and prediction results.

8. A patent tag information generation system based on transfer learning, characterized in that, The system includes a task configuration module, a model design module, a corpus construction module, a model training module, and an execution prediction module, which specifically include: The task configuration module is used to select patent prediction tasks. Based on all patent texts, the tasks are divided according to the technical fields in the IPC classification system, and appropriate source and target domains are selected based on the patent data. The model design module is used to design feature encoding models and prediction models based on task configuration information. The feature encoding models include a baseline encoding model and a momentum encoding model. The baseline encoding model encodes tagged patents and their corresponding citation patents in the source domain, while the momentum encoding model keeps synchronized with the baseline encoding model through periodic updates. The prediction models include a classification prediction model, a prototype prediction model, and a covariance prediction model. The feature vectors generated by the feature encoding models are input into these models, respectively. The classification prediction model is used to predict the classification of patents, the prototype prediction model is used to predict the representativeness of patents in the source domain, and the covariance prediction model is used to predict the covariance matrix of patents in the source domain. The corpus construction module is used to prepare training corpus data for the source domain and the target domain, respectively. The source domain corpus is constructed using labeled patents and corresponding patent citation data, while the target domain corpus is constructed using a small number of labeled patents and a large number of unlabeled original patent texts. The model training module is used to train the model using the corresponding training corpus data for the selected source and target domains. The execution prediction module is used to apply the model interface after model training to predict unlabeled patents in the target domain, generate predicted label information and store it; wherein, the predicted label information includes at least the patent's IPC classification number, technical field and legal status; The momentum encoding model parameter update algorithm is implemented through iterative updates. Specifically, the calculation process uses the parameters m of the momentum encoder F′ at the (k-1)th step. k-1 v k-1 , n k-1 , θ′ k-1 Update the parameters θ at the k-th and (k-1)-th steps of the baseline encoder F. k θ k-1 : m k =β1m k-1 +(1-β1)θ k v k =β2v k-1 +(1-β2)(θ k -θ k-1 ) n k =β3n k-1 +(1-β3)[θ k +(1–β2)(θ k -θ k-1 )] 2 m k v k , n k , θ′ k Let θk be the parameters of the momentum encoder F′ at step k, θ0 be the initialization model parameters, k be the number of iterations, η be the step size, and (β1, β2, β2) ∈ [0, 1]. 3 Represents momentum, stability parameter ε > 0, and weight decay λ. k >0.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

Images