Text detection model training method and apparatus

Abstract

This specification provides a text detection model training method and apparatus. The text detection model training method includes: obtaining pre-training parameters of a pre-trained model; performing a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data; adding label parameters to the initial parameters to obtain target parameters, and obtaining labels for the sample data, wherein the labels are type representations of the text data; and performing a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain a target model. Because label parameters are added to the parameters, text data and its labels can be used as training samples, i.e., training can be performed on different types of text data, allowing for low-cost transfer to multiple new domains and reducing the difficulty of model transfer.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of model training technology, and in particular to a method for training a text detection model. Background Technology

[0002] In current machine learning practices, pre-trained models and attention mechanisms have achieved good performance in various text tasks and have strong generalization capabilities. However, in scenarios involving heterogeneous text data (text data with different grammatical structures), it is difficult to fine-tune a pre-trained model using general texts to texts with different language structures and expression patterns. This is especially true when the language structure of the downstream task differs significantly from the corpus used in the pre-training task, making model generalization even more challenging. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a text detection model training method. One or more embodiments of this specification also relate to a text detection model training apparatus, a category detection method, a category detection device, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a text detection model training method is provided, comprising:

[0005] Obtain the pre-training parameters of the pre-trained model, and perform a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data;

[0006] Add a label parameter to the initial parameters to obtain the target parameters, and obtain the label of the sample data, wherein the label is a type representation of the text data;

[0007] A second pre-training task is performed based on the target parameters, the sample data, and the labels of the sample data to obtain the target model.

[0008] According to a second aspect of the embodiments of this specification, a text detection model training apparatus is provided, comprising:

[0009] The first training module is configured to obtain the pre-training parameters of the pre-trained model, and perform a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data;

[0010] The parameter determination module is configured to add a label parameter to the initial parameters to obtain the target parameters, and to obtain the label of the sample data, wherein the label is a type representation of the text data;

[0011] The second training module is configured to perform a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain the target model.

[0012] According to a third aspect of the embodiments of this specification, a category detection method is provided, comprising:

[0013] The description information of the target object is input into the embedding layer of the category detection model to obtain the description encoding vector, wherein the category of the description information is at least two.

[0014] The description encoding vector is input into the attention mechanism layer of the category detection model to obtain the description output data;

[0015] According to a fourth aspect of the embodiments of this specification, a category detection apparatus is provided, the apparatus comprising:

[0016] The embedding module is configured to input the description information of the target object into the embedding layer of the category detection model to obtain a description encoding vector, wherein the categories of the description information are at least two.

[0017] An attention module is configured to input the description encoding vector into the attention mechanism layer of the category detection model to obtain description output data;

[0018] The detection module is configured to input the description output data into the detection layer of the category detection model to obtain the category information of the target object.

[0019] The description output data is input into the detection layer of the category detection model to obtain the category information of the target object.

[0020] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0021] Memory and processor;

[0022] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described text detection model training method.

[0023] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described text detection model training method.

[0024] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described text detection model training method.

[0025] This specification provides a text detection model training method and apparatus. The text detection model training method includes: obtaining pre-training parameters of a pre-trained model; performing a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data; adding label parameters to the initial parameters to obtain target parameters, and obtaining labels for the sample data, wherein the labels are type representations of the text data; and performing a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain a target model. By adding label parameters to the initial parameters to obtain target parameters, the second pre-training task can be performed based on the target parameters, sample data, and the labels of the sample data to obtain the target model. Since label parameters are added to the parameters, text data and its labels can be used as training samples. That is, training can be performed on different types of text data, and the model can be transferred to multiple new domains (application scenarios) at a lower cost. This eliminates the need for separate model training for each type of text data, reducing the difficulty of model transfer and improving the model's generalization ability. Attached Figure Description

[0026] Figure 1a This is a schematic diagram illustrating a scenario of a text detection model training method provided in one embodiment of this specification;

[0027] Figure 1b This is a training diagram illustrating a text detection model training method provided in one embodiment of this specification;

[0028] Figure 2a This is a flowchart illustrating a text detection model training method provided in one embodiment of this specification;

[0029] Figure 2b This is a schematic diagram of the initial parameters of a text detection model training method provided in one embodiment of this specification;

[0030] Figure 2c This is a schematic diagram of the target parameters of a text detection model training method provided in one embodiment of this specification;

[0031] Figure 2d This is a schematic diagram of the detection layer of a text detection model training method provided in one embodiment of this specification;

[0032] Figure 3This is a flowchart of a category detection method provided in one embodiment of this specification;

[0033] Figure 4 This is a schematic diagram of the structure of a text detection model training device provided in one embodiment of this specification;

[0034] Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0035] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0036] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0038] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0039] Heterogeneous text: Text data with different meta-features such as language structure and expression patterns.

[0040] Attention mechanism: A network structure in deep learning with the form E = g(Q,K,V) = softmax(normalize(QK^T))V.

[0041] Self-attention mechanism: For sequential data, it is a special case of the attention mechanism where Q=K=V=X.

[0042] Multihead Attention is a variant of attention mechanisms that divides the softmax function into multiple groups on an average basis.

[0043] Pre-training: A training paradigm for large models, in which the model is first trained on a large number of samples using a pre-training task in an unsupervised or minimally supervised manner, and then fine-tuned for downstream tasks.

[0044] Transformer: A sequence model structure based on an attention mechanism.

[0045] BERT: A pre-trained model based on bidirectional Transformer.

[0046] Word embedding, also known as word embedding, transforms a word or feature into a vector.

[0047] MLM / NSP: Two pre-training tasks.

[0048] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that include convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning.

[0049] Recurrent Neural Network (RNN): A type of recursive neural network that takes sequence data as input, recurses in the direction of sequence evolution, and connects all nodes (recurrent units) in a chain-like manner.

[0050] Multilayer Perceptron (MLP): A feedforward artificial neural network model that maps multiple input datasets to a single output dataset.

[0051] CLS stands for classification, which can be understood as being used for downstream classification tasks.

[0052] The field of text processing models has evolved from traditional W2V (word2vec, a type of word embedding that maps a word to a fixed-dimensional vector (unchanging with the corpus) and reflects the relationships between words to some extent) to language models, and then to CNNs or some RNN-based deep learning models. Currently, the Transformer, as a completely MLP-based implementation, overcomes a series of shortcomings of CNN / RNN models, such as their difficulty in parallelization, thus becoming the state-of-the-art (SOTA) text processing model. BERT, as an implementation of the Transformer, utilizes two pre-training tasks, MLM and NSP, to distill the general structures existing in text data into the model, making them usable for downstream tasks.

[0053] Currently, there are some methods in academia for fusion of heterogeneous texts, such as PTE (Predictive Text Embedding), but most of these solutions cannot be directly fused with BERT, the current state-of-the-art (SOTA) text model. In the deep learning or Transformer paradigm, some works attempt to inject meta-features from different sources into the representation vector by predicting the source labels of multi-source texts. However, such approaches alter the meaning of the original BERT CLS token, leading to a cold-start problem and impacting downstream tasks, even causing catastrophic forgetting. In non-Transformer paradigms, there are also some graph-based fusion methods.

[0054] As mentioned earlier, current solutions for heterogeneous texts are difficult to integrate with attention mechanisms, which increases the transfer difficulty of many large pre-trained models.

[0055] Based on this, a method is proposed that enables a model pre-trained on a certain corpus to be transferred to multiple new domains at a low cost, and that information sharing between multiple different domains can be achieved. This specification provides a text detection model training method, and also relates to a text detection model training device, a category detection method, a category detection device, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments.

[0056] See Figure 1a , Figure 1a This diagram illustrates a scenario illustrating a text detection model training method according to one embodiment of this specification. A company possesses a wealth of information, including business licenses, job postings, and patent information. By inputting this company information into a company classification model, the company's category can be determined. Before classifying the company, the company classification model must be trained. See also... Figure 1b , Figure 1bThe diagram illustrates the training of the company's classification model, which includes a parameter loading module, a pre-training task module, an additional parameter initialization module, an additional parameter pre-training task module, a model head initialization module, and a downstream task fine-tuning module.

[0057] In the company classification scenario, the parameters of the pre-trained model are obtained in the loading parameter module, the model to be trained is determined, and the information from different sources of the company (such as recruitment information, business license information, etc.) is encoded to obtain the encoded vector X, which is input into the pre-training task module to execute the pre-training task and obtain the first model.

[0058] In the initialization of additional parameters module, new label parameters are added to the first model so that in the additional parameters pre-training task module, the encoding vector X and the label vector Y corresponding to the encoding vector X can be input to perform the training task and obtain the second model.

[0059] Because the application scenario is company classification, the model head initialization module adds a model head for the company classification scenario to the second model to obtain the third model. Then, in the downstream task fine-tuning module, the encoding vector X, the label vector Y corresponding to the encoding vector X, and the global vector Z (e.g., company name) are input into the third model to obtain the company classification model. Then, the company classification model can be used to classify companies.

[0060] By adding a label parameter to the parameters, text data and its labels can be used as training samples. That is, training can be performed on different types of text data, and the model can be transferred to multiple new domains at a lower cost, reducing the difficulty of model transfer.

[0061] See Figure 2a , Figure 2a A flowchart of a text detection model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0062] Step 202: Obtain the pre-training parameters of the pre-trained model, and perform the first pre-training task according to the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data.

[0063] The pre-trained model can be an attention-based model, such as the BERT model; the pre-trained parameters can be the parameters of the pre-trained model, such as the K, Q, and V matrices; the first pre-training task can be a training task that takes text of multiple structural types as input. Correspondingly, the aforementioned multiple structural types of text can be at least two types of text data, and these at least two types of text data can be text data with different grammatical structures. It should be noted that language structure can refer to the way language is composed. For example, one language structure is: "We need you to have certain professional abilities," which contains words in the form of subject, verb, and object. Another language structure is: "Party A, Party B...", which only includes the subject and does not contain words in the form of predicate or object. For example, the text data in a contract can be: Party A, Party B, signature, etc., and the text data corresponding to a company's recruitment information can be: "We need you to have certain professional abilities."

[0064] In practical applications, you can use a pre-trained BERT model, extract the parameters of the BERT model, and perform the first pre-training task on other corresponding models. Alternatively, you can directly use the pre-trained BERT model to perform the first pre-training task.

[0065] For example, to obtain a pre-trained BERT model, input the company's business qualification information, such as "cultural and artistic activity exchange planning, network, multimedia" and the company's recruitment information, such as "we need you to have certain professional abilities," into the BERT model. This means that the BERT model has performed the first pre-training task.

[0066] The embodiments in this specification improve the generalization ability of the model by training it with at least two types of text data, without having to train the model separately for each type of text data.

[0067] Specifically, because there may be other training needs, the pre-trained model is not used directly when training the model. The parameters of the BERT model can be extracted by performing the first training task and then executed on other corresponding models. The specific implementation method is as follows.

[0068] The step of performing the first pre-training task based on the pre-training parameters and sample data to obtain initial parameters includes:

[0069] A first training model is determined based on the pre-training parameters, wherein the parameters of the first training model correspond to the parameters of the pre-training parameters;

[0070] The pre-trained parameters are used as training parameters for the first training model and added to the first training model.

[0071] The first training model is trained based on the sample data to obtain the initial parameters of the first training model after training.

[0072] The first training model can be an attention-based model; the parameters of the first training model correspond to the pre-training parameters, which can be understood as the first training model requiring the same parameters as the pre-training parameters.

[0073] In practical applications, the extracted pre-trained parameters can be loaded into the corresponding model, and the first pre-training task can be performed based on the model loaded with the pre-trained parameters to obtain the initial parameters. The specific training implementation method is as follows.

[0074] In one possible implementation, training the first training model based on the sample data to obtain the initial parameters of the trained first training model includes:

[0075] The sample data is input into the embedding layer of the first training model to obtain the first sample encoding vector;

[0076] The first sample encoding vector is input into the attention mechanism layer of the first training model to obtain the first output data and the initial parameters after training.

[0077] The embedding layer can be understood as a network layer that converts text data into encoded vectors; the sample encoded vector can be a vector of sample data after encoding; the first output data is the data output by the first trained model; and the initial parameters can be the parameters of the model obtained after the first pre-training task.

[0078] In practical applications, a core module of BERT is the SelfAttention mechanism. Based on a sequence representation vector X, it obtains a transformed sequence representation tilde{X} by mapping g(KX,QX,VX), where K, Q, and V are matrices, typically compressed mappings called the key map, query map, and value map, respectively. On a homogeneous training set, K, Q, and V are global and independent of the text source.

[0079] For example, a BERT model matching the pre-trained parameters is determined, and the pre-trained parameters are loaded into the BERT model. The business qualification information of the company, such as "cultural and artistic activity exchange planning, network, multimedia", and the recruitment information of the company, such as "we need you to have certain professional abilities", are input into the BERT model. In the embedding layer of the BERT model, these statement data will be converted into encoding vectors X. Then, the encoding vector X is multiplied by the K, Q, and V matrices, and after mapping g(KX,QX,VX), the transformed sequence representation tilde{X} is obtained.

[0080] The specific implementation method of directly using the pre-trained BERT model to perform the first pre-training task is as follows.

[0081] In one possible implementation, the step of performing a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters includes:

[0082] The sample data is input into the embedding layer of the pre-trained model to obtain the second sample encoding vector, wherein the parameters of the pre-trained model are the pre-training parameters;

[0083] The second sample encoding vector is input into the attention mechanism layer of the pre-trained model to obtain the second output data and the initial parameters after training.

[0084] In practical applications, there may be situations where a pre-trained model is obtained through pre-training. In such cases, the pre-trained model can be used directly to perform the first pre-training task.

[0085] For example, see Figure 2b , Figure 2b This diagram illustrates the initial parameters of a text detection model training method described in this specification. After pre-training, a pre-trained model is obtained, which is a BERT model. The business qualification information of the company, such as "cultural and artistic activity exchange planning, network, multimedia", and the recruitment information of the company, such as "we need you to have certain professional abilities", are input into the BERT model. In the embedding layer of the BERT model, these sentence data are converted into encoding vectors X. Then, the encoding vector X is multiplied by the K, Q, and V matrices, and passed through the multi-head attention module g, that is, through the mapping g(KX,QX,VX), the transformed sequence representation is obtained, that is, the output data: tilde{X}.

[0086] The embodiments in this specification utilize a pre-trained model for the first training task, reducing training time and improving training efficiency.

[0087] Step 204: Add a label parameter to the initial parameters to obtain the target parameters, and obtain the label of the sample data, wherein the label is a type representation of the text data.

[0088] The initial parameters can be the parameters of the model after the first pre-training task; the label parameters are the parameters processed for the labels of the sample data, for example, the label parameters include K1, Q1, and V1 matrices; the target parameters can include both the parameters processed for the sample data and the parameters processed for the labels of the sample data, for example, the target parameters include K, Q, and V matrices and K1, Q1, and V1 matrices; the labels can be a summary description of the text data, for example, if the text data is "cultural and artistic activity exchange planning, network, multimedia", then the label corresponding to this text data is "business license information".

[0089] In practical applications, when dealing with heterogeneous texts, our training data is no longer a uniform set of N data points in the form of {X_i}, but rather a labeled set in the form of {X_i,Y_i}, where Y_i is the label indicating that the text X_i belongs to a certain source. For example, in the industry classification module we plan to use, Y_i can represent a description of the business scope, patent information, etc.

[0090] For example, the initial parameters only include processing parameters for the encoding vector X of the text data, and then label parameters are added for processing the label vector.

[0091] In one possible implementation, the initial parameters include a query matrix, a key matrix, and a value matrix;

[0092] Accordingly, the step of adding label parameters to the initial parameters to obtain target parameters includes:

[0093] The target parameters are obtained by adding a query tag matrix, a key tag matrix, and a value tag matrix to the initial parameters.

[0094] In this context, the query map can be the Q-matrix in the above embodiment, the key map can be the K-matrix in the above embodiment, and the value map can be the V-matrix in the above embodiment.

[0095] In practical applications, to enable a large pre-trained model to adapt to this heterogeneity, we embed Yi into a dense matrix identical to K, Q, and V, where the parameters are trainable free parameters, referred to as label embedding. In extreme cases, different label embeddings can be introduced into each SelfAttention module of BERT, i.e.

[0096]

[0097] Here, g represents an original Multihead Attention module, tilde{K} represents the key map embedding of Y_i for the corresponding layer, K represents the original parameters obtained from the pre-trained model, X represents the sequence embedding output of a certain layer, and Y represents the source label embedding output of a certain layer. Since the dimension of the representation vector obtained by the above module changes compared to the dimension of the input X, in order to maintain comparability with the original BERT structure, the subsequent MLP will recompress the output of this module to the dimension of the input.

[0098] It should be noted that new parameters can be obtained by freezing the pre-trained parameters and reusing the upstream model's pre-training task for separate training.

[0099] For example, label parameters can be added to the initial parameters, including the K1, Q1, and V1 matrices.

[0100] The embodiments in this specification add parameters that can be processed for label variables, so that the label data can be processed through the label parameters in the future. This enables the capture of different interaction structures on heterogeneous texts and integrates them into pre-trained representation vectors, thereby obtaining a unified embedding space structure for texts of different properties.

[0101] Step 206: Perform a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain the target model.

[0102] The second pre-training task can be a task that trains using sample data and the labels of the sample data; the target model can be a pre-trained attention-based model, such as the BERT model, or other transform models.

[0103] In practical applications, fine-tuning is performed using {X_i} and {X_i,Y_i}. The original parameters are either frozen directly or a small learning rate is used. For cold-start parameters of newly added modules, a normal learning rate is used; this is the pre-training process. Taking BERT as an example, the MLM task can be directly reused, and the NSP task can be reconstructed using the current dataset. For instance, texts describing the same object can be used to construct ordered pairs (X_ij,X_ij') to supervise the NSP task.

[0104] Specifically, the first training model can be used directly for further training, as described below.

[0105] After obtaining the initial parameters of the first trained model, the process also includes:

[0106] Add label parameters to the parameters of the first trained model after training to obtain the second trained model;

[0107] The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

[0108] The second training model can be a model that performs the second pre-training task.

[0109] In practical applications, the first training model, trained through the first pre-training task, can be directly used to perform the second pre-training task.

[0110] For example, after the first pre-training task, a second training model is obtained. This second training model is a BERT model. The business qualification information of the company, such as "cultural and artistic activity exchange planning, network, multimedia," and the recruitment information of the company, "We need you to have certain professional skills," are input into the BERT model. In the embedding layer of the BERT model, this statement data is converted into an encoding vector X, which is then multiplied by matrices K, Q, and V. Additionally, the labels corresponding to "cultural and artistic activity exchange planning, network, multimedia" ("business license information") and "recruitment information" ("we need you to have certain professional skills") are input. These labels are converted into label vectors Y in the embedding layer of the BERT model. The encoding vector X is then multiplied by matrices K1, Q1, and V1, and passed through the attention module g, i.e., through the mapping g(KX,QX,VX), to obtain the transformed sequence representation, i.e., the output data: tilde{X}.

[0111] In one feasible approach, target parameters can also be obtained, and other models can be configured using these target parameters to perform a second pre-training task, as described in the following implementation.

[0112] The step of performing a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain the target model includes:

[0113] A second training model is determined based on the target parameters, wherein the parameters of the second training model correspond to the parameters of the target parameters;

[0114] The target parameters are used as training parameters for the second training model and added to the second training model.

[0115] The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

[0116] In practical applications, the target parameters can be loaded into the corresponding model, and a second pre-training task can be performed based on the model loaded with the target parameters to obtain the target model. The specific training implementation method is described below.

[0117] The step of training the second training model based on the sample data and the labels of the sample data includes:

[0118] The sample data and its labels are input into the embedding layer of the second training model to obtain the third sample encoding vector and the label encoding vector.

[0119] The third sample encoding vector and the label encoding vector are input into the attention mechanism layer of the second training model to obtain the third output data and the trained target model.

[0120] For example, see Figure 2c , Figure 2c This diagram illustrates the target parameters of a text detection model training method described in this specification. A BERT model matching the target parameters is determined and loaded into this BERT model. The business qualification information of the company, such as "Cultural and Artistic Activity Exchange Planning, Network, Multimedia," and the recruitment information, "We need you to have certain professional skills," are input into the BERT model. In the embedding layer of the BERT model, this text data is converted into an encoding vector X, which is then multiplied by matrices K, Q, and V. Additionally, the tags corresponding to "Cultural and Artistic Activity Exchange Planning, Network, Multimedia" ("Business License Information") and "We need you to have certain professional skills" ("Recruitment Information") are input. These tags are converted into a label vector Y in the embedding layer of the BERT model. The encoding vector X is then multiplied by matrices K1, Q1, and V1, and passed through a multi-head attention module g, i.e., through mapping g(KX,QX,VX), to obtain the transformed sequence representation, i.e., the output data: tilde{X}.

[0121] The embodiments in this specification enable the fine-tuning model to fully utilize the common features of the pre-trained model and the downstream task corpus by embedding text sources, while also capturing differences, thereby improving the performance of the pre-trained model on the downstream task.

[0122] Since the target model outputs a sequence representation, it cannot be directly applied to specific problems. Therefore, after obtaining the target model, the following steps are also included:

[0123] Determine the target detection layer based on task requirements;

[0124] The target detection layer is added after the attention mechanism layer of the target model to obtain the application model.

[0125] The task requirements can be the specific application scenarios of the model, such as company classification; the target detection layer can be the detection head (model head).

[0126] In practical applications, the trained target model needs to be fitted with a model head that corresponds to the usage scenario before it can be used in practice.

[0127] For example, see Figure 2d , Figure 2d This diagram illustrates a detection layer for training a text detection model, comprising an input layer, an embedding layer, and an attention mechanism layer. Text data and its labels are input into the input layer; for example, the text data could be "We need you to have certain professional skills," and the corresponding label could be "recruitment information." The embedding layer transforms the text data into an encoding vector X and a label vector Y. These vectors are then input into the attention mechanism layer, passing through N stacked transform modules to output a sequence representation: tilde{X}. This sequence representation is then passed through a detection layer for company classification to determine the company's category, such as "technology company."

[0128] By adding a detection head, the embodiments in this specification enable the target model to be applied to different scenarios, thus expanding the application range of the model.

[0129] In one feasible approach, after obtaining the application model, the following is also included:

[0130] A third training task is performed based on the application model, the sample data, and the labels of the sample data to obtain a category detection model.

[0131] In practical applications, for downstream tasks such as text classification or text similarity tasks, normal model fine-tuning can be performed based on the first stage to achieve better recognition and detection results.

[0132] For example, for text classification tasks, you can use a pre-trained Pooler (a type of output in the BERT model) or a cold-start Pooler to optimize the model head using label information.

[0133] This specification provides a text detection model training method and apparatus. The text detection model training method includes: obtaining pre-training parameters of a pre-trained model; performing a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data; adding label parameters to the initial parameters to obtain target parameters, and obtaining labels for the sample data, wherein the labels are type representations of the text data; and performing a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain a target model. By adding label parameters to the initial parameters to obtain target parameters, the second pre-training task can be performed based on the target parameters, sample data, and the labels of the sample data to obtain the target model. Since label parameters are added to the parameters, text data and its labels can be used as training samples. That is, training can be performed on different types of text data, and the model can be transferred to multiple new domains at a lower cost, reducing the difficulty of model transfer.

[0134] See appendix Figure 3 , Figure 3 A flowchart of a category detection method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0135] Step 302: Input the description information of the target object into the embedding layer of the category detection model to obtain the description encoding vector, wherein the category of the description information is at least two.

[0136] The target object can be an individual, a company, or other similar entity; the descriptive information can be a business license, patent information, or other similar information; and the category detection model can be the category detection model trained in the above embodiments.

[0137] In practical applications, companies have multiple information sources, such as business license information, patent information, recruitment information, etc. By using this information through a trained category detection model, the company's category information can be obtained.

[0138] For example, the company's description is "We need you to have certain professional skills", and the corresponding tag for this description is "recruitment information". The description is converted into an encoding vector X and a tag vector Y through an embedding layer.

[0139] Step 304: Input the description encoding vector into the attention mechanism layer of the category detection model to obtain description output data.

[0140] The category detection model can be an attention-based model, such as the BERT model; the output data can be a sequence representation of descriptive information.

[0141] For example, inputting the encoding vector X and the label vector Y into the attention mechanism layer, and passing through N stacked transform modules in the attention mechanism layer, the output sequence representation is: tilde{X}.

[0142] Step 306: Input the description output data into the detection layer of the category detection model to obtain the category information of the target object.

[0143] The detection layer can be a detection head (model head).

[0144] For example, the sequence tilde{X} is passed through a detection layer used for company classification to output the company category, such as "technology company".

[0145] The embodiments in this specification use a category detection model for category detection. The integration of the category detection model does not change the parameter structure of the original model, and it can be effectively loaded and utilized. This overcomes the shortcomings of methods such as source label prediction that change the meaning of the representation vector, thereby improving the detection accuracy.

[0146] Corresponding to the above method embodiments, this specification also provides embodiments of a text detection model training device. Figure 4 A schematic diagram of a text detection model training device according to one embodiment of this specification is shown. Figure 4 As shown, the device includes:

[0147] The first training module 402 is configured to obtain the pre-training parameters of the pre-trained model, perform a first pre-training task based on the pre-training parameters and sample data, and obtain initial parameters, wherein the sample data includes at least two types of text data;

[0148] The parameter determination module 404 is configured to add a label parameter to the initial parameters to obtain the target parameters, and to obtain the label of the sample data, wherein the label is a type representation of the text data;

[0149] The second training module 406 is configured to perform a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain the target model.

[0150] In one possible implementation, the first training module 402 is further configured as follows:

[0151] A first training model is determined based on the pre-training parameters, wherein the parameters of the first training model correspond to the pre-training parameters;

[0152] The pre-trained parameters are used as training parameters for the first training model and added to the first training model.

[0153] The first training model is trained based on the sample data to obtain the initial parameters of the first training model after training.

[0154] In one possible implementation, the first training module 402 is further configured as follows:

[0155] The sample data is input into the embedding layer of the first training model to obtain the first sample encoding vector;

[0156] The first sample encoding vector is input into the attention mechanism layer of the first training model to obtain the first output data and the initial parameters after training.

[0157] In one possible implementation, the first training module 402 is further configured as follows:

[0158] The sample data is input into the embedding layer of the pre-trained model to obtain the second sample encoding vector, wherein the parameters of the pre-trained model are the pre-training parameters;

[0159] The second sample encoding vector is input into the attention mechanism layer of the pre-trained model to obtain the second output data and the initial parameters after training.

[0160] In one possible implementation, the parameter determination module 404 is also configured as follows:

[0161] The initial parameters include a query matrix, a key matrix, and a value matrix;

[0162] Accordingly, the step of adding label parameters to the initial parameters to obtain target parameters includes:

[0163] The target parameters are obtained by adding a query tag matrix, a key tag matrix, and a value tag matrix to the initial parameters.

[0164] In one possible implementation, the second training module 406 is further configured as follows:

[0165] Add label parameters to the parameters of the first trained model after training to obtain the second trained model;

[0166] The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

[0167] In one possible implementation, the second training module 406 is further configured as follows:

[0168] A second training model is determined based on the target parameters, wherein the parameters of the second training model correspond to the target parameters;

[0169] The target parameters are used as training parameters for the second training model and added to the second training model.

[0170] The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

[0171] In one possible implementation, the second training module 406 is further configured as follows:

[0172] The sample data and its labels are input into the embedding layer of the second training model to obtain the third sample encoding vector and the label encoding vector.

[0173] The third sample encoding vector and the label encoding vector are input into the attention mechanism layer of the second training model to obtain the third output data and the trained target model.

[0174] In one possible implementation, the second training module 406 is further configured as follows:

[0175] Determine the target detection layer based on task requirements;

[0176] The target detection layer is added after the attention mechanism layer of the target model to obtain the application model.

[0177] In one possible implementation, the second training module 406 is further configured as follows:

[0178] A third training task is performed based on the application model, the sample data, and the labels of the sample data to obtain a category detection model.

[0179] This specification provides a text detection model training method and apparatus. The text detection model training apparatus includes: acquiring pre-training parameters of a pre-trained model; performing a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data; adding label parameters to the initial parameters to obtain target parameters, and acquiring labels for the sample data, wherein the labels are type representations of the text data; and performing a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain a target model. By adding label parameters to the initial parameters to obtain target parameters, the second pre-training task can be performed based on the target parameters, sample data, and the labels of the sample data to obtain the target model. Since label parameters are added to the parameters, text data and its labels can be used as training samples, i.e., training can be performed on different types of text data, and the model can be transferred to multiple new domains at a lower cost, reducing the difficulty of model transfer.

[0180] Corresponding to the above method embodiments, this specification also provides embodiments of a category detection device. Figure 3 A schematic diagram of a category detection device according to one embodiment of this specification is shown. Figure 3 As shown, the device includes:

[0181] The embedding module is configured to input the description information of the target object into the embedding layer of the category detection model to obtain a description encoding vector, wherein the categories of the description information are at least two.

[0182] An attention module is configured to input the description encoding vector into the attention mechanism layer of the category detection model to obtain description output data;

[0183] The detection module is configured to input the description output data into the detection layer of the category detection model to obtain the category information of the target object.

[0184] In one possible implementation, a model training module is also included, which is configured as follows:

[0185] Obtain the pre-training parameters of the pre-trained model, and perform a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data;

[0186] Add a label parameter to the initial parameters to obtain the target parameters, and obtain the label of the sample data, wherein the label is a type representation of the text data;

[0187] A second pre-training task is performed based on the target parameters, the sample data, and the labels of the sample data to obtain the target model.

[0188] In one possible implementation, the model training module is also configured as follows:

[0189] A first training model is determined based on the pre-training parameters, wherein the parameters of the first training model correspond to the parameters of the pre-training parameters;

[0190] The pre-trained parameters are used as training parameters for the first training model and added to the first training model.

[0191] The first training model is trained based on the sample data to obtain the initial parameters of the first training model after training.

[0192] In one possible implementation, the model training module is also configured as follows:

[0193] The sample data is input into the embedding layer of the first training model to obtain the first sample encoding vector;

[0194] The first sample encoding vector is input into the attention mechanism layer of the first training model to obtain the first output data and the initial parameters after training.

[0195] In one possible implementation, the model training module is also configured as follows:

[0196] The sample data is input into the embedding layer of the pre-trained model to obtain the second sample encoding vector, wherein the parameters of the pre-trained model are the pre-training parameters;

[0197] The second sample encoding vector is input into the attention mechanism layer of the pre-trained model to obtain the second output data and the initial parameters after training.

[0198] In one possible implementation, the model training module is also configured as follows:

[0199] Accordingly, the step of adding label parameters to the initial parameters to obtain target parameters includes:

[0200] The target parameters are obtained by adding a query tag matrix, a key tag matrix, and a value tag matrix to the initial parameters.

[0201] In one possible implementation, the model training module is also configured as follows:

[0202] Add label parameters to the parameters of the first trained model after training to obtain the second trained model;

[0203] The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

[0204] In one possible implementation, the model training module is also configured as follows:

[0205] A second training model is determined based on the target parameters, wherein the parameters of the second training model correspond to the parameters of the target parameters;

[0206] The target parameters are used as training parameters for the second training model and added to the second training model.

[0207] The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

[0208] In one possible implementation, the model training module is also configured as follows:

[0209] The sample data and its labels are input into the embedding layer of the second training model to obtain the third sample encoding vector and the label encoding vector.

[0210] The third sample encoding vector and the label encoding vector are input into the attention mechanism layer of the second training model to obtain the third output data and the trained target model.

[0211] In one possible implementation, the model training module is also configured as follows:

[0212] Determine the target detection layer based on task requirements;

[0213] The target detection layer is added after the attention mechanism layer of the target model to obtain the application model.

[0214] In one possible implementation, the model training module is also configured as follows:

[0215] A third training task is performed based on the application model, the sample data, and the labels of the sample data to obtain a category detection model.

[0216] By adding label parameters to the initial parameters, target parameters are obtained, which enable the second pre-training task to be performed based on the target parameters, sample data, and the labels of the sample data to obtain the target model. Since label parameters are added to the parameters, text data and the labels of the text data can be used as training samples. That is, training can be performed on different types of text data, and the model can be transferred to multiple new domains at a lower cost, reducing the difficulty of model transfer.

[0217] The above is an illustrative scheme of a text detection model training device according to this embodiment. It should be noted that the technical solution of this text detection model training device and the technical solution of the text detection model training method described above belong to the same concept. For details not described in detail in the technical solution of the text detection model training device, please refer to the description of the technical solution of the text detection model training method described above.

[0218] Figure 5 A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0219] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0220] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0221] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 500 can also be a mobile or stationary server.

[0222] The processor 520 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described text detection model training method.

[0223] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the text detection model training method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the text detection model training method described above.

[0224] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described text detection model training method.

[0225] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the text detection model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the text detection model training method described above.

[0226] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described text detection model training method.

[0227] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the text detection model training method described above. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the text detection model training method described above.

[0228] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0229] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0230] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0231] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0232] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a text detection model, comprising: Obtain the pre-training parameters of the pre-trained model, and perform a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data; Adding label parameters to the initial parameters yields target parameters, and obtaining labels for the sample data, wherein the labels represent the type of the text data, and the initial parameters include a query matrix, a key matrix, and a value matrix; correspondingly, adding label parameters to the initial parameters to obtain target parameters includes: adding a query label matrix, a key label matrix, and a value label matrix to the initial parameters to obtain target parameters; A second pre-training task is performed based on the target parameters, the sample data, and the labels of the sample data to obtain the target model.

2. The method according to claim 1, wherein performing the first pre-training task based on the pre-training parameters and sample data to obtain initial parameters comprises: A first training model is determined based on the pre-training parameters, wherein the parameters of the first training model correspond to the parameters of the pre-training parameters; The pre-trained parameters are used as training parameters for the first training model and added to the first training model. The first training model is trained based on the sample data to obtain the initial parameters of the first training model after training.

3. The method according to claim 2, wherein training the first training model based on the sample data to obtain the initial parameters of the first training model after training includes: The sample data is input into the embedding layer of the first training model to obtain the first sample encoding vector; The first sample encoding vector is input into the attention mechanism layer of the first training model to obtain the first output data and the initial parameters after training.

4. The method according to claim 1, wherein performing the first pre-training task based on the pre-training parameters and sample data to obtain initial parameters includes: The sample data is input into the embedding layer of the pre-trained model to obtain the second sample encoding vector, wherein the parameters of the pre-trained model are the pre-training parameters; The second sample encoding vector is input into the attention mechanism layer of the pre-trained model to obtain the second output data and the initial parameters after training.

5. The method according to claim 2, further comprising, after obtaining the initial parameters of the first trained model, including: Add label parameters to the parameters of the first trained model after training to obtain the second trained model; The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

6. The method according to claim 1, wherein performing a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain the target model comprises: A second training model is determined based on the target parameters, wherein the parameters of the second training model correspond to the parameters of the target parameters; The target parameters are used as training parameters for the second training model and added to the second training model. The second training model is trained based on the sample data and the labels of the sample data to obtain the target model.

7. The method according to claim 6, wherein training the second training model based on the sample data and the labels of the sample data comprises: The sample data and its labels are input into the embedding layer of the second training model to obtain the third sample encoding vector and the label encoding vector. The third sample encoding vector and the label encoding vector are input into the attention mechanism layer of the second training model to obtain the third output data and the trained target model.

8. The method according to claim 1, further comprising, after obtaining the target model: Determine the target detection layer based on task requirements; The target detection layer is added after the attention mechanism layer of the target model to obtain the application model.

9. The method according to claim 8, further comprising, after obtaining the application model: A third training task is performed based on the application model, the sample data, and the labels of the sample data to obtain a category detection model.

10. A class detection method, comprising: The description information of the target object is input into the embedding layer of the category detection model to obtain the description encoding vector, wherein the category of the description information is at least two types, and the category detection model is obtained based on the training method of claim 1; The description encoding vector is input into the attention mechanism layer of the category detection model to obtain the description output data; The description output data is input into the detection layer of the category detection model to obtain the category information of the target object.

11. A text detection model training device, comprising: The first training module is configured to obtain the pre-training parameters of the pre-trained model, and perform a first pre-training task based on the pre-training parameters and sample data to obtain initial parameters, wherein the sample data includes at least two types of text data; The parameter determination module is configured to add label parameters to the initial parameters to obtain target parameters, and to obtain labels for the sample data, wherein the labels are type representations of the text data, and the initial parameters include a query matrix, a key matrix, and a value matrix; correspondingly, adding label parameters to the initial parameters to obtain target parameters includes: adding a query label matrix, a key label matrix, and a value label matrix to the initial parameters to obtain target parameters; The second training module is configured to perform a second pre-training task based on the target parameters, the sample data, and the labels of the sample data to obtain the target model.

12. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the model training method according to any one of claims 1 to 9 or the category detection method according to claim 10.

13. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the model training method of any one of claims 1 to 9 or the category detection method of claim 10.

14. A computer program product, characterised in that, It includes computer instructions that, when executed by a processor, implement the steps of the model training method of any one of claims 1 to 9 or the category detection method of claim 10.

Images