A model training method, a named entity recognition method, a medium, and an apparatus.

Abstract

This invention discloses a model training method, a named entity recognition method, a medium, and an apparatus, relating to the field of named entity recognition in hydrogeology. The model training method mainly includes the following steps: obtaining a generative PLM; obtaining a source sequence and a target sequence; feeding the source sequence into an encoder; and sequentially feeding the target sequence into a decoder; obtaining the conditional probability of the c-th word output by the decoder; obtaining a loss function; adjusting the model parameters; comparing the values ​​of the loss function under different model parameters; and determining the model parameters when the loss function converges to obtain the trained model. Implementing the model training method, named entity recognition method, medium, and apparatus provided by this invention can overcome the shortcomings of current named entity recognition methods in the field of hydrogeology, such as low efficiency and poor accuracy.

Description

Technical Field

[0001] This invention relates to the field of named entity recognition in hydrogeology, and more specifically, to a model training method, a named entity recognition method, a medium, and a device. Background Technology

[0002] Sufficient hydrogeological data is fundamental to hydrogeological research. Most hydrogeological data is organized in the form of massive amounts of unstructured text. Because each hydrogeological document contains limited key information about its content, much of this information is stored in unstructured text and cannot be directly applied to hydrogeological research. In practice, manually identifying and extracting data from documents is time-consuming and labor-intensive. Currently, automatically extracting hydrogeological data from emerging texts presents many challenges and is highly valuable for hydrogeological research. Named Entity Recognition (NER), a leading method for extracting information from unstructured text, has been used for hydrogeological entity extraction, extracting geological information from geological texts such as scientific literature to support key tasks in data analysis and geological interpretation, including identifying and extracting important geological concepts such as rock type, geological structure, geological age, and place names. Previous NER methods faced significant challenges in extracting entities from hydrogeological texts due to the significant differences between hydrogeological entities and texts and available public datasets. Early NER methods were dictionary- and rule-based. However, entities in the geological field are diverse and constantly updated, making it difficult for these methods to enumerate all rules and efficiently acquire emerging entities. Currently, the development of deep learning has made neural networks the mainstream method for Neural Emission Reduction (NER). In practical hydrogeological fields, manually labeled samples not only require the knowledge of domain experts but also a large amount of manual input, resulting in a shortage of usable labeled samples at present. However, neural NER models are highly dependent on a large number of manually labeled training samples to train the model, and current NER methods suffer from low efficiency and poor accuracy in the field of hydrogeology. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a named entity recognition method, a model training method, a medium, and a device to address the shortcomings of current NER methods in the field of hydrogeology, such as low efficiency and poor accuracy.

[0004] The technical solution adopted by this invention to solve its technical problem is to construct a model training method. This model is used for hydrogeological named entity recognition, including the following steps: S101: Obtain a generative PLM, which includes an encoder and a decoder. The encoder is used to encode the input sequence, and the decoder is used to generate text. The encoder and decoder are connected through a self-attention mechanism; S102: Obtain the source sequence and the target sequence; feed the source sequence into the encoder to obtain the hidden representation of the sentence; feed the target sequence into the decoder sequentially to obtain the decoded output of the target sequence; and combine the hidden representation of the sentence with the first (c-1) words t output by the decoder. 1:c-1 As input to the decoder, the output of the decoder is obtained; S103: Obtain the conditional probability of the c-th word in the decoder output, as shown in the formula. Where p(t) c |t 1:c-1 S) represents the conditional probability mentioned above, and t c For the c-th word output by the decoder, t 1:c-1 Let S be the first (c-1) words output by the decoder, S be the source sequence, and the softmax function be used to calculate the probability distribution. W is the output of the current decoder. lm b lm These are model parameters used to train the model; S104: Obtain the loss function and adjust the model parameters W. lm and b lm By comparing the values ​​of the loss function under different model parameters, the model parameters are determined when the loss function converges, and the trained model is obtained.

[0005] Furthermore, in the above model training method, the source sequence and target sequence are obtained by the following steps: obtaining the tag word set L = {l1,…,l...} |L|}; Obtain a one-to-one mapping function M:l→y, used to convert L into a label set Y={y1,…,y |L|}, where l is the tag word, y is the label of the category represented by the tag word, |L| is the number of categories, the category label is used to represent category semantics, and the tag word is used to represent each entity category in natural language form; obtain a one-to-one mapping function M′:y→l, used for mapping from label to tag word; obtain the preset sentence X={x1,…,x n The preset sentence was obtained through manual annotation. i It is the i-th word in X; obtain the entity set E = {e1,…,e...} k}, e o It is the o-th entity in E, as shown in the formula. Where 1≤i≤j≤n; obtain the input prefix P. x ={px1,…,px u} is used to describe the input task, where px i It is P x The i-th word in the sequence; obtain the output prefix P. r ={pr1,…,pr v} is used to describe the output task, where pr i It is P r The i-th word element in P, where i, j, k, n, u, and v are integers, is used to represent P. x Concatenating with X yields the source sequence S = {px1, ...,px} u ,x1,…,x n};Let X = {x1,…,x} n Entity e in} o Replace with label y o y o For the o-th category label in the label set Y, the replacement sentence R = {r1, ..., r...} is obtained. m}={x1,…,y o ,…,x n} will output the prefix P r Connecting the target sequence T with the replacement sentence R, we obtain the target sequence T = {pr1, ..., pr...} v ,r1,…,r m}

[0006] Furthermore, in the above model training method, the set of label terms is {Location, AquiferType, GroundwaterType, HydrochemicalType, RockType, StratigraphicGroupType, GeologicAge}, where Location is the location, AquiferType is the aquifer type, GroundwaterType is the groundwater type, HydrochemicalType is the hydrochemical type, RockType is the rock type, StratigraphicGroupType is the stratigraphic group type, and GeologicAge is the geological age. The label set is {Loc, Aqu, Gro, Hyd, Roc, Str, Geo}, where Loc corresponds to Location, Aqu corresponds to AquiferType, Gro corresponds to GroundwaterType, Hyd corresponds to HydrochemicalType, Roc corresponds to RockType, Str corresponds to StratigraphicGroupType, and Geo corresponds to GeologicAge.

[0007] Furthermore, the loss function in the above model training method is given by the formula... We obtain, where Σ is the summation symbol, m+v is the length of the target sequence, log is the natural logarithm, and p(t) is the summation value. c |t 1:c-1 S) represents the conditional probability described above. Let T be the loss function between the decoder output and the target sequence T.

[0008] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described model training method.

[0009] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the model training method described above.

[0010] This invention also provides a named entity recognition method for hydrogeological named entity recognition. The method includes the following steps: S701: Obtain the sentence sequence to be recognized, obtain the input prefix, which is used to describe the input task, and concatenate the input prefix with the sentence sequence to be recognized to obtain the input sequence; S702: Train the above-mentioned generative PLM according to the above model training method to obtain the training model; S703: Feed the input sequence into the training model and perform entity reasoning to obtain the labels corresponding to the named entities in the sentence sequence to be recognized.

[0011] Furthermore, in the aforementioned named entity recognition method, step S703, which involves feeding the input sequence into the training model and performing entity reasoning, includes the following steps: feeding the input sequence into the training model decoder to obtain a predicted sequence; using a sequence matching algorithm to obtain a set of matching sequences between the input sequence and the predicted sequence; and obtaining a set of input difference sequences, as shown in formula Dif. i =x-Mat={di0,di1,…,di n}, where Dif i Let x be the input set of differing sequences, Mat be the input sequence, and di be the set of matching sequences. o It is the input prefix, di n It is Dif i The nth element; obtain the set of predicted differential sequences, as shown in the formula Dif. p =p-Mat={dp0,dp1,…,dp n}, where Dif p Let p be the set of predicted differential sequences, p be the predicted sequence, dp0 be the output prefix, and dp be the output sequence. n It is Dif p The nth element; obtain the predicted tag word, as shown in formulas (1) and (2):

[0012]

[0013]

[0014] The Argmax function is used to find the index m of the tag word corresponding to the maximum value of the cosine similarity function. It is a cosine similarity function used to calculate the similarity between the i-th predicted word in the set of predicted difference sequences and the set of tag words L = {l1, ..., l2}. |L| The word similarity of each tag in the set is given by |L|, where |L| is the number of tag words in the tag word set. The encoder outputs the prediction of the i-th word in the set of differencing sequences. The output of the encoder is the tag word, α is a hyperparameter used to set the threshold for entity selection, w is the predicted tag word, and l is the output of the encoder. m Let w be the m-th tag word in the tag word set; map the predicted tag word w back to tag y, and this tag y is the recognition result.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the named entity recognition method described above.

[0016] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the named entity recognition method described above.

[0017] Implementing the model training method, named entity recognition method, medium, and device provided by this invention has the following beneficial effects:

[0018] This invention introduces a generative PLM (pre-trained language model) to implement a template-free NER (Named Entity Recognition) method. The NER task is treated as a seq2seq (sequence-to-sequence) task. Specific prefix sequences are added to sentences to construct source and target sequence samples for training, making the model more focused on the current target task. The source and target sequences are used as the input and output of the model's encoder-decoder structure, respectively, to fine-tune the language model. In the inference phase, the source sequence constructed from the input sentence is used as the input to the language model, from which target entities are extracted and decoded to obtain the target sequence. Finally, hydrogeological entities are extracted from unstructured text, thus solving the aforementioned challenges encountered by NER methods in the field of hydrogeology and overcoming the shortcomings of current NER methods in hydrogeology, such as low efficiency and poor accuracy. Attached Figure Description

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0020] Figure 1 This is a flowchart of a model training method for the field of named entity recognition in hydrogeology provided by the present invention;

[0021] Figure 2 This is a schematic diagram illustrating the construction process of the source sequence and target sequence provided by the present invention;

[0022] Figure 3 This is a schematic diagram illustrating the process of fine-tuning and training a generative pre-trained language model provided by the present invention.

[0023] Figure 4 This is a schematic diagram illustrating the process of entity reasoning on text provided by the present invention;

[0024] Figure 5 This is a structural block diagram of the computer device of the present invention. Detailed Implementation

[0025] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] Figure 1 This diagram illustrates one implementation of the model training method of this embodiment. The model training method is used in the field of hydrogeological named entity recognition and includes the following steps: obtaining a generative PLM, which includes an encoder and a decoder. The encoder is used to encode the input sequence, and the decoder is used to generate text. The encoder and decoder are connected through a self-attention mechanism; obtaining a source sequence and a target sequence; feeding the source sequence into the encoder to obtain the hidden representation of the sentence; feeding the target sequence sequentially into the decoder to obtain the decoded output of the target sequence; and combining the hidden representation of the sentence with the first (c-1) words t output by the decoder. 1:c-1 As input to the decoder, we obtain the decoder's output; we then obtain the conditional probability of the c-th word in the decoder's output, as shown in the formula. Where p(t) c |t 1:c-1 S) is the conditional probability, t c For the c-th word output by the decoder, t 1:c-1 Let S be the first (c-1) words output by the decoder, S be the source sequence, and the softmax function be used to calculate the probability distribution. W is the output of the current decoder. lm b lm These are model parameters used to train the model; obtain the loss function and adjust the model parameters W. lm and b lm By comparing the values ​​of the loss function under different model parameters, the model parameters are determined when the loss function converges, and the trained model is obtained.

[0027] Specifically, the method described above for obtaining the source and target sequences includes the following steps: obtaining a set of tag words L = {l1, ..., l...} |L|}; Obtain a one-to-one mapping function M:l→y, used to convert L into a label set Y={y1,…,y |L|}, where l is the tag word, y is the label of the category represented by the tag word, |L| is the number of categories, the category label is used to represent category semantics, and the tag word is used to represent each entity category in natural language form; obtain a one-to-one mapping function M′:y→l, used for mapping from label to tag word; obtain the preset sentence X={x1,…,x n The preset sentence was obtained through manual annotation. i It is the i-th word in X; obtain the entity set E = {e1,…,e...} k}, e o It is the o-th entity in E, as shown in the formula. Where 1≤i≤j≤n; obtain the input prefix P. x ={px1,…,px u} is used to describe the input task, where px i It is P x The i-th word in the sequence; obtain the output prefix P. r ={pr1,…,pr v} is used to describe the output task, where pr i It is P r The i-th word element in P, where i, j, k, n, u, and v are integers, is used to represent P. x Concatenating with X yields the source sequence S = {px1, ...,px} u ,x1,…,x n};Let X = {x1,…,x} n Entity e in} o Replace with label y o y o For the o-th category label in the label set Y, the replacement sentence R = {r1, ..., r...} is obtained. m}={x1,…,y o ,…,x n} will output the prefix P r Connecting the sentence R with the replacement sentence yields the target sequence T = {pr} 12 ,…,pr v ,r 12 ,…,r m}

[0028] Specifically, in the aforementioned model training method, the set of label terms is {Location, AquiferType, GroundwaterType, HydrochemicalType, RockType, StratigraphicGroupType, GeologicAge}, where Location is the location, AquiferType is the aquifer type, GroundwaterType is the groundwater type, HydrochemicalType is the hydrochemical type, RockType is the rock type, StratigraphicGroupType is the stratigraphic group type, and GeologicAge is the geological age. The set of label terms is {Loc, Aqu, Gro, Hyd, Roc, Str, Geo}, where Loc corresponds to Location, Aqu corresponds to AquiferType, Gro corresponds to GroundwaterType, Hyd corresponds to HydrochemicalType, Roc corresponds to RockType, Str corresponds to StratigraphicGroupType, and Geo corresponds to GeologicAge.

[0029] Specifically, the loss function described in the above model training method is given by the formula... We obtain, where Σ is the summation symbol, m+v is the length of the target sequence, log is the natural logarithm, and p(t) is the summation value. c |t 1:c-1 S) is the conditional probability. is the loss function between the decoder output and the target sequence T, used to represent how close the model's prediction is to the true result. The smaller the value of the loss function, the greater the closeness.

[0030] Specifically, the above model training method can also be implemented through the following steps:

[0031] like Figure 2 As shown, obtain the tag word set L = {l1, ..., l |L|}; Obtain a one-to-one mapping function M:l→y, used to convert L into a label set Y={y1,…,y |L|}, where l is the tag word, y is the category label represented by the tag word, |L| is the number of categories, the category label is used to represent category semantics, and the tag word is used to represent each entity category in natural language form; obtain a one-to-one mapping function M′:y→l, which is used for mapping from tag to tag word;

[0032] Get the sentence X = {x1, ..., x} n}, x iIt is the i-th word in X; obtain the entity set E = {e1,…,e...} k}, e o It is the o-th entity in E, as shown in the formula. Where 1≤i≤j≤n; obtain the input prefix P. x ={px1,…,px u} is used to describe the input task, where px i It is P x The i-th word in the sequence; obtain the output prefix P. r ={pr1,…,pr v} is used to describe the output task, where pr i It is P r The i-th word element in P, where i, j, k, n, u, and v are integers, is used to represent P. x Concatenating with X yields the source sequence S = {px1, ...,px} u ,x1,…,x n};

[0033] Let X = {x1, ..., x} n Entity e in} o Replace with label y o y o For the o-th category label in the label set Y, the replacement sentence R = {r1, ..., r...} is obtained. m}={x1,…,y o ,…,x n} will output the prefix P r Connecting the target sequence T with the replacement sentence R, we obtain the target sequence T = {pr1, ..., pr...} v ,r1,…,r m The source and target sequences are used for model training.

[0034] like Figure 3 As shown, a generative pre-trained language model is obtained. This generative pre-trained language model is a language model that adopts the standard Transformer structure, which includes an encoder and a decoder. The encoder is used to encode the input sequence, and the decoder is used to generate text. The encoder and decoder are connected through a self-attention mechanism.

[0035] The source sequence is fed into the encoder to obtain the hidden representation of the sentence, as shown in the formula. Where h Enc This is a hidden representation of the sentence. This represents the encoding operations of the encoder, used to transform text into a computable vector. The source sequence is represented as n1; the target sequence is fed into the decoder sequentially to obtain the decoded output of the target sequence; the hidden representation h of the sentence is then processed. EncThe first (c-1) words t output by the decoder 1:c-1 As input to the decoder, the output of the decoder is obtained. This indicates the decoding operation of the decoder;

[0036] Get t c The conditional probability is given by the formula. Where p(t) c |t 1:c-1 X) is to obtain t c The conditional probability, The softmax function is used to calculate the probability distribution, d h It is the hidden dimension of the encoder, |V| is the vocabulary size of the pre-trained model, and W is the hidden dimension of the encoder. lm b lm These are model parameters used for fine-tuning and training the model.

[0037] Obtain the loss function, as shown in the formula. in The loss function between the decoder output and the target sequence T represents how close the model's prediction is to the actual result. The smaller the value of the loss function, the closer the result is to the actual result.

[0038] Adjust the model parameters, compare the values ​​of the loss function under different model parameters, determine the model parameters when the loss function converges, and obtain the trained model.

[0039] This embodiment provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the model training method described above.

[0040] The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memory.

[0041] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the model training method used in the named entity recognition method described above to obtain the steps of the model training method described above.

[0042] like Figure 5As shown, the computer device may include: at least one processor 151, such as a CPU (Central Processing Unit), at least one communication interface 153, memory 154, and at least one communication bus 152. The communication bus 152 is used to enable communication between these components. The communication interface 153 may include a display screen and a keyboard; optionally, the communication interface 153 may also include a standard wired interface or a wireless interface. The memory 154 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk drive. Optionally, the memory 154 may also be at least one storage device located remotely from the processor 151. The memory 154 stores application programs, and the processor 151 calls the program code stored in the memory 154 to execute any of the above-described method steps.

[0043] The communication bus 152 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 152 can be divided into an address bus, a data bus, and a control bus, etc. For ease of representation, Figure 5 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0044] The memory 154 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 154 may also include a combination of the above types of memory.

[0045] The processor 151 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0046] The processor 151 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0047] Optionally, the memory 154 is also used to store program instructions. The processor 151 can call the program instructions to implement the model training method as described in this embodiment.

[0048] Below, this embodiment will briefly describe the named entity recognition method: This named entity recognition method is used for hydrogeological named entity recognition, and the method includes the following steps: S701: Obtain the sentence sequence to be recognized, obtain the input prefix, which is used to describe the input task, and concatenate the input prefix with the sentence sequence to be recognized to obtain the input sequence; S702: Train the generative PLM according to the above model training method to obtain the training model; S703: Feed the input sequence into the training model and perform entity reasoning to obtain the labels corresponding to the named entities in the sentence sequence to be recognized.

[0049] Specifically, in the named entity recognition method described above, step S703, which involves feeding the input sequence into the training model and performing entity reasoning, includes the following steps: feeding the input sequence into the decoder of the training model to obtain a predicted sequence; using a sequence matching algorithm to obtain a set of matching sequences between the input sequence and the predicted sequence; and obtaining a set of input difference sequences, as shown in the formula Dif. i =x-Mat={di0,di1,…,di n}, where Dif i Let x be the input set of differing sequences, Mat be the input sequence, and di be the set of matching sequences. o It is the input prefix, di n It is Dif i The nth element; obtain the set of predicted differential sequences, as shown in the formula Dif. p =p-Mat={dp0,dp1,…,dp n}, where Dif pLet p be the set of predicted differential sequences, p be the predicted sequence, dp0 be the output prefix, and dp be the output sequence. n It is Dif p The nth element; obtain the predicted tag word, as shown in formulas (1) and (2):

[0050]

[0051] The Argmax function is used to find the index m of the tag word corresponding to the maximum value of the cosine similarity function. It is a cosine similarity function used to calculate the similarity between the i-th predicted word in the set of predicted difference sequences and the set of tag words L = {l1, ..., l2}. |L| The word similarity of each tag in the set is given by |L|, where |L| is the number of tag words in the tag word set. The encoder outputs the prediction of the i-th word in the set of differencing sequences. The output of the encoder is the tag word, α is a hyperparameter used to set the threshold for entity selection, w is the predicted tag word, and l is the output of the encoder. m Let w be the m-th tag word in the tag word set; map the predicted tag word w back to tag y, and this tag y is the recognition result.

[0052] In some embodiments, the sequence matching algorithm in this named entity recognition method is the SequenceMatcher function.

[0053] This embodiment provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the named entity recognition method described above.

[0054] The aforementioned storage medium may be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0055] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the named entity recognition method described above to obtain the steps of the named entity recognition method described above.

[0056] like Figure 5As shown, the computer device may include: at least one processor 151, such as a CPU (Central Processing Unit), at least one communication interface 153, memory 154, and at least one communication bus 152. The communication bus 152 is used to enable communication between these components. The communication interface 153 may include a display screen and a keyboard; optionally, the communication interface 153 may also include a standard wired interface or a wireless interface. The memory 154 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk drive. Optionally, the memory 154 may also be at least one storage device located remotely from the processor 151. The memory 154 stores application programs, and the processor 151 calls the program code stored in the memory 154 to execute any of the above-described method steps.

[0057] The communication bus 152 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 152 can be divided into an address bus, a data bus, and a control bus, etc. For ease of representation, Figure 5 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0058] The memory 154 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 154 may also include a combination of the above types of memory.

[0059] The processor 151 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0060] The processor 151 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0061] Optionally, the memory 154 is also used to store program instructions. The processor 151 can invoke the program instructions to implement the named entity recognition method as described in this embodiment.

[0062] In some embodiments, the named entity recognition method described above can be implemented in the following manner.

[0063] This embodiment proposes a named entity recognition method, particularly suitable for the field of hydrogeology. This method introduces cue learning (PL) to improve entity extraction from geoscientific texts. PL has shown great potential in handling few-shot problems by bridging the gap between downstream and pre-training tasks. This embodiment utilizes generative PLM to transform the NER task into a natural language generation task. During training, it fine-tunes the language model by using cue samples as input to the generative model. Cue prefixes are designed for the input text to describe the named entity recognition task, aiming to encourage the model to focus on the named entity recognition task and guide the generation of sentences oriented towards the named entity recognition task. During inference, the replaced phrases in the generated sentences are extracted as entities, and their types are labeled with the tag words that replaced them in the sentence.

[0064] The goal of the named entity recognition task is to identify entities in a given sentence X = {x1, ..., x2}. n In the}, identify entity E = {e1,…,e k}, where x i It is the i-th token in X, and the o-th entity is represented as a... The NER task predicts each entity e as entity label l, and then maps l to class label y through a mapping function.

[0065] I. Overall Framework

[0066] This method treats the NER task as a sequence-to-sequence (seq2seq) task using generative PLM to extract hydrogeological entities, and encodes and decodes the sequences using the PLM encoder and decoder respectively. The seq2seq model requires two sentence sequences as cue samples for training. The source sequences for this model are... From the input prefix P x And the input statement X = {x1, ..., x n Composed of}. The target sequence is m1=m+|P r |, derived from the output prefix P r And the replacement sentence R = {r1, ..., r m} is composed of R, which is transformed by X.

[0067] This embodiment consists of three modules: cue sequence construction, backbone network training, and result inference. To obtain the cue sequences used for training, namely the source sequence and the target sequence, this embodiment constructs the cue sequences in the cue sequence construction module and uses them to train the backbone network. In the backbone network training module, the pre-trained model is fine-tuned to obtain the fine-tuned network. Since this method differs from traditional sequence labeling models, this embodiment performs entity inference on the sentences generated by the network in the result inference module, ultimately obtaining the target entity. These three modules will be described in the following three subsections.

[0068] II. Hints for Sequence Construction

[0069] Typically, seq2seq training samples consist of two sentences: a source sequence and a target sequence. Therefore, it is necessary to construct a cue sequence, consisting of the source sequence and the target sequence, for model training. The steps for constructing the cue sequence are as follows:

[0070] 1. Since the labels may not be sufficient to contain the category semantics, in order to adapt to the seq2seq task, before constructing the prompt sequence, this embodiment manually defines a label word for each category according to the category semantics, so as to represent each entity category in natural language form. The corresponding label words for each category are shown in the table below.

[0071] The table shows the tag settings for each category.

[0072]

[0073] 2. To establish a mapping function between categories and tags: A one-to-one mapping function M: l → y is established, which maps the tag set L = {l1, ..., l...} used when constructing the cue sequence. |L|} is converted into a label set Y = {y1, ..., y2}. |L|Let l be a tag word, y be the category label represented by the tag word, and |L| be the number of categories. The rules are as follows: category label “Loc” corresponds to the tag word “location”, category label “Aqu” corresponds to the tag word “aquifer”, and so on. Simultaneously, a one-to-one mapping function M′:y→l is established to map the tag word to the category label during inference, such as tag word “location” corresponding to category label “Loc”, tag word “aquifer” corresponding to category label “Aqu”, and so on.

[0074] 3. For an input sentence, the process of obtaining the tags is as follows: For a given input sentence X, its entity set is represented as follows: Where k is the number of entities, and n is the sentence length. Entity e o The category label is denoted as y o Its tag words can be mapped to l according to the mapping function. o =M′(y o ).

[0075] 4. To obtain the source sequence from the prompt sequence, the construction method is as follows: First, use the task description text "replaceentity to label:" as the input prefix P. x ={px1,…,px5}, helps the PLM model match tasks. Then P x Concatenating x yields the source sequence S = {px1,…,px5,x1,…,x} n}

[0076] 5. To obtain the target sequence from the prompt sequence, the construction method is as follows: Automatically replace the entity words in X with their corresponding tag words using a script based on the mapping function M, resulting in the replaced sentence R. Specifically, replace the span of each entity word in S with its corresponding tag word, while leaving non-entity words unchanged. Formally, X = {x1, ..., x...} n Entity e in} o Replace the continuous span with the tag word y o To form R = {r1, ..., r} m}={x1,…,y o ,…,x n The similar task description text "replaced sentence is:" is denoted as the output prefix P. r ={pr1,…,pr4}, aiming to make the model understand the output format and correspond it to the input prefix. Finally, by P r Connecting R and R yields the target sequence T = {pr1,…,pr4,r1,…,r} m}

[0077] The overall process is as follows Figure 2 As shown, for the given input sentence "Turbidite occurs in Apenninemountains.", the span "Apennine mountains" is an entity, and its corresponding category label is "Loc". The label word obtained through M′ is "Location". The replaced sentence "Turbidite occurs in Location." is obtained through the above process. Finally, the prompt prefix is ​​added to the input sentence and the replacement sentence to obtain the source sequence and the target sequence, respectively. The constructed source sequence is "replace entity to label:Turbiditeoccurs in Apenninemountains.", and the constructed target sequence is "replaced sentence is:Turbidite occurs inLocation.".

[0078] like Figure 2 As shown, the construction process of the prompt sequence mainly includes: ① obtaining the replacement sentence by replacing entities in the input sentence; in processes ② and ③, the input sentence and the input prefix are concatenated, and the replacement sentence and the output prefix are concatenated to obtain the source sequence and the target sequence.

[0079] III. Backbone Network Training

[0080] In this method, this embodiment selects a generative PLM as the pre-trained model. Here, generative PLM refers to a language model using the standard Transformer structure. The difference between BERT and BERT is that BERT only includes the encoder in the Transformer structure and is commonly used for sequence classification tasks; while BERT includes both an encoder and a decoder. The encoder encodes the input sequence, and the decoder generates the text. The encoder and decoder are connected through a self-attention mechanism. Representative models include MASS, T5, and BART, and are typically used for text generation tasks. Its pre-training task is to add noise to the original text and restore it. Taking BART as an example, BART defines the pre-training objective as destructive document recovery—in the encoder-decoder architecture, the encoder's input is the destructive document sequence, and the decoder's output is the recovered document sequence. BART constructs a cross-entropy loss between the recovered document and the undestructed document. The suggestion strategy proposed in this method essentially replaces entities in the input sentence and reconstructs them into labeled words, thereby narrowing the gap between the downstream task and the pre-training task and reducing the dependence on the target data. In addition, there are a large number of multi-token entities in the field of hydrogeology, and natural language generation tasks can achieve a more appropriate and flexible mapping between entities and token words.

[0081] The overall process is shown in the figure. A generative PLM with an encoder-decoder structure fine-tunes by receiving input and output sentences. The encoder obtains a semantic vector C, and the decoding stage uses this semantic vector C to obtain individual outputs. The training steps for the backbone network are as follows:

[0082] 1. Constructing training samples for model input: During the training phase, given the input sentence X, the source sequence S and the target sequence T are constructed according to the prompt sequence method in Part 2.

[0083] 2. To enable the model to receive sentences, a hidden representation of the sentence is needed: For a cue sequence pair (S, T), this embodiment passes S through the model's encoder and obtains the hidden representation of the sentence using the formula... Obtain its hidden representation h Enc ,in This represents the encoding operation of the encoder, which aims to transform text into a computable vector.

[0084] 3. To complete the model training task, T needs to be input into the model's decoder, and h needs to be calculated. Enc Conditional probability of predicting a word: If we want to predict the c-th word t in a sentence... c The hidden representation of the sentence h Enc The first c words t output by the decoder 1:c-1 As input to the decoder, the Transformer's self-attention mechanism is utilized, allowing it to adjust the model's attention to each word based on its importance, focusing on key information parts to obtain the decoder's output. Calculate t c conditional probability in |V| represents the vocabulary size of the pre-trained model, and W... lm ,b lm These are the parameters of the pre-trained model.

[0085] 4. To update the model parameters, it is necessary to calculate the loss function and optimize the objective function: using cross-entropy loss. The loss function, acting as the link between the decoder output and the target sequence T, takes the model's predictions and the actual results as inputs and outputs a non-negative real number. A smaller loss function value indicates that the model's predictions are closer to the actual results. The objective function, which is the function this embodiment aims to optimize, includes the loss function and other terms such as regularization. By optimizing the objective function, this embodiment can find the optimal model parameters that minimize the loss function value.

[0086] like Figure 3As shown, training is performed on the backbone network. This method uses a generative PLM as a pre-trained model, whose encoder-decoder structure is fine-tuned by receiving source and target sequences.

[0087] IV. Result Reasoning

[0088] Since this method treats sequence labeling as a text generation task, it requires inference to obtain the predicted entities from the generated text. During the inference phase, based on the second part, an input sequence x is constructed from the given input text as the source sequence for the model, such as... Figure 4 As shown, this method treats the result reasoning process as a sequence alignment process, and the reasoning steps are as follows:

[0089] 1. To infer entities, text needs to be input into a fine-tuned model to obtain output: input x into the fine-tuned model and use the output of the model decoder as the predicted sequence p. Based on the form of the target sequence, the structure of the output sequence after decoding by the fine-tuned model will be consistent with the target sequence. This is equivalent to transforming the entity prediction process into the process of the model replacing entities in the input sequence with label words.

[0090] 2. After obtaining the output text, it is necessary to combine it with the input text and use a sequence matching algorithm to infer the entity words: After x passes through the model, the fine-tuned network generates a predicted sequence p, and then the sequence matching algorithm is used to find the matching sequence set Mat = Match(x,p) between x and p. Here, Match(x,p) is the sequence matching algorithm. In this embodiment, the SequenceMatcher function is used, a function that compares two strings and returns data based on their similarity, to achieve sequence matching, finding the same subsequences between two given sequences, and naming them the matching sequence set.

[0091] 3. During the matching process, the predicted entity is obtained by finding the differences: by calculating the difference between x and Mat and the difference between y and Mat, the set of difference sequences Dif is obtained respectively. i =x-Mat and Dif p =p-Mat. Where Dif i ={di0,di1,…di n},Dif p ={dp0,dp1,…dp n},di0 is the input prefix P x ,di n It is Dif i The i-th element, It is the set of input subsequences before prediction that correspond to the predicted sequence, and dp0 is the output prefix P. r dp n It is Dif pThe i-th element, It is the set of predicted sequences.

[0092] Unlike traditional NER tasks, most entity words and tag words in the hydrogeological field are not in the PLM vocabulary. The words generated by this model are sometimes in the tag word set (described in Part II), but some words and tags are similar but have been scrambled or modified, i.e., "suspected" tag words. To distinguish whether a "suspected" tag word represents a word in the tag word set, this embodiment sets an entity selection threshold. When the similarity between a "suspected" tag word and an existing tag word exceeds this threshold, the "suspected" tag word is considered an existing tag word. Otherwise, it is considered a non-entity tag word, i.e., no tag was predicted.

[0093] 4. To eliminate the influence of suspected entities, entity selection threshold filtering is required: before calculating similarity, [the threshold is set to - the ... Elements in the text are considered "suspected" entities. Elements within the range are considered "suspected" tags. DP is calculated using cosine similarity sim(h1,h2). i With the tag set L = {l1, ..., l |L| The word similarity of each tag in the table is calculated. The tag words are obtained using the following formula:

[0094]

[0095] Where dp i It is Dif p The i-th predicted word in the text, α is a hyperparameter representing the threshold for entity selection. This embodiment obtains the threshold value from dp. i The index m of the tag word with the highest similarity that exceeds the entity selection threshold is used to assign the corresponding tag word l in L. m w is used as the predicted tag word.

[0096] 5. To obtain the category corresponding to the entity, a mapping function is needed: Finally, the mapping function M is used to map the predicted label w back to the label l. According to the label settings in Part 2, if w is the label "location", it is mapped back to the label "LOC" according to M. The input sequence can generate all entities at once to obtain the output sentence.

[0097] The overall process is shown in the figure. Given the sequence "replace entity to label: Turbidite occurs in Apennine mountains.", if the model generates the sequence "replaced sentence is: Turbidite occurs in Location.", after sequence matching, the predicted label word for the span "Apennine mountains" is inferred to be "Location", and finally the entity category for that span is obtained.

[0098] like Figure 4 The following describes the reasoning process: The input sequence is fed into a seq2seq model to obtain the predicted sequence. Then, sequence matching is performed on the two text sequences to extract entity words and predicted words from processes ① and ②, respectively. Process ③ calculates the similarity between each predicted word and each tag word in the tag word set. In process ④, the tag word with the highest similarity is selected and mapped back to a tag. Finally, process ⑤ obtains the tags corresponding to the real words in the input sequence.

[0099] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A model training method, wherein the model is used for hydrogeological named entity recognition, characterized in that, Includes the following steps: S101: Obtain a generative PLM, which includes an encoder and a decoder. The encoder is used to encode the input sequence, and the decoder is used to generate text. The encoder and decoder are connected through a self-attention mechanism. S102: Obtain the source sequence and target sequence; feed the source sequence into the encoder to obtain the hidden representation of the sentence; feed the target sequence into the decoder sequentially to obtain the decoded output of the target sequence; combine the hidden representation of the sentence with the first (c-1) words output by the decoder. As input to the decoder, the output of the decoder is obtained; S103: Obtain the conditional probability of the c-th word output by the decoder, formula ,in Let be the conditional probability. For the c-th word output by the decoder, Let S be the first (c-1) words output by the decoder, S be the source sequence, and the softmax function be used to calculate the probability distribution. W is the output of the current decoder. lm b lm These are model parameters used to train the model; S104: Obtain the loss function and adjust the model parameters W. lm and b lm By comparing the values ​​of the loss function under different model parameters, the model parameters are determined when the loss function converges, and the trained model is obtained. The source sequence and target sequence are obtained through the following steps: Get the set of tags ; Get a one-to-one mapping function , used to Convert to a collection of tags ,in For tag words, The label represents the category of the tag word. The number of categories is given, where the category labels represent category semantics, and the label terms represent each entity category in natural language form; a one-to-one mapping function is obtained. , used for mapping from tags to tag words; Get preset sentences The preset sentences were obtained through manual annotation. yes The first in Each word element; obtain the entity set , yes The o-th entity in the formula Get the input prefix , used to describe the input task, where yes The first in Each word; obtain the output prefix. , used to describe the output task, where yes The first in There are 1 word elements, where i, j, k, n, u, and v are integers. and The source sequence is obtained by joining. ; Will medium entity Replace with tags , For a set of tags The o-th category label yields the replacement sentence. , will output prefix and replace sentences Connect to obtain the target sequence ; The tag set is {Location, AquiferType, GroundwaterType, HydrochemicalType, RockType, StratigraphicGroupType, GeologicAge}, where Location is the location, AquiferType is the aquifer type, GroundwaterType is the groundwater type, HydrochemicalType is the hydrochemical type, RockType is the rock type, StratigraphicGroupType is the stratigraphic group type, and GeologicAge is the geological age. The tag set is {Loc, Aqu, Gro, Hyd, Roc, Str, Geo}, where Loc corresponds to Location, Aqu corresponds to AquiferType, Gro corresponds to GroundwaterType, Hyd corresponds to HydrochemicalType, Roc corresponds to RockType, Str corresponds to StratigraphicGroupType, and Geo corresponds to GeologicAge.

2. The model training method according to claim 1, characterized in that, The loss function is given by the formula We obtain, where Σ is the summation symbol, m+v is the length of the target sequence, and log is the natural logarithm. Let be the conditional probability. For decoder output and target sequence The loss function between them.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the model training method as described in any one of claims 1-2.

4. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the model training method as described in any one of claims 1-2.

5. A named entity recognition method for hydrogeological named entity recognition, characterized in that, The method includes the following steps: S701: Obtain the sentence sequence to be recognized, obtain the input prefix, the input prefix is ​​used to describe the input task, and concatenate the input prefix with the sentence sequence to be recognized to obtain the input sequence; S702: The model training method as described in any one of claims 1-2 is used to train the generative PLM to obtain a trained model; S703: Feed the input sequence into the training model and perform entity reasoning to obtain the labels corresponding to the named entities in the sentence sequence to be identified.

6. The named entity recognition method according to claim 5, characterized in that, Step S703, which involves feeding the input sequence into the training model and performing entity reasoning, includes the following steps: The input sequence is fed into the decoder of the trained model to obtain the predicted sequence; Using sequence matching algorithms, obtain a set of matching sequences between the input sequence and the predicted sequence; Obtain the set of input difference sequences, formula Dif i = x – Mat = {di0, di1,…,di n }, where Dif i Let x be the input set of differing sequences, Mat be the input sequence, and di be the set of matching sequences. o It is the input prefix, di n It is Dif i The nth element; Obtain the predicted differential sequence set, formula Dif p = p – Mat = {dp0, dp1,…, dp n }, where Dif p Let p be the set of predicted differential sequences. It is the output prefix. yes The nth element; The predicted tag words are obtained, as shown in formulas (1) and (2): （1） （2） The argmax function is used to find the index of the tag word corresponding to the maximum value of the cosine similarity function. , It is a cosine similarity function used to calculate the first digit of the predicted difference sequence in the set. A set of predicted words and tags The word similarity of each tag word in the set, where |L| is the number of tag words in the tag word set. To predict the first in the set of differential sequences The predicted words are output by the encoder. The output of the encoder for the tag words. These are hyperparameters used to set the threshold for entity selection. To predict tag words, l m The m-th tag in the tag set; Predicted tags Map back to label The label y is the recognition result.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the named entity recognition method as described in any one of claims 5-6.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the named entity recognition method as described in any one of claims 5-6.

Images