Text information extraction method and device, electronic equipment and storage medium

Abstract

This application provides a text information extraction method, apparatus, electronic device, and storage medium, belonging to the field of financial technology technology. The method includes: clustering target words in the original corpus according to a preset clustering center vector to obtain clustered upper-level word segments; masking the original corpus according to the clustered upper-level word segments to obtain a first masked corpus and masked category labels; masking the original corpus according to the target words to obtain a second masked corpus; inputting the first masked corpus into an initial language model for prediction, and adjusting the parameters of the initial language model based on the prediction data to obtain a candidate language model; inputting the second masked corpus into the candidate language model for prediction, and adjusting the parameters of the candidate language model based on the prediction data to obtain a target language model; and inputting the target text into the target language model to extract text information to obtain extracted text data. The embodiments of this application can effectively improve the accuracy of text information extraction.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a method and apparatus for extracting text information, an electronic device, and a storage medium. Background Technology

[0002] With the increasing application of technologies (such as big data, cloud computing, and blockchain) in the financial sector, the development of banking systems, transaction systems, and order systems under fintech is accelerating, generating ever-increasing volumes of business data, transaction data, and payment data. To achieve machine intelligence technology for text structuring and better manage these massive amounts of text data, entity information extraction is currently used to extract entity information such as parts of speech, location, and company from the text data, and then construct a text structure based on the extracted entity text. Entity information extraction is a crucial task in natural language processing. Related technologies typically use pre-trained language models as encoders when extracting entity information from text, and then use an integrated decoder to determine the content to be extracted. Furthermore, the language models in these technologies are usually constructed using structures such as pre-trained Bidirectional Encoder Representation from Transformers (BERT) and RoBERTa. Language models are trained through pre-training tasks, primarily using Masked Language Modeling (MLM). This involves masking a word in the text and then using context words to predict the current word. However, when extracting information from the text, MLM tasks often fail to learn the relationships between multiple related words, leading to incomplete or incorrect word extraction and reduced accuracy. Therefore, improving the accuracy of text information extraction has become a pressing technical problem. Summary of the Invention

[0003] The main objective of this application is to provide a text information extraction method, apparatus, electronic device, and storage medium, which aims to improve the accuracy of text information extraction.

[0004] To achieve the above objectives, a first aspect of this application proposes a text information extraction method, the method comprising:

[0005] Obtain the original corpus and the target corpus words of the original corpus;

[0006] The target corpus words are clustered according to the preset cluster center vector to obtain clustered upper-level word segmentation;

[0007] The original corpus is subjected to a first masking process based on the clustering and upper-level word segmentation to obtain a first masked corpus and a masking category label for the first masked corpus.

[0008] The original corpus is subjected to a second masking process based on the target corpus words to obtain a second masked corpus.

[0009] The first masked corpus is input into a preset initial language model for corpus category prediction processing to obtain category prediction data. The parameters of the initial language model are adjusted according to the category prediction data and the masked category labels to obtain a candidate language model.

[0010] The second masked corpus is input into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data. The parameters of the candidate language model are adjusted according to the word segmentation prediction data and the target corpus words to obtain the target language model.

[0011] The acquired target text is input into the target language model for text information extraction, resulting in extracted text data.

[0012] In some embodiments, the step of clustering the target corpus words according to a preset clustering center vector to obtain clustered upper-level word segmentation includes:

[0013] Obtain the preset cluster center vector;

[0014] The target corpus words are segmented and their features are extracted to obtain target corpus word vectors;

[0015] The vector distance between the target corpus word vectors and the cluster center vectors is calculated to obtain corpus word clustering data.

[0016] The clustered upper-level words are determined based on the corpus word clustering data.

[0017] In some embodiments, before performing a first masking process on the original corpus based on the clustering-based upper-level word segmentation to obtain a first masked corpus and masked category labels for the first masked corpus, the method further includes:

[0018] Obtain the target corpus word vectors under the cluster center vectors to obtain the clustered corpus word vectors;

[0019] The target clustering vector is obtained by calculating the vector mean of the word vectors in the clustered corpus.

[0020] Update the cluster center vector based on the target cluster vector.

[0021] In some embodiments, the initial language model includes a corpus encoding layer and a first fully connected layer. The step of inputting the first masked corpus into a preset initial language model for corpus category prediction processing to obtain category prediction data, and adjusting the parameters of the initial language model based on the category prediction data and the masked category labels to obtain a candidate language model, includes:

[0022] The first masked corpus is processed by the corpus coding layer to obtain the first corpus coding features;

[0023] The first fully connected layer is used to perform corpus category prediction on the encoded features of the first corpus to obtain category prediction data.

[0024] The corpus category loss is calculated based on the category prediction data and the mask category labels to obtain category loss data;

[0025] The parameters of the initial language model are adjusted based on the category loss data to obtain the candidate language model.

[0026] In some embodiments, the initial language model further includes a second fully connected layer, wherein the weight dimension of the first fully connected layer is different from the weight dimension of the second fully connected layer;

[0027] The step of inputting the second masked corpus into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data, and adjusting the parameters of the candidate language model based on the word segmentation prediction data and the target corpus words to obtain the target language model includes:

[0028] The second masked corpus is processed by the corpus coding layer to obtain the second corpus coding features;

[0029] The second fully connected layer is used to perform word segmentation prediction on the encoded features of the second corpus to obtain word segmentation prediction data.

[0030] Based on the word segmentation prediction data and the target corpus words, the word segmentation loss is calculated to obtain the word segmentation loss data;

[0031] The candidate language model is adjusted based on the word segmentation loss data to obtain the target language model.

[0032] In some embodiments, the step of calculating the segmentation loss based on the segmentation prediction data and the target corpus words to obtain segmentation loss data includes:

[0033] The number of masking codes is calculated on the second masked corpus to obtain the number of word segmentation masks;

[0034] The word segmentation loss is calculated based on the word segmentation prediction data and the number of word segmentation masks to obtain the word segmentation loss data.

[0035] In some embodiments, obtaining the original corpus and the target corpus words of the original corpus includes:

[0036] Obtain the original corpus;

[0037] The original corpus is segmented to obtain the target corpus words.

[0038] To achieve the above objectives, a second aspect of this application provides a text information extraction apparatus, the apparatus comprising:

[0039] The corpus acquisition module is used to acquire the original corpus and the target corpus words of the original corpus;

[0040] The clustering module is used to cluster the target corpus words according to the preset cluster center vector to obtain clustered upper-level word segments;

[0041] The first masking processing module is used to perform a first masking process on the original corpus based on the clustering upper-level word segmentation to obtain a first masked corpus and a masking category label for the first masked corpus.

[0042] The second masking processing module is used to perform second masking processing on the original corpus according to the target corpus words to obtain the second masked corpus;

[0043] The corpus category prediction module is used to input the first masked corpus into a preset initial language model for corpus category prediction processing to obtain category prediction data, and to adjust the parameters of the initial language model according to the category prediction data and the masked category labels to obtain a candidate language model.

[0044] The corpus word segmentation prediction module is used to input the second masked corpus into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data, and adjust the parameters of the candidate language model according to the word segmentation prediction data and the target corpus words to obtain the target language model;

[0045] The information extraction module is used to input the acquired target text into the target language model to extract text information and obtain text extraction data.

[0046] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0047] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0048] This application proposes a text information extraction method, apparatus, electronic device, and storage medium. It acquires original corpus and target words from the original corpus, and clusters the target words according to a preset clustering center vector to obtain clustered upper-level word segments. Then, it performs a first masking process on the original corpus based on the clustered upper-level word segments to obtain a first masked corpus and its masked category labels. Next, it performs a second masking process on the original corpus based on the target words to obtain a second masked corpus. Then, in a first pre-training task phase, the first masked corpus is input into a preset initial language model for corpus category prediction processing to obtain category prediction data. The parameters of the initial language model are adjusted based on the category prediction data and the masked category labels to obtain a candidate language model. In a second pre-training task phase, the second masked corpus is input into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data. The parameters of the candidate language model are adjusted based on the word segmentation prediction data and the target words to obtain the target language model. Finally, the acquired target text is input into the target language model for text information extraction, yielding extracted text data. This embodiment of the application, through a two-stage pre-training task, can obtain a target language model with higher accuracy in text information extraction. This target language model enhances the correlation between multiple related words in the text, effectively avoiding problems such as incomplete or erroneous word extraction. Therefore, when the acquired target text is input into the target language model of this application for text information extraction, the accuracy of text information extraction can be effectively improved. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a text information extraction method provided in an embodiment of this application;

[0050] Figure 2 yes Figure 1 The flowchart of step S110 in the middle;

[0051] Figure 3 yes Figure 1 The flowchart of step S120 in the middle;

[0052] Figure 4 This is another flowchart illustrating the text information extraction method provided in this application embodiment;

[0053] Figure 5 yes Figure 1 The flowchart of step S150 in the middle;

[0054] Figure 6 yes Figure 1 The flowchart of step S160 in the process;

[0055] Figure 7 yes Figure 6 The flowchart of step S630 in the middle;

[0056] Figure 8 This is a schematic diagram of the structure of the text information extraction device provided in the embodiments of this application;

[0057] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] First, let's analyze some of the terms used in this application:

[0062] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0063] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0064] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0065] K-means clustering algorithm is an iterative clustering analysis algorithm. Its steps are as follows: First, randomly select K candidate objects as initial cluster centers to divide the initial objects into K groups. Then, calculate the distance between each initial object and each initial cluster center, and assign each initial object to the nearest cluster center. Each cluster center and the objects assigned to it represent a cluster. Each time an object is assigned, the cluster centers are recalculated based on the existing objects in the cluster. This process is repeated until a termination condition is met. Furthermore, the termination condition can be that no objects are reassigned to different clusters, or a preset number of cluster centers are reached, at which point the sum of squared errors reaches a local minimum.

[0066] With the increasing application of technologies (such as big data, cloud computing, and blockchain) in the financial sector, the development of banking systems, transaction systems, and order systems under fintech is accelerating, generating ever-increasing volumes of business data, transaction data, and payment data. To achieve machine intelligence technology for text structuring and better manage these massive amounts of text data, entity information extraction is currently used to extract entity information such as parts of speech, location, and company from the text data, and then construct a text structure based on the extracted entity text. Entity information extraction is a crucial task in natural language processing. Related technologies typically use pre-trained language models as encoders when extracting entity information from text, and then use an integrated decoder to determine the content to be extracted. Furthermore, the language models in these technologies are usually constructed using structures such as pre-trained Bidirectional Encoder Representation from Transformers (BERT) and RoBERTa. Language models are trained through pre-training tasks, primarily using Masked Language Modeling (MLM). This involves masking a word in the text and then using context words to predict the current word. However, when extracting information from the text, MLM tasks often fail to learn the relationships between multiple related words, leading to incomplete or incorrect word extraction and reduced accuracy. Therefore, improving the accuracy of text information extraction has become a pressing technical problem.

[0067] Based on this, embodiments of this application provide a text information extraction method and apparatus, electronic device and storage medium, aiming to improve the accuracy of text information extraction.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The text information extraction method provided in this application relates to the field of artificial intelligence technology. This text information extraction method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing the text information extraction method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0073] Please see Figure 1 , Figure 1This is an optional flowchart of the text information extraction method provided in the embodiments of this application. In some embodiments of this application, Figure 1 The method may include, but is not limited to, steps S110 to S170, as described below. Figure 1 These seven steps will be explained in detail.

[0074] Step S110: Obtain the original corpus and the target corpus words of the original corpus;

[0075] Step S120: Cluster the target corpus words according to the preset cluster center vector to obtain clustered upper-level word segmentation;

[0076] Step S130: Perform a first masking process on the original corpus based on the clustering upper-level word segmentation to obtain the first masked corpus and the masking category labels of the first masked corpus;

[0077] Step S140: Perform a second masking process on the original corpus based on the target corpus words to obtain the second masked corpus;

[0078] Step S150: Input the first masked corpus into the preset initial language model for corpus category prediction processing to obtain category prediction data, and adjust the parameters of the initial language model according to the category prediction data and the masked category labels to obtain a candidate language model.

[0079] Step S160: Input the second masked corpus into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data, and adjust the parameters of the candidate language model according to the word segmentation prediction data and the target corpus words to obtain the target language model;

[0080] Step S170: Input the obtained target text into the target language model to extract text information and obtain text extraction data.

[0081] Steps S110 to S170, as illustrated in this embodiment, involve acquiring the original corpus and target words from the original corpus, and clustering the target words according to a preset clustering center vector to obtain clustered upper-level word segments. Then, the original corpus undergoes a first masking process based on the clustered upper-level word segments to obtain a first masked corpus and its masked category labels; the original corpus undergoes a second masking process based on the target words to obtain a second masked corpus. Subsequently, in the first pre-training task stage, the first masked corpus is input into a preset initial language model for corpus category prediction processing to obtain category prediction data. The parameters of the initial language model are then adjusted based on the category prediction data and the masked category labels to obtain a candidate language model. In the second pre-training task stage, the second masked corpus is input into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data. The parameters of the candidate language model are then adjusted based on the word segmentation prediction data and the target words to obtain the target language model. Finally, the acquired target text is input into the target language model for text information extraction, resulting in extracted text data. This embodiment of the application, through a two-stage pre-training task, can obtain a target language model with higher accuracy in text information extraction. This target language model enhances the correlation between multiple related words in the text, effectively avoiding problems such as incomplete or erroneous word extraction. Therefore, when the acquired target text is input into the target language model of this application for text information extraction, the accuracy of text information extraction can be effectively improved, thereby enabling text structuring based on the accurately extracted text information.

[0082] In step S110 of some embodiments, the original corpus is used to represent the sample corpus for model training. The original corpus can be derived from a pre-set training corpus set, or it can be obtained by writing a web crawler or script program to crawl data in a targeted manner.

[0083] It should be noted that, in practical applications, this target language model can be used in downstream specific devices that include functions such as sentiment analysis, intelligent question answering, and reading comprehension. These downstream specific devices can be smart speakers, smartphones, etc. Specifically, before actual use on the downstream specific device, the target language model needs to be debugged. This application embodiment does not limit the specific device type for debugging. Therefore, the corresponding original corpus can be natural corpus generated in real time by the debugged downstream specific device, that is, it can be related corpus in the same field or corpus in different fields. This application embodiment does not impose any restrictions on this. For example, in the intelligent navigation scenario of fintech, when using the target language model on a downstream specific device that includes intelligent navigation functions, its corresponding original corpus can be "I want to go to City A", then the target corpus words extracted from the original corpus include "City A"; as another example, when using the target language model on a downstream specific device that includes intelligent question answering functions, its corresponding original corpus can be "What is the weather now?", then the target corpus words extracted from the original corpus include "now", "weather", and "what".

[0084] In step S120 of some embodiments, since there are many types of target corpus words of the same type, the model's pre-training task may not learn the correlation between different text words, resulting in incomplete or incorrect extraction of text words. This application clusters the target corpus words according to a preset cluster center vector to obtain cluster upper-level words, that is, it obtains the same cluster upper-level words corresponding to different target corpus words of the same category. For example, in the intelligent navigation scenario of fintech, when using a target language model on a downstream device containing intelligent navigation functions, if the target corpus words are "City A", "City B", "Province C", etc., which all belong to the address category, then their cluster upper-level words can be set to LOC; or, when the target corpus words are all specific person names, then their cluster upper-level words can be set to PER, without specific limitations here.

[0085] In step S130 of some embodiments, in order to improve the accuracy of text information extraction, during the first pre-training task of the model, this application first performs a first masking process on the original corpus based on the clustering upper-level word segmentation to obtain the first masked corpus. Specifically, the target corpus words in the original corpus are replaced with the mask symbol [MASK1], and the mask category label of [MASK1] is used to represent the category label of the clustering upper-level word segmentation of the target corpus word at the corresponding position.

[0086] For example, in a fintech intelligent question-answering scenario, if the original corpus is "I want to go to City A to find B to play", the target corpus words are "City A" and "B". Based on a preset clustering center vector, the target corpus words are clustered, resulting in the higher-order word segment for "B" being PER (person name category) and the higher-order word segment for "City A" being LOC (address category). Therefore, the original corpus is first masked based on the higher-order word segmentation, resulting in the first masked corpus "I want to go to [MASK1] to find [MASK1] to play", where the mask category label for [MASK] represents the category label of the higher-order word segmentation of the target corpus word at the corresponding position.

[0087] It should be noted that the target words in the original raw material are replaced with their corresponding clustering higher-level words to obtain the clustering higher-level corpus, which is used for subsequent comparison with the predicted higher-level corpus. Specifically, when labeling the target words of entities in the original corpus with their corresponding clustering higher-level words, BIO represents the specific mask category label, "B" indicates the beginning of the entity, "I" indicates the entity excluding the beginning, and "O" indicates a non-entity. For example, when the original corpus is "CC went to E city", the corresponding clustering higher-level corpus label is [PER-B,PER-I,O,O,LOC-B,LOC-I].

[0088] In step S140 of some embodiments, in order to improve the accuracy of text information extraction, during the second pre-training task of the model, the original corpus is subjected to a second masking process based on the target corpus words to obtain the second masked corpus, that is, to accurately predict the characters of the target corpus words. For example, when the original corpus is "CC went to EF city", the corresponding second masked corpus is "[MASK2][MASK2] went to [MASK2][MASK2][MASK2]", and the mask category label of [MASK2] is used to represent the category label of the target corpus word at the corresponding position.

[0089] It should be noted that the original corpus is also retained as the label corpus when predicting the second mask corpus.

[0090] In step S150 of some embodiments, during the first MLM task stage of the model, the first masked corpus is input into a preset initial language model for corpus category prediction processing, that is, prediction of the clustered upper-level words at the target corpus words. This category prediction data includes predicted upper-level words and predicted category probabilities. The predicted category probability represents the probability value that the predicted upper-level word matches the clustered upper-level words of the corresponding target corpus words. Afterwards, the parameters of the initial language model are adjusted based on the category prediction data and the masked category labels to obtain a candidate language model. In the first pre-training task stage of this application embodiment, the model is first trained on the corpus after the original corpus has been replaced using clustered upper-level words. This allows the model to associate the contextual information of all words under the same upper-level word category, making the model's feature predictions for similar words more similar and accurate.

[0091] In step S160 of some embodiments, during the second MLM task stage of the model, the second masked corpus is input into the candidate language model. This candidate language model is a model trained on corpus obtained from clustering and higher-level word segmentation of the target corpus words. The candidate language model performs word segmentation prediction processing on the second masked corpus to obtain word segmentation prediction data. This word segmentation prediction data includes predicted word segments and predicted word segmentation probabilities. The predicted word segments represent the predicted word segmentation at the position of the target corpus word, and the predicted word segmentation probabilities represent the probability value that the predicted word segmentation matches the corresponding target corpus word. Then, the parameters of the candidate language model are adjusted based on the word segmentation prediction data and the target corpus words to obtain the target language model. In this embodiment, after performing the first MLM task stage on the model, a more complex MLM task of predicting specific token words is performed. This can effectively associate contextual information with the features of the specific token to make accurate word prediction. Furthermore, the improved target language model enhances the correlation between multiple related words in the text, which can effectively avoid problems such as incomplete or incorrect extraction of text words, thereby effectively improving the accuracy of text information extraction.

[0092] It should be noted that the initial language model used in the embodiments of this application can be a Transformer model, a Long Short-Term Memory (LSTM) model, a Generative Pre-Training-2 (GPT-2) model, etc., without being specifically limited here, and the obtained target language model can be applied to specific downstream tasks using language models such as BERT model, LSTM model, GPT-2 model, etc., such as sentiment analysis, question answering, reading comprehension, etc.

[0093] In step S170 of some embodiments, in specific downstream tasks such as sentiment analysis, question answering, and reading comprehension, the target text can be input into a terminal containing a text information extraction method by the target object; alternatively, the target object can input target speech through a language input port, allowing the terminal to collect the target object's speech, convert the target speech into target text according to a preset speech-to-text conversion model, and then extract text information from the target text according to the target language model in the terminal to obtain text extraction data. This application does not limit the specific method of obtaining the target text, and will not elaborate further here.

[0094] Please see Figure 2 , Figure 2 This is an optional flowchart of step S110 provided in the embodiments of this application. In some embodiments of this application, step S110 may specifically include, but is not limited to, steps S210 and S220, which are described below in conjunction with... Figure 2 These two steps will be explained in detail.

[0095] Step S210: Obtain the original corpus;

[0096] Step S220: Perform word segmentation on the original corpus to obtain the target corpus words.

[0097] In steps S210 and S220 of some embodiments, after obtaining the original corpus by selectively crawling data according to a preset training corpus set or by writing a web crawler or script program, the original corpus is segmented to mark each word in the original corpus and determine the target corpus words of the preset marking type.

[0098] It should be noted that, specifically, the Jiabe tool can be used to first segment the text words in the original corpus, and then the segmented text words in the original corpus can be tagged with parts of speech according to the preset part-of-speech dictionary in the Jiabe tool to determine the type of each word. In order to more accurately identify the keywords in the original corpus, text words with preset tag types are used as target corpus words. The preset tag types can be nouns, personal names, place names, organizations, etc., without specific limitations.

[0099] It should be noted that entity extraction is an important task in natural language processing. It can extract entity information from text based on preset tag types, such as key text words like names, locations, and companies.

[0100] Please see Figure 3 , Figure 3This is an optional flowchart of step S120 provided in the embodiments of this application. In some embodiments of this application, step S120 may specifically include, but is not limited to, steps S310 to S340, as described below. Figure 3 These four steps will be explained in detail.

[0101] Step S310: Obtain the preset cluster center vector;

[0102] Step S320: Extract word segmentation features from the target corpus words to obtain target corpus word vectors;

[0103] Step S330: Calculate the vector distance between the word vectors and cluster center vectors of the target corpus to obtain the word clustering data of the corpus;

[0104] Step S340: Determine the clustering superordinate word segmentation based on the corpus word clustering data.

[0105] In step S310 of some embodiments, the present application embodiments may use the K-means clustering algorithm to cluster the target corpus words. The number of cluster center vectors is the same as the number of preset tag types, both being K, and each cluster center vector corresponds to a higher-level word segment, such as LOC, PER, etc. in the above embodiments.

[0106] In step S320 of some embodiments, a word vector model based on the word2vec model can be used to extract word segmentation features from the target corpus words to obtain target corpus word vectors. Other word vector feature extraction methods can also be used, which are not specifically limited here.

[0107] In steps S330 and S340 of some embodiments, vector distance is calculated between the target corpus word vectors and each cluster center vector to obtain K corpus word cluster data corresponding to the target corpus word vectors. This corpus word cluster data is used to represent the vector distance value between the target corpus word vectors and their corresponding cluster center vectors. All corpus word cluster data are numerically compared, and the superordinate word corresponding to the cluster center vector with the smallest value is taken as the superordinate word of the cluster.

[0108] Please see Figure 4 , Figure 4 This is another optional flowchart of the text information extraction method provided in the embodiments of this application. In some embodiments of this application, before step S130, the text information extraction method provided in the embodiments of this application also includes, but is not limited to, steps S410 to S430, which are described below in conjunction with... Figure 4 These three steps will be explained in detail.

[0109] Step S410: Obtain the target corpus word vectors under the cluster center vectors to obtain the clustered corpus word vectors;

[0110] Step S420: Calculate the vector mean of the word vectors in the clustered corpus to obtain the target clustering vector;

[0111] Step S430: Update the cluster center vector based on the target cluster vector.

[0112] In steps S410 to S430 of some embodiments, after determining the cluster superordinate word segment of a target corpus word, the target corpus word vectors under the cluster center vector corresponding to the cluster superordinate word are obtained, and these word vectors are recorded as cluster corpus word vectors. The cluster corpus word vectors are summed, and the vector mean is calculated based on the number of cluster corpus word vectors to obtain the target cluster vector. Then, the cluster center vector is updated according to the target cluster vector, that is, the target cluster vector is used as the new cluster center vector under the current category, and the next cyclic clustering is performed, repeating until the clustering termination condition is met. The clustering termination condition can be that no target corpus word vectors or a preset number of target corpus word vectors are reassigned to different clusters, that is, the cluster center vectors with no target corpus word vectors or a preset number of target corpus word vectors change, at which point the clustering error is minimized.

[0113] Please see Figure 5 , Figure 5 This is an optional flowchart of step S150 provided in the embodiments of this application. In some embodiments of this application, the initial language model includes a corpus encoding layer and a first fully connected layer, and step S150 may specifically include, but is not limited to, steps S510 to S540, as described below. Figure 5 These four steps will be explained in detail.

[0114] Step S510: Perform corpus encoding processing on the first masked corpus through the corpus encoding layer to obtain the first corpus encoding features;

[0115] Step S520: Perform corpus category prediction on the encoded features of the first corpus through the first fully connected layer to obtain category prediction data;

[0116] Step S530: Calculate the corpus category loss based on the category prediction data and the mask category labels to obtain the category loss data;

[0117] Step S540: Adjust the parameters of the initial language model based on the category loss data to obtain a candidate language model.

[0118] In step S510 of some embodiments, in the first MLM task stage of the model, firstly, the first masked corpus is processed by corpus coding layer to obtain the first corpus coding feature, which is to convert the text in the corpus into a fixed-length coding vector.

[0119] In step S520 of some embodiments, the fully connected layer maps the output vectors of the preceding layers into vectors of a specified dimension and outputs the corresponding predicted probabilities through the normalization function of the connections. The first fully connected layer performs corpus category prediction on the encoded features of the first corpus, obtaining category prediction data. This category prediction data includes predicted hypersegmentation words and predicted category probabilities. The predicted category probability represents the probability value that the predicted hypersegmentation word matches the clustered hypersegmentation words of the corresponding target corpus words. The normalization function can be a Softmax activation function, a sigmoid activation function, etc., and is not specifically limited here; it can be flexibly adjusted according to actual needs.

[0120] It's important to note that the differences between fully connected layers lie in their weight dimensions, resulting in varying output dimensions for each prediction. The first fully connected layer is used for predicting higher-order words. When building an initial language model based on BERT, the last dimension of its output token is 768, and the number of higher-order word categories is K when using K-means clustering. Therefore, for example, when a sentence is 20 characters long, the output dimension of the first fully connected layer is (20, K), and its corresponding weight dimension is (768, K).

[0121] In steps S530 and S540 of some embodiments, the corpus category loss is calculated based on the category prediction data and the mask category label to obtain category loss data, denoted as L. CEL1 Then, the parameters of the initial language model are adjusted based on the category loss data until the category loss data is greater than or equal to a preset loss threshold. Optimization of the initial language model then stops, resulting in a candidate language model that meets the requirements. The preset loss threshold can be a probability threshold such as 0.98 or 0.99, etc., and is not specifically limited here. In the first pre-training stage of this application embodiment, the model is first trained on the corpus after the original corpus has been replaced using clustering-based hypernym segmentation. This allows the model to associate the contextual information of all words within the same hypernym category, making the model's feature predictions for similar words more similar and accurate.

[0122] It should be noted that, in order to improve the accuracy of the initial language model in predicting the higher-level categories, the mask symbols at corresponding positions in the first masked corpus can be replaced with the obtained predicted higher-level words to obtain the predicted category corpus. The similarity value between the predicted category corpus and the clustered higher-level corpus is calculated using a preset similarity function, and the initial language model is optimized based on this similarity value and the category loss data to obtain the candidate language model.

[0123] Please see Figure 6 , Figure 6This is an optional flowchart of step S160 provided in the embodiments of this application. In some embodiments of this application, the initial language model further includes a second fully connected layer, and the weight dimensions of the first fully connected layer are different from those of the second fully connected layer. Step S160 may specifically include, but is not limited to, steps S610 to S640. The following describes the steps in conjunction with... Figure 6 These four steps will be explained in detail.

[0124] Step S610: Perform corpus encoding processing on the second masked corpus through the corpus encoding layer to obtain the second corpus encoding features;

[0125] Step S620: Perform word segmentation prediction on the encoded features of the second corpus through the second fully connected layer to obtain word segmentation prediction data;

[0126] Step S630: Calculate the segmentation loss of the corpus based on the word segmentation prediction data and the target corpus words to obtain the segmentation loss data;

[0127] Step S640: Adjust the parameters of the candidate language model based on the word segmentation loss data to obtain the target language model.

[0128] In step S610 of some embodiments, in the second MLM task stage of the model, firstly, the second mask corpus is processed by corpus coding layer to obtain second corpus coding features, which are the text in the corpus converted into a fixed-length coding vector.

[0129] In step S620 of some embodiments, the initial language model further includes a second fully connected layer, and the weight dimension of the first fully connected layer is different from that of the second fully connected layer. The second fully connected layer performs word segmentation prediction on the encoded features of the second corpus to obtain word segmentation prediction data. This word segmentation prediction data includes predicted word segments and predicted word segmentation probabilities. The predicted word segments represent the predicted word segments at the target corpus word positions, and the predicted word segmentation probabilities represent the probability value that the predicted word segments match the corresponding target corpus word.

[0130] It should be noted that the second fully connected layer is used for predicting specific tokens. When the initial language model is built based on the BERT model, the last dimension of the output token is 768, while the preset token dictionary has a dimension of M. Therefore, for example, when a sentence is 20 characters long, the output dimension of the second fully connected layer is (20, M), and its corresponding weight dimension is (768, M). The token dictionary is used to represent the dictionary storing all the characters.

[0131] In steps S630 and S640 of some embodiments, word segmentation loss is calculated based on word segmentation prediction data and target corpus words to obtain word segmentation loss data, denoted as L. CEL2Then, the parameters of the candidate language model are adjusted based on the word segmentation loss data until the word segmentation loss data is greater than or equal to a preset loss threshold. Optimization of the candidate language model then stops, resulting in a target language model that meets the requirements. The preset loss threshold can be a probability threshold such as 0.98 or 0.99, etc., and is not specifically limited here. In this embodiment, after performing the first MLM task stage on the model, a more complex MLM task of predicting specific token characters is performed. This effectively associates contextual information with the features of the specific token for accurate character prediction. Furthermore, the improved target language model enhances the correlation between multiple related words in the text, effectively avoiding problems such as incomplete or erroneous word extraction, thereby significantly improving the accuracy of text information extraction.

[0132] It should be noted that, in order to improve the prediction accuracy of the candidate language model for specific tokens, the mask symbols at corresponding positions in the second mask corpus can be replaced based on the obtained predicted word segments to obtain the predicted word segment corpus. The similarity value between the predicted word segment corpus and the original corpus is calculated using a preset similarity function, and the candidate language model is optimized based on this similarity value and the word segmentation loss data to obtain the target language model.

[0133] Please see Figure 7 , Figure 7 This is an optional flowchart of step S630 provided in the embodiments of this application. In some embodiments of this application, step S630 may include, but is not limited to, steps S710 and S720, as described below. Figure 7 These two steps will be explained in detail.

[0134] Step S710: Calculate the number of masks on the second masked corpus to obtain the number of word segmentation masks;

[0135] Step S720: Calculate the word segmentation loss based on the word segmentation prediction data and the number of word segmentation masks to obtain the word segmentation loss data.

[0136] In steps S710 and S720 of some embodiments, the present application embodiments can calculate the category loss data L based on the cross-entropy loss function. CEL1 And word segmentation loss data L CEL2 Specifically, as shown in formula (1), the number of masks is calculated for the second masked corpus to obtain the number of word segmentation masks m, which represents the total number of text characters that are masked. Then, based on the predicted word segmentation probabilities predicted at each [MASK] position, the corpus word segmentation loss is calculated on the cross-entropy loss function to obtain the word segmentation loss data L. CEL2 .

[0137] L CEL =∑m log(p) (1)

[0138] Among them, L CEL Including category loss data L CEL1 And word segmentation loss data L CEL2 m represents the number of segmentation masks, and p represents the predicted segmentation probability or predicted category probability at each [MASK] position.

[0139] For example, in the insurance claims scenario of fintech, the original text description is "E was injured in a car accident and received compensation F", where E is a specific person's name and F is the amount of compensation received. The target words in this text description are "E", "car accident", and "F". The target words are clustered according to a preset cluster center vector, i.e., the cluster hypersegmentation of "E" is PER, the cluster hypersegmentation of "car accident" is the specific event (EVE), and the cluster hypersegmentation of "F" is LOC. Then, the original text is first masked based on the cluster hypersegmentation to obtain the first masked text "[MASK1] was injured and received compensation [MASK1]", where the mask category label of [MASK1] represents the category label of the cluster hypersegmentation of the target words at the corresponding position. That is, this application can first train the model on the first masked text constructed from the cluster hypersegmentation to obtain a candidate language model. Subsequently, the original corpus is subjected to a second masking process based on the target corpus words, resulting in the second mask prediction "[MASK2] was injured and received compensation [MASK2]", where the mask category label of [MASK2] is used to represent the category label of the target corpus word at the corresponding position. That is, after training the candidate language model, this application then trains the candidate language model a second time based on the second mask corpus to obtain the target language model.

[0140] Therefore, in this embodiment, after performing the first MLM task stage on the model, a more complex MLM task of predicting specific tokens is then performed. The resulting target language model can effectively associate contextual information with the features of that specific token for accurate word prediction. Furthermore, the improved target language model enhances the correlation between multiple related words in the text, effectively avoiding problems such as incomplete or erroneous word extraction, and significantly improving the accuracy of text information extraction.

[0141] Please see Figure 8 , Figure 8This is a block diagram of a text information extraction device provided in some embodiments of this application. In some embodiments, the text information extraction device may specifically include a corpus acquisition module 810, a clustering module 820, a first mask processing module 830, a second mask processing module 840, a corpus category prediction module 850, a corpus word segmentation prediction module 860, and an information extraction module 870. This text information extraction device can implement the above-described text information extraction method.

[0142] The corpus acquisition module 810 is used to acquire the original corpus and the target corpus words of the original corpus;

[0143] Clustering module 820 is used to cluster words in the target corpus according to the preset clustering center vector to obtain clustered upper-level word segments;

[0144] The first masking processing module 830 is used to perform first masking processing on the original corpus according to the clustering upper-level word segmentation to obtain the first masked corpus and the masking category label of the first masked corpus;

[0145] The second masking processing module 840 is used to perform second masking processing on the original corpus based on the target corpus words to obtain the second masked corpus;

[0146] The corpus category prediction module 850 is used to input the first masked corpus into a preset initial language model for corpus category prediction processing, obtain category prediction data, and adjust the parameters of the initial language model according to the category prediction data and the masked category labels to obtain a candidate language model.

[0147] The corpus word segmentation prediction module 860 is used to input the second masked corpus into the candidate language model for corpus word segmentation prediction processing, obtain word segmentation prediction data, and adjust the parameters of the candidate language model according to the word segmentation prediction data and the target corpus words to obtain the target language model;

[0148] The information extraction module 870 is used to input the acquired target text into the target language model to extract text information and obtain text extraction data.

[0149] The specific implementation of this text information extraction device is basically the same as the specific embodiment of the text information extraction method described above, and will not be repeated here.

[0150] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described text information extraction method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0151] Please see Figure 9 , Figure 9The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0152] The processor 910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0153] The memory 920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910 using the text information extraction method of the embodiments of this application.

[0154] The input / output interface 930 is used to implement information input and output;

[0155] The communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0156] Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940);

[0157] The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0158] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described text information extraction method.

[0159] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] The text information extraction method, apparatus, electronic device, and storage medium provided in this application embodiment obtain target corpus words by acquiring original corpus, performing word segmentation on the original corpus, and obtaining target corpus words. A preset cluster center vector is acquired, and word segmentation features are extracted from the target corpus words to obtain target corpus word vectors. Vector distance is calculated between the target corpus word vectors and the cluster center vectors to obtain corpus word clustering data, which is used to determine the higher-level word segmentation for the cluster. Then, the target corpus word vectors under the cluster center vectors are acquired to obtain clustered corpus word vectors. The vector mean of the clustered corpus word vectors is calculated to update the cluster center vectors based on the obtained target cluster vectors. Afterwards, the original corpus is subjected to a first masking process based on the higher-level word segmentation for the cluster, resulting in a first masked corpus and its mask category label; the original corpus is then subjected to a second masking process based on the target corpus words to obtain a second masked corpus. The initial language model consists of a corpus encoding layer, a first fully connected layer, and a second fully connected layer. In the first pre-training task, the corpus encoding layer encodes the first masked corpus to obtain first corpus encoded features. The first fully connected layer then uses these first corpus encoded features to predict corpus categories, obtaining category prediction data. Based on this category prediction data and the masked category labels, the parameters of the initial language model are adjusted to obtain a candidate language model. In the second pre-training task, the corpus encoding layer encodes the second masked corpus to obtain second corpus encoded features. The second fully connected layer then uses these second corpus encoded features to predict word segmentation, obtaining word segmentation prediction data. Based on this word segmentation prediction data and the target corpus words, the parameters of the candidate language model are adjusted to obtain the target language model. Finally, the acquired target text is input into the target language model for text information extraction, yielding extracted text data. This application embodiment employs a two-stage pre-training task: introducing a task to replace the word with a hypernym category and perform MLM (Multi-Level Modeling). This results in a target language model with higher accuracy in extracting text information. This target language model enhances the correlation between multiple related words in the text, effectively avoiding problems such as incomplete or erroneous word extraction. In the first pre-training stage, the model is trained on the replaced corpus using clustering-based hypernym segmentation. This allows the model to associate the contextual information of all words within the same hypernym category, making the model's feature predictions for similar words more similar and accurate. After the first MLM stage, a more complex MLM task predicting specific tokens is performed. This effectively associates contextual information with the features of that specific token for accurate word prediction. Furthermore, the improved target language model enhances the correlation between multiple related words in the text, effectively avoiding problems such as incomplete or erroneous word extraction.Therefore, the technical solution of this application embodiment is more suitable for information extraction tasks and can effectively improve the accuracy of text information extraction.

[0161] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0162] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0163] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0165] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0166] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0168] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0170] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for extracting text information, characterized in that, The method includes: Obtain the original corpus, perform word segmentation on the original corpus, and obtain the target corpus words; Clustering is performed on the target corpus words according to a preset cluster center vector to obtain clustered upper-level word segments; wherein, the step of clustering the target corpus words according to the preset cluster center vector to obtain clustered upper-level word segments includes: obtaining the preset cluster center vector; extracting word segmentation features from the target corpus words to obtain target corpus word vectors; calculating the vector distance between the target corpus word vectors and the cluster center vectors to obtain corpus word clustering data; and determining the clustered upper-level word segments based on the corpus word clustering data; The original corpus is subjected to a first masking process based on the clustering and upper-level word segmentation to obtain a first masked corpus and a masking category label for the first masked corpus. The original corpus is subjected to a second masking process based on the target corpus words to obtain a second masked corpus. The process involves inputting the first masked corpus into a preset initial language model for corpus category prediction processing to obtain category prediction data. Then, the parameters of the initial language model are adjusted based on the category prediction data and the masked category labels to obtain a candidate language model. The initial language model includes a corpus encoding layer and a first fully connected layer. The steps of inputting the first masked corpus into the preset initial language model for corpus category prediction processing to obtain category prediction data, and adjusting the parameters of the initial language model based on the category prediction data and the masked category labels to obtain a candidate language model, include: performing corpus encoding processing on the first masked corpus through the corpus encoding layer to obtain first corpus encoding features; performing corpus category prediction on the first corpus encoding features through the first fully connected layer to obtain category prediction data; calculating corpus category loss based on the category prediction data and the masked category labels to obtain category loss data; and adjusting the parameters of the initial language model based on the category loss data to obtain the candidate language model. The second masked corpus is input into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data. The parameters of the candidate language model are adjusted according to the word segmentation prediction data and the target corpus words to obtain the target language model. The acquired target text is input into the target language model for text information extraction, resulting in extracted text data.

2. The method according to claim 1, characterized in that, Before performing a first masking process on the original corpus based on the clustering-based upper-level word segmentation to obtain a first masked corpus and masked category labels for the first masked corpus, the method further includes: Obtain the target corpus word vectors under the cluster center vectors to obtain the clustered corpus word vectors; The target clustering vector is obtained by calculating the vector mean of the word vectors in the clustered corpus. Update the cluster center vector based on the target cluster vector.

3. The method according to claim 1, characterized in that, The initial language model also includes a second fully connected layer, wherein the weight dimension of the first fully connected layer is different from the weight dimension of the second fully connected layer; The step of inputting the second masked corpus into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data, and adjusting the parameters of the candidate language model based on the word segmentation prediction data and the target corpus words to obtain the target language model includes: The second masked corpus is processed by the corpus coding layer to obtain the second corpus coding features; The second fully connected layer is used to perform word segmentation prediction on the encoded features of the second corpus to obtain word segmentation prediction data. Based on the word segmentation prediction data and the target corpus words, the word segmentation loss is calculated to obtain the word segmentation loss data; The candidate language model is adjusted based on the word segmentation loss data to obtain the target language model.

4. The method according to claim 3, characterized in that, The step of calculating the segmentation loss based on the word segmentation prediction data and the target corpus words to obtain segmentation loss data includes: The number of masking codes is calculated on the second masked corpus to obtain the number of word segmentation masks; The word segmentation loss is calculated based on the word segmentation prediction data and the number of word segmentation masks to obtain the word segmentation loss data.

5. A text information extraction device, characterized in that, The device includes: The corpus acquisition module is used to acquire raw corpus, perform word segmentation on the raw corpus, and obtain target corpus words. A clustering module is used to cluster the target corpus words according to a preset clustering center vector to obtain clustered upper-level word segments; wherein, the step of clustering the target corpus words according to the preset clustering center vector to obtain clustered upper-level word segments includes: obtaining the preset clustering center vector; extracting word segmentation features from the target corpus words to obtain target corpus word vectors; calculating the vector distance between the target corpus word vectors and the clustering center vectors to obtain corpus word clustering data; and determining the clustered upper-level word segments based on the corpus word clustering data; The first masking processing module is used to perform a first masking process on the original corpus based on the clustering upper-level word segmentation to obtain a first masked corpus and a masking category label for the first masked corpus. The second masking processing module is used to perform second masking processing on the original corpus according to the target corpus words to obtain the second masked corpus; A corpus category prediction module is used to input the first masked corpus into a preset initial language model for corpus category prediction processing to obtain category prediction data, and to adjust the parameters of the initial language model according to the category prediction data and the masked category labels to obtain a candidate language model. The initial language model includes a corpus encoding layer and a first fully connected layer. The process of inputting the first masked corpus into the preset initial language model for corpus category prediction processing to obtain category prediction data, and adjusting the parameters of the initial language model according to the category prediction data and the masked category labels to obtain a candidate language model, includes: performing corpus encoding processing on the first masked corpus through the corpus encoding layer to obtain first corpus encoding features; performing corpus category prediction on the first corpus encoding features through the first fully connected layer to obtain category prediction data; calculating corpus category loss according to the category prediction data and the masked category labels to obtain category loss data; and adjusting the parameters of the initial language model according to the category loss data to obtain the candidate language model. The corpus word segmentation prediction module is used to input the second masked corpus into the candidate language model for corpus word segmentation prediction processing to obtain word segmentation prediction data, and adjust the parameters of the candidate language model according to the word segmentation prediction data and the target corpus words to obtain the target language model; The information extraction module is used to input the acquired target text into the target language model to extract text information and obtain text extraction data.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a text information extraction method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a text information extraction method according to any one of claims 1 to 4.

Images