A domain term expansion method and system based on pre-training model

Abstract

The present invention discloses a domain word expansion method and system based on a pre-trained model. The method is based on a word model pre-trained with massive external general corpus, designs and superimposes an internal flow text word model fine-tuning algorithm, and is supplemented by a natural language processing tool flow to achieve word understanding of a small amount of industry-specific text and improve the efficiency of artificial experience word library construction, effectively improving the recognition ability of non-operating income flow. In this way, keywords related to non-operating income can be automatically mined from the flow data, achieving the expansion of operating flow keywords, improving the recognition of non-operating flow, and having high accuracy and generalization ability, thereby effectively portraying the user's operating ability.

Description

Technical Field

[0001] The present invention relates to the field of intelligent word expansion technology, and in particular to a domain word expansion method and system based on a pre-training model. Background Art

[0002] In credit scenarios, assessing a borrower's debt repayment ability is crucial. The collection and analysis of cash flow is a powerful tool for assessing a user's debt repayment ability in credit scenarios. Operating cash flow is a crucial indicator of a user's operating ability in operating loans. Operating cash flow is a crucial data source reflecting a user's operating status. However, since cash flow often contains a significant amount of non-operating income, such as transfers, loans, and repayments, this income can affect an accurate assessment of a user's operating authenticity and ability.

[0003] Currently, the most common method for evaluating operating cash flow is to manually construct a keyword lexicon based on experience, then filter out non-operating income based on the terms in the lexicon. However, this approach has the following problems: First, manually constructed lexicons are inefficient and require constant updating and maintenance; second, they have low coverage, making it easy to miss some non-operating income; and third, they have poor generalization capabilities and are difficult to adapt to cash flow data from different industries, regions, and time periods. Using model training and expansion methods also presents challenges such as small datasets, strong domain attributes of target terms, and difficulty migrating and reusing general corpora.

[0004] Therefore, a domain term expansion scheme based on pre-training models is desired. Summary of the Invention

[0005] The embodiment of the present invention provides a method and system for expanding domain terms based on a pre-trained model. The method and system are based on a word model pre-trained on a large amount of external general corpus, and a fine-tuning algorithm for an internal flow text word model is designed and superimposed, and supplemented by a natural language processing tool flow, to achieve word understanding of a small amount of industry-specific text and improve the efficiency of constructing an artificial experience vocabulary, effectively improving the ability to identify non-operating income flow. In this way, keywords related to non-operating income can be automatically mined from the flow data, achieving the expansion of operating flow keywords, improving the recognition of non-operating flow, and having high accuracy and generalization ability, thereby effectively portraying the user's operating ability.

[0006] The embodiment of the present invention also provides a method for expanding domain terms based on a pre-training model, which includes:

[0007] Obtain user transaction flow text data;

[0008] Preprocessing the user transaction flow text data to obtain preprocessed user transaction flow text data;

[0009] Sorting and arranging the pre-processed user transaction flow text data according to word frequency and filtering the text data according to part of speech to obtain a first seed pool to be evaluated;

[0010] Obtaining a first word set and a first universal word vector set corresponding to the first word set from a language model pre-trained based on an external massive corpus;

[0011] Performing similarity correlation analysis on the first general word vector set, the first word set, and the first seed pool to be evaluated to obtain a second seed pool to be evaluated;

[0012] Introducing internal stream text data and building an industry word model through the Fine-tuning method to obtain a set of industry-specific words and a set of industry-specific word vectors;

[0013] Performing similarity correlation analysis on the industry-specific word set and the industry-specific word vector set with the first seed pool to be evaluated to obtain a third seed pool to be evaluated; and

[0014] The first seed pool to be evaluated, the second seed pool to be evaluated, and the third seed pool to be evaluated are merged to obtain a final seed pool.

[0015] The embodiment of the present invention further provides a domain term expansion system based on a pre-trained model, which includes:

[0016] A text data acquisition module is used to obtain user transaction flow text data;

[0017] A preprocessing module, configured to preprocess the user transaction flow text data to obtain preprocessed user transaction flow text data;

[0018] A part-of-speech processing module, configured to sort and arrange the pre-processed user transaction flow text data according to word frequency and filter the pre-processed user transaction flow text data according to word frequency to obtain a first seed pool to be evaluated;

[0019] A word set generation module, configured to obtain a first word set and a first universal word vector set corresponding to the first word set from a language model pre-trained based on an external massive corpus;

[0020] A first association analysis module is configured to perform similarity association analysis on the first general word vector set, the first word set, and the first seed pool to be evaluated to obtain a second seed pool to be evaluated;

[0021] The industry word model construction module is used to introduce internal stream text data and construct an industry word model through the Fine-tuning method to obtain a set of industry-specific words and a set of industry-specific word vectors;

[0022] a second association analysis module, configured to perform similarity association analysis on the industry-specific word set and the industry-specific word vector set with the first seed pool to be evaluated to obtain a third seed pool to be evaluated; and

[0023] A merging module is configured to merge the first seed pool to be evaluated, the second seed pool to be evaluated, and the third seed pool to be evaluated to obtain a final seed pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 The present invention provides a flowchart of a method for expanding domain terms based on a pre-training model in an embodiment of the present invention.

[0026] Figure 2 A schematic diagram of the system architecture of a domain term expansion method based on a pre-training model provided in an embodiment of the present invention.

[0027] Figure 3 This is a block diagram of a domain term expansion system based on a pre-training model provided in an embodiment of the present invention.

[0028] Figure 4 This is an application scenario diagram of a domain term expansion method based on a pre-training model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0030] Unless otherwise specified, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art in the art of the present application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the scope of this application.

[0031] In the description of the embodiments of this application, it should be noted that, unless otherwise specified and limited, the term "connection" should be understood in a broad sense. For example, it can be an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meaning of the above terms can be understood according to the specific circumstances.

[0032] It should be noted that the terms "first, second, and third" in the embodiments of the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the terms "first, second, and third" can be interchanged to represent a specific order or precedence where permitted. It should be understood that the objects distinguished by "first, second, and third" can be interchanged where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] The collection and analysis of cash flow is a powerful tool for evaluating the user's debt repayment ability in the credit scenario, and operating cash flow is an important reflection of the user's operating ability in operating loans. In actual cash flow, there are many non-operating incomes that interfere with the judgment of the authenticity of the operation. If we simply rely on manual experience to circle keywords to eliminate non-operating income, there will be problems such as high missed judgment rate and low efficiency. If we adopt the model training and expansion method, we will face problems such as too small data set, strong domain attributes of target words, and difficulty in migrating and reusing general corpus. The present invention is based on a word model pre-trained with massive external general corpus, designs a fine-tuned word model that superimposes internal cash flow text, and is supplemented by a natural language processing tool flow to achieve the expansion of operating cash flow keywords, improve the recognition of non-operating cash flow, and effectively characterize the user's operating ability.

[0034] In this application, the solutions provided include:

[0035] (1) Obtain user transaction flow text data and perform text preprocessing such as word segmentation, word form normalization, and part-of-speech tagging;

[0036] (2) Sort by descending word frequency, provide tools such as part-of-speech filtering, and efficiently build the target seed pool based on manual experience1;

[0037] (3) Obtaining a language model and universal word vector based on pre-training of external massive corpus1;

[0038] (4) In the general word vector 1, the cosine similarity method is used to output words with high similarity to the artificial experience seed pool 1. Assisted by tools such as part-of-speech filtering, artificial screening is performed to form the seed pool 2 to be evaluated, thereby achieving the expansion of general similar words;

[0039] (5) Based on the above-mentioned external massive corpus model, internal stream text data is further introduced, and the industry word model 2 is constructed through the Fine-tuning method. In order to avoid overfitting when there is less industry text data, L2 regularization is added to the Fine-tuning loss function to obtain industry-specific word vectors 2;

[0040] (6) In the industry-specific word vector 2, the cosine similarity method is used to output words with high similarity to the artificial experience seed pool 1. Assisted by tools such as part-of-speech filtering, manual screening is performed to form the seed pool 3 to be evaluated, thereby expanding the industry-similar words;

[0041] (7) Seed pools 1, 2, and 3 are merged and the final seed pool is obtained after business validity verification. The above process can be repeated multiple times.

[0042] This application combines a massive external corpus pre-trained word model with an internal flow text word model fine-tuning algorithm to achieve word understanding of a small amount of industry-specific text; designs a natural language processing tool flow to achieve efficient construction of artificial experience vocabulary; and effectively improves the recognition ability of non-operating income flow.

[0043] In one embodiment of the present invention, Figure 1 The present invention provides a flowchart of a method for expanding domain terms based on a pre-training model in an embodiment of the present invention. Figure 2 Schematic diagram of the system architecture of a domain word expansion method based on a pre-training model provided in an embodiment of the present invention. Figure 1 and Figure 2 As shown, the domain word expansion method based on the pre-training model according to an embodiment of the present invention includes: 110, obtaining user transaction flow text data; 120, pre-processing the user transaction flow text data to obtain pre-processed user transaction flow text data; 130, sorting the pre-processed user transaction flow text data according to word frequency and filtering the part of speech to obtain a first seed pool to be evaluated; 140, obtaining a first word set and a first general word vector set corresponding to the first word set from a language model pre-trained based on an external massive corpus; 150, combining the first general word vector set and the first general word vector set. A word set is subjected to similarity association analysis with the first seed pool to be evaluated to obtain a second seed pool to be evaluated; 160, internal flow text data is introduced and an industry word model is constructed through a Fine-tuning method to obtain an industry-specific word set and a set of industry-specific word vectors; 170, the industry-specific word set and the set of industry-specific word vectors are subjected to similarity association analysis with the first seed pool to be evaluated to obtain a third seed pool to be evaluated; and, 180, the first seed pool to be evaluated, the second seed pool to be evaluated and the third seed pool to be evaluated are merged to obtain a final seed pool.

[0044] In step 110, ensure that a sufficient amount and diversity of user transaction flow text data is obtained to cover situations in different industries, regions and time periods. Obtain real user transaction flow data to provide a basis for subsequent analysis and modeling. In step 120, preprocessing steps such as text cleaning, word segmentation, stop word removal, and stemming are included to prepare text data for subsequent analysis and modeling. Extract effective text features, reduce noise and redundant information, and provide clean data for subsequent steps. In step 130, high-frequency words are selected as candidate words based on word frequency sorting, and part-of-speech filtering is performed to screen out words related to business flow. High-frequency words related to business flow are extracted as the initial word set of the first seed pool to be evaluated. In step 140, a pre-trained language model, such as BERT, GPT, etc., is used to extract word vector representations of the words in the first seed pool to be evaluated. Obtain the semantic representation of the words in the first seed pool to be evaluated to provide a basis for subsequent similarity association analysis. In the step 150, similarity calculation method (such as cosine similarity) is used to calculate the similarity between the first general word vector set and the vocabulary in the first seed pool to be evaluated, and the vocabulary with high similarity to the vocabulary in the seed pool is screened out. The first seed pool to be evaluated is expanded, and a general vocabulary with high similarity to the vocabulary in the seed pool is introduced to increase the coverage capability of non-operating income. In the step 160, the pre-training model is further trained by the Fine-tuning method using internal flow text data to adapt to the language characteristics and vocabulary of a specific industry. The set of industry-specific word sets and industry-specific word vectors is obtained to improve the understanding and recognition capability of specific industry flow data. In the step 170, similarity calculation method is used to calculate the similarity between the industry-specific word vector set and the vocabulary in the first seed pool to be evaluated, and the vocabulary with high similarity to the vocabulary in the seed pool is screened out. The first seed pool to be evaluated is further expanded, and industry-specific vocabulary with high similarity to the vocabulary in the seed pool is introduced to improve the accurate recognition capability of non-operating income. In step 180, the words in the first, second, and third seed pools to be evaluated are merged, and duplicate words are removed to obtain a final seed pool. This final seed pool, which includes common words, industry-specific words, and high-frequency words, serves as an expanded set of business flow keywords, improving the ability to accurately identify non-operating income and assess operating capacity.

[0045] Through the above steps, a domain term expansion solution based on the pre-trained model can be constructed to improve the ability to accurately evaluate user business flows.

[0046] In response to the above technical problems, the technical concept of this application is to design a word model based on pre-training of massive external general corpus, superimpose an internal flow text word model fine-tuning algorithm, and supplement it with natural language processing tool flow to achieve word understanding of a small amount of industry-specific text and improve the efficiency of artificial experience vocabulary construction, effectively improving the ability to identify non-operating income flow. In this way, it is possible to automatically mine keywords related to non-operating income from the flow data, realize the expansion of operating flow keywords, improve non-operating flow recognition, and have high accuracy and generalization ability, thereby effectively portraying the user's operating ability.

[0047] Specifically, in the technical solution of the present application, first, the user transaction flow text data is obtained. Then, the user transaction flow text data is preprocessed to obtain the preprocessed user transaction flow text data. Specifically, the preprocessing steps may include word segmentation, word form normalization, part-of-speech tagging, etc., wherein word segmentation is to segment the text according to the boundaries of words and divide the continuous character sequence into independent words. Word segmentation can extract the basic units in the text, which is convenient for subsequent processing and analysis. Word form normalization is to normalize words of different forms to their original forms. For example, the plural form, tense changes, etc. of a word are converted into its basic form to reduce the interference of word variants on subsequent processing. Part-of-speech tagging means assigning each word its own part-of-speech label, such as noun, verb, adjective, etc. Part-of-speech tagging can help understand the role and grammatical relationship of words in sentences, which is helpful for subsequent semantic analysis and feature extraction.

[0048] In one embodiment of the present application, the user transaction flow text data is preprocessed to obtain preprocessed user transaction flow text data, including: performing word segmentation, word form normalization and part-of-speech tagging processing on the user transaction flow text data to obtain the preprocessed user transaction flow text data.

[0049] The pre-processed user transaction flow text data is then sorted and arranged by word frequency and filtered by part of speech to screen out words of certain importance and representativeness, forming the first seed pool to be evaluated. Specifically, word frequency sorting is performed by sorting the frequency of words in the text, so as to determine which words appear more frequently in the text and may have more important significance and representativeness. High-frequency words usually contain key information in the text. For business capability assessment, these high-frequency words are often closely related to business activities. Part of speech filtering means that in the business capability assessment, the focus is on words related to business activities, rather than words that interfere with non-business income. By filtering words by part of speech, some words unrelated to business activities can be eliminated, reducing the interference of non-business income. For example, nouns, verbs, and other words related to business activities can be filtered out, while adjectives, adverbs, and other words with a weak relationship with business activities can be filtered out.

[0050] Furthermore, a first word set and a first general word vector set corresponding to the first word set are obtained from a language model pre-trained based on an external massive corpus, so as to obtain a wider vocabulary and more comprehensive semantic information. These word sets and word vector sets can be used for subsequent word expansion and identification of non-operating income. Specifically, through the language model pre-trained based on an external massive corpus, a large-scale word set can be obtained, which contains words from various fields and topics. These words can be used as candidate words to expand the first seed pool to be evaluated to increase the coverage of business activities. At the same time, the word vectors learned by the language model can capture the semantic relationship and similarity between words. These word vectors can be used to calculate the similarity between words, thereby helping to determine whether a word is related to business activities, which is very useful for the identification of non-operating income.

[0051] In one embodiment of the present application, a similarity association analysis is performed on the first general word vector set and the first word set with the first seed pool to be evaluated to obtain a second seed pool to be evaluated, including: in the first general word vector set and the first word set, a cosine similarity method is used to output words whose similarity with each word in the first seed pool to be evaluated exceeds a predetermined threshold to obtain the second seed pool to be evaluated.

[0052] Then, in the first general word vector set and the first word set, the cosine similarity method is used to output words whose similarity with each word in the first seed pool to be evaluated exceeds a predetermined threshold, so as to screen out words related to the first seed pool to be evaluated, and thus construct the second seed pool to be evaluated. In this way, the seed pool to be evaluated can be further expanded and enriched to improve the accuracy of the assessment of business capabilities. Specifically, the cosine similarity method can be used to measure the degree of similarity between two word vectors. Here, each word in the first seed pool to be evaluated is calculated for similarity with the words in the first general word vector set. If the similarity between a word and a general word vector exceeds a predetermined threshold, they are considered to be semantically similar or related. In this way, the accuracy of the assessment of business capabilities can be further improved, ensuring more comprehensive coverage of possible business activity areas.

[0053] It should be understood that in a credit scenario, the assessment of a borrower's operating capacity needs to take into account the relevant terminology, domain knowledge, and industry-specific operating activities of a specific industry. In order to more accurately identify and understand texts related to operating capacity, in the technical solution of this application, internal flow text data is further introduced and an industry word model is constructed through the Fine-tuning method to obtain a set of industry-specific words and a set of industry-specific word vectors to better adapt to the needs of operating activity assessment in credit scenarios. Specifically, through the Fine-tuning method, the pre-trained general language model can be further trained using internal flow text data to make it more suitable for operating activity assessment tasks in credit scenarios. This can improve the model's ability to understand specific industry fields and learn industry-specific vocabulary and semantic information.

[0054] In one embodiment of the present application, a similarity association analysis is performed on the set of industry-specific words and the set of industry-specific word vectors with the first seed pool to be evaluated to obtain a third seed pool to be evaluated, including: in the set of industry-specific words and the set of industry-specific word vectors, a cosine similarity method is used to output words whose similarity with each word in the first seed pool to be evaluated exceeds a predetermined threshold to obtain the third seed pool to be evaluated.

[0055] Considering the importance of industry-specific vocabulary and terminology in understanding and evaluating a borrower's business operations during business capability assessments, to filter out industry-specific terms related to the first seed pool for evaluation, the cosine similarity method is used to output words whose similarity with each word in the first seed pool for evaluation exceeds a predetermined threshold from the set of industry-specific terms and the set of industry-specific term vectors, thereby generating a third seed pool for evaluation. It should be understood that by using the cosine similarity method, the similarity between each word in the first seed pool for evaluation and a word in the industry-specific term set can be calculated. If the similarity between a word and a word in the industry-specific term set exceeds a predetermined threshold, the two words are considered semantically similar or related. This indicates that the word is likely related to the business activities of a specific industry. By outputting words whose similarity exceeds the predetermined threshold, the third seed pool for evaluation is generated, which contains industry-specific terms related to the first seed pool for evaluation. These terms can provide more specific and accurate industry background information, further enriching and expanding the seed pool for evaluation, thereby improving the accuracy of the business capability assessment.

[0056] In one embodiment of the present application, the first seed pool to be evaluated, the second seed pool to be evaluated, and the third seed pool to be evaluated are merged to obtain a final seed pool, including: passing the first seed pool to be evaluated, the second seed pool to be evaluated, and the third seed pool to be evaluated through a text semantic encoder to obtain a first text feature vector, a second text feature vector, and a third text feature vector; fusing the first text feature vector, the second text feature vector, and the third text feature vector to obtain a cascade feature vector; and passing the cascade feature vector through a text semantic decoder to obtain the final seed pool.

[0057] Furthermore, the first, second, and third seed pools to be evaluated are merged to form a final seed pool, which serves as the basis for the subsequent business capability assessment. It should be understood that in a business capability assessment, a seed pool is an initial set of terms used to build a model or perform feature extraction. The first seed pool is a set of terms manually selected based on business needs and domain knowledge. The second seed pool is a set of terms related to business capability obtained through a pre-trained model and fine-tuned algorithms. The third seed pool is a set of industry-specific terms related to the first seed pool, obtained through cosine similarity screening. Merging these three seed pools creates a more comprehensive and richer seed pool encompassing a wider range of business capability-related terms. This merging operation removes duplicate terms and integrates terms from different sources, ensuring the completeness and diversity of the final seed pool. In particular, the construction of the final seed pool provides a more comprehensive and accurate vocabulary foundation to support the subsequent business capability assessment, thereby facilitating a better understanding and assessment of the borrower's business capability.

[0058] In one embodiment of the present application, the domain word expansion method based on the pre-training model further includes a training step: for training the text semantic encoder and the text semantic decoder. The training step includes: obtaining training data, which includes training user transaction flow text data and real data of the final seed pool; preprocessing the training user transaction flow text data to obtain preprocessed training user transaction flow text data; sorting and arranging the preprocessed training user transaction flow text data according to word frequency and filtering the part of speech to obtain a training first seed pool to be evaluated; obtaining a training first word set and a training first general word vector set corresponding to the training first word set from a language model pre-trained based on the external massive corpus; using the cosine similarity method to output words whose similarity with each word in the training first seed pool to be evaluated exceeds a predetermined threshold in the training first general word vector set and the training first word set to obtain a training second seed pool to be evaluated; introducing the internal flow text data and constructing an industry word model through the Fine-tuning method to obtain a training industry-specific word set and a set of training industry-specific word vectors; using the cosine similarity method to output words whose similarity with each word in the training first seed pool to be evaluated exceeds a predetermined threshold in the training industry-specific word set and the set of training industry-specific word vectors Output words whose similarity with each word in the training first seed pool to be evaluated exceeds a predetermined threshold to obtain a training third seed pool to be evaluated; pass the training first seed pool to be evaluated, the training second seed pool to be evaluated and the training third seed pool to be evaluated through the text semantic encoder to obtain a training first text feature vector, a training second text feature vector and a training third text feature vector; fuse the training first text feature vector, the training second text feature vector and the training third text feature vector to obtain a training cascade feature vector; perform feature accuracy alignment based on scale representation and inverse recovery on the training cascade feature vector to obtain an optimized training cascade feature vector; pass the optimized training cascade feature vector through the text semantic decoder to obtain a training final seed pool; calculate the cross entropy loss function value between the training final seed pool and the real data of the final seed pool to obtain the final seed pool data cross entropy loss function value; train the text semantic encoder and the text semantic decoder based on the final seed pool data cross entropy loss function value and through gradient descent direction propagation.

[0059] In particular, in the technical solution of the present application, when the training first seed pool to be evaluated, the training second seed pool to be evaluated and the training third seed pool to be evaluated are merged to obtain the final seed pool, in order to perform semantic fusion at the text semantic feature level, a sampling encoder-decoder structure is used to perform seed pool merging, that is, first, a text semantic encoder is used to obtain a training first text feature vector, a training second text feature vector and a training third text feature vector from the training first seed pool to be evaluated, the training second seed pool to be evaluated and the training third seed pool to be evaluated, and then a text semantic decoder is used to obtain the final seed pool from the fused feature vector of the training first text feature vector, the training second text feature vector and the training third text feature vector, such as a training cascade feature vector.

[0060] However, considering that when the cascade feature vector is obtained by concatenating the training first text feature vector, the training second text feature vector and the training third text feature vector, the training first text feature vector, the training second text feature vector and the training third text feature vector have different text semantic association accuracy differences when passing through the decoder due to the different text semantic feature representations of the corresponding training first seed pool to be evaluated, the training second seed pool to be evaluated and the training third seed pool to be evaluated, it may affect the training effect of the training cascade feature vector when it passes through the decoder training. Therefore, during the training process, the applicant of this application performs feature accuracy alignment based on scale representation and inverse recovery on the training cascade feature vector, for example, denoted as V, which is specifically expressed as: the feature accuracy alignment based on scale representation and inverse recovery is performed on the training cascade feature vector using the following optimization formula to obtain the optimized training cascade feature vector; wherein, the optimization formula is:

[0061]

[0062] Where V is the training concatenated feature vector, v i is the eigenvalue of the i-th position of the training cascade feature vector, ‖V‖0 represents the zero norm of the training cascade feature vector, L is the length of the training cascade feature vector, and α is a weight hyperparameter, v ′i is the eigenvalue of the i-th position of the optimized training cascade feature vector.

[0063] Here, in response to the accuracy contradiction between the scale-based high-dimensional feature encoding of text semantic feature parameters and the semantic editing of associated features, the feature accuracy alignment based on scale representation and inverse recovery is generated by treating the semantic editing of associated features as the inverse embedding of the high-dimensional feature encoding of text semantic feature parameters, thereby equipping the feature values ​​represented as the encoding with sparse distribution balance of scale representation, and performing inverse recovery of associated details based on vector counts, so as to achieve adaptive alignment of accuracy differences during the training process and improve the training effect of the cascade feature vector when it is decoded and regressed by the decoder. In this way, keywords related to non-operating income can be automatically mined from the flow data, and the expansion of operating flow keywords can be achieved, thereby improving the ability to identify non-operating flow, so as to better understand and evaluate the borrower's operating ability.

[0064] In summary, the domain word expansion method based on the pre-trained model according to the embodiment of the present invention is explained, which can automatically mine keywords related to non-operating income from the flow data, realize the expansion of operating flow keywords, improve the recognition of non-operating flow, and has high accuracy and generalization ability, thereby effectively portraying the user's operating ability.

[0065] Figure 3 FIG1 is a block diagram of a domain word expansion system based on a pre-training model provided in an embodiment of the present invention. Figure 3 As shown, the domain word expansion system 200 based on the pre-trained model includes: a text data acquisition module 210 for acquiring user transaction flow text data; a pre-processing module 220 for pre-processing the user transaction flow text data to obtain pre-processed user transaction flow text data; a part-of-speech processing module 230 for sorting and arranging the pre-processed user transaction flow text data according to word frequency and filtering the part-of-speech to obtain a first seed pool to be evaluated; a word set generation module 240 for obtaining a first word set and a first general word vector set corresponding to the first word set from a language model pre-trained based on an external massive corpus; a first association analysis module 250 for assigning the first general word to the first seed pool; The quantity set and the first word set are subjected to similarity association analysis with the first seed pool to be evaluated to obtain a second seed pool to be evaluated; an industry word model construction module 260 is used to introduce internal flow text data and construct an industry word model through a Fine-tuning method to obtain an industry-specific word set and a set of industry-specific word vectors; a second association analysis module 270 is used to perform a similarity association analysis on the industry-specific word set and the set of industry-specific word vectors with the first seed pool to be evaluated to obtain a third seed pool to be evaluated; and a merging module 280 is used to merge the first seed pool to be evaluated, the second seed pool to be evaluated and the third seed pool to be evaluated to obtain a final seed pool.

[0066] Those skilled in the art will appreciate that the specific operations of each step in the above-mentioned domain word expansion system based on the pre-training model have been described in detail above. Figures 1 to 2 It has been introduced in detail in the description of the domain term expansion method based on the pre-trained model, and therefore, its repeated description will be omitted.

[0067] As described above, the domain term expansion system 200 based on the pre-trained model according to an embodiment of the present invention can be implemented in various terminal devices, such as a server for domain term expansion based on the pre-trained model. In one example, the domain term expansion system 200 based on the pre-trained model according to an embodiment of the present invention can be integrated into a terminal device as a software module and / or a hardware module. For example, the domain term expansion system 200 based on the pre-trained model can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the domain term expansion system 200 based on the pre-trained model can also be one of the many hardware modules of the terminal device.

[0068] Alternatively, in another example, the domain vocabulary expansion system 200 based on the pre-trained model and the terminal device may also be separate devices, and the domain vocabulary expansion system 200 based on the pre-trained model may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0069] Figure 4 This is an application scenario diagram of a domain word expansion method based on a pre-training model provided in an embodiment of the present invention. Figure 4 As shown, in this application scenario, first, obtain the user transaction flow text data (for example, Figure 4 The first word set (e.g., C1) is obtained from a language model pre-trained based on an external massive corpus. Figure 4 C2) and a first common word vector set corresponding to the first word set (e.g., Figure 4 In C3, we introduce internal stream text data and build an industry word model through the Fine-tuning method to obtain an industry-specific word set (for example, Figure 4 C4 as shown in ) and a collection of industry-specific word vectors (e.g., Figure 4 Then, the obtained user transaction flow text data, the first word set, the first general word vector set, the industry-specific word set and the industry-specific word vector set are input to a server deployed with a domain word expansion algorithm based on a pre-trained model (for example, Figure 4In S) shown in , the server is capable of processing the user transaction flow text data, the first word set, the first general word vector set, the industry-specific word set and the set of industry-specific word vectors based on the domain word expansion algorithm of the pre-trained model to merge the first seed pool to be evaluated, the second seed pool to be evaluated and the third seed pool to be evaluated to obtain a final seed pool.

[0070] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A domain word expansion method based on a pre-training model, characterized in that: include: Obtain user transaction flow text data; Preprocessing the user transaction flow text data to obtain preprocessed user transaction flow text data; Sorting and arranging the pre-processed user transaction flow text data according to word frequency and filtering the text data according to part of speech to obtain a first seed pool to be evaluated; Obtaining a first word set and a first universal word vector set corresponding to the first word set from a language model pre-trained based on an external massive corpus; Performing similarity association analysis on the first general word vector set and the first word set with the first seed pool to be evaluated to obtain a second seed pool to be evaluated, including: using a cosine similarity method to output words in the first general word vector set and the first word set whose similarity with each word in the first seed pool to be evaluated exceeds a predetermined threshold to obtain the second seed pool to be evaluated; Introducing internal stream text data and building an industry word model through the Fine-tuning method to obtain a set of industry-specific words and a set of industry-specific word vectors; Performing a similarity association analysis on the industry-specific word set and the set of industry-specific word vectors with the first seed pool to be evaluated to obtain a third seed pool to be evaluated, including: using a cosine similarity method to output words in the industry-specific word set and the set of industry-specific word vectors whose similarity with each word in the first seed pool to be evaluated exceeds a predetermined threshold to obtain the third seed pool to be evaluated; The first seed pool to be evaluated, the second seed pool to be evaluated, and the third seed pool to be evaluated are merged to obtain a final seed pool, including: passing the first seed pool to be evaluated, the second seed pool to be evaluated, and the third seed pool to be evaluated through a text semantic encoder to obtain a first text feature vector, a second text feature vector, and a third text feature vector; fusing the first text feature vector, the second text feature vector, and the third text feature vector to obtain a cascade feature vector; and passing the cascade feature vector through a text semantic decoder to obtain the final seed pool.

2. The domain term expansion method based on the pre-training model according to claim 1 is characterized in that: Preprocessing the user transaction flow text data to obtain preprocessed user transaction flow text data includes: The user transaction flow text data is processed by word segmentation, word form normalization and part-of-speech tagging to obtain the pre-processed user transaction flow text data.

3. The domain term expansion method based on the pre-training model according to claim 2 is characterized in that: It also includes a training step: used to train the text semantic encoder and the text semantic decoder.

4. The domain term expansion method based on the pre-training model according to claim 3 is characterized in that: The training step comprises: Acquire training data, the training data including transaction flow text data of training users and real data of the final seed pool; Preprocessing the training user transaction flow text data to obtain preprocessed training user transaction flow text data; Sorting and arranging the pre-processed training user transaction flow text data according to word frequency and filtering the text data according to part of speech to obtain a first seed pool to be evaluated for training; Obtaining a first training word set and a first training universal word vector set corresponding to the first training word set from a language model pre-trained based on the external massive corpus; In the training first general word vector set and the training first word set, a cosine similarity method is used to output words whose similarity with each word in the training first seed pool to be evaluated exceeds a predetermined threshold to obtain a training second seed pool to be evaluated; Introducing the internal flow text data and constructing an industry word model through a Fine-tuning method to obtain a set of training industry-specific words and a set of training industry-specific word vectors; In the training industry-specific word set and the training industry-specific word vector set, a cosine similarity method is used to output words whose similarity with each word in the training first seed pool to be evaluated exceeds a predetermined threshold to obtain a training third seed pool to be evaluated; Passing the training first seed pool to be evaluated, the training second seed pool to be evaluated, and the training third seed pool to be evaluated through the text semantic encoder to obtain a training first text feature vector, a training second text feature vector, and a training third text feature vector; fusing the training first text feature vector, the training second text feature vector, and the training third text feature vector to obtain a training cascade feature vector; Performing feature accuracy alignment based on scale representation and inversion recovery on the training cascade feature vector to obtain an optimized training cascade feature vector; Passing the optimized training cascade feature vector through the text semantic decoder to obtain a final training seed pool; Calculating a cross entropy loss function value between the training final seed pool and the real data of the final seed pool to obtain a final seed pool data cross entropy loss function value; The text semantic encoder and the text semantic decoder are trained based on the final seed pool data cross entropy loss function value and propagated in the direction of gradient descent.

5. The method for expanding domain terms based on a pre-training model according to claim 4, characterized in that: Performing feature accuracy alignment based on scale representation and inversion recovery on the training cascade feature vector to obtain an optimized training cascade feature vector, including: Performing feature accuracy alignment based on scale representation and inversion recovery on the training cascade feature vector using the following optimization formula to obtain the optimized training cascade feature vector; Wherein, the optimization formula is: in, is the training concatenated feature vector, is the first of the training concatenated feature vectors The eigenvalues ​​at the positions, represents the zero norm of the training concatenated feature vector, is the length of the training concatenated feature vector, and is the weight hyperparameter, is the first of the optimized training cascade feature vectors The eigenvalues ​​at each position.

6. A domain word expansion system based on a pre-training model, using the domain word expansion method based on a pre-training model according to claim 1, characterized in that: include: A text data acquisition module is used to obtain user transaction flow text data; A preprocessing module, configured to preprocess the user transaction flow text data to obtain preprocessed user transaction flow text data; A part-of-speech processing module, configured to sort and arrange the pre-processed user transaction flow text data according to word frequency and filter the pre-processed user transaction flow text data according to word frequency to obtain a first seed pool to be evaluated; A word set generation module, configured to obtain a first word set and a first universal word vector set corresponding to the first word set from a language model pre-trained based on an external massive corpus; A first association analysis module is configured to perform similarity association analysis on the first general word vector set, the first word set, and the first seed pool to be evaluated to obtain a second seed pool to be evaluated; The industry word model construction module is used to introduce internal stream text data and construct an industry word model through the Fine-tuning method to obtain a set of industry-specific words and a set of industry-specific word vectors; a second association analysis module, configured to perform similarity association analysis on the industry-specific word set and the industry-specific word vector set with the first seed pool to be evaluated to obtain a third seed pool to be evaluated; as well as A merging module is configured to merge the first seed pool to be evaluated, the second seed pool to be evaluated, and the third seed pool to be evaluated to obtain a final seed pool.

7. The domain term expansion system based on the pre-training model according to claim 6, characterized in that: The pre-processing module is used to: The user transaction flow text data is processed by word segmentation, word form normalization and part-of-speech tagging to obtain the pre-processed user transaction flow text data.

Images