Automatic extraction and analysis method for key information of massive credit short messages

Abstract

The invention relates to the technical field of data analysis, in particular to a method for automatically extracting and analyzing key information of massive credit short messages. The method comprises the following steps: replacing specific information in a short message text with a preset semantic tag, and performing word segmentation and vectorization processing on the short message text to obtain a structured text representation; according to each segmented word in the semantic unit, a corresponding preset semantic tag and a template semantic anchor point, determining a semantic anchor point feature index; screening out a target semantic anchor point; for each semantic analysis window, determining semantic description strength according to the number of target semantic anchor points in the window and semantic anchor point feature indexes; determining template similarity according to semantic difference characteristics of a target semantic anchor point and a template semantic anchor point in the window; according to the semantic description strength and the template similarity, determining the regional attention weight of the window; and inputting the word vector of each text slice and the attention weight of the corresponding region into a pre-training language model, and outputting text key information, thereby improving the accuracy of the key information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and particularly relates to a key information automatic extraction and analysis method for massive credit messages. BACKGROUND

[0002] In the field of financial credit, the massive credit messages contain key information such as user repayment reminders, credit changes, overdue notifications, and the like. Accurate extraction of these information is crucial for risk control, customer service, and business analysis.

[0003] At present, the related art relies on manual preset fixed templates to match the message format. However, credit messages have various text formats (such as large differences in notification templates of different institutions), frequent new scams or marketing messages, and the like. The use of fixed templates not only leads to a high template matching missed judgment rate, but also cannot adapt to the semantic analysis needs of unstructured text. In addition, credit messages contain sensitive user information (such as card numbers and amounts), and the existing solutions lack systematic design in privacy protection and compliance processing, which poses a risk of data leakage. Moreover, a pre-trained language model (PLM) is used to extract key information from message text. However, direct processing of message text has high computational cost and insufficient key information focusing capability, which seriously affects the efficiency of extracting key information. SUMMARY

[0004] To solve the technical problem of relying on manual preset fixed templates to match the message format, resulting in high computational cost and insufficient key information focusing capability, which affects the efficiency of extracting key information, the present application provides a key information automatic extraction and analysis method for massive credit messages, and the technical solution adopted is as follows:

[0005] The present application provides a key information automatic extraction and analysis method for massive credit messages, which comprises the following steps: Obtaining credit message text and a preset template containing template semantic labels and corresponding template semantic anchor points; replacing specific information in the message text with preset semantic labels, and performing word segmentation and vectorization processing on the message text to obtain a structured text representation; Dividing the structured text representation into semantic units based on punctuation marks as boundaries; determining semantic anchor point feature indicators of each word based on the semantic association relationship between each word in the semantic unit and the corresponding preset semantic label and template semantic anchor point; and selecting target semantic anchors representing each preset semantic label from each word according to the semantic anchor point feature indicators; constructing a semantic analysis window based on semantic units associated with each preset semantic label; for each window, determining a semantic description strength according to the number of target semantic anchors and the semantic anchor feature indicators within the window; determining a template similarity according to the semantic difference characteristics of the target semantic anchors and the template semantic anchors within the window; determining the regional attention weight of the window according to the semantic description strength and the template similarity; The structured text representation is divided into multiple text slices according to windows, and the word vectors of each slice and the corresponding regional attention weight are input into a pre-trained language model to output text key information.

[0006] Further, before replacing the specific information in the short message text with the preset semantic label, a mass credit short message key information automatic extraction analysis method further comprises: unifying the coding format of the short message text; converting various currency symbols into a standard form; converting Chinese numerals and special numeral expressions into Arabic numeral form.

[0007] Further, the specific information includes numerical data and institution identification information, and the replacing of the specific information in the short message text with the preset semantic label comprises: numerical data is matched using a regular expression, and institution identification information is matched using a financial institution dictionary, and the matched information is replaced with a preset semantic label to form a standardized text.

[0008] Further, the short message text is subjected to word segmentation and vectorization processing to obtain a structured text representation, comprising: adopting a preset word segmentation tool to perform word segmentation processing on the standardized text, removing stop words and obtaining a word segmentation sequence; converting each word in the word segmentation sequence into a fixed-dimensional word vector to generate a corresponding word vector sequence; The standardized text, the word segmentation sequence and the word vector sequence jointly constitute the structured text representation.

[0009] Further, after the structured text representation is divided into semantic units based on punctuation marks, comprising:

[0010] If there is a first semantic unit that does not contain a preset semantic label, the edge word segmentation of the first semantic unit and the word segmentation distance and semantic similarity of each preset semantic label in the adjacent semantic unit are calculated; the preset semantic label with the smallest word segmentation distance and the semantic similarity greater than a preset threshold is preferentially selected as the preset semantic label of the first semantic unit; if the semantic similarity of all adjacent preset semantic labels is lower than the preset threshold, the preset semantic label with the smallest word segmentation distance is selected as the preset semantic label of the first semantic unit.

[0011] Further, the semantic anchor feature indicator determination process comprises: For each word in the semantic unit, calculate the word distance between the word and the preset semantic label corresponding to the semantic unit as the label description distance of the word; From the preset templates, select a template semantic label that is consistent with the preset semantic label type, and obtain all template semantic anchor points corresponding to the template semantic label; calculate the cosine similarity between the word vector of the segmentation and each template semantic anchor point, take the maximum value of the cosine similarity as the anchor semantic similarity coefficient of the segmentation, and take the template semantic anchor point corresponding to the maximum value as the target template semantic anchor point of the segmentation; The anchor feature vector is constructed by combining the segmentation distance and relative word order relationship between the target template semantic anchor point and its corresponding template semantic label; the segmentation feature vector of each segmentation is constructed by combining the segmentation distance and relative word order relationship between each segmentation and the preset semantic label; the similarity between the segmentation feature vector of each segmentation and the corresponding anchor feature vector is calculated as the anchor position feature coefficient of each segmentation; The product of the anchor semantic similarity coefficient and the anchor position feature coefficient is calculated as the template anchor matching index of each word segmentation; the ratio of the template anchor matching index and the label description distance is calculated as the semantic anchor feature index.

[0012] Furthermore, the method of selecting target semantic anchors representing preset semantic labels from each word segment based on the semantic anchor feature index includes: For all the segmented words corresponding to the same preset semantic label, sort them in descending order according to the semantic anchor feature index of each segmented word to obtain a segmented word sorting list; Calculate the difference between adjacent elements in the word segmentation sort list and obtain the two target elements associated with the maximum difference; Filter out the target segmentation in the segmentation sorting list whose semantic anchor feature index is not less than the larger semantic anchor feature index of the two target elements, and use the target segmentation as the target semantic anchor representing the preset semantic label.

[0013] Furthermore, the semantic description strength determination process includes: For each target semantic anchor point, extract the word vectors of all the interval segmentations between the target semantic anchor point and the associated preset semantic label, calculate the variance of the word vectors of all the interval segmentations and perform normalization processing to obtain the modifying feature parameters of the target semantic anchor point; Count the total number of segmented words between each target semantic anchor point and the associated preset semantic label, and normalize the ratio of the total number of segmented words to the corresponding modification feature parameter to obtain the modification index of the target semantic anchor point; The semantic description strength is obtained by summing the product of the modification index of all target semantic anchor points and the feature index of their respective semantic anchor points.

[0014] Further, the template similarity determination process comprises: From the preset template, extract the target template semantic anchor point corresponding to each target semantic anchor point in the semantic analysis window; based on the target template semantic anchor point and its corresponding template semantic label, and the semantic unit associated with the template semantic label, construct a reference window; According to the text sequence, the target semantic anchor points and the preset semantic labels in the semantic analysis window are arranged in sequence to form a first sequence; the target template semantic anchor points and the template semantic labels in the reference window are arranged according to the same rule to form a second sequence; Calculate the average value of the absolute value of the difference between the original sequence numbers of the elements with the same index in the first sequence and the second sequence in the corresponding window; normalize the average value to obtain the structure consistency index; Extract all interval tokens between each target semantic anchor point and its associated preset semantic label in the semantic analysis window to form a first token set; extract all interval tokens between each target template semantic anchor point and its corresponding template semantic label in the reference window to form a second token set; calculate the average cosine similarity of the word vectors of all tokens in the first token set and the second token set as the modification consistency index; Calculate the product of the structure consistency index and the modification consistency index as the template similarity of the semantic analysis window.

[0015] Further, the structured text representation is divided into multiple text slices according to the window, and the word vectors and the corresponding regional attention weights of each slice are input into the pre-trained language model to output the text key information, comprising: The structured text representation is divided into multiple text slices, and each text slice contains corresponding regional attention weights; Multiply each word vector in the text slice by the regional attention weight to obtain a modified word vector; Element-wise multiply the self-attention weight and the regional attention weight of each token in the text slice to obtain a modified attention matrix; input the modified word vector and the modified attention matrix into the subsequent layer of the pre-trained language model to output the text key information.

[0016] The present application has the following beneficial effects:

[0017] The application replaces the value type data and the institutional identification information by preset semantic labels, combines the semantic anchor point screening technology, accurately locates the semantic anchor points with strong association with key information from the short message text, effectively extracts the core semantic features in the text, and then compares with the traditional preset template, realizes the generalization ability of new short message format without relying on fixed format, greatly improves the recall rate and accuracy of text key information, constructs the semantic window and sets the regional attention weight, only inputs the high value text slice (semantic unit containing text key information) into the pre-training language model, focuses on the key semantic unit by correcting the attention weight, reduces the subsequent invalid data processing amount, at the same time, combines the template similarity semantic description intensity, effectively corrects the model attention distribution, avoids irrelevant information interference, significantly improves the mass data processing efficiency under the premise of ensuring the extraction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 A flow chart of a key information automatic extraction and analysis method of a mass credit short message provided by an embodiment of the present application is provided.

[0020] Figure 2 An example diagram of a semantic anchor point feature index determination process provided by an embodiment of the present application is provided.

[0021] Figure 3 An example diagram of a semantic description intensity determination process provided by an embodiment of the present application is provided.

[0022] Figure 4 An example diagram of a template similarity determination process provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0023] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purpose, the following combines the drawings and the preferred embodiments to specifically describe the specific implementation, structure, features and effects of the key information automatic extraction and analysis method of the mass credit short message according to the present application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0025] The application provides a mass credit short message key information automatic extraction and analysis method.

[0026] Referring to Figure 1 , a mass credit short message key information automatic extraction and analysis method flow chart is shown, and the method comprises the following steps:

[0027] S101: Obtain credit short message text and a preset template containing template semantic label and corresponding template semantic anchor point; replace specific information in the short message text with a preset semantic label, and perform word segmentation and vectorization processing on the short message text to obtain a structured text representation.

[0028] It should be understood that the collected credit short message data comes from multiple channels, and the credit short message usually contains numerical data such as amount, date, card number tail number / contract number, state words (such as loan / repayment / overdue / credit / interest rate), and characteristic nouns such as institution name. The original credit short message data is disordered and contains sensitive information. Therefore, in order to ensure the integrity and accuracy of the data, protect the privacy of the user, and accurately analyze the structured text representation representing the key semantic information, a series of preprocessing is required on the original credit short message data.

[0029] It should be noted that the collection channel of the credit short message data is not limited in the embodiment, for example, it can be the credit short message data collected from the credit institution short message gateway, user terminal log, risk control system historical database, etc. Credit short message database, while recording the short message sending time, institution ID, user ID, etc.

[0030] In the embodiment, a series of preprocessing is performed on the original credit short message data, including: constructing a unified data structure containing a unique short message identifier, a desensitized user identifier, a sender identifier, a receiving timestamp, and short message text content; performing privacy and compliance control, only collecting short message meta information and text content necessary for key information extraction, not containing sensitive information directly associated with personal identity, and ensuring data security through desensitization, hashing, permission grading, and closed environment processing; using a high-concurrency message queue to realize real-time storage of streaming data, combining extensible object storage to store original data and meta information for offline analysis, forming a real-time and offline parallel storage architecture; and normalizing the short message text.

[0031] It should be noted that the unified data structure is used to integrate multi-source credit short message data.

[0032] It should be noted that the desensitization processing is achieved by removing the identity in the user association information, the hashing processing converts the user unique identification into an irreversible character sequence, the permission grading mechanism limits the data operation range of different roles, and all data processing is carried out in a closed and controlled environment.

[0033] It should be noted that real-time storage adopts a message queue with high concurrent processing capability, which is used to carry streaming credit message data, guarantee low-latency access of data, and support subsequent real-time analysis requirements; offline storage adopts an extensible object storage service, which is used to store raw messages and meta information processed in compliance for a long time, and meet the data requirements of offline analysis, template mining and other scenarios.

[0034] It should be noted that the normalization processing of the message text includes: unifying the message text encoding format; converting various currency symbols to standard form; converting Chinese numerals and special numeral expressions to Arabic numeral form.

[0035] It should be noted that the specific information includes numerical data and institution identification information, wherein the institution identification information refers to all institution names and associated expressions related to credit business, which are used to explicitly indicate business participants (such as fund providers and service providers) in the message.

[0036] In this embodiment, regular expressions are used to match numerical data, and a financial institution dictionary is used to match institution identification information. The matched information is replaced with a preset semantic label to form a standardized text.

[0037] It should be noted that the specific numerical data in the credit message is determined according to the industry knowledge commonly used in the credit message field, such as amount, date, number, etc.

[0038] The financial institution dictionary usually includes the full name, abbreviation and common alias of credit-related institutions such as banks, consumer finance companies and microfinance companies. The financial institution dictionary is the industry knowledge commonly used in this technical field, and this embodiment will not be described again.

[0039] It should be noted that for different numerical data and institution identification information, the replaced preset semantic label is also different. For example, the preset semantic label includes an amount label, a date label, an institution label, and a number label, which are respectively used to replace the amount value, date information, financial institution name, and contract or card number in the message text.

[0040] For example, the message text is "Dear customer, your A bank credit card with tail number 1234 has been billed this period, and the amount to be paid is 3000 yuan. The payment date is December 20, 2024. Please pay on time." Assuming that the amount label is <amt>, date tag is <date>, the institutional label is <bank>then match "3000 yuan" with a regular expression and replace it with <amt>, "A Bank" is replaced by matching the financial institution dictionary <bank>At the same time, the regular expression is used to replace "2024-12-20" with <date>, get "Dear customer, your account number 1234 is <bank>Credit card, this period's bill has been issued and should be paid <amt>Repayment date is <date>Please pay back in time.

[0041] The preset template is a set of predefined structured rules for guiding the semantic analysis of the short message text. The specific content of the preset template is summarized according to the public knowledge in the technical field, and the embodiment will not be described herein.

[0042] For example, the preset template includes <amt>(Amount), <date>(Date), and <bank>(BANK, institution) three types of template semantic tags and corresponding template semantic anchor points (such as <amt>corresponding "to be returned" "amount"; <date>corresponding to the "due date" "due date").

[0043] In this embodiment, the standardized text is processed by a preset word segmentation tool to remove stop words and obtain a word segmentation sequence; each word in the word segmentation sequence is converted into a fixed-dimensional word vector to generate a corresponding word vector sequence; the standardized text, the word segmentation sequence, and the word vector sequence jointly constitute the structured text representation.

[0044] It should be noted that the preset word segmentation tool can use a publicly available word segmentation tool, such as the jieba word segmentation tool.

[0045] It should be noted that the specific method of word vector conversion is a well-known technical means to those skilled in the art, and will not be described in detail in this embodiment. For example, each word is converted into a fixed-dimensional (such as 100-300 dimensional) word vector using a word2vec model.

[0046] For example, the standardized text is processed by the jieba word segmentation tool to remove the stop words "of" "for" "please" and obtain the word segmentation sequence ["respectful", "customer", "tail number", "1234", "BANK", "credit card", "this period", "bill", "has been issued", "should be returned", "AMT", "repayment date", "DATE", "on time", "repayment"], and then the words are converted into 128-dimensional word vectors to form the structured text representation.

[0047] S102: dividing the structured text representation into semantic units based on punctuation marks as boundaries; determining semantic anchor feature indicators of each word based on semantic association relationships between the words in the semantic units and corresponding preset semantic labels and template semantic anchors; and selecting target semantic anchors representing each preset semantic label from the words based on the semantic anchor feature indicators.

[0048] It should be noted that the punctuation marks at least include a period, a semicolon, and an exclamation point.

[0049] For example, the structured text representation is divided into three semantic units based on punctuation marks as boundaries: semantic unit 1: "Dear customer, your tail number 1234 <bank>Credit card, this bill has been issued"; semantic unit 2: "should be paid back <amt>" ; semantic unit 3: "payment date is <date>, please repay on time."

[0050] In this embodiment, if there is a first semantic unit that does not contain a preset semantic tag, the segmentation distance and semantic similarity between the edge segmentations of the first semantic unit and each preset semantic tag in the adjacent semantic unit are calculated; the preset semantic tag with the smallest segmentation distance and a semantic similarity greater than a preset threshold is preferentially selected as the preset semantic tag of the first semantic unit; if the semantic similarities of all adjacent preset semantic tags are lower than the preset threshold, the preset semantic tag with the smallest segmentation distance is used as the preset semantic tag of the first semantic unit.

[0051] Marginal participle, the sentence-initial or sentence-final participle in the first semantic unit. For example, if the first semantic unit is "warm reminder", the marginal participle of "warm reminder" can be "tips".

[0052] Semantic similarity is used to indicate the cosine similarity between the edge segmentation of the first semantic unit and the word vector of the preset semantic label in the adjacent semantic unit.

[0053] It should be noted that the specific value of the preset threshold is determined according to actual conditions and is not specifically limited in this embodiment. For example, the preset threshold may be 0.6.

[0054] The word segmentation distance refers to the number of intervals between two word segments in the word segmentation sequence of a text message. It is used to quantitatively represent the relative position relationship of different word segments in the text. The smaller the word segmentation distance between two word segments, the closer they are in the text and the closer their semantic relationship may be.

[0055] It is understandable that the preset semantic tags are also word segmentations in the text.

[0056] For example, the segmented words obtained after the SMS text is segmented are arranged in the order of the original text to form an ordered segmented word sequence. Each segmented word corresponds to a unique position number in the sequence (starting from 0 or 1 and increasing in sequence). For any two segmented words A and B, assuming that the position number of A is i and the position number of B is j, then the segmented word distance between the two is │i-j│ (that is, the absolute value of the difference in the position numbers).

[0057] The process of determining semantic anchor feature indicators is as follows: Figure 2 As shown, including: S102-1: For each word segment in the semantic unit, calculate the word segment distance between the word segment and the preset semantic label corresponding to the semantic unit as the label description distance of the word segment.

[0058] It is understandable that the tag description distance of the preset semantic tag is not calculated.

[0059] For example, "repayment date" and the preset semantic tag of semantic unit 3 <date>The word segmentation sequence is adjacent, the word segmentation distance is 1, and therefore the label description distance is 1.

[0060] S102-2: From the preset template, a template semantic label consistent with the preset semantic label type is selected, and all template semantic anchor points corresponding to the template semantic label are obtained; a word vector cosine similarity between the word segmentation and each template semantic anchor point is calculated, the maximum value in the cosine similarity is taken as an anchor point semantic similarity coefficient of the word segmentation, and the template semantic anchor point corresponding to the maximum value is taken as a target template semantic anchor point of the word segmentation.

[0061] For example, in the preset template <date>The template semantic anchors corresponding to (date) are "repayment date" and "maturity". The cosine similarity between the word vector of the word segmentation (repayment date) and each template semantic anchor is calculated. Among them, the cosine similarity with the template semantic anchor (repayment date) is the maximum, and the target template semantic anchor of the word segmentation (repayment date) is "repayment date".

[0062] It should be noted that the specific calculation method of the word vector cosine similarity is a technical means well known to those skilled in the art and will not be repeated in this embodiment.

[0063] S102-3: Construct an anchor feature vector based on the segmentation distance and relative word order relationship between the target template semantic anchor and its corresponding template semantic label; construct a segmentation feature vector for each segmentation based on the segmentation distance and relative word order relationship between each segmentation and the preset semantic label; calculate the similarity between the segmentation feature vector of each segmentation and the corresponding anchor feature vector as the anchor position feature coefficient of each segmentation.

[0064] It should be noted that if the template semantic label is before the target template semantic anchor, the relative word order relationship is negative; if the template semantic label is after the target template semantic anchor, the relative word order relationship is positive. For example, the target template semantic anchor "repayment date" and its corresponding template semantic label <date>the word segmentation distance is 1 (preset in the template), and <date>If the "repayment date" is after the "due date" (the relative order relationship is positive), the anchor feature vector is (1, positive).

[0065] Similarly, if the preset semantic label is before the segmentation, the relative order relationship is negative; if the preset semantic label is after the segmentation, the relative order relationship is positive, for example, the segmentation "repayment date" and its corresponding preset semantic label <date>the word segmentation distance of " " is 1, and <date>If the "repayment date" is after the relative sequence relationship (positive), the segmentation feature vector of the segmented word (repayment date) is (1, positive).

[0066] It should be noted that the specific method of measuring the similarity of the two vectors is a technology known to those skilled in the art, and will not be described in detail in the present embodiment. For example, the cosine similarity of the segmentation feature vector and the corresponding anchor feature vector is calculated.

[0067] S102-4: Calculate the product of the anchor semantic similarity coefficient and the anchor position feature coefficient as the template anchor matching degree index of each segmented word; calculate the ratio of the template anchor matching degree index and the label description distance as the semantic anchor feature index.

[0068] It should be noted that the greater the anchor semantic similarity coefficient of a segmented word, the higher the semantic similarity of the segmented word and the semantic anchor of the target template in the preset template, the greater the probability of the segmented word as a semantic anchor in the current structured text representation. If the anchor position feature coefficient of a segmented word is greater, it means that the "position relationship" of the segmented word and the target template semantic anchor is closer (such as 1 position before the semantic label), and the spatial position of the segmented word also has the characteristics of the semantic anchor. If the label description distance of a segmented word is greater, it means that the position of the segmented word and the key information (preset semantic label) is farther, and the possibility of the segmented word as a semantic anchor is smaller. Therefore, by comprehensively analyzing each segmented word from the three dimensions of "semantic similarity, position proximity, and template feature", if the comprehensive performance of a segmented word is better, the segmented word is more likely to be a real semantic anchor.

[0069] It should be noted that when calculating the semantic anchor feature index, if the label description distance is zero (i.e., the position of the target semantic anchor and the preset semantic label coincides in the segmented word sequence), the label description distance is directly assigned a preset minimum positive value (such as 0.001). The preset minimum positive value is a constant much smaller than 1, and its value does not affect the relative size order of the semantic anchor feature index, but only ensures the mathematical validity of the division operation.

[0070] In the present embodiment, for all segmented words corresponding to the same preset semantic label, the segmented words are sorted in descending order of the semantic anchor feature index of each segmented word to obtain a segmented word sorting list. The difference between adjacent elements in the segmented word sorting list is calculated, and the two target elements associated with the maximum difference value are obtained. The target segmented word in the segmented word sorting list whose semantic anchor feature index is not less than the larger semantic anchor feature index of the two target elements is selected as the target semantic anchor representing the preset semantic label.

[0071] It should be noted that the same preset semantic label can correspond to multiple segmented words.

[0072] For example, for "all words corresponding to the same preset semantic label", the words are sorted in descending order according to the semantic anchor feature indicators, and a word sorting list is obtained, which is assumed to be [1.8, 1.7, 1.0, 0.9]. The two elements with the largest adjacent element difference (i.e., "inflection point") are found, wherein the inflection point with the largest adjacent difference is 1.7 and 1.0 (difference 0.7). The target word with a semantic anchor feature indicator not less than 1.7 is selected as the target semantic anchor.

[0073] S103: Construct a semantic analysis window based on the semantic units associated with each preset semantic label; for each window, determine the semantic description strength according to the number of target semantic anchors in the window and the semantic anchor feature indicators; determine the template similarity according to the semantic difference characteristics of the target semantic anchors and the template semantic anchors in the window; determine the regional attention weight of the window according to the semantic description strength and the template similarity.

[0074] It should be understood that in the complex context of the short message text, the preset semantic label is often closely related to its context, and simply identifying the semantic label is not enough to analyze the key business features of the short message text. Since one label may correspond to multiple target semantic anchors, and the real business meaning needs to be analyzed in combination with the semantic label and the context, it is necessary to take the preset semantic label as the center, combine the surrounding semantic anchors, and establish an analysis window to further focus on the key information.

[0075] It should be noted that the analysis window needs to contain two types of semantic units: one type of semantic unit is the semantic unit divided according to the punctuation marks; the other type of semantic unit is the semantic unit to which the target semantic anchors corresponding to the preset semantic label belong (for example <amt>The corresponding target semantic anchor "should return" is in the "should return <amt>"semantic unit").

[0076] For example, if the preset semantic label <date>(date) is "Please return the completed form to <date>"front payment", the corresponding target semantic anchor point "due" where the sentence (i.e., semantic unit) is "bill is due", then <date>the analysis window must contain "please <date>"Please pay the outstanding balance by the due date" and "The bill is due" form the complete context "The bill is due. Please pay the outstanding balance by the due date." <date>"front payoff".

[0077] Summing up the modification indicators of all target semantic anchors with the multiplication of the respective semantic anchor feature indicators, the semantic description intensity is obtained.

[0078] It should be understood that the measurement of whether a target semantic anchor of a semantic analysis window can clearly and accurately describe the key information can be analyzed from two dimensions of "message context itself" and "similarity with the template".

[0079] The semantic description intensity determination process, as shown in Figure 3 includes: It should be understood that focusing on the message context itself can be achieved by focusing on the semantic quality of the window itself, that is, the more dense and strong the target semantic anchors in the window are, and the more modification words (such as "must" and "as soon as possible") there are, the more business urgency can be reflected, and the more sufficient the semantic description of the window to the preset semantic label is, and the stronger the business indication is.

[0080] S103-1: For each target semantic anchor, extract the word vectors of all interval tokens between the target semantic anchor and the associated preset semantic label, calculate the variance of all interval token word vectors and perform normalization processing to obtain the modification feature parameter of the target semantic anchor.

[0081] It should be understood that in the context of credit messages, the character distance between the semantic label and its corresponding semantic anchor does not simply reflect the strength of semantic relevance. When the character distance is large, there are usually more modification words indicating the same direction between them. These modification words are often used to emphasize urgency or importance, such as "must immediately", "please as soon as possible", "must be completed before × × day", etc. Unlike the rule that the relevance decreases with the increase of distance in general text, in the context of credit messages, the existence of such modification words often gives the semantic anchor stronger business indication and urgency. Therefore, in the analysis process, higher weight should be given to such long-distance semantic anchors to reflect their actual importance in semantic interpretation and business risk prompt.

[0082] It should be noted that the specific calculation method of the variance of the word vector is a well-known technical means to those skilled in the art, and will not be described here.

[0083] For example, assuming that a semantic analysis window is: "penalty <amt>, the interest is calculated by day", wherein the preset semantic label is <amt>, the target semantic anchor is "penalty interest", the window is <amt>The total number of associated target semantic anchor points is two for "penalty": "penalty" and <amt>There is no interval between the words, the word vector variance is 0, and the normalized modification feature parameter = 1.0 (the smaller the variance, the smaller the interference of interval word segmentation on the target semantic anchor point, and the higher the modification feature parameter); for "penalty interest": "penalty interest" and <amt>1 wordpiece, the variance of the word vector is small, and the normalized modification feature parameter = 0.8.

[0084] S103-2: Count the total number of interval wordpieces between each target semantic anchor point and the associated preset semantic label, normalize the ratio of the total number of interval wordpieces to the corresponding modification feature parameter, and obtain the modification index of the target semantic anchor point.

[0085] It should be noted that after normalization, even if the variance is 0 (such as no interval wordpieces or interval wordpieces are completely consistent), the normalized parameter will be mapped to the maximum value (such as 1.0), and the modification feature parameter cannot be zero.

[0086] S103-3: Sum the product of the modification index of all target semantic anchor points and the respective semantic anchor point feature index to obtain the semantic description strength.

[0087] It should be noted that the more the total number of interval wordpieces of a certain target semantic anchor point, the more semantic clues related to the key information in the window; the larger the ratio, the stronger the "business indication strengthening effect" of the modifying words on the semantic anchor point (such as multiple groups of same-direction modifying words superimposed, highlighting the urgency of the repayment date), and the larger the modification index.

[0088] The template similarity determination process is as shown in Figure 4 , which includes: S103-4: Extract target template semantic anchor points corresponding to each target semantic anchor point in the semantic analysis window from the preset template; based on the target template semantic anchor points and their corresponding template semantic labels, and the semantic units associated with the template semantic labels, construct a reference window.

[0089] It should be noted that there are a large number of fixed structures in the credit SMS (such as the amount of money after "should be paid" and the date after "due"), even if the overall SMS does not match the template, if the local window is similar to the preset template structure, it still has high key information value, therefore, in order to measure the matching degree of the semantic analysis window and the preset template, the structure of the analysis window and whether the modifying words conform to the typical expression mode of the key information in the credit business can be analyzed.

[0090] For example, based on the extracted target template semantic anchor point (such as "should be paid"), find its corresponding template semantic label preset template <amt>In the preset template, find the one containing <amt>and semantic units of the target template semantic anchor. It is assumed that there is "should be returned" in the preset template <amt>Please pay on time" such as the expression, the target template semantic anchor, template semantic label and the associated semantic unit are integrated, build out the reference window "should be returned <amt>Please pay on time.

[0091] S103-5: According to the text sequence, arrange the target semantic anchor points and preset semantic tags in the semantic analysis window in turn to form a first sequence; arrange the target template semantic anchor points and template semantic tags in the reference window according to the same rule to form a second sequence.

[0092] Text sequence, the arrangement order of the word segmentation in the text, maintains the order of the word segmentation in the original short message text, ignores irrelevant word segmentation (such as punctuation, auxiliary word), and only arranges the core word segmentation in turn.

[0093] For example, assume that the window content of the semantic analysis window is: "penalty <amt>, the fine interest is calculated per day", core element: target semantic anchor point "penalty" "fine interest", preset semantic label <amount>, original word order (from left to right): "penalty" → <amt>→"penalty interest", then the first sequence: ["liquidated damages", <amt>, "penalty interest"] ; assume reference window window content: "shall be paid <amt>, overdue by the day interest", core element: target template semantic anchor point " should be returned" " overdue", template semantic label <amt>, the primitive sequence (from left to right): "should return" -> "should return" <amt>→ "overdue", then the second sequence: ["due", "overdue", <amt>, "overdue" ].

[0094] S103-6: Calculate the average value of the absolute value of the difference between the original sequence numbers of the elements with the same index in the first sequence and the second sequence in the respective corresponding window, normalize the average value, and obtain the structural consistency index.

[0095] It should be noted that if the number of elements is inconsistent, virtual elements are filled to make the sequence length equal.

[0096] It should be noted that before calculating the average value, the first sequence and the second sequence are aligned in sequence, taking the "element index" of the sequence as the reference, and the elements with the same index in the first sequence and the second sequence are one-to-one corresponding, for example, the first element in the first sequence corresponds to the first element in the second sequence, the second element in the first sequence corresponds to the second element in the second sequence, and so on.

[0097] In this embodiment, the window corresponding to the elements of the first sequence is the semantic analysis window; and the window corresponding to the elements of the second sequence is the reference window.

[0098] The original sequence number refers to the real position sequence number of the element in the word segmentation sequence, which is not the index within the sequence.

[0099] For example, it is assumed that: the original sequence number of the first sequence element is "penalty" (the real position sequence number is 5), <amt>(real position number is 6), "penalty interest" (real position number is 8); the original number of the second sequence element: "should be repaid" (real position number is 2), <amt>If the original sequence number difference is 3, the absolute value of the difference is: 1st position ("penalty" and "due"): |5-2|=3; 2nd position ("overdue" and "due"): |3-5|=2; 3rd position ("overdue" and "due"): |3-3|=0. Thus, the 1st position is the most significant position, the 2nd position is the second most significant position, and the 3rd position is the least significant position. <amt>With <amt>|6-3|=3; 3rd position ("penalty" vs. "overdue"): |8-5|=3.

[0100] S103-7: Extract all interval tokens between each target semantic anchor and its associated preset semantic label within the semantic analysis window, to form a first token set; extract all interval tokens between each target template semantic anchor and its corresponding template semantic label within the reference window, to form a second token set; calculate the average cosine similarity of the word vectors of all tokens in the first token set and the second token set as the modification consistency index.

[0101] It should be noted that in the case of the first token set or the second token set being empty, the average cosine similarity of the word vectors of all tokens in the first token set and the second token set is temporarily set to 1.

[0102] It should be noted that in the case of the number of elements in the first token set and the second token set being different, the set with fewer elements needs to be completed first (completion rule: repeat the elements already in the set until the number of elements in the first token set and the second token set is consistent), and then calculate the cosine similarity of the word vectors of the corresponding tokens.

[0103] In this embodiment, the cosine similarity of the word vectors of the corresponding tokens in the first token set and the second token set is calculated as the first similarity, and the average of all first similarities is calculated as the modification consistency index.

[0104] S103-8: Calculate the product of the structure consistency index and the modification consistency index as the template similarity of the semantic analysis window.

[0105] It should be noted that "cosine similarity" is used to measure the angle between two vectors. The smaller the angle, the closer the similarity is to 1 (the more similar the semantics); the larger the angle, the closer the similarity is to 0 (the less similar the semantics). Therefore, if the modification consistency index is larger, it means that the semantic pattern and word usage habit of the modifying words in the semantic analysis window are more consistent with the preset template (for example, the preset template "please as soon as possible" is more consistent with the semantic pattern "please as soon as possible" in the semantic analysis window). <date>, semantic analysis window also appears "please as soon as possible”); if the structural consistency index is greater, the semantic analysis window is more consistent with the preset semantic label-target semantic anchor sequence structure and template, and the key information expression of the credit message is more consistent, and the template similarity of the semantic analysis window is also higher.

[0106] In this embodiment, the product of the semantic description intensity and the template similarity is normalized as the window value score of the window; and the window value score is used as the regional attention weight for determining the window.

[0107] It should be understood that the greater the semantic description intensity of a semantic analysis window, the richer the semantic clues related to the key information in the semantic analysis window and the stronger the "business indication reinforcement” of the modifier words on the semantic anchor, and the more important the semantic analysis window is to the overall structured text representation; the higher the template similarity of a semantic analysis window, the higher the matching degree of the semantic analysis window with the preset template, and the more important the semantic analysis window.

[0108] S104: The structured text representation is divided into multiple text slices according to windows, and the word vectors of each slice and the corresponding regional attention weight are input into a pre-trained language model to output the text key information.

[0109] It should be understood that the pre-trained language model has the same attention degree for all words in the text by default. When directly processing a large amount of credit messages, the pre-trained language model is easy to be distracted by irrelevant information (such as the greeting "Dear customer” and the redundant prompt "Please handle it in time”), and the accuracy of text key information extraction is reduced. It is known that the values of different text slices are different, and the high-value text slice contains the main key information, and the pre-trained language model should be guided to focus on the high-value text slice first. Therefore, the word vectors of the high-weight window can be amplified by the regional attention weight, the self-attention of the words in the high-weight window can be strengthened and focused, and finally the attention distribution is adjusted to reduce irrelevant information interference and improve the recall rate of key information.

[0110] In this embodiment, the structured text representation is divided into multiple text slices according to windows, and each text slice contains a corresponding regional attention weight; each word vector in the text slice is multiplied by the regional attention weight to obtain a modified word vector; the self-attention weight of each word in the text slice is multiplied by the regional attention weight element by element to obtain a modified attention matrix; and the modified word vector and the modified attention matrix are input into the subsequent layer of the pre-trained language model to output the text key information.

[0111] It should be noted that one text slice corresponds to one semantic analysis window, and contains the complete short message text segment in the semantic analysis window.

[0112] It should be noted that the text slice carries not only the regional attention weight, but also the start and end indexes, which are used to locate the real position of the text key information in the original short message text when the text key information is output subsequently.

[0113] It should be noted that the self-attention weight of each word is determined based on the self-attention matrix output by the subsequent (encoder) layer of the pre-trained language model, and the embodiment will not be described again.

[0114] It should be noted that the above-mentioned embodiment sequence of the application is only for description, and does not represent the advantages and disadvantages of the embodiment. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous.

[0115] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments.< / date> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / amt> < / date> < / date> < / date> < / date> < / date> < / amt> < / amt> < / date> < / date> < / date> < / date> < / date> < / date> < / date> < / amt> < / bank> < / date> < / amt> < / bank> < / date> < / amt> < / date> < / amt> < / bank> < / date> < / bank> < / amt> < / bank> < / date> < / amt>

Claims

1. A method for automatically extracting and analyzing key information from massive amounts of credit text messages, characterized in that: The method comprises: Obtain the credit SMS text and a preset template containing template semantic tags and corresponding template semantic anchors; replace specific information in the SMS text with the preset semantic tags, and perform word segmentation and vectorization on the SMS text to obtain a structured text representation; The structured text representation is divided into semantic units based on punctuation marks. The semantic anchor feature index of each word is determined based on the semantic association between each word and the corresponding preset semantic label and template semantic anchor within the semantic unit. Based on the semantic anchor feature index, the target semantic anchor representing each preset semantic label is selected from each word. A semantic analysis window is constructed based on the semantic units associated with each preset semantic tag. For each window, the semantic description strength is determined based on the number of target semantic anchors in the window and the semantic anchor feature indicators. The template similarity is determined based on the semantic difference characteristics between the target semantic anchor in the window and the template semantic anchor. The regional attention weight of the window is determined based on the semantic description strength and template similarity. The structured text representation is divided into multiple text slices by window, the word vector of each slice and the corresponding area attention weight are input into the pre-trained language model, and the key information of the text is output.

2. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 1, characterized in that: Before replacing the specific information in the SMS text with the preset semantic tag, the method further includes: Unify the SMS text encoding format; convert various currency symbols into standard form; convert Chinese numbers and special number expressions into Arabic numerals.

3. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 2, characterized in that: The specific information includes numerical data and organization identification information. The specific information in the text of the SMS message is replaced by a preset semantic tag, including: Regular expressions are used to match numerical data, and the financial institution dictionary is used to match institution identification information. The matched information is replaced with preset semantic tags to form standardized text.

4. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 3, characterized in that: The word segmentation and vectorization processing of the SMS text to obtain a structured text representation includes: Use the preset word segmentation tool to segment the standardized text, remove stop words and obtain the word segmentation sequence; Convert each word in the word segmentation sequence into a fixed-dimensional word vector to generate the corresponding word vector sequence; Normalized text, word segmentation sequence and word vector sequence together constitute structured text representation.

5. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 1, characterized in that: After dividing the structured text representation into semantic units based on punctuation marks, the following steps are included: If there is a first semantic unit that does not contain a preset semantic label, calculate the segmentation distance and semantic similarity between the edge segmentations of the first semantic unit and the preset semantic labels in the adjacent semantic units; preferentially select the preset semantic label with the smallest segmentation distance and a semantic similarity greater than a preset threshold as the preset semantic label of the first semantic unit; if the semantic similarities of all adjacent preset semantic labels are lower than the preset threshold, select the preset semantic label with the smallest segmentation distance as the preset semantic label of the first semantic unit.

6. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 5, characterized in that: The process of determining the semantic anchor feature index includes: For each word in the semantic unit, calculate the word distance between the word and the preset semantic label corresponding to the semantic unit as the label description distance of the word; From the preset templates, select a template semantic label that is consistent with the preset semantic label type, and obtain all template semantic anchor points corresponding to the template semantic label; calculate the cosine similarity between the word vector of the segmentation and each template semantic anchor point, take the maximum value of the cosine similarity as the anchor semantic similarity coefficient of the segmentation, and take the template semantic anchor point corresponding to the maximum value as the target template semantic anchor point of the segmentation; The anchor feature vector is constructed by combining the segmentation distance and relative word order relationship between the target template semantic anchor point and its corresponding template semantic label; the segmentation feature vector of each segmentation is constructed by combining the segmentation distance and relative word order relationship between each segmentation and the preset semantic label; the similarity between the segmentation feature vector of each segmentation and the corresponding anchor feature vector is calculated as the anchor position feature coefficient of each segmentation; The product of the anchor semantic similarity coefficient and the anchor position feature coefficient is calculated as the template anchor matching index of each word segmentation; the ratio of the template anchor matching index and the label description distance is calculated as the semantic anchor feature index.

7. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 6, characterized in that: The target semantic anchor points representing the preset semantic labels are selected from each word segment based on the semantic anchor feature index, including: For all the segmented words corresponding to the same preset semantic label, sort them in descending order according to the semantic anchor feature index of each segmented word to obtain a segmented word sorting list; Calculate the difference between adjacent elements in the word segmentation sort list and obtain the two target elements associated with the maximum difference; Filter out the target segmentation in the segmentation sorting list whose semantic anchor feature index is not less than the larger semantic anchor feature index of the two target elements, and use the target segmentation as the target semantic anchor representing the preset semantic label.

8. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 4, characterized in that: The semantic description strength determination process includes: For each target semantic anchor point, extract the word vectors of all the interval segmentations between the target semantic anchor point and the associated preset semantic label, calculate the variance of the word vectors of all the interval segmentations and perform normalization processing to obtain the modifying feature parameters of the target semantic anchor point; Count the total number of segmented words between each target semantic anchor point and the associated preset semantic label, and normalize the ratio of the total number of segmented words to the corresponding modification feature parameter to obtain the modification index of the target semantic anchor point; The semantic description strength is obtained by summing the product of the modification index of all target semantic anchor points and the feature index of their respective semantic anchor points.

9. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 6, characterized in that: The template similarity determination process includes: Extracting target template semantic anchors corresponding to target semantic anchors in the semantic analysis window from the preset template; constructing a reference window based on the target template semantic anchors and their corresponding template semantic labels, as well as semantic units associated with the template semantic labels; Arrange the target semantic anchors and preset semantic labels in the semantic analysis window in sequence according to the word order of the text to form a first sequence; and arrange the target template semantic anchors and template semantic labels in the reference window in the same order to form a second sequence; Calculating the average of the absolute values ​​of the original sequence number differences of the elements with the same index in the first sequence and the second sequence in their respective corresponding windows, and normalizing the average value to obtain a structural consistency index; Extract all the intervening segmentations between each target semantic anchor and its associated preset semantic label in the semantic analysis window to form a first segmentation set; extract all the intervening segmentations between each target template semantic anchor and its corresponding template semantic label in the reference window to form a second segmentation set; calculate the average cosine similarity of the word vectors of all segmentations in the first segmentation set and the second segmentation set as a modification consistency indicator; The product of the structural consistency index and the modification consistency index is calculated as the template similarity of the semantic analysis window.

10. The method for automatically extracting and analyzing key information from massive credit SMS messages according to claim 4, characterized in that: The structured text representation is divided into multiple text slices by window, the word vector of each slice and the corresponding regional attention weight are input into the pre-trained language model, and the key information of the text is output, including: The structured text representation is divided into multiple text slices by window, and each text slice contains the corresponding regional attention weight; Multiply each word vector in the text slice by the regional attention weight to obtain the modified word vector; The self-attention weight of each word in the text slice is multiplied element-by-element by the regional attention weight to obtain the corrected attention matrix; the corrected word vector and the corrected attention matrix are passed into the subsequent layer of the pre-trained language model to output the key information of the text.

Images