Business application text classification method and system based on multilevel feature fusion

Through the multi-level feature fusion business application text classification method, the RoBERTa model is used to extract deep semantic information and keyword position coding, which solves the problems of low efficiency and limited accuracy of business application approval, and realizes intelligent analysis of business application text and Real-time review improves recognition accuracy.

CN119066150APending Publication Date: 2024-12-03EB INFORMATION TECH
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410989070.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing technology has problems of low efficiency and limited accuracy in business application approval, mainly because the manual approval method requires time and human resources, and the results are affected by subjective factors of the approval personnel.

Method used

Using a multi-level feature fusion business application text classification method, by constructing and training a text classification model, keywords are extracted from the input text and fusion features are generated, combined with the RoBERTa model for deep semantic information extraction and keyword position coding, realizing business application Intelligent analysis and real-time review of text.

Benefits of technology

It improves the recognition accuracy of business application text, reduces the false alarm and missed report rates, and enhances the ability to analyze context semantics, especially when the text data set is incomplete, effectively avoiding missed reports, alleviating the problem of category imbalance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066150A_ABST
    Figure CN119066150A_ABST
Patent Text Reader

Abstract

The invention discloses a multilevel feature fusion service application text classification method and system, and the method comprises the steps: constructing and training a text classification model, firstly extracting keywords from an input text through the text classification model, splicing and coding the input text and the extracted keywords, then generating text embedding features of fusion keyword information, and carrying out the classification of the text embedding features; the text embedded features pass through a plurality of coding sub-layers to obtain output features of the last layer and a plurality of middle layers, the output features of the last layer and the output features of the middle layers are fused to generate text fusion features, and finally classification labels which the text fusion features belong to are calculated and output; and obtaining business application information input by a user, combining the business application information into a to-be-classified application text, and inputting the to-be-classified application text into the text classification model so as to obtain a classification label output by the text classification model. The invention relates to the technical field of computers, and can efficiently and intelligently realize real-time review of service applications submitted by users.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a business application text classification method and system with multi-level feature fusion, and relates to the field of computer technology. Background Art

[0002] In various current business application fields, users often need to fill in business application texts that comply with regulations to apply for related operations. Taking the treasury business application as an example, Article 79 of the "China Mobile IT Data Security Management Measures" stipulates that when the operator performs sensitive operations to trigger the treasury application, there must be clear and reasonable business needs and complete the reasons for the operation. For unknown reasons or unreasonable business needs, the authorized person should not approve it. However, since the business applications submitted by users are currently mainly approved manually, manual approval requires a certain amount of time and human resources. With the rapid development of business needs, the amount of approval data has increased dramatically, which will greatly affect the speed and efficiency of business application approval. In addition, the results and accuracy of manual approval will also be affected by the subjective factors of the approver.

[0003] Therefore, how to efficiently and intelligently implement real-time review of business applications submitted by users has become a technical issue that technical personnel focus on. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a multi-level feature fusion business application text classification method and system, which can efficiently and intelligently realize real-time review of business applications submitted by users.

[0005] In order to achieve the above object, the present invention provides a business application text classification method with multi-level feature fusion, comprising:

[0006] Step 1: construct and train a text classification model, wherein the text classification model first extracts keywords from the input text, and concatenates and encodes the input text and the extracted keywords to generate text embedding features that integrate keyword information, and then passes the text embedding features through several encoding sublayers to obtain output features of the last layer and multiple intermediate layers, and then fuses the output features of the last layer with the multiple intermediate layers to generate text fusion features, and finally calculates and outputs classification labels to which the text fusion features belong, and the classification labels include: sufficient application reasons, insufficient application reasons, and non-compliant application reasons;

[0007] Step 2: Obtain the business application information input by the user and combine it into an application text to be classified, and then input the application text to be classified into the text classification model to obtain the classification label output by the text classification model.

[0008] In order to achieve the above object, the present invention also provides a multi-level feature fusion business application text classification system, including:

[0009] A text classification model construction device is used to construct and train a text classification model. The working process of the text classification model is as follows: first, keywords are extracted from an input text, and the input text and the extracted keywords are concatenated and encoded to generate text embedding features that integrate keyword information. Then, the text embedding features are passed through several encoding sub-layers to obtain output features of the last layer and multiple intermediate layers, and then the output features of the last layer and multiple intermediate layers are integrated to generate text fusion features. Finally, the classification label to which the text fusion feature belongs is calculated and output, and the classification label includes: sufficient application reasons, insufficient application reasons, and non-compliant application reasons.

[0010] The business application text judgment device is used to obtain the business application information input by the user and combine it into an application text to be classified, and then input the application text to be classified into a text classification model to obtain a classification label output by the text classification model.

[0011] In order to achieve the above object, the present invention further provides a computing device, comprising:

[0012] Memory and processor;

[0013] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the business application text classification method with multi-level feature fusion are implemented.

[0014] In order to achieve the above-mentioned object, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the business application text classification method with multi-level feature fusion.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: the text classification model of the present invention extracts keywords from the business application text filled in by the user, and then generates text embedding features that integrate the keyword information after splicing and encoding the business application text and the extracted keywords, and finally realizes intelligent analysis and real-time review of the business application text through multi-level feature extraction and fusion; traditional text sensitive information detection methods usually rely on keyword matching or statistical features of the text, but lack in-depth judgment of the semantics of the text content, resulting in high false alarm and omission rates when processing complex scenarios. The present invention can also comprehensively use multi-mode matching and text classification models to classify and identify business application texts, which not only realizes the analysis of contextual semantics, but also avoids omissions as much as possible when the text data set is incomplete, effectively improving the recognition accuracy; the present invention also guides the large model to generate sample data for training and testing by specifying a keyword list, which not only ensures the quality of the generated content, but also alleviates the problem of imbalance in the existing text data categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a business application text classification method using multi-level feature fusion shown in an exemplary embodiment of the present invention.

[0017] Figure 2 It is a structural diagram of a text classification model shown in an exemplary embodiment of the present invention.

[0018] Figure 3 The figure is a flowchart showing a specific process of input text by a text classification model according to an exemplary embodiment of the present invention.

[0019] Figure 4 An exemplary embodiment of the present invention is shown Figure 3 Detailed flowchart of step 13.

[0020] Figure 5 It is a structural diagram of a business application text classification system with multi-level feature fusion shown in an exemplary embodiment of the present invention.

[0021] Figure 6 It is a schematic diagram of the structure of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, the present invention provides a method for classifying business application texts with multi-level feature fusion, including:

[0024] Step 1: construct and train a text classification model, wherein the text classification model first extracts keywords from the input text, and concatenates and encodes the input text and the extracted keywords to generate text embedding features that integrate keyword information, and then passes the text embedding features through several encoding sub-layers to obtain output features of the last layer and multiple intermediate layers, and then fuses the output features of the last layer with the multiple intermediate layers to generate text fusion features, and finally calculates and outputs classification labels to which the text fusion features belong, and the classification labels may include: sufficient application reasons, insufficient application reasons, and non-compliant application reasons;

[0025] Step 2: Obtain the business application information input by the user and combine it into an application text to be classified, and then input the application text to be classified into the text classification model to obtain the classification label output by the text classification model.

[0026] For example, the business application information input by the user includes the application reason, operation content, etc. The application reason and operation content can be combined according to the format "Due to XXXXX reason, perform XXXXX operation, please approve" to generate an application text to be classified. That is, each application text to be classified consists of two parts: application reason and operation content.

[0027] As an optional implementation, the present invention can also construct a multi-mode string feature set to determine whether the business application text is a recognition result of the compliance of the application reason based on the calculation of the classification label using the text classification model, and also includes:

[0028] Step A: Obtain all non-compliant business-sensitive words, divide all business-sensitive words into multiple non-compliant categories, and form a pattern string feature set for each non-compliant category, wherein the pattern string feature set is composed of all business-sensitive words under the non-compliant category, and then construct and train a multi-modal non-compliant recognition model, wherein the multi-modal non-compliant recognition model is used to match the input text with the business-sensitive words in the pattern string feature sets of all non-compliant categories to determine whether the input text is a recognition result of compliance of the application reason.

[0029] Thus, step 2 further includes:

[0030] The business application information input by the user is obtained and combined into an application text to be classified, and then the application text to be classified is input into the multi-modal non-compliance recognition model to determine whether the recognition result output by the multi-modal non-compliance recognition model is that the application reason is compliant. If so, the application text to be classified is further input into the text classification model, and the corresponding prompt information is returned to the user according to the classification label output by the text classification model; if not, a prompt information indicating that the application reason is non-compliant is returned to the user.

[0031] As an optional implementation, in step A, the multi-mode non-compliant recognition model can construct a Trie dictionary tree and a mismatch pointer fail based on the pattern string feature set corresponding to each non-compliant category, so that when the input text fails to match the current business sensitive word, it can jump to the business sensitive word with the longest common prefix and suffix to continue matching. The specific processing process of the multi-mode non-compliant recognition model on the input text is as follows:

[0032] The input text is matched with the pattern string feature set of each non-compliant category. If the number of successful matches p_s (that is, the input text contains p_s business-sensitive words in the pattern string feature set) exceeds a certain threshold, the recognition result that the application reason is non-compliant is output; if the number of successful matches with the pattern string feature set of each non-compliant category is less than the threshold, the recognition result that the application reason is compliant is output.

[0033] The present invention can collect the contents of sensitive word lists publicly released by government agencies, laws and regulations, social media platforms, online forums, etc., classify them according to different dimensions, and supplement non-standard or informal application words (including multiple reduplications, Internet buzzwords, etc.).

[0035] If the input text contains the above-mentioned sensitive words, the user's business application will be directly rejected and the "application reason is not compliant" will be output; if it does not contain sensitive words, the input text will be transferred to the text classification model for the next step of processing.

[0036] In existing research, deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN) are usually used to extract text features. However, the network structures of these models are relatively simple, and it is difficult to learn deeper features. Text classification based on BERT can effectively solve the problem of long-distance contextual information dependency, but BERT obtains the semantic features of the overall information of the input text and lacks feature representation at the phrase level. Since the business application text involves certain keyword information, the position of these keywords in the text will also indirectly affect the classification judgment of the business application text. For example, Text 1: "Due to the reason of handling channel remuneration inquiries, user query information operations are performed, please approve" and Text 2: "Due to user query information reasons, user query information operations are performed, please approve" both contain the keyword "user query information", but the classification labels of these two texts should be "sufficient application reasons" and "inadequate application reasons" respectively. In order to solve the above problems, the present invention utilizes the deep semantic information of the middle layer of the RoBERTa model, and enhances the model's ability to capture key information through keyword-sensitive position encoding, thereby significantly improving the classification performance in text classification tasks. Further as an optional implementation method, such as Figure 2As shown, the text classification model in step 1 can be based on the RoBERTa model, and further includes: a keyword fusion layer, a feature extraction layer, a multi-level feature fusion layer and a classification layer. Figure 3 As shown in the figure, the specific processing process of the text classification model on the input text is as follows:

[0037] Step 11: Input the text into the keyword fusion layer, extract all keywords from the input text, then concatenate the input text and all keywords into an input sequence, convert the input sequence into a token sequence, and encode the token sequence to output the text embedding feature E that fused the keyword information;

[0038] The present invention concatenates the input text and the keywords, for example, using special characters "[CLS]" and "[SEP]" to separate the input text and the keyword information, and the format example of the input sequence is: "[CLS]<text content>[SEP]<keyword content>[SEP]"; the concatenated input sequence is then converted into a token sequence, and the entire token sequence is encoded, so that the position encoding of the keywords can be used to add the order information of the words to the embedded representation of the input text, and the token is input into the embedding layer, and finally the text embedding feature E integrating the keyword information is obtained, so as to realize accurate and intelligent analysis of the business application text;

[0039] Step 12: Input the text embedding feature E to the feature extraction layer. The feature extraction layer consists of 12 Transformer encoders. The output feature calculation formula of each encoding sublayer is: i =Encoder i (T i-1 ), i=1, 2,...,12, and T0=E, Encoder i represents the i-th coding sublayer, T i Represents the output features of the i-th encoding sublayer;

[0040] Step 13: Extract the output features T3, T6, T9, T12 of the 3rd, 6th, 9th, and 12th encoding sublayers of the feature extraction layer. 12 , and input to the multi-level feature fusion layer: T is calculated through the self-attention mechanism 12 The degree of correlation between T3, T6, and T9, and according to the degree of correlation, T3, T6, T9, T 12 Perform feature fusion to obtain text fusion features;

[0041] Step 14: Input the text fusion features into the classification layer, calculate the probability values ​​of the text fusion features in the three classification labels of sufficient application reasons, insufficient application reasons, and non-compliant application reasons, and then output the classification label corresponding to the maximum probability value.

[0042] To prevent the model from overfitting, the dropout function can be used to select the optimal model. In the classification layer, the feature vector output by the fully connected layer is passed through the Softmax classifier to calculate the probability value of the input text fusion feature in different classification labels.

[0043] As an optional implementation, in step 13, when performing text classification tasks based on the RoBERTa model, usually only the hidden vector of the last layer of the pre-trained model is used as the final output of the model. In order to make full use of the intermediate layer encoding information of the RoBERTa pre-trained model, the present invention extracts the output features of the 3rd, 6th, 9th and 12th encoding sublayers respectively, such as Figure 4 As shown, Figure 2 The specific process of step 13 can be as follows:

[0044] Step 131: Based on the self-attention mechanism, the conversion matrix W is used q , W k and W v , calculate T 12 The query vector Q 12 , and T3, T6, T9, T 12 The key and value vectors can be calculated as follows: 12 =W q ×T 12 , Kb=W k ×T b , V b =W v ×T b , where W q , W k and W v are all weight matrices, b = 3, 6, 9, 12, K b 、V b are the key and value vectors of the output features of the b-th encoding sublayer respectively;

[0045] Step 132: According to T 12 The degree of correlation with T3, T6, and T9, and the weight of the output features of each encoding sublayer on feature fusion are calculated: where d k is the dimension of the key vector, α b is the weight of the output feature of the b-th encoding sublayer on feature fusion;

[0046] Step 133: According to the weights of the feature fusion of the output features of each coding sublayer, T3, T6, T9, T 12 Perform feature fusion to obtain text fusion features: T out =∑ b=3,6,9,12 αb ·V b .

[0047] As an optional implementation, the text classification model may also adjust the text fusion features according to different position features of the keyword context in the input text to improve the classification accuracy, and further includes:

[0048] Step B1, using the RoBERTa model to determine the position of each word in the input text, and assigning a basic position code to each word;

[0049] Step B2: Based on Gaussian distribution, according to the position of the word in the input text and the position of the keyword, calculate the position weight of each word in the input text corresponding to each keyword: Among them, w jk is the position weight of word j corresponding to keyword k, s j 、s k are the positions of word j and keyword k respectively. δ is the standard deviation of Gaussian distribution, which is used to control the speed of weight decay and obtain the position weight of each word accordingly.

[0050] The size of the position weight depends on whether the word is a keyword and its importance in the input text. Using Gaussian distribution as the position weight distribution function can make the keyword and its neighboring words obtain higher position weights, while the position weights of words far away from the keyword are lower; when there are multiple keywords in the input text, each keyword will affect the position weights of the words near it, resulting in the possibility that words in certain positions may be jointly affected by multiple keywords. In order to deal with the multiple influences caused by multiple keywords, the maximum value strategy or the average value strategy can be adopted to merge the position weights to avoid over-enhancement. Taking the maximum value strategy as an example, obtaining the position weight of each word can further include: selecting the maximum value from the position weights of the word corresponding to all keywords as the position weight of the word;

[0051] Step B3: Normalize the position weight of each word in the text, and then calculate the position code of each word in the text: P j , They are the position code and base position code of word j, and w j is the normalized position weight of word j;

[0052] Step B4: Calculate the weight of each word's position encoding for feature fusion: Among them, d p is the dimension of the positional encoding, β j is the weight of the position encoding of word j on feature fusion;

[0053] The introduction of position encoding for feature fusion weight calculation can give reasonable attention to features at different positions according to the keyword context in the text;

[0054] Step B5: Adjust text fusion features: Among them, T out ′ is the adjusted text fusion feature, n is the total number of words in the input text, H j It is the hidden state of the position of word j in the input text.

[0055] The present invention can also collect user business application data in different business scenarios, and after data cleaning, data normalization and data labeling, use it as training and test data for text classification models, where:

[0056] Data cleaning: remove duplicate and abnormal data from the original data; delete meaningless characters in the text, such as "?", "·", and spaces;

[0057] Data normalization: Normalize the text data according to the format of "For XXXXX reason, perform XXXXX operation, please approve", that is, each application text content consists of two parts: "application reason" and "operation content";

[0058] Data labeling: The text data is labeled according to its characteristics. The classification labels are divided into three types: "application reasons are not compliant", "application reasons are insufficient" and "application reasons are sufficient". The judgment rules for whether the application content is sufficient are determined based on business experience. That is, as long as the application text involves specific business content, the application text is considered sufficient; otherwise, it is insufficient.

[0059] Since the number of samples of different categories in the collected original data set is seriously unbalanced (500:2), and there are problems such as inconsistent text content formats, the present invention can also use a method combining EDA (synonym replacement, random exchange, random insertion, etc.) and large model technology to enhance the sample data. For some short text data in the data set, the text data is expanded by splicing or reorganizing the text in the text format "Due to XXXXX reasons, perform XXXXX operations, please approve". In addition, in different business scenarios, the business content involved in the application text is different. Since there are large differences in the number of categories in the collected manually annotated data sets, in order to solve the text imbalance problem, especially to increase the number of samples in smaller categories, while saving manpower and time costs, as an optional implementation method, the present invention can also use a large model combined with the prompt method for data enhancement. The specific steps are as follows:

[0060] Build business keyword lists and non-business keyword lists, for example, keyword information corresponding to the reason for application:

[0061] ist1 = ["Group arrears payment"; "Tariff depreciation assessment"; "Paperless contract signing abnormality monitoring"; "Poor quality campus verification"; "User tariff data acquisition"; "Special line shutdown verification"; "Batch speed test exemption flow extraction"; "Late payment fee reduction service"; ...], the keyword information corresponding to the operation content: ist2 = ["Query details of arrears of government and enterprise"; "Query user traffic"; "Process account cancellation and refund"; "View new anti-fraud video specifications"; "Query details of all units"; "Notification of number quality data"; "Data acquisition of CD groups"; "Data acquisition of global communication details"; ...], keywords that do not involve specific business: list3 = ["Business development needs"; "Process business query"; "Process XXX business"; "Customer requirement query"; "Business quality analysis"; "Data analysis needs"; "Data acquisition for analysis"; "Business development"; ...], and construct prompt words based on this, for example, construct a prompt for text data with sufficient application reasons: "You are a text generation expert with rich experience in text creation. Now, we have business Keyword list list1 and non-business keyword list list3, in order to increase the diversity and richness of the text, you can not only directly use the keywords in these two lists, but also generate similar alternatives based on these keywords. When generating text data, for each keyword in list1 and list3, you can consider using its synonyms, related terms or phrases as alternatives. Please combine these original keywords and their similar alternatives to generate at least N text data in the format of "Due to <similar or original list1 keyword>, perform <similar or original list3 keyword> operation, please approve" and then input the prompt words into the large language model. The large language model generates and outputs the corresponding business application text as sample data. By providing a specified keyword list to guide the large language model to generate text content related to a specific business, the problem of the large language model generating irrelevant or meaningless text can be effectively avoided, ensuring the quality of the generated content and improving the efficiency of text generation.

[0062] like Figure 5 As shown, the present invention provides a multi-level feature fusion business application text classification system, including:

[0063] A text classification model construction device is used to construct and train a text classification model. The working process of the text classification model is as follows: first, keywords are extracted from an input text, and the input text and the extracted keywords are concatenated and encoded to generate text embedding features that integrate keyword information. Then, the text embedding features are passed through several encoding sub-layers to obtain output features of the last layer and multiple intermediate layers, and then the output features of the last layer and multiple intermediate layers are integrated to generate text fusion features. Finally, a classification label to which the text fusion feature belongs is calculated and output. The classification label may include: sufficient application reasons, insufficient application reasons, and non-compliant application reasons.

[0064] The business application text judgment device is used to obtain the business application information input by the user and combine it into an application text to be classified, and then input the application text to be classified into a text classification model to obtain a classification label output by the text classification model.

[0065] See also Figure 6 , Figure 6 6 is a block diagram of a computing device 600 shown in an exemplary embodiment of the present specification. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0066] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0067] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0068] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 600 may also be a mobile or stationary server or a cloud server, etc.

[0069] The processor 620 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned multi-level feature fusion business application text classification method.

[0070] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned multi-level feature fusion business application text classification method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned multi-level feature fusion business application text classification method.

[0071] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned multi-level feature fusion business application text classification method.

[0072] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the above-mentioned multi-level feature fusion business application text classification method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned multi-level feature fusion business application text classification method or system.

[0073] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned multi-level feature fusion business application text classification method.

[0074] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned multi-level feature fusion business application text classification method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned multi-level feature fusion business application text classification method or system.

[0075] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] The computer instructions include computer program codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0077] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A business application text classification method with multi-level feature fusion, characterized in that: Included are: Step 1: construct and train a text classification model, wherein the text classification model first extracts keywords from the input text, and concatenates and encodes the input text and the extracted keywords to generate text embedding features that integrate keyword information, and then passes the text embedding features through several encoding sublayers to obtain output features of the last layer and multiple intermediate layers, and then fuses the output features of the last layer with the multiple intermediate layers to generate text fusion features, and finally calculates and outputs classification labels to which the text fusion features belong, and the classification labels include: sufficient application reasons, insufficient application reasons, and non-compliant application reasons; Step 2: Obtain the business application information input by the user and combine it into an application text to be classified, and then input the application text to be classified into the text classification model to obtain the classification label output by the text classification model.

2. The method according to claim 1, characterized in that Also included are: Step A: Obtain all non-compliant business-sensitive words, divide all business-sensitive words into multiple non-compliant categories, and form a pattern string feature set for each non-compliant category, wherein the pattern string feature set is composed of all business-sensitive words under the non-compliant category, and then construct and train a multi-modal non-compliant recognition model, wherein the multi-modal non-compliant recognition model is used to match the input text with the business-sensitive words in the pattern string feature sets of all non-compliant categories to determine whether the input text is a recognition result of compliance of the application reason. Step 2 further includes: The business application information input by the user is obtained and combined into an application text to be classified, and then the application text to be classified is input into the multi-modal non-compliance recognition model to determine whether the recognition result output by the multi-modal non-compliance recognition model is that the application reason is compliant. If so, the application text to be classified is further input into the text classification model, and the corresponding prompt information is returned to the user according to the classification label output by the text classification model; if not, a prompt information indicating that the application reason is non-compliant is returned to the user.

3. The method according to claim 2, characterized in that In step A, the multi-mode non-compliant identification model constructs a Trie dictionary tree and a mismatch pointer fail based on the pattern string feature set corresponding to each non-compliant category, so that when the input text fails to match the current business sensitive word, it can jump to the business sensitive word with the longest common prefix and suffix to continue matching. The specific processing process of the multi-mode non-compliant identification model on the input text is as follows: The input text is matched with the pattern string feature set of each non-compliant category. If the number of successful matches exceeds a certain threshold, the recognition result that the application reason is non-compliant is output; if the number of successful matches with the pattern string feature set of each non-compliant category is less than the threshold, the recognition result that the application reason is compliant is output.

4. The method according to claim 1, characterized in that The text classification model is based on the RoBERTa model, which includes a keyword fusion layer, a feature extraction layer, a multi-level feature fusion layer, and a classification layer. The specific processing process of the text classification model on the input text is as follows: Step 11: Input the text into the keyword fusion layer, extract all keywords from the input text, then concatenate the input text and all keywords into an input sequence, convert the input sequence into a token sequence, and encode the token sequence to output the text embedding feature E that fused the keyword information; Step 12: Input the text embedding feature E to the feature extraction layer. The feature extraction layer consists of 12 Transformer encoders. The output feature calculation formula of each encoding sublayer is: i =Encoder i (T i-1 ), i=1, 2,...,12, and T0=E, Encoder i represents the i-th coding sublayer, T i Represents the output features of the i-th encoding sublayer; Step 13: Extract the output features T3, T6, T9, T12 of the 3rd, 6th, 9th, and 12th encoding sublayers of the feature extraction layer. 12 , and input to the multi-level feature fusion layer: T is calculated through the self-attention mechanism 12 The degree of correlation between T3, T6, and T9, and according to the degree of correlation, T3, T6, T9, T 12 Perform feature fusion to obtain text fusion features; Step 14: Input the text fusion features into the classification layer, calculate the probability values ​​of the text fusion features in the three classification labels of sufficient application reasons, insufficient application reasons, and non-compliant application reasons, and then output the classification label corresponding to the maximum probability value.

5. The method according to claim 4, characterized in that The specific process of step 13 is as follows: Step 131: Based on the self-attention mechanism, the conversion matrix W is used q , W k and W v , calculate T 12 The query vector Q 12 , and T3, T6, T9, T 12 The key and value vectors are calculated as follows: 12 =W q ×T 12 , K b =W k ×T b , V b =W v ×T b , where W q , W k and W v are all weight matrices, b = 3, 6, 9, 12, K b 、V b are the key and value vectors of the output features of the b-th encoding sublayer respectively; Step 132: According to T 12 The degree of correlation with T3, T6, and T9, and the weight of the output features of each encoding sublayer on feature fusion are calculated: where d k is the dimension of the key vector, α b is the weight of the output feature of the b-th encoding sublayer on feature fusion; Step 133: According to the weights of the feature fusion of the output features of each coding sublayer, T3, T6, T9, T 12 Perform feature fusion to obtain text fusion features: T out =∑ b=3,6,9,12 α b ·V b .

6. The method according to claim 5, characterized in that Also included are: Step B1, using the RoBERTa model to determine the position of each word in the input text, and assigning a basic position code to each word; Step B2: Based on Gaussian distribution, according to the position of the word in the input text and the position of the keyword, calculate the position weight of each word in the input text corresponding to each keyword: Among them, w jk is the position weight of word j corresponding to keyword k, s j 、s k are the positions of word j and keyword k respectively, δ is the standard deviation of Gaussian distribution, and the position weight of each word is obtained accordingly; Step B3: Normalize the position weight of each word in the text, and then calculate the position code of each word in the text: P j , They are the position code and base position code of word j, and w j is the normalized position weight of word j; Step B4: Calculate the weight of each word's position encoding for feature fusion: Among them, d p is the dimension of the positional encoding, β j is the weight of the position encoding of word j on feature fusion; Step B5: Adjust text fusion features: Among them, T out ′ is the adjusted text fusion feature, n is the total number of words in the input text, H j It is the hidden state of the position of word j in the input text.

7. The method according to claim 1, characterized in that Also included are: A business keyword list and a non-business keyword list are constructed, and prompt words are constructed based on them. The prompt words are then input into the large language model, and the large language model generates and outputs the corresponding business application text as sample data.

8. A multi-level feature fusion business application text classification system, characterized in that: Included are: A text classification model construction device is used to construct and train a text classification model. The working process of the text classification model is as follows: first, keywords are extracted from an input text, and the input text and the extracted keywords are concatenated and encoded to generate text embedding features that integrate keyword information. Then, the text embedding features are passed through several encoding sub-layers to obtain output features of the last layer and multiple intermediate layers, and then the output features of the last layer and multiple intermediate layers are integrated to generate text fusion features. Finally, the classification label to which the text fusion feature belongs is calculated and output, and the classification label includes: sufficient application reasons, insufficient application reasons, and non-compliant application reasons. The business application text judgment device is used to obtain the business application information input by the user and combine it into an application text to be classified, and then input the application text to be classified into a text classification model to obtain a classification label output by the text classification model.

9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the business application text classification method with multi-level feature fusion described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the business application text classification method with multi-level feature fusion as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Method for text classification through sensitive word detection and illegal content recognition

    CN110851590A

  • Text classification method and device and electronic device

    CN111104516A

  • Text classification method and device, equipment and storage medium

    CN113886577A

  • Case judgment prediction method and system based on BERT hidden layer information and medium

    CN114219682A

  • Text classification method, classifier and system based on key information and dynamic routing

    CN114461804A