Alarm text classification method based on CBERT-MDPCNN model

Abstract

The invention discloses an alarm text classification method based on a CBERT-MDPCNN model, and the method employs a GPT4 large language model to carry out alarm text data enhancement, obtains a text feature matrix through a word embedding layer, converts the text feature matrix into a multi-convolution kernel text feature vector which can be processed by the MDPCNN through a BERT pre-training model, and carries out the classification of the alarm text through the multi-convolution kernel text feature vector. And finally, inputting an MDPCNN multi-head classifier model to realize an alarm text classification task. According to the method, the alarm text can be quickly classified, the classification accuracy of the model on the alarm text is improved, the requirement of model training on the data sample size of the alarm text is greatly reduced, a certain model generalization ability is achieved, and the method is suitable for the alarm text of a network-related alarm condition, a non-network-related alarm condition and the like.

Description

Technical Field

[0001] The present invention relates to the field of police text classification, and specifically refers to a police text classification problem based on a CBERT-MDPCNN model. Background Art

[0002] In the past decade, with the rapid development of computer and artificial intelligence technology, the application of computer intelligence technology in the information and intelligence of modern police work is showing an increasingly important role. In the public security system, the efficient classification of police text data is crucial to improving the efficiency of case retrieval and investigation, and also plays a key supporting role in the in-depth analysis and mining of crime patterns. However, facing thousands of new alarm information every day, the traditional manual classification method is often inefficient, easily affected by subjective factors, and has a certain degree of uncertainty. Therefore, there is an urgent need for a method that can quickly and accurately classify police text intelligently to help police officers understand and handle police more quickly and accurately, improve the response speed and efficiency of the public security system, and provide more targeted data support for relevant departments, so as to better maintain social order and public safety. Therefore, in-depth research on the classification of police text has important practical significance and application prospects.

[0003] At present, the research on police text classification is still in its preliminary stage, covering research results on traffic police, criminal cases and hierarchical classification. With the rapid development of the Internet, new Internet-related crimes are on the rise. Criminal activities carried out through new Internet technologies and platforms have caused huge property losses to victims. These new Internet-related criminal cases have varied methods and complex charges, which poses a great challenge to prevention and crackdown. The subdivision of such cases is helpful for the management and detection of cases. For example, emotional fraud is usually carried out by the same person in multiple roles, so setting up emotional fraud categories can help police officers focus on investigating similar cases and obtain more clues. Therefore, it is necessary to study a model suitable for police text classification.

[0004] In terms of general text classification, it has gradually developed from the traditional machine learning method to the deep learning method based on neural network, and has been widely used in information mining fields such as spam classification, sentiment analysis, news topic classification, and identification of key elements of judicial cases. Among them, Wang Qian et al. proposed a news topic text classification method based on the multi-head attention pooling mechanism of RoBERTaRCNN, and verified the feasibility of the model in news classification tasks through ablation experiments. Mao Xingliang et al. proposed a key case element identification method that integrates global and local information. Based on the test of judicial public data set, it effectively improved the accuracy of element entity classification. Zhengxuan Wu et al. combined the semantics of structured self-attention weight encoding with sentiment analysis, and two tests showed that the method encoded rich semantics and matched the human interpretation of semantics. In view of the problem that traditional models cannot recognize the polysemy of words, Chen Zhiqun et al. proposed a Weibo comment tendency analysis method based on BERT and bidirectional LSTM, and used Sina Weibo comments for comparative experiments, showing that the accuracy of this method is higher than other mainstream tendency analysis models.

[0005] In terms of police text classification, in recent years, the public security industry has used big data technology to build business platforms for various police types. The convergence of various data information enables more efficient dispatch and command in public security, investigation and traffic. The data collected by the business system includes a large amount of police text information, which provides important data support for intelligence analysis and case investigation. However, in the face of massive police text data, letting the machine "learn" intelligent recognition of police text is one of the effective ways to improve the speed of police processing and reduce the manual processing process of police personnel. Wang Mengxuan et al. built a text classification system for 110 police texts and proposed a model based on improved convolutional recurrent neural network, which effectively improved the accuracy of case classification. Li Yunxuan et al. proposed an automatic processing method for traffic police information based on multi-task transfer learning for traffic police data, which realized the automatic processing of key information, types and semantics in traffic police. Qiu Kaikai et al. proposed a text classification method based on ERNIE-SA-DPCNN for the case text of new Internet-related criminal cases, and proved through comparative experiments that its performance in the classification of case text of new Internet-related criminal cases is better than other models. Ayinga et al. proposed a text classification method based on the GRU-Glove algorithm for cybercrime cases, and used the public data set of Chinese word segmentation experiments for experiments, showing that this method effectively improved the accuracy and efficiency of complex text classification of new vocabulary. Wang Yue et al. proposed a named entity recognition method for police text based on BERT to address the problem of low accuracy in identifying key entity information in police reports, and used public data sets for training and testing, effectively improving the recognition accuracy.

[0006] However, the above studies have focused too much on the effectiveness of the model itself, and have not done a good job of adapting the characteristics of the police text to the model. The police text is written by the police according to the description and question-and-answer information of the caller in accordance with certain specifications, and the expression is objective and concise. In addition, due to the difference in the frequency of the case itself, compared with the text data obtained from the Internet such as Weibo and news, it has the characteristics of high information density, incoherence, and data imbalance, which leads to gradient explosion and difficulties in extracting key features. Summary of the invention

[0007] Based on the above needs, new Internet-related criminal cases will be further subdivided into nine common categories, including credit card fraud and fraud by impersonating public officials, according to case classification rules, in order to provide more targeted guidance and support for case investigation and crackdown.

[0008] The present invention provides a warning text classification method based on the CBERT-MDPCNN model, which comprises the following steps:

[0009] Step 1: Obtain police texts within a historical period as model training data;

[0010] Step 2: Pre-process and screen the description information of the obtained alarm text;

[0011] Step 3: Classify the preprocessed police texts by category, and mark the corresponding category labels for each classified police text; the classification includes three rough categories and eleven detailed categories;

[0012] Step 4: For the classified police text data, use the GPT4 language model to perform data enhancement to increase the amount of police text data in the minority sample category;

[0013] Step 5: Use the BERT pre-trained model including the word embedding layer and the Transformer layer to extract the semantic information of the police text and obtain the text feature matrix, where the word embedding layer is used to generate sentence embedding word vectors, and the Transformer layer is used to generate word vector encodings; add the intermediate word vector encoding output by the Transformer layer to the word vector encoding output by the BERT pre-trained model, and use the police text data to perform MLM pre-training on it to generate a CBERT pre-trained model;

[0014] Step 6: Convert the text feature matrix output by the CBERT pre-training model into a text feature vector of multiple convolutional kernels that can be processed by MDPCNN; use the word embedding layer as the backbone network, and construct an MDPCNN multi-head classifier model through a branch network consisting of an eleven-classification pyramid pooling structure and a three-classification pyramid pooling structure. The CBERT pre-training model and the MDPCNN multi-head classifier model constitute a CBERT-MDPCNN police text classifier model;

[0015] Step 7: Divide the enhanced police text data in step 4 into a training data set and a validation data set, initialize the model parameters, and iteratively train the CBERT-MDPCNN police text classifier model;

[0016] Step 8: Obtain the police text data that needs to be classified, perform preprocessing in step 2, input it into the CBERT-MDPCNN police text classifier model trained in step 7, and output the segmented classification results.

[0017] The beneficial effects of the present invention are mainly manifested in:

[0018] The method of the present invention includes a data enhancement step based on the GPT4 model, constructs corresponding system prompt words, and enables a large language model to automatically generate text data of corresponding categories, greatly increasing the amount of data in the small sample category, which is conducive to the model to fully learn the characteristics of various samples.

[0019] The present invention proposes an MLM method to fine-tune the Transformer layer of the BERT pre-training model, thereby improving the performance of the pre-training model in natural language processing tasks in the field of police texts. A BERT aggregation operator is constructed to improve the multi-layer expression ability of the pre-training model.

[0020] Based on the unique binary structure of DPCNN classifier, the present invention uses the text region embedding layer as the backbone model and the pyramid structure as the branch network to construct the MDPCNN multi-head classifier model, thereby ensuring the balance of training data while making more full use of the original data. Multi-task focus operators are introduced into the MDPCNN multi-task classifier model, so that the model can better recognize various types of text in eleven classification tasks.

[0021] In order to alleviate the gradient explosion problem caused by the high information density and incoherence of the police text, the present invention selects a smoother Mish function as the activation function to help the model better capture the key features of the text and improve the model classification effect.

[0022] During the training process of the BERT pre-training model, in order to better adapt to the training requirements of the Transformer architecture, the present invention adopts the BERTAdam dynamic learning rate method. On the basis of AdamW, the warmup learning rate decay strategy is introduced to remove the weight decay of all bias items of the model, thereby achieving faster speed and better effect.

[0023] In summary, the present invention can quickly classify police texts, improve the accuracy of the model in classifying police texts, greatly reduce the requirements for the sample size of police text data for model training, and has a certain model generalization ability, which is suitable for police texts such as Internet-related police incidents and non-Internet-related police incidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is the overall architecture of the CBERT-MDPCNN model;

[0025] Figure 2 It is a data enhancement implementation process based on a large language model;

[0026] Figure 3 This is the Loss value change diagram of MLM fine-tuning training;

[0027] Figure 4 This is a comparison chart of the accuracy distribution of the comparative experiment;

[0028] Figure 5 This is a comparison chart of the accuracy of the experimental training set. DETAILED DESCRIPTION

[0029] The present invention will be described in detail below with reference to the accompanying drawings.

[0030] This embodiment discloses a warning text classification method based on the CBERT-MDPCNN model. Figure 1 As shown, it mainly includes the following steps:

[0031] Step 1: Obtain the police information text from the 110 police call receiving platform of a certain public security bureau within a period of time.

[0032] Step 2: Screen the descriptive information of the obtained alarm text, check whether the alarm category is consistent with the alarm description, delete duplicate alarms and invalid alarms, retain valid alarms, and then use regular matching and wildcard matching to desensitize and mask unnecessary information.

[0033] Step 3: After screening and analyzing the police text, they are divided into three categories: non-Internet-related cases, Internet-related cases, and other cases; among them, non-Internet-related cases refer to criminal cases that do not involve the Internet or network technology, Internet-related cases refer to criminal cases involving the Internet or network technology, and other cases refer to Internet-related cases other than those that have been classified; further, for Internet-related cases, by analogy with traditional property security, a detailed classification method is proposed, and the Internet-related police texts are further divided into nine categories: credit card fraud, ordinary identity fraud, fraud by impersonating public officials, emotional fraud, investment and financial fraud, false transaction fraud, fraud involving illegal activities, theft through information means, extortion, etc., combined as data labels.

[0034] Step 4: For the classified police text data, such as Figure 2 As shown in the figure, the flowchart shows the complete process of text enhancement using GPT-4. First, the system calculates the number of text categories to initialize the enhancement configuration, and determines the amount of data to be added based on the set parameter n3. Next, the system selects a specific text category H to be enhanced, and samples several data entries in this category as examples, including the name and description information of the category. This information will be input into the "prompt word generator" to generate specific prompt words, which are then passed to GPT-4 to generate enhanced text data. These generated data are expanded or modified based on the initial text to obtain enhanced text. Users can affect the number and length of generated results by entering two parameters: the number of text items n1 and the shortest length of the text n2, to meet personalized needs. System prompts and user prompts are responsible for data initialization, selection, and parameter setting, respectively.

[0035] Step 5: Use the BERT pre-trained model including the word embedding layer and the Transformer layer to extract the semantic information of the police text and obtain the text feature matrix, where the word embedding layer is used to generate sentence embedding word vectors, and the Transformer layer is used to generate word vector encodings; add the intermediate word vector encoding output by the Transformer layer to the word vector encoding output by the BERT pre-trained model, and use the police text data to perform MLM pre-training on it to generate a CBERT pre-trained model;

[0036] The CBERT pre-training model construction method is as follows:

[0037] 1) Obtain sentence embedding word vector.

[0038] The input data of the word embedding layer is the police text data after data enhancement, which consists of police text sentences and corresponding classification labels. In the word embedding layer, each police text sentence is calculated by adding the word embedding vector, the sentence embedding vector and the position embedding vector to finally obtain the embedded word vector of each police text sentence.

[0039] 2) Use the Transformer layer to generate word vector encoding.

[0040] The Transformer layer uses a multi-head attention layer and multiple attention mechanisms to focus on different positions of the input sequence, associate the vectors of each position in the input sequence, and generate a new set of vector representations. Then, the feedforward neural network sublayer is used to perform nonlinear transformation on the output of the multi-head self-attention sublayer. Residual links and layer normalization are used to connect layers to speed up the convergence of the model. At the same time, the MLM method is used to fine-tune it, randomly replace some words in the model with [MASK] tags, and let the model predict the replaced words. Strengthen the consideration of the relevance of context words when the model corresponds to the field text, and improve the performance of the trained model in natural language processing tasks in the field of police text.

[0041] 3) The BERT aggregation operator is introduced to improve its multi-level expression ability. After adding the aggregation operator, BERT is recorded as CBERT. The aggregation operator can be expressed as:

[0042]

[0043] Where O is the final output of the BERT pre-training model after adding the operator, that is, the text feature matrix, T j is the word vector encoding output by the j-th Transformer layer in the BERT pre-training model, and n is the total number of Transformer layers in the BERT pre-training model.

[0044] Step 6: Use the text region embedding layer as the backbone network, convert the text feature matrix output by CBERT into a text feature vector of multiple convolution kernels that can be processed by MDPCNN, and after two layers of equal-length convolution, input the pyramid structure to form a branch network to construct the MDPCNN multi-head classifier model. The CBERT pre-trained model and the MDPCNN multi-head classifier model constitute the CBERT-MDPCNN police text classifier model;

[0045] In a specific embodiment of the present invention, step six is ​​as follows:

[0046] 1) A non-monotonic and smooth Mish function is used in the backbone network of MDPCNN to improve the expressiveness of the model. The formula is as follows:

[0047] f 2 (x) = xtanh(ln(1+e x )) (2)

[0048] Where x is the input value of the model, f 2 (x) is the output value after passing through the activation function.

[0049] 2) The branch network composed of two pyramid pooling structures performs maximum pooling on the text feature vector output by the backbone network to halve the length of the text feature vector, and constructs a loop until the text feature vector of each convolution kernel is compressed to 1×1. In this process, the text feature vectors before and after the convolution block are added through residual connection, thereby alleviating the problem of gradient disappearance during model training. Subsequently, a multi-task focus operator is introduced to integrate the data of the two branch networks to improve the model classification performance. The specific formula of the multi-task focus operator is as follows:

[0050]

[0051] Where R is the output tensor of the MDPCNN multi-head classifier model, A is the output tensor of the eleven-classification pyramid structure, and B is the output tensor of the three-classification pyramid structure. is a variable parameter optimized by gradient descent.

[0052] 3) The output tensor of the MDPCNN multi-head classifier model is transformed through the fully connected layer to output the scores of the corresponding categories. The Softmax function is then used to convert these scores into probability functions to obtain the eleven classification results. The Softmax function is as follows:

[0053]

[0054] where y i is the output of each node, and m is the number of output nodes, that is, the number of classification categories.

[0055] Step 7: Divide the enhanced text data into training data set and validation data set according to a certain ratio, initialize model parameters such as learning rate, and iteratively train the current CBERT-MDPCNN-based police text classification model.

[0056] In order to achieve faster speed and better results, and better adapt to the training requirements of Transformer architectures such as BERT, BERTAdam is used in the training process of the CBERT-MDPCNN police text classifier model. On the basis of AdamW, the warmup learning rate decay strategy is introduced, and the weight decay of all bias items of the model is removed. The formula is as follows:

[0057]

[0058] Where v is the momentum vector, s is the adaptive learning rate parameter, is the gradient of the weight, β 1 and β 2 is the momentum coefficient, and its value is close to 1; Equations (7) and (8) are the deviation corrections of the first-order moment estimation of the gradient; η 0 is the initial value of the learning rate, η t is the learning rate after executing the warmup strategy, decay is the decay factor; is the updated weight, ε is a very small value, and λ is the regularization coefficient.

[0059] Step 8: Obtain the police text data that needs to be classified, preprocess it, input it into the trained CBERT-MDPCNN-based police text classification model, and output the classification results.

[0060] In order to evaluate the classification method of the present invention, the present invention conducts comparative analysis on the classification results output by the model, and adopts comparative experiment and parameter experiment methods for evaluation.

[0061] 1) In order to make the performance evaluation of the model comparable, the present invention builds a relevant experimental environment and deploys it on the Ubuntu-20.04.1 system. The GPU uses NVIDIAA6000 45G, and the code is written in Python 3.9.18 version. The detailed information of the experimental environment is shown in Table 1.

[0062] Table 1 Experimental environment

[0063]

[0064] 2) This paper uses top1, top3 accuracy (Acc), precision (P), recall (R) and F 1 The value is used to evaluate the model. The specific formula is:

[0065]

[0066] Among them, TP is the number of entities recognized correctly, TN is the number of entities recognized incorrectly, FP is the number of other tags recognized as this tag, and FN is the number of this tag recognized as other tags.

[0067] For multi-category text classification tasks, we also need to examine the pros and cons of classifiers under different categories. We use the evaluation function classification-report in the sklearn module to calculate the macro weighted average, which is recorded as the macro precision (P w ), macro recall (R w ) and macro F 1 Value (F w), and its calculation formula is shown in equations (15), (16), and (17):

[0068]

[0069] where ω i , P i and R i , i=1,2,…,n are the weight, precision and recall of each category respectively.

[0070] Since model training has a certain degree of randomness, in order to eliminate the influence of experimental result variance and improve experimental stability, the evaluation index of the present invention not only selects the optimal values ​​of all related indicators of the same type of parameter adjustment experiments, but also calculates the average accuracy of the top 10% of the accuracy scores of the same type of parameter adjustment experiments, with the specific symbol Acc 10％ .

[0071] 3) In order to make the BERT pre-trained model more suitable for the police text dataset, the present invention fine-tunes the BERT pre-trained model through the MLM method, and the BERT pre-trained model used in subsequent comparisons is the fine-tuned model. The relevant experimental parameter settings are shown in Table 2.

[0072] Table 2 MLM fine-tuning experimental parameters

[0073]

[0074] Where max_seq_length is the maximum input length of each sentence, train_epoches is the number of training rounds, batch_size is the number of input sentences per iteration, and mlm_probability is the proportion of [MASK] words set in each sentence. The loss function used in training is the negative log-likelihood estimate, as shown in formula (18).

[0075]

[0076] Among them, L MLM is the Loss value, N is the number of masked words, w i is the i-th masked word, C i is the context of the i-th masked word, P(w i |C i ) is the probability distribution predicted by the model.

[0077] In order to reflect the effect of MLM fine-tuning pre-training, the present invention draws a graph of the change of the MLM fine-tuning training Loss value, such as Figure 3 As shown. Figure 3It can be seen that after MLM pre-training fine-tuning, the Loss value of the BERT pre-training model for the police text dataset has been reduced from 2.2 to about 1.2, indicating that in the field of police text, the BERT pre-training model's word prediction ability has been improved. Through MLM fine-tuning training, the BERT pre-training model can learn the knowledge of police text based on the original NLP knowledge, so that it can better adapt to the downstream police text classification tasks and improve the classification accuracy.

[0078] 4) To verify the classification performance of the CBERT-MDPCNN model, it is compared with the following benchmark models under the same experimental environment. BERT-FC: Use BERT to extract global semantic information and add a fully connected layer (FC) for classification. ERNIE-FC: Use ERNIE to extract global semantic information and add a fully connected layer (FC) for classification. BERT-RCNN: Use BERT to extract global semantic information and use the residual network (RCNN) to process local context information and classify. BERT-RNN: Use BERT to extract global semantic information and RNN to process local context information and classify. BERT-LSTM: Use BERT to extract global semantic information and add an LSTM layer to process local context information and classify.

[0079] 5) The parameter experiment of the present invention mainly tests the performance of different models under different hyperparameters by setting different learn_rate and batch_size. The present invention selects the learning rate (learn_rate) and the number of input model samples (batch_size) as variable hyperparameters. Among them, learn_rate selects seven types: 0.00003, 0.0009, 0.0007, 0.0005, 0.0003, 0.0001, and 0.003; batch_size selects five types: 16, 32, 64, 128, and 256, and a total of 35 parameter experiments are performed on each model in pairs.

[0080] In order to make the computing resources allocation more reasonable when training each model, the number of epochs of each group of experimental training in the experiment of the present invention follows formula (19).

[0081]

[0082] After preliminary experiments, Hidden Layer Size is uniformly set to 768, and Pad_size is uniformly set to 21. For BERT-RNN, BERT-RCNN: Dropout = 0.1, Conv Kernel Size: 2x3x4, Activation: ReLU; for CBERT-MDPCNN: Conv Kernel Num = 250, Activation: Mish / ReLU, Adam Warmup = 0.5.

[0083] Through experiments, we can get a comparison chart of the accuracy distribution of the comparative experiment, such as Figure 4 As shown, the horizontal axis in the figure represents the parameter Acc top1 The experiment sorts the numbers in ascending order according to the value, and the vertical axis represents Acc top1 Therefore, this figure is monotonically increasing for each model, which can reflect the Acc of each model parameter experiment to a certain extent. top1 Since ERNIE-FC performs poorly in this figure and affects the display effect of other models, it is not presented.

[0084] from Figure 4 It can be seen that the overall accuracy of BERT-RCNN and BERT-RNN models is higher than that of BERT-LSTM and BERT-FC, while the accuracy distribution curve of CBERT-MDPCNN exceeds BERT-RCNN at the ascending order number 3, and then remains at a high position at the same order number. This shows that the CBERT-MDPCNN model not only has a high accuracy in the police text classification task but also has good robustness to hyperparameters.

[0085] After the above experimental tests, according to the Acc top1 The optimal values ​​of the hyperparameters are shown in Table 3.

[0086] Table 3 Optimal parameter settings for comparative experiments

[0087]

[0088] 6) Based on the optimal parameters of the comparative experiment obtained in Table 3, another comparative experiment was conducted to obtain the optimal comparative experiment indicators, as shown in Table 4.

[0089] Table 4 Comparison of the best comparative experimental indicators

[0090]

[0091] As can be seen from Table 4, BERT-FC is significantly better than ERNIE-FC. Therefore, the pre-training models used in other control group models are all BERT pre-training models. In addition, except for the accuracy rate that is lower than BERT-RCNN, CBERT-MDPCNN is better than other BERT-based models in Acc. top1 、Acc 10％ , P w , R w 、F w There have been improvements in all indicators, which verifies the effectiveness of the CBERT-MDPCNN model in the eleven-category police text classification task.

[0092] Figure 5 It is represented as the training accuracy of different models on the validation set for the eleven classification tasks. From the enlarged part of the figure, it can be seen that although the CBERT-MDPCNN model has some fluctuations, its overall validation set accuracy is higher than that of the BERT-RNN and BERT-RCNN models below during training, which proves that the CBERT-MDPCNN model has a better effect on the validation set during training.

[0093] Table 5 CBERT-MDPCNN police text test set indicators for each category

[0094]

[0095] In the table, bold indicates an index greater than or equal to 90, and italic and underlined indicates an index less than or equal to 80.

[0096] Table 5 shows the three major indicators of CBERT-MDPCNN in the police text test set. From the analysis in the above table, we can find that the categories with better model performance are: credit card fraud, fraud by impersonating public officials, emotional fraud, fraud involving illegal activities, theft through information means, and extortion. The text features of these categories may be more obvious. Due to the limitation of the amount of training data, for investment and financial fraud, false transaction fraud, and other categories, the amount of training data can be further increased to improve the model capabilities.

[0097] In addition, the model has high precision for non-Internet-related cases but low recall, which shows that the model is very accurate in predicting non-Internet-related cases, but it will miss some positive examples. The model is more conservative in predicting this category, which may be caused by the imbalance in the amount of data between non-Internet-related cases and Internet-related cases in the eleven-category dataset.

[0098] In general, on the police text test set, CBERT-MDPCNN has more categories with better performance than those with worse performance, which shows that CBERT-MDPCNN has good prediction capabilities for various types of text data.

[0099] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A police text classification method based on the CBERT-MDPCNN model, characterized in that: The following steps are involved: Step 1: Obtain police texts within a historical period as model training data; Step 2: Pre-process and screen the description information of the obtained alarm text; Step 3: Classify the preprocessed police texts by category, and mark the corresponding category labels for each classified police text; the classification includes three rough categories and eleven detailed categories; Step 4: For the classified police text data, use the GPT4 language model to perform data enhancement to increase the amount of police text data in the minority sample category; Step 5: Use the BERT pre-trained model including the word embedding layer and the Transformer layer to extract the semantic information of the police text and obtain the text feature matrix. The word embedding layer is used to generate sentence embedding word vectors, and the Transformer layer is used to generate word vector encodings. The intermediate word vector encoding output by the Transformer layer is added to the word vector encoding output by the BERT pre-trained model, and MLM pre-training is performed on it using police text data to generate a CBERT pre-trained model. Step 6: Convert the text feature matrix output by the CBERT pre-training model into a text feature vector of multiple convolutional kernels that can be processed by MDPCNN; use the word embedding layer as the backbone network, and construct an MDPCNN multi-head classifier model through a branch network consisting of an eleven-classification pyramid pooling structure and a three-classification pyramid pooling structure. The CBERT pre-training model and the MDPCNN multi-head classifier model constitute a CBERT-MDPCNN police text classifier model; Step 7: Divide the enhanced police text data in step 4 into a training data set and a validation data set, initialize the model parameters, and iteratively train the CBERT-MDPCNN police text classifier model; Step 8: Obtain the police text data that needs to be classified, perform preprocessing in step 2, input it into the CBERT-MDPCNN police text classifier model trained in step 7, and output the segmented classification results.

2. The method for classifying police information text according to claim 1, characterized in that: The step 2 includes: checking whether the alarm category is consistent with the alarm description one by one, deleting duplicate alarms and invalid alarms, retaining valid alarm texts, and then using regular matching and wildcard matching to desensitize and mask unnecessary information.

3. The method for classifying police information text according to claim 1, characterized in that: In the step three, the police report text is first divided into three categories: non-Internet-related cases, Internet-related cases, and other cases, and marked with three classification labels. Then, the Internet-related cases are subdivided into nine subcategories: credit card fraud, ordinary identity fraud, fraud by impersonating public officials, emotional fraud, investment and financial fraud, false transaction fraud, fraud involving illegal activities, theft through information means, and extortion, and marked with eleven classification labels.

4. The method for classifying police information text according to claim 1, characterized in that: The step four includes: constructing corresponding prompt words according to different police text categories, handing them over to the prompt word generator for processing, allowing the GPT4 language model to automatically generate police text data of the corresponding category, and then through the identification of categories with fewer samples, cyclically generating and enhancing related police text data, and finally obtaining enhanced police text data, thereby increasing the amount of police text data in categories with fewer samples.

5. The method for classifying police information text according to claim 1, characterized in that: The step 5 is implemented by the following sub-steps: (5.1) Obtain sentence embedding word vector; The input data of the word embedding layer is the police text data after data enhancement, which consists of police text sentences and corresponding classification labels. In the word embedding layer, each police text sentence is calculated by adding the word embedding vector, the sentence embedding vector and the position embedding vector to finally obtain the embedded word vector of each police text sentence. (5.2) Use the Transformer layer to generate word vector encoding; The Transformer layer uses a multi-head attention sublayer and multiple attention mechanisms to focus on the position of the input sentence embedding word vector, associate the vectors of each position in the input sentence embedding word vector, and generate a new set of vector representations; then the feedforward neural network sublayer is used to perform nonlinear transformation on the output of the multi-head self-attention sublayer to obtain the intermediate word vector encoding; the layers are connected by residual links and layer normalization to speed up the convergence of the Transformer layer; (5.3) The BERT aggregation operator is introduced to improve its multi-level expression ability. After adding the aggregation operator, BERT is denoted as CBERT; the aggregation operator is expressed as: Where O is the final output of the BERT pre-training model after adding the operator, that is, the text feature matrix, T j is the word vector encoding output by the j-th Transformer layer in the BERT pre-training model, and n is the total number of Transformer layers in the BERT pre-training model.

6. The method for classifying police information text according to claim 5, characterized in that: In step (5.2), the MLM method is used to fine-tune the Transformer layer, randomly replace the words in some sentences input to the Transformer layer with [MASK] tags, and let the Transformer layer predict the replaced words; strengthen the consideration of the relevance of context words when the Transformer layer corresponds to the domain text.

7. The method for classifying police information text according to claim 1, characterized in that: The step six is ​​implemented by the following sub-steps: (7.1) A non-monotonic and smooth Mish function is used in the backbone network of MDPCNN to improve the expressiveness of the model. The formula is as follows: f2(x)=xtanh(ln(1+e x )) Where x is the input value of the model, and f2(x) is the output value after passing through the activation function; (7.2) The branch network composed of two pyramid pooling structures performs maximum pooling on the text feature vector output by the backbone network to halve the length of the text feature vector, and constructs a loop until the text feature vector of each convolution kernel is compressed to 1×1. In this process, the text feature vectors before and after the convolution block are added together through residual connections, thereby alleviating the problem of gradient disappearance during model training. Subsequently, a multi-task focus operator is introduced to integrate the data of the two branch networks to improve the model classification performance. The specific formula of the multi-task focus operator is as follows: Where R is the output tensor of the MDPCNN multi-head classifier model, A is the output tensor of the eleven-classification pyramid structure, and B is the output tensor of the three-classification pyramid structure. is a variable parameter optimized by gradient descent. (7.3) The output tensor of the MDPCNN multi-head classifier model is transformed through the fully connected layer to output the scores of the corresponding categories. The Softmax function is then used to transform these scores into probability functions to obtain the eleven classification results. The Softmax function is as follows: where y i is the output of each node, and m is the number of output nodes, that is, the number of classification categories.

8. The method for classifying police information text according to claim 1, characterized in that: In the step 7, the BERTAdam optimizer is used in the training process of the CBERT-MDPCNN police text classifier model. The BERTAdam optimizer introduces a warmup learning rate decay strategy based on AdamW and removes the weight decay of all bias items of the model.

Images