Text classification method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a text classification method and device, computer equipment and a storage medium, belongs to the technical field of text classification, and is suitable for the field of financial science and technology. The method comprises the steps of obtaining a training text; inputting the training text into a large-scale language model comprising a feature extraction network and a multi-head attention network; performing feature extraction on the training text through a feature extraction network to obtain preliminary text features; performing attention processing on the preliminary text features through a multi-head attention network to obtain selected text features; training a preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model; obtaining a target text, and performing feature extraction on the target text through the large-scale language model to obtain target text features; and performing classification processing on the target text through the target decision tree model and the target text features to obtain text category information. According to the embodiment of the invention, the text classification accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of text classification technology, applicable to the field of financial technology, and in particular to a text classification method and apparatus, computer equipment and storage medium. Background Art

[0002] In the financial field, a large amount of text-based financial data will be generated. In order to quickly complete the analysis of financial data, it is necessary to use large-scale language models to extract features from financial texts, and then classify the texts based on the text features. In related technologies, financial text classification achieved only through large-scale language models is affected by the special content of financial texts, thereby affecting the accuracy of text classification. For example, if the financial text involves mathematical concepts, mathematical symbols, etc., it is difficult for large-scale language models to understand the mathematical concepts and mathematical symbols in the financial text, thereby affecting the classification accuracy of the financial text. Therefore, how to improve the accuracy of text classification has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a text classification method and apparatus, a computer device and a storage medium, aiming to improve the accuracy of text classification.

[0004] To achieve the above purpose, a first aspect of an embodiment of the present application proposes a text classification method, the method comprising:

[0005] Get training text;

[0006] Inputting the training text into a preset large-scale language model; wherein the large-scale language model includes: a feature extraction network and a multi-head attention network;

[0007] Extracting features from the training text through the feature extraction network to obtain preliminary text features;

[0008] Performing attention processing on the preliminary text features through the multi-head attention network to obtain selected text features;

[0009] Training a preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model;

[0010] Acquire a target text, and extract features of the target text using the large-scale language model to obtain features of the target text;

[0011] The target text is classified by using the target decision tree model and the target text features to obtain text category information.

[0012] In some embodiments, the feature extraction network includes: a preprocessing subnetwork and a feature extraction subnetwork; extracting features from the training text through the feature extraction network to obtain preliminary text features includes:

[0013] Performing word segmentation processing on the training text through the preprocessing subnetwork to obtain a word unit sequence; wherein the word unit sequence includes at least one word unit;

[0014] Performing feature conversion on each of the word units through the feature extraction subnetwork and preset candidate word features to obtain selected word features;

[0015] The selected word features are concatenated to obtain the preliminary text features.

[0016] In some embodiments, the feature extraction subnetwork includes: an index conversion layer and a feature conversion layer, and the feature conversion of each word unit is performed through the feature extraction subnetwork and preset candidate word features to obtain the selected word features, including:

[0017] Filtering a target word index from preset candidate word indexes through the index conversion layer and the word unit;

[0018] The feature conversion is performed on the word unit through the feature conversion layer and the target word index to obtain the selected word feature.

[0019] In some embodiments, the multi-head attention network includes: a linear transformation subnetwork, at least one self-attention layer, and a feed-forward neural subnetwork;

[0020] The step of performing attention processing on the preliminary text features through the multi-head attention network to obtain selected text features includes:

[0021] The preliminary text features are linearly transformed by the linear transformation subnetwork to obtain a text vector set; wherein the text vector set includes: a query vector, a key vector and a value vector;

[0022] Performing self-attention processing on the query vector, the key vector and the value vector through each of the self-attention layers to obtain a candidate text vector;

[0023] Concatenate the candidate text vectors to obtain a target text vector;

[0024] The target text vector is feature mapped by the feedforward neural sub-network to obtain the selected text feature.

[0025] In some embodiments, performing self-attention processing on the query vector, the key vector, and the value vector through each of the self-attention layers to obtain a candidate text vector includes:

[0026] Performing similarity calculation on the query vector and the key vector through the self-attention layer to obtain preliminary similarity data;

[0027] Scaling the preliminary similarity data by the self-attention layer and a preset scaling factor to obtain candidate similarity data;

[0028] Normalizing the self-attention layer and the candidate similarity data to obtain an attention weight;

[0029] The query vector, the key vector and the value vector are concatenated according to the attention weight to obtain the candidate text vector.

[0030] In some embodiments, the step of training a preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model includes:

[0031] Calculating information gain for the selected text feature to obtain feature information gain; wherein the feature information gain is used to measure the amount of information reduction of the selected text feature in text classification;

[0032] Filtering selected text features from the selected text features according to the feature information gain;

[0033] Dividing the training text into sets according to the selected text features to obtain a reference text set;

[0034] The original decision tree model is trained according to the reference text set and the selected text features to obtain the target decision tree model.

[0035] In some embodiments, the target text is classified by the target decision tree model and the target text features to obtain text category information, including:

[0036] Searching a text set by using the target decision tree model and the target text features to obtain a selected text set and a classification probability value of the selected text set;

[0037] The target text is classified according to the classification probability value and the selected text set to obtain the text category information.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a text classification device, the device comprising:

[0039] A text acquisition module, used to acquire training text;

[0040] A text input module, used to input the training text into a preset large-scale language model; wherein the large-scale language model includes: a feature extraction network and a multi-head attention network;

[0041] A preliminary feature extraction module, used to extract features from the training text through the feature extraction network to obtain preliminary text features;

[0042] An attention processing module, used for performing attention processing on the preliminary text features through the multi-head attention network to obtain selected text features;

[0043] A model training module, used for training a preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model;

[0044] A target feature extraction module is used to obtain a target text and extract features from the target text using the large-scale language model to obtain target text features;

[0045] The text classification module is used to classify the target text through the target decision tree model and the target text features to obtain text category information.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0048] The text classification method and device, computer equipment and storage medium proposed in the present application extract the selected text features of the training text through a large-scale language model, and then train the original decision tree model based on the selected text features and the training text to construct a target decision tree model that can achieve accurate text classification. When classifying text, the target text features of the target text are extracted through a large-scale language model, and then the target decision tree model and the target text features complete the classification of the target text. Therefore, by combining a large-scale language model and a decision tree model to complete text classification, text classification is performed by combining the natural language understanding ability of the large-scale language model and the high interpretability and high analytical ability of the decision tree model, making the text classification operation more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of a text classification method provided in an embodiment of the present application;

[0050] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0051] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0052] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0053] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0054] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0055] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0056] Figure 8 is a structural diagram of a text classification device provided in an embodiment of the present application;

[0057] Fig. 9 is a system structure diagram of the text classification method provided in the embodiment of the present application;

[0058] Fig.10 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0065] Large Language Model (LLM): A natural language processing model trained with massive text data using deep learning technology. Large language models can understand and generate human language and complete various natural language processing tasks, such as text generation, translation, question answering, summarization, etc.

[0066] Decision Tree: A model that uses a tree-like data structure to display decision rules and classification results. As an inductive learning algorithm, its focus is to transform seemingly disordered and messy known data into a tree model that can predict unknown data through some technical means. Each path from the root node (the attribute that contributes the most to the final classification result) to the leaf node (the final classification result) represents a decision rule.

[0067] Self-Attention Mechanism: It is a mechanism that allows the model to focus on information at different positions in the sequence when processing sequence data. Unlike recurrent neural networks, the self-attention mechanism does not process data sequentially, but is able to consider information at all positions in parallel, thereby better capturing long-distance dependencies.

[0068] Feedforward neural network: It is the simplest neural network. Neurons are arranged in layers, and each neuron is only connected to neurons in the previous layer. It receives the output of the previous layer and outputs it to the next layer. There is no feedback between layers.

[0069] Information Gain: It is an indicator that measures the reduction in the amount of information brought by a certain feature when dividing a data set. It is often used in decision tree algorithms to select split points in order to divide the data set into purer subsets. The concept of information gain is based on information entropy, which is used to measure the uncertainty or confusion of a data set. Information gain measures the amount of reduction in uncertainty after the data set is divided under a certain feature.

[0070] In the financial field, a large amount of text-based financial data is generated. In order to improve the efficiency of financial analysis, it is necessary to classify text-based financial data so that financial data can be selected for different categories during financial analysis. In related technologies, the classification operation of financial data mainly extracts the characteristics of financial data through large-scale language models, and completes the classification operation based on the characteristics of financial data. However, large-scale language models have the problems of insufficient ability to process structured information, large data requirements, high computing resource requirements, difficulty in interpretation, and poor performance on scarce or specific field data, which affects the accuracy of classification. For example, large-scale language models can perform well in natural language processing tasks, but there are some obvious limitations in performing complex mathematical operations and solving mathematical problems, which affects the classification accuracy of financial data with special content.

[0071] Based on this, the embodiments of the present application provide a text classification method and apparatus, a computer device, and a storage medium, which are intended to extract selected text features in a training text through a large-scale language model, and train the original decision tree model with the selected text features and the training text into a target decision tree model with more accurate text classification. Therefore, in the text classification process, the text features in the target text are first extracted through a large-scale language model, and then the target decision tree model and the target text features complete the classification operation of the target text, so that the large-scale language model and the decision tree model are combined to complete the text classification, combining the semantic understanding ability of the large-scale language model and the feature interpretation ability of the decision tree model, so that the text classification is more accurate and can be applied to the classification of texts containing difficult-to-interpret content.

[0072] The text classification method and apparatus, computer equipment and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the text classification method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The text classification method provided in the embodiment of the present application relates to the field of text classification technology and is applicable to the field of financial technology. The text classification method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the text classification method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] Figure 1 is an optional flowchart of the text classification method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0079] Step S101, obtaining training text;

[0080] Step S102, inputting the training text into a preset large-scale language model; wherein the large-scale language model includes: a feature extraction network and a multi-head attention network;

[0081] Step S103, extracting features from the training text through a feature extraction network to obtain preliminary text features;

[0082] Step S104, performing attention processing on the preliminary text features through a multi-head attention network to obtain selected text features;

[0083] Step S105, training the preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model;

[0084] Step S106, obtaining a target text, and extracting features of the target text using a large-scale language model to obtain features of the target text;

[0085] Step S107, classifying the target text using the target decision tree model and the target text features to obtain text category information.

[0086] Steps S101 to S107 shown in the embodiment of the present application, by forming a large-scale language model by a feature extraction network and a multi-head attention network, in the feature extraction process of the training text, the preliminary text features in the training text are first extracted by the feature extraction network, and then the preliminary text features are processed by the multi-head attention network to obtain selected text features that characterize the training text more accurately and more representatively. The original decision tree model is trained by the selected text features and the training text to construct a target decision tree model that can complete the accurate classification of the text based on the text features. Therefore, in the text classification application, the target text features of the target text are first extracted by the large-scale language model, and then the classification of the target text is completed by the target decision tree model and the target text features. Therefore, by combining the semantic understanding ability of the large-scale language model and the analytical ability and strong explanatory ability of the decision tree model, it can not only be applied to the text classification with difficult-to-explain content, but also make the text classification more accurate, and provide more comprehensive and accurate text processing and analysis capabilities.

[0087] In step S101 of some embodiments, text can be collected in real time as training text, or the training text can be obtained in advance by other means. For example, a training database is constructed in advance, a large amount of training text is stored in the training database, and when a large-scale language model and an original decision tree model need to be trained, the training text is extracted from the training database, and this embodiment does not specifically limit the method of obtaining the training text.

[0088] Specifically, if the text classification method is applied to the financial field, the specific scenario can be a bank private placement scenario, and the text-based private placement object data needs to be classified, so the collected training text is the private placement object text, and the private placement object text records all aspects of the private placement object information, including: object basic identity information, object financial data, investment behavior data, risk tolerance assessment data and object credit assessment data, etc. Therefore, the information of each investor in private placement is obtained through the bank private placement platform as training text, so as to facilitate the subsequent classification of private placement objects, so as to realize the classification of qualified investors and unqualified investors, and qualified investors can be further classified into professional investors and ordinary investors.

[0089] It should be noted that after obtaining the training text, the training text needs to be preprocessed to obtain a clean training text that is convenient for large-scale language models to extract features. Specifically, the preprocessing includes at least one of the following: format conversion, removal of duplicate content, correction of typos and correction of incorrect symbols, etc. Among them, if the preprocessing is format conversion, it is necessary to convert the input training text into a format acceptable to the large-scale language model, that is, to obtain the target format acceptable to the large-scale language model, and obtain the original format of the training text, and convert the format of the training text according to the original format and the target format to obtain the training text in the target format, which is convenient for the large-scale language model to extract features.

[0090] In step S102 of some embodiments, the large-scale language model is a natural language processing model with powerful semantic understanding and natural language processing capabilities, which can well understand and process text and extract key text features from the text. The large-scale language model includes: a feature extraction network and a multi-head attention network, the feature extraction network is used to extract key features in the text, and the multi-head attention network is used to calculate the correlation between text features, capture more subtle differences between text features, and output more representative text features.

[0091] In some embodiments, the feature extraction network includes: a preprocessing subnetwork and a feature extraction subnetwork. It should be noted that the preprocessing subnetwork is used to preprocess the training text, and the preprocessing performed by the preprocessing subnetwork is different from the preprocessing performed in advance on the training text, mainly including word segmentation operations to facilitate feature extraction of the training text. It should be noted that the feature extraction network is used to extract key features from the training text.

[0092] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0093] Step S201, performing word segmentation processing on the training text through the preprocessing sub-network to obtain a word unit sequence; wherein the word unit sequence includes at least one word unit;

[0094] Step S202, performing feature conversion on each word unit through the feature extraction sub-network and the preset candidate word features to obtain the selected word features;

[0095] Step S203, concatenating the selected word features to obtain preliminary text features.

[0096] In step S201 of some embodiments, the preprocessing subnetwork performs word segmentation processing on the training text in advance, that is, the training text is split into words or subwords, and then the split words or subwords are combined into a word unit sequence. Therefore, the word unit sequence includes word units as words or subwords. It should be noted that in the training text word segmentation process, the punctuation marks in the training text will be removed first, and then the uppercase and lowercase letters will be unified. Specifically, uppercase letters can be converted to lowercase letters, or lowercase letters can be converted to uppercase letters to achieve a text with a unified format, and then the punctuation marks are removed and the training text with unified uppercase and lowercase letters is segmented to facilitate the text segmentation operation.

[0097] In step S202 of some embodiments, the candidate word feature is a pre-constructed word feature, different word units correspond to different candidate word features, and the candidate word feature is specifically represented as an embedded vector corresponding to the word unit, and the word unit is represented in the form of a vector, which is convenient for computer processing. Therefore, the word unit is feature-converted through the feature extraction subnetwork and the candidate word feature to convert the word unit into a selected word feature represented by the embedded vector. Specifically, the selected word feature that matches the word unit is screened out from the candidate word features through the feature extraction subnetwork to achieve accurate acquisition of the selected word feature.

[0098] In step S203 of some embodiments, because the training text is composed of multiple word units, each word unit corresponds to a selected word feature, it is necessary to splice the selected word features into preliminary text features.

[0099] It should be noted that after the preliminary text features are generated, because the selected word features in the preliminary text features do not have explicit order information, it is necessary to introduce position information in the preliminary text features, and the position information is introduced by adding position encoding. It should be noted that the position encoding is a matrix with the same length as the input sequence, and each row represents the position vector of a position. Specifically, the position encoding method uses sine and cosine functions to generate the position encoding matrix. In addition, the preliminary text features also set a mask vector, and the mask vector is applied to the attention score. It should be noted that the mask vector is a vector with the same length as the preliminary text features, which is used to control the attention weight of each selected word feature in the preliminary text features. Specifically, in the subsequent attention processing process, the mask vector m[j] = -∞ (or other very small values) indicates that the information at position j is masked in the attention calculation of the current position, and the mask vector m[j] = 0 indicates that the information at position j can be accessed. Therefore, by adding position encoding and mask vectors to the preliminary text features, the selected word features that need to be processed for attention in the preliminary text features, as well as the attention weights, can be determined, which facilitates the subsequent attention processing of the preliminary text features.

[0100] In steps S201 to S203 shown in this embodiment, the training text is first split into at least one word unit, and then selected word features matching the word unit are extracted from the candidate word features, and finally the selected word features are spliced ​​into preliminary text features to achieve accurate generation of preliminary text features and facilitate subsequent processing.

[0101] See also Figure 3 In some embodiments, the feature extraction subnetwork includes: an index conversion layer and a feature conversion layer, and step S202 may include but is not limited to steps S301 to S302:

[0102] Step S301, filtering out a target word index from a preset candidate word index through an index conversion layer and a word unit;

[0103] Step S302, performing feature conversion on the word unit through the feature conversion layer and the target word index to obtain the selected word feature.

[0104] In step S301 of some embodiments, before extracting the preliminary text features in the training text, a vocabulary is first constructed, by indexing the words or characters that appear in all the training texts, and then constructing a mapping relationship between each word or character and the index number to form a vocabulary, so the vocabulary records the mapping relationship between each word or character and the candidate word index. The candidate word index that matches each word unit is found in the vocabulary through the index conversion layer as the target word index, so it is easier and faster to complete the word feature conversion by means of the target word index.

[0105] In step S302 of some embodiments, as disclosed above, the candidate word index corresponding to each word or character is recorded in the vocabulary, and the candidate word index is independent and unique, and the candidate word feature corresponding to each candidate word index is also set in the vocabulary. Therefore, the candidate word feature corresponding to each target word index is found in the vocabulary through the feature conversion layer as the selected word feature, so as to realize the rapid search and determination of the selected word feature.

[0106] In steps S301 to S302 shown in the present embodiment, the target word index corresponding to the word unit is first found in the candidate word index, and then the selected word feature corresponding to the word unit is screened out from the candidate word features based on the target word index, so that the selected word feature screening operation is simple and a unique and representative selected word feature is generated.

[0107] In some embodiments, the multi-head attention network includes: a linear transformation subnetwork, at least one self-attention layer and a feedforward neural subnetwork; the linear transformation subnetwork is used to convert preliminary text features into an embedded representation for attention processing. It should be noted that the embedded representation of attention processing is mainly divided into embedded representations of queries, keys and values, so the linear transformation subnetwork includes three different linear transformation layers, so that the preliminary text features are converted into the same embedded representations of queries, keys and values ​​through three different linear transformation layers, which is convenient for subsequent attention processing. Further, the attention layer is used to calculate the importance of the preliminary text features after linear transformation, and each attention layer uses a self-attention mechanism to complete the attention processing, and at least one attention layer is set, and the output of at least one attention layer needs to be spliced ​​and then input into the feedforward neural subnetwork. It should be noted that the feedforward neural self-network is used to capture the complex relationship between the outputs of at least one attention layer, so as to output more representative selected text features and obtain key features that are more in line with the training text.

[0108] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S404:

[0109] Step S401, performing linear transformation on preliminary text features through a linear transformation sub-network to obtain a text vector set; wherein the text vector set includes: a query vector, a key vector and a value vector;

[0110] Step S402, performing self-attention processing on the query vector, the key vector and the value vector through each self-attention layer to obtain a candidate text vector;

[0111] Step S403, concatenating the candidate text vectors to obtain a target text vector;

[0112] Step S404, feature mapping the target text vector through a feedforward neural sub-network to obtain selected text features.

[0113] In step S401 of some embodiments, for subsequent self-attention processing, the preliminary text features need to be converted into embedded representations for self-attention processing. Specifically, three linear transformation layers are set, the first linear transformation layer converts the preliminary text features into query vectors, the second linear transformation layer converts the preliminary text features into key vectors, and the third linear transformation layer converts the preliminary text features into value vectors. Specifically, the selected word features at each position in the preliminary text features are linearly transformed through a preset query weight matrix to obtain a query vector, and the calculation formula of the query vector is: Q = XW Q , where X represents the initial text features of the input, W Qis the query weight matrix. The key vector is obtained by linearly transforming the selected word features at each position in the preliminary text features through the preset key weight matrix, and the calculation formula of the key vector is: K = XW K , where X represents the preliminary text features of the input, W K Represents the key weight matrix. The value vector is obtained by linearly transforming each selected word feature in the preliminary text feature through the preset value weight matrix, and the calculation formula of the value vector is: V = XW V , where X represents the preliminary text features of the input, W V Represents the value weight matrix.

[0114] In steps S402 to S403 of some embodiments, by inputting the query vector, the key vector and the value vector into each self-attention layer, each self-attention layer performs self-attention processing on the query vector, the key vector and the value vector to output a candidate text vector. It should be noted that each self-attention layer performs independent self-attention calculations and generates an attention weight matrix, so the candidate text vectors are spliced, that is, the candidate text vectors output by all self-attention layers are spliced ​​into a target text vector according to the attention weight matrix.

[0115] In step S404 of some embodiments, the target text vector is the feature representation of each position in the training text after being processed by the self-attention mechanism. In order to learn the complex nonlinear representation between the feature representations of each position and output a higher-level feature representation, the target text vector is feature mapped through a feedforward neural sub-network to perform a more complex mapping of the feature representation of each position, enhance the modeling ability of the large-scale language model for local features, and obtain more critical selected text features.

[0116] Specifically, the feedforward neural subnetwork includes: a first linear transformation layer, an activation layer, and a second linear transformation layer. The first linear transformation layer maps the input target text vector to a feature space with a higher dimension to obtain an updated text vector, and then the activation layer activates the updated text vector to obtain an activated text vector. Finally, the second linear transformation layer maps the activated text vector to a feature space with the same dimension as the target text vector to obtain the selected text feature. In the activation layer, the activation layer uses an activation function to activate the updated text vector, and the activation function can be a ReLU function, a Sigmoid function, a Tanh function, etc. This embodiment does not specifically limit the activation function.

[0117] In step S401 to step S404 shown in this embodiment, the preliminary text features are first linearly transformed into a query vector, a key vector and a value vector, and then the query vector, the key vector and the value vector are self-attention processed through each self-attention layer to obtain a candidate text vector. The candidate text vectors are weighted and summed to obtain a target text vector, and finally the target text vector is mapped to the selected text features to obtain a higher-level feature representation, and obtain text features that represent more critical training texts, thereby improving the accuracy of text classification.

[0118] See also Figure 5 In some embodiments, step S402 may include but is not limited to steps S501 to S504:

[0119] Step S501, calculating the similarity between the query vector and the key vector through the self-attention layer to obtain preliminary similarity data;

[0120] Step S502, scaling the preliminary similarity data using a self-attention layer and a preset scaling factor to obtain candidate similarity data;

[0121] Step S503, normalizing the self-attention layer and the candidate similarity data to obtain the attention weight;

[0122] Step S504, concatenating the query vector, the key vector and the value vector according to the attention weight to obtain a candidate text vector.

[0123] It should be noted that attention processing is performed on the query vector, key vector and value vector, that is, the similarity between the query vector and the key vector is first calculated to obtain the attention score, and the importance of each position to the current query is determined according to the attention score, so as to determine the attention weight according to the attention score. Therefore, by calculating the attention weight, the query vector, key vector and value vector are concatenated according to the attention weight to obtain a candidate text vector that more accurately represents the training text features.

[0124] In steps S501 to S502 of some embodiments, as disclosed above, the similarity between the query vector and the key vector is first calculated to obtain preliminary similarity data, and the preliminary similarity data can be obtained using dot product calculation or linear transformation and dot product calculation. In order to scale the preliminary similarity data, a scaling factor is set in advance. The role of the scaling factor is to balance the scale of the query vector and the key vector to prevent the problem of gradient disappearance or gradient explosion in the similarity calculation. Therefore, the preliminary similarity data is scaled according to the scaling factor, that is, the preliminary similarity data is divided by the scaling factor to obtain the candidate similarity data.

[0125] In step S503 of some embodiments, in order to characterize the importance of each position to the current query vector, the candidate similarity data needs to be converted into attention weights. Therefore, the candidate similarity data is normalized to obtain the attention weights. In this embodiment, the candidate similarity data is converted into the attention weights through the softmax function. Other embodiments may use other normalization functions. This embodiment does not specifically limit the normalization function of the candidate similarity data.

[0126] Specifically, the process of converting candidate similarity data into attention weights through the softmax function is:

[0127] Attention(Q,K,V)=softmax((QK^T) / √d_k)*V;

[0128] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and d_k represents the dimension of the query vector and the key vector. The attention score represents the relevance or importance between each query vector and the key vector.

[0129] In step S504 of some embodiments, after the attention weight is obtained by the aforementioned public calculation, the query vector, the key vector and the value vector are concatenated according to the attention weight, and the concatenation is weighted summation to obtain the candidate text vector.

[0130] In the step S501 to step S504 shown in the present embodiment, in the process of generating the candidate text vector, the similarity between the query vector and the key vector is first calculated, and then the similarity is scaled according to the scaling factor to prevent the problem of gradient vanishing or gradient exploding in the similarity calculation process. Then the scaled similarity is normalized to obtain the attention weight to determine the importance of each position to the current query vector. Finally, the query vector, the key vector and the value vector are weighted and summed according to the attention weight to obtain the candidate text vector, so that the candidate text vector generation operation is stable and can accurately represent the key feature representations in the training text.

[0131] In some embodiments, a residual connection and layer normalization operation are set between the self-attention layer and the feedforward neural subnetwork, by introducing a residual connection between the input and output of each self-attention layer, and the residual connection is added by adding the text vector set and the candidate text vector, and then inputting the text vector set together into the self-attention layer. It should be noted that the role of the residual connection is to solve the problem of gradient disappearance and gradient explosion, so that the gradient can flow more easily in the network, and it also helps to retain the details of the underlying features, avoid information loss in the text features, and improve the representation and learning capabilities of large-scale language models. Layer normalization is to normalize the vector of each position of the candidate text vector output by each self-attention layer and then input it into the feedforward neural subnetwork. It should be noted that layer normalization realizes the normalization operation by calculating the mean and variance of the vector at each position.

[0132] In some embodiments, when the large-scale language model outputs the selected text features to complete the feature extraction, in order to improve the feature extraction effect of the large-scale language model, the large-scale language model needs to be trained. Specifically, the loss data between the preset verification text features and the selected text features is calculated to obtain the feature loss data, and the model parameters of the large-scale language model are adjusted according to the feature loss data. Among them, the loss function that can be used in the feature loss data calculation process is used for calculation, and the loss function can be a cross entropy loss function, or a mean square error loss function, a log-likelihood loss function, etc. This embodiment does not make specific restrictions on the loss function.

[0133] It should be noted that in the process of adjusting the model parameters of the large-scale language model, the gradient of the feature loss data to the model parameters of the large-scale language model can be calculated through the back propagation algorithm, and the gradient represents the sensitivity of the feature loss data to the change of the model parameters, so the large-scale language model is adjusted in the direction of the gradient by reducing the feature loss data. Therefore, according to the direction and size of the gradient, through multiple iterative training and parameter adjustment of the large-scale language model, the deviation between the selected text features and the verification text features is reduced, so as to output a large-scale language model with more accurate feature extraction effect.

[0134] It should be noted that after the large-scale language model extracts features from each training text, the model parameters of the large-scale language model are adjusted once, and the selected text features output by the large-scale language model are also input into the original decision tree model, so that the original decision tree model is trained synchronously when the large-scale language model is trained. The training process of the original decision tree model is described in detail below.

[0135] See also Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S604:

[0136] Step S601, calculating information gain of the selected text feature to obtain feature information gain; wherein the feature information gain is used to measure the amount of information reduction of the selected text feature in text classification;

[0137] Step S602, selecting a selected text feature from selected text features according to feature information gain;

[0138] Step S603, dividing the training text into sets according to the selected text features to obtain a reference text set;

[0139] Step S604, training the original decision tree model according to the reference text set and the selected text features to obtain a target decision tree model.

[0140] It should be noted that the training of the original decision tree model is to train the original decision tree model into a target decision tree model related to the training text field, and the target decision tree model can achieve accurate text classification in this field. For example, if the application scenario corresponding to the training text is the classification of private equity object data, then the constructed target decision tree model can be divided into qualified investors and unqualified investors based on the characteristics of private equity objects in various aspects, and the further classification of qualified investors can be professional investors and ordinary investors, so as to achieve accurate classification of private equity objects, and adopt the structural representation of the decision tree to clearly understand the characteristics of private equity objects of different categories.

[0141] In step S601 of some embodiments, when training a target decision tree model related to the training text field, the large-scale language model will continuously input a large number of selected text features into the original decision tree model, so the information gain of each selected text feature is calculated to obtain the feature information gain, and the feature information gain is used to measure the amount of information reduction obtained after using the selected text feature for text classification, so the key role of each selected text feature in text classification is determined by the feature information gain. Among them, the information gain calculation formula of the selected text feature is:

[0142] Information gain = information entropy of the parent node – total information entropy of the child nodes;

[0143] Among them, the calculation formula of information entropy is: information entropy = -Σ(p_i*log2(p_i));

[0144] Among them, p_i represents the proportion of the selected text feature in each category, and Σ represents the sum of all child nodes.

[0145] In step S602 of some embodiments, as disclosed above, the feature information gain is characterized by the importance of the selected text feature in each text category, so the maximum information gain is selected from the feature information gain, and the selected text feature corresponding to the maximum information gain is used as the selected text feature to determine the most representative text feature of the text category.

[0146] In step S603 of some embodiments, the training texts are divided into sets by selecting text features, that is, the training texts with the same text features are divided into one category to obtain multiple reference text sets, each of which corresponds to a text category.

[0147] In step S604 of some embodiments, the original decision tree model is trained according to the reference text set and the selected text feature, that is, the selected text feature is used as the root node in the original decision tree model, the reference text set is used as the leaf node of the original decision tree model, and the setting of the root node and the leaf node in the original decision tree model is repeated. Specifically, the original decision tree model training process mainly repeats the following operations: first determine the selected text feature as the root node, if the training texts in the reference text set all contain the selected text feature, then the reference text set is a leaf node, and the above operations are repeated until the stop condition is met to obtain the target decision tree model. It should be noted that the stop condition is the maximum depth of the decision tree model, the number of training texts, etc. Therefore, by constructing the target decision tree model, the text features existing in the same text category can be determined.

[0148] In steps S601 to S604 shown in this embodiment, the information gain of each selected text feature is calculated by setting, and the selected text feature with the largest information gain is selected as the selected text feature, and the original decision tree model is trained based on the selected text feature and the reference text set divided by the selected text feature, so as to construct a target decision tree model that can accurately perform text classification, so as to improve the accuracy of text classification.

[0149] In step S106 of some embodiments, when the target decision tree model is trained and the large-scale language model is trained, more accurate and critical target text features in the target text are extracted by the trained large-scale language model. It should be noted that the process of extracting target text features by the large-scale language model is the same as the process of extracting selected text features, which will not be repeated here.

[0150] See also Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S702:

[0151] Step S701, searching a text set by using a target decision tree model and target text features to obtain a selected text set and a classification probability value of the selected text set;

[0152] Step S702, classify the target text according to the classification probability value and the selected text set to obtain text category information.

[0153] In step S701 of some embodiments, as disclosed above, the root node of the target decision tree model is the reference text set, and the leaf node is the selected text feature, so the target similarity data is obtained by performing similarity comparison based on the target text feature and the selected text feature, and the selected text features corresponding to the preset number before the target similarity data are used as the reference text features, and then the reference text set corresponding to the reference text features is used as the selected text set. At the same time, the target similarity data on each selected text set is used as the classification probability value.

[0154] In step S702 of some embodiments, the category of the target text is determined by extracting the maximum probability value from the classification probability values, taking the selected text set corresponding to the maximum probability value as the target text set, and then obtaining the text category of the target text set as text category information.

[0155] In steps S701 to S702 shown in this embodiment, a reference text set related to the target text features and a classification probability value of the reference text set are found by using a target decision tree model, so that a target text set is selected from the reference text set according to the classification probability value, and the category corresponding to the target text set is used as text category information, so that the classification operation of the target text category is simple and accurate.

[0156] For example, in the bank private placement scenario, the selected text features corresponding to professional investors in the target decision tree model include: the basic identity information of the object meets the preset identity information, the financial data of the object is greater than the preset financial threshold, the investment behavior data meets the preset investment conditions, the risk tolerance assessment data is greater than the preset risk threshold, and the credit assessment data of the object is greater than the preset credit threshold. Therefore, when classifying private placement objects, it is judged whether the characteristics of the private placement object meet the selected text features and the degree of compliance, so as to judge whether the private placement object belongs to one of professional investors, ordinary investors and unqualified investors, so that the classification operation of private placement objects is accurate and simple.

[0157] Including: Figure 8 As shown, Figure 8 The system structure diagram of the text classification method of the embodiment of the present application is shown. Figure 8It can be seen that the text classification operation involves a large-scale language model and a target decision tree model, and the large-scale language model includes a feature extraction network and a multi-head attention network. Among them, the feature extraction network includes: a preprocessing subnetwork and a feature extraction subnetwork, the feature extraction subnetwork includes: an index conversion layer and a feature conversion layer, and the multi-head attention network includes: a linear transformation subnetwork, at least one self-attention layer and a feedforward neural subnetwork. After the training of the large-scale language model and the target decision tree model is completed, the classification operation of each acquired target text is started. First, the target text is obtained and converted into a format accepted by the large-scale language model, and then the punctuation marks are removed and the uppercase and lowercase letters are unified for the target text after the format conversion. Then, the processed target text is segmented by the preprocessing subnetwork to obtain at least one phrase, and the candidate word index corresponding to each phrase is found in the vocabulary as a reference word index, and then the target word feature is found in the vocabulary based on the reference word index, and the target word feature is spliced ​​into the initial text feature. In order to further improve the criticality and accuracy of text features, the initial text features are input into three different linear transformation layers to output query vectors, key vectors and value vectors, and self-attention processing is performed on the query vectors, key vectors and text vectors to learn the importance of phrase features at each position in the initial text features and the correlation between phrase features, so as to achieve the readjustment of the initial text features to obtain more representative target text features. Finally, the target decision tree model is used to determine the selected text features with high similarity to the target text features as reference text features, and the reference text set corresponding to the reference text features with the highest similarity is used as the target text set, so as to use the text category corresponding to the target text set as the text category information of the target text, so as to achieve the classification operation of the target text. Therefore, in the text classification process, the large-scale language model and the target decision tree model are combined to complete the text classification, that is, the advantages of natural language processing and decision analysis are combined, which can not only better understand and process the text and extract the key information in the text, but also realize the classification operation of the text through the high interpretability and analysis ability of the target decision tree model, making the text classification more accurate.

[0158] See also Fig. 9 The present application also provides a text classification device that can implement the above-mentioned text classification method. The device includes:

[0159] A text acquisition module 901 is used to acquire training text;

[0160] The text input module 902 is used to input the training text into a preset large-scale language model; wherein the large-scale language model includes: a feature extraction network and a multi-head attention network;

[0161] A preliminary feature extraction module 903 is used to extract features from the training text through a feature extraction network to obtain preliminary text features;

[0162] An attention processing module 904 is used to perform attention processing on the preliminary text features through a multi-head attention network to obtain selected text features;

[0163] The model training module 905 is used to train the preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model;

[0164] A target feature extraction module 906 is used to obtain a target text and extract features from the target text using a large-scale language model to obtain target text features;

[0165] The text classification module 907 is used to classify the target text through the target decision tree model and the target text features to obtain text category information.

[0166] The specific implementation of the text classification device is basically the same as the specific implementation of the above-mentioned text classification method, and will not be repeated here.

[0167] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above text classification method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0168] See also Fig.10 , Fig.10 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0169] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0170] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the text classification method of the embodiment of this application;

[0171] Input / output interface 1003, used to implement information input and output;

[0172] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0173] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0174] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0175] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned text classification method is implemented.

[0176] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The text classification method and device, computer equipment and storage medium provided in the embodiments of the present application extract the selected text features of the training text through a large-scale language model, and then train the original decision tree model based on the selected text features and the training text to build a target decision tree model that can achieve text classification. When classifying text, a large-scale language model is applied to extract the target text features of the target text, and then the target decision tree model and the target text features that accurately perform text classification complete the classification operation of the target text. Therefore, by combining a large-scale language model and a decision tree model to complete text classification, the text classification operation is made more accurate by combining the natural language understanding ability of the large-scale language model and the high interpretability and high analytical ability of the decision tree model.

[0178] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0179] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0180] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0182] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0183] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0188] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A text classification method, characterized in that: The method comprises: Get training text; Inputting the training text into a preset large-scale language model; wherein the large-scale language model includes: a feature extraction network and a multi-head attention network; Extracting features from the training text through the feature extraction network to obtain preliminary text features; Performing attention processing on the preliminary text features through the multi-head attention network to obtain selected text features; Training a preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model; Acquire a target text, and extract features of the target text using the large-scale language model to obtain features of the target text; The target text is classified by using the target decision tree model and the target text features to obtain text category information.

2. The method according to claim 1, characterized in that The feature extraction network includes: a preprocessing subnetwork and a feature extraction subnetwork; the feature extraction network is used to extract features from the training text to obtain preliminary text features, including: Performing word segmentation processing on the training text through the preprocessing subnetwork to obtain a word unit sequence; wherein the word unit sequence includes at least one word unit; Performing feature conversion on each of the word units through the feature extraction subnetwork and preset candidate word features to obtain selected word features; The selected word features are concatenated to obtain the preliminary text features.

3. The method according to claim 2, characterized in that The feature extraction sub-network includes: an index conversion layer and a feature conversion layer. The feature extraction sub-network and the preset candidate word features are used to perform feature conversion on each word unit to obtain the selected word features, including: Filtering a target word index from a preset candidate word index through the index conversion layer and the word unit; The feature conversion is performed on the word unit through the feature conversion layer and the target word index to obtain the selected word feature.

4. The method according to claim 1, characterized in that: The multi-head attention network includes: a linear transformation subnetwork, at least one self-attention layer and a feedforward neural subnetwork; The step of performing attention processing on the preliminary text features through the multi-head attention network to obtain selected text features includes: The preliminary text features are linearly transformed by the linear transformation subnetwork to obtain a text vector set; wherein the text vector set includes: a query vector, a key vector and a value vector; Performing self-attention processing on the query vector, the key vector and the value vector through each of the self-attention layers to obtain a candidate text vector; Concatenate the candidate text vectors to obtain a target text vector; The target text vector is feature mapped by the feedforward neural sub-network to obtain the selected text feature.

5. The method according to claim 4, characterized in that The step of performing self-attention processing on the query vector, the key vector, and the value vector through each of the self-attention layers to obtain a candidate text vector includes: Performing similarity calculation on the query vector and the key vector through the self-attention layer to obtain preliminary similarity data; Scaling the preliminary similarity data by the self-attention layer and a preset scaling factor to obtain candidate similarity data; Normalizing the self-attention layer and the candidate similarity data to obtain an attention weight; The query vector, the key vector and the value vector are concatenated according to the attention weight to obtain the candidate text vector.

6. The method according to any one of claims 1 to 5, characterized in that: The step of training a preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model includes: Calculating information gain for the selected text feature to obtain feature information gain; wherein the feature information gain is used to measure the amount of information reduction of the selected text feature in text classification; Filtering selected text features from the selected text features according to the feature information gain; Dividing the training text into sets according to the selected text features to obtain a reference text set; The original decision tree model is trained according to the reference text set and the selected text features to obtain the target decision tree model.

7. The method according to any one of claims 1 to 5, characterized in that: The target text is classified by the target decision tree model and the target text features to obtain text category information, including: Searching a text set by using the target decision tree model and the target text features to obtain a selected text set and a classification probability value of the selected text set; The target text is classified according to the classification probability value and the selected text set to obtain the text category information.

8. A text classification device, characterized in that: The device comprises: A text acquisition module is used to acquire training text; A text input module, used to input the training text into a preset large-scale language model; wherein the large-scale language model includes: a feature extraction network and a multi-head attention network; A preliminary feature extraction module, used to extract features from the training text through the feature extraction network to obtain preliminary text features; An attention processing module, used for performing attention processing on the preliminary text features through the multi-head attention network to obtain selected text features; A model training module, used for training a preset original decision tree model according to the selected text features and the training text to obtain a target decision tree model; A target feature extraction module is used to obtain a target text and extract features from the target text using the large-scale language model to obtain target text features; The text classification module is used to classify the target text through the target decision tree model and the target text features to obtain text category information.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the text classification method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text classification method according to any one of claims 1 to 7 is implemented.

Images