Text classification method and related device based on large model and dilated convolution fusion fine-tuning

Abstract

The present application discloses a text classification method and related devices that integrate a large model and dilated convolution and fine-tune, and relates to the field of text classification, including: extracting a hidden layer matrix of training text content using a large language model pre-trained with large-scale corpus, obtaining a training text vector based on the extracted hidden layer matrix, training a pre-built neural network model with the training text vector and the text classification results corresponding to the manually annotated training text content as training data, obtaining a text classification model, extracting a hidden layer matrix of the text content to be classified using the large language model, obtaining a text vector to be classified based on the extracted hidden layer matrix, inputting the text vector to be classified into the text classification model, and obtaining a text classification result corresponding to the text content to be classified. The present application uses a large language model to extract a hidden layer matrix containing rich semantic and syntactic information, and splices the text vector obtained based on the hidden layer matrix with a text classification model based on dilated convolution, thereby improving the classification accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of text classification, and in particular to a text classification method and related devices that combine a large model with dilated convolution and fine-tuning. Background Art

[0002] Text classification is the process of classifying text data into one or more predefined categories. It can be used in a variety of text classification scenarios, such as spam detection, sentiment analysis, news classification, and topic classification.

[0003] Currently, the following three text classification methods are provided.

[0004] The first method is a data classification prediction method and system for multi-model fusion (CN118069852A), which preprocesses text data and extracts information. The corresponding summary highlights and a list of keywords involved are extracted using a large language model technology. The manually labeled data is fine-tuned using a pre-trained model of a BERT model or a variant of the BERT model. The category of the most similar question is queried based on the prediction results of the traditional classification pre-training module and the semantic similarity of the questions fed back by users. If the threshold of the corresponding category is high, the category of the most similar question is used as the final classification. Otherwise, the classification predicted by the BERT model is compared with the category queried after text semantic similarity comparison. If the two are the same, the same category is used as the final predicted category. Otherwise, the keyword list extracted from the two and the keyword list extracted from the user feedback information are compared for similarity, and the keyword with the highest similarity to the keywords in the merged list in the keyword group extracted by the large model is used as the final classification. In this method, the key points and keywords of the summary extracted by the large language model are actually the result of the large language model decoding the encoded hidden layer matrix. However, the result decoded by the large language model may have certain errors compared with the actual application. Therefore, using the key points and keywords of the summary extracted by the large language model for subsequent classification tasks will affect the accuracy of the classification results.

[0005] The second method is a text classification method based on feature vector sparsity (CN117932072A), which obtains a text data set with known category labels, preprocesses the text data set to obtain a dictionary data set, wherein the dictionary data set includes multiple terms, extracts typical features from the dictionary data set, obtains a class-typical feature vector, matches all terms in the text to be classified with the class-typical feature vector, obtains the corresponding feature vector, calculates the vector sparsity of the feature vector, and calculates the text category of the text to be classified. This method uses statistical analysis methods in the text classification process, lacks feature extraction of text context relationships, and results in low classification accuracy.

[0006] The third method is a weakly supervised text classification method and system based on prompt engineering (CN117932073A), which obtains an unlabeled text set, obtains the initial pseudo-label of the unlabeled text set, classifies and fine-tunes the pre-trained model according to the text set containing the initial pseudo-label to generate a target pseudo-label text set, filters noise samples of the target pseudo-label text set, obtains a target classifier according to the filtered target pseudo-label text set, and classifies the text to be classified through the target classifier. Since the accuracy of the pseudo-label generation results automatically generated by computer-related technology is lower than that of the manual annotation classification results, this method applies the pseudo-label generation results to the subsequent process, which will interfere with the recognition effect of the model and reduce the accuracy.

[0007] In summary, the above three methods all have the problem of low classification accuracy, which needs to be solved urgently. Summary of the invention

[0008] In view of the above problems, this application provides a text classification method and related devices that combine a large model with dilated convolution and fine-tuning to improve the accuracy of the classification results. The specific solution is as follows:

[0009] The first aspect of the present application provides a text classification method that combines a large model with dilated convolution and fine-tuning, including:

[0010] A large language model pre-trained with large-scale corpus is used to extract a hidden layer matrix of the training text content, so as to obtain a training text vector based on the extracted hidden layer matrix;

[0011] The training text vector and the text classification results corresponding to the manually annotated training text content are used as training data to train a pre-constructed neural network model to obtain a text classification model, wherein the pre-constructed neural network model includes a dilated convolution layer, and the dilated convolution layer is used to perform dilated convolution processing on the input vector to obtain a sparse matrix that can reflect the text category to which the input vector belongs;

[0012] Extracting a hidden layer matrix of the text content to be classified using the large language model, so as to obtain a text vector to be classified based on the extracted hidden layer matrix;

[0013] The text vector to be classified is input into the text classification model to obtain a text classification result corresponding to the text content to be classified output by the model.

[0014] In a possible implementation, the process of extracting a hidden layer matrix of target text content by using the large language model to obtain a target text vector based on the extracted hidden layer matrix includes:

[0015] Segmenting the target text content into a plurality of basic elements, and converting the plurality of basic elements into numerical vectors;

[0016] Processing the numerical vector into a hidden layer matrix using the large language model;

[0017] The hidden layer matrix is ​​encoded into the target text vector, wherein if the target text content is the training text content, the target text vector is the training text vector, and if the target text content is the text content to be classified, the target text vector is the text vector to be classified.

[0018] In a possible implementation, the pre-built neural network model includes a plurality of the dilated convolutional layers, each of which uses two convolution functions with the same structure but non-shared weights to perform dilated convolution processing on the input vector, one of the two convolution functions is activated with a first activation function, and the other convolution function is not activated, and the dilation rate of each of the dilated convolutional layers is related to the dimension of the input vector.

[0019] In a possible implementation, the training text vector and the text classification result corresponding to the manually annotated training text content are used as training data to train a pre-built neural network model to obtain a text classification model, including:

[0020] Inputting the training text vector into the pre-built neural network model, so that the training text vector is subjected to multiple dilated convolution processing in multiple dilated convolution layers, obtaining a sparse matrix output by the last dilated convolution layer as a training sparse matrix, and obtaining a prediction classification result based on the training sparse matrix;

[0021] The pre-constructed neural network model is trained based on the predicted classification result and the text classification result corresponding to the manually annotated training text content to obtain the text classification model, wherein the loss function of the training process adopts cross entropy loss.

[0022] In a possible implementation, obtaining a prediction classification result based on the training sparse matrix includes:

[0023] Converting the training sparse matrix into a training sparse vector through a fully connected layer in the pre-built neural network model;

[0024] The training sparse vector is converted into the probability of a preset text classification label through a second activation function in the pre-built neural network model, so as to obtain the predicted classification result based on the probability.

[0025] In a possible implementation, the calculation formula used by the dilated convolution layer when performing dilated convolution processing on the input vector is: ,in, represents the input vector, represents the convolution function without activation function, represents the convolution function activated by the first activation function, represents the first activation function, represents the sparse matrix output by the dilated convolutional layer, Represents bit-wise multiplication.

[0026] The second aspect of the present application provides a text classification device that combines a large model with dilated convolution and fine-tuning, including:

[0027] A training data preprocessing module is used to extract a hidden layer matrix of the training text content using a large language model pre-trained with a large-scale corpus, so as to obtain a training text vector based on the extracted hidden layer matrix;

[0028] A model training module, used to train a pre-constructed neural network model using the training text vector and the text classification results corresponding to the manually annotated training text content as training data to obtain a text classification model, wherein the pre-constructed neural network model includes a dilated convolution layer, and the dilated convolution layer is used to perform dilated convolution processing on the input vector to obtain a sparse matrix that can reflect the text category to which the input vector belongs;

[0029] A preprocessing module for data to be classified, used for extracting a hidden layer matrix of the text content to be classified using the large language model, so as to obtain a text vector to be classified based on the extracted hidden layer matrix;

[0030] The text classification module is used to input the text vector to be classified into the text classification model to obtain the text classification result corresponding to the text content to be classified output by the model.

[0031] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the text classification method of the large model and dilated convolution fusion fine-tuning described in the first aspect or any implementation of the first aspect.

[0032] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program so that the electronic device can implement the text classification method of the large model and dilated convolution fusion fine-tuning described in the first aspect or any implementation method of the first aspect.

[0035] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can use the text classification method of the large model of the first aspect or any implementation of the first aspect to fuse fine-tune the dilated convolution.

[0036] By means of the above technical scheme, the text classification method provided by the present application combines a large model with dilated convolution and fine-tuning, uses a large language model pre-trained with large-scale corpus to extract the hidden layer matrix of the training text content, and obtains the training text vector based on the extracted hidden layer matrix. The text classification results corresponding to the training text vector and the manually annotated training text content are used as training data to train the pre-built neural network model to obtain a text classification model, and uses the large language model to extract the hidden layer matrix of the text content to be classified, and obtains the text vector to be classified based on the extracted hidden layer matrix. The text vector to be classified is input into the text classification model to obtain the text classification result corresponding to the text content to be classified output by the model.

[0037] The entire text classification process of the present application utilizes a large language model pre-trained with large-scale corpus, which saves the time of training the large language model in the text classification process and improves the efficiency of text classification. Since the large language model has been pre-trained with large-scale corpus, the large language model can learn more contextual data and knowledge, so that the hidden layer matrix extracted by the large language model contains richer semantic and syntactic information. The text vector obtained based on the hidden layer matrix is ​​used in the subsequent neural network model or text classification model, which can improve the accuracy of the text classification results.

[0038] Furthermore, the dilated convolution layer included in the neural network model and the text classification model can perform dilated convolution processing on the input vector, so that the neural network model and the text classification model can better capture the positional relationship and semantic relationship between characters or words that are farther away in the input vector, so that the obtained sparse matrix can more accurately reflect the text category to which the input vector belongs, so that text classification based on the sparse matrix makes the text classification result more accurate. Since the manually annotated text classification result is more accurate, it is used to train the neural network model, so that the text classification result of the trained text classification model is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0040] Figure 1 A schematic diagram of a system architecture provided for this application;

[0041] Figure 2 A schematic diagram of an optional hardware structure of the terminal 100 provided in this application;

[0042] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0043] Figure 4 A flowchart of a text classification method that combines a large model with dilated convolution and fine-tuning is provided in this application;

[0044] Figure 5 (a) is a schematic diagram of the ordinary convolution process;

[0045] Figure 5 (b) is a schematic diagram of the process of performing dilated convolution processing for each dilated convolution layer;

[0046] Figure 6 A schematic diagram of a hidden layer matrix provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the overall process of text classification provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a text classification device that combines a large model with dilated convolution and fine-tuning provided in this application;

[0049] Fig. 9 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0054] Among them, an application corresponding to the third-party system can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0055] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0056] Next describe Figure 1 The product form of the mid-terminal 100;

[0057] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0058] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0059] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0060] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0061] Among them, the input device 132 can receive input data and the like.

[0062] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0063] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0064] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0065] Among them, the memory 120 can be used to store software codes related to the text classification method of fusing a large model with dilated convolution and fine-tuning, the processor 170 can execute the steps of the text classification method of fusing a large model with dilated convolution and fine-tuning, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to implement corresponding functions.

[0066] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile Communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0067] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0068] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0069] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0070] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0071] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0072] Next describe Figure 1 The product form of the server 200;

[0073] Figure 3 A schematic diagram of the structure of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0074] The bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0075] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0076] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0077] Among them, the memory 204 can be used to store software codes related to the text classification method of fusing a large model with dilated convolution and fine-tuning, the processor 202 can execute the steps of the text classification method of fusing a large model with dilated convolution and fine-tuning of the chip, and can also schedule other units to implement corresponding functions.

[0078] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, DSPs, microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0079] The present application provides a text classification method that combines a large model with dilated convolution and fine-tuning. The following is a detailed introduction to the text classification method that combines a large model with dilated convolution and fine-tuning in the embodiment of the present application in conjunction with the accompanying drawings.

[0080] Reference Figure 4 , Figure 4 A flowchart of a text classification method for fusing a large model with dilated convolution and fine-tuning is provided in an embodiment of the present application. The method may include:

[0081] Step S401: extract the hidden layer matrix of the training text content using the large language model pre-trained with large-scale corpus, so as to obtain the training text vector based on the extracted hidden layer matrix.

[0082] Here, the training text content refers to the text content used to train the text classification model in a specific industry classification scenario. A specific industry refers to an industry with a large number of professional terms and knowledge in a specific field. Since it contains a large number of professional terms and knowledge in a specific field, the text classification results of the specific industry by the traditional text classification method may be greatly different from the actual category, so the text classification method in this application is proposed.

[0083] In the process of implementing the present application, the inventors of this case initially thought of using a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model to extract the hidden layer matrix of the training text content. However, the BERT model has less training corpus, resulting in the extracted hidden layer matrix containing less semantic and syntactic information of the training text content. Furthermore, the training text vectors obtained based on the hidden layer matrix were used to train the neural network model below, resulting in a low accuracy rate in the classification results of the trained text classification model.

[0084] To this end, the inventor of this case conducted further research and found that it is possible to obtain a large language model pre-trained with large-scale corpus, and then extract the hidden layer matrix of the training text content based on the obtained large language model. Here, pre-training refers to training a large language model with large-scale corpus, so that the large language model can learn richer semantic and syntactic information in the large-scale corpus, so as to extract a hidden layer matrix with richer semantic and syntactic information.

[0085] It should be understood that the hidden layer matrix extracted by the large language model is intermediate data, and the data actually output by the large language model is generally the data decoded from the hidden layer matrix. However, the decoding process of the large language model may be wrong or the decoding result may have a certain error compared with the actual application. In order to avoid various problems that may arise in the decoding process, this application does not use the decoding result of the hidden layer matrix, but directly uses the hidden layer matrix to obtain the training text vector (optionally, the dimension of the training text vector is 32000), which helps to improve the accuracy of subsequent text classification results.

[0086] That is to say, the process of extracting the hidden layer matrix of the training text content using the large language model in the embodiment of the present application specifically refers to: extracting the hidden layer matrix of the training text content using the encoder of the large language model.

[0087] Optionally, the large language model may be a large model structure such as a general language model (GLM), a Qwen large language model, a YI large language model, or a LLM (Large Language Model) large language model.

[0088] Optionally, the encoder is a Transformer encoder, which uses a self-attention mechanism to capture long-distance dependencies in sequence data. It can process all words in the input sequence in parallel and calculate the correlation between each word and other words through the self-attention mechanism. This mechanism enables the Transformer to have higher efficiency and better performance when processing target text content.

[0089] The main advantages of using a large language model pre-trained with large-scale corpus to extract the hidden layer matrix in this step include the following: (1) Through large-scale corpus pre-training, the large language model can learn more contextual data and knowledge, thereby improving the generalization ability of the large language model for unknown data (such as training text content), so that the large language model can extract a hidden layer matrix containing richer semantic and syntactic information; (2) The embodiment of the present application directly uses the large language model pre-trained with large-scale corpus to extract the hidden layer matrix, and the extracted hidden layer matrix can be directly used in the subsequent training and application process of the text classification model, because there is no need to train the large language model when training the text classification model later. (3) Since the large language model is pre-trained using corpus that can be applied to various natural language processing tasks (such as text classification, sentiment analysis, machine translation, etc.), the versatility and scalability of natural language processing technology are improved; (4) The encoder part of the large language model can extract hidden layer matrices with richer semantic and syntactic information. The extracted hidden layer matrices are used in subsequent text classification models, which can effectively improve the accuracy and performance of the text classification model, making the text classification model have better performance on various text classification tasks.

[0090] Step S402: Using the training text vector and the text classification results corresponding to the manually annotated training text content as training data, the pre-built neural network model is trained to obtain a text classification model.

[0091] Since the human brain has powerful learning and memory abilities, can continuously learn and improve from experience, and can better understand and process contextual information, compared to the prior art method of using pseudo-labels to generate results, this application uses a method of manually annotating the text classification results corresponding to the training text content, so that the trained text classification model can perform text classification more accurately.

[0092] In an embodiment of the present application, the pre-built neural network model includes a dilated convolution layer, which can perform dilated convolution processing on the input vector to obtain a sparse matrix that can reflect the text category to which the input vector belongs.

[0093] In an optional embodiment, the pre-built neural network model includes multiple dilated convolution layers, each of the multiple dilated convolution layers uses two convolution functions with the same structure (for example, the same number of convolution kernels and the same convolution window size) but non-shared weights to perform dilated convolution processing on the input vector, wherein one of the two convolution functions is activated with a first activation function, and the other convolution function is not activated, and the dilation rate of each dilated convolution layer is related to the dimension of the input vector.

[0094] When the embodiment of the present application trains a pre-built neural network model based on a training text vector, the training text vector can be used as an input vector of a first dilated convolutional layer among multiple dilated convolutional layers, the sparse matrix output by the first dilated convolutional layer can be used as an input vector of a second dilated convolutional layer among multiple dilated convolutional layers, the sparse matrix output by the second dilated convolutional layer can be used as an input vector of a third dilated convolutional layer among multiple dilated convolutional layers, and so on.

[0095] Optionally, the network depth of multiple dilated convolutional layers is 32, and the dimension of the sparse matrix output by the last dilated convolutional layer is 512.

[0096] In an optional embodiment, the calculation formula used by each dilated convolution layer to perform dilated convolution processing on the input vector is: ,in, represents the input vector, represents the convolution function without activation function, represents the convolution function activated by the first activation function, represents the first activation function, represents the sparse matrix of the output of the dilated convolutional layer, Represents bit-wise multiplication.

[0097] See Figure 5 (a) and Figure 5 (b). Figure 5 (a) is a schematic diagram of the ordinary convolution processing process, and Figure 5 (b) is a schematic diagram of the process of performing dilated convolution processing on each dilated convolution layer. As shown in the figure, each node of the ordinary convolution (a circle in the figure represents a node, and each node can represent a word or a character) can only capture the information of the three nodes before and after, and the higher the number of layers, the lower the correlation with the longer distance time series. However, the dilated convolution can control the interval of the input time series by adjusting the dilation rate, so that it can capture more distant position information in higher convolution layers without increasing model parameters.

[0098] It can be seen that compared with ordinary convolution, the embodiment of the present application performs dilated convolution on the input vector, which can capture the positional relationship and semantic relationship between characters or words that are farther away in the input vector during the dilated convolution process, so that the sparse matrix can contain richer semantic and structural information, and thus the sparse matrix can more accurately reflect the text category to which the input vector belongs.

[0099] This step fine-tunes the parameters of the pre-built neural network model through training data, so that the trained text classification model can learn the semantic and structural information contained in the training data, and apply the learned information knowledge to subsequent text classification prediction, which can effectively improve the accuracy of the classification results.

[0100] Step S403: using the large language model to extract the hidden layer matrix of the text content to be classified, so as to obtain the text vector to be classified based on the extracted hidden layer matrix.

[0101] Step S404: input the text vector to be classified into the text classification model to obtain the text classification result corresponding to the text content to be classified output by the model.

[0102] The text classification method provided by the application combines a large model with dilated convolution and fine-tuning. The large language model pre-trained with a large-scale corpus is used to extract the hidden layer matrix of the training text content, and the training text vector is obtained based on the extracted hidden layer matrix. The text classification results corresponding to the training text vector and the manually annotated training text content are used as training data to train a pre-built neural network model to obtain a text classification model. The large language model is used to extract the hidden layer matrix of the text content to be classified, and the text vector to be classified is obtained based on the extracted hidden layer matrix. The text vector to be classified is input into the text classification model to obtain the text classification result corresponding to the text content to be classified output by the model.

[0103] The entire text classification process of the present application utilizes a large language model pre-trained with large-scale corpus, which saves the time of training the large language model in the text classification process and improves the efficiency of text classification. Since the large language model has been pre-trained with large-scale corpus, the large language model can learn more contextual data and knowledge, so that the hidden layer matrix extracted by the large language model contains richer semantic and syntactic information. The text vector obtained based on the hidden layer matrix is ​​used in the subsequent neural network model or text classification model, which can improve the accuracy of the text classification results.

[0104] Furthermore, the dilated convolution layer included in the neural network model and the text classification model can perform dilated convolution processing on the input vector, so that the neural network model and the text classification model can better capture the positional relationship and semantic relationship between characters or words that are farther away in the input vector, so that the obtained sparse matrix can more accurately reflect the text category to which the input vector belongs, so that text classification based on the sparse matrix makes the text classification result more accurate. Since the manually annotated text classification result is more accurate, it is used to train the neural network model, so that the text classification result of the trained text classification model is more accurate.

[0105] In some embodiments of the present application, a process of extracting a hidden layer matrix of target text content using a large language model to obtain a target text vector based on the extracted hidden layer matrix is ​​introduced.

[0106] Here, if the target text content is the training text content, the target text vector is the training text vector; if the target text content is the text content to be classified, the target text vector is the text vector to be classified.

[0107] Optionally, the process of "using a large language model to extract a hidden layer matrix of the target text content to obtain a target text vector based on the extracted hidden layer matrix" may include: dividing the target text content into a number of basic elements, and converting the number of basic elements into numerical vectors; using a large language model to process the numerical vector into a hidden layer matrix; encoding the hidden layer matrix into a target text vector.

[0108] Specifically, the embodiment of the present application first divides the target text content into several basic elements, namely tokens.

[0109] Optionally, the basic element can be a word or a character.

[0110] For example, if the target text content is "This product is very good", then the several basic elements segmented out may be "This product is very good"; for another example, if the target text content is "Today's weather is very bad", then the several basic elements segmented out may be "Today's weather is very bad".

[0111] Next, the application may convert several basic elements into numerical vectors.

[0112] For example, taking the basic elements "This product is very good" as an example, the numerical vectors converted from these basic elements are: tensor([[151331,151333,151336,198,99637,101791,98443,109854,1773,151337]]).

[0113] For another example, taking the basic elements "Today's weather is very bad" as an example, the numerical vectors converted from these basic elements are: tensor([[151331,151333,151336,198,102662,98961,118185,1773,151337]]).

[0114] Next, the application can use a large language model to extract the hidden state embedding matrix corresponding to the target text content based on the numerical vector. The extraction result is as follows: Figure 6 shown. Figure 6 In the example, the hidden layer matrix in the first tensor corresponds to "this product is very good", and the hidden layer matrix in the second tensor corresponds to "today's weather is very bad".

[0115] After the hidden layer matrix is ​​obtained, the embodiment of the present application can encode the hidden layer matrix into a target text vector.

[0116] The embodiment of the present application utilizes a pre-trained large language model to obtain the hidden state output of the target text content. These hidden states can effectively represent the semantic and syntactic information in the target text content. The target text vector encoded by the hidden layer matrix extracted by the large language model is used as the embedding vector of the target text content, and then spliced ​​with the subsequent pre-built neural network model or text classification model, which can improve the accuracy and robustness of the model classification.

[0117] In other embodiments of the present application, the process of "using the text classification results corresponding to the training text vectors and the manually annotated training text contents as training data, training the pre-built neural network model, and obtaining a text classification model" in step S402 is introduced.

[0118] As introduced above, the pre-built neural network model includes multiple dilated convolution layers. Then, the embodiment of the present application can input the training text vector into the pre-built neural network model, so that the training text vector undergoes multiple dilated convolution processes in multiple dilated convolution layers, and obtains the sparse matrix output by the last dilated convolution layer as the training sparse matrix, and obtains the predicted classification result based on the training sparse matrix.

[0119] In a possible implementation, the pre-built neural network model also includes a fully connected layer and a second activation function. Then, the above process of "obtaining a predicted classification result based on the training sparse matrix" may include: converting the training sparse matrix into a training sparse vector through the fully connected layer in the pre-built neural network model; converting the training sparse vector into the probability of a preset text classification label through the second activation function in the pre-built neural network model, so as to obtain a predicted classification result based on the probability.

[0120] Furthermore, the embodiments of the present application can train a pre-constructed neural network model based on the text classification results corresponding to the predicted classification results and the manually annotated training text content to obtain a text classification model, wherein the loss function of the training process adopts cross entropy loss.

[0121] Optionally, in order to verify the classification accuracy of the text classification model, the embodiment of the present application can also obtain several test text contents, use the above large language model to extract the hidden layer matrix of the test text contents, and obtain a test text vector, input the test text vector into the text classification model, and obtain a text classification result corresponding to the test text content. If the proportion of the text classification results corresponding to the test text content output by the model and the text classification results corresponding to the manually determined test text content is consistent with expectations, it is considered that the text classification model can obtain a relatively accurate text classification result, and it can be used in the subsequent step S404. Otherwise, the training data is used to continue to train the text classification model until the test consistency ratio reaches the expected level.

[0122] In order to make those skilled in the art understand the present application better, Figure 7 The overall process of text classification for this application is introduced.

[0123] like Figure 7 As shown, it is a schematic diagram of the overall process of text classification provided in an embodiment of the present application, which shows an optional network structure of the text classification model, including m dilated convolutional layers, a fully connected layer and a second activation function.

[0124] First, the text classification model is trained. The training process can refer to the previous embodiment and will not be repeated here.

[0125] Next, the encoder Transformer of the large language model pre-trained with large-scale corpus is used to extract the hidden layer matrix of the text content to be classified, and the text vector to be classified is obtained based on the extracted hidden layer matrix. Figure 7 Expressed by (x1,x2,x3,…,xn).

[0126] After that, the text vector to be classified can be input into the first dilated convolutional layer of the text classification model. In the first dilated convolutional layer, the text vector to be classified (x1, x2, x3, …, xn) is used as the formula In , the sparse vector Y1 output by the first dilated convolutional layer is calculated; where the dilation rate Dilation of the first dilated convolutional layer is set to 2, and the convolution window window is set to 3.

[0127] The sparse vector Y1 output by the first dilated convolutional layer can be input to the second dilated convolutional layer of the text classification model. In the second dilated convolutional layer, the sparse vector Y1 is used as the formula In , the sparse vector Y2 output by the second dilated convolutional layer is calculated; where the dilation rate Dilation of the second dilated convolutional layer is set to 4 and the convolution window window is set to 3.

[0128] And so on, until the sparse vector output by the m-1th dilated convolution layer can be input into the mth dilated convolution layer of the text classification model. In the mth dilated convolution layer, the sparse vector output by the m-1th dilated convolution layer is used as the formula In , the sparse vector Ym output by the m-th dilated convolutional layer is calculated; where the dilation rate Dilation of the m-th dilated convolutional layer is set to , is the dimension of the sparse vector output by the m-1th dilated convolutional layer, and the convolution window is set to 3.

[0129] The sparse vector Ym output by the m-th dilated convolutional layer is input into the fully connected layer to obtain a sparse vector (y1, y2, y3, …, yn).

[0130] Finally, the sparse vector (y1, y2, y3, …, yn) is converted into the probability of the preset text classification label through the second activation function, so as to obtain the text classification result corresponding to the text content to be classified based on the probability.

[0131] It should also be noted that in the above multiple dilated convolutional layers, the dilation rates are 2, 4, 8, ..., It increases gradually. This is to make the dilated convolution process follow the principle of "try not to repeat or miss" for each node of the input vector (one node represents a word or a character), so as to better extract the eigenvalues ​​in the sparse dictionary to obtain a sparse vector.

[0132] In summary, this application proposes a text classification method based on Transformer and dilated convolution fusion fine-tuning, which can efficiently classify text content in specific industry classification scenarios with high classification accuracy.

[0133] A text classification method provided by an embodiment of the present application is introduced above, and a device for executing the above text classification method will be introduced below.

[0134] See also Figure 8 , Figure 8 A structural diagram of a text classification device that combines a large model with dilated convolution and fine-tuning is provided in an embodiment of the present application. Figure 8 As shown, the device may include:

[0135] The training data preprocessing module 801 is used to extract the hidden layer matrix of the training text content using the large language model pre-trained with large-scale corpus, so as to obtain the training text vector based on the extracted hidden layer matrix;

[0136] A model training module 802 is used to train a pre-built neural network model using the training text vector and the text classification results corresponding to the manually annotated training text content as training data to obtain a text classification model, wherein the pre-built neural network model includes a dilated convolution layer, and the dilated convolution layer is used to perform dilated convolution processing on the input vector to obtain a sparse matrix that can reflect the text category to which the input vector belongs;

[0137] The data preprocessing module 803 is used to extract the hidden layer matrix of the text content to be classified by using the large language model, so as to obtain the text vector to be classified based on the extracted hidden layer matrix;

[0138] The text classification module 804 is used to input the text vector to be classified into the text classification model to obtain the text classification result corresponding to the text content to be classified output by the model.

[0139] In one possible implementation, the training data preprocessing module uses a large language model pre-trained with large-scale corpus to extract a hidden layer matrix of the training text content, so as to obtain a training text vector based on the extracted hidden layer matrix. Specifically, it is used to: divide the training text content into a number of basic elements, and convert the several basic elements into numerical vectors, use the large language model to process the numerical vector into a hidden layer matrix, and encode the hidden layer matrix into a training text vector.

[0140] Correspondingly, the data preprocessing module to be classified uses a large language model to extract the hidden layer matrix of the text content to be classified, so as to obtain the text vector to be classified based on the extracted hidden layer matrix. Specifically, it is used to: divide the text content to be classified into several basic elements, and convert the several basic elements into numerical vectors, use the large language model to process the numerical vector into a hidden layer matrix, and encode the hidden layer matrix into the text vector to be classified.

[0141] In a possible implementation, the pre-built neural network model in the model training module includes multiple dilated convolution layers, each of which uses two convolution functions with the same structure but non-shared weights to perform dilated convolution processing on the input vector, one of the two convolution functions is activated by a first activation function, and the other convolution function is not activated, and the dilation rate of each dilated convolution layer is related to the dimension of the input vector.

[0142] In a possible implementation, the model training module uses the training text vector and the text classification results corresponding to the manually annotated training text content as training data to train the pre-built neural network model. When the text classification model is obtained, it is specifically used for:

[0143] The training text vector is input into the pre-built neural network model, so that the training text vector is subjected to multiple dilated convolution processing in multiple dilated convolution layers, and the sparse matrix output by the last dilated convolution layer is obtained as the training sparse matrix, and the prediction classification result is obtained based on the training sparse matrix;

[0144] A pre-built neural network model is trained based on the predicted classification results and the text classification results corresponding to the manually annotated training text content to obtain a text classification model, wherein the loss function of the training process adopts the cross entropy loss.

[0145] In a possible implementation, when the model training module obtains the prediction classification result based on the training sparse matrix, it is specifically used to:

[0146] Convert the training sparse matrix into training sparse vector through the fully connected layer in the pre-built neural network model;

[0147] The training sparse vector is converted into the probability of a preset text classification label through the second activation function in the pre-built neural network model to obtain a predicted classification result based on the probability.

[0148] In one possible implementation, the calculation formula used by each dilated convolution layer to perform dilated convolution on the input vector is: ,in, represents the input vector, represents the convolution function without activation function, represents the convolution function activated by the first activation function, represents the first activation function, represents the sparse matrix of the output of the dilated convolutional layer, Represents bit-wise multiplication.

[0149] The present application also provides an electronic device in an embodiment. Fig. 9 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0150] like Fig. 9 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0151] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a memory card, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0152] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the large model and dilated convolution fusion fine-tuning text classification methods provided in the embodiments of the present application.

[0153] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the large model and dilated convolution fusion fine-tuning text classification methods provided in the embodiments of the present application.

[0154] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0155] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0156] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0157] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A text classification method combining a large model with dilated convolution and fine-tuning, characterized in that: include: A large language model pre-trained with large-scale corpus is used to extract a hidden layer matrix of the training text content, so as to obtain a training text vector based on the extracted hidden layer matrix; The training text content refers to the text content used to train the text classification model in a specific industry classification scenario; the large language model is the Qwen large language model or the YI large language model; the dimension of the training text vector is 32000; The training text vector and the text classification results corresponding to the manually annotated training text content are used as training data to train the pre-constructed neural network model to obtain a text classification model, wherein the pre-constructed neural network model includes a plurality of dilated convolution layers, the dilated convolution layers are used to perform dilated convolution processing on the input vector to obtain a sparse matrix that can reflect the text category to which the input vector belongs; in the plurality of dilated convolution layers, the dilation rates are 2, 4, 8, ..., Gradually increasing, is the dimension of the sparse vector output by the m-1th dilated convolutional layer, where m represents the number of dilated convolutional layers; the convolution window is set to 3, the network depth of multiple dilated convolutional layers is 32, and the dimension of the sparse matrix output by the last dilated convolutional layer is 512; Extracting a hidden layer matrix of the text content to be classified using the large language model, so as to obtain a text vector to be classified based on the extracted hidden layer matrix; Inputting the to-be-classified text vector into the text classification model to obtain a text classification result corresponding to the to-be-classified text content output by the model; The pre-built neural network model comprises a plurality of the dilated convolutional layers, each of which uses two convolutional functions with the same structure but no shared weights to perform dilated convolution processing on the input vector, one of the two convolutional functions is activated by a first activation function, and the other convolutional function is not activated, and the dilation rate of each dilated convolutional layer is related to the dimension of the input vector; The training text vector and the text classification result corresponding to the manually annotated training text content are used as training data to train the pre-built neural network model to obtain a text classification model, including: Inputting the training text vector into the pre-built neural network model, so that the training text vector is subjected to multiple dilated convolution processes in multiple dilated convolution layers, and obtaining a sparse matrix output by the last dilated convolution layer as a training sparse matrix; The training sparse matrix is ​​converted into a training sparse vector through a fully linked layer in the pre-constructed neural network model, and the training sparse vector is converted into a probability of a preset text classification label through a second activation function in the pre-constructed neural network model, so as to obtain a predicted classification result based on the probability; The pre-constructed neural network model is trained based on the predicted classification result and the text classification result corresponding to the manually annotated training text content to obtain the text classification model, wherein the loss function of the training process adopts cross entropy loss.

2. The text classification method of the large model and dilated convolution fusion fine-tuning according to claim 1 is characterized in that: The process of extracting the hidden layer matrix of the target text content by using the large language model to obtain the target text vector based on the extracted hidden layer matrix includes: Segmenting the target text content into a plurality of basic elements, and converting the plurality of basic elements into numerical vectors; Processing the numerical vector into a hidden layer matrix using the large language model; The hidden layer matrix is ​​encoded into the target text vector, wherein if the target text content is the training text content, the target text vector is the training text vector, and if the target text content is the text content to be classified, the target text vector is the text vector to be classified.

3. The text classification method of the large model and dilated convolution fusion fine-tuning according to any one of claims 1 to 2 is characterized in that: The calculation formula used by the dilated convolution layer to perform dilated convolution processing on the input vector is: ,in, represents the input vector, represents the convolution function without activation function, represents the convolution function activated by the first activation function, represents the first activation function, represents the sparse matrix output by the dilated convolutional layer, Represents bit-wise multiplication.

4. A text classification device that combines a large model with dilated convolution and fine-tuning, characterized in that: include: A training data preprocessing module is used to extract a hidden layer matrix of the training text content using a large language model pre-trained with a large-scale corpus, so as to obtain a training text vector based on the extracted hidden layer matrix; The training text content refers to the text content used to train the text classification model in a specific industry classification scenario; the large language model is the Qwen large language model or the YI large language model; the dimension of the training text vector is 32000; The model training module is used to train the pre-constructed neural network model using the training text vector and the text classification results corresponding to the manually annotated training text content as training data to obtain a text classification model, wherein the pre-constructed neural network model includes a plurality of dilated convolution layers, the dilated convolution layers are used to perform dilated convolution processing on the input vector to obtain a sparse matrix that can reflect the text category to which the input vector belongs; in the plurality of dilated convolution layers, the dilation rates are 2, 4, 8, ..., Gradually increasing, is the dimension of the sparse vector output by the m-1th dilated convolutional layer, where m represents the number of dilated convolutional layers; the convolution window is set to 3, the network depth of multiple dilated convolutional layers is 32, and the dimension of the sparse matrix output by the last dilated convolutional layer is 512; A preprocessing module for data to be classified, used for extracting a hidden layer matrix of the text content to be classified using the large language model, so as to obtain a text vector to be classified based on the extracted hidden layer matrix; A text classification module, used for inputting the text vector to be classified into the text classification model to obtain a text classification result corresponding to the text content to be classified output by the model; The pre-built neural network model comprises a plurality of the dilated convolutional layers, each of which uses two convolutional functions with the same structure but no shared weights to perform dilated convolution processing on the input vector, one of the two convolutional functions is activated by a first activation function, and the other convolutional function is not activated, and the dilation rate of each dilated convolutional layer is related to the dimension of the input vector; The model training module uses the training text vector and the text classification results corresponding to the manually annotated training text content as training data to train the pre-built neural network model to obtain a text classification model, including: Inputting the training text vector into the pre-built neural network model, so that the training text vector is subjected to multiple dilated convolution processes in multiple dilated convolution layers, and obtaining a sparse matrix output by the last dilated convolution layer as a training sparse matrix; The training sparse matrix is ​​converted into a training sparse vector through a fully linked layer in the pre-constructed neural network model, and the training sparse vector is converted into a probability of a preset text classification label through a second activation function in the pre-constructed neural network model, so as to obtain a predicted classification result based on the probability; The pre-constructed neural network model is trained based on the predicted classification result and the text classification result corresponding to the manually annotated training text content to obtain the text classification model, wherein the loss function of the training process adopts cross entropy loss.

5. A computer program product, characterized in that It includes computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the text classification method of fine-tuning the fusion of a large model and dilated convolution as described in any one of claims 1 to 3.

6. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the text classification method of the large model and dilated convolution fusion fine-tuning as described in any one of claims 1 to 3.

7. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the text classification method of the large model and dilated convolution fusion fine-tuning as described in any one of claims 1 to 3.

Images