Text classification method, device, electronic device and storage medium
By splitting the sample set into positive and negative sample pairs and training the text classification model according to the similarity difference, the problem of low classification accuracy in the prior art is solved and higher text classification accuracy is achieved.
Patent Information
- Application Number
- CN202210432887.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-21
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-04-21
AI Technical Summary
The existing text classification model does not consider the correlation between samples during the training process, resulting in low classification accuracy.
By splitting the sample set of each field into positive sample pairs and negative sample pairs, the target loss function is determined based on the similarity difference between the positive sample pair and the negative sample pair, the text classification model of each field is trained, and when classification is performed, the classification model is performed is performed, Calculate weights based on sample distribution equilibrium to improve prediction accuracy.
By enhancing the interaction between samples, the model can learn more accurately the similarity between positive sample pairs and the dissimilarity between negative sample pairs, thereby improving the accuracy of text classification.
Smart Images

Figure CN114706985B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a text classification method, device, electronic device and storage medium. Background Art
[0002] Text classification is an important topic in the field of natural language processing. Text classification can be widely used in many scenarios such as search engines, intelligent question and answer, knowledge retrieval, and dialogue systems.
[0003] Currently, the mainstream approach is to use a text classification model to classify text. The training process of a text classification model is usually to input samples with labels into the model and optimize the text classification model by minimizing the difference between the label and the predicted value. However, this method does not consider the correlation between samples, resulting in low classification accuracy. Therefore, a text classification method is urgently needed to improve the accuracy of text classification. Summary of the invention
[0004] In view of the above, it is necessary to provide a text classification method, device, electronic device and storage medium, aiming to improve the accuracy of text classification.
[0005] The text classification method provided by the present invention comprises the following steps:
[0006] Obtain a sample set carrying text category information corresponding to each field, and split each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information;
[0007] Determine a target loss function according to the similarity difference between the positive sample pair and the negative sample pair, and train an initial classification model corresponding to each field by minimizing the target loss function to obtain a text classification model corresponding to each field;
[0008] Receive a text to be classified, identify a target field corresponding to the text to be classified, determine whether a sample set corresponding to the target field is evenly distributed across text categories, and if not, calculate a weight corresponding to each text category in the target field;
[0009] The text to be classified is input into a text classification model corresponding to the target domain, and a predicted value of each text category of the text to be classified in the target domain is obtained; a target value of each text category of the text to be classified in the target domain is determined based on the predicted value and the weight; and a target text category corresponding to the text to be classified is determined based on the target value.
[0010] Optionally, determining the target loss function according to the similarity difference between the positive sample pair and the negative sample pair includes:
[0011] Selecting a sample set in a field, inputting the selected sample set into the encoding network of the initial classification model to perform encoding processing, and obtaining an encoding vector for each sample in the selected sample set;
[0012] Calculate a first similarity value of each positive sample pair and a second similarity value of each negative sample pair corresponding to the selected sample set based on the encoding vector;
[0013] Determine a first loss function corresponding to the selected sample set based on a difference between the first similarity value and the second similarity value;
[0014] Input the encoding vector into the classification network of the initial classification model to perform classification processing, obtain the probability value of each text category of each sample in the selected sample set in the corresponding field, and determine the second loss function corresponding to the selected sample set based on the probability value;
[0015] A target loss function corresponding to the selected sample set is determined based on the first loss function and the second loss function.
[0016] Optionally, the identifying the target field corresponding to the text to be classified includes:
[0017] Performing word segmentation processing on the text to be classified to obtain a word set;
[0018] Match each word in the word set with the word library corresponding to each field to obtain a matching word set corresponding to each field;
[0019] The field with the largest number of matching words in the matching word set is taken as the target field corresponding to the text to be classified.
[0020] Optionally, splitting each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information includes:
[0021] Combine each sample in each sample set with samples of the same text category in pairs to obtain multiple positive sample pairs corresponding to each sample set;
[0022] Each sample in each sample set is combined with samples of different text categories in pairs to obtain multiple negative sample pairs corresponding to each sample set.
[0023] Optionally, the determining whether the sample set corresponding to the target domain is evenly distributed in the text category includes:
[0024] Counting the number of samples in each text category of the sample set corresponding to the target field;
[0025] Calculate the sample average corresponding to the target field based on the sample quantity;
[0026] Calculate the variance of the sample set corresponding to the target field based on the sample quantity and the sample average;
[0027] It is determined whether the sample set corresponding to the target domain is evenly distributed in the text category according to the variance.
[0028] Optionally, after determining whether the sample set corresponding to the target domain is evenly distributed in the text categories, the method further includes:
[0029] If the judgment is yes, the text to be classified is input into the text classification model corresponding to the target field to obtain the predicted value of the text to be classified in each text category, and the target text category corresponding to the text to be classified is determined based on the predicted value.
[0030] Optionally, the first loss function is:
[0031]
[0032] Among them, L is the first loss function corresponding to the selected sample set, S i is the second similarity value of the i-th negative sample pair corresponding to the selected sample set, S j is the first similarity value of the jth positive sample pair corresponding to the selected sample set, m is the total number of negative sample pairs in the selected sample set, and n is the total number of positive sample pairs in the selected sample set.
[0033] In order to solve the above problem, the present invention further provides a text classification device, the device comprising:
[0034] A splitting module is used to obtain a sample set carrying text category information corresponding to each field, and split each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information;
[0035] A training module, used for determining a target loss function according to the similarity difference between the positive sample pair and the negative sample pair, and training an initial classification model corresponding to each field by minimizing the target loss function to obtain a text classification model corresponding to each field;
[0036] A judgment module is used to receive the text to be classified, identify the target field corresponding to the text to be classified, and judge whether the sample set corresponding to the target field is evenly distributed in the text category. If the judgment is not, calculate the weight corresponding to each text category in the target field;
[0037] A classification module is used to input the text to be classified into a text classification model corresponding to the target field, obtain a predicted value of each text category of the text to be classified in the target field, determine a target value of each text category of the text to be classified in the target field based on the predicted value and the weight, and determine a target text category corresponding to the text to be classified based on the target value.
[0038] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:
[0039] at least one processor; and,
[0040] a memory communicatively connected to the at least one processor; wherein,
[0041] The memory stores a text classification program that can be executed by the at least one processor. The text classification program is executed by the at least one processor so that the at least one processor can perform the above-mentioned text classification method.
[0042] In order to solve the above problem, the present invention further provides a computer-readable storage medium, on which a text classification program is stored. The text classification program can be executed by one or more processors to implement the above text classification method.
[0043] Compared with the prior art, the present invention first splits the sample set corresponding to each field into multiple positive sample pairs and multiple negative sample pairs; then, the target loss function is determined according to the similarity difference between the positive sample pairs and the negative sample pairs, and the text classification model corresponding to each field is trained by minimizing the target loss function; then, the target field corresponding to the text to be classified is identified, and it is judged whether the sample set corresponding to the target field is evenly distributed in the text category. If it is judged to be not, the weight corresponding to each text category in the target field is calculated; finally, the text to be classified is input into the text classification model corresponding to the target field, and the target text category corresponding to the text to be classified is determined based on the predicted value output by the text classification model and the weight. The present invention determines the target loss function according to the similarity difference between the positive sample pairs and the negative sample pairs, so that the model can learn the similarity between the positive sample pairs and the dissimilarity between the negative sample pairs, deepen the interaction between the samples, and improve the classification accuracy of the model. Therefore, the present invention improves the accuracy of text classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 A flowchart of a text classification method provided by an embodiment of the present invention;
[0045] Figure 2 A schematic diagram of modules of a text classification device provided by an embodiment of the present invention;
[0046] Figure 3 A schematic diagram of the structure of an electronic device for implementing a text classification method provided by an embodiment of the present invention.
[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0048] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0049] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0050] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0051] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0052] The present invention provides a text classification method. Figure 1 FIG. 1 is a flow chart of a text classification method provided by an embodiment of the present invention. The method can be executed by an electronic device, and the electronic device can be implemented by software and / or hardware.
[0053] In this embodiment, the text classification method includes the following steps S1-S4:
[0054] S1. Obtain a sample set carrying text category information corresponding to each field, and split each sample set into multiple positive sample pairs and multiple negative sample pairs according to the text category information.
[0055] In this embodiment, a corresponding sample set is collected in advance for each field, which may include medical care, sports, finance, education, tourism, etc. The samples in the sample set carry text category information labels. For example, the text categories in the medical field include internal medicine, surgery, emergency department, fitness department, neurology, etc.
[0056] The step of splitting each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information includes the following steps A11-A12:
[0057] A11. Combine each sample in each sample set with samples of the same text category in pairs to obtain multiple positive sample pairs corresponding to each sample set;
[0058] For example, in the medical field, if the samples of the internal medicine, surgery, emergency department, fitness department, and brain department text categories are 500, 1000, 1200, 1500, and 1600 respectively, then the number of positive sample pairs is
[0059] A12. Combine each sample in each sample set with samples of different text categories in pairs to obtain multiple negative sample pairs corresponding to each sample set.
[0060] For sample 1 of the internal medicine text category in the medical field, it can be combined with any of the 5300 samples (where 5300 = 1000 + 1200 + 1500 + 1600) corresponding to the surgery, emergency, fitness, and brain text categories to form a negative sample pair. The number of negative sample pairs corresponding to sample 1 is The number of negative sample pairs corresponding to the medical field
[0061] S2. Determine a target loss function according to the similarity difference between the positive sample pair and the negative sample pair, and train an initial classification model corresponding to each field by minimizing the target loss function to obtain a text classification model corresponding to each field.
[0062] The positive sample pair is two samples of the same text category, and the negative sample pair is two samples of different text categories. In this embodiment, increasing the similarity distance between the positive sample pair and the negative sample pair is used as the first goal to construct a first loss function, and reducing the difference between the predicted value and the label value of each sample is used as the second goal to construct a second loss function. The target loss function is determined based on the first and second loss functions, and the model is trained by minimizing the target loss function. In this way, the model can learn the similarities between the positive sample pairs and the dissimilarity between the negative sample pairs, deepen the interaction between the samples, and also shorten the distance between the predicted value and the label value, so that the trained text classification model is more accurate.
[0063] Determining the target loss function according to the similarity difference between the positive sample pair and the negative sample pair includes the following steps B11-B15:
[0064] B11, selecting a sample set in a field, inputting the selected sample set into the encoding network of the initial classification model to perform encoding processing, and obtaining an encoding vector for each sample in the selected sample set;
[0065] In this embodiment, the initial classification model can be a transformer model, which includes an encoding network and a classification network. The encoding network is used to encode the input text (including character encoding, position encoding, and semantic encoding) to obtain an encoding vector of the input text.
[0066] B12. Calculate, based on the encoding vector, a first similarity value of each positive sample pair and a second similarity value of each negative sample pair corresponding to the selected sample set;
[0067] In this embodiment, cosine similarity, Manhattan distance, Euclidean distance, or explicit distance algorithm may be used to calculate a first similarity value between two samples in a positive sample pair and a second similarity value between two samples in a negative sample pair.
[0068] B13. Determine a first loss function corresponding to the selected sample set based on a difference between the first similarity value and the second similarity value;
[0069] The first loss function is:
[0070]
[0071] Among them, L is the first loss function corresponding to the selected sample set, S i is the second similarity value of the i-th negative sample pair corresponding to the selected sample set, S j is the first similarity value of the jth positive sample pair corresponding to the selected sample set, m is the total number of negative sample pairs in the selected sample set, and n is the total number of positive sample pairs in the selected sample set.
[0072] The outermost log(1+x) of the first loss function is a smooth function of the function max(x). When L decreases, S i Reduce, S j Increase, thereby achieving that the second similarity value of any negative sample pair corresponding to the sample set is less than the first similarity value of any positive sample pair, that is, the distance between the similarity values of the negative sample pair and the positive sample pair is widened.
[0073] B14, inputting the encoding vector into the classification network of the initial classification model to perform classification processing, obtaining the probability value of each text category of each sample in the selected sample set in the corresponding field, and determining the second loss function corresponding to the selected sample set based on the probability value;
[0074] The classification network is used to predict the probability value of each text category of the input sample in the corresponding field. The true value of the input sample in each text category can be determined according to the label of the input sample, and the second loss function corresponding to the selected sample set is determined based on the predicted probability value and the true value.
[0075] The second loss function is:
[0076]
[0077] Among them, Y is the second loss function corresponding to the selected sample, is the true value of the jth text category in the corresponding field of the i-th sample in the selected sample set, is the predicted probability value of the jth text category in the corresponding field for the i-th sample in the selected sample set, a is the total number of samples in the selected sample set, and b is the total number of text categories in the corresponding field of the selected sample set.
[0078] B15. Determine a target loss function corresponding to the selected sample set based on the first loss function and the second loss function.
[0079] In this embodiment, the first and second loss functions are summed to obtain the target loss function. In other embodiments, corresponding weights may be set for the first loss function and the second loss function, respectively, and the target loss function is determined by weighted summation.
[0080] S3, receiving the text to be classified, identifying the target field corresponding to the text to be classified, judging whether the sample set corresponding to the target field is evenly distributed in the text category, and if not, calculating the weight corresponding to each text category in the target field.
[0081] In this embodiment, steps S1-S2 are used to train a corresponding text classification model for each field. However, due to the different sample distributions in various fields, the classification accuracy of the text classification model is different. For fields where the sample distribution of each text category is balanced, the classification accuracy of the text classification model is higher. For fields where the distribution of text categories is uneven, the classification accuracy of the text classification model is not high.
[0082] In order to improve the accuracy of text classification in the field of uneven sample distribution, this embodiment sets a weight parameter for each text category in the field, and uses the Powell method to solve the optimal value of the weight parameter, and uses the product of the output value of the model in each text category and the value of the corresponding weight parameter as the target value of the input text in each text category, which solves the problem of inaccurate model text classification in this field. The Powell method is a method for solving parameter values by iterating step by step to find the optimal value of the parameter so that the loss reaches the minimum value. Using the Powell method to solve the parameter value is a prior art and will not be repeated here.
[0083] The step of identifying the target domain corresponding to the text to be classified includes the following steps C11-C13:
[0084] C11, performing word segmentation processing on the text to be classified to obtain a word set;
[0085] In this embodiment, the forward maximum matching method, the reverse maximum matching method or the minimum segmentation method may be used to perform word segmentation processing on the text to be classified.
[0086] C12, matching each word in the word set with the word library corresponding to each field, to obtain a matching word set corresponding to each field;
[0087] In this embodiment, a corresponding word library is pre-configured for each field.
[0088] C13. The field with the largest number of matching words in the matching word set is taken as the target field corresponding to the text to be classified.
[0089] For example, if the number of matching words in the matching word set corresponding to the medical field is the largest, then the medical field is taken as the target field corresponding to the text to be classified.
[0090] The step of judging whether the sample set corresponding to the target domain is evenly distributed in the text category includes the following steps D11-D14:
[0091] D11, counting the number of samples in each text category of the sample set corresponding to the target field;
[0092] Specifically, the number of samples in each text category can be counted according to the label information of the samples.
[0093] D12. Calculate the sample average value corresponding to the target field based on the sample quantity;
[0094] D13, calculating the variance of the sample set corresponding to the target field based on the sample quantity and the sample average;
[0095] D14. Determine whether the sample set corresponding to the target domain is evenly distributed in the text category according to the variance.
[0096] In this embodiment, if the variance is less than a predetermined variance threshold, the sample set corresponding to the target domain is evenly distributed in the text categories; if the variance is greater than the variance threshold, the sample set corresponding to the target domain is unevenly distributed in the text categories.
[0097] S4. Input the text to be classified into the text classification model corresponding to the target domain, obtain the predicted value of each text category of the text to be classified in the target domain, determine the target value of each text category of the text to be classified in the target domain based on the predicted value and the weight, and determine the target text category corresponding to the text to be classified based on the target value.
[0098] In this embodiment, the product of the predicted value of each text category and the corresponding weight is used as the target value of the text to be classified in each text category, and the text category with the largest target value is used as the target text category corresponding to the text to be classified.
[0099] After determining whether the sample set corresponding to the target domain is evenly distributed in the text categories, the method further includes:
[0100] If the judgment is yes, the text to be classified is input into the text classification model corresponding to the target field to obtain the predicted value of the text to be classified in each text category, and the target text category corresponding to the text to be classified is determined based on the predicted value.
[0101] It can be seen from the above embodiments that the text classification method proposed by the present invention firstly splits the sample set corresponding to each field into multiple positive sample pairs and multiple negative sample pairs; then, determines the target loss function according to the similarity difference between the positive sample pairs and the negative sample pairs, and trains the text classification model corresponding to each field by minimizing the target loss function; then, identifies the target field corresponding to the text to be classified, judges whether the distribution of the sample set corresponding to the target field on the text category is balanced, and if it is judged to be not, calculates the weight corresponding to each text category of the target field; finally, inputs the text to be classified into the text classification model corresponding to the target field, and determines the target text category corresponding to the text to be classified based on the predicted value output by the text classification model and the weight. The present invention determines the target loss function according to the similarity difference between the positive sample pairs and the negative sample pairs, so that the model can learn the similarity between the positive sample pairs and the dissimilarity between the negative sample pairs, deepens the interaction between the samples, and improves the classification accuracy of the model. Therefore, the present invention improves the accuracy of text classification.
[0102] like Figure 2 FIG. 1 is a schematic diagram of modules of a text classification device provided in one embodiment of the present invention.
[0103] The text classification device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the text classification device 100 can include a splitting module 110, a training module 120, a judgment module 130 and a classification module 140. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, which are stored in the memory of the electronic device.
[0104] In this embodiment, the functions of each module / unit are as follows:
[0105] The splitting module 110 is used to obtain a sample set carrying text category information corresponding to each field, and split each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information.
[0106] The step of splitting each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information includes the following steps A21-A22:
[0107] A21. Combine each sample in each sample set with samples of the same text category in pairs to obtain multiple positive sample pairs corresponding to each sample set;
[0108] A22. Combine each sample in each sample set with samples of different text categories in pairs to obtain multiple negative sample pairs corresponding to each sample set.
[0109] The training module 120 is used to determine a target loss function according to the similarity difference between the positive sample pair and the negative sample pair, and to train an initial classification model corresponding to each field by minimizing the target loss function to obtain a text classification model corresponding to each field.
[0110] Determining the target loss function according to the similarity difference between the positive sample pair and the negative sample pair includes the following steps B21-B25:
[0111] B21, selecting a sample set in a field, inputting the selected sample set into the encoding network of the initial classification model to perform encoding processing, and obtaining an encoding vector for each sample in the selected sample set;
[0112] B22. Calculate, based on the encoding vector, a first similarity value of each positive sample pair and a second similarity value of each negative sample pair corresponding to the selected sample set;
[0113] B23. Determine a first loss function corresponding to the selected sample set based on a difference between the first similarity value and the second similarity value;
[0114] The first loss function is:
[0115]
[0116] Among them, L is the first loss function corresponding to the selected sample set, S i is the second similarity value of the i-th negative sample pair corresponding to the selected sample set, S j is the first similarity value of the jth positive sample pair corresponding to the selected sample set, m is the total number of negative sample pairs in the selected sample set, and n is the total number of positive sample pairs in the selected sample set.
[0117] B24, inputting the encoding vector into the classification network of the initial classification model to perform classification processing, obtaining the probability value of each text category of each sample in the selected sample set in the corresponding field, and determining the second loss function corresponding to the selected sample set based on the probability value;
[0118] B25. Determine a target loss function corresponding to the selected sample set based on the first loss function and the second loss function.
[0119] The judgment module 130 is used to receive the text to be classified, identify the target field corresponding to the text to be classified, and judge whether the sample set corresponding to the target field is evenly distributed in the text category. If not, calculate the weight corresponding to each text category of the target field.
[0120] The step of identifying the target domain corresponding to the text to be classified includes the following steps C21-C23:
[0121] C21, performing word segmentation processing on the text to be classified to obtain a word set;
[0122] C22, matching each word in the word set with the word library corresponding to each field, to obtain a matching word set corresponding to each field;
[0123] C23. The field with the largest number of matching words in the matching word set is taken as the target field corresponding to the text to be classified.
[0124] The step of judging whether the sample set corresponding to the target domain is evenly distributed in the text category includes the following steps D21-D24:
[0125] D21, counting the number of samples in each text category of the sample set corresponding to the target field;
[0126] D22. Calculate the sample average value corresponding to the target field based on the sample quantity;
[0127] D23, calculating the variance of the sample set corresponding to the target field based on the sample quantity and the sample average;
[0128] D24. Determine whether the sample set corresponding to the target domain is evenly distributed in the text category according to the variance.
[0129] The classification module 140 is used to input the text to be classified into a text classification model corresponding to the target field, obtain a predicted value of each text category of the text to be classified in the target field, determine a target value of each text category of the text to be classified in the target field based on the predicted value and the weight, and determine a target text category corresponding to the text to be classified based on the target value.
[0130] After determining whether the sample set corresponding to the target domain is evenly distributed in the text categories, the classification module 140 is further used to:
[0131] If the judgment is yes, the text to be classified is input into the text classification model corresponding to the target field to obtain the predicted value of the text to be classified in each text category, and the target text category corresponding to the text to be classified is determined based on the predicted value.
[0132] like Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a text classification method provided by an embodiment of the present invention.
[0133] The electronic device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The electronic device 1 can be a computer, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets.
[0134] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected and communicated via a system bus. The memory 11 stores a text classification program 10, and the text classification program 10 can be executed by the processor 12. Figure 3 Only the electronic device 1 having components 11-13 and the text classification program 10 is shown. It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0135] The memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. In this embodiment, the readable storage medium of the memory 11 is generally used to store an operating system and various application software installed on the electronic device 1, such as storing the code of the text classification program 10 in an embodiment of the present invention, etc. In addition, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.
[0136] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the text classification program 10.
[0137] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a client (not shown in the figure).
[0138] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.
[0139] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0140] The text classification program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, and when executed in the processor 12, the above-mentioned text classification method can be implemented.
[0141] Specifically, the processor 12 can refer to the specific implementation method of the text classification program 10. Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.
[0142] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be non-volatile or non-volatile. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0143] The computer-readable storage medium stores a text classification program 10 , which can be executed by one or more processors to implement the above-mentioned text classification method.
[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0145] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0146] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0147] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0148] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.
[0149] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0150] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.
[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A text classification method, characterized in that: The method comprises: Obtain a sample set carrying text category information corresponding to each field, and split each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information; Determine a target loss function according to the similarity difference between the positive sample pair and the negative sample pair, and train an initial classification model corresponding to each field by minimizing the target loss function to obtain a text classification model corresponding to each field; Receive a text to be classified, identify a target field corresponding to the text to be classified, determine whether the sample set corresponding to the target field is evenly distributed in the text category, and if not, calculate the weight corresponding to each text category in the target field, and the weight corresponding to each text category is solved by the Powell method; Input the text to be classified into a text classification model corresponding to the target domain, obtain a predicted value of each text category of the text to be classified in the target domain, determine a target value of each text category of the text to be classified in the target domain based on the predicted value and the weight, and determine a target text category corresponding to the text to be classified based on the target value; Wherein, the method of determining the target loss function according to the similarity difference between the positive sample pair and the negative sample pair includes: selecting a sample set in a field, inputting the selected sample set into the encoding network of the initial classification model to perform encoding processing, and obtaining the encoding vector of each sample in the selected sample set; calculating the first similarity value of each positive sample pair and the second similarity value of each negative sample pair corresponding to the selected sample set based on the encoding vector; determining the first loss function corresponding to the selected sample set based on the difference between the first similarity value and the second similarity value; inputting the encoding vector into the classification network of the initial classification model to perform classification processing, and obtaining the probability value of each text category of each sample in the selected sample set in the corresponding field, and determining the second loss function corresponding to the selected sample set based on the probability value; determining the target loss function corresponding to the selected sample set based on the first loss function and the second loss function; The determining whether the sample set corresponding to the target field is evenly distributed on the text category includes: counting the number of samples of the sample set corresponding to the target field in each text category; calculating the sample average corresponding to the target field based on the sample number; calculating the variance of the sample set corresponding to the target field based on the sample number and the sample average; and determining whether the sample set corresponding to the target field is evenly distributed on the text category based on the variance; After determining whether the sample set corresponding to the target domain is evenly distributed in the text category, the method further includes: if the determination is yes, inputting the text to be classified into a text classification model corresponding to the target domain, obtaining a predicted value of the text to be classified in each text category, and determining the target text category corresponding to the text to be classified based on the predicted value.
2. The text classification method according to claim 1, characterized in that: The identifying the target domain corresponding to the text to be classified includes: Performing word segmentation processing on the text to be classified to obtain a word set; Match each word in the word set with the word library corresponding to each field to obtain a matching word set corresponding to each field; The field with the largest number of matching words in the matching word set is taken as the target field corresponding to the text to be classified.
3. The text classification method according to claim 1, characterized in that: The step of splitting each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information comprises: Combine each sample in each sample set with samples of the same text category in pairs to obtain multiple positive sample pairs corresponding to each sample set; Each sample in each sample set is combined with samples of different text categories in pairs to obtain multiple negative sample pairs corresponding to each sample set.
4. The text classification method according to claim 1, characterized in that: The first loss function is: Among them, L is the first loss function corresponding to the selected sample set, is the second similarity value of the i-th negative sample pair corresponding to the selected sample set, is the first similarity value of the jth positive sample pair corresponding to the selected sample set, m is the total number of negative sample pairs in the selected sample set, and n is the total number of positive sample pairs in the selected sample set.
5. A text classification device, used to implement the text classification method according to any one of claims 1 to 4, characterized in that: The device comprises: A splitting module is used to obtain a sample set carrying text category information corresponding to each field, and split each sample set into a plurality of positive sample pairs and a plurality of negative sample pairs according to the text category information; A training module, used for determining a target loss function according to the similarity difference between the positive sample pair and the negative sample pair, and training an initial classification model corresponding to each field by minimizing the target loss function to obtain a text classification model corresponding to each field; A judgment module is used to receive the text to be classified, identify the target field corresponding to the text to be classified, and judge whether the sample set corresponding to the target field is evenly distributed in the text category. If the judgment is not, calculate the weight corresponding to each text category in the target field; A classification module is used to input the text to be classified into a text classification model corresponding to the target field, obtain a predicted value of each text category of the text to be classified in the target field, determine a target value of each text category of the text to be classified in the target field based on the predicted value and the weight, and determine a target text category corresponding to the text to be classified based on the target value.
6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a text classification program executable by the at least one processor, and the text classification program is executed by the at least one processor so that the at least one processor can perform the text classification method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a text classification program, which can be executed by one or more processors to implement the text classification method according to any one of claims 1 to 4.
