Text processing method and device, equipment and medium
By combining multiple pre-trained language models with the number threshold to determine the statement type, the problem that the NLP model cannot accurately identify statement categories and types is solved, and the accuracy of classification is improved.
Patent Information
- Application Number
- CN202411145103.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-15
AI Technical Summary
Existing natural language processing (NLP) models cannot accurately identify the categories of statements in the original text, and using a single model to judge can easily lead to the absence of a certain type of statement.
Multiple pre-trained language models are used to input the original text into these models separately, combining the output results and the number threshold of each model, the type of statement is determined, and classification extraction and generalization is performed based on the statement type.
It improves the accuracy of statement classification, avoids the problem of missing types caused by using a single model, and enhances the accurate identification of statement categories.
Smart Images

Figure CN118964605A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and specifically to a text processing method, device, equipment and medium. Background Art
[0002] After a certain device obtains the original text, other devices may also need some sentence information in the original text. Therefore, the sentence information required by other devices can be classified and extracted from the original text and then sent to other devices that need the sentence information.
[0003] Currently, the method of extracting some information from the original text is usually to use a single natural language processing (NLP) model to classify and extract the sentence information in the original text. However, the ordinary NLP model cannot accurately identify the categories corresponding to some sentences. In addition, if there are sentences in the original text corresponding to multiple types, using a single model for judgment may easily lead to the missing of sentences corresponding to a certain type. Summary of the invention
[0004] The embodiments of the present application provide a text processing method, apparatus, device and medium for solving the problem that the common NLP model cannot accurately identify the categories corresponding to some sentences, and the use of a single model for judgment easily leads to the missing of sentences corresponding to a certain type.
[0005] In a first aspect, an embodiment of the present application provides a text processing method, comprising:
[0006] Obtaining a first group of multiple trained language models; wherein the first language model is used to determine whether an input sentence is a first type of sentence; the first language model is any one of the multiple language models;
[0007] Inputting the original text into the multiple language models respectively, and obtaining the sentence type of each sentence in the original text output by each language model in the multiple language models;
[0008] Determining the text type of the original text according to the sentence type of each sentence;
[0009] Based on the sentence type corresponding to each sentence, multiple sentences contained in the original text are classified and extracted to obtain a classified extracted text;
[0010] The classified extracted text is summarized based on the semantic information of the classified extracted text to obtain a target text.
[0011] In a possible implementation, the inputting the original text into the multiple language models respectively to obtain the sentence type of each sentence in the original text output by each language model in the multiple language models includes:
[0012] Determining a first number of language models among the plurality of language models that classify a first sentence as a second type; the first sentence being any one of the plurality of sentences; and the second type being any one of a plurality of sentence types;
[0013] If the first quantity is greater than or equal to the quantity threshold corresponding to the second type, determining that the type of the first statement is the second type;
[0014] If the first number is smaller than a number threshold corresponding to the second type, the first statement is removed from the statements included in the second type.
[0015] In a possible implementation, the classifying and extracting the multiple sentences contained in the original text based on the sentence type corresponding to each sentence includes:
[0016] Obtain a second set of multiple language models that have been trained;
[0017] Inputting the original text into a plurality of language models of the second group respectively, and obtaining a target sentence judgment result of each sentence in the original text output by each language model in the plurality of language models; the target sentence judgment result is used to indicate whether the corresponding sentence contains the set content;
[0018] Based on the sentence type corresponding to each sentence and the target sentence judgment result, multiple sentences contained in the original text are classified and extracted.
[0019] In a possible implementation, the type of language models included in the second group of multiple trained language models is the same as the type of language models included in the first group of multiple trained language models.
[0020] In a possible implementation, the classifying and extracting the multiple sentences included in the original text based on the sentence type corresponding to each sentence and the target sentence judgment result includes:
[0021] If the target sentence judgment result for the first sentence is less than the target threshold, the first sentence is removed from the sentence type corresponding to the first sentence;
[0022] If the target sentence judgment result of each sentence in the original text is greater than or equal to the target threshold, then each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type according to the sentence type corresponding to each sentence.
[0023] In a possible implementation, obtaining a first group of multiple trained language models includes:
[0024] Determine a plurality of large language models of different types, and input a training set into each of the plurality of large language models; the training set includes a plurality of original texts and a sentence type of each sentence in the plurality of original texts; after receiving the training set, each language model compares a model output result with a result of the training set, determines a corresponding loss value, and repeatedly adjusts training parameters until the loss value reaches a set value;
[0025] Wait until each of the large language models completes training using the training set to obtain the first group of trained multiple language models.
[0026] In a possible implementation, the classifying and extracting the multiple sentences contained in the original text based on the sentence type corresponding to each sentence includes:
[0027] Based on the statement type corresponding to each statement, determine multiple statements corresponding to each statement type, and retain one of the same statements among the multiple statements corresponding to each statement type;
[0028] According to the multiple sentences corresponding to each sentence type, each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type.
[0029] In a possible implementation, after classifying and extracting the multiple sentences contained in the original text based on the sentence type corresponding to each sentence, the method further includes:
[0030] If an information viewing request is received from any request end, the multiple sentences extracted by classification are sent to the request end.
[0031] In a possible implementation manner, determining the text type of the original text according to the sentence type of each sentence includes:
[0032] The number of sentences corresponding to each sentence type is determined, and the text type of the original text is determined according to the sentence type corresponding to the maximum value among the number of sentences.
[0033] In a possible embodiment, summarizing the classified extracted text based on the semantic information of the classified extracted text to obtain the target text includes:
[0034] The classified extracted text is input into a target language model to obtain a target text; the target language model is used to identify important information of the classified extracted text, determine semantic information of the classified extracted text based on the important information, and combine the semantic information to obtain the target text; the important information includes a description subject in the classified extracted text and partial information related to the description subject.
[0035] In a second aspect, the present application provides a text processing device, the device comprising:
[0036] A model acquisition unit, used to acquire a first group of multiple language models that have been trained; wherein the first language model is used to determine whether an input sentence is a first type of sentence; and the first language model is any one of the multiple language models;
[0037] a type determination unit, configured to input the original text into the multiple language models respectively, and obtain the sentence type of each sentence in the original text output by each language model in the multiple language models;
[0038] Determining the text type of the original text according to the sentence type of each sentence;
[0039] A classification extraction unit, configured to classify and extract multiple sentences contained in the original text based on the sentence type corresponding to each sentence, to obtain a classified extracted text;
[0040] The semantic summarization unit is used to summarize the classified extracted text based on the semantic information of the classified extracted text to obtain a target text.
[0041] In a possible implementation manner, the type determination unit is specifically configured to:
[0042] Determining a first number of language models among the plurality of language models that classify a first sentence as a second type; the first sentence being any one of the plurality of sentences; and the second type being any one of a plurality of sentence types;
[0043] If the first quantity is greater than or equal to the quantity threshold corresponding to the second type, determining that the type of the first statement is the second type;
[0044] If the first number is smaller than a number threshold corresponding to the second type, the first statement is removed from the statements included in the second type.
[0045] In a possible implementation, the classification and extraction unit is specifically used for:
[0046] Obtain a second set of multiple language models that have been trained;
[0047] Inputting the original text into a plurality of language models of the second group respectively, and obtaining a target sentence judgment result of each sentence in the original text output by each language model in the plurality of language models; the target sentence judgment result is used to indicate whether the corresponding sentence contains the set content;
[0048] Based on the sentence type corresponding to each sentence and the target sentence judgment result, multiple sentences contained in the original text are classified and extracted.
[0049] In a possible implementation, the classification and extraction unit is specifically used for:
[0050] If the target sentence judgment result for the first sentence is less than the target threshold, the first sentence is removed from the sentence type corresponding to the first sentence;
[0051] If the target sentence judgment result of each sentence in the original text is greater than or equal to the target threshold, then each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type according to the sentence type corresponding to each sentence.
[0052] In a possible implementation, the model acquisition unit is specifically used to determine a plurality of large language models of different types, and input a training set into each of the plurality of large language models; the training set includes a plurality of original texts and a sentence type of each sentence in the plurality of original texts; after receiving the training set, each language model compares a model output result with a result of the training set, determines a corresponding loss value, and repeatedly adjusts the training parameters until the loss value reaches a set value;
[0053] Wait until each of the large language models completes training using the training set to obtain the first group of trained multiple language models.
[0054] In a possible implementation, the classification and extraction unit is specifically used for:
[0055] Based on the statement type corresponding to each statement, determine multiple statements corresponding to each statement type, and retain one of the same statements among the multiple statements corresponding to each statement type;
[0056] According to the multiple sentences corresponding to each sentence type, each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type.
[0057] In a third aspect, the present application provides an electronic device, including:
[0058] A memory for storing program instructions;
[0059] The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods in the first aspect according to the obtained program instructions.
[0060] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods described in the first aspect.
[0061] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect.
[0062] The beneficial effects of this application are as follows:
[0063] The embodiments of the present application provide a text processing method, apparatus, device and medium. In the embodiments of the present application, a first group of multiple trained language models can be obtained, each language model is used to determine a sentence type, and the original text is input into multiple language models respectively, and the sentence type of each sentence in the original text output by each language model can be obtained, and based on the sentence type corresponding to each sentence, the multiple sentences contained in the original text are classified and extracted. By preparing multiple pre-trained language models, each pre-trained language model only needs to determine whether the sentence belongs to a type, avoiding the situation where the type is missing when using a single model, and changing the model trained by NLP to a large language model that has been trained, the category of each sentence can be accurately identified, improving the accuracy of classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] Figure 1 A schematic diagram of an application scenario of a text processing method provided in an embodiment of the present application;
[0065] Figure 2 A flowchart of a text processing method provided in an embodiment of the present application;
[0066] Figure 3 A flowchart of a specific text processing method provided in an embodiment of the present application;
[0067] Figure 4 A structural diagram of a classification extraction device provided in an embodiment of the present application;
[0068] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0069] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.
[0070] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.
[0071] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0072] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.
[0073] Before introducing the text processing method provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.
[0074] After a certain device obtains the original text, other devices may also need some sentence information in the original text. Therefore, the sentence information required by other devices can be classified and extracted from the original text and then sent to other devices that need the sentence information.
[0075] In the related art, in order to solve the above problems, the method of extracting sentence information from the original text is usually to use a single NLP model. The trained NLP model can classify and extract the sentence information in the original text.
[0076] Although this method can filter sentence information from the original text, ordinary NLP models cannot accurately identify the categories corresponding to some sentences, nor can they accurately extract the required information from the relevant information. Errors often occur, resulting in other devices or other applications being unable to directly use the extracted sentence information. Moreover, if there are sentences in the original text that correspond to multiple types, using a single model for judgment may easily lead to the missing of sentences corresponding to a certain type.
[0077] For example, if the original text contains five sentences, namely sentence 1, sentence 2, sentence 3, sentence 4 and sentence 5, among which sentence 2, sentence 3, sentence 4 and sentence 5 are sentences that need to be classified, the ordinary NLP model may not be able to associate sentence 2 with the sentences that need to be classified, and can only filter out sentence 3, sentence 4 and sentence 5 as sentences that need to be classified, resulting in the lack of sentences that need to be classified and affecting subsequent steps; if sentence 2, sentence 3 and sentence 4 belong to type 1, and sentence 3, sentence 4 and sentence 5 belong to type 2, a single ordinary NLP model may not be able to classify sentence 4 as both type 1 and type 2, resulting in the lack of sentences corresponding to a certain type.
[0078] In view of this, the embodiments of the present application provide a text processing method, device, equipment and medium. In the embodiments of the present application, a first group of multiple language models that have been trained can be obtained, each language model is used to determine a sentence type, and the original text is input into multiple language models respectively, and the sentence type of each sentence in the original text output by each language model can be obtained, and based on the sentence type corresponding to each sentence, the multiple sentences contained in the original text are classified and extracted. By preparing multiple pre-trained language models, each pre-trained language model only needs to determine whether the sentence belongs to a type, avoiding the situation where the type is missing when using a single model, and changing the model trained using NLP to a large language model that has been trained, the category of each sentence can be accurately identified, improving the accuracy of classification.
[0079] The following first briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.
[0080] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes four terminal devices, namely terminal device 100, terminal device 200, terminal device 300 and terminal device 400. Among them, the text processing method provided in the embodiment of the present application can be applied to terminal device 200, the original text can be sent from terminal device 100 to terminal device 200, and terminal device 200 can obtain the target sentence corresponding to each classification after performing relevant operations on the original text. After receiving a request for the target sentence corresponding to each classification from any terminal device in terminal device 300 or terminal device 400, terminal device 200 can send the target sentence corresponding to each classification to the corresponding terminal device.
[0081] It should be noted that the text processing method provided in the embodiment of the present application can be deployed in a web page or an application. Figure 1 The terminal device 100 and the terminal device 200 in the embodiment may be the same terminal device, that is, the original text may be directly input into an application or webpage in the same terminal device, so that relevant operations may be performed on the original text, which is not limited in the present application; and the terminal device 300 and the terminal device 400 serve as receiving ends of the target sentence corresponding to each classification, and fewer or more devices may be used to receive the target sentence through a transmission method such as a local area network, the Internet, or a bus, etc., and the number of receiving devices and the transmission method of information are not limited in the present application; Figure 1 The terminal device may also be a server, or any other electronic device that can execute the text processing method provided in the embodiment of the present application, and the present application is not limited thereto.
[0082] The text processing method provided by the exemplary embodiment of the present application is described below in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.
[0083] Figure 2 A flowchart of a text processing method provided by an embodiment of the present application is shown. Figure 2 As shown, the method may include the following steps:
[0084] Step S201, obtaining a first group of multiple trained language models.
[0085] In a possible embodiment, multiple large language models of different types may be determined first, such as a chatglm model, a baichuan model, or a T5 model, etc., which are not limited in the present application. For example, two chatglm models may be determined to be model 1 and model 2, two baichuan models may be determined to be model 3 and model 4, and two T5 models may be determined to be model 5 and model 6.
[0086] After determining a plurality of large language models of different types, the training set may be input into each of the plurality of large language models so that each large language model is trained on the training set. Among them, the training set may include multiple original texts, such as 8 or 10 or other numbers of original texts, and the training set also includes the type of sentences in each original text. For example, one of the original texts contains 6 sentences, namely, sentence A, sentence B, sentence C, sentence D, sentence E and sentence F. These 6 sentences contain 3 types of sentences, namely, type 1, type 2 and type 3. If sentence B, sentence C and sentence D belong to type 1, sentence D and sentence E belong to type 2, and sentence F belongs to type 3, then there are two training methods, namely, method 1, inputting each original text and the type of sentence in each original text into each large language model; method 2, each large language model is divided into three parts for training for different types, such as 9 chat models, using any three models to train for the training set of type 1, using another three different models to train for the training set of type 2, and using the last three models to train for the training set of type 3. The same operation can be performed on other types of large language models. It should be noted that the following description will explain the above two methods separately.
[0087] In a possible embodiment, after each large language model receives the training set, the model output result can be compared with the result in the training set to determine the corresponding loss value. The more the output result differs from the training set, the greater the loss value; the smaller the difference between the output result and the training set, the smaller the loss value. Each large language model can be repeatedly trained and the parameters can be adjusted until the loss value reaches the set value. When the loss value of any model reaches the set value, the model training can be considered to be completed. When each large language model has completed training, a first group of multiple trained language models can be obtained.
[0088] Step S202: input the original text into a plurality of language models respectively, and obtain the sentence type of each sentence in the original text output by each of the plurality of language models.
[0089] In a possible embodiment, after obtaining a first group of multiple trained language models, the original text to be classified can be input into the first group of multiple trained language models respectively, so as to obtain the sentence type of each sentence in the original text output by each of the multiple language models.
[0090] In one possible embodiment, if the training method of each language model in the first group of multiple language models that have been trained is the first one in step S201, that is, each language model can judge multiple types, then after the original text to be classified is input into the first group of multiple language models that have been trained, each language model can output the type of each sentence contained in the original text. For example, the result output by model 1 for the types of the 6 sentences contained in the original text is that sentence A has no type, sentence B belongs to type 1, sentence C belongs to type 1, sentence D belongs to type 1 and type 2, sentence E has no type, and sentence F belongs to type 2; the result output by model 2 for the type of each sentence contained in the original text is that sentence A has no type, sentence B has no type, sentence C belongs to type 1, sentence D belongs to type 1 and type 2, sentence E belongs to type 2, and sentence F belongs to type 2; the result output by model 3 for the type of each sentence contained in the original text is that sentence A has no type, sentence B has no type, sentence C belongs to type 1, sentence D belongs to type 1 and type 2, sentence E has no type, and sentence F belongs to type 2.
[0091] For the above possible embodiments, the following method can be used to analyze the results of the above examples, determine the first number of language models that classify any sentence as any type in multiple language models, and if the first number is greater than or equal to the number threshold of the corresponding type, determine that the sentence belongs to the corresponding type; if the first number is less than the number threshold of the corresponding type, determine that the sentence does not belong to the corresponding type. For example, for type 1, if 1 / 3 of the language models believe that any sentence belongs to type 1, it can be determined that the sentence belongs to type 1. For example, if one or two or three of the three language models believe that sentence B belongs to type 1, it can be determined that sentence B belongs to type 1, and if no language model believes that sentence A belongs to type 1, it can be determined that sentence A does not belong to type 1; for type 2, if 2 / 3 of the language models believe that any sentence belongs to type 2, it can be determined that the sentence belongs to type 2. If two or three of the three language models believe that sentence D belongs to type 2, it can be determined that sentence D belongs to type 2, and if only one or no language model of the three language models believes that sentence D belongs to type 2, it can be determined that sentence D does not belong to type 2. It should be noted that the quantity threshold 1 / 3 corresponding to type 1 and the quantity threshold 2 / 3 corresponding to type 2 can also be any other quantity threshold, such as 1 / 4 or 2 / 5, etc., and can be set differently according to the number of language models or the application scenarios, and this application does not limit this.
[0092] In another possible embodiment, if the training method of each language model in the first group of multiple trained language models is the second method in step S201, that is, each language model can judge a single type, then after the original text to be classified is input into the first group of multiple trained language models respectively, each language model can output the type to which each sentence contained in the original text belongs. For example, there are nine language models in total, among which any three language models can be used to judge whether multiple sentences belong to type 1, another three language models can be used to judge whether multiple sentences belong to type 2, and the last three language models can be used to judge whether multiple sentences belong to type 3.
[0093] For the above possible embodiments, the results of the above examples can also be analyzed in the following manner to determine the first number of language models that classify any sentence into any type among multiple language models. If the first number is greater than or equal to the quantity threshold of the corresponding type, it is determined that the sentence belongs to the corresponding type; if the first number is less than the quantity threshold of the corresponding type, it is determined that the sentence does not belong to the corresponding type. For example, for type 1, if 1 / 3 of the language models believe that any sentence belongs to type 1, then it can be determined that the sentence belongs to type 1. If models 1, 2, and 3 can all be used to determine whether any sentence belongs to type 1, and any one, two, or three models among models 1, 2, and 3 determine that sentence B belongs to type 1, then it can be determined that sentence B belongs to type 1. If no model among models 1, 2, and 3 determines that sentence A belongs to type 1, then it can be determined that sentence A does not belong to type 1. For type 2, if 2 / 3 of the language models believe that any sentence belongs to type 2, then it can be determined that the sentence belongs to type 2. If models 4, 5, and 6 can all be used to determine whether any sentence belongs to type 2, and any two or three models among models 4, 5, and 6 determine that sentence D belongs to type 2, then it can be determined that sentence D belongs to type 2. If only one or no language model among models 4, 5, and 6 believes that sentence D belongs to type 2, then it can be determined that sentence D does not belong to type 2. It should also be noted that the quantity threshold 1 / 3 corresponding to type 1 and the quantity threshold 2 / 3 corresponding to type 2 can also be any other quantity threshold, such as 1 / 4 or 2 / 5, etc., and can be set differently according to the number of language models or the application scenarios, and this application does not limit this.
[0094] By using multiple language models and setting a quantity threshold, we can effectively avoid the situation where using a single model for judgment may easily lead to the missing of sentences corresponding to a certain type. The quantity threshold can also avoid the impact of misjudgment of a small number of models among multiple models on the results.
[0095] Step S203: determining the text type of the original text according to the sentence type of each sentence.
[0096] In a possible embodiment, the number of statements corresponding to each statement type can be determined, and the text type of the original text can be determined according to the statement type corresponding to the maximum value in the number of statements. For example, there are three statement types, namely statement type A, statement type B, and statement type C. If statement type A contains 5 statements, statement type B contains 4 statements, and statement type C contains 6 statements, then the text type corresponding to the original text can be determined according to statement type C. If statement type A is the advantage of thing 1, statement type B is a neutral evaluation of thing 1, and statement type C is the determination of thing 1, then it can be determined that in the original text, the disadvantages of thing 1 are more than the neutral evaluation and advantages, so the text type of the original text can be obtained as: thing 1 contains more disadvantages.
[0097] Step S204: Based on the sentence type corresponding to each sentence, multiple sentences included in the original text are classified and extracted to obtain a classified extracted text.
[0098] In a possible embodiment, when executing step S204, a second group of multiple language models that have been trained may be prepared first. It should be noted that the types of language models included in the second group of multiple language models that have been trained are the same as the types of language models included in the first group of multiple language models that have been trained. For example, the first group of multiple language models that have been trained include models 1, 2, and 3 of the chatglm type, models 4, 5, and 6 of the baichuan type, and models 7, 8, and 9 of the T5 type. Then the types of language models included in the second group of multiple language models that have been trained should also be chatglm, baichuan, and T5. The number of the second group of multiple language models that have been trained should be at least the same as the number of types of language models included in the first group of multiple language models that have been trained, and it can be guaranteed that there is at least one language model of each type. The specific number is not limited in this application.
[0099] In a possible embodiment, the training set of the multiple large language models in the second group may include multiple original texts, such as 8 or 10 or other numbers of original texts, and the training set also includes whether the sentences in each original text are target sentences. For example, type 1 can screen sentences containing "indicator A rises". If there is a sentence "the user points to the rising mark", this has nothing to do with type 1, that is, the sentence does not belong to the target sentence, but the multiple language models trained in the first group may classify this sentence as type 1, so the sentence needs to be eliminated. It should be noted that the training method of the multiple language models in the second group is the same as the training method of the multiple language models in the first group, and this application will not go into details here.
[0100] In a possible embodiment, multiple classified sentences can be input into each language model of the second group of trained multiple language models, and the target sentence judgment results for the multiple classified sentences can be obtained. For example, the classified sentences are sentence B, sentence C, sentence D, sentence E, and sentence F, then five sentences from sentence B to sentence F can be input into each language model of the second group of trained multiple language models, and each language model can output the target sentence judgment result for each sentence. If for sentence B, 8 out of 10 language models judge that sentence B is the target sentence, then the target sentence judgment result output by each of the 8 language models is 1, and 2 out of 10 language models judge that sentence B is not the target sentence, then the target sentence judgment result output by each of the 2 language models is 0. On the whole, the target sentence judgment result for sentence B is 8.
[0101] In a possible embodiment, the target sentence judgment result of any sentence can be compared with the target threshold. If the target sentence judgment result of any sentence is less than the target threshold, the sentence is eliminated from the corresponding type; if the target sentence judgment result of any sentence is greater than or equal to the target threshold, it is determined that the type corresponding to the sentence includes the sentence. For example, if the target sentence judgment result of sentence B is 8 and the target threshold is 8, then it is determined that the type corresponding to sentence B includes sentence B; and if the target sentence judgment result of sentence B is 6 and the target threshold is 8, then sentence B is eliminated from the sentence type corresponding to sentence B. It should be noted that the target threshold can be changed according to the number of language models in the second group of multiple trained language models, and can also be adjusted according to different needs. The target threshold can be 6 or 10, and this application is not limited here.
[0102] In a possible embodiment, after the second group of trained multiple language models completes the target sentence judgment for multiple classified sentences, the multiple sentences contained in the original text can be classified and extracted based on the sentence type corresponding to each sentence. Since each model in the first group of trained multiple language models will judge the same sentence separately, multiple identical sentences may be included in one sentence type. Based on the sentence type corresponding to each sentence, multiple sentences corresponding to each sentence type can be determined, and only one of the identical sentences in the multiple sentences corresponding to each sentence type can be retained. After deleting the repeated sentences, the multiple sentences contained in the original text can be classified and extracted based on the multiple sentences corresponding to each sentence type. Specifically, according to the multiple sentences corresponding to each sentence type, each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type. For example, the result that type 1 includes sentence B, sentence C, and sentence D, type 2 includes sentence D and sentence E, and type 3 includes sentence F can be obtained.
[0103] Step S205 , summarizing the classified extracted text based on the semantic information of the classified extracted text to obtain the target text.
[0104] In a possible embodiment, the classified extracted text obtained in step S204 can be input into the target language model to obtain the target text. Similarly, the target language model can also be a chatglm model, a baichuan model or a T5 model, etc., which is not limited in this application. The target language model can identify the important information of the classified extracted text, that is, the description subject in the classified extracted text and the partial information related to the description subject, wherein the partial information can include the trend of the description subject and the state of the description subject, while the non-important information can be some limiting conditions or some adjectives. For example, there is a sentence "Thing A has shown a certain downward trend every year in the past ten years, but in the future, thing A will achieve a reversal of the downward trend at a faster speed and will show a steady upward trend", then the target language model can extract the important information "Thing A", "Decline" and "Rise", and then the semantic information of the classified extracted text "Past Decline" and "Future Rise" can be determined according to the extracted important information, and finally the semantic information can be connected to obtain the target text, "Thing A fell in the past and rose in the future".
[0105] In a possible embodiment, after obtaining the target text, if an information viewing request is received from any request end, the target text can be sent to the request end corresponding to the information viewing request. For example, terminal device C sends an information viewing request to the device that executes the text processing method of the present application. After receiving the information viewing request, the device that executes the text processing method of the present application can send the target text to terminal device C.
[0106] The embodiments of the present application provide a text processing method, apparatus, device and medium. In the embodiments of the present application, a first group of multiple trained language models can be obtained, each language model is used to determine a sentence type, and the original text is input into multiple language models respectively, and the sentence type of each sentence in the original text output by each language model can be obtained, and based on the sentence type corresponding to each sentence, the multiple sentences contained in the original text are classified and extracted. By preparing multiple pre-trained language models, each pre-trained language model only needs to determine whether the sentence belongs to a type, avoiding the situation where the type is missing when using a single model, and changing the model trained by NLP to a large language model that has been trained, the category of each sentence can be accurately identified, improving the accuracy of classification.
[0107] Based on the same inventive concept, the present application embodiment provides a flowchart of a specific text processing method, such as Figure 3 As shown, the steps include:
[0108] Step S301 : determining a plurality of large language models of different types, and inputting a training set into each of the plurality of large language models.
[0109] Step S302, waiting until each large language model completes training using the training set, to obtain a first group of multiple trained language models.
[0110] Step S303: determining a first number of language models that classify the first sentence as a second type among the multiple language models.
[0111] In a possible embodiment, the first statement is any one of a plurality of statements, and the second type is any one of a plurality of statement types.
[0112] Step S304, determining whether the first quantity is greater than or equal to the quantity threshold corresponding to the second type, if so, executing step S305; if not, executing step S306.
[0113] Step S305: determine that the type of the first statement is the second type.
[0114] Step S306: remove the first statement from the statements included in the second type, and return to execute step S304.
[0115] Step S307, obtaining a second group of multiple trained language models.
[0116] Step S308, inputting the original text into the plurality of language models of the second group respectively, and obtaining the target sentence judgment result of each sentence in the original text output by each of the plurality of language models.
[0117] Step S309, determining whether the target sentence judgment result for the first sentence is less than the target threshold, if so, executing step S310; if not, executing step S311.
[0118] Step S310, remove the first statement from the statement type corresponding to the first statement, and return to execute step S309.
[0119] Step S311 , determining the text type of the original text based on the sentence type corresponding to each sentence, and retaining only one sentence of the same sentence among multiple sentences corresponding to each sentence type.
[0120] Step S312, according to the multiple sentences corresponding to each sentence type, each sentence in the original text is saved in a first file in correspondence with the corresponding sentence type to obtain a classified extracted text.
[0121] Step S313, summarizing the classified extracted text based on the semantic information of the classified extracted text to obtain the target text, and if an information viewing request sent by any request end is received, the target text is sent to the corresponding request end.
[0122] Based on the same inventive concept, the present application embodiment provides a text processing device, please refer to Figure 4 , the text processing device 400 comprises:
[0123] A model acquisition unit 401 is used to acquire a first group of multiple language models that have been trained; wherein the first language model is used to determine whether an input sentence is a first type of sentence; and the first language model is any one of the multiple language models;
[0124] A type determination unit 402 is used to input the original text into the multiple language models respectively, and obtain the sentence type of each sentence in the original text output by each language model in the multiple language models;
[0125] Determining the text type of the original text according to the sentence type of each sentence;
[0126] A classification extraction unit 403, configured to classify and extract multiple sentences contained in the original text based on the sentence type corresponding to each sentence, to obtain a classified extracted text;
[0127] The semantic summarization unit 404 is used to summarize the classified extracted text based on the semantic information of the classified extracted text to obtain a target text.
[0128] In a possible implementation manner, the type determination unit 402 is specifically configured to:
[0129] Determining a first number of language models among the plurality of language models that classify a first sentence as a second type; the first sentence being any one of the plurality of sentences; and the second type being any one of a plurality of sentence types;
[0130] If the first quantity is greater than or equal to the quantity threshold corresponding to the second type, determining that the type of the first statement is the second type;
[0131] If the first number is smaller than a number threshold corresponding to the second type, the first statement is removed from the statements included in the second type.
[0132] In a possible implementation, the classification extraction unit 403 is specifically configured to:
[0133] Obtain a second set of multiple language models that have been trained;
[0134] Inputting the original text into a plurality of language models of the second group respectively, and obtaining a target sentence judgment result of each sentence in the original text output by each language model in the plurality of language models; the target sentence judgment result is used to indicate whether the corresponding sentence contains the set content;
[0135] Based on the sentence type corresponding to each sentence and the target sentence judgment result, multiple sentences contained in the original text are classified and extracted.
[0136] In a possible implementation, the classification extraction unit 403 is specifically configured to:
[0137] If the target sentence judgment result for the first sentence is less than the target threshold, the first sentence is removed from the sentence type corresponding to the first sentence;
[0138] If the target sentence judgment result of each sentence in the original text is greater than or equal to the target threshold, then each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type according to the sentence type corresponding to each sentence.
[0139] In a possible implementation, the model acquisition unit 401 is specifically used to determine a plurality of large language models of different types, and input a training set into each of the plurality of large language models; the training set includes a plurality of original texts and a sentence type of each sentence in the plurality of original texts; after receiving the training set, each language model compares a model output result with a result of the training set, determines a corresponding loss value, and repeatedly adjusts the training parameters until the loss value reaches a set value;
[0140] Wait until each of the large language models completes training using the training set to obtain the first group of trained multiple language models.
[0141] In a possible implementation, the classification extraction unit 403 is specifically configured to:
[0142] Based on the statement type corresponding to each statement, determine multiple statements corresponding to each statement type, and retain one of the same statements among the multiple statements corresponding to each statement type;
[0143] According to the multiple sentences corresponding to each sentence type, each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type.
[0144] In a possible implementation manner, the type determination unit 402 is specifically configured to:
[0145] The number of sentences corresponding to each sentence type is determined, and the text type of the original text is determined according to the sentence type corresponding to the maximum value among the number of sentences.
[0146] In a possible implementation, the semantic summarization unit 404 is specifically used to:
[0147] The classified extracted text is input into a target language model to obtain a target text; the target language model is used to identify important information of the classified extracted text, determine semantic information of the classified extracted text based on the important information, and combine the semantic information to obtain the target text; the important information includes a description subject in the classified extracted text and partial information related to the description subject.
[0148] Based on the same inventive concept, the present application embodiment provides an electronic device, which can realize the functions of the classification extraction device discussed above, please refer to Figure 5 , the device includes a memory 501, a processor 502 and a bus 503.
[0149] The memory 501 is used to store computer programs executed by the processor 502. The memory 501 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.
[0150] The memory 501 may be a volatile memory, such as a random-access memory (RAM); the memory 501 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 501 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 501 may be a combination of the above memories.
[0151] The processor 502 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 502 is used to implement the resource checking method in the above embodiment when calling the computer program stored in the memory 501 .
[0152] The specific connection medium between the memory 501 and the processor 502 is not limited in the embodiment of the present application. Figure 5 In the embodiment, the memory 501 and the processor 502 are connected via a bus 503. The bus 503 is connected to the processor 502 via a bus 503. Figure 5 The connections between other components are shown in bold lines, and are not intended to be limiting. Bus 503 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0153] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium, and the computer program product includes: computer program code, when the computer program code is run on a computer, the computer executes any of the text processing methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the text processing method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.
[0154] Based on the same inventive concept, the embodiment of the present application further provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the text processing methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the text processing method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.
[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0159] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A text processing method, It is characterized in that include: Obtain a first group of multiple trained language models; wherein the first language model is used to determine whether an input sentence is a first type of sentence; the first language model is any one of the multiple language models; Inputting the original text into the multiple language models respectively, and obtaining the sentence type of each sentence in the original text output by each language model in the multiple language models; Determining the text type of the original text according to the sentence type of each sentence; Based on the sentence type corresponding to each sentence, multiple sentences contained in the original text are classified and extracted to obtain a classified extracted text; The classified extracted text is summarized based on the semantic information of the classified extracted text to obtain a target text.
2. The method according to claim 1, It is characterized in that The step of inputting the original text into the multiple language models respectively to obtain the sentence type of each sentence in the original text output by each language model in the multiple language models includes: Determining a first number of language models among the plurality of language models that classify a first sentence as a second type; the first sentence is any one of the plurality of sentences; the second type is any one of a plurality of sentence types; If the first quantity is greater than or equal to the quantity threshold corresponding to the second type, determining that the type of the first statement is the second type; If the first number is smaller than a number threshold corresponding to the second type, the first statement is removed from the statements included in the second type.
3. The method according to claim 1, It is characterized in that The classifying and extracting the multiple sentences contained in the original text based on the sentence type corresponding to each sentence includes: Obtain a second set of multiple language models that have been trained; Inputting the original text into a plurality of language models of the second group respectively, and obtaining a target sentence judgment result of each sentence in the original text output by each language model in the plurality of language models; the target sentence judgment result is used to indicate whether the corresponding sentence contains the set content; Based on the sentence type corresponding to each sentence and the target sentence judgment result, multiple sentences contained in the original text are classified and extracted.
4. The method according to claim 3, It is characterized in that The types of language models included in the second set of multiple trained language models are the same as the types of language models included in the first set of multiple trained language models.
5. The method according to any one of claims 3 to 4, It is characterized in that The step of classifying and extracting the multiple sentences contained in the original text based on the sentence type corresponding to each sentence and the target sentence judgment result includes: If the target sentence judgment result for the first sentence is less than the target threshold, the first sentence is removed from the sentence type corresponding to the first sentence; If the target sentence judgment result of each sentence in the original text is greater than or equal to the target threshold, then each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type according to the sentence type corresponding to each sentence.
6. The method according to claim 1, It is characterized in that The obtaining of a first set of multiple trained language models includes: Determine a plurality of large language models of different types, and input a training set into each of the plurality of large language models; the training set includes a plurality of original texts and a sentence type of each sentence in the plurality of original texts; after receiving the training set, each language model compares a model output result with a result of the training set, determines a corresponding loss value, and repeatedly adjusts training parameters until the loss value reaches a set value; Wait until each of the large language models completes training using the training set to obtain the first group of trained multiple language models.
7. The method according to claim 1, It is characterized in that The classifying and extracting the multiple sentences contained in the original text based on the sentence type corresponding to each sentence includes: Based on the statement type corresponding to each statement, determine multiple statements corresponding to each statement type, and retain one of the same statements among the multiple statements corresponding to each statement type; According to the multiple sentences corresponding to each sentence type, each sentence in the original text is saved in the first file in correspondence with the corresponding sentence type.
8. The method according to claim 1, It is characterized in that After classifying and extracting the multiple sentences contained in the original text based on the sentence type corresponding to each sentence, the method further includes: If an information viewing request is received from any request end, the multiple sentences extracted by classification are sent to the request end.
9. The method according to claim 1, It is characterized in that The determining the text type of the original text according to the sentence type of each sentence includes: The number of sentences corresponding to each sentence type is determined, and the text type of the original text is determined according to the sentence type corresponding to the maximum value among the number of sentences.
10. The method according to claim 1, It is characterized in that The step of summarizing the classified extracted text based on the semantic information of the classified extracted text to obtain the target text includes: The classified extracted text is input into a target language model to obtain a target text; the target language model is used to identify important information of the classified extracted text, determine semantic information of the classified extracted text based on the important information, and combine the semantic information to obtain the target text; the important information includes a description subject in the classified extracted text and partial information related to the description subject.
11. A text processing device, It is characterized in that include: A model acquisition unit, used to acquire a first group of multiple language models that have been trained; wherein the first language model is used to determine whether an input sentence is a first type of sentence; and the first language model is any one of the multiple language models; a type determination unit, configured to input the original text into the multiple language models respectively, and obtain the sentence type of each sentence in the original text output by each language model in the multiple language models; Determining the text type of the original text according to the sentence type of each sentence; A classification extraction unit, configured to classify and extract multiple sentences contained in the original text based on the sentence type corresponding to each sentence, to obtain a classified extracted text; The semantic summarization unit is used to summarize the classified extracted text based on the semantic information of the classified extracted text to obtain a target text.
12. An electronic device, It is characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of claims 1-10 according to the obtained program instructions.
13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 10.
14. A computer program product, It is characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Text classification method and device, electronic equipment and storage medium
CN112883190A
Context-based dialogue text classification method, system and equipment and storage medium
CN114357162A
Text processing method and device, model training method and device, equipment and storage medium
CN114691862A
Automatic abstracting method and system for legal instruments based on semantic segmentation
CN116894088A
Text classification method and device, equipment and storage medium
CN118093868A
