Extended text processing method and device
By setting the text expansion model outputs more extended text than the preset target and calculating the diversity parameter value, the problem of insufficient diversity of extended text in the prior art is solved, and semantically relevant and diverse extended text generation is achieved.
Patent Information
- Application Number
- CN202210162598.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-22
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-02-22
AI Technical Summary
Existing text extension models cannot ensure the output extended text diversity, resulting in a lack of diversity in generated extended text.
By setting the text extension model outputs more than the preset target, calculate the diversity parameter value of the candidate extended text, and select the target extended text from large to small according to the diversity parameter value to ensure the diversity of the extended text.
Effectively improve the diversity of extended text and ensure that the generated extended text has diversity while semantic relevance.
Smart Images

Figure CN114548069B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of information processing, and in particular to an extended text processing method and device. Background Art
[0002] The existing text expansion model can automatically generate an extended text that is different from the input text but has the same semantics as the input text. For example, if the input text is "how much is the shipping fee", the extended text output by the text expansion model can be "how much do I need to pay". Summary of the invention
[0003] The inventors note that the output result of the text expansion model can only ensure that the expanded text is semantically relevant to the input text, but cannot ensure the diversity of the expanded text.
[0004] Accordingly, the present disclosure provides an extended text processing solution that can effectively improve the diversity of extended text.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an extended text processing method, comprising: setting a number m1 of extended texts output by a text extension model, wherein the m1 is greater than a preset number m2 of target extended texts; inputting text information into the text extension model to obtain m1 candidate extended texts; extracting a plurality of candidate extended texts associated with the text information from the m1 candidate extended texts to generate a candidate extended text set; calculating a diversity parameter value of each candidate extended text in the candidate extended text set; and selecting m2 candidate extended texts from the candidate extended text set as target extended texts according to the order of the diversity parameter values from large to small.
[0006] In some embodiments, calculating the diversity parameter value of each candidate extended text in the candidate extended text set includes: in the candidate extended text set, respectively calculating the text similarity parameters between the i-th candidate extended text and other N-1 candidate extended texts, 1≤i≤N, N is the total number of candidate extended texts in the candidate extended text set; and determining the diversity parameter value of the i-th candidate extended text using the obtained N-1 text similarity parameters.
[0007] In some embodiments, using the obtained N-1 text similarity parameters to determine the diversity parameter value of the i-th candidate extended text includes: using an average value of the obtained N-1 text similarity parameters as the diversity parameter value of the i-th candidate extended text.
[0008] In some embodiments, the text similarity parameter is Jaccard distance.
[0009] In some embodiments, extracting multiple candidate extended texts associated with the text information from the m1 candidate extended texts to generate a candidate extended text set includes: among the m1 candidate extended texts, determining whether the semantics of the jth candidate extended text matches the semantics of the text information, where 1≤j≤m1; if the semantics of the jth candidate extended text matches the semantics of the text information, placing the jth candidate extended text into the candidate extended text set.
[0010] In some embodiments, determining whether the semantics of the jth candidate extended text matches the semantics of the text information includes: determining whether the semantics of the jth candidate extended text is the same as the semantics of the text information; if the semantics of the jth candidate extended text is the same as the semantics of the text information, determining that the semantics of the jth candidate extended text matches the semantics of the text information.
[0011] In some embodiments, if the semantics of the jth candidate extended text is different from the semantics of the text information, it is determined whether the semantics of the jth candidate extended text is included in the semantics of the text information; if the semantics of the jth candidate extended text is included in the semantics of the text information, it is determined that the semantics of the jth candidate extended text matches the semantics of the text information.
[0012] In some embodiments, if the semantics of the jth candidate extended text is not included in the semantics of the text information, it is determined whether the semantics of the jth candidate extended text includes the semantics of the text information; if the semantics of the jth candidate extended text includes the semantics of the text information, it is determined that the semantics of the jth candidate extended text matches the semantics of the text information.
[0013] In some embodiments, it is determined whether the jth candidate extended text has any semantic error; if the jth candidate extended text has no semantic error, it is determined whether the semantics of the jth candidate extended text matches the semantics of the text information.
[0014] According to a second aspect of an embodiment of the present disclosure, an extended text processing device is provided, comprising: a first processing module, configured to set the number m1 of extended texts output by a text extension model, wherein the m1 is greater than a preset number m2 of target extended texts; a second processing module, configured to input text information into the text extension model to obtain m1 candidate extended texts; and a third processing module, configured to extract a plurality of candidate extended texts associated with the text information from the m1 candidate extended texts to generate a candidate extended text set, calculate a diversity parameter value of each candidate extended text in the candidate extended text set, and select m2 candidate extended texts from the candidate extended text set as target extended texts according to the order of the diversity parameter values from large to small.
[0015] According to a third aspect of an embodiment of the present disclosure, there is provided an extended text processing apparatus, comprising: a memory configured to store instructions; a processor coupled to the memory, the processor being configured to execute a method as described in any of the above embodiments based on the instructions stored in the memory.
[0016] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method involved in any of the above embodiments is implemented.
[0017] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0019] Figure 1 A flowchart of an extended text processing method according to an embodiment of the present disclosure;
[0020] Figure 2 A flowchart of an extended text processing method according to another embodiment of the present disclosure;
[0021] Figure 3 A schematic diagram of the structure of an extended text processing device according to an embodiment of the present disclosure;
[0022] Figure 4 It is a structural schematic diagram of an extended text processing device according to another embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0024] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.
[0025] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0026] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.
[0027] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0028] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0029] Figure 1 The following is a flow chart of an extended text processing method according to an embodiment of the present disclosure. In some embodiments, the following extended text processing method is executed by an extended text processing device.
[0030] In step 101, the number m1 of extended texts output by the text extension model is set, wherein m1 is greater than a preset target extended text number m2.
[0031] In some embodiments, m1 is a multiple of m2. For example, m2 is 10 and m1 is 20.
[0032] In step 102, text information is input into a text expansion model to obtain m1 candidate expansion texts.
[0033] It should be noted here that since the text expansion model itself is not the inventive point of the present disclosure, it will not be described in detail here.
[0034] In step 103, multiple candidate extended texts associated with the text information are extracted from the m1 candidate extended texts to generate a candidate extended text set.
[0035] In some embodiments, step 103 is as Figure 2 shown.
[0036] In step 201, among the m1 candidate extended texts, it is determined whether the j-th candidate extended text has a semantic error, where 1 ≤ j ≤ m1.
[0037] If the j-th candidate extended text does not have a semantic error, step 202 is executed; otherwise, step 204 is executed.
[0038] For example, due to an error in the processing of the file extension model, the output candidate extended text is incomplete. For example, the candidate extended text output by the file extension model is "my white" instead of "how much is my white bar limit". By removing the candidate extended texts with semantic errors, the quality of the target extended text is ensured.
[0039] In step 202, it is determined whether the semantics of the j-th candidate extended text match the semantics of the text information.
[0040] If the semantics of the j-th candidate extended text match the semantics of the text information, step 203 is executed; otherwise, step 204 is executed.
[0041] In some embodiments, it is determined whether the semantics of the j-th candidate extended text are the same as the semantics of the text information. If the semantics of the j-th candidate extended text are the same as the semantics of the text information, it is determined that the semantics of the j-th candidate extended text match the semantics of the text information.
[0042] In some embodiments, if the semantics of the j-th candidate extended text are not the same as the semantics of the text information, it is determined whether the semantics of the j-th candidate extended text are included in the semantics of the text information. If the semantics of the j-th candidate extended text are included in the semantics of the text information, it is determined that the semantics of the j-th candidate extended text match the semantics of the text information.
[0043] In some embodiments, if the semantics of the j-th candidate extended text are not included in the semantics of the text information, it is determined whether the semantics of the j-th candidate extended text include the semantics of the text information. If the semantics of the j-th candidate extended text include the semantics of the text information, it is determined that the semantics of the j-th candidate extended text match the semantics of the text information.
[0044] For example, if the text information is "white bar", and the candidate extended text is "white bar on the e-commerce platform", the semantics of the candidate extended text are included in the semantics of the text information. In this case, it is determined that the semantics of the candidate extended text match the semantics of the text information.
[0045] In step 203, the jth candidate extension text is put into a candidate extension text set.
[0046] In step 204, the jth candidate extension text is removed.
[0047] return Figure 1 In step 104, a diversity parameter value of each candidate extended text in the candidate extended text set is calculated.
[0048] In some embodiments, in the candidate extended text set, the text similarity parameters of the i-th candidate extended text and other N-1 candidate extended texts are calculated respectively, 1≤i≤N, N is the total number of candidate extended texts in the candidate extended text set. Next, the diversity parameter value of the i-th candidate extended text is determined using the obtained N-1 text similarity parameters.
[0049] For example, the average value of the obtained N-1 text similarity parameters is used as the diversity parameter value of the i-th candidate extended text.
[0050] In some embodiments, the text similarity parameter is Jaccard distance.
[0051] In step 105, m2 candidate extended texts are selected from the candidate extended text set according to the order of the diversity parameter values from large to small to serve as target extended texts.
[0052] For example, text information is input into the text expansion model to obtain 20 candidate expansion texts, from which 15 candidate expansion texts associated with the text information are selected to generate a candidate expansion text set. Next, the diversity parameter value of each candidate expansion text in the candidate expansion text set is calculated, and 10 candidate expansion texts are selected from the candidate expansion text set as target expansion texts according to the order of the diversity parameter values from large to small. This ensures the diversity of the target expansion text.
[0053] Figure 3 3 , the extended text processing device includes a first processing module 31 , a second processing module 32 and a third processing module 33 .
[0054] The first processing module 31 sets the text expansion model to output the number m1 of expanded texts, wherein m1 is greater than a preset target number m2 of expanded texts.
[0055] In some embodiments, m1 is a multiple of m2. For example, m2 is 10 and m1 is 20.
[0056] The second processing module 32 inputs the text information into the text expansion model to obtain m1 candidate expansion texts.
[0057] It should be noted here that since the text expansion model itself is not the inventive point of the present disclosure, it will not be described in detail here.
[0058] The third processing module 33 extracts multiple candidate extended texts associated with the text information from the m1 candidate extended texts to generate a candidate extended text set, calculates the diversity parameter value of each candidate extended text in the candidate extended text set, and selects m2 candidate extended texts from the candidate extended text set as target extended texts according to the order of the diversity parameter values from large to small.
[0059] In some embodiments, the third processing module 33 determines whether the jth candidate extended text has a semantic error among the m1 candidate extended texts, where 1≤j≤m1. If the jth candidate extended text has a semantic error, the jth candidate extended text is removed, otherwise it is determined whether the semantics of the jth candidate extended text matches the semantics of the text information. If the semantics of the jth candidate extended text does not match the semantics of the text information, the jth candidate extended text is removed, otherwise the jth candidate extended text is put into the candidate extended text set.
[0060] For example, the third processing module 33 determines whether the semantics of the jth candidate extended text is the same as the semantics of the text information. If the semantics of the jth candidate extended text is the same as the semantics of the text information, it is determined that the semantics of the jth candidate extended text matches the semantics of the text information. If the semantics of the jth candidate extended text is not the same as the semantics of the text information, the third processing module 33 determines whether the semantics of the jth candidate extended text is included in the semantics of the text information. If the semantics of the jth candidate extended text is included in the semantics of the text information, it is determined that the semantics of the jth candidate extended text matches the semantics of the text information. If the semantics of the jth candidate extended text is not included in the semantics of the text information, the third processing module 33 determines whether the semantics of the jth candidate extended text includes the semantics of the text information. If the semantics of the jth candidate extended text includes the semantics of the text information, it is determined that the semantics of the jth candidate extended text matches the semantics of the text information.
[0061] In some embodiments, the third processing module 33 calculates the text similarity parameters of the i-th candidate extended text and other N-1 candidate extended texts in the candidate extended text set, 1≤i≤N, N is the total number of candidate extended texts in the candidate extended text set. Next, the diversity parameter value of the i-th candidate extended text is determined using the obtained N-1 text similarity parameters.
[0062] For example, the average value of the obtained N-1 text similarity parameters is used as the diversity parameter value of the i-th candidate extended text.
[0063] In some embodiments, the text similarity parameter is Jaccard distance.
[0064] For example, text information is input into the text expansion model to obtain 20 candidate expansion texts, from which 15 candidate expansion texts associated with the text information are selected to generate a candidate expansion text set. Next, the diversity parameter value of each candidate expansion text in the candidate expansion text set is calculated, and 10 candidate expansion texts are selected from the candidate expansion text set as target expansion texts according to the order of the diversity parameter values from large to small. This ensures the diversity of the target expansion text.
[0065] Figure 4 FIG. 1 is a schematic diagram of the structure of an extended text processing device according to another embodiment of the present disclosure. Figure 4 As shown, the extended text processing device includes a memory 41 and a processor 42.
[0066] The memory 41 is used to store instructions. The processor 42 is coupled to the memory 41. The processor 42 is configured to execute the instructions stored in the memory to implement the following. Figure 1 or Figure 2 The method of any one of the embodiments.
[0067] like Figure 4 As shown, the extended text processing device also includes a communication interface 43 for information exchange with other devices. At the same time, the extended text processing device also includes a bus 44, through which the processor 42, the communication interface 43, and the memory 41 communicate with each other.
[0068] The memory 41 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory. The memory 41 may also be a memory array. The memory 41 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.
[0069] In addition, the processor 42 may be a central processing unit (CPU), or may be an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.
[0070] The present disclosure also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, which are executed by a processor to implement the following Figure 1 or Figure 2 The method of any one of the embodiments.
[0071] In some embodiments, the functional unit module described above can be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components or any appropriate combination thereof for performing the functions described in the present disclosure.
[0072] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0073] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.
Claims
1. An extended text processing method, comprising: Setting the number m1 of extended texts output by the text extension model, wherein m1 is greater than a preset number m2 of target extended texts; Inputting text information into the text expansion model to obtain m1 candidate expansion texts; Extracting a plurality of candidate extended texts associated with the text information from the m1 candidate extended texts to generate a candidate extended text set; Calculating a diversity parameter value of each candidate extended text in the candidate extended text set; According to the order of the diversity parameter values from large to small, m2 candidate extended texts are selected from the candidate extended text set as target extended texts; The step of calculating the diversity parameter value of each candidate extended text in the candidate extended text set includes: In the candidate extended text set, respectively calculating text similarity parameters between the i-th candidate extended text and other N-1 candidate extended texts, 1≤i≤N, N being the total number of candidate extended texts in the candidate extended text set; The obtained N-1 text similarity parameters are used to determine the diversity parameter value of the i-th candidate extended text.
2. The method according to claim 1, wherein: Determining the diversity parameter value of the i-th candidate extended text by using the obtained N-1 text similarity parameters includes: The average value of the obtained N-1 text similarity parameters is used as the diversity parameter value of the i-th candidate extended text.
3. The method according to claim 1, wherein: The text similarity parameter is the Jaccard distance.
4. The method according to any one of claims 1 to 3, wherein: Extracting a plurality of candidate extended texts associated with the text information from the m1 candidate extended texts to generate a candidate extended text set includes: Among the m1 candidate extension texts, determining whether the semantics of the jth candidate extension text matches the semantics of the text information, where 1≤j≤m1; If the semantics of the j-th candidate extended text matches the semantics of the text information, the j-th candidate extended text is put into the candidate extended text set.
5. The method according to claim 4, wherein: Determining whether the semantics of the j-th candidate extended text matches the semantics of the text information includes: Determining whether the semantics of the j-th candidate extended text is the same as the semantics of the text information; If the semantics of the jth candidate extended text is the same as the semantics of the text information, it is determined that the semantics of the jth candidate extended text matches the semantics of the text information.
6. The method according to claim 5, further comprising: If the semantics of the j-th candidate extended text is different from the semantics of the text information, determining whether the semantics of the j-th candidate extended text is included in the semantics of the text information; If the semantics of the j-th candidate extended text is included in the semantics of the text information, it is determined that the semantics of the j-th candidate extended text matches the semantics of the text information.
7. The method according to claim 6, further comprising: If the semantics of the j-th candidate extended text is not included in the semantics of the text information, determining whether the semantics of the j-th candidate extended text includes the semantics of the text information; If the semantics of the j-th candidate extended text includes the semantics of the text information, it is determined that the semantics of the j-th candidate extended text matches the semantics of the text information.
8. The method according to claim 4, further comprising: Determining whether the j-th candidate extension text has a semantic error; If there is no semantic error in the j-th candidate extended text, it is determined whether the semantics of the j-th candidate extended text matches the semantics of the text information.
9. An extended text processing device, comprising: A first processing module is configured to set the number m1 of extended texts output by the text extension model, wherein m1 is greater than a preset target extended text number m2; A second processing module is configured to input text information into the text expansion model to obtain m1 candidate expansion texts; The third processing module is configured to extract multiple candidate extended texts associated with the text information from the m1 candidate extended texts to generate a candidate extended text set, calculate the diversity parameter value of each candidate extended text in the candidate extended text set, and select m2 candidate extended texts from the candidate extended text set as target extended texts according to the order of the diversity parameter values from large to small, wherein in the candidate extended text set, the text similarity parameters of the i-th candidate extended text and other N-1 candidate extended texts are respectively calculated, 1≤i≤N, N is the total number of candidate extended texts in the candidate extended text set, and the diversity parameter value of the i-th candidate extended text is determined by using the obtained N-1 text similarity parameters.
10. An extended text processing device, comprising: a memory configured to store instructions; A processor is coupled to the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 based on instructions stored in the memory.
11. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.
