Correlation prediction model training method and device, and abstract generation method and device
By extracting semantic feature vectors of sentences and predicting their relevance using the BERT model, and combining this with the TextRank algorithm to calculate weight values, the problem of incomplete semantics in existing summaries is solved, resulting in higher-quality summaries.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
- Filing Date
- 2022-02-10
- Publication Date
- 2026-05-22
AI Technical Summary
Existing summary generation methods fail to effectively guarantee the semantic integrity of summaries, resulting in the loss of semantics for important statements and reducing the quality of summaries.
By extracting semantic feature vectors from sentences, using the BERT model to predict semantic relevance, generating training samples and determining the loss function, training the relevance prediction model, and combining the TextRank algorithm to calculate the weight values of candidate summaries, the relevance and importance of sentences are ensured, and a complete summary is generated.
It improves the semantic integrity and quality of the abstract, ensures the preservation of important information, and generates more accurate abstracts.
Smart Images
Figure CN114461789B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of information processing, and in particular to a method and apparatus for training a correlation prediction model, and a method and apparatus for generating summaries. Background Technology
[0002] Current summary generation methods mainly extract important sentences from text to serve as a summary of the text. Summary of the Invention
[0003] The inventors discovered through research that existing summarization methods do not consider the semantic completeness of the extracted sentences. For example, if a text includes the sentence "A certain company has annual revenue exceeding 10 billion, surpassing industry standards. However, its profits are only a few million," and the summary only extracts the second sentence, "However, its profits are only a few million," the loss of semantics in the first sentence, "A certain company has annual revenue exceeding 10 billion, surpassing industry standards," will result in a semantically incomplete summary, thus reducing its quality.
[0004] Accordingly, this disclosure provides a summary generation scheme that can effectively guarantee the semantic integrity of the summary.
[0005] According to a first aspect of the present disclosure, a method for training an association prediction model is provided, comprising: extracting a first semantic feature vector of a first statement sample and a second semantic feature vector of a second statement sample; generating training samples, wherein the training samples include the first semantic feature vector and the second semantic feature vector, and preset semantic association annotations of the first statement sample and the second statement sample; inputting the training samples into a machine learning model to obtain semantic association prediction results of the first statement sample and the second statement sample; determining a loss function based on the semantic association annotations and the semantic association prediction results; and training the machine learning model using the loss function to obtain the association prediction model.
[0006] In some embodiments, the training samples further include a delimiter, which is set between the first semantic feature vector and the second semantic feature vector.
[0007] In some embodiments, the machine learning model is a converter-based bidirectional encoding representation BERT model.
[0008] According to a second aspect of the present disclosure, a training apparatus for an association prediction model is provided, comprising: a first processing module configured to extract a first semantic feature vector of a first statement sample and a second semantic feature vector of a second statement sample, and generate training samples, wherein the training samples include the first semantic feature vector and the second semantic feature vector, and preset semantic association annotations of the first statement sample and the second statement sample; and a second processing module configured to input the training samples into a machine learning model to obtain semantic association prediction results of the first statement sample and the second statement sample, determine a loss function based on the semantic association annotations and the semantic association prediction results, and train the machine learning model using the loss function to obtain an association prediction model.
[0009] In some embodiments, the training samples further include a delimiter, which is set between the first semantic feature vector and the second semantic feature vector.
[0010] In some embodiments, the machine learning model is a converter-based bidirectional encoding representation BERT model.
[0011] According to a third aspect of the present disclosure, an apparatus for training an association prediction model is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any of the above embodiments.
[0012] According to a fourth aspect of the present disclosure, a summary generation method is provided, comprising: extracting all sentences from a target text; inputting the semantic feature vectors of all adjacent sentences into a correlation prediction model according to the order of the sentences in the target text to obtain semantic correlation prediction results of all adjacent sentences, wherein the correlation prediction model is obtained by the training method described in any of the above embodiments; taking all adjacent sentences with semantic correlation prediction results greater than a preset threshold as candidate summaries; calculating the weight value of each sentence in all candidate summaries in the target text; determining the weight value of each candidate summary according to the weight values of all sentences included in each candidate summary; and taking the candidate summary with the largest weight value as the summary of the target text.
[0013] In some embodiments, before calculating the weight value of each statement in the target text among all candidate summaries, the method further includes: determining whether there are two candidate summaries with the same statement; if there are two candidate summaries with the same statement, then merging the two candidate summaries.
[0014] In some embodiments, the weight of each candidate summary is the average of the weights of all statements included in each candidate summary.
[0015] In some embodiments, calculating the weight value of each statement in all candidate summaries in the target text includes: calculating the weight value of each statement in all candidate summaries in the target text using the TextRank algorithm.
[0016] According to a fifth aspect of the present disclosure, a summary generation apparatus is provided, comprising: an extraction module configured to extract all statements from a target text; a prediction module configured to input the semantic feature vectors of all adjacent statements into a correlation prediction model according to the order of the statements in the target text, to obtain semantic correlation prediction results of all adjacent statements, wherein the correlation prediction model is obtained by the training method described in any of the above embodiments; and a generation module configured to take all adjacent statements with semantic correlation prediction results greater than a preset threshold as candidate summaries, calculate the weight value of each statement in all candidate summaries in the target text, determine the weight value of each candidate summary based on the weight values of all statements included in each candidate summary, and take the candidate summary with the largest weight value as the summary of the target text.
[0017] In some embodiments, the generation module is configured to determine whether there are two candidate digests with the same statement, and if there are two candidate digests with the same statement, then the two candidate digests are merged.
[0018] In some embodiments, the weight of each candidate summary is the average of the weights of all statements included in each candidate summary.
[0019] In some embodiments, the generation module is configured to use the TextRank algorithm to calculate the weight value of each statement in the target text among all candidate summaries.
[0020] According to a sixth aspect of the present disclosure, a summary generation apparatus is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any of the above embodiments.
[0021] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.
[0022] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a flowchart illustrating a correlation prediction model training method according to an embodiment of the present disclosure;
[0025] Figure 2 This is a schematic diagram of the structure of a correlation prediction model training device according to an embodiment of the present disclosure;
[0026] Figure 3 This is a schematic diagram of the structure of a correlation prediction model training device according to another embodiment of the present disclosure;
[0027] Figure 4 This is a schematic flowchart of a summary generation method according to an embodiment of the present disclosure;
[0028] Figure 5 This is a schematic diagram of the structure of an abstract generation apparatus according to an embodiment of the present disclosure;
[0029] Figure 6 This is a schematic diagram of the structure of a summary generation apparatus according to another embodiment of the present disclosure. Detailed Implementation
[0030] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.
[0031] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of this disclosure.
[0032] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0033] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0034] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0035] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0036] Figure 1 This is a schematic flowchart illustrating a correlation prediction model training method according to an embodiment of the present disclosure. In some embodiments, the following correlation prediction model training method is performed by a correlation prediction model training apparatus.
[0037] In step 101, the first semantic feature vector of the first statement sample and the second semantic feature vector of the second statement sample are extracted.
[0038] In step 102, training samples are generated, wherein the training samples include a first semantic feature vector and a second semantic feature vector, as well as preset semantic association annotations of the first statement sample and the second statement sample.
[0039] In some embodiments, the training samples include semantic association annotations, a first semantic feature vector, a second semantic feature vector, and a separator, wherein the separator is set between the first semantic feature vector and the second semantic feature vector.
[0040] For example, the training samples are ([CLS], x 11 x 12 , ..., x 1n [SEP], x 21 x 22 , ..., x 1m ), where [CLS] is the semantic association tag, (x 11 x 12 , ..., x 1n ) is the first semantic feature vector, [SEP] is the delimiter, (x 21 x 22 , ..., x 1m ) is the second semantic feature vector.
[0041] In step 103, the training samples are input into the machine learning model to obtain the semantic relevance prediction results of the first sentence sample and the second sentence sample.
[0042] In some embodiments, the machine learning model is the BERT (Bidirectional Encoder Representations from Transformers) model.
[0043] In step 104, the loss function is determined based on the semantic association annotation and semantic association prediction results.
[0044] In step 105, the machine learning model is trained using a loss function to obtain an association prediction model.
[0045] In the association prediction model training method provided in the above embodiments of this disclosure, a machine learning model is trained using two sentence samples to obtain a prediction model capable of predicting the semantic association between the two sentences.
[0046] Figure 2 This is a schematic diagram of the structure of a correlation prediction model training device according to an embodiment of this disclosure. Figure 2 As shown, the correlation prediction model training device includes a first processing module 21 and a second processing module 22.
[0047] The first processing module 21 is configured to extract the first semantic feature vector of the first statement sample and the second semantic feature vector of the second statement sample to generate training samples, wherein the training samples include the first semantic feature vector and the second semantic feature vector, as well as preset semantic association annotations of the first statement sample and the second statement sample.
[0048] In some embodiments, the training samples include semantic association annotations, a first semantic feature vector, a second semantic feature vector, and a separator, wherein the separator is set between the first semantic feature vector and the second semantic feature vector.
[0049] The second processing module 22 is configured to input training samples into a machine learning model to obtain semantic relevance prediction results of the first sentence sample and the second sentence sample, determine a loss function based on the semantic relevance annotation and semantic relevance prediction results, and use the loss function to train the machine learning model to obtain a relevance prediction model.
[0050] In some embodiments, the machine learning model is the BERT model.
[0051] Figure 3 This is a schematic diagram of the structure of a correlation prediction model training device according to another embodiment of this disclosure. Figure 3 As shown, the correlation prediction model training device includes a memory 31 and a processor 32.
[0052] Memory 31 is used to store instructions, and processor 32 is coupled to memory 31. Processor 32 is configured to execute instructions based on memory storage, as shown in the example. Figure 1 The method involved in any of the embodiments.
[0053] like Figure 3 As shown, the correlation prediction model training device also includes a communication interface 33 for information exchange with other devices. Simultaneously, the correlation prediction model training device also includes a bus 34, through which the processor 32, communication interface 33, and memory 31 communicate with each other.
[0054] The memory 31 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. The memory 31 may also be a memory array. The memory 31 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.
[0055] Furthermore, processor 32 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.
[0056] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figure 1 The method involved in any of the embodiments.
[0057] Figure 4 This is a schematic flowchart illustrating a summary generation method according to an embodiment of the present disclosure. In some embodiments, the following summary generation method is performed by a summary generation apparatus.
[0058] In step 401, all statements in the target text are extracted.
[0059] For example, extracting statements based on periods in the target file.
[0060] In step 402, according to the order of each statement in the target text, the semantic feature vectors of all adjacent statements are input into the correlation prediction model to obtain the semantic correlation prediction results of all adjacent statements.
[0061] It should be noted here that the correlation prediction model uses... Figure 1 The training method of any of the embodiments is obtained.
[0062] In step 403, all two adjacent statements whose semantic relevance prediction results are greater than a preset threshold are selected as candidate summaries.
[0063] In step 404, the weight value of each statement in all candidate summaries in the target text is calculated.
[0064] In some embodiments, the TextRank algorithm is used to calculate the weight value of each statement in the target text among all candidate summaries.
[0065] Since the TextRank algorithm itself is not the inventive point of this disclosure, it will not be described in detail here.
[0066] In step 405, the weight value of each candidate summary is determined based on the weight values of all statements included in each candidate summary.
[0067] In some embodiments, the weight value of each candidate digest is the average of the weight values of all statements included in each candidate digest.
[0068] In step 406, the candidate summary with the highest weight value is used as the summary of the target text.
[0069] For example, a target text contains 10 sentences, numbered a1-a10 if arranged in order of appearance. A relevance prediction model is used to predict the semantic relevance between a1 and a2, a2 and a3, a3 and a4, a4 and a5, a5 and a6, a6 and a7, a7 and a8, a8 and a9, and a9 and a10. Based on the prediction results, if the predicted semantic relevance between a1 and a2, and between a4 and a5, exceeds a preset threshold, then a1 and a2 are selected as the first candidate summary, and a4 and a5 as the second candidate summary. Next, the weight values of a1, a2, a4, and a5 in the target text are calculated. The average weight value of a1 and a2 is used as the weight value of the first candidate summary, and the average weight value of a4 and a5 is used as the weight value of the second candidate summary. If the weight value of the first candidate summary is greater than the weight value of the second candidate summary, then the first candidate summary (including a1 and a2) is selected as the summary of the target text. Conversely, the second candidate abstract (including a4 and a5) will be used as the abstract of the target text.
[0070] In some embodiments, before calculating the weight value of each statement in the target text among all candidate summaries, it is further determined whether there are two candidate summaries with the same statement. If there are two candidate summaries with the same statement, the two candidate summaries are merged.
[0071] For example, a target text contains 10 sentences, which, if arranged in the order of their appearance, would be a1-a10. The semantic relevance of a1 and a2, a2 and a3, a3 and a4, a4 and a5, a5 and a6, a6 and a7, a7 and a8, a8 and a9, and a9 and a10 is predicted sequentially using a relevance prediction model. Based on the prediction results, the semantic relevance predictions for a1 and a2, a4 and a5, and a5 and a6 are found to be greater than a preset threshold. Therefore, a1 and a2 are selected as the first candidate summary, a4 and a5 as the second candidate summary, and a5 and a6 as the third candidate summary. Since both the second and third candidate summaries include a5, it indicates that a4-a6 are semantically related. In this case, a4, a5, and a6 are selected as the fourth candidate summary. Next, the weight values of a1, a2, a4, a5, and a6 in the target text are calculated respectively. The average weight value of a1 and a2 is used as the weight value of the first candidate summary, and the average weight value of a4, a5, and a6 is used as the weight value of the fourth candidate summary. If the weight value of the first candidate summary is greater than the weight value of the fourth candidate summary, then the first candidate summary (including a1 and a2) is used as the summary of the target text. Otherwise, the fourth candidate summary (including a4, a5, and a6) is used as the summary of the target text.
[0072] Figure 5 This is a schematic diagram of the structure of a summary generation apparatus according to an embodiment of this disclosure. Figure 5 As shown, the abstract generation device includes an extraction module 51, a prediction module 52, and a generation module 53.
[0073] Extraction module 51 is configured to extract all statements from the target text.
[0074] Prediction module 52 is configured to input the semantic feature vectors of all adjacent pairs of statements into the relevance prediction model according to the order of the statements in the target text, in order to obtain the semantic relevance prediction results of all adjacent pairs of statements, wherein the relevance prediction model is obtained through... Figure 1 The training method of any of the embodiments is obtained.
[0075] The generation module 53 is configured to take all two adjacent statements whose semantic relevance prediction results are greater than a preset threshold as candidate summaries, calculate the weight value of each statement in all candidate summaries in the target text, determine the weight value of each candidate summary based on the weight values of all statements included in each candidate summary, and take the candidate summary with the largest weight value as the summary of the target text.
[0076] In some embodiments, the generation module 53 uses the TextRank algorithm to calculate the weight value of each statement in the target text among all candidate summaries.
[0077] In some embodiments, the weight value of each candidate digest is the average of the weight values of all statements included in each candidate digest.
[0078] In some embodiments, before calculating the weight value of each statement in all candidate summaries in the target text, the generation module 53 further determines whether there are two candidate summaries with the same statement. If there are two candidate summaries with the same statement, the two candidate summaries are merged.
[0079] Figure 6 This is a schematic diagram of the structure of a summary generation apparatus according to another embodiment of this disclosure. Figure 6 As shown, the abstract generation device includes a memory 61, a processor 62, a communication interface 63, and a bus 64. Figure 6 and Figure 3 The difference is that, in Figure 6 In the illustrated embodiment, processor 62 is configured to implement memory-based instruction execution as follows: Figure 4 The method involved in any of the embodiments.
[0080] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement as follows: Figure 4 The method involved in any of the embodiments.
[0081] In some embodiments, the functional unit modules described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.
[0082] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0083] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.
Claims
1. A method for generating summaries, comprising: Extract all sentences from the target text; According to the order of each statement in the target text, the semantic feature vectors of all adjacent statements are input into the relevance prediction model to obtain the semantic relevance prediction results of all adjacent statements, wherein the relevance prediction model is obtained through a training method. All adjacent sentences whose semantic relevance prediction results are greater than a preset threshold are selected as candidate summaries; Determine if two candidate digests with the same statement exist; If two candidate digests with the same statement exist, then the two candidate digests are merged. Calculate the weight value of each statement in the target text from all candidate summaries; The weight value of each candidate digest is determined based on the weight values of all statements included in each candidate digest, wherein the weight value of each candidate digest is the average of the weight values of all statements included in each candidate digest; The candidate summary with the highest weight value is used as the summary of the target text; The training method includes: Extract the first semantic feature vector of the first statement sample and the second semantic feature vector of the second statement sample; Generate training samples, wherein the training samples include the first semantic feature vector and the second semantic feature vector, as well as preset semantic association annotations of the first statement sample and the second statement sample; The training samples are input into the machine learning model to obtain the semantic relevance prediction results of the first statement sample and the second statement sample; The loss function is determined based on the semantic association annotation and the semantic association prediction results; The machine learning model is trained using the loss function to obtain the correlation prediction model.
2. The method according to claim 1, wherein, Calculating the weight value of each statement in the target text from all candidate summaries includes: The TextRank algorithm is used to calculate the weight value of each statement in the target text among all candidate summaries.
3. The method according to claim 1, wherein, The training samples also include a separator, which is set between the first semantic feature vector and the second semantic feature vector.
4. The method according to claim 1 or 3, wherein, The machine learning model is a converter-based bidirectional encoding representation BERT model.
5. A summary generation apparatus, comprising: The extraction module is configured to extract all statements from the target text; The prediction module is configured to input the semantic feature vectors of all adjacent pairs of statements into an association prediction model according to the order of the statements in the target text, so as to obtain the semantic association prediction results of all adjacent pairs of statements. The association prediction model is obtained through a training method, which includes: extracting a first semantic feature vector from a first statement sample and a second semantic feature vector from a second statement sample; generating training samples, wherein the training samples include the first semantic feature vector and the second semantic feature vector, and preset semantic association annotations for the first and second statement samples; inputting the training samples into a machine learning model to obtain the semantic association prediction results for the first and second statement samples; determining a loss function based on the semantic association annotations and the semantic association prediction results; and training the machine learning model using the loss function to obtain the association prediction model. The generation module is configured to take any two adjacent statements whose semantic relevance prediction results are greater than a preset threshold as candidate summaries, determine whether there are two candidate summaries with the same statement, and if so, merge the two candidate summaries. It then calculates the weight value of each statement in all candidate summaries within the target text, determines the weight value of each candidate summary based on the weight values of all statements included in each candidate summary, where the weight value of each candidate summary is the average of the weight values of all statements included in each candidate summary, and takes the candidate summary with the highest weight value as the summary of the target text.
6. The apparatus according to claim 5, wherein, The generation module is configured to use the TextRank algorithm to calculate the weight value of each statement in the target text among all candidate summaries.
7. The apparatus according to claim 5, wherein, The training samples also include a separator, which is set between the first semantic feature vector and the second semantic feature vector.
8. The apparatus according to claim 5 or 7, wherein, The machine learning model is a converter-based bidirectional encoding representation BERT model.
9. A summary generation apparatus, comprising: The memory is configured to store instructions; A processor, coupled to memory, configured to implement the method as described in any one of claims 1-4 based on memory-stored instruction execution.
10. A computer-readable storage medium, wherein, A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-4.