Text generation method and device, equipment and storage medium
By generating and weighting sentence vectors, combining similarity and cluster vector calculations, selecting text to form abstract text, solving the problem of low accuracy in generating summary results in the existing text analysis system, and realizing abstract text generation that is more in line with user needs.
Patent Information
- Application Number
- CN202411915526.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-03-28
AI Technical Summary
The text summary results generated by the existing text analysis system have poor accuracy, resulting in the summary results that do not match the user's needs and ignore valid data.
By generating sentence vectors for each sentence in the original text and setting weights for each sentence vector, selecting multiple sentences from the original text to form a summary text based on similarity and cluster vector calculations.
Improve the accuracy of the summary text, making it more in line with user needs, and reduces the situation of ignoring effective data.
Smart Images
Figure CN119719363A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a text generation method, device, equipment and storage medium. Background Art
[0002] With the popularization of the Internet, various information resources and data are also scattered in various network systems, and the development of enterprises is inseparable from the effective use of various data.
[0003] In some technologies, in order to extract key information from Internet data information, existing text analysis systems are usually used to extract core information from large amounts of text and generate concise and meaningful summaries.
[0004] However, the accuracy of text summary results generated by existing text analysis systems is poor, which makes the obtained text summary results deviate from the user's demand for data, resulting in the obtained text summary results ignoring some valid data. Summary of the invention
[0005] The purpose of the present invention is to at least provide a text generation method, device, equipment and storage medium, which can at least solve the technical problem that the accuracy of text summary results generated by existing text analysis systems is poor, so that the obtained text summary results deviate from the user's demand for data, thereby causing the obtained text summary results to ignore certain valid data, and at least achieve the technical effect of effectively improving the accuracy of summary text formation, making it more in line with user needs, and reducing the probability of ignoring certain valid data when generating summary text.
[0006] To solve the above technical problems, at least one embodiment of the present application provides a text generation method, comprising:
[0007] Generate a sentence vector for each sentence in the original text, and an original text vector corresponding to the original text, and set a weight for each of the sentence vectors;
[0008] Based on the weights of the sentence vectors of the sentences and a preset selection condition, selecting a plurality of sentences from the original text to form a first text, and generating a first text vector of the first text;
[0009] Calculating a first similarity between a sentence vector of each sentence in the original text and the first text vector;
[0010] Clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating a second similarity between each cluster vector and the original text vector;
[0011] Based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each sentence belongs, multiple sentences are selected from the original text to form a summary text.
[0012] At least one embodiment of the present application further provides a text generation device, comprising:
[0013] A sentence vector generation module, used to generate a sentence vector for each sentence in the original text, and an original text vector corresponding to the original text, and set a weight for each sentence vector;
[0014] A first text forming module, configured to select a plurality of sentences from the original text to form a first text based on the weights of the sentence vectors of the sentences and a preset selection condition, and generate a first text vector of the first text;
[0015] A first similarity calculation module, used for calculating a first similarity between a sentence vector of each sentence in the original text and the first text vector;
[0016] A second similarity calculation module, used for clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating a second similarity between each cluster vector and the original text vector;
[0017] The summary text forming module is used to select multiple sentences from the original text to form a summary text based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each sentence belongs.
[0018] At least one embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned text generation method.
[0019] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned text generation method when executed by a processor.
[0020] The text generation method provided in the embodiment of the present application selects multiple sentences in the original text to form a summary text based on the first similarity between the sentence vector of each sentence in the original text and the first text vector and the second similarity between each cluster vector and the original text vector, thereby effectively improving the accuracy of the summary text formation and making it more in line with user needs, so as to reduce the probability of ignoring certain valid data when generating the summary text.
[0021] In some optional embodiments, clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating the second similarity between each cluster vector and the original text vector includes:
[0022] The sentence vectors of each sentence in the original text are divided into a plurality of clusters by using a clustering algorithm, and an average value of the sentence vectors of each cluster is calculated to determine the cluster vector, and a second similarity between the cluster vector and the original text vector is calculated.
[0023] The second similarity between the cluster vector and the original text vector is calculated by a clustering algorithm, so that the obtained second similarity is used as a standard for selecting sentences from the original text, making the selection of sentences more accurate, and through clustering calculation, the amount of calculation is effectively reduced and the calculation efficiency is improved.
[0024] In some optional embodiments, the selecting a plurality of sentences from the original text to form a summary text based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster to which the sentence vector of each sentence belongs includes:
[0025] Based on the weights of the sentence vectors of the sentences, the sentence corresponding to the sentence vector with the largest weight is selected from the original text as the first selected sentence to be used to form the summary text, and the remaining sentences in the original text except the selected sentence are recorded as candidate sentences;
[0026] The process of selecting a sentence other than the first one selected from the original text for forming the summary text comprises:
[0027] Calculating a third similarity between the text vector of the text formed by all the selected sentences and the sentence vector of each current candidate sentence;
[0028] Based on the first similarity corresponding to each of the candidate sentences, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each of the candidate sentences is located, and the third similarity corresponding to each of the candidate sentences, at least one candidate sentence is selected from the candidate sentences as the non-first selected sentence for forming the summary text;
[0029] The summary text is formed based on all the selected sentences.
[0030] Based on the first similarity between the sentence vector of each sentence in the original text and the first text vector, the second similarity between each cluster vector and the original text vector, and the third similarity corresponding to each of the current candidate sentences, the first similarity and the second similarity obtained by combining the TextRank algorithm and the clustering algorithm are double-authenticated to improve the accuracy of the similarity results, reduce the amount of calculation, and improve calculation efficiency.
[0031] In some optional embodiments, forming the summary text based on all the selected sentences includes:
[0032] The first selected sentence selected to form the summary text is used as the first sentence of the summary text;
[0033] For the selected sentences other than the first one selected to form the summary text, determining the order of each of the selected sentences in the summary text based on the first similarity corresponding to each of the selected sentences, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each of the selected sentences belongs, and the third similarity corresponding to each of the selected sentences;
[0034] The summary text is formed based on the ranking of all the selected sentences in the summary text.
[0035] By defining the order of each selected sentence, a summary text is formed according to the order of the selected sentences, so that the summary text is formed more accurately.
[0036] In some optional embodiments, selecting a plurality of sentences from the original text to form a first text based on the weights of the sentence vectors of the sentences and a preset selection condition, and generating a first text vector of the first text includes:
[0037] Obtaining keywords input by the user, and detecting whether the keywords exist in each sentence of the original text;
[0038] When detecting that the keyword exists in a sentence in the original text, determining that the weight value of the sentence vector of the sentence in which the keyword exists in the original text is a first weight value, and the weight value of the sentence vector of the sentence in which the keyword does not exist in the original text is a second weight value, wherein the first weight value is greater than the second weight value, and the first weight value is positively correlated with the number of the keywords contained in the sentence;
[0039] Based on the weight value of the sentence vector of each sentence in the original text, multiple sentences meeting the selection condition are selected from the original text to form a first text, and a first text vector of the first text is generated.
[0040] Through the keywords defined by the user, the generated first text is made more in line with the needs of the user, so that the finally formed summary text is made more in line with the needs of the user.
[0041] In some optional embodiments, generating a sentence vector for each sentence in the original text includes:
[0042] Acquire text data of a web page, and generate a plurality of original texts based on the text data;
[0043] The original text is segmented into sentences and words to obtain sentence vectors of the plurality of sentences, wherein the sentence vectors of the sentences are composed of word vectors of the words contained in the sentences.
[0044] By segmenting the original text into sentences and words, the error of the resulting sentence vector is reduced, which facilitates the formation of the subsequent summary text.
[0045] In some optional embodiments, obtaining text data of a web page includes:
[0046] Extracting data from the text content of the webpage to obtain the webpage text content and the corresponding text type;
[0047] The webpage text content is filled into a preset data structure, and the webpage text content is labeled according to the text type, so as to obtain the text data of the webpage.
[0048] By labeling the webpage text content according to the text type, it is easy to understand the data source of the webpage text content. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.
[0050] Figure 1It is a flowchart of a text generation method provided by an embodiment of the present application;
[0051] Figure 2 is a structural block diagram of a text generation device provided by an embodiment of the present application;
[0052] Figure 3 is a structural schematic diagram of an electronic device provided by another embodiment of the present application;
[0053] Figure 4 It is a schematic diagram of a flow chart of obtaining text data of a web page in a text generation method provided by an embodiment of the present application;
[0054] Figure 5 It is a schematic diagram of a flow chart of forming a summary text of a web page in a text generation method provided by an embodiment of the present application;
[0055] Figure 6 It is a structural diagram of a text generation system provided by another embodiment of the present application. DETAILED DESCRIPTION
[0056] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.
[0057] In order to facilitate understanding of the embodiments of the present application, relevant content about the text generation method is first introduced here.
[0058] With the popularization of the Internet, various information resources and data are also scattered in various network systems, and the development of enterprises is inseparable from the effective use of various data.
[0059] In some technologies, in order to extract key information from Internet data information, existing text analysis systems are usually used to extract core information from large amounts of text and generate concise and meaningful summaries.
[0060] However, the accuracy of text summary results generated by existing text analysis systems is poor, which makes the obtained text summary results deviate from the user's demand for data, resulting in the obtained text summary results ignoring some valid data.
[0061] In order to solve the technical problem that the accuracy of text summary results generated by the above-mentioned existing text analysis system is poor, so that the obtained text summary results deviate from the user's demand for data, thereby causing the obtained text summary results to ignore certain valid data, the present invention proposes a text generation method. The implementation details of the text generation method of this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.
[0062] Embodiment 1:
[0063] The text generation method of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 As shown, including:
[0064] Step 110, generating a sentence vector for each sentence in the original text and an original text vector corresponding to the original text, and setting a weight for each sentence vector.
[0065] Specifically, the original text contains text data extracted from a web page or website. After generating the sentence vectors of each sentence in the original text, the original text vector of the original text is generated based on the sentence vectors of each sentence in the original text. And the TextRank algorithm is used to configure the weight for each sentence vector so that each sentence vector has a corresponding weight value. In some examples, after obtaining the original text, the original text needs to be normalized, such as converting letters to lowercase letters, defining a custom vocabulary based on industry professional vocabulary, and avoiding the splitting of professional vocabulary.
[0066] Step 120 , based on the weight of the sentence vector of each sentence and a preset selection condition, multiple sentences are selected from the original text to form a first text, and a first text vector of the first text is generated.
[0067] Specifically, the preset selection conditions are used to define the prerequisites for selecting sentences in the original text. After weights are configured for each sentence vector so that each sentence vector has a corresponding weight value, multiple sentences that meet the selection conditions are selected from the original text based on the weights and selection conditions of the sentence vectors of each sentence, and a first text is formed based on the multiple sentences, and the sentence vectors of each sentence in the first text are converted into the first text vector of the first text.
[0068] Step 130, calculating a first similarity between the sentence vector of each sentence in the original text and the first text vector.
[0069] Specifically, after forming the first text and generating the first text vector of the first text, the first similarity between the sentence vector of each sentence in the original text and the first text vector is calculated by simulating t(D i ,S) is calculated, where D i is the sentence vector of the sentence to be selected in the original text, S is the first text vector, sim t (D i , S) is the first similarity between the sentence vector of the sentence to be selected and the first text vector.
[0070] Step 140, clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating the second similarity between each cluster vector and the original text vector.
[0071] Specifically, the sentence vectors of each sentence in the original text are clustered using a clustering method, each cluster contains at least one sentence vector, and at least one sentence vector in each cluster is converted into a cluster vector of each cluster. After obtaining the cluster vector of each cluster, the second similarity between each cluster vector and the original text vector is calculated. In some examples, the second similarity between each cluster vector and the original text vector is calculated by simulating D (N Di ,Q) is calculated, where N Di is the cluster vector to be selected, Q is the original text vector, sim D (N Di , Q) is the second similarity between the selected cluster vector and the original text vector.
[0072] Step 150 , based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster to which the sentence vector of each sentence belongs, multiple sentences are selected from the original text to form a summary text.
[0073] Specifically, the first similarity and the second similarity are weighted to obtain a comprehensive similarity between any candidate sentence and the original text. The comprehensive similarity is calculated by Sim1(α*sim t (D i ,S)+(1-α)*sim D (N Di ,Q)) is calculated, where sim t (D i , S) is the first similarity between the sentence vector of the sentence to be selected and the first text vector, sim D (N Di ,Q) is the second similarity between the selected cluster vector and the original text vector, α is a constant, and Sim1(α*sim t (D i ,S)+(1-α)*sim D (N Di,Q)) is the comprehensive similarity between any candidate sentence and the original text.
[0074] In this embodiment, after obtaining the comprehensive similarity, multiple sentences are selected from the original text according to the comprehensive similarity to form a summary text. In some examples, the summary text formed by the multiple sentences selected from the original text needs to meet the required summary length, or, when selecting the multiple sentences, all sentences in the original text have been considered.
[0075] In this embodiment, a sentence vector of each sentence in the original text and an original text vector corresponding to the original text are generated, and a weight is set for each sentence vector. Then, based on the weight of the sentence vector of each sentence and the preset selection condition, multiple sentences are selected from the original text to form a first text, and a first text vector of the first text is generated, and the first similarity between the sentence vector of each sentence in the original text and the first text vector is calculated. Then, the sentence vector of each sentence in the original text is clustered, a cluster vector corresponding to the sentence vector of each cluster is generated, and the second similarity between each cluster vector and the original text vector is calculated. Based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each sentence is located, multiple sentences are selected from the original text to form a summary text. In this way, based on the first similarity between the sentence vector of each sentence in the original text and the first text vector and the second similarity between each cluster vector and the original text vector, multiple sentences are selected from the original text to form a summary text, which effectively improves the accuracy of the summary text formation, making it more in line with the needs of users, so as to reduce the probability of ignoring some valid data when generating the summary text.
[0076] In some embodiments, clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating the second similarity between each cluster vector and the original text vector includes:
[0077] The sentence vectors of each sentence in the original text are divided into multiple clusters by using a clustering algorithm, and the average value of the sentence vectors of each cluster is calculated to determine the cluster vector, and the second similarity between the cluster vector and the original text vector is calculated.
[0078] Specifically, the clustering algorithm is used to divide the sentence vectors of each sentence in the original text into multiple clusters, so that the similarity of sentences in the same cluster is as large as possible, and the difference of sentences in different clusters is as large as possible. That is, sentences of the same category after being processed by the clustering algorithm are clustered together as much as possible, and sentences of different categories are separated as much as possible. After the sentence vectors of each sentence in the original text are clustered, the average value of the sentence vector of each cluster is calculated to obtain the average value of the sentence vector of each cluster, that is, the cluster vector, and then the average value of the sentence vector of each cluster is calculated by sim D(N Di ,Q) Calculate the second similarity between the cluster vector and the original text vector. The second similarity between the cluster vector and the original text vector is calculated by a clustering algorithm, so that the obtained second similarity is used as a criterion for selecting sentences from the original text, so that the selection of sentences is more accurate, and through clustering calculation, the amount of calculation is effectively reduced and the calculation efficiency is improved.
[0079] In some embodiments, based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster to which the sentence vector of each sentence belongs, multiple sentences are selected from the original text to form a summary text, including:
[0080] Based on the weight of the sentence vector of each sentence, the sentence corresponding to the sentence vector with the largest weight is selected from the original text as the first selected sentence to be used to form the summary text, and the remaining sentences in the original text except the selected sentences are recorded as candidate sentences;
[0081] Specifically, after weights are assigned to each sentence vector so that each sentence vector has a corresponding weight value, the sentence corresponding to the sentence vector with the largest weight value is selected from the original text as the first selected sentence to be used to form the summary text, that is, the sentence corresponding to the sentence vector with the largest weight is the starting sentence of the summary text. The remaining sentences in the original text except the selected sentences are candidate sentences, and the sentence located after the starting sentence in the summary text needs to be selected from the candidate sentences to form the summary text.
[0082] The process of selecting the selected sentences other than the first one selected from the original text for forming the summary text includes:
[0083] Calculate the third similarity between the text vector of the text formed by all the selected sentences and the sentence vector of each current candidate sentence;
[0084] Based on the first similarity corresponding to each current candidate sentence, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each current candidate sentence belongs, and the third similarity corresponding to each current candidate sentence, at least one candidate sentence is selected from the current candidate sentences as a non-first selected sentence for forming a summary text;
[0085] A summary text is formed based on all selected sentences.
[0086] Specifically, after selecting the sentence corresponding to the sentence vector with the largest weight as the starting sentence of the summary text, it is necessary to select the sentence in the summary text that is located after the starting sentence from the candidate sentences. Specifically, the process of selecting the selected sentence that is not the first selected sentence from the original text to form the summary text is as follows: the third similarity between the text vector of the text formed by all the selected sentences and the sentence vector of each current candidate sentence is calculated, and the third similarity between the text vector of the text formed by all the selected sentences and the sentence vector of each current candidate sentence is calculated by Calculated, where λ is a hyperparameter, D i is the sentence vector of the sentence to be selected in the original text, D j is the text vector of the text formed by all selected sentences, is the third similarity between the sentence vector of any candidate sentence and all selected sentences. In some examples, the value range of λ is [0,1], and the specific value of this hyperparameter can be defined by the user to balance the relevance between the sentence and the entire document and the diversity between sentences. λ is preferably set to 0.5.
[0087] In this embodiment, after the third similarity is calculated, based on the first similarity corresponding to each current candidate sentence, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each current candidate sentence belongs, and the third similarity corresponding to each current candidate sentence, the candidate sentence with higher similarity is calculated as the selected sentence that is not the first selected to form the summary text, and then the summary text is formed based on all the selected sentences. In some examples, the MMR value is calculated using the following relationship:
[0088]
[0089] In the formula, is the cluster vector to be selected, Q is the original text vector, is the second similarity between the cluster vector to be selected and the original text vector; sim t (D i , S) is the first similarity between the sentence vector of the sentence to be selected and the first text vector, is the second similarity between the selected cluster vector and the original text vector, α is a constant; λ is a hyperparameter, D i is the sentence vector of the sentence to be selected in the original text, D j is the text vector of the text formed by all selected sentences, is the third similarity between the sentence vector of any candidate sentence and all selected sentences, It is the difference between the sentence vector of any candidate sentence and all the selected sentences.
[0090] In this embodiment, based on the first similarity between the sentence vector of each sentence in the original text and the first text vector and the second similarity between each cluster vector and the original text vector, the first similarity and the second similarity obtained by combining the TextRank algorithm and the clustering algorithm are double-authenticated to improve the accuracy of the similarity result, reduce the amount of calculation, and improve efficiency. In some examples, when calculating the third similarity between any candidate sentence and the selected sentence, the user can pre-set the k value, and the specific k is the number of selected sentences to be selected for calculation in each calculation process. For example, the number of currently selected sentences is, and k is set to 3, then the last 3 selected sentences are selected to participate in the calculation part of the third similarity, thereby reducing the amount of calculation of this part, improving efficiency, and selecting the candidate sentence with the maximum similarity with the k selected sentences to maintain the diversity of the summary. In this way, based on the first similarity between the sentence vector of each sentence in the original text and the first text vector and the second similarity between each cluster vector and the original text vector, as well as the third similarity corresponding to each of the current candidate sentences, the first similarity and the second similarity obtained by combining the TextRank algorithm and the clustering algorithm are double-authenticated to improve the accuracy of the similarity results, reduce the amount of calculation, and improve calculation efficiency.
[0091] In this embodiment, the candidate sentences whose MMR values are greater than the preset threshold value calculated by the above relationship (1) are used as the selected sentences other than the first selected sentences for forming the summary text, and then the summary text is formed based on all the selected sentences. The summary text consists of the first selected sentence and at least one non-first selected sentence.
[0092] In some embodiments, a summary text is formed based on all selected sentences, including:
[0093] The first selected sentence selected for forming the summary text is used as the first sentence of the summary text;
[0094] For the selected sentences that are not the first ones selected to form the summary text, based on the first similarity corresponding to each selected sentence, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each selected sentence belongs, and the third similarity corresponding to each selected sentence, determine the order of each selected sentence in the summary text;
[0095] Based on the ranking of all selected sentences in the summary text, a summary text is formed.
[0096] Specifically, after selecting the sentence corresponding to the sentence vector with the largest weight as the first selected sentence to be used to form the summary text, the first selected sentence to be used to form the summary text is used as the first sentence of the summary text (i.e., the starting sentence). For the selected sentences that are not the first selected sentences to form the summary text, the first similarity corresponding to each selected sentence, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each selected sentence is located, and the third similarity corresponding to each selected sentence are substituted into the relational formula (1) to calculate, and the ranking of each selected sentence in the summary text is obtained. In some examples, the first similarity corresponding to each selected sentence, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each selected sentence is located, and the third similarity corresponding to each selected sentence are substituted into the relational formula (1) to calculate, and the MMR value of each selected sentence is obtained, and the ranking of each selected sentence in the summary text is determined according to the MMR value. For example, the selected sentence with the highest MMR value is the second sentence of the summary text, and so on, to determine the ranking of all selected sentences in the summary text. In this way, by defining the order of each selected sentence, a summary text is formed according to the order of the selected sentences, so that the formed summary text is more accurate.
[0097] In some embodiments, based on the weight of the sentence vector of each sentence and a preset selection condition, multiple sentences are selected from the original text to form a first text, and a first text vector of the first text is generated, including:
[0098] Get the keywords input by the user and detect whether the keywords exist in each sentence of the original text;
[0099] When detecting the presence of keywords in sentences in the original text, determining that the weight value of the sentence vector of the sentence in which the keywords exist in the original text is a first weight value, and the weight value of the sentence vector of the sentence in which the keywords do not exist in the original text is a second weight value, wherein the first weight value is greater than the second weight value, and the first weight value is positively correlated with the number of keywords contained in the sentence;
[0100] Based on the weight value of the sentence vector of each sentence in the original text, a plurality of sentences meeting the selection condition are selected from the original text to form a first text, and a first text vector of the first text is generated.
[0101] Specifically, the keywords input by the user are used to define the subject content of the summary text. It can be understood that sentences with keywords are more in line with the sentence selection requirements of the summary text. After obtaining the keywords input by the user, it is detected whether there are keywords in each sentence in the original text. When the keywords are detected in the sentences in the original text, the weight value of the sentence vector of the sentence with keywords in the original text is determined to be the first weight value, and the weight value of the sentence vector of the sentence without keywords in the original text is the second weight value, and the first weight value is greater than the second weight value. In other words, the sentence vector of the sentence containing keywords has a larger weight value and is more likely to be selected as the summary text. In addition, if a sentence contains multiple keywords, the first weight value of the sentence is greater than the first weight value of other sentences with fewer keywords. For example, sentence A contains 3 keywords and sentence B contains 1 keyword, then the first weight value of sentence A is greater than the first weight value of sentence B.
[0102] In this embodiment, according to the weight value of the sentence vector of each sentence in the original text, multiple sentences that meet the selection conditions are selected from the original text to form a first text. The preset selection condition is to select sentences with weight values greater than the preset weight value, or to select sentences with the top weight values. In some examples, after obtaining the weight value of the sentence vector of the sentence, the importance score of each sentence is iteratively calculated using the TextRank algorithm, and the corresponding sentences with the top M importance scores are selected to form the first text, and the first text vector of the first text is generated. In this way, through the user-defined keywords, the generated first text is more in line with the user's needs, so that the final summary text is more in line with the user's needs.
[0103] In some embodiments, generating a sentence vector for each sentence in the original text includes:
[0104] Acquire text data of a web page, and generate multiple original texts based on the text data;
[0105] The original text is segmented into sentences and words to obtain sentence vectors of multiple sentences, where the sentence vector of a sentence is composed of the word vectors of the words contained in the sentence.
[0106] Specifically, the text data of a certain web page can be collected, or the text data of a certain website can also be collected. By accessing the link of the web page, requests or selenium, or a combination of both can be used to obtain the return result of the network-marked text displayed on the web page corresponding to the link. Using selenium can simulate the browser method to display the corresponding text, which can effectively reduce the time consumed by parsing requests during data collection and improve the data collection efficiency. Then, by accessing the link url of other response result types of the web page, the return result of the corresponding link is obtained, and then the return result of the network-marked text is parsed and extracted together with the return result of the corresponding link. Finally, the text content obtained by extraction is filled into the corresponding data structure to generate the text data of the web page, and the text data of the web page is stored in the database.
[0107] In this embodiment, the text data of the web page is obtained from the database, and multiple original texts are generated based on the text data, and the original texts are normalized. Then, the normalized original document is segmented into sentences and words to obtain the sentence vectors of multiple sentences. Among them, the sentence vector of a sentence is composed of the word vectors of the words included in the sentence. In some examples, when segmenting the normalized original text into sentences and words, the original text is first segmented into sentences to obtain multiple sentences and their corresponding sentence vectors, and then the obtained sentences are segmented into words to obtain multiple words and their corresponding word vectors. When performing word segmentation, stop words are removed from the word segmentation results. Here, stop words are words or punctuation marks without actual representative meaning, such as "de" "le" "!" and so on. In this way, by segmenting the original text into sentences and words, the error of the obtained sentence vectors is small, which is convenient for the formation of the subsequent abstract text.
[0108] In some examples, when performing word segmentation, the bag-of-words model method is used, and the obtained words are all unique words, and the obtained words are converted into the representation form of word vectors to obtain the sentence vector of each sentence.
[0109] In some embodiments, obtaining the text data of the web page includes:
[0110] Performing data extraction on the text content of the web page to obtain the web page text content and the corresponding text type;
[0111] Filling the web page text content into a preset data structure, and performing label identification on the web page text content according to the text type, so as to obtain the text data of the web page.
[0112] Specifically, data is extracted from the text content of the web page to obtain the web page text content, and the data sources of the web page text content are different, so the text type of the web page text content needs to be obtained. After obtaining the web page text content, the web page text content is filled into the corresponding preset data structure according to the text type, and then the web page text content is labeled according to the text type, so as to understand the data source of the web page text content. In some examples, the theme label and sub-category label of the web page text content are specified according to the data source. Assuming that the website data comes from an e-commerce website, a government announcement website, etc., the data obtained from different websites can correspond to different themes (product information, policies, etc.). According to the content of the acquired website data, the corresponding theme label and sub-category label are set in the acquisition subroutine (the theme of the e-commerce website can be product information, and the sub-category label can be product specifications, product reviews, etc.), such as product name, specification, price, theme, sub-category, file / picture: storage path and other information. The data structure can be understood as follows {product name: **, specification: **, price: **, theme: **, sub-category: **, file / picture: storage path}. In this way, by labeling the webpage text content according to the text type, it is easy to understand the data source of the webpage text content.
[0113] Embodiment 2:
[0114] like Figure 4 and Figure 5 As shown, this embodiment provides exemplary content of Embodiment 1, that is, provides an exemplary process of a text generation method, and the specific content includes:
[0115] like Figure 6 As shown, the text generation system includes a data acquisition module, an exception handling module, a data storage module and a text summary generation module, wherein the data acquisition module and the exception handling module belong to the network data acquisition part, store the internal data of the enterprise in the data storage module, design a specific storage structure according to the above data, and specify the specified topic label and sub-category label at the same time. The data storage module is the repository of all data, and the user-defined parameters are used to obtain the user-defined weights, parameters and other data. The text summary generation module generates text summaries for the data selected by the user according to the user-defined weight vocabulary and parameter data, and feeds back the text summary generation results generated in the previous step to the user.
[0116] Specifically, the data collection module is composed of k network data collection programs. A subprogram can be updated or offline at any time, and multiple data collection programs can be added at any time. When processing a subprogram, other tasks are not affected, so more data information can be obtained in the network. The exception handling module means that when a data collection subprogram fails to obtain data, it can prompt the developer of the program abnormality and update the abnormal collection subprogram in time. The text summary generation module extracts the text content in the database selected by the user to form a document (i.e., the original text), processes the document, uses the graph-based TextRank algorithm and uses the user-defined weight vocabulary, parameters and other information to generate the corresponding text summary result, and returns the result to the user.
[0117] like Figure 4 As shown, the process of obtaining the text data of a web page is as follows:
[0118] Step 101, specify a website / webpage to be collected;
[0119] Step 102 refers to accessing the link of the website; this access method can use requests or selenium, or a combination of the two, to obtain the return result of the network markup text displayed on the website page corresponding to the link. The page text of some websites may be returned to the browser by multiple network requests. Using selenium can simulate the browser method to display the corresponding text, which can effectively reduce the time spent on parsing requests during data collection and improve data collection efficiency;
[0120] Step 103: Access other response result type link URLs of the website, such as text, JSON, binary files, etc., to obtain the return result of the corresponding link;
[0121] Step 104, parsing and extracting the results in step 102 and step 103;
[0122] Step 105, fill the content obtained in step 104 into the corresponding data structure, and specify the subject label and sub-category label of the data according to the data source. The data source here refers to website data, such as e-commerce websites, government announcement websites, etc. The data obtained from different websites can correspond to different topics (product information, policies, etc.). According to the content of the website data obtained, the corresponding subject label and sub-category label are set in the acquisition subroutine (the subject of the e-commerce website can be product information, and the sub-category label can be product specifications, product reviews, etc.), such as product name, specifications, price, theme, sub-category, file / picture: storage path and other information. The data structure can be understood as follows {product name: **, specifications: **, price: **, theme: **, sub-category: **, file / picture: storage path};
[0123] Step 106: Store the data results generated in step 105 into a database.
[0124] Specifically, the data of step 102 can be extracted in a variety of ways, such as the attribute value of the web page tag (ID, NAME, CLASSNAME), or it can be located using rules such as xpath, css, and fuzzy matching. The response content of step 103 can be obtained by other means. The text text can be directly used or regular expressions can be used to select the target content. The json text can use a json parser to extract the target field. For example, a response json text is {'id':'****','name':'**','price':**}. Here, only the name and price data contents need to be selected. The return response content corresponding to the binary file is a string of binary encoded strings. We can get the corresponding picture or other file by creating a new file object and storing this binary string. For the text content of the page, valuable and meaningful texts are extracted respectively to generate the data structure corresponding to the website, such as the e-commerce product page, and the name, price, brand, specification description and other data of the product are extracted. If there is image data or file data, the image data or file data can be stored in the file system, and the corresponding storage path is recorded. It should be noted that step 104 is to extract content, which may obtain content (product name, price, brand, specification description, pictures, files) through multiple network requests, specifically referring to valuable data content, and step 105 assembles these contents into a complete set data structure.
[0125] like Figure 5 As shown in the figure, the process of forming the summary text is:
[0126] Step 301: extracting the data selected by the user from the data repository, and combining the text data and the text file content into several documents (i.e., original text);
[0127] Step 302: Preprocess the document generated in Step 301. Normalize the document by converting all letters to lowercase. Define a custom vocabulary based on industry-specific terms to prevent professional terms from being split. Use the Jieba Chinese tool to process the document, load the above custom vocabulary, split the document into sentences and words, and remove stop words from the word segmentation results. Here, stop words are words or punctuation marks with no actual representative meaning, such as "的" (de), "了" (le), "!" (!), etc.;
[0128] Step 303: For the word segmentation results processed in Step 302, use the bag-of-words model method. The unique words (after removing duplicates) of all word segmentation results are used as the vocabulary. Convert each sentence in Step 302 into a word vector representation to obtain the word vector of each sentence;
[0129] Step 304: Use the word vector results in the result of Step 303 to construct a graph. Each sentence is used as a node, calculate the similarity of each sentence to construct edges. According to the user-defined keyword list obtained, if a sentence contains keywords in the vocabulary, the initial weight value of the sentence is correspondingly adjusted and increased. Then use the TextRank algorithm to iteratively calculate the importance score of each sentence. Sentences containing keywords are more likely to be selected as important sentences;
[0130] Step 305: Select the first M sentences obtained in Step 304 to form a new document S (i.e., the first text) and convert it into a word vector representation of the document;
[0131] Step 306: Use the clustering method (DBSCAN) to divide the sentence vectors in Step 303 into N clusters. Calculate the average value of the sentence vectors contained in each cluster as the cluster center vector. Calculate the similarity between this cluster center vector and the document vector. Here, the similarity between each cluster N and the original document Q represents the similarity between all sentence vectors in the cluster and the original document, which is applied to the simD part in Sim1 to reduce the computational amount each time;
[0132] Step 307, select the sentence with the highest score in step 304 as the starting sentence of the summary, apply the MMR formula, iteratively calculate and select the next sentence, select the sentence with the highest MMR score and add it to the summary, where λ is a hyperparameter with a value range of [0,1]. The specific value of this parameter can be defined by the user to balance the relevance of the sentence and the entire document and the diversity between sentences. The default value is 0.5. Sim1 represents the similarity between any sentence to be selected and the document, which is obtained by weighted calculation of the similarity between any sentence to be selected and the document S in step 350 and the similarity between the cluster where the sentence to be selected belongs and the original document Q. The weighted result of the two similarities is used as the result of the Sim1 part, which can be combined with the results of textrank and clustering, and double authentication can be performed to improve the accuracy of the similarity result, reduce the amount of calculation, and improve efficiency. Sim2 represents the similarity between any sentence to be selected and the most recent k selected summary sentences. The k most recent selected sentences refer to, for example, 5 sentences have been selected so far. Assuming k is 3, the last 3 selected sentences are selected to participate in the calculation part of sim2, so as to reduce the amount of calculation in this part and improve efficiency. The maximum similarity between the sentence to be selected and the k summary sentences is selected to maintain the diversity of the summary. Thus, the MMR score of each sentence to be selected can be obtained and the sentence with the highest score is selected. The sentence with the highest MMR calculation score is the summary sentence selected for this step and added to the summary sentence set. At this time, the sentence to be selected and the selected summary sentence have changed. If the number of summary sentences does not meet the number of results, the MMR is calculated again to select the sentence with the highest score to be selected and put into the summary sentence.
[0133] Step 308, repeat the calculation in step 307 and select the next sentence until the required summary length is met or all sentences have been considered;
[0134] Step 309: combine all selected summary sentences into a summary text in the order of selection and return it.
[0135] In this embodiment, a summary generation method is customized based on user needs. The special point is that the user can customize the keyword table and calculation parameters, adjust the initial weight value of the sentence to calculate the sentence score, so that the summary generation result is biased towards the user's search target. In the step of selecting the summary sentence, the sentence with the highest textrank score is used as the starting sentence, and the first M highest-scoring sentences are converted into a new document S. The clustering calculation result is added, and the similarity between the sentence and the document S and the similarity between the cluster cluster and the original document Q are combined. The user-defined MMR parameter value is used to make the generated summary result more in line with the user's search needs.
[0136] In this embodiment, a method of constructing multiple subtasks corresponds to the design of corresponding collection subroutines for different websites, which can handle abnormal problems of a certain subtask separately, effectively avoid affecting the overall collection task, and store the corresponding storage structure of the enterprise's internal specific information data synchronously with the collected data. The user can select batch data, define the keyword table, parameters and other information required for summary generation, and generate corresponding text summary results for users to use, thereby meeting the user's analysis needs for specific subject texts, realizing effective use of data, and improving the efficiency of information retrieval.
[0137] Embodiment three:
[0138] Another embodiment of the present application relates to a text generation device. The implementation details of the text generation device of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details, and is not necessary for the implementation of this solution. The schematic diagram of the text generation device of this embodiment can be as follows Figure 2 As shown, including:
[0139] A sentence vector generation module 210 is used to generate a sentence vector for each sentence in the original text and an original text vector corresponding to the original text, and to set a weight for each sentence vector;
[0140] A first text forming module 220, configured to select a plurality of sentences from the original text to form a first text based on the weight of the sentence vector of each sentence and a preset selection condition, and generate a first text vector of the first text;
[0141] A first similarity calculation module 230, used to calculate a first similarity between a sentence vector of each sentence in the original text and the first text vector;
[0142] A second similarity calculation module 240, for clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating a second similarity between each cluster vector and the original text vector;
[0143] The summary text forming module 250 is used to select multiple sentences from the original text to form a summary text based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each sentence belongs.
[0144] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.
[0145] Embodiment 4:
[0146] Another embodiment of the present application relates to an electronic device, such as Figure 3 As shown, it includes: at least one processor 901; and a memory 902 that is communicatively connected to the at least one processor 901; wherein the memory 902 stores instructions that can be executed by the at least one processor 901, and the instructions are executed by the at least one processor 901 so that the at least one processor 901 can execute the text generation method in the above-mentioned embodiments.
[0147] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.
[0148] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.
[0149] Embodiment five:
[0150] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.
[0151] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.
[0152] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.
Claims
1. A text generation method, characterized in that: include: Generate a sentence vector for each sentence in the original text, and an original text vector corresponding to the original text, and set a weight for each of the sentence vectors; Based on the weights of the sentence vectors of the sentences and a preset selection condition, selecting a plurality of sentences from the original text to form a first text, and generating a first text vector of the first text; Calculating a first similarity between a sentence vector of each sentence in the original text and the first text vector; Clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating a second similarity between each cluster vector and the original text vector; Based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each sentence belongs, multiple sentences are selected from the original text to form a summary text.
2. The text generation method according to claim 1, characterized in that: The clustering of the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating the second similarity between each cluster vector and the original text vector includes: The sentence vectors of each sentence in the original text are divided into a plurality of clusters by using a clustering algorithm, and an average value of the sentence vectors of each cluster is calculated to determine the cluster vector, and a second similarity between the cluster vector and the original text vector is calculated.
3. The text generation method according to claim 1, characterized in that: The method of selecting a plurality of sentences from the original text to form a summary text based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each sentence belongs to comprises: Based on the weights of the sentence vectors of the sentences, the sentence corresponding to the sentence vector with the largest weight is selected from the original text as the first selected sentence to be used to form the summary text, and the remaining sentences in the original text except the selected sentence are recorded as candidate sentences; The process of selecting a sentence other than the first one selected from the original text for forming the summary text comprises: Calculating a third similarity between the text vector of the text formed by all the selected sentences and the sentence vector of each current candidate sentence; Based on the first similarity corresponding to each of the candidate sentences, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each of the candidate sentences is located, and the third similarity corresponding to each of the candidate sentences, at least one candidate sentence is selected from the candidate sentences as the non-first selected sentence for forming the summary text; The summary text is formed based on all the selected sentences.
4. The text generation method according to claim 3, characterized in that: The forming of the summary text based on all the selected sentences comprises: The first selected sentence selected to form the summary text is used as the first sentence of the summary text; For the selected sentences other than the first one selected to form the summary text, determining the order of each of the selected sentences in the summary text based on the first similarity corresponding to each of the selected sentences, the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each of the selected sentences belongs, and the third similarity corresponding to each of the selected sentences; The summary text is formed based on the ranking of all the selected sentences in the summary text.
5. The text generation method according to claim 1, characterized in that: The step of selecting a plurality of sentences from the original text to form a first text based on the weights of the sentence vectors of the sentences and a preset selection condition, and generating a first text vector of the first text includes: Obtaining keywords input by the user, and detecting whether the keywords exist in each sentence of the original text; When detecting that the keyword exists in a sentence in the original text, determining that the weight value of the sentence vector of the sentence in which the keyword exists in the original text is a first weight value, and the weight value of the sentence vector of the sentence in which the keyword does not exist in the original text is a second weight value, wherein the first weight value is greater than the second weight value, and the first weight value is positively correlated with the number of the keywords contained in the sentence; Based on the weight value of the sentence vector of each sentence in the original text, multiple sentences meeting the selection condition are selected from the original text to form a first text, and a first text vector of the first text is generated.
6. The text generation method according to claim 1, characterized in that: The generating of sentence vectors of each sentence in the original text includes: Acquire text data of a web page, and generate a plurality of original texts based on the text data; The original text is segmented into sentences and words to obtain sentence vectors of the plurality of sentences, wherein the sentence vectors of the sentences are composed of word vectors of the words contained in the sentences.
7. The text generation method according to claim 6, characterized in that: The step of obtaining text data of a web page includes: Extracting data from the text content of the webpage to obtain the webpage text content and the corresponding text type; The webpage text content is filled into a preset data structure, and the webpage text content is labeled according to the text type, so as to obtain the text data of the webpage.
8. A text generation device, characterized in that: include: A sentence vector generation module, used to generate a sentence vector for each sentence in the original text, and an original text vector corresponding to the original text, and set a weight for each sentence vector; A first text forming module, configured to select a plurality of sentences from the original text to form a first text based on the weights of the sentence vectors of the sentences and a preset selection condition, and generate a first text vector of the first text; A first similarity calculation module, used for calculating a first similarity between a sentence vector of each sentence in the original text and the first text vector; A second similarity calculation module, used for clustering the sentence vectors of each sentence in the original text, generating a cluster vector corresponding to the sentence vector of each cluster, and calculating a second similarity between each cluster vector and the original text vector; The summary text forming module is used to select multiple sentences from the original text to form a summary text based on the first similarity corresponding to each sentence in the original text and the second similarity corresponding to the cluster vector of the cluster where the sentence vector of each sentence belongs.
9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the text generation method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text generation method according to any one of claims 1 to 7 is implemented.