Text extraction method and device based on hybrid compression, equipment and medium

Abstract

The invention discloses a text extraction method and device based on hybrid compression, equipment and a medium. The method comprises the steps that a first text set of to-be-compressed texts is obtained; performing semantic expansion and replacement on paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets; respectively compressing and clipping the paragraph set of the first text set and the paragraph set of the second text set according to a mixed compression ratio to obtain a plurality of third text sets; screening the plurality of third text sets to obtain a plurality of fourth text sets; according to the to-be-compressed text, the paragraph sets of the fourth text sets and the number of words of the paragraph sets, compressed extraction texts corresponding to the to-be-compressed text are screened out from the fourth text sets. The invention belongs to the field of text compression. According to the method, text compression can be guaranteed, and meanwhile semantic closeness is achieved.

Description

Technical Field

[0001] The present invention relates to the field of text compression, and in particular to a text extraction method, device, equipment and medium based on hybrid compression. Background Art

[0002] Text compression refers to the process of making the text shorter and more concise by streamlining, rewriting or deleting redundant content while retaining the core semantics and key information of the original text as much as possible. It is not simply reducing the number of words, but purposefully refining information to make the content more compact but still readable and expressive.

[0003] If the compressed content is too much, the original text will be lost; if the compressed content is too little, the compression requirements will not be met. How to achieve a balance between text compression effect and text semantic effect is an urgent problem to be solved. Summary of the invention

[0004] The present invention solves the technical problem of being unable to balance the text compression effect and the text semantic effect in the prior art by providing a text extraction method, device, equipment and medium based on hybrid compression, and achieves the technical effect of balancing the text compression effect and the text semantic effect.

[0005] In a first aspect, the present invention provides a text extraction method based on hybrid compression, comprising: Determine the text to be compressed, divide the text to be compressed into paragraphs and extract paragraph keywords to obtain a first text set of the text to be compressed, the first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set; Based on a preset semantic replacement and expansion model, semantic expansion and replacement are performed on paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets; Determine a mixed compression ratio, and compress and trim the paragraph set of the first text set and the paragraph set of the second text set respectively with the mixed compression ratio to obtain a plurality of third text sets, each of which includes a compressed paragraph set and a paragraph keyword set; According to the third text set, the paragraph set in the first text set and the weight distribution of each paragraph in the first text set, the plurality of third text sets are screened to obtain a plurality of fourth text sets, wherein the fourth text sets include a paragraph set and a paragraph keyword set; According to the text to be compressed, the paragraph set of the fourth text set and the number of words in the paragraph set, the compressed extracted text corresponding to the text to be compressed is screened out from the plurality of fourth text sets.

[0006] Further, according to the third text set, the paragraph set in the first text set, and the weight distribution of each paragraph in the first text set, several third text sets are screened to obtain several fourth text sets, including: Compare the paragraphs in the third text set with the corresponding paragraphs in the first text set to obtain the paragraph similarity of the paragraph in the third text set; Determine the paragraph keyword coverage of the paragraph according to the paragraph in the third text set and the corresponding paragraph keyword set; Determine the weight of each paragraph in the third text set according to the weight distribution of each paragraph in the first text set; Determine the mixed compression value of the paragraph according to the paragraph similarity, paragraph keyword coverage, and weight of the paragraph in the third text set; Determine the mixed compression value of the third text set according to the mixed compression values of the paragraphs in the third text set; Sort each third text set according to the mixed compression value of each third text set, and select the third text sets before the preset position as the fourth text set.

[0007] Further, the calculation of the mixed compression value includes:

[0008]

[0009]

[0010] Among them, is the mixed compression value of the th paragraph of the th third text set, is the weight of the th paragraph of the first text set, is the th paragraph similarity of the th paragraph of the and are both preset weights, and and sum to 1, is the th paragraph keyword coverage of the th paragraph of the is the th paragraph keyword total number of the th paragraph keyword set in the is the th paragraph of the The number of paragraph keywords in a paragraph, is the mixed compression value of the th third text set,

[0011] Furthermore, according to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set, the compression extraction text corresponding to the text to be compressed is screened from several fourth text sets, including: Merge the paragraphs in the paragraph set of the fourth text set in order, and delete the paragraph keyword set in the fourth text set to obtain the fourth text corresponding to the fourth text set; Perform semantic similarity analysis on the fourth text and the text to be compressed to obtain the text similarity of the fourth text; According to the number of words in the paragraph of the fourth text set, the paragraph weight, and the number of words in the corresponding paragraph in the first text set, determine the numerical value of the number of words in the paragraph of the fourth text set; According to the word count ratio of each paragraph in the fourth text set and the text similarity of the fourth text, determine the text value of the fourth text; Use the fourth text corresponding to the highest text value as the compression extraction text corresponding to the text to be compressed.

[0012] Furthermore, the calculation of the text value includes:

[0013] Among them, is the text value of the th fourth text, is the number of words in the th paragraph in the first text set, is the th number of words in the and are both preset weights, and and sum to 1.

[0014] Furthermore, compress and trim the paragraph sets of the first text set and the second text set respectively with the mixed compression ratio to obtain several third text sets, including: Arbitrarily cut the paragraphs in the first text set and the paragraphs in the second text set with a mixed compression ratio, so that the total number of words in several paragraphs of the first text set after cutting and the total number of words in several paragraphs of the second text set after cutting meet the mixed compression ratio.

[0015] Further, perform paragraph division and paragraph keyword extraction on the text to be compressed to obtain a first text set of the text to be compressed, including: Perform paragraph division on the text to be compressed with a preset punctuation mark to obtain several paragraphs and a paragraph set composed of several paragraphs; After the paragraph division is completed, use a preset keyword extraction model to perform paragraph keyword extraction on each paragraph of the text to be compressed to obtain a paragraph keyword set corresponding to the paragraph.

[0016] In a second aspect, the present invention provides a text extraction device based on hybrid compression, including: A division and extraction module, configured to determine the text to be compressed, perform paragraph division and paragraph keyword extraction on the text to be compressed to obtain a first text set of the text to be compressed, where the first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set; A replacement module, configured to perform semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set based on a preset semantic replacement and expansion model to obtain several second text sets; A hybrid compression module, configured to determine a hybrid compression ratio, and perform compression and cutting on the paragraph sets of the first text set and the paragraph sets of the second text set respectively with the hybrid compression ratio to obtain several third text sets, and each third text set includes a compressed paragraph set and a paragraph keyword set; A screening module, configured to screen several third text sets according to the third text sets, the paragraph sets in the first text set, and the weight distribution of each paragraph in the first text set to obtain several fourth text sets, and the fourth text sets include a paragraph set and a paragraph keyword set; A text extraction module, configured to screen out the compressed extraction text corresponding to the text to be compressed from several fourth text sets according to the text to be compressed, the paragraph sets of the fourth text sets, and the number of words in the paragraph sets.

[0017] In a third aspect, the present invention provides an electronic device, including: A processor; A memory for storing processor-executable instructions; Wherein, the processor is configured to execute to implement the text extraction method based on hybrid compression provided in the first aspect.

[0018] Fourthly, the present invention provides a non-transitory computer-readable storage medium. When the instructions in the non-transitory computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the text extraction method based on hybrid compression provided in the first aspect.

[0019] One or more technical solutions provided in the present invention have at least the following technical effects or advantages: The present invention provides a large number of reference documents with semantically similar possibilities for the text to be compressed through the second text set obtained by text expansion and the third text obtained by arbitrary tailoring. Then, the hybrid compression value and the text value are screened twice, ensuring the maximization of the compression effect while retaining the core semantics.

[0020] The present invention arbitrarily tailors the paragraphs in the first text set and the paragraphs in the second text set with a hybrid compression ratio, realizing a huge expansion of the text volume and providing text volume support for the extraction of the optimal compressed text.

[0021] The present invention realizes the quantitative evaluation of the compression effect based on the hybrid compression value, which is convenient for comparison and selection between different compression results. The higher the hybrid compression value, the more it indicates that while reducing the text length, no key information is lost, improving the readability and practicality of the compressed text. The hybrid compression value comprehensively evaluates the quality of the compressed text through three dimensions: paragraph similarity, keyword coverage, and paragraph weight, reflecting a comprehensive consideration of the semantic retention degree, the integrity of key information, and the content importance priority during the compression process.

[0022] The present invention improves the semantic similarity between the compressed text and the text to be compressed based on a two-fold comparison method of comparing the semantics of the compressed paragraphs and the semantics of the compressed text. While ensuring that the compression and tailoring are satisfied, the text semantics are made closest.

[0023] The present invention reflects the compression and tailoring effect of the paragraph when ensuring semantic proximity by introducing the word count value of the paragraph, and maximizes the compression and tailoring by introducing the word count value of the paragraph.

[0024] The present invention avoids the loss of core information caused by simply pursuing word count compression by introducing double constraints of semantic similarity and structural ratio. The text value provided by the present invention comprehensively considers two key factors: the overall semantic similarity and the paragraph word count ratio, reflecting the semantic retention ability of the fourth text for the original content and the matching degree of the information distribution during the compression or reconstruction process. Description of the Drawings

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0026] Figure 1 It is a schematic flowchart of the text extraction method based on hybrid compression of the present invention; Figure 2 It is a schematic structural diagram of the text extraction device based on hybrid compression of the present invention. Detailed implementation manners

[0027] The embodiments of the present invention provide a text extraction method based on hybrid compression, which solves the technical problem in the prior art that it is impossible to balance the text compression effect and the text semantic effect.

[0028] The technical solution of the present invention to solve the above technical problem has the following general idea: The text extraction method based on hybrid compression includes: determining the text to be compressed, performing paragraph division and paragraph keyword extraction on the text to be compressed to obtain a first text set of the text to be compressed. The first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set; based on a preset semantic replacement and expansion model, performing semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set to obtain a number of second text sets; determining the hybrid compression ratio, and respectively compressing and trimming the paragraph sets of the first text set and the paragraph sets of the second text sets with the hybrid compression ratio to obtain a number of third text sets. Each third text set includes a compressed paragraph set and a paragraph keyword set; screening a number of third text sets according to the third text sets, the paragraph sets in the first text set, and the weight distribution of each paragraph in the first text set to obtain a number of fourth text sets. The fourth text set includes a paragraph set and a paragraph keyword set; screening out the compressed extraction text corresponding to the text to be compressed from a number of fourth text sets according to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set.

[0029] To better understand the above technical solution, the following will detail the above technical solution in combination with the accompanying drawings of the specification and specific implementation manners.

[0030] First, it should be noted that the term "and / or" appearing in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the front and rear associated objects.

[0031] The present invention provides a text extraction method based on hybrid compression as shown in Figure 1 and includes steps S11 - S15: Step S11: Determine the text to be compressed, perform paragraph division and paragraph keyword extraction on the text to be compressed, and obtain a first text set of the text to be compressed. The first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set.

[0032] Specifically, it includes: using a preset punctuation mark to divide the text to be compressed into several paragraphs and obtaining a paragraph set composed of several paragraphs; after completing the paragraph division, using a preset keyword extraction model to extract paragraph keywords for each paragraph of the text to be compressed, and obtaining a paragraph keyword set corresponding to the paragraph.

[0033] The objective of step S11 is to structurally process an original text to be compressed, divide it into multiple paragraphs, and extract a set of keywords for each paragraph, thereby constructing a data set (referred to as the first text set) containing paragraphs and corresponding keywords.

[0034] The text to be compressed refers to selecting a paragraph of original text that needs to be compressed from the input. For example: text content such as a news article, an essay, a report, etc.

[0035] Preset punctuation marks (such as a period, a question mark, an exclamation mark, or a line break) can be used as delimiters to cut the entire text into several paragraphs.

[0036] After completing the paragraph division, use a preset keyword extraction model (such as TF - IDF, TextRank, or a BERT - based keyword extraction model, etc.) to extract keywords for each paragraph respectively. One paragraph corresponds to one paragraph keyword set. In addition to using the keyword extraction model, in special fields (such as medicine), screening can also be performed based on the work experience of relevant personnel.

[0037] For example: Content of paragraph 1: "The weather is very good today. We went for a walk in the park." The extracted keywords may be: [weather, park, walk], then [weather, park, walk] is the paragraph keyword set of paragraph 1. The paragraph keyword set contains at least one paragraph keyword, and the paragraph set is composed of all the divided paragraphs.

[0038] It should also be noted that the divided paragraphs are arranged in order, that is, all the following paragraph sets are also arranged in order. That is, the paragraph sets of the first text set to the fourth text set below are all arranged in order.

[0039] Step S12, based on a preset semantic replacement and expansion model, perform semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set to obtain a number of second text sets.

[0040] For each paragraph and its corresponding paragraph keywords, use the semantic replacement and expansion model to find words or phrases with similar semantics. The preset semantic replacement and expansion model can be a Word2Vec, GloVe, or BERT model, etc.

[0041] It can be understood that when performing semantic expansion and replacement on a paragraph, the number of words in the paragraph being semantically expanded and replaced may change.

[0042] For example: Content of paragraph 1: "The weather is nice today. We went for a walk in the park." After replacement: "The climate is pleasant today, suitable for outdoor activities, so we went for a stroll in the city green space." Or, "The sky is clear and the temperature is suitable. The family went for a walk in the natural scenic area outside the city." Or, "The weather condition is excellent. My friend and I chose to take a leisurely walk in the botanical garden." It can be understood that the paragraph keywords of paragraph 1 are also significantly different after semantic expansion and replacement. For example, if the content of paragraph 1 after replacement is: "The weather condition is excellent. My friend and I chose to take a leisurely walk in the botanical garden.", then the set of paragraph keywords can be {weather, botanical garden, me, friend, walk}.

[0043] In addition, it is also possible to first perform semantic expansion and replacement on the paragraph keywords in the original paragraph 1, and then expand paragraph 1 according to the semantically expanded and replaced paragraph keywords.

[0044] For example: Content of paragraph 1: "The weather is nice today. We went for a walk in the park."; Set of paragraph keywords: ["weather", "park", "walk"]; Then the set of replaced paragraph keywords can be: {climate, green space, walk}, {meteorology, scenic area, stroll}, etc.; Taking {climate, green space, walk} as an example: Paragraph 1 after replacement can be "The climate is pleasant today. We went to the green space for a relaxing walk." After the above semantic expansion and replacement operations, the first text set is converted into multiple possible versions, each version representing a different but semantically possibly related form of expression.

[0045] Step S13: Determine the hybrid compression ratio, and compress and trim the paragraph sets of the first text set and the second text set respectively according to the hybrid compression ratio to obtain a number of third text sets. Each third text set includes a compressed paragraph set and a paragraph keyword set.

[0046] Specifically, it includes: arbitrarily trimming the paragraphs in the first text set and the paragraphs in the second text set according to the hybrid compression ratio, so that the total number of words of several paragraphs in the trimmed first text set and the total number of words of several paragraphs in the trimmed second text set meet the hybrid compression ratio.

[0047] The first text set is composed of paragraphs before semantic expansion and replacement and their corresponding paragraph keyword sets; the second text set is composed of paragraphs obtained through semantic expansion and replacement and their corresponding paragraph keyword sets. A paragraph in the second text set also corresponds to a paragraph keyword set.

[0048] The hybrid compression ratio is a preset trimming ratio. For example, if the hybrid compression ratio is 60%, then it is necessary to trim 60% of the word count of the paragraph sets of the second text set and the first text set. The hybrid compression ratio can be determined according to actual needs.

[0049] It should be emphasized that the trimming method is random trimming. That is to say, it is possible that a certain paragraph is completely trimmed. All paragraph keyword sets do not participate in trimming, and in the case where a certain paragraph is completely trimmed, it still has a paragraph number, but its content is trimmed and compressed.

[0050] For example: Content of Paragraph 1: "The weather is very good today. We went for a walk in the park." Paragraph keyword set of Paragraph 1: [weather, park, walk] If Paragraph 1 is completely compressed and trimmed, then Paragraph 1 cannot have content, but Paragraph 1 still exists, and the paragraph keyword set of Paragraph 1 is still [weather, park, walk] Still taking Paragraph 1 as an example, Paragraph 1 may not be compressed and trimmed at all. For example: "The weather is good, went for a walk in the park." It can be understood that as long as the word count of the paragraph set of the first text set corresponding to Paragraph 1 meets the remaining 40%.

[0051] It can be understood that since it is random trimming and compression, the paragraph sets of the first text set and the second text set can be compressed and trimmed several times according to the hybrid compression ratio.

[0052] That is to say, when the paragraph set of the first text set is compressed and trimmed 10 times, 10 different third text sets can be obtained (it can be specified that the content of the compression and trimming is different from the previous time). Similarly, for the second text set, the number of times of randomly trimming and compressing the same text set may be determined according to the actual situation. For example, it can be set to 5000 times, 10000 times, 20000 times, 30000 times or 50000 times, etc.

[0053] For example, if there is 1 first text set and 10 second text sets, and the number of times of randomly trimming and compressing is 5000 times, a total of 55000 first text sets are obtained.

[0054] After compression and trimming, several third text sets are obtained. Each third text set includes a compressed paragraph set and a paragraph keyword set. A compressed paragraph also corresponds to a paragraph keyword set.

[0055] The present invention arbitrarily trims the paragraphs in the first text set and the paragraphs in the second text set with a mixed compression ratio, realizing a huge expansion of the text volume and providing text volume support for the extraction of the optimal compressed text.

[0056] Step S14: According to the third text set, the paragraph set in the first text set, and the weight distribution of each paragraph in the first text set, screen several third text sets to obtain several fourth text sets. The fourth text set includes a paragraph set and a paragraph keyword set.

[0057] Specifically, it includes: Compare the paragraphs in the third text set with the corresponding paragraphs in the first text set to obtain the paragraph similarity of the paragraph in the third text set.

[0058] It can be understood that since the paragraphs in the first text set and the third text set are both arranged in order, that is to say, each paragraph in the first text set and the third text set can correspond one by one (even if the paragraph in the third text set has no content).

[0059] For example: Paragraph 1 of a certain third text set: "The weather is good today. Went to the park for a walk.", corresponding to Paragraph 1 of the first text set: "The weather is very good today. We went for a walk in the park." Also for example: Paragraph 1 of a certain third text set has no content, which is: "", and still corresponds to Paragraph 1 of the first text set: "The weather is very good today. We went for a walk in the park." In addition to the paragraphs corresponding one by one, the paragraph keyword sets also correspond one by one (but the paragraph keywords are not involved in the calculation of the paragraph similarity).

[0060] For example: The set of paragraph keywords corresponding to paragraph 1 of a certain third text set is: {climate, green space, walking}, corresponding to the set of paragraph keywords of paragraph 1 of the first text set: [weather, park, walking].

[0061] Paragraphs can be converted into vector representations (such as using models like BERT, Sentence - BERT, etc.), and then the vector similarity (such as cosine similarity) between two paragraphs can be calculated to obtain the paragraph similarity of a certain paragraph in the third text set.

[0062] Each paragraph in each third text set needs to be compared for similarity with the corresponding paragraph in the first text set.

[0063] According to the set of paragraph keywords corresponding to a paragraph in the third text set, determine the paragraph keyword coverage of this paragraph.

[0064] The paragraph keyword coverage reflects the existence rate of keywords in the paragraph after compression and pruning compared to the paragraph before compression and pruning.

[0065] For example: The set of paragraph keywords corresponding to paragraph 1 of a certain third text set is: [weather, park, walking]; paragraph 1 of a certain third text set: "The weather is good today. Went to the park for a walk.", then there are no paragraph keywords in paragraph 1 of the third text set, so the paragraph keyword coverage is 0 / 3 = 0%.

[0066] Each paragraph in each third text set needs to be compared with the set of paragraph keywords corresponding to this paragraph in this third text set, and the paragraph keyword coverage of this paragraph is obtained.

[0067] According to the weight distribution of each paragraph in the first text set, determine the weights of each paragraph in the third text set.

[0068] The weight distribution of each paragraph in the first text set can be set according to the work experience of relevant personnel. The greater the weight of a certain paragraph, the more important this paragraph is in the first text set.

[0069] For example: There are a total of 10 paragraphs in the first text set, and the weight distribution is {0.1, 0.15, 0.05, 0.28, 0.34, 0.04, 0.01, 0.01, 0.01, 0.01}, and the sum of the weights of each paragraph is 1.

[0070] According to the paragraph similarity, paragraph keyword coverage, and weight of a paragraph in the third text set, determine the mixed compression value of this paragraph.

[0071] The calculation of the mixed compression value includes:

[0072]

[0073]

[0074] wherein, is the mixed compression value of the th paragraph of the th third text set, is the weight of the th paragraph of the first text set, is the paragraph similarity of the th paragraph of the th third text set, and are both preset weights, and and sum to 1, is the paragraph keyword coverage of the th paragraph of the th third text set, is the total number of paragraph keywords in the paragraph keyword set of the th paragraph in the th third text set, is the number of paragraph keywords in the th paragraph of the th third text set, is the mixed compression value of the th third text set, is the number of paragraphs in the th third text set.

[0075] The mixed compression value comprehensively evaluates the quality of the compressed text through three dimensions: paragraph similarity, keyword coverage, and paragraph weight, reflecting a comprehensive consideration of the degree of semantic retention, the integrity of key information, and the priority of content importance during the compression process.

[0076] Based on the mixed compression value, the present invention realizes a quantitative evaluation of the compression effect, which is convenient for comparison and selection between different compression results. The higher the mixed compression value, the more it indicates that while reducing the text length, key information is not lost, improving the readability and practicality of the compressed text.

[0077] Determine the mixed compression value of the third text set according to the mixed compression values of the paragraphs in the third text set.

[0078] Determine the mixed compression value of the third text set according to the sum of the mixed compression values of the paragraphs in the third text set.

[0079] Sort each third text set according to the mixed compression value of each third text set, and select the third text sets before the preset ranking as the fourth text set.

[0080] For example, if the preset ranking is 100 and there are a total of 55,000 third text sets, then continue to sort according to the mixed compression values corresponding to each third text set, and select the third text sets with the top 100 mixed compression values of the third text sets.

[0081] Step S15: Screen out the compressed extraction text corresponding to the text to be compressed from several fourth text sets according to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set.

[0082] Specifically, it includes: Merge the paragraphs in the paragraph set of the fourth text set in order, and delete the paragraph keyword set in the fourth text set to obtain the fourth text corresponding to the fourth text set.

[0083] Since the paragraphs in the paragraph set of the fourth text set are arranged in order, after merging in order, a text corresponding to the text to be compressed can also be obtained.

[0084] Perform semantic similarity analysis on the fourth text and the text to be compressed to obtain the text similarity of the fourth text.

[0085] It can be understood that although semantic similarity comparison is performed on paragraphs in the third text, after merging the paragraphs, the meaning of the text may change. Therefore, the present invention improves the semantic similarity between the compressed text and the text to be compressed based on the dual comparison method of compressed paragraph semantic comparison and compressed text semantic comparison, and realizes the closest text semantics while ensuring that compression and tailoring are satisfied.

[0086] Determine the word count value of the paragraph of the fourth text set according to the word count of the paragraph of the fourth text set, the paragraph weight, and the word count of the corresponding paragraph in the first text set.

[0087] As the example above, after semantic expansion and replacement, the word count of the corresponding paragraph may be different from that of the paragraph before replacement, which means that the total word count of the paragraph set in the second text set may be different from the total word count of the paragraph set in the first text set. Therefore, the total word counts of the paragraph sets of the third text sets obtained may all be different, that is, the total word counts of the paragraph sets of the fourth text sets selected from the third text sets may be different.

[0088] The word count value of the paragraphs in the fourth text set reflects the compression and trimming effect of the paragraph when ensuring semantic proximity.

[0089] Determine the text value of the fourth text according to the word count ratio of each paragraph in the fourth text set and the text similarity of the fourth text; The calculation of the text value includes:

[0090] Among them, is the text value of the th fourth text, is the text similarity of the th fourth text, is the word count of the th paragraph of the first text set, is the th paragraph of the th fourth text set, is the th paragraph number of the fourth text set, and are both preset weights, and and sum to 1.

[0091] Among them, is the th paragraph's word count value of the th fourth text set. The minimum value of

[0092] The text value comprehensively considers two key factors: the overall semantic similarity and the paragraph word count ratio, reflecting the semantic retention ability of the fourth text during the compression or reconstruction process and the matching degree of the information distribution.

[0093] The present invention avoids the loss of core information caused by simply pursuing word count compression by introducing double constraints of semantic similarity and structural ratio.

[0094] Use the fourth text corresponding to the highest text value as the compressed extraction text corresponding to the text to be compressed.

[0095] The present invention obtains the second text set through text expansion and obtains the third text through arbitrary trimming, providing a large number of reference documents with close semantics for the text to be compressed. Then, through two screenings of the mixed compression value and the text value, while ensuring the compression effect, the core semantics are retained.

[0096] In summary, the present invention provides a text extraction method based on hybrid compression, including: determining the text to be compressed, dividing the text to be compressed into paragraphs and extracting paragraph keywords to obtain a first text set of the text to be compressed; based on a preset semantic replacement and expansion model, performing semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets; determining a hybrid compression ratio, and respectively compressing and trimming the paragraph sets of the first text set and the paragraph sets of the second text sets with the hybrid compression ratio to obtain a plurality of third text sets; screening the plurality of third text sets according to the third text sets, the paragraph sets in the first text set, and the weight distribution of each paragraph in the first text set to obtain a plurality of fourth text sets; and screening out the compressed extraction text corresponding to the text to be compressed from the plurality of fourth text sets according to the text to be compressed, the paragraph sets of the fourth text sets, and the number of words in the paragraph sets. The present invention arbitrarily trims the paragraphs in the first text set and the paragraphs in the second text set with the hybrid compression ratio, realizing a huge expansion of the text volume and providing text volume support for the extraction of the optimal compressed text. The present invention realizes a quantitative evaluation of the compression effect based on the hybrid compression value, facilitating the comparison and preference between different compression results. The higher the hybrid compression value, the more it indicates that while reducing the text length, no key information is lost, improving the readability and practicality of the compressed text. The hybrid compression value comprehensively evaluates the quality of the compressed text through three dimensions: paragraph similarity, keyword coverage, and paragraph weight, reflecting a comprehensive consideration of the degree of semantic retention, the integrity of key information, and the priority of content importance during the compression process. The present invention improves the semantic similarity between the compressed text and the text to be compressed based on a two-fold comparison method of comparing the semantics of the compressed paragraphs and the semantics of the compressed text. While ensuring that the compression and trimming are satisfied, the text semantics are made closest. The present invention reflects the compression and trimming effect of the paragraph when the semantics are close by introducing the numerical value of the number of words in the paragraph, and maximizes the compression and trimming by introducing the numerical value of the number of words in the paragraph. The present invention avoids the loss of core information caused by simply pursuing word count compression by introducing dual constraints of semantic similarity and structural ratio. The text value provided by the present invention comprehensively considers two key factors: overall semantic similarity and paragraph word count ratio, reflecting the semantic retention ability of the fourth text with respect to the original content and the matching degree of information distribution during the compression or reconstruction process. The present invention obtains a second text set through text expansion and obtains a third text through arbitrary trimming, providing a large number of reference documents with close semantics for the text to be compressed. Then, two screenings are performed on the hybrid compression value and the text value, ensuring the compression effect while retaining the core semantics.

[0097] Based on the same inventive concept, the present invention provides a text extraction device based on hybrid compression as shown in Figure 2 and includes: A partitioning and extraction module 21, configured to determine a text to be compressed, partition the text to be compressed into paragraphs, and extract paragraph keywords, so as to obtain a first text set of the text to be compressed. The first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set; A replacement module 22, configured to perform semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set based on a preset semantic replacement and expansion model, so as to obtain a plurality of second text sets; A hybrid compression module 23, configured to determine a hybrid compression ratio, and perform compression and clipping on the paragraph sets of the first text set and the paragraph sets of the second text sets respectively according to the hybrid compression ratio, so as to obtain a plurality of third text sets. Each third text set includes a compressed paragraph set and a paragraph keyword set; A screening module 24, configured to screen a plurality of third text sets according to the third text sets, the paragraph sets in the first text set, and the weight distribution of each paragraph in the first text set, so as to obtain a plurality of fourth text sets. The fourth text set includes a paragraph set and a paragraph keyword set; A text extraction module 25, configured to screen out a compressed extraction text corresponding to the text to be compressed from a plurality of fourth text sets according to the text to be compressed, the paragraph sets of the fourth text sets, and the number of words in the paragraph sets.

[0098] Based on the same inventive concept, the present invention further provides an electronic device, including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to execute to implement the text extraction method based on hybrid compression as provided above.

[0099] Based on the same inventive concept, the present invention further provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute to implement the text extraction method based on hybrid compression as provided above.

[0100] Since the electronic device introduced in this embodiment is the electronic device adopted for implementing the information processing method in the embodiments of the present invention, based on the information processing method introduced in the embodiments of the present invention, those skilled in the art can understand the specific implementation manners and various variations of the electronic device in this embodiment. Therefore, the specific implementation of how this electronic device implements the method in the embodiments of the present invention will not be described in detail here. As long as the electronic device adopted by those skilled in the art to implement the information processing method in the embodiments of the present invention falls within the scope of protection of the present invention.

[0101] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0102] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0103] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0105] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0106] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A text extraction method based on hybrid compression, characterized in that Including: Determine the text to be compressed, perform paragraph division and paragraph keyword extraction on the text to be compressed, and obtain a first text set of the text to be compressed. The first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set; Based on a preset semantic replacement and expansion model, perform semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set to obtain a number of second text sets; Determine a hybrid compression ratio, and perform compression and clipping on the paragraph sets of the first text set and the paragraph sets of the second text sets respectively with the hybrid compression ratio to obtain a number of third text sets. Each third text set includes a compressed paragraph set and a paragraph keyword set; According to the third text sets, the paragraph sets in the first text set, and the weight distribution of each paragraph in the first text set, screen the number of third text sets to obtain a number of fourth text sets. The fourth text sets include a paragraph set and a paragraph keyword set; According to the text to be compressed, the paragraph sets of the fourth text sets, and the number of words in the paragraph sets, screen out the compressed extraction text corresponding to the text to be compressed from a number of fourth text sets.

2. The text extraction method based on hybrid compression according to claim 1, characterized in that, According to the third text sets, the paragraph sets in the first text set, and the weight distribution of each paragraph in the first text set, screen the number of third text sets to obtain a number of fourth text sets, including: Compare the similarity of the paragraphs in the third text set with the corresponding paragraphs in the first text set to obtain the paragraph similarity of the paragraph in the third text set; According to the paragraphs in the third text set and the corresponding paragraph keyword set of the paragraph, determine the paragraph keyword coverage of the paragraph; According to the weight distribution of each paragraph in the first text set, determine the weight of each paragraph in the third text set; According to the paragraph similarity, paragraph keyword coverage, and weight of the paragraphs in the third text set, determine the hybrid compression value of the paragraph; According to the hybrid compression values of the paragraphs in the third text set, determine the hybrid compression value of the third text set; According to the hybrid compression values of each third text set, sort each third text set, and select the third text sets before a preset position as the fourth text sets.

3. The text extraction method based on hybrid compression according to claim 2, wherein The calculation of the hybrid compression value includes: Wherein, is the mixed compression value of the th paragraph of the th third text set, is the weight of the th paragraph of the first text set, is the paragraph similarity of the th paragraph of the th third text set, and are both preset weights, and and sum to 1, is the paragraph keyword coverage of the th paragraph of the th third text set, is the total number of paragraph keywords in the paragraph keyword set of the th paragraph in the th third text set, is the number of paragraph keywords in the th paragraph of the th third text set, is the mixed compression value of the th third text set, is the number of paragraphs in the th third text set.

4. The text extraction method based on hybrid compression according to claim 2, wherein According to the text to be compressed, the paragraph sets of the fourth text sets, and the number of words in the paragraph sets, screen out the compressed extraction text corresponding to the text to be compressed from a number of fourth text sets, including: Merge the paragraphs in the paragraph set of the fourth text set in order, and delete the paragraph keyword set in the fourth text set to obtain the fourth text corresponding to the fourth text set; Perform semantic similarity analysis on the fourth text and the text to be compressed to obtain the text similarity of the fourth text; According to the number of words in the paragraphs of the fourth text set, the weight of the paragraph, and the number of words in the corresponding paragraphs in the first text set, determine the number value of the paragraph in the fourth text set; According to the proportion of the number of words in each paragraph in the fourth text set and the text similarity of the fourth text, determine the text value of the fourth text; Use the fourth text corresponding to the highest text value as the compressed extraction text corresponding to the text to be compressed.

5. The text extraction method based on hybrid compression according to claim 4, characterized in that The calculation of the text value includes: wherein, is the text value of the th fourth text, is the text similarity of the th fourth text, is the number of words in the th paragraph of the first text set, is the number of words in the th paragraph of the th fourth text set, is the number of paragraphs in the th fourth text set, and are both preset weights, and and sum to 1.

6. The text extraction method based on hybrid compression according to claim 1, characterized in that, Compress and clip the paragraph sets of the first text set and the second text set respectively with a mixed compression ratio to obtain several third text sets, including: Arbitrarily clip the paragraphs in the first text set and the paragraphs in the second text set with a mixed compression ratio so that the total number of words in several paragraphs of the clipped first text set and the total number of words in several paragraphs of the clipped second text set meet the mixed compression ratio.

7. The text extraction method based on hybrid compression according to claim 1, wherein Perform paragraph division and paragraph keyword extraction on the text to be compressed to obtain the first text set of the text to be compressed, including: Use preset punctuation marks to divide the text to be compressed into several paragraphs and a paragraph set composed of several paragraphs; After the paragraph division is completed, use a preset keyword extraction model to extract paragraph keywords for each paragraph of the text to be compressed to obtain a paragraph keyword set corresponding to the paragraph.

8. A text extraction device based on hybrid compression, characterized in that, Include: A division and extraction module for determining the text to be compressed, performing paragraph division and paragraph keyword extraction on the text to be compressed to obtain the first text set of the text to be compressed. The first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set; A replacement module for performing semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set based on a preset semantic replacement and expansion model to obtain several second text sets; A mixed compression module for determining a mixed compression ratio and respectively compressing and clipping the paragraph sets of the first text set and the paragraph sets of the second text set with the mixed compression ratio to obtain several third text sets. Each third text set includes a compressed paragraph set and a paragraph keyword set; A screening module for screening several third text sets according to the third text sets, the paragraph sets in the first text set, and the weight distribution of each paragraph in the first text set to obtain several fourth text sets. The fourth text sets include a paragraph set and a paragraph keyword set; A text extraction module for screening out the compressed extraction text corresponding to the text to be compressed from several fourth text sets according to the text to be compressed, the paragraph sets of the fourth text sets, and the number of words in the paragraph sets.

9. An electronic device, characterized in that, Include: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to execute to implement the text extraction method based on mixed compression as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the non-temporary computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute to implement the text extraction method based on mixed compression as described in any one of claims 1 to 7.

Images