Text Extraction Method, Device, Equipment and Medium Based on Hybrid Compression

Abstract

The present invention discloses a text extraction method, apparatus, device, and medium based on hybrid compression, comprising: obtaining a first text set of text to be compressed; semantically expanding and replacing paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets; compressing and trimming the paragraph sets of the first text set and the paragraph sets of the second text set using a hybrid compression ratio to obtain a plurality of third text sets; screening the plurality of third text sets to obtain a plurality of fourth text sets; and screening compressed extracted text corresponding to the text to be compressed from the plurality of fourth text sets based on the text to be compressed, the paragraph sets of the fourth text set, and the word count of the paragraph sets. The present invention belongs to the field of text compression. The present invention can ensure text compression while achieving the closest semantic approximation.

Description

Technical Field

[0001] The present invention relates to the field of text compression, and in particular to a text extraction method, device, equipment and medium based on hybrid compression. Background Art

[0002] Text compression is the process of making a text shorter and more concise by streamlining, rewriting, or removing redundant content, while preserving the original text's core meaning and key information as much as possible. It's not simply about reducing the number of words, but rather about purposefully refining information to make the content more compact while still being readable and expressive.

[0003] If too much content is compressed and trimmed, the original text will be lost; if too little is compressed and trimmed, the compression requirements will not be met. How to achieve a balance between text compression effect and text semantic effect is an urgent problem to be solved. Summary of the Invention

[0004] The present invention solves the technical problem of being unable to balance the text compression effect and the text semantic effect in the prior art by providing a text extraction method, apparatus, equipment and medium based on hybrid compression, and achieves the technical effect of balancing the text compression effect and the text semantic effect.

[0005] In a first aspect, the present invention provides a text extraction method based on hybrid compression, comprising:

[0006] Determine a text to be compressed, divide the text to be compressed into paragraphs, and extract paragraph keywords to obtain a first text set of the text to be compressed, wherein the first text set includes a paragraph set and a paragraph keyword set, and each paragraph corresponds to a paragraph keyword set;

[0007] Based on a preset semantic replacement and expansion model, semantic expansion and replacement are performed on paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets;

[0008] Determining a mixed compression ratio, and compressing and trimming the paragraph set of the first text set and the paragraph set of the second text set using the mixed compression ratio to obtain a plurality of third text sets, each of which includes a compressed paragraph set and a paragraph keyword set;

[0009] screening the plurality of third text sets according to the third text set, the paragraph set in the first text set, and the weight distribution of each paragraph in the first text set to obtain a plurality of fourth text sets, the fourth text sets including the paragraph set and the paragraph keyword set;

[0010] According to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set, a compressed extracted text corresponding to the text to be compressed is screened out from the plurality of fourth text sets.

[0011] Furthermore, based on the third text set, the paragraph set in the first text set, and the weight distribution of each paragraph in the first text set, several third text sets are screened to obtain several fourth text sets, including:

[0012] Comparing the similarity between the paragraph in the third text set and the corresponding paragraph in the first text set to obtain the paragraph similarity of the paragraph in the third text set;

[0013] Determining the paragraph keyword coverage of the paragraph based on the paragraph in the third text set and the paragraph keyword set corresponding to the paragraph;

[0014] Determining the weight of each paragraph in the third text set according to the weight distribution of each paragraph in the first text set;

[0015] determining a mixed compression value of the paragraph according to paragraph similarity, paragraph keyword coverage, and weight of the paragraph in the third text set;

[0016] determining a mixed compression value of the third text set according to the mixed compression value of each paragraph in the third text set;

[0017] The third text sets are sorted according to the mixed compression values of the third text sets, and the third text sets before the preset ranking are extracted as the fourth text set.

[0018] Furthermore, the calculation of the mixed compression value includes:

[0019]

[0020]

[0021]

[0022] in, For the The third text set The mixed compression value of the paragraphs, The first text set The weight of the paragraph, For the The third text set The paragraph similarity of the paragraphs, as well as are all preset weights, and and The sum is 1, For the The third text set The keyword coverage of each paragraph, For the The third text set The total number of paragraph keywords in the paragraph keyword set, For the The third text set The number of paragraph keywords in each paragraph, For the A mixed compression value of a third text set, For the The number of paragraphs in the third text set.

[0023] Furthermore, based on the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set, a compressed extracted text corresponding to the text to be compressed is screened out from the plurality of fourth text sets, including:

[0024] Merging the paragraphs in the paragraph set of the fourth text set in order, and deleting the paragraph keyword set in the fourth text set, to obtain a fourth text corresponding to the fourth text set;

[0025] Performing semantic similarity analysis on the fourth text and the text to be compressed to obtain text similarity of the fourth text;

[0026] Determining a word count value of the paragraph in the fourth text set according to the word count of the paragraph in the fourth text set, the paragraph weight, and the word count of the corresponding paragraph in the first text set;

[0027] Determining a text value of the fourth text based on a word count ratio of each paragraph in the fourth text set and a text similarity of the fourth text;

[0028] The fourth text corresponding to the highest text value is used as the compressed extracted text corresponding to the text to be compressed.

[0029] Furthermore, the calculation of text values includes:

[0030]

[0031] in, For the The text value of the fourth text, For the The text similarity of the fourth text, The first text set The number of words in a paragraph, For the The fourth text set The number of words in a paragraph, For the The number of paragraphs in the fourth text set, and are all preset weights, and and The sum is 1.

[0032] Furthermore, the paragraph set of the first text set and the paragraph set of the second text set are compressed and trimmed respectively with a mixed compression ratio to obtain a plurality of third text sets, including:

[0033] The paragraphs in the first text set and the paragraphs in the second text set are arbitrarily trimmed with a mixed compression ratio so that the total number of words in the trimmed paragraphs of the first text set and the total number of words in the trimmed paragraphs of the second text set meet the mixed compression ratio.

[0034] Furthermore, the text to be compressed is divided into paragraphs and keywords are extracted from the paragraphs to obtain a first text set of the text to be compressed, including:

[0035] Divide the compressed text into paragraphs using preset punctuation marks to obtain a number of paragraphs and a paragraph set consisting of a number of paragraphs;

[0036] After the paragraph division is completed, the preset keyword extraction model is used to extract paragraph keywords from each paragraph of the compressed text to obtain a paragraph keyword set corresponding to the paragraph.

[0037] In a second aspect, the present invention provides a text extraction device based on hybrid compression, comprising:

[0038] a division and extraction module for determining a text to be compressed, dividing the text to be compressed into paragraphs, and extracting paragraph keywords to obtain a first text set of the text to be compressed, wherein the first text set includes a paragraph set and a paragraph keyword set, and each paragraph corresponds to a paragraph keyword set;

[0039] A replacement module, configured to perform semantic expansion and replacement on paragraphs and paragraph keywords in the first text set based on a preset semantic replacement expansion model to obtain a plurality of second text sets;

[0040] a hybrid compression module, configured to determine a hybrid compression ratio, and compress and trim the paragraph set of the first text set and the paragraph set of the second text set using the hybrid compression ratio to obtain a plurality of third text sets, each of which includes a compressed paragraph set and a paragraph keyword set;

[0041] a screening module configured to screen the plurality of third text sets based on the third text set, the paragraph set in the first text set, and the weight distribution of each paragraph in the first text set to obtain a plurality of fourth text sets, the fourth text sets including the paragraph set and the paragraph keyword set;

[0042] The text extraction module is used to screen out compressed extracted text corresponding to the text to be compressed from a plurality of fourth text sets according to the text to be compressed, the paragraph set of the fourth text set and the number of words in the paragraph set.

[0043] In a third aspect, the present invention provides an electronic device, comprising:

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] The processor is configured to execute to implement the text extraction method based on hybrid compression as provided in the first aspect.

[0047] In a fourth aspect, the present invention provides a non-temporary computer-readable storage medium, which, when the instructions in the non-temporary computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to implement the text extraction method based on hybrid compression as provided in the first aspect.

[0048] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0049] The present invention obtains a second text set through text expansion and a third text obtained by arbitrary trimming, providing a large number of reference files with possibly similar semantics for the text to be compressed, and then performs two screenings by mixing the compression value and the text value, thereby ensuring the maximization of the compression effect while retaining the core semantics.

[0050] The present invention arbitrarily trims paragraphs in the first text set and paragraphs in the second text set with a mixed compression ratio, thereby achieving a massive expansion of the text volume and providing text volume support for extracting the optimal compressed text.

[0051] This invention uses a hybrid compression value to quantitatively evaluate compression effectiveness, facilitating comparison and optimization of different compression results. A higher hybrid compression value indicates that while reducing text length, key information is retained, improving the readability and usability of the compressed text. The hybrid compression value comprehensively assesses the quality of the compressed text using three dimensions: paragraph similarity, keyword coverage, and paragraph weight. This reflects the comprehensive consideration of semantic preservation, key information integrity, and content importance prioritization during the compression process.

[0052] The present invention is based on a dual comparison method of semantic comparison of compressed paragraphs and semantic comparison of compressed texts, which improves the semantic similarity between the compressed text and the text to be compressed, and achieves the closest text semantics while ensuring that compression and tailoring are met.

[0053] The present invention reflects the compression and tailoring effect of the paragraph while ensuring semantic closeness by introducing the word count value of the paragraph, and maximizes the compression and tailoring by introducing the word count value of the paragraph.

[0054] This invention introduces the dual constraints of semantic similarity and structural proportion to avoid the loss of core information caused by simply pursuing word count compression. The text value provided by this invention comprehensively considers two key factors: overall semantic similarity and paragraph word count ratio. It reflects the ability of the fourth text to preserve the semantics of the original content during compression or reconstruction, as well as the degree of matching of information distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 Schematic diagram of the flow of the text extraction method based on hybrid compression of the present invention;

[0057] Figure 2 It is a structural diagram of the text extraction device based on hybrid compression of the present invention. DETAILED DESCRIPTION

[0058] The embodiment of the present invention solves the technical problem in the prior art that the text compression effect and the text semantic effect cannot be balanced by providing a text extraction method based on hybrid compression.

[0059] The technical solution of the present invention is to solve the above technical problems, and the overall idea is as follows:

[0060] A text extraction method based on hybrid compression includes: determining a text to be compressed, dividing the text to be compressed into paragraphs and extracting paragraph keywords to obtain a first text set of the text to be compressed, wherein the first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set; based on a preset semantic replacement expansion model, semantically expanding and replacing the paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets; determining a hybrid compression ratio, and compressing and trimming the paragraph set of the first text set and the paragraph set of the second text set using the hybrid compression ratio to obtain a plurality of third text sets, wherein each third text set includes a compressed paragraph set and a paragraph keyword set; screening the plurality of third text sets according to the weight distribution of the third text sets, the paragraph set in the first text set, and the paragraphs in the first text set to obtain a plurality of fourth text sets, wherein the fourth text sets include a paragraph set and a paragraph keyword set; and screening compressed extracted texts corresponding to the text to be compressed from the plurality of fourth text sets according to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set.

[0061] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0062] First, the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0063] The present invention provides Figure 1 The text extraction method based on hybrid compression shown includes steps S11-S15:

[0064] Step S11 , determining the text to be compressed, dividing the text into paragraphs and extracting paragraph keywords, and obtaining a first text set of the text to be compressed. The first text set includes a paragraph set and a paragraph keyword set, and one paragraph corresponds to one paragraph keyword set.

[0065] Specifically, the method includes: dividing the compressed text into paragraphs using preset punctuation marks to obtain a number of paragraphs and a paragraph set consisting of a number of paragraphs; after completing the paragraph division, extracting paragraph keywords from each paragraph of the compressed text using a preset keyword extraction model to obtain a paragraph keyword set corresponding to the paragraph.

[0066] The goal of step S11 is to perform structural processing on a piece of original text to be compressed, divide it into multiple paragraphs, and extract a set of keywords for each paragraph, thereby constructing a data set containing paragraphs and corresponding keywords (called the first text set).

[0067] The text to be compressed refers to a piece of original text that needs to be compressed selected from the input, such as a news article, report, etc.

[0068] You can use preset punctuation marks (such as period, question mark, exclamation mark, or line break) as separators to split the entire text into several paragraphs.

[0069] After completing paragraph division, use a preset keyword extraction model (such as TF-IDF, TextRank, or BERT-based keyword extraction model) to extract keywords for each paragraph. Each paragraph corresponds to a paragraph keyword set. In addition to using the keyword extraction model, in special fields (such as medicine), you can also screen based on the work experience of relevant personnel.

[0070] For example:

[0071] Paragraph 1: "The weather is nice today. We went for a walk in the park."

[0072] The extracted keywords may be: [weather, park, walk], then [weather, park, walk] is the paragraph keyword set of paragraph 1, the paragraph keyword set contains at least one paragraph keyword, and the paragraph set is composed of all the divided paragraphs.

[0073] It should also be noted that the split paragraphs are arranged in order, that is, all the paragraph sets below are also arranged in order, that is, the paragraph sets of the first text set to the paragraph sets of the fourth text set below are all arranged in order.

[0074] Step S12: Based on a preset semantic replacement and expansion model, semantic expansion and replacement are performed on the paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets.

[0075] For each paragraph and its corresponding paragraph keywords, a semantic replacement expansion model is used to find words or phrases with similar semantics. The preset semantic replacement expansion model can be Word2Vec, GloVe, or BERT model, etc.

[0076] It is understandable that when a paragraph is semantically expanded or replaced, the number of words in the semantically expanded or replaced paragraph may change.

[0077] For example:

[0078] Paragraph 1: "The weather is nice today. We went for a walk in the park."

[0079] After replacement:

[0080] "Today's weather is pleasant and suitable for outdoor activities, so we went for a walk in the city's green space." Or,

[0081] "The sky is clear, the temperature is pleasant, and the family is going for a walk in a natural scenic area in the suburbs." Or,

[0082] “The weather was excellent, so my friend and I took a leisurely walk around the botanical gardens.”

[0083] It is understandable that the keywords of the semantically expanded and replaced paragraph 1 are also significantly different. For example, the content of the replaced paragraph 1 is:

[0084] “The weather was excellent, so my friend and I chose to take a leisurely walk in the botanical garden.” Then, the paragraph keyword set could be {weather, botanical garden, me, friend, walk}.

[0085] In addition, you can first semantically expand and replace the paragraph keywords in the original paragraph 1, and then expand paragraph 1 based on the semantically expanded and replaced paragraph keywords.

[0086] For example:

[0087] Paragraph 1 content: "The weather is nice today. We went for a walk in the park."; Paragraph keyword set: ["weather", "park", "walk"];

[0088] The replacement paragraph keyword set can be: {climate, green space, walking}, {weather, scenic spot, stroll}, etc.

[0089] Take {climate, green space, walking} as an example: the replaced paragraph 1 could be "The weather was pleasant today, so we went for a relaxing walk in the green space."

[0090] After the above semantic expansion and replacement operations, the first text set is converted into multiple possible versions, each version representing a different but possibly semantically related expression form.

[0091] Step S13, determining a mixed compression ratio, and compressing and trimming the paragraph set of the first text set and the paragraph set of the second text set respectively using the mixed compression ratio to obtain a plurality of third text sets, each of which includes a compressed paragraph set and a paragraph keyword set.

[0092] Specifically, the method includes arbitrarily trimming paragraphs in the first text set and paragraphs in the second text set using a mixed compression ratio, so that the total number of words in the trimmed paragraphs of the first text set and the total number of words in the trimmed paragraphs of the second text set meet the mixed compression ratio.

[0093] The first text set consists of paragraphs before semantic expansion and replacement and their corresponding paragraph keyword sets; the second text set consists of paragraphs obtained through semantic expansion and replacement and their corresponding paragraph keyword sets. A paragraph in the second text set also corresponds to a paragraph keyword set.

[0094] The mixed compression ratio is a preset trimming ratio. For example, if the mixed compression ratio is 60%, 60% of the words in the paragraph set of the second text set and the paragraph set of the first text set need to be trimmed. The mixed compression ratio can be determined based on actual needs.

[0095] It is important to emphasize that the clipping method is random, that is, some paragraphs may be completely clipped. All paragraph keyword sets are not involved in the clipping. In addition, if a paragraph is completely clipped, it still has a paragraph number, but its content is clipped and compressed.

[0096] For example:

[0097] Paragraph 1: "The weather is nice today. We went for a walk in the park."

[0098] Paragraph 1 keyword set: [weather, park, walk]

[0099] If paragraph 1 is completely compressed and trimmed, then paragraph 1 cannot have any content, but paragraph 1 still exists, and the paragraph keyword set of paragraph 1 is still [weather, park, walk].

[0100] Still taking paragraph 1 as an example, paragraph 1 may not be compressed or trimmed at all, for example: "The weather is good today, I'm going to go to work." It can be understood that as long as the number of words in the paragraph set of the first text set corresponding to paragraph 1 meets the remaining 40%, it will be sufficient.

[0101] It is understandable that, due to the random trimming and compression, the paragraph set of the first text set and the paragraph set of the second text set may be compressed and trimmed several times with a mixed compression ratio.

[0102] That is to say, when the paragraph set of the first text set is compressed and trimmed 10 times, 10 different third text sets can be obtained (the content of the compression and trimming can be limited to be different from the previous compression and trimming). The same is true for the second text set. The number of random trimming and compression of the same text set may be determined according to actual conditions. For example, it can be set to 5,000 times, 10,000 times, 20,000 times, 30,000 times or 50,000 times, etc.

[0103] For example, there is one first text set and ten second text sets, and the number of random cropping and compression is 5,000, then a total of 55,000 first text sets are obtained.

[0104] After compression and trimming, several third text sets are obtained. Each third text set includes a compressed paragraph set and a paragraph keyword set. A compressed paragraph also corresponds to a paragraph keyword set.

[0105] The present invention arbitrarily trims paragraphs in the first text set and paragraphs in the second text set with a mixed compression ratio, thereby achieving a massive expansion of the text volume and providing text volume support for extracting the optimal compressed text.

[0106] Step S14 , screening the third text sets according to the third text set, the paragraph set in the first text set and the weight distribution of each paragraph in the first text set to obtain several fourth text sets, wherein the fourth text sets include a paragraph set and a paragraph keyword set.

[0107] Specifically include:

[0108] A paragraph in the third text set is compared with a corresponding paragraph in the first text set in similarity to obtain a paragraph similarity of the paragraph in the third text set.

[0109] It is understandable that since the paragraphs in the first text set and the third text set are arranged in order, each paragraph in the first text set can correspond one-to-one with each paragraph in the third text set (even if the paragraph in the third text set has no content).

[0110] For example:

[0111] Paragraph 1 of a third text set: "The weather is good today, let's go to the office." corresponds to paragraph 1 of the first text set: "The weather is good today. Let's go for a walk in the park."

[0112] Also for example:

[0113] Paragraph 1 of a third text set has no content and is: "", which still corresponds to paragraph 1 of the first text set: "The weather is very good today. We went for a walk in the park."

[0114] In addition to the one-to-one correspondence between paragraphs, the paragraph keyword sets also have a one-to-one correspondence (but paragraph keywords do not participate in the calculation of paragraph similarity).

[0115] For example:

[0116] The paragraph keyword set corresponding to paragraph 1 of a certain third text set is: {climate, green space, walking}, and the paragraph keyword set corresponding to paragraph 1 of the first text set is: [weather, park, walk].

[0117] The paragraphs can be converted into vector representations (such as using models such as BERT and Sentence-BERT), and then the vector similarity between two paragraphs (such as cosine similarity) can be calculated to obtain the paragraph similarity of a paragraph in the third text collection.

[0118] Each paragraph in each third text set needs to be compared with the corresponding paragraph in the first text set for similarity.

[0119] The paragraph keyword coverage of the paragraph is determined according to the paragraph in the third text set and the paragraph keyword set corresponding to the paragraph in the third text set.

[0120] The paragraph keyword coverage reflects the presence rate of keywords in the compressed and trimmed paragraph and the paragraph before compression and trimming.

[0121] For example:

[0122] The paragraph keyword set corresponding to paragraph 1 of a certain third text set is: [weather, park, walk]; paragraph 1 of a certain third text set: "The weather is good today, let's go to work.", then there is no paragraph keyword in paragraph 1 of the third text set, and the paragraph keyword coverage is 0 / 3=0%.

[0123] Each paragraph in each third text set needs to be compared with the paragraph keyword set corresponding to the paragraph in the third text set, and the paragraph keyword coverage of the paragraph is obtained.

[0124] The weight of each paragraph in the third text set is determined according to the weight distribution of each paragraph in the first text set.

[0125] The weight distribution of each paragraph in the first text set can be set according to the work experience of relevant personnel. The greater the weight of a paragraph, the more important the paragraph is in the first text set.

[0126] For example:

[0127] There are 10 paragraphs in the first text set, with weight distribution of {0.1, 0.15, 0.05, 0.28, 0.34, 0.04, 0.01, 0.01, 0.01, 0.01}, and the sum of the weights of each paragraph is 1.

[0128] The mixed compression value of the paragraph is determined according to the paragraph similarity, paragraph keyword coverage, and weight of the paragraph in the third text set.

[0129] The calculation of the hybrid compression value includes:

[0130]

[0131]

[0132]

[0133] in, For the The third text set The mixed compression value of the paragraphs, The first text set The weight of the paragraph, For the The third text set The paragraph similarity of the paragraphs, as well as are all preset weights, and and The sum is 1, For the The third text set The keyword coverage of each paragraph, For the The third text set The total number of paragraph keywords in the paragraph keyword set, For the The third text set The number of paragraph keywords in each paragraph, For the A mixed compression value of a third text set, For the The number of paragraphs in the third text set.

[0134] The hybrid compression value comprehensively evaluates the quality of the compressed text through three dimensions: paragraph similarity, keyword coverage, and paragraph weight, reflecting the comprehensive consideration of the degree of semantic retention, key information integrity, and content importance priority during the compression process.

[0135] The present invention realizes quantitative evaluation of compression effect based on hybrid compression value, which facilitates comparison and optimization between different compression results. The higher the hybrid compression value, the more it means that while reducing the length of the text, key information is not lost, thereby improving the readability and practicality of the compressed text.

[0136] The mixed compression value of the third text set is determined according to the mixed compression value of each paragraph in the third text set.

[0137] The mixed compression value of the third text set is determined according to the sum of the mixed compression values of the paragraphs in the third text set.

[0138] The third text sets are sorted according to the mixed compression values of the third text sets, and the third text sets before the preset ranking are extracted as the fourth text set.

[0139] For example, if the preset ranking is 100 and there are 55,000 third text sets in total, the third text sets are further sorted according to their corresponding mixed compression values, and the third text sets with the top 100 mixed compression values are extracted.

[0140] Step S15 , selecting compressed extracted text corresponding to the text to be compressed from the plurality of fourth text sets according to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set.

[0141] Specifically include:

[0142] The paragraphs in the paragraph set of the fourth text set are merged in order, and the paragraph keyword set in the fourth text set is deleted to obtain a fourth text corresponding to the fourth text set.

[0143] Since the paragraphs in the paragraph set of the fourth text set are arranged in sequence, after merging them in sequence, a text corresponding to the text to be compressed can also be obtained.

[0144] A semantic similarity analysis is performed on the fourth text and the text to be compressed to obtain text similarity of the fourth text.

[0145] It is understandable that although the paragraphs are compared for semantic similarity in the third text, the meaning of the text may change after the paragraphs are merged.

[0146] Therefore, the present invention improves the semantic similarity between the compressed text and the text to be compressed based on the dual comparison method of semantic comparison of compressed paragraphs and semantic comparison of compressed texts, and achieves the closest text semantics while ensuring that compression and tailoring are met.

[0147] The word count value of the paragraph in the fourth text set is determined according to the word count of the paragraph in the fourth text set, the paragraph weight, and the word count of the corresponding paragraph in the first text set.

[0148] As in the example above, after semantic expansion and replacement, the number of words in the corresponding paragraph may be different from the number of words in the paragraph before replacement, which means that the total number of words in the paragraph set in the second text set may be different from the total number of words in the paragraph set in the first text set. Therefore, the total number of words in the paragraph set of the third text set may be different. In other words, the total number of words in the paragraph set of the fourth text set obtained by filtering the third text set may be different.

[0149] The word count of the paragraph in the fourth text set reflects the compression and trimming effect of the paragraph while ensuring semantic proximity.

[0150] Determining a text value of the fourth text based on a word count ratio of each paragraph in the fourth text set and a text similarity of the fourth text;

[0151] Calculations on text values, including:

[0152]

[0153] in, For the The text value of the fourth text, For the The text similarity of the fourth text, The first text set The number of words in a paragraph, For the The fourth text set The number of words in a paragraph, For the The number of paragraphs in the fourth text set, and are all preset weights, and and The sum is 1.

[0154] in, For the The fourth text set The word count of each paragraph. The minimum value is 1.

[0155] The text value comprehensively considers two key factors: overall semantic similarity and the proportion of words in a paragraph. It reflects the fourth text's ability to preserve the semantics of the original content during compression or reconstruction and the degree of matching of information distribution.

[0156] The present invention introduces dual constraints of semantic similarity and structural ratio to avoid the loss of core information caused by simply pursuing word count compression.

[0157] The fourth text corresponding to the highest text value is used as the compressed extracted text corresponding to the text to be compressed.

[0158] The present invention obtains a second text set by text expansion and obtains a third text by arbitrary trimming, providing a large number of semantically similar comparison files for the text to be compressed, and then mixing the compression value and the text value for two screenings, thereby ensuring the compression effect while retaining the core semantics.

[0159] In summary, the present invention provides a text extraction method based on hybrid compression, including: determining the text to be compressed, and dividing the text to be compressed into paragraphs and extracting paragraph keywords to obtain a first text set of the text to be compressed; based on a preset semantic replacement expansion model, semantically expanding and replacing the paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets; determining a hybrid compression ratio, and compressing and trimming the paragraph set of the first text set and the paragraph set of the second text set respectively with the hybrid compression ratio to obtain a plurality of third text sets; screening the plurality of third text sets according to the weight distribution of the third text set, the paragraph set in the first text set, and each paragraph of the first text set to obtain a plurality of fourth text sets; screening the compressed extracted text corresponding to the text to be compressed from the plurality of fourth text sets according to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set. The present invention arbitrarily trims the paragraphs in the first text set and the paragraphs in the second text set with the hybrid compression ratio, thereby realizing a massive expansion of the text volume and providing text volume support for achieving the extraction of the optimal compressed text. The present invention realizes the quantitative evaluation of the compression effect based on the hybrid compression value, which is convenient for comparison and optimization between different compression results. The higher the hybrid compression value, the less key information is lost while reducing the length of the text, and the readability and practicality of the compressed text are improved. The hybrid compression value comprehensively evaluates the quality of the compressed text through three dimensions: paragraph similarity, keyword coverage and paragraph weight, reflecting the comprehensive consideration of the degree of semantic retention, key information integrity and content importance priority in the compression process. The present invention is based on a dual comparison method of semantic comparison of compressed paragraphs and semantic comparison of compressed texts, which improves the semantic similarity between the compressed text and the text to be compressed, and achieves the closest semantics of the text while ensuring that the compression and trimming are met. The present invention reflects the compression and trimming effect of the paragraph when ensuring semantic proximity by introducing the word count value of the paragraph, and maximizes the compression and trimming by introducing the word count value of the paragraph. The present invention avoids the loss of core information caused by the simple pursuit of word count compression by introducing the dual constraints of semantic similarity and structural proportion. The text value provided by this invention comprehensively considers two key factors: overall semantic similarity and paragraph word count ratio. This reflects the fourth text's ability to preserve the semantics of the original content during compression or reconstruction, as well as the degree of matching of information distribution. By expanding the second text set and arbitrarily trimming the third text, this invention provides a large number of semantically similar reference files for the text to be compressed. The compression value and text value are then mixed and filtered twice, ensuring compression effectiveness while preserving core semantics.

[0160] Based on the same inventive concept, the present invention provides Figure 2 The text extraction device based on hybrid compression shown includes:

[0161] The division and extraction module 21 is used to determine the text to be compressed, divide the text to be compressed into paragraphs, and extract paragraph keywords to obtain a first text set of the text to be compressed. The first text set includes a paragraph set and a paragraph keyword set, and each paragraph corresponds to a paragraph keyword set.

[0162] A replacement module 22 is configured to perform semantic expansion and replacement on paragraphs and paragraph keywords in the first text set based on a preset semantic replacement expansion model to obtain a plurality of second text sets;

[0163] a hybrid compression module 23 for determining a hybrid compression ratio and compressing and trimming the paragraph set of the first text set and the paragraph set of the second text set using the hybrid compression ratio to obtain a plurality of third text sets, each of which includes a compressed paragraph set and a paragraph keyword set;

[0164] a screening module 24 configured to screen the plurality of third text sets based on the third text set, the paragraph set in the first text set, and the weight distribution of each paragraph in the first text set to obtain a plurality of fourth text sets, the fourth text sets including the paragraph set and the paragraph keyword set;

[0165] The text extraction module 25 is configured to select compressed extracted text corresponding to the text to be compressed from the plurality of fourth text sets according to the text to be compressed, the paragraph set of the fourth text set, and the number of words in the paragraph set.

[0166] Based on the same inventive concept, the present invention further provides an electronic device, comprising:

[0167] processor;

[0168] a memory for storing processor-executable instructions;

[0169] The processor is configured to execute to implement the text extraction method based on hybrid compression as provided above.

[0170] Based on the same inventive concept, the present invention also provides a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to implement the text extraction method based on hybrid compression as provided above.

[0171] Since the electronic device described in this embodiment is an electronic device used to implement the information processing method in the embodiment of the present invention, based on the information processing method described in the embodiment of the present invention, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present invention will not be described in detail here. As long as the electronic device used by those skilled in the art to implement the information processing method in the embodiment of the present invention falls within the scope of protection of the present invention.

[0172] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0177] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A text extraction method based on hybrid compression, characterized in that: include: Determine a text to be compressed, divide the text to be compressed into paragraphs, and extract paragraph keywords from the text to be compressed to obtain a first text set of the text to be compressed, wherein the first text set includes a paragraph set and a paragraph keyword set, and each paragraph corresponds to a paragraph keyword set; Based on a preset semantic replacement and expansion model, semantically expand and replace the paragraphs and paragraph keywords in the first text set to obtain a plurality of second text sets; Determining a mixed compression ratio, and compressing and trimming the paragraph set of the first text set and the paragraph set of the second text set using the mixed compression ratio to obtain a plurality of third text sets, each of which includes a compressed paragraph set and a paragraph keyword set; According to the third text set, the paragraph set in the first text set and the weight distribution of each paragraph in the first text set, several third text sets are screened to obtain several fourth text sets, where the fourth text set includes a paragraph set and a paragraph keyword set, including: comparing the similarity of the paragraph in the third text set with the corresponding paragraph in the first text set to obtain the paragraph similarity of the paragraph in the third text set; determining the paragraph keyword coverage of the paragraph according to the paragraph in the third text set and the paragraph keyword set corresponding to the paragraph; determining the weight of each paragraph in the third text set according to the weight distribution of each paragraph in the first text set; determining the mixed compression value of the paragraph according to the paragraph similarity, paragraph keyword coverage and weight of the paragraph in the third text set; determining the mixed compression value of the third text set according to the mixed compression value of each paragraph in the third text set; sorting each third text set according to the mixed compression value of each third text set, and extracting the third text set before the preset position as the fourth text set; According to the text to be compressed, the paragraph set of the fourth text set and the number of words in the paragraph set, a compressed extracted text corresponding to the text to be compressed is screened out from several fourth text sets, including: merging the paragraphs in the paragraph set of the fourth text set in order, and deleting the paragraph keyword set in the fourth text set to obtain a fourth text corresponding to the fourth text set; performing semantic similarity analysis on the fourth text and the text to be compressed to obtain the text similarity of the fourth text; determining the word count value of the paragraph in the fourth text set according to the word count of the paragraph in the fourth text set, the paragraph weight and the word count of the corresponding paragraph in the first text set; determining the text value of the fourth text according to the word count proportion of each paragraph in the fourth text set and the text similarity of the fourth text; and using the fourth text corresponding to the highest text value as the compressed extracted text corresponding to the text to be compressed.

2. The text extraction method based on hybrid compression according to claim 1, characterized in that: The calculation of the mixed compression value includes: in, For the The third text set The mixed compression value of the paragraphs, The first text set The weight of the paragraph, For the The third text set The paragraph similarity of the paragraphs, as well as are all preset weights, and and The sum is 1, For the The third text set The keyword coverage of each paragraph, For the The third text set The total number of paragraph keywords in the paragraph keyword set, For the The third text set The number of paragraph keywords in each paragraph, For the A mixed compression value of a third text set, For the The number of paragraphs in the third text set.

3. The text extraction method based on hybrid compression according to claim 2, characterized in that: Calculations on text values, including: in, For the The text value of the fourth text, For the The text similarity of the fourth text, The first text set The number of words in a paragraph, For the The fourth text set The number of words in a paragraph, For the The number of paragraphs in the fourth text set, and are all preset weights, and and The sum is 1.

4. The text extraction method based on hybrid compression according to claim 1, characterized in that: The paragraph set of the first text set and the paragraph set of the second text set are compressed and trimmed respectively with a mixed compression ratio to obtain a plurality of third text sets, including: The paragraphs in the first text set and the paragraphs in the second text set are arbitrarily trimmed with a mixed compression ratio so that the total number of words in the trimmed paragraphs of the first text set and the total number of words in the trimmed paragraphs of the second text set meet the mixed compression ratio.

5. The text extraction method based on hybrid compression according to claim 1, characterized in that: Dividing the text to be compressed into paragraphs and extracting keywords from the paragraphs to obtain a first text set of the text to be compressed, including: Dividing the text to be compressed into paragraphs using preset punctuation marks to obtain a plurality of paragraphs and a paragraph set consisting of the plurality of paragraphs; After the paragraph division is completed, a preset keyword extraction model is used to extract paragraph keywords from each paragraph of the text to be compressed to obtain a paragraph keyword set corresponding to the paragraph.

6. A text extraction device based on hybrid compression, characterized in that: The method for implementing the hybrid compression-based text extraction method according to any one of claims 1 to 5 comprises: a division and extraction module for determining a text to be compressed, dividing the text to be compressed into paragraphs and extracting paragraph keywords to obtain a first text set of the text to be compressed, wherein the first text set includes a paragraph set and a paragraph keyword set, and each paragraph corresponds to a paragraph keyword set; a replacement module, configured to perform semantic expansion and replacement on the paragraphs and paragraph keywords in the first text set based on a preset semantic replacement expansion model to obtain a plurality of second text sets; a hybrid compression module, configured to determine a hybrid compression ratio, and compress and trim the paragraph set of the first text set and the paragraph set of the second text set using the hybrid compression ratio to obtain a plurality of third text sets, each of which includes a compressed paragraph set and a paragraph keyword set; a screening module configured to screen the plurality of third text sets based on the third text set, the paragraph set in the first text set, and the weight distribution of each paragraph in the first text set to obtain a plurality of fourth text sets, the fourth text sets including the paragraph set and the paragraph keyword set; The text extraction module is used to screen out compressed extracted text corresponding to the text to be compressed from a plurality of fourth text sets according to the text to be compressed, the paragraph set of the fourth text set and the number of words in the paragraph set.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute to implement the text extraction method based on hybrid compression as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that When the instructions in the non-transitory computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to implement the text extraction method based on hybrid compression as described in any one of claims 1 to 5.

Images