Sentence compression method, device, electronic device and readable storage medium

Abstract

The present invention relates to data processing, and discloses a sentence compression method, comprising: performing de-colloquialization processing on the compressed sentences to obtain a target sentence set, and judging whether the number of sentences in the target sentence set is greater than a first threshold; when it is judged that the number of sentences in the target sentence set is greater than the first threshold, sorting the sentences in the target sentence set by importance, extracting key sentences based on the sorting results, and judging whether the sentence length of the key sentences is greater than a second threshold; when it is judged that the sentence length of the key sentences is greater than the second threshold, extracting the main words of the key sentences, splicing the main words, and obtaining the target sentences. The present invention also provides a sentence compression device, an electronic device, and a readable storage medium. The present invention reduces the annotation cost and ensures the semantic accuracy of the compressed sentences.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a sentence compression method, device, electronic equipment and readable storage medium. Background Art

[0002] Sentence compression technology is an important research direction in the field of natural language processing. Through sentence compression processing, redundant information in the sentence can be eliminated, the main idea can be retained, and it is easier for users to read and machines to recognize. It can be used in many fields such as summary generation, question matching and topic extraction.

[0003] At present, generative or extractive sentence compression processing methods are usually used. However, the generative sentence compression method requires a large amount of annotated corpus for supervised learning. This method is not applicable to situations where the project size and cost are limited, the amount of business data is small, and the annotated data is missing. The traditional extractive sentence compression method is related to the length of the sentence. When the sentence is long, the compression effect of the extractive compression method is not ideal and the semantic information cannot be accurately retained. Therefore, a sentence compression method is urgently needed to reduce the annotation cost and ensure the semantic accuracy of the compressed sentence. Summary of the invention

[0004] In view of the above, it is necessary to provide a sentence compression method to reduce the annotation cost and ensure the semantic accuracy of the compressed sentences.

[0005] The statement compression method provided by the present invention comprises:

[0006] Parsing a sentence compression request issued by a user based on a client, obtaining a sentence to be compressed carried in the request, performing a de-colloquialization process on the sentence to be compressed to obtain a target sentence set, and determining whether the number of sentences in the target sentence set is greater than a first threshold;

[0007] When it is determined that the number of sentences in the target sentence set is greater than a first threshold, the sentences in the target sentence set are sorted by importance, key sentences are extracted based on the sorting result, and it is determined whether the sentence length of the key sentence is greater than a second threshold;

[0008] When it is determined that the sentence length of the key sentence is greater than the second threshold, the main words of the key sentence are extracted, and the main words are concatenated to obtain the target sentence.

[0009] Optionally, performing de-colloquialization processing on the sentence to be compressed includes:

[0010] Acquire a spoken sentence dictionary from a first database, compare each first clause in the sentence to be compressed with the spoken sentence dictionary, and if a specified first clause matches a sentence in the spoken sentence dictionary, delete the specified first clause to obtain an initial sentence set;

[0011] Performing word segmentation processing on the sentences in the initial sentence set to obtain a first word sequence;

[0012] identifying spoken words in the first word sequence based on a spoken word recognition model, deleting the spoken words, and obtaining a second word sequence;

[0013] The words in the second word sequence are concatenated according to the positions of the words in the sentence to be compressed to obtain a plurality of second clauses, and a set of the plurality of second clauses is used as a target sentence set.

[0014] Optionally, the step of ranking the sentences in the target sentence set by importance and extracting key sentences based on the ranking result includes:

[0015] Combining each sentence in the target sentence set with other sentences in pairs to obtain a plurality of combination pairs;

[0016] Calculating similarity values ​​of two sentences of each combination pair in the plurality of combination pairs, and determining a similarity matrix corresponding to the target sentence set based on the similarity values;

[0017] The importance score of each sentence in the target sentence set is calculated based on the similarity matrix, and the sentences in the target sentence set are sorted in descending order of importance score, and the sentence at the top of the sorting is used as the key sentence.

[0018] Optionally, after determining whether the number of sentences in the target sentence set is greater than a first threshold, the method further includes:

[0019] If it is determined that the number of sentences in the target sentence set is less than or equal to the first threshold, the sentence type of the sentence to be compressed is determined, and extraction rules corresponding to the sentence type are obtained from the second database. Sentences are extracted from the target sentence set based on the extraction rules, and the extracted sentences are spliced ​​to obtain key sentences.

[0020] Optionally, extracting the main words of the key sentence includes:

[0021] Performing word segmentation processing on the key sentence to obtain a third word sequence;

[0022] The part of speech of each word in the third word sequence is identified in turn, the syntactic structure of the third word sequence is determined based on the part of speech and a preset syntactic analysis strategy, and the main words in the third word sequence are extracted based on the syntactic structure.

[0023] Optionally, after determining whether the sentence length of the key sentence is greater than a second threshold, the method further includes:

[0024] If it is determined that the sentence length of the key sentence is less than or equal to the second threshold, the key sentence is used as the target sentence.

[0025] In order to solve the above problem, the present invention further provides a sentence compression device, the device comprising:

[0026] A parsing module, configured to parse a sentence compression request issued by a user based on a client, obtain a sentence to be compressed carried in the request, perform a de-colloquialization process on the sentence to be compressed, obtain a target sentence set, and determine whether the number of sentences in the target sentence set is greater than a first threshold;

[0027] A sorting module, configured to sort the sentences in the target sentence set by importance when it is determined that the number of sentences in the target sentence set is greater than a first threshold, extract key sentences based on the sorting result, and determine whether the sentence length of the key sentences is greater than a second threshold;

[0028] The extraction module is used to extract the main words of the key sentence and concatenate the main words to obtain the target sentence when it is determined that the sentence length of the key sentence is greater than the second threshold.

[0029] Optionally, the step of ranking the sentences in the target sentence set by importance and extracting key sentences based on the ranking result includes:

[0030] Combining each sentence in the target sentence set with other sentences in pairs to obtain a plurality of combination pairs;

[0031] Calculating similarity values ​​of two sentences of each combination pair in the plurality of combination pairs, and determining a similarity matrix corresponding to the target sentence set based on the similarity values;

[0032] The importance score of each sentence in the target sentence set is calculated based on the similarity matrix, and the sentences in the target sentence set are sorted in descending order of importance score, and the sentence at the top of the sorting is used as the key sentence.

[0033] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0034] at least one processor; and,

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores a statement compression program that can be executed by the at least one processor, and the statement compression program is executed by the at least one processor so that the at least one processor can execute the above statement compression method.

[0037] In order to solve the above problem, the present invention also provides a computer-readable storage medium, on which a statement compression program is stored. The statement compression program can be executed by one or more processors to implement the above statement compression method.

[0038] Compared with the prior art, the present invention first performs de-colloquialization processing on the sentences to be compressed to obtain a target sentence set. This step removes colloquial sentences and colloquial words without semantic information in the sentences to be compressed by performing de-colloquialization processing, thereby achieving preliminary compression of the sentences to be compressed; then, when it is judged that the number of sentences in the target sentence set is greater than the first threshold, the sentences in the target sentence set are sorted by importance, and key sentences are extracted based on the sorting results. This step extracts the key sentences in the sentences to be compressed by importance sorting, further removes redundant information, and retains the semantic information of the sentences to be compressed; finally, when it is judged that the sentence length of the key sentences is greater than the second threshold, the main words of the key sentences are extracted, and the main words are spliced ​​to obtain the target sentences. This step further removes redundant words by extracting the main words of the key sentences, thereby ensuring the semantic accuracy of the target sentences. At the same time, the present invention does not need to label samples. Therefore, the present invention reduces the labeling cost and ensures the semantic accuracy of the compressed sentences. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a statement compression method provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a module of a sentence compression device provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the structure of an electronic device for implementing a sentence compression method provided by an embodiment of the present invention;

[0042] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0045] The present invention provides a method for compressing sentences. Figure 1 FIG. 1 is a flow chart of a method for compressing sentences according to an embodiment of the present invention. The method can be executed by an electronic device, and the electronic device can be implemented by software and / or hardware.

[0046] In this embodiment, the statement compression method includes:

[0047] S1. Parse a sentence compression request issued by a user based on a client, obtain the sentence to be compressed carried in the request, perform de-colloquialization processing on the sentence to be compressed, obtain a target sentence set, and determine whether the number of sentences in the target sentence set is greater than a first threshold.

[0048] The performing the de-colloquialization process on the sentence to be compressed includes:

[0049] A11, obtaining a spoken sentence dictionary from a first database, comparing each first clause in the sentence to be compressed with the spoken sentence dictionary, and if a specified first clause matches a sentence in the spoken sentence dictionary, deleting the specified first clause to obtain an initial sentence set;

[0050] A12, performing word segmentation processing on the sentences in the initial sentence set to obtain a first word sequence;

[0051] In this embodiment, a statistical probability model or / and a word segmentation method based on an N-gram language model may be used to segment the sentences in the initial sentence set.

[0052] A13, identifying spoken words in the first word sequence based on a spoken word recognition model, deleting the spoken words, and obtaining a second word sequence;

[0053] In this embodiment, the spoken word recognition model is a deep neural network model, which identifies the part-of-speech tag of each word in the first word sequence and removes spoken language (modal particles, etc.) based on the part-of-speech tags.

[0054] A14. Concatenate the words in the second word sequence according to the position of each word in the sentence to be compressed to obtain a plurality of second clauses, and use the set of the plurality of second clauses as a target sentence set.

[0055] In this embodiment, the sentence to be compressed is a long sentence composed of multiple sentences, and the spoken sentence dictionary stores multiple slang sentences without semantic information.

[0056] For example, if the sentence to be compressed is "Uh, I know all this, and I have borrowed all I can. I don't get paid until the 15th of every month. Uh, I can only wait until the 15th to get paid. I have to pay it back on the 15th of every month. There is no other way."

[0057] The set of sentences remaining after removing the sloppy sentence "I know all this" and the modal particle "eh" is taken as the target sentence set.

[0058] S2. When it is determined that the number of sentences in the target sentence set is greater than a first threshold, the sentences in the target sentence set are sorted by importance, key sentences are extracted based on the sorting result, and it is determined whether the sentence length of the key sentence is greater than a second threshold.

[0059] In this embodiment, the first threshold may be 5.

[0060] The step of sorting the sentences in the target sentence set by importance and extracting key sentences based on the sorting results includes:

[0061] B11, combining each sentence in the target sentence set with other sentences in pairs to obtain a plurality of combination pairs;

[0062] B12, calculating the similarity value of two sentences of each combination pair in the multiple combination pairs, and determining the similarity matrix corresponding to the target sentence set based on the similarity value;

[0063] B13. Calculate the importance score of each sentence in the target sentence set based on the similarity matrix, sort the sentences in the target sentence set in descending order of importance score, and use the sentence with the highest importance score as the key sentence.

[0064] Assume that the similarity values ​​corresponding to each combination pair in the target sentence set are shown in Table 1 below:

[0065] Similarity value Sentence 1 Sentence 2 Sentence 3 Sentence 1 1 0.63 0.44 Sentence 2 0.63 1 0.78 Sentence 3 0.44 0.78 1

[0066] Table 1

[0067] Then the similarity matrix corresponding to the target sentence set is

[0068]

[0069] The calculation formula of the importance score is:

[0070] w i =(1-d)+d*s*w i′

[0071] Among them, w i is the importance score of the i-th sentence in the target sentence set, d is the damping coefficient (ranging from 0 to 1, generally 0.85), s is the similarity matrix corresponding to the target sentence set, and w i′ is the importance score obtained in the previous iteration of the i-th sentence in the target sentence set.

[0072] In this embodiment, the initial importance score of each sentence is 1, and the final importance score of each sentence is iteratively propagated according to the above importance score calculation formula. Convergence is reached when the error rate of any sentence is less than a given limit value (for example, 0.0001).

[0073] In this embodiment, cosine similarity, Euclidean distance, Manhattan distance and Minkowski distance algorithms may be used to calculate the similarity value of two sentences in each combination pair.

[0074] After determining whether the number of sentences in the target sentence set is greater than a first threshold, the method further includes:

[0075] If it is determined that the number of sentences in the target sentence set is less than or equal to the first threshold, the sentence type of the sentence to be compressed is determined, and extraction rules corresponding to the sentence type are obtained from the second database. Sentences are extracted from the target sentence set based on the extraction rules, and the extracted sentences are spliced ​​to obtain key sentences.

[0076] In this embodiment, when the number of sentences in the target sentence set is less than a first threshold value (for example, 5), the sentence type of the sentence to be compressed is first determined, and the sentence type includes a question sentence pattern, an answer sentence pattern, and a declarative sentence pattern. The second database pre-stores extraction rules corresponding to various sentence types. For example, the extraction rule corresponding to the question sentence pattern may be to extract the two sentences at the end of the sentence to be compressed in the target sentence set, the extraction rule corresponding to the answer sentence pattern may be to extract the two sentences at the beginning and end of the sentence to be compressed in the target sentence set, and the descriptive sentence pattern may be to extract the three sentences at the beginning, end, and middle of the sentence to be compressed in the target sentence set.

[0077] In this embodiment, there is no restriction on the extraction rules, and the user can set corresponding extraction rules according to specific scenarios.

[0078] The extracted sentences are concatenated according to their order in the sentences to be compressed to obtain key sentences.

[0079] S3. When it is determined that the sentence length of the key sentence is greater than the second threshold, extract the main words of the key sentence, and concatenate the main words to obtain a target sentence.

[0080] In this embodiment, the main words of the key sentence are extracted including:

[0081] C11, performing word segmentation processing on the key sentence to obtain a third word sequence;

[0082] C12. Identify the part of speech of each word in the third word sequence in turn, determine the syntactic structure of the third word sequence based on the part of speech and a preset syntactic analysis strategy, and extract the main words in the third word sequence based on the syntactic structure.

[0083] In this embodiment, the parts of speech include nouns, verbs, adjectives, prepositions, negative words, adverbs, auxiliary words, etc.

[0084] The preset syntactic analysis strategy is dependency syntactic analysis, and the determining of the syntactic structure of the third word sequence based on the part of speech and the preset syntactic analysis strategy includes:

[0085] D11, determining a core word in the third word sequence based on the part of speech;

[0086] Usually, the verb is the core word (there is usually only one verb in a sentence).

[0087] D12, determining the subordinate relationship between each word in the third word sequence;

[0088] For example, if the key sentence is: I ate a big apple, then the third word sequence is {I, ate, a, big, apple}, and its core word is "eat".

[0089] When analyzing the subordinate relationship, "one" is subordinate to "apple" and "big" is also subordinate to "apple".

[0090] D13. Determine the syntactic structure of the third word sequence based on the core words and the subordinate relationship.

[0091] The syntactic structure includes: subject-predicate relationship, verb-object relationship, intermezzo-object relationship, prepositioned object, attributive-predicate relationship, adverbial-predicate structure, verb-complement structure, parallel relationship, preposition-object relationship, left-attachment relationship and right-attachment relationship.

[0092] The syntactic structure corresponding to the third word sequence {I, ate, a, big, apple} is {2: subject-predicate relationship, 6: verb-object relationship, 2: verb-complement relationship, 6, attributive relationship, 6: attributive relationship, 2: verb-object relationship}.

[0093] The main words extracted according to the above syntactic structure are "I", "eat", and "apple".

[0094] In this embodiment, after determining whether the sentence length of the key sentence is greater than the second threshold, the method further includes:

[0095] If it is determined that the sentence length of the key sentence is less than or equal to the second threshold, the key sentence is used as the target sentence.

[0096] It can be seen from the above embodiments that the sentence compression method proposed by the present invention first performs de-colloquialization processing on the sentences to be compressed to obtain a target sentence set. This step removes colloquial sentences and colloquial words without semantic information in the sentences to be compressed by performing de-colloquialization processing, thereby achieving preliminary compression of the sentences to be compressed; then, when it is judged that the number of sentences in the target sentence set is greater than the first threshold, the sentences in the target sentence set are sorted by importance, and key sentences are extracted based on the sorting results. This step extracts the key sentences in the sentences to be compressed by importance sorting, further removes redundant information, and retains the semantic information of the sentences to be compressed; finally, when it is judged that the sentence length of the key sentence is greater than the second threshold, the main words of the key sentence are extracted, and the main words are spliced ​​to obtain the target sentence. This step further removes redundant words by extracting the main words of the key sentence, thereby ensuring the semantic accuracy of the target sentence. At the same time, the present invention does not require labeled samples. Therefore, the present invention reduces the labeling cost and ensures the semantic accuracy of the compressed sentence.

[0097] like Figure 2 FIG. 1 is a schematic diagram of a module of a sentence compression device provided in one embodiment of the present invention.

[0098] The sentence compression device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the sentence compression device 100 may include a parsing module 110, a sorting module 120 and an extraction module 130. The module of the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and is stored in the memory of the electronic device.

[0099] In this embodiment, the functions of each module / unit are as follows:

[0100] The parsing module 110 is used to parse the sentence compression request issued by the user based on the client, obtain the sentence to be compressed carried in the request, perform de-colloquialization processing on the sentence to be compressed, obtain a target sentence set, and determine whether the number of sentences in the target sentence set is greater than a first threshold.

[0101] The performing the de-colloquialization process on the sentence to be compressed includes:

[0102] A21, obtaining a spoken sentence dictionary from a first database, comparing each first clause in the sentence to be compressed with the spoken sentence dictionary, and if a specified first clause matches a sentence in the spoken sentence dictionary, deleting the specified first clause to obtain an initial sentence set;

[0103] A22, performing word segmentation processing on the sentences in the initial sentence set to obtain a first word sequence;

[0104] In this embodiment, a statistical probability model or / and a word segmentation method based on an N-gram language model may be used to segment the sentences in the initial sentence set.

[0105] A23, identifying spoken words in the first word sequence based on a spoken word recognition model, deleting the spoken words, and obtaining a second word sequence;

[0106] In this embodiment, the spoken word recognition model is a deep neural network model, which identifies the part-of-speech tag of each word in the first word sequence and removes spoken language (modal particles, etc.) based on the part-of-speech tags.

[0107] A24. Concatenate the words in the second word sequence according to the position of each word in the sentence to be compressed to obtain a plurality of second clauses, and use the set of the plurality of second clauses as a target sentence set.

[0108] In this embodiment, the sentence to be compressed is a long sentence composed of multiple sentences, and the spoken sentence dictionary stores multiple slang sentences without semantic information.

[0109] For example, if the sentence to be compressed is "Uh, I know all this, and I have borrowed all I can. I don't get paid until the 15th of every month. Uh, I can only wait until the 15th to get paid. I have to pay it back on the 15th of every month. There is no other way."

[0110] The set of sentences remaining after removing the sloppy sentence "I know all this" and the modal particle "eh" is taken as the target sentence set.

[0111] The sorting module 120 is used to sort the sentences in the target sentence set by importance when it is determined that the number of sentences in the target sentence set is greater than a first threshold, extract key sentences based on the sorting result, and determine whether the sentence length of the key sentences is greater than a second threshold.

[0112] In this embodiment, the first threshold may be 5.

[0113] The step of sorting the sentences in the target sentence set by importance and extracting key sentences based on the sorting results includes:

[0114] B21, combining each sentence in the target sentence set with other sentences in pairs to obtain a plurality of combination pairs;

[0115] B22, calculating the similarity value of two sentences of each combination pair in the multiple combination pairs, and determining the similarity matrix corresponding to the target sentence set based on the similarity value;

[0116] B23. Calculate the importance score of each sentence in the target sentence set based on the similarity matrix, sort the sentences in the target sentence set in descending order of importance score, and use the sentence with the highest importance score as the key sentence.

[0117] Assume that the similarity values ​​corresponding to each combination pair in the target sentence set are as shown in Table 1 above.

[0118] Then the similarity matrix corresponding to the target sentence set is

[0119]

[0120] The calculation formula of the importance score is:

[0121] w i =(1-d)+d*s*w i′

[0122] Among them, w i is the importance score of the i-th sentence in the target sentence set, d is the damping coefficient (ranging from 0 to 1, generally 0.85), s is the similarity matrix corresponding to the target sentence set, and w i′ is the importance score obtained in the previous iteration of the i-th sentence in the target sentence set.

[0123] In this embodiment, the initial importance score of each sentence is 1, and the final importance score of each sentence is iteratively propagated according to the above importance score calculation formula. Convergence is reached when the error rate of any sentence is less than a given limit value (for example, 0.0001).

[0124] In this embodiment, cosine similarity, Euclidean distance, Manhattan distance and Minkowski distance algorithms may be used to calculate the similarity value of two sentences in each combination pair.

[0125] After determining whether the number of sentences in the target sentence set is greater than a first threshold, the sorting module 120 is further configured to:

[0126] If it is determined that the number of sentences in the target sentence set is less than or equal to the first threshold, the sentence type of the sentence to be compressed is determined, and extraction rules corresponding to the sentence type are obtained from the second database. Sentences are extracted from the target sentence set based on the extraction rules, and the extracted sentences are spliced ​​to obtain key sentences.

[0127] In this embodiment, when the number of sentences in the target sentence set is less than a first threshold value (for example, 5), the sentence type of the sentence to be compressed is first determined, and the sentence type includes a question sentence pattern, an answer sentence pattern, and a declarative sentence pattern. The second database pre-stores extraction rules corresponding to various sentence types. For example, the extraction rule corresponding to the question sentence pattern may be to extract the two sentences at the end of the sentence to be compressed in the target sentence set, the extraction rule corresponding to the answer sentence pattern may be to extract the two sentences at the beginning and end of the sentence to be compressed in the target sentence set, and the descriptive sentence pattern may be to extract the three sentences at the beginning, end, and middle of the sentence to be compressed in the target sentence set.

[0128] In this embodiment, there is no restriction on the extraction rules, and the user can set corresponding extraction rules according to specific scenarios.

[0129] The extracted sentences are concatenated according to their order in the sentences to be compressed to obtain key sentences.

[0130] The extraction module 130 is used to extract the main words of the key sentence and concatenate the main words to obtain the target sentence when it is determined that the sentence length of the key sentence is greater than the second threshold.

[0131] In this embodiment, the main words of the key sentence are extracted including:

[0132] C21, performing word segmentation processing on the key sentence to obtain a third word sequence;

[0133] C22. Identify the part of speech of each word in the third word sequence in turn, determine the syntactic structure of the third word sequence based on the part of speech and a preset syntactic analysis strategy, and extract the main words in the third word sequence based on the syntactic structure.

[0134] In this embodiment, the parts of speech include nouns, verbs, adjectives, prepositions, negative words, adverbs, auxiliary words, etc.

[0135] The preset syntactic analysis strategy is dependency syntactic analysis, and the determining of the syntactic structure of the third word sequence based on the part of speech and the preset syntactic analysis strategy includes:

[0136] D21, determining a core word in the third word sequence based on the part of speech;

[0137] Usually, the verb is the core word (there is usually only one verb in a sentence).

[0138] D22, determining the subordinate relationship between each word in the third word sequence;

[0139] For example, if the key sentence is: I ate a big apple, then the third word sequence is {I, ate, a, big, apple}, and its core word is "eat".

[0140] When analyzing the subordinate relationship, "one" is subordinate to "apple" and "big" is also subordinate to "apple".

[0141] D23. Determine the syntactic structure of the third word sequence based on the core words and the subordinate relationship.

[0142] The syntactic structure includes: subject-predicate relationship, verb-object relationship, intermezzo-object relationship, prepositioned object, attributive-predicate relationship, adverbial-predicate structure, verb-complement structure, parallel relationship, preposition-object relationship, left-attachment relationship and right-attachment relationship.

[0143] The syntactic structure corresponding to the third word sequence {I, ate, a, big, apple} is {2: subject-predicate relationship, 6: verb-object relationship, 2: verb-complement relationship, 6, attributive relationship, 6: attributive relationship, 2: verb-object relationship}.

[0144] The main words extracted according to the above syntactic structure are "I", "eat", and "apple".

[0145] In this embodiment, after determining whether the sentence length of the key sentence is greater than the second threshold, the extraction module 130 is further configured to:

[0146] If it is determined that the sentence length of the key sentence is less than or equal to the second threshold, the key sentence is used as the target sentence.

[0147] It can be seen from the above embodiments that the sentence compression device 100 proposed by the present invention first performs de-colloquialization processing on the sentences to be compressed to obtain a target sentence set. This step removes colloquial sentences and colloquial words without semantic information in the sentences to be compressed by performing de-colloquialization processing, thereby achieving preliminary compression of the sentences to be compressed; then, when it is judged that the number of sentences in the target sentence set is greater than the first threshold, the sentences in the target sentence set are sorted by importance, and key sentences are extracted based on the sorting results. This step extracts the key sentences in the sentences to be compressed by importance sorting, further removes redundant information, and retains the semantic information of the sentences to be compressed; finally, when it is judged that the sentence length of the key sentence is greater than the second threshold, the main words of the key sentence are extracted, and the main words are spliced ​​to obtain the target sentence. This step further removes redundant words by extracting the main words of the key sentence, thereby ensuring the semantic accuracy of the target sentence. At the same time, the present invention does not require labeled samples. Therefore, the present invention reduces the labeling cost and ensures the semantic accuracy of the compressed sentence.

[0148] like Figure 3 , which is a schematic diagram of the structure of an electronic device for implementing a sentence compression method provided by an embodiment of the present invention.

[0149] The electronic device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The electronic device 1 can be a computer, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets.

[0150] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected and communicated via a system bus. The memory 11 stores a statement compression program 10 that can be executed by the processor 12. Figure 3 Only the electronic device 1 having components 11-13 and the statement compression program 10 is shown, and those skilled in the art can understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0151] Among them, the memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. In this embodiment, the readable storage medium of the memory 11 is generally used to store an operating system and various application software installed on the electronic device 1, such as storing the code of the statement compression program 10 in an embodiment of the present invention, etc. In addition, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.

[0152] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the statement compression program 10.

[0153] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a client (not shown in the figure).

[0154] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0155] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0156] The statement compression program 10 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 12, it can achieve:

[0157] Parsing a sentence compression request issued by a user based on a client, obtaining a sentence to be compressed carried in the request, performing a de-colloquialization process on the sentence to be compressed to obtain a target sentence set, and determining whether the number of sentences in the target sentence set is greater than a first threshold;

[0158] When it is determined that the number of sentences in the target sentence set is greater than a first threshold, the sentences in the target sentence set are sorted by importance, key sentences are extracted based on the sorting result, and it is determined whether the sentence length of the key sentence is greater than a second threshold;

[0159] When it is determined that the sentence length of the key sentence is greater than the second threshold, the main words of the key sentence are extracted, and the main words are concatenated to obtain the target sentence.

[0160] Specifically, the processor 12 can refer to the specific implementation method of the above statement compression program 10. Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0161] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0162] The computer-readable storage medium stores a statement compression program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the above-mentioned statement compression method embodiments, and will not be described in detail here.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0164] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0166] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0167] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0168] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A sentence compression method, characterized in that: The method comprises: Parsing a sentence compression request issued by a user based on a client, obtaining a sentence to be compressed carried in the request, performing a de-colloquialization process on the sentence to be compressed to obtain a target sentence set, and determining whether the number of sentences in the target sentence set is greater than a first threshold; When it is determined that the number of sentences in the target sentence set is greater than a first threshold, the sentences in the target sentence set are sorted by importance, and key sentences are extracted based on the sorting result. If it is determined that the number of sentences in the target sentence set is less than or equal to the first threshold, the sentence type of the sentence to be compressed is determined, and the sentence type includes a question sentence pattern, an answer sentence pattern, and a declarative sentence pattern. An extraction rule corresponding to the sentence type is obtained from a second database, and sentences are extracted from the target sentence set based on the extraction rule. The extracted sentences are spliced ​​to obtain key sentences, and it is determined whether the sentence length of the key sentence is greater than a second threshold. When it is determined that the sentence length of the key sentence is greater than the second threshold, performing word segmentation processing on the key sentence to obtain a third word sequence, identifying the part of speech of each word in the third word sequence in turn, determining a core word in the third word sequence based on the part of speech, determining a subordinate relationship between each word in the third word sequence, determining a syntactic structure of the third word sequence based on the core word and the subordinate relationship, extracting stem words in the third word sequence based on the syntactic structure, and concatenating the stem words to obtain a target sentence; When it is determined that the sentence length of the key sentence is less than or equal to the second threshold, the key sentence is used as the target sentence.

2. The sentence compression method according to claim 1, characterized in that: The performing the de-colloquialization process on the sentence to be compressed includes: Acquire a spoken sentence dictionary from a first database, compare each first clause in the sentence to be compressed with the spoken sentence dictionary, and if a specified first clause matches a sentence in the spoken sentence dictionary, delete the specified first clause to obtain an initial sentence set; Performing word segmentation processing on the sentences in the initial sentence set to obtain a first word sequence; identifying spoken words in the first word sequence based on a spoken word recognition model, deleting the spoken words, and obtaining a second word sequence; The words in the second word sequence are concatenated according to the positions of the words in the sentence to be compressed to obtain a plurality of second clauses, and a set of the plurality of second clauses is used as a target sentence set.

3. The sentence compression method according to claim 1, characterized in that: The step of sorting the sentences in the target sentence set by importance and extracting key sentences based on the sorting results includes: Combining each sentence in the target sentence set with other sentences in pairs to obtain a plurality of combination pairs; Calculating similarity values ​​of two sentences of each combination pair in the plurality of combination pairs, and determining a similarity matrix corresponding to the target sentence set based on the similarity values; The importance score of each sentence in the target sentence set is calculated based on the similarity matrix, and the sentences in the target sentence set are sorted in descending order of importance score, and the sentence at the top of the sorting is used as the key sentence.

4. A sentence compression device, characterized in that: The device comprises: A parsing module, configured to parse a sentence compression request issued by a user based on a client, obtain a sentence to be compressed carried in the request, perform a de-colloquialization process on the sentence to be compressed, obtain a target sentence set, and determine whether the number of sentences in the target sentence set is greater than a first threshold; A sorting module, used for sorting the sentences in the target sentence set by importance when it is determined that the number of sentences in the target sentence set is greater than a first threshold, extracting key sentences based on the sorting result, and if it is determined that the number of sentences in the target sentence set is less than or equal to the first threshold, determining the sentence type of the sentence to be compressed, the sentence type including a question sentence pattern, an answer sentence pattern and a declarative sentence pattern, obtaining an extraction rule corresponding to the sentence type from a second database, extracting sentences from the target sentence set based on the extraction rule, splicing the extracted sentences to obtain key sentences, and determining whether the sentence length of the key sentence is greater than a second threshold; The extraction module is used to perform word segmentation processing on the key sentence when it is determined that the sentence length of the key sentence is greater than a second threshold value to obtain a third word sequence, identify the part of speech of each word in the third word sequence in turn, determine the core words in the third word sequence based on the part of speech, determine the subordinate relationship between the words in the third word sequence, determine the syntactic structure of the third word sequence based on the core words and the subordinate relationship, extract the trunk words in the third word sequence based on the syntactic structure, splice the trunk words to obtain a target sentence, and when it is determined that the sentence length of the key sentence is less than or equal to the second threshold value, use the key sentence as the target sentence.

5. The sentence compression device according to claim 4, characterized in that: The step of sorting the sentences in the target sentence set by importance and extracting key sentences based on the sorting results includes: Combining each sentence in the target sentence set with other sentences in pairs to obtain a plurality of combination pairs; Calculating similarity values ​​of two sentences of each combination pair in the plurality of combination pairs, and determining a similarity matrix corresponding to the target sentence set based on the similarity values; The importance score of each sentence in the target sentence set is calculated based on the similarity matrix, and the sentences in the target sentence set are sorted in descending order of importance score, and the sentence at the top of the sorting is used as the key sentence.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a statement compression program that can be executed by the at least one processor, and the statement compression program is executed by the at least one processor so that the at least one processor can execute the statement compression method according to any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a statement compression program, which can be executed by one or more processors to implement the statement compression method according to any one of claims 1 to 3.

Images