Text processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a text processing method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining a to-be-processed text; performing semantic segmentation on the to-be-processed text to obtain a plurality of text blocks; wherein the semantics of the adjacent text blocks are different; generating a text block abstract of each text block; wherein one text block corresponds to one text block abstract; and generating an abstract of the to-be-processed text based on the text block abstract of each text block. Through the method, the accuracy of the to-be-processed text abstract can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of text processing technology, and in particular to a text processing method, device, electronic device, storage medium, and program product. Background Art

[0002] With the continuous advancement of natural language processing technology, more and more people use natural language processing technology to process text, such as using natural language processing technology to generate text summaries. When generating summaries for long texts, in order to reduce the memory usage of data generated during the text processing process and improve processing efficiency, the long texts are often segmented, and summaries are generated for the segmented texts, and then spliced ​​into summaries of the long texts. However, the text summaries generated by the current segmentation strategies for long texts are less accurate. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a text processing method, device, electronic device, storage medium and program product.

[0004] According to a first aspect of an embodiment of the present disclosure, a text processing method is provided, the method comprising:

[0005] Get the text to be processed;

[0006] Performing semantic segmentation on the text to be processed to obtain a plurality of text blocks, wherein adjacent text blocks have different semantics;

[0007] Generate a text block summary for each text block; wherein, one text block corresponds to one text block summary;

[0008] Based on the text block summaries of the text blocks, a summary of the text to be processed is generated.

[0009] In some embodiments, the semantic segmentation of the text to be processed to obtain multiple text blocks includes:

[0010] Dividing the text to be processed into a plurality of text segments and determining semantic features corresponding to each text segment; wherein the characters in each text segment are continuous;

[0011] Determining semantic similarity of adjacent text segments based on semantic features of adjacent text segments;

[0012] Based on the semantic similarity of adjacent text segments in the multiple text segments, semantic segmentation is performed on the text to be processed to obtain the multiple text blocks.

[0013] In some embodiments, dividing the to-be-processed text into a plurality of text segments includes:

[0014] The text to be processed is segmented based on a preset character length threshold to obtain a plurality of text units; wherein the number of characters included in each text unit is less than the preset character length threshold, and the last character included in the previous text unit and the first character included in the next text unit in adjacent text units are consecutively positioned in the text to be processed;

[0015] Adjacent text units in the multiple text units are merged in sequence to obtain multiple text segments; wherein the same text units exist in adjacent text segments.

[0016] In some embodiments, the segmentation of the text to be processed based on a preset character length threshold is guided to obtain multiple text units, including:

[0017] Segmenting the text to be processed into paragraphs to obtain multiple paragraphs;

[0018] Segmenting the paragraphs whose number of characters is greater than or equal to the preset character length threshold among the multiple paragraphs to obtain multiple sentences;

[0019] Segmenting the sentences corresponding to the paragraph, wherein the number of characters included in the sentences is greater than or equal to the preset character length threshold, to obtain a plurality of words; wherein the number of characters included in each word is less than the preset character length threshold;

[0020] The paragraphs, sentences and words whose number of characters is less than the preset character length threshold are determined as the multiple text units.

[0021] In some embodiments, sequentially merging adjacent text units in the plurality of text units to obtain a plurality of text segments includes:

[0022] For a first text unit among the plurality of text units, merging the first text unit with a first preset number of adjacent text units to obtain a text segment corresponding to the first text unit;

[0023] For each text unit after the first text unit, selecting a second preset number of adjacent text units before and a first preset number of adjacent text units after the first text unit and merging them with the text unit to obtain a text segment corresponding to the text unit;

[0024] The text segments corresponding to the text units form the multiple text segments.

[0025] In some embodiments, the semantic segmentation of the to-be-processed text based on the semantic similarity of adjacent text segments in the multiple text segments to obtain the multiple text blocks includes:

[0026] For a non-first text segment, in response to a semantic similarity between the text segment and an adjacent previous text segment being greater than a preset similarity threshold, dividing text units that are different from those in the text segment and the adjacent previous text segment into the same text block as text units included in the adjacent previous text segment;

[0027] In response to the semantic similarity between the text segment and the adjacent previous text segment being less than or equal to the preset similarity threshold, the text units that are different from those in the text segment and the adjacent previous text segment and the text units included in the adjacent previous text segment are divided into different text blocks.

[0028] In some embodiments, generating a text block summary for each text block includes:

[0029] Generate an initial summary for each text block, and generate a summary topic for each text block based on the initial summary of each text block; wherein one text block corresponds to one initial summary, and one initial summary corresponds to one summary topic;

[0030] Guided by the summary topic of each text block, a preset large language model is used to generate a text block summary for each text block.

[0031] In some embodiments, generating a summary of the to-be-processed text based on the text block summary of each text block includes:

[0032] The text block summaries of the text blocks are integrated using a preset large language model to generate a summary of the text to be processed.

[0033] In some embodiments, generating a text block summary for each text block includes:

[0034] Generate a text block summary for each text block using edge devices.

[0035] In some embodiments, generating a text block summary for each text block using an edge device includes:

[0036] Multiple edge devices are used to generate text block summaries for each text block in parallel.

[0037] According to a second aspect of an embodiment of the present disclosure, a text processing device is provided, the device comprising:

[0038] An acquisition module configured to acquire text to be processed;

[0039] a semantic segmentation module configured to perform semantic segmentation on the text to be processed to obtain a plurality of text blocks; wherein adjacent text blocks have different semantics;

[0040] A first generating module is configured to generate a text block summary for each text block; wherein one text block corresponds to one text block summary;

[0041] The second generating module is configured to generate a summary of the to-be-processed text based on the text block summary of each text block.

[0042] In some embodiments, the semantic segmentation module is further configured to divide the text to be processed into multiple text segments and determine the semantic features corresponding to each text segment; wherein the characters in each text segment are continuous; based on the semantic features of adjacent text segments, the semantic similarity of adjacent text segments is determined; based on the semantic similarity of each adjacent text segment in the multiple text segments, the text to be processed is semantically segmented to obtain the multiple text blocks.

[0043] In some embodiments, the semantic segmentation module is further configured to segment the text to be processed based on a preset character length threshold to obtain multiple text units; wherein the number of characters included in each text unit is less than the preset character length threshold, and the last character included in the previous text unit and the first character included in the next text unit in adjacent text units are continuous in the position of the text to be processed; adjacent text units in the multiple text units are merged in sequence to obtain multiple text fragments; wherein the same text units exist in adjacent text fragments.

[0044] In some embodiments, the semantic segmentation module is further configured to perform paragraph segmentation on the text to be processed to obtain multiple paragraphs; perform sentence segmentation on the paragraphs among the multiple paragraphs whose number of characters is greater than or equal to the preset character length threshold to obtain multiple sentences; perform sentence segmentation on the multiple sentences corresponding to the paragraphs whose number of characters is greater than or equal to the preset character length threshold to obtain multiple words; wherein the number of characters included in each word is less than the preset character length threshold; and determine the paragraphs, sentences and words whose number of characters is less than the preset character length threshold as the multiple text units.

[0045] In some embodiments, the semantic segmentation module is further configured to, for the first text unit among the multiple text units, merge the first text unit with the adjacent first preset number of text units to obtain a text fragment corresponding to the first text unit; for each text unit after the first text unit, select the adjacent first second preset number of text units and the adjacent first preset number of text units and merge them with the text unit to obtain a text fragment corresponding to the text unit; wherein, the text fragments corresponding to each text unit form the multiple text fragments.

[0046] In some embodiments, the semantic segmentation module is further configured to, for a non-first text segment, in response to the semantic similarity between the text segment and the adjacent previous text segment being greater than a preset similarity threshold, divide the text units that are different from the text segment and the adjacent previous text segment, and the text units included in the adjacent previous text segment into the same text block; in response to the semantic similarity between the text segment and the adjacent previous text segment being less than or equal to the preset similarity threshold, divide the text units that are different from the text segment and the adjacent previous text segment, and the text units included in the adjacent previous text segment into different text blocks.

[0047] In some embodiments, the first generation module is further configured to generate an initial summary for each text block, and generate a summary topic for each text block based on the initial summary of each text block; wherein, one text block corresponds to one initial summary, and one initial summary corresponds to one summary topic; guided by the summary topic of each text block, a preset large language model is used to generate a text block summary for each text block.

[0048] In some embodiments, the second generating module is further configured to integrate the text block summaries of each text block using a preset large language model to generate a summary of the text to be processed.

[0049] In some embodiments, the first generation module is further configured to generate a text block summary for each text block using an edge device.

[0050] In some embodiments, the first generation module is further configured to utilize multiple edge devices to generate text block summaries for each text block in parallel.

[0051] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0052] processor;

[0053] memory for storing computer programs or instructions;

[0054] The processor executes the computer program or instructions to implement the steps of the text processing method described in the first aspect above.

[0055] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, which stores a computer program or instructions. When the computer program or instructions in the storage medium are executed by a processor, the steps of the text processing method described in the first aspect above are implemented.

[0056] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program or instructions, which, when executed by a processor, implement the steps of the text processing method described in the first aspect above.

[0057] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0058] In an embodiment of the present disclosure, an electronic device obtains a text to be processed; performs semantic segmentation on the text to be processed to obtain multiple text blocks, wherein adjacent text blocks have different semantics, and generates a text block summary for each text block; and then generates a summary of the text to be processed based on the text block summary of each text block. Compared with the related art which adopts a simple segmentation method based on the number of characters or line breaks, the embodiment of the present disclosure performs semantic segmentation to obtain multiple adjacent text blocks with different semantics, which can make each text block a semantically complete text block, thereby reducing problems such as missing or misunderstanding of the generated text block summary information due to improper segmentation, thereby improving the readability and accuracy of the text block summary, and thus improving the accuracy of the summary of the text to be processed.

[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0061] Figure 1 The figure is a flowchart of a text processing method according to an exemplary embodiment.

[0062] Figure 2 This is a principle diagram of a method for determining a text block according to an exemplary embodiment.

[0063] Figure 3 This is a principle diagram of a method for segmenting text to be processed according to an exemplary embodiment.

[0064] Figure 4 The figure is a block diagram of a text processing device according to an exemplary embodiment.

[0065] Figure 5 The figure is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0066] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0067] When using natural language processing to generate summaries for long texts, each phrase in the long text must be preprocessed to generate a corresponding query vector, key vector, and value vector. The similarity between each phrase's query vector and the key vectors of other phrases is then calculated to obtain an attention distribution weight for each phrase, reflecting its relative importance in the current context. Based on the attention distribution weights of each phrase, the value vectors of all phrases are weighted and summed to obtain a vector representation of the long text. This is then used to generate a text summarization model based on the vector representation of the long text.

[0068] In the above method, after generating the query vector, key vector and value vector corresponding to each phrase, it is necessary to store the query vector, key vector and value vector corresponding to each phrase for use in the subsequent generation of text summaries. However, for long texts, the storage space required for the query vectors, key vectors and value vectors of all phrases is usually large, and a large amount of memory is required. For example, it is measured through experiments that when the sequence length of the text is 8192, the query vector, key vector and value vector corresponding to each phrase in the text occupy 4G of memory, and the query vector, key vector and value vector corresponding to each phrase in the text need to be generated before subsequent processing can be performed, and the processing efficiency is low.

[0069] Therefore, to reduce the memory usage of data generated during text processing and improve processing efficiency, a text processing method exists in the related art that first segments long text, then generates summaries for the segmented text blocks and concatenates them into summaries of the long text. However, this method typically uses a simple segmentation method based on character count or line breaks. While this improves processing efficiency, it disrupts the context and logical relationships of the text, resulting in incoherent summaries of long texts or omissions of key content, and the resulting summaries are less accurate.

[0070] In this regard, the present disclosure provides a text processing method. Figure 1 FIG. 1 is a flowchart of a text processing method according to an exemplary embodiment. Figure 1 As shown, the method mainly includes the following steps:

[0071] S11, obtaining the text to be processed;

[0072] S12, performing semantic segmentation on the text to be processed to obtain a plurality of text blocks; wherein adjacent text blocks have different semantics;

[0073] S13, generating a text block summary for each text block; wherein each text block corresponds to one text block summary;

[0074] S14. Generate a summary of the text to be processed based on the text block summary of each text block.

[0075] The text processing method provided in the embodiments of the present disclosure may be executed by terminal devices such as user equipment (UE), mobile devices, user terminals, mobile phones, tablet computers, personal digital assistants (PDAs), handheld devices, computing devices, vehicle-mounted devices, and wearable devices; cloud devices such as cloud servers; and edge devices such as gateways, switches, and routers. The embodiments of the present disclosure do not limit the execution entities. For ease of description, the embodiments of the present disclosure are described using electronic devices as the execution entities.

[0076] In step S11 , the electronic device obtains a text to be processed, wherein the text to be processed may be an article published on the Internet, or may be an edited but unpublished article, or may be an article recorded in a publication.

[0077] In the disclosed embodiments, the electronic device can directly acquire the text to be processed, such as the aforementioned published articles, edited but unpublished articles, and articles in publications. The electronic device can also acquire image data, audio data, and the like, such as photographs, product images, movie clips, short videos, audiobooks, audio blogs, and song clips. The acquired image data and audio data can then be converted into the text to be processed. The disclosed embodiments are not limited to this.

[0078] In the embodiments of the present disclosure, the electronic device may obtain the text to be processed by means of a web crawler; the electronic device may also call the text to be processed through a predetermined programming interface (Application Programming Interface, API); the electronic device may also obtain the text to be processed from a database or data set storing the text to be processed through a data processing tool, and the embodiments of the present disclosure are not limited to this.

[0079] In step S12, the electronic device performs semantic segmentation on the text to be processed to obtain multiple text blocks, wherein adjacent text blocks have different semantics. The electronic device may first pre-process the text to be processed, such as removing irrelevant characters such as punctuation marks and spaces.

[0080] In some embodiments, the electronic device may first perform an initial segmentation on the text to be processed based on a preset segmentation rule, wherein the preset segmentation rule may be segmentation based on the number of characters, segmentation based on line breaks, etc. After the initial segmentation is completed, the electronic device may sequentially determine whether the semantics of each segmented text content are complete, and thereby determine the semantic similarity between each text content and the adjacent text content. If the semantics of the segmented text content are complete and the semantic similarity with the adjacent text content is less than a preset similarity threshold, the segmented text content is treated as a text block; if the semantics of the segmented text content are incomplete, the text content is merged with the adjacent text content, and then it is determined whether the semantics of the merged text content are complete. If the semantics are complete but the semantic similarity with the adjacent text content is greater than or equal to the preset similarity threshold, the text content is also merged with the adjacent text content. The above process is repeated until multiple text blocks with different semantics of adjacent text blocks are determined.

[0081] In other embodiments, the electronic device may first segment the text to be processed into multiple phrases, wherein the electronic device may segment the text to be processed into multiple phrases using a preset word segmentation tool, such as a dictionary (Jieba, etc.) or a natural language processing toolkit (NLTK). Parts of speech are tagged for each segmented phrase, such as noun, verb, adjective, etc., and then semantic segmentation is performed on the text to be processed based on the part-of-speech tagging results for each phrase. For example, when a noun phrase is encountered, a new text block is generated, and then it is determined whether the semantics of adjacent text blocks are the same. If they are the same, they are merged, thereby obtaining multiple text blocks with different semantics of adjacent text blocks.

[0082] In other embodiments, the electronic device may first segment the text to be processed into multiple phrases and determine the dependency relationship between the parts of speech of adjacent phrases. Since the subject, predicate, and object usually represent a complete sentence, the electronic device may perform semantic segmentation on the text to be processed based on the dependency relationship between the parts of speech of adjacent phrases. For example, when encountering a subject, predicate, and object, a new text block is generated, and then it is determined whether the semantics of the adjacent text blocks are the same. If they are the same, they are merged to obtain multiple text blocks with different semantics of adjacent text blocks.

[0083] In other embodiments, the electronic device may use a preset semantic segmentation model to perform semantic segmentation on the text to be processed to obtain multiple text blocks with different semantics of adjacent text blocks, wherein the preset semantic segmentation model may be obtained by training a large language model based on sample text.

[0084] In step S13, the electronic device generates a text block summary for each text block; wherein, each text block corresponds to a text block summary. In some embodiments, the electronic device may generate a text block summary for each text block using a preset large language model, wherein the preset large language model may be a model based on a transformer architecture, such as a Bidirectional Encoder Representations from Transformers (BERT) model, a Generative Pre-trained Transformer (GPT) model, etc.

[0085] In other embodiments, the electronic device may generate a text block summary for each text block based on a preset algorithm, wherein the preset algorithm may include an extractive summarization method, wherein the electronic device may identify each sentence or phrase in the text block, and evaluate the importance of each sentence or phrase based on information such as sentence position and phrase frequency, and select the most important sentences or phrases to be integrated into the summary of the text block. The preset algorithm may also include a clustering method, wherein the electronic device may identify each sentence or phrase in the text block, extract features of each sentence or phrase, cluster each sentence or phrase based on the features, and select the cluster center or sentences or phrases close to the cluster center from each cluster to be integrated into the summary of the text block.

[0086] In some embodiments, generating a text block summary for each text block includes:

[0087] Generate a text block summary for each text block using edge devices.

[0088] In the disclosed embodiments, edge devices are used to generate a block summary for each text block. Edge devices primarily refer to physical devices installed on edge networks that are responsible for transmitting data packets between access devices and network devices, such as switches, routers, and gateways. Edge devices can generate a block summary for each text block using a preset large language model or a preset algorithm. The large language model and algorithm can be the models and algorithms described above.

[0089] In some embodiments, only one edge device may be deployed, and the edge device may be used to generate a text block summary for each text block. For example, the edge device may be used to generate a text block summary for each text block according to the order of the text blocks in the text to be processed.

[0090] In some other embodiments, generating a text block summary for each text block using an edge device includes:

[0091] Multiple edge devices are used to generate text block summaries for each text block in parallel.

[0092] In embodiments of the present disclosure, multiple edge devices can be deployed at the edge of the network, where each edge device can be of the same or different types. The number of edge devices and text blocks can be determined first, and then the text blocks can be evenly distributed to each edge device so that each edge device processes the same number of text blocks. The number of text blocks processed by each edge device can also be determined based on the computing power and / or memory of the edge device.

[0093] In the embodiment of the present disclosure, a plurality of edge devices are used to generate text block summaries for each text block in parallel. On the one hand, since the edge devices are deployed at the edge of the network, after the text to be processed for summaries is generated, there is no need to upload the text to be processed to cloud devices such as cloud servers for processing, nor is there any need to occupy the memory and process and other resources of the terminal device. This can improve the processing efficiency of the text to be processed, and can also reduce problems such as data leakage caused by transmitting data to cloud devices, and has higher security. On the other hand, deploying multiple edge devices and generating text block summaries for each text block in parallel can further improve the processing efficiency of the text to be processed, speed up the generation of summaries of the text to be processed, and has higher intelligence.

[0094] In step S14, the electronic device generates a summary of the text to be processed based on the text block summaries of each text block. In some embodiments, the electronic device may concatenate the text block summaries of each text block based on the text block summaries and the positions of each text block in the text to be processed to obtain a summary of the text to be processed. It should be noted that after concatenation, post-processing such as deduplication, logical order adjustment, addition of transitional sentences, and vocabulary optimization may be performed to improve the quality of the summary of the text to be processed.

[0095] In some other embodiments, generating a summary of the to-be-processed text based on the text block summary of each text block includes:

[0096] The text block summaries of the text blocks are integrated using a preset large language model to generate a summary of the text to be processed.

[0097] In an embodiment of the present disclosure, the electronic device uses a preset large language model to integrate the text block summaries of each text block to generate a summary of the text to be processed, wherein the preset large language model can be a model based on a converter architecture, such as a BERT model, a GPT model, etc. The preset large language model integrates the text block summaries of each text block to generate a summary of the text to be processed. It is not only a simple splicing of the text block summaries of each text block, but also performs the above-mentioned deduplication, logical order adjustment, transition sentence addition, vocabulary optimization and other post-processing after the splicing is completed.

[0098] It can be understood that the preset large language model is obtained through training. Therefore, using the preset large language model to integrate the text block summaries of each text block to generate a summary of the text to be processed can improve the quality of the generated summary of the text to be processed and is more intelligent.

[0099] In other embodiments, the electronic device may determine the weight of each text block relative to the text to be processed based on factors such as the position of each text block in the text to be processed, the number of characters, and the semantic importance, and perform weighted fusion on the summaries of each text block based on the weight of each text block to obtain a summary of the text to be processed.

[0100] In other embodiments, the similarity between the summaries of each text block can be determined, and the summaries of each text block can be clustered based on the similarity between the summaries of each text block, and the cluster center of each clustered cluster can be determined. The cluster center of each cluster or the summaries of text blocks whose distance from the cluster center is within a preset distance threshold can be selected and fused to obtain the summary of the text to be processed.

[0101] In an embodiment of the present disclosure, an electronic device obtains a text to be processed; performs semantic segmentation on the text to be processed to obtain multiple text blocks, wherein adjacent text blocks have different semantics, and generates a text block summary for each text block; and then generates a summary of the text to be processed based on the text block summary of each text block. Compared with the related art which adopts a simple segmentation method based on the number of characters or line breaks, the embodiment of the present disclosure performs semantic segmentation to obtain multiple adjacent text blocks with different semantics, which can make each text block a semantically complete text block, thereby reducing problems such as missing or misunderstanding of the generated text block summary information due to improper segmentation, thereby improving the readability and accuracy of the text block summary, and thus improving the accuracy of the summary of the text to be processed.

[0102] In some embodiments, the semantic segmentation of the text to be processed to obtain multiple text blocks includes:

[0103] Dividing the text to be processed into a plurality of text segments and determining semantic features corresponding to each text segment; wherein the characters in each text segment are continuous;

[0104] Determining semantic similarity of adjacent text segments based on semantic features of adjacent text segments;

[0105] Based on the semantic similarity of adjacent text segments in the multiple text segments, semantic segmentation is performed on the text to be processed to obtain the multiple text blocks.

[0106] In an embodiment of the present disclosure, the electronic device divides a text to be processed into multiple text segments and determines semantic features corresponding to each text segment; wherein the characters in each text segment are continuous. In some embodiments, the electronic device may first segment the text to be processed by the number of characters or line breaks, dividing the text to be processed into multiple text segments, wherein the characters in each text segment are continuous on the text to be processed, and characters in adjacent text segments may partially overlap. For example, when the text to be processed is "Today I went to the supermarket to buy things. There were a lot of people in the supermarket, and I waited for a long time before I bought what I wanted.", the text to be processed can be divided into multiple non-overlapping text segments. For example, based on punctuation marks, it can be divided into three text segments: "Today I went to the supermarket to buy things.", "There were a lot of people in the supermarket," and "I waited for a long time before I bought what I wanted." The text to be processed can also be divided into multiple text segments in which the text in adjacent text segments overlaps. For example, based on the number of characters, it can be divided into multiple text segments: "Today I went to the supermarket," "Go to the supermarket to buy things," "Buy things. In the supermarket," "There were a lot of people in the supermarket," "There were a lot of people, and I waited," ..., "What I want." The last three characters in the previous text segment are the same as the last three characters in the adjacent next text segment.

[0107] In an embodiment of the present disclosure, after the electronic device divides the text to be processed into multiple text segments, it also determines the semantic features corresponding to each text segment. In some embodiments, the electronic device can use an embedding model to generate a vector corresponding to each text segment, which can represent the semantic features of each text segment.

[0108] In other embodiments, the electronic device may generate semantic features corresponding to each text segment based on the bag-of-words model, such as determining the number of times each phrase in the text segment appears in the text segment for each text segment, and analyzing the theme or keywords of the text segment based on the number of times each phrase appears in the text segment, thereby determining the semantic features of the text segment.

[0109] In an embodiment of the present disclosure, an electronic device determines the semantic similarity of adjacent text segments based on the semantic features of the adjacent text segments. The electronic device may represent the determined semantic features corresponding to each text segment as a vector and calculate the similarity of the vectors corresponding to the adjacent text segments to determine the semantic similarity of the adjacent text segments. The method for calculating the similarity of the vectors corresponding to the adjacent text segments may be a cosine similarity method, a Pearson correlation coefficient method, a Euclidean distance method, or the like, which is not limited in the embodiment of the present disclosure.

[0110] In an embodiment of the present disclosure, the electronic device may also determine the semantic similarity of adjacent text segments based on the semantic features of the adjacent text segments using a preset similarity model, wherein the preset similarity model may be a model trained based on sample data.

[0111] In the embodiment of the present disclosure, the electronic device performs semantic segmentation on the text to be processed based on the semantic similarity of each adjacent text segment in a plurality of text segments to obtain the plurality of text blocks. The electronic device may preset a similarity threshold value and compare the semantic similarity of each adjacent text segment in a plurality of text segments with the relationship of the preset similarity threshold value. When the semantic similarity of the adjacent text segments is greater than the preset similarity threshold value, the adjacent text segments are merged, and based on the relationship between the similarity between the merged text segment and the adjacent text segment and the preset similarity threshold value, it is determined whether to merge again, until the similarity between the merged text segment and the adjacent text segment is less than or equal to the preset similarity threshold value, and the currently merged text segment is regarded as a text block. According to the above steps, a plurality of text blocks are obtained.

[0112] In an embodiment of the present disclosure, the electronic device divides the text to be processed into multiple text segments, and determines the semantic features corresponding to each text segment, determines the semantic similarity of adjacent text segments based on the semantic features of adjacent text segments, and performs semantic segmentation on the text to be processed based on the semantic similarity of each adjacent text segment in the multiple text segments to obtain multiple text blocks, which can improve the accuracy of the determined text blocks and is highly intelligent.

[0113] In some embodiments, dividing the to-be-processed text into a plurality of text segments includes:

[0114] The text to be processed is segmented based on a preset character length threshold to obtain a plurality of text units; wherein the number of characters included in each text unit is less than the preset character length threshold, and the last character included in the previous text unit and the first character included in the next text unit in adjacent text units are consecutively positioned in the text to be processed;

[0115] Adjacent text units in the multiple text units are merged in sequence to obtain multiple text segments; wherein the same text units exist in adjacent text segments.

[0116] In an embodiment of the present disclosure, the electronic device segments the text to be processed based on a preset character length threshold as a guide to obtain a plurality of text units. In some embodiments, the electronic device may pre-set a preset character length threshold, such as setting the preset character length threshold to 20, and segment the text to be processed based on the preset character length threshold as a guide to obtain a plurality of text units. For example, the electronic device may segment the text to be processed starting from the first character of the text to be processed with a character length less than the preset character length threshold. For example, when the preset character length threshold is 20, the text to be processed may be segmented into a plurality of text units starting from the first character of the text to be processed, wherein the character length of each text unit is 19, and the positions of the last character included in the previous text unit and the first character included in the next text unit in adjacent text units are continuous in the text to be processed.

[0117] In some other embodiments, the text to be processed is segmented based on a preset character length threshold to obtain multiple text units, including:

[0118] Segmenting the text to be processed into paragraphs to obtain multiple paragraphs;

[0119] Segmenting the paragraphs whose number of characters is greater than or equal to the preset character length threshold among the multiple paragraphs to obtain multiple sentences;

[0120] Segmenting the sentences corresponding to the paragraph, wherein the number of characters included in the sentences is greater than or equal to the preset character length threshold, to obtain a plurality of words; wherein the number of characters included in each word is less than the preset character length threshold;

[0121] The paragraphs, sentences and words whose number of characters is less than the preset character length threshold are determined as the multiple text units.

[0122] In the embodiment of the present disclosure, the electronic device first divides the text to be processed into paragraphs to obtain multiple paragraphs. The electronic device may divide the text to be processed into paragraphs based on line breaks to obtain multiple paragraphs.

[0123] In an embodiment of the present disclosure, the electronic device performs sentence segmentation on paragraphs in which the number of characters included in a plurality of paragraphs is greater than or equal to a preset character length threshold, thereby obtaining a plurality of sentences, wherein the preset character length threshold is a set value. The electronic device determines the relationship between the number of characters included in each paragraph and the preset character length threshold, and performs sentence segmentation on paragraphs in which the number of characters included in the paragraphs is greater than or equal to the preset character length threshold, thereby obtaining a plurality of sentences. The electronic device may perform sentence segmentation on paragraphs in which the number of characters included in the paragraphs is greater than or equal to the preset character length threshold based on punctuation marks, such as performing sentence segmentation based on “.”, thereby obtaining a plurality of sentences.

[0124] In an embodiment of the present disclosure, the electronic device segments the sentences corresponding to the paragraph, wherein the number of characters included in the sentences is greater than or equal to a preset character length threshold, to obtain multiple words; wherein the number of characters included in each word is less than the preset character length threshold. In some embodiments, the electronic device may directly segment the sentences including the number of characters greater than or equal to the preset character length threshold. For example, when the preset character length threshold is 6, the electronic device may segment the sentences including the number of characters greater than or equal to the preset character length threshold starting from the beginning of the sentence and dividing the sentences into multiple words, wherein the number of characters included in each word is the same.

[0125] In other embodiments, the electronic device may segment a sentence based on the part of speech of each word in the sentence where the number of characters included in the sentence is greater than or equal to a preset character length threshold, obtain multiple initial words, determine the relationship between each initial word and the preset character length threshold, and segment the initial words where the number of characters included in the initial words is greater than or equal to the preset character length threshold, to obtain multiple words where the number of characters is less than the preset character length threshold.

[0126] In an embodiment of the present disclosure, the electronic device determines paragraphs, sentences, and words whose number of characters is less than a preset character length threshold as multiple text units.

[0127] In the disclosed embodiment, the electronic device segments the text to be processed in the order of paragraphs, sentences, and words, which can improve the coherence of the text units and retain the original structure of the text to be processed to a great extent.

[0128] In an embodiment of the present disclosure, an electronic device sequentially merges adjacent text units from a plurality of text units to obtain a plurality of text segments, wherein identical text units exist within adjacent text segments. Taking the example of a text to be processed including five text units, which are identified as A, B, C, D, and E, the electronic device can pre-set the number of text units to be merged each time and the number of identical text units within adjacent text segments. For example, if the number of text units to be merged each time is set to 3 and the number of identical text units within adjacent text segments is set to 2, the text unit can be divided into three text segments: "A, B, C", "B, C, D", and "C, D, E".

[0129] In an embodiment of the present disclosure, an electronic device segments a text to be processed based on a preset character length threshold as a guide, obtains a plurality of text units whose number of characters is less than the preset character length threshold, and sequentially merges adjacent text units in the plurality of text units to obtain a plurality of text fragments; wherein, the same text units exist in adjacent text fragments. On the one hand, the length of each text unit is less than the preset character length threshold, which can prevent the text unit from being too long and facilitate subsequent processing; on the other hand, the existence of the same text units between adjacent text fragments can make it possible to combine the association or contextual relationship between the text fragments when processing the text fragments subsequently, which is highly intelligent.

[0130] In some embodiments, sequentially merging adjacent text units in the plurality of text units to obtain a plurality of text segments includes:

[0131] For a first text unit among the plurality of text units, merging the first text unit with a first preset number of adjacent text units to obtain a text segment corresponding to the first text unit;

[0132] For each text unit after the first text unit, selecting a second preset number of adjacent text units before and a first preset number of adjacent text units after the first text unit and merging them with the text unit to obtain a text segment corresponding to the text unit;

[0133] The text segments corresponding to the text units form the multiple text segments.

[0134] In an embodiment of the present disclosure, the electronic device merges the first text unit among multiple text units with the first preset number of adjacent text units to obtain a text fragment corresponding to the first text unit, wherein the first preset number is a set value, such as the first preset number can be set to 1. When the first preset number is 1, the text fragment corresponding to the first text unit includes the first text unit and the next text unit adjacent to the first text unit.

[0135] In an embodiment of the present disclosure, the electronic device selects the adjacent first second preset number of text units and the adjacent last first preset number of text units and merges them with the text unit for each text unit after the first text unit to obtain a text fragment corresponding to the text unit, wherein the second preset number also has no set value. The second preset number may be the same as or different from the first preset number, and the embodiment of the present disclosure does not impose any restrictions on this.

[0136] In the embodiment of the present disclosure, the first preset number and the second preset number are both 1. For each text unit after the first text unit, for example, the electronic device stores the second text unit in a text segment corresponding to the second text unit, including the first text unit, the second text unit, and the third text unit. The text segments corresponding to each text unit form a plurality of text segments.

[0137] In an embodiment of the present disclosure, for the first text unit, the first text unit is merged with the adjacent first preset number of text units to obtain a text fragment corresponding to the first text unit. For each text unit after the first text unit, the adjacent first second preset number of text units and the adjacent first preset number of text units are merged to obtain a text fragment corresponding to the text unit. This enables the text fragment corresponding to each text unit to include the text unit and the context of the text unit, and enables the fusion of context information to improve the accuracy of the semantic features of the generated text fragment.

[0138] In some embodiments, the semantic segmentation of the to-be-processed text based on the semantic similarity of adjacent text segments in the multiple text segments to obtain the multiple text blocks includes:

[0139] For a non-first text segment, in response to a semantic similarity between the text segment and an adjacent previous text segment being greater than a preset similarity threshold, dividing text units that are different from those in the text segment and the adjacent previous text segment into the same text block as text units included in the adjacent previous text segment;

[0140] In response to the semantic similarity between the text segment and the adjacent previous text segment being less than or equal to the preset similarity threshold, the text units that are different from those in the text segment and the adjacent previous text segment and the text units included in the adjacent previous text segment are divided into different text blocks.

[0141] In an embodiment of the present disclosure, for a non-first text segment, the electronic device, in response to the semantic similarity between the text segment and the adjacent previous text segment being greater than a preset similarity threshold, divides the text segment and the text units that are different from the adjacent previous text segment, and the text units included in the adjacent previous text segment, into the same text block, wherein the preset similarity threshold is a set value. For example, the text to be processed includes 5 text units, and the 5 text units are identified as A, B, C, D, and E. The text segments after the text units are merged are "A, B", "A, B, C", "B, C, D", and "C, D, E". When the semantic similarity between the text segments "A, B, C" and the text segment "A, B" is greater than the preset similarity threshold, C and A, B are divided into the same text block. When the semantic similarity between the text segments "B, C, D" and the text segments "A, B, C" is less than or equal to the preset similarity threshold, the segmentation is performed from D, and A, B, C are divided into the same text block, and D is divided into the next text block of the text blocks A, B, and C.

[0142] Figure 2 : This is a schematic diagram of a method for determining a text block according to an exemplary embodiment, wherein L21, L22, L23, L24, and L25 are text units of the text to be processed, L26 is a text segment corresponding to the text unit L21, L27 is a text segment corresponding to the text unit L22, L28 is a text segment corresponding to the text unit L23, and L29 is a text segment corresponding to the text unit L24. Figure 2 It can be seen that the first preset number and the second preset number are both 1, and the distances between text segments L26 and L27, text segments L27 and L28, and text segments L28 and L29 are determined respectively. The distance represents the semantic similarity between text segments. The larger the distance, the smaller the semantic similarity. The preset distance threshold is set to 0.1. Figure 2 It can be seen that the distance between text segment L26 and text segment L27 is 0.07, which is less than 0.1, so the text unit L23 that is different from the text segment L26 and the text units L21 and L22 included in the text segment L26 are divided into the same text block; the distance between text segment L27 and text segment L28 is 0.010, which is less than 0.1, so the text unit L24 that is different from the text segment L28 and the text segment L27 is divided into the same text block. The text units L21, L22, and L23 included in this fragment L27 are divided into the same text block; the distance between text fragment L29 and text fragment L28 is 0.2, which is greater than 0.1, so the text unit L25 that is different from the text fragment L29 and the text fragment L28, and the text units L22, L23, and L24 included in the text fragment L28 are divided into different text blocks, and the divided text block 1 includes text units L21, L22, L23, and L24.

[0143] It should be noted that the electronic device can use the embedding model to determine the vector of each text segment. The vector of each text segment represents the semantic features of each text segment. The vector of the text segment corresponding to each text unit can be determined based on formula (1), and the distance between the text segment and the adjacent previous text segment can be determined based on formula (2).

[0144] emed[i]=Embedding(seq[i-1]+seq[i]+seq[i+1]) (1)

[0145] Among them, emed[i] is the vector of the text fragment corresponding to the i-th text unit, and seq[i] is the i-th text unit.

[0146]

[0147] Among them, distance[i][j] is the distance between the i-th text segment and the j-th text segment, Represents the semantic similarity between the i-th text segment and the j-th text segment.

[0148] In the embodiment of the present disclosure, by comparing the semantic similarity between a text segment and an adjacent previous text segment, the accuracy of text block division can be improved. When the similarity is greater than a preset threshold, different text units and text units in the previous text segment are divided into the same text block, which helps to maintain the coherence and integrity of the text content.

[0149] In some embodiments, generating a text block summary for each text block includes:

[0150] Generate an initial summary for each text block, and generate a summary topic for each text block based on the initial summary of each text block; wherein one text block corresponds to one initial summary, and one initial summary corresponds to one summary topic;

[0151] Guided by the summary topic of each text block, a preset large language model is used to generate a text block summary for each text block.

[0152] In an embodiment of the present disclosure, the electronic device can generate an initial summary of each text block based on a model such as a BERT for summarization model, whose model size is smaller than the model size of a preset language model, and generate a summary topic of each text block based on the initial summary of each text block; wherein, one text block corresponds to one initial summary, and one initial summary corresponds to one summary topic.

[0153] In the embodiment of the present disclosure, in order to further reduce the memory usage of the preset large language model in the electronic device, the preset large language model can also be quantized, such as FP16 quantization, INT8 quantization, INT4 quantization, etc.

[0154] In the disclosed embodiment, the electronic device uses the summary theme of each text block as a guide and utilizes a preset large language model to generate a text block summary of each text block, thereby improving the accuracy of the generated text block summary of each text block.

[0155] Figure 3 is a schematic diagram showing a method for segmenting text to be processed according to an exemplary embodiment. Figure 3 As shown, L31, L32, L33, L34, and L35 are text blocks of the text to be processed respectively, L36 is a preset distance threshold, representing a preset similarity threshold, and the horizontal axis represents the distance between each text segment other than the first text segment and the adjacent previous text segment. Taking the distance between the second text segment and the first text segment as an example, L37 represents that the distance between the second text segment and the first text segment is 0.041, which is less than the preset distance threshold, representing that the semantic similarity between the second text segment and the first text segment is greater than the preset similarity threshold, and the text units that are different from the second text segment and the first text segment and the text units included in the first text segment are divided into the same text block. After comparing the distance between each text segment and the adjacent previous text segment in turn, the text to be processed is divided into L31, L32, L33, L34, and L35 text blocks.

[0156] Figure 4 FIG. 4 is a block diagram of a text processing device 400 according to an exemplary embodiment. Figure 4 As shown, the device mainly includes:

[0157] An acquisition module 401 is configured to acquire a text to be processed;

[0158] Semantic segmentation module 402 is configured to perform semantic segmentation on the text to be processed to obtain a plurality of text blocks; wherein adjacent text blocks have different semantics;

[0159] The first generating module 403 is configured to generate a text block summary for each text block; wherein one text block corresponds to one text block summary;

[0160] The second generating module 404 is configured to generate a summary of the text to be processed based on the text block summary of each text block.

[0161] In some embodiments, the semantic segmentation module 402 is further configured to divide the text to be processed into multiple text segments and determine the semantic features corresponding to each text segment; wherein the characters in each text segment are continuous; based on the semantic features of adjacent text segments, the semantic similarity of adjacent text segments is determined; based on the semantic similarity of each adjacent text segment in the multiple text segments, the text to be processed is semantically segmented to obtain the multiple text blocks.

[0162] In some embodiments, the semantic segmentation module 402 is further configured to segment the text to be processed based on a preset character length threshold to obtain multiple text units; wherein the number of characters included in each text unit is less than the preset character length threshold, and the last character included in the previous text unit and the first character included in the next text unit in adjacent text units are continuous in the position of the text to be processed; adjacent text units in the multiple text units are merged in sequence to obtain multiple text fragments; wherein the same text units exist in adjacent text fragments.

[0163] In some embodiments, the semantic segmentation module 402 is further configured to perform paragraph segmentation on the text to be processed to obtain multiple paragraphs; perform sentence segmentation on the paragraphs among the multiple paragraphs whose number of characters is greater than or equal to the preset character length threshold to obtain multiple sentences; perform sentence segmentation on the multiple sentences corresponding to the paragraphs whose number of characters is greater than or equal to the preset character length threshold to obtain multiple words; wherein the number of characters included in each word is less than the preset character length threshold; and determine the paragraphs, sentences and words whose number of characters is less than the preset character length threshold as the multiple text units.

[0164] In some embodiments, the semantic segmentation module 402 is further configured to, for the first text unit among the multiple text units, merge the first text unit with the adjacent first preset number of text units to obtain a text fragment corresponding to the first text unit; for each text unit after the first text unit, select the adjacent first second preset number of text units and the adjacent first preset number of text units and merge them with the text unit to obtain a text fragment corresponding to the text unit; wherein, the text fragments corresponding to each text unit form the multiple text fragments.

[0165] In some embodiments, the semantic segmentation module 402 is further configured to, for a non-first text segment, in response to the semantic similarity between the text segment and the adjacent previous text segment being greater than a preset similarity threshold, divide the text units that are different from the text segment and the adjacent previous text segment, and the text units included in the adjacent previous text segment into the same text block; in response to the semantic similarity between the text segment and the adjacent previous text segment being less than or equal to the preset similarity threshold, divide the text units that are different from the text segment and the adjacent previous text segment, and the text units included in the adjacent previous text segment into different text blocks.

[0166] In some embodiments, the first generation module 403 is further configured to generate an initial summary for each text block, and generate a summary topic for each text block based on the initial summary of each text block; wherein, one text block corresponds to one initial summary, and one initial summary corresponds to one summary topic; guided by the summary topic of each text block, a preset large language model is used to generate a text block summary for each text block.

[0167] In some embodiments, the second generating module 404 is further configured to integrate the text block summaries of each text block using a preset large language model to generate a summary of the text to be processed.

[0168] In some embodiments, the first generating module 403 is further configured to generate a text block summary of each text block using an edge device.

[0169] In some embodiments, the first generating module 403 is further configured to utilize multiple edge devices to generate text block summaries for each text block in parallel.

[0170] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0171] Figure 5 This is a block diagram illustrating the structure of an electronic device 500 according to an exemplary embodiment. For example, electronic device 500 may be a terminal device such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a tablet device, a personal digital assistant, or the like; it may also be an edge device such as a gateway, a switch, or a router; or it may be a cloud device such as a server.

[0172] Reference Figure 5 , electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0173] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with at least one of display, phone calls, data communications, camera operation, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 502 may include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0174] The memory 504 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include at least one of the following: instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, and videos. The memory 504 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0175] The power supply component 506 provides power to various components of the electronic device 500. The power supply component 506 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0176] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0177] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0178] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as a keyboard, click wheel, and buttons. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0179] The sensor assembly 514 includes one or more sensors for providing various aspects of the status assessment of the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500. The sensor assembly 514 can also detect changes in the position of the electronic device 500 or a component thereof, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and changes in the temperature of the electronic device 500. The sensor assembly 514 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 can also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 can also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, and a temperature sensor.

[0180] The communication component 516 is configured to facilitate communication between the electronic device 500 and other devices in a wired or wireless manner. The electronic device 500 can access a wireless network based on a communication standard, such as Wi-Fi, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0181] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0182] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as a memory 504 including executable instructions or a computer program. The instructions or computer program can be executed by the processor 520 of the electronic device 500 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0183] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the above-mentioned text processing methods of the embodiments of the present disclosure. For example, the method includes:

[0184] Get the text to be processed;

[0185] Performing semantic segmentation on the text to be processed to obtain a plurality of text blocks, wherein adjacent text blocks have different semantics;

[0186] Generate a text block summary for each text block; wherein, one text block corresponds to one text block summary;

[0187] Based on the text block summaries of the text blocks, a summary of the text to be processed is generated.

[0188] The present disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the above-described text processing methods of the present disclosure.

[0189] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0190] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A text processing method, characterized in that: The method comprises: Get the text to be processed; Performing semantic segmentation on the text to be processed to obtain a plurality of text blocks, wherein adjacent text blocks have different semantics; Generate a text block summary for each text block; wherein, one text block corresponds to one text block summary; Based on the text block summaries of the text blocks, a summary of the text to be processed is generated.

2. The method according to claim 1, characterized in that The semantic segmentation of the text to be processed is performed to obtain multiple text blocks, including: Dividing the text to be processed into a plurality of text segments and determining semantic features corresponding to each text segment; wherein the characters in each text segment are continuous; Determining semantic similarity of adjacent text segments based on semantic features of adjacent text segments; Based on the semantic similarity of adjacent text segments in the multiple text segments, semantic segmentation is performed on the text to be processed to obtain the multiple text blocks.

3. The method according to claim 2, characterized in that The step of dividing the text to be processed into a plurality of text segments includes: The text to be processed is segmented based on a preset character length threshold to obtain a plurality of text units; wherein the number of characters included in each text unit is less than the preset character length threshold, and the last character included in the previous text unit and the first character included in the next text unit in adjacent text units are consecutively positioned in the text to be processed; Adjacent text units in the multiple text units are merged in sequence to obtain multiple text segments; wherein the same text units exist in adjacent text segments.

4. The method according to claim 3, characterized in that The text to be processed is segmented based on a preset character length threshold to obtain multiple text units, including: Segmenting the text to be processed into paragraphs to obtain multiple paragraphs; Segmenting the paragraphs whose number of characters is greater than or equal to the preset character length threshold among the multiple paragraphs to obtain multiple sentences; Segmenting the sentences corresponding to the paragraph, wherein the number of characters included in the sentences is greater than or equal to the preset character length threshold, to obtain a plurality of words; wherein the number of characters included in each word is less than the preset character length threshold; The paragraphs, sentences and words whose number of characters is less than the preset character length threshold are determined as the multiple text units.

5. The method according to claim 3 or 4, characterized in that The step of sequentially merging adjacent text units in the plurality of text units to obtain a plurality of text segments includes: For a first text unit among the plurality of text units, merging the first text unit with a first preset number of adjacent text units to obtain a text segment corresponding to the first text unit; For each text unit after the first text unit, selecting a second preset number of adjacent text units before and a first preset number of adjacent text units after the first text unit and merging them with the text unit to obtain a text segment corresponding to the text unit; The text segments corresponding to the text units form the multiple text segments.

6. The method according to claim 3, characterized in that The step of performing semantic segmentation on the text to be processed based on the semantic similarity of adjacent text segments in the multiple text segments to obtain the multiple text blocks includes: For a non-first text segment, in response to a semantic similarity between the text segment and an adjacent previous text segment being greater than a preset similarity threshold, dividing text units that are different from those in the text segment and the adjacent previous text segment into the same text block as text units included in the adjacent previous text segment; In response to the semantic similarity between the text segment and the adjacent previous text segment being less than or equal to the preset similarity threshold, the text units that are different from those in the text segment and the adjacent previous text segment and the text units included in the adjacent previous text segment are divided into different text blocks.

7. The method according to claim 1, characterized in that Generating a text block summary for each text block includes: Generate an initial summary for each text block, and generate a summary topic for each text block based on the initial summary of each text block; wherein one text block corresponds to one initial summary, and one initial summary corresponds to one summary topic; Guided by the summary topic of each text block, a preset large language model is used to generate a text block summary for each text block.

8. The method according to claim 1, characterized in that The step of generating a summary of the text to be processed based on the text block summary of each text block includes: The text block summaries of the text blocks are integrated using a preset large language model to generate a summary of the text to be processed.

9. The method according to claim 1, characterized in that Generating a text block summary for each text block includes: Generate a text block summary for each text block using edge devices.

10. The method according to claim 9, characterized in that The generating of a text block summary for each text block by using an edge device includes: Multiple edge devices are used to generate text block summaries for each text block in parallel.

11. A text processing device, characterized in that: The device comprises: An acquisition module configured to acquire text to be processed; a semantic segmentation module configured to perform semantic segmentation on the text to be processed to obtain a plurality of text blocks; wherein adjacent text blocks have different semantics; A first generating module is configured to generate a text block summary for each text block; wherein one text block corresponds to one text block summary; The second generating module is configured to generate a summary of the to-be-processed text based on the text block summary of each text block.

12. An electronic device, characterized in that: include: processor; memory for storing computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the text processing method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing a computer program or instruction, characterized in that: When the computer program or instructions in the storage medium are executed by a processor, the steps of the text processing method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the text processing method according to any one of claims 1 to 10 are implemented.

Images