Text segmentation method, question answering method and related devices and equipment

Abstract

The invention discloses a text segmentation method, a question answering method and related devices and equipments.The text segmentation method comprises the steps that the confusion degree of each sentence text in a to-be-processed text is obtained; for each sentence text, obtaining a confusion degree comparison result between the sentence text and the adjacent sentence text, and determining whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result; and segmenting the to-be-processed chapter based on each segmentation point to obtain a plurality of text blocks. According to the scheme, the segmentation adaptivity of the chapter text can be improved, and the semantic and logic coherence of the text blocks obtained through segmentation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text processing, in particular to a text segmentation method and a question answering method and related devices and equipment. BACKGROUND

[0002] With the development of artificial intelligence technology, large language models have been widely applied in many fields. However, the answering effect of large language models for strong time-sensitive problems and specific professional field problems is insufficient, therefore, how to improve the generation effect of large language models is paid more and more attention.

[0003] In the prior art, an external knowledge base is constructed to store new knowledge, a retrieval enhancement generation technology is adopted, a large language model is combined with a retriever module, and the retrieval enhancement technology is used to select a text block in the external knowledge base as additional information provided by the large language model. For long text stored in the knowledge base, the long text is usually segmented based on a fixed length to obtain a text block. However, such segmentation lacks adaptability and is prone to cause poor semantic and logical coherence of the text block, thereby affecting subsequent question answering based on the segmentation. Therefore, how to improve the segmentation adaptability of the long text and improve the semantic and logical coherence of the segmented text block has become a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is to provide a text segmentation method and a question answering method and related devices and equipment, which can improve the segmentation adaptability of the long text and improve the semantic and logical coherence of the segmented text block.

[0005] To solve the above technical problem, the first aspect of the present application provides a text segmentation method, comprising: obtaining the perplexity of each sentence text in a to-be-processed long text; for each sentence text, obtaining a comparison result of the perplexity between the sentence text and the adjacent sentence text, and determining whether to select the sentence text as a segmentation point of the to-be-processed long text based on the comparison result; and segmenting the to-be-processed long text based on each segmentation point to obtain a plurality of text blocks.

[0006] To solve the above technical problem, the second aspect of the present application provides a question answering method, comprising: obtaining a to-be-answered question and a plurality of text blocks of a target long text; wherein the plurality of text blocks are obtained by segmenting the target long text as a to-be-processed long text by the text segmentation method of the first aspect; searching the plurality of text blocks based on the to-be-answered question to obtain a target block; and generating an answering content of the to-be-answered question based on the to-be-answered question and the target block.

[0007] To solve the above technical problems, the third aspect of the present application provides a text segmentation device, comprising: an acquisition module, a comparison module and a segmentation module, the acquisition module is used to acquire the perplexity of each sentence text in a to-be-processed chapter; the comparison module is used to acquire the comparison result between the perplexity of each sentence text and the perplexity of the adjacent sentence text of the sentence text, and determine whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result; and the segmentation module is used to segment the to-be-processed chapter based on each segmentation point to obtain a plurality of text blocks.

[0008] To solve the above technical problems, the fourth aspect of the present application provides a question answering device, comprising: a processing module, a search module and a generation module, the processing module is used to acquire a to-be-answered question and a plurality of text blocks of a target chapter; wherein the plurality of text blocks are obtained by segmenting the target chapter as a to-be-processed chapter by the text segmentation method of the first aspect; the search module is used to search in the plurality of text blocks based on the to-be-answered question to obtain a target block; and the generation module is used to generate the answer content of the to-be-answered question based on the to-be-answered question and the target block.

[0009] To solve the above technical problems, the fifth aspect of the present application provides an electronic device, comprising at least a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to realize the text segmentation method of the first aspect or the question answering method of the second aspect.

[0010] To solve the above technical problems, the sixth aspect of the present application provides a computer readable storage medium, which stores program instructions capable of being executed by a processor, and the program instructions are used to realize the text segmentation method of the first aspect or the question answering method of the second aspect.

[0011] The above scheme acquires the perplexity of each sentence text in a to-be-processed chapter, acquires the comparison result between the perplexity of each sentence text and the perplexity of the adjacent sentence text of the sentence text, and determines whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result, and segments the to-be-processed chapter based on each segmentation point to obtain a plurality of text blocks. Therefore, the comparison result between the perplexity of each sentence text and the perplexity of the adjacent sentence text is used as reference information for determining the segmentation point, which can reflect the logical relationship between the sentence text and the context based on the perplexity, measure the logical coherence between each sentence text in the to-be-processed chapter, and facilitate the logical structure of the text semantic of the clear chapter, thereby improving the accuracy and flexibility of the segmentation point determination. Therefore, the segmentation adaptability of the chapter text can be improved, and the semantic and logical coherence of the segmented text block can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 FIG. 1 is a flowchart of an embodiment of the text segmentation method of the present application; Figure 2 is a flowchart of an embodiment of the problem answering method of the present application; Figure 3 is a framework diagram of an embodiment of the text segmentation device of the present application; Figure 4 is a framework diagram of an embodiment of the problem answering device of the present application; Figure 5 is a framework diagram of an embodiment of the electronic device of the present application; Figure 6 is a framework diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0014] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to provide a thorough understanding of the present application for the sake of explanation, but not for the purpose of limiting the present application.

[0015] The terms "system" and "network" are often used interchangeably herein. The term "and / or" herein is merely an associated relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the segment " / " herein generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" herein means two or more than two.

[0016] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the text segmentation method of the present application. Specifically, it can include the following steps: Step S11: Obtain the perplexity of each sentence text in the to-be-processed chapter.

[0017] In the embodiments of the present disclosure, the to-be-processed chapter refers to a language unit composed of multiple sentences or paragraphs, having complete meaning and logical structure. It is not only a simple pile of sentences, but also an organic whole formed by coherence and cohesion, which can express a complete theme, thought or communicative intention. It should be noted that the physical parameters of the to-be-processed chapter, such as the number of characters, data structure, language information, etc. are not limited in the present application.

[0018] In an implementation scenario, the perplexity of the sentence text can intuitively reflect the uncertainty or confusion degree of the model in processing the corresponding sentence text. When the model understands the text well, the perplexity is correspondingly lower, indicating that the model is more confident and accurate in processing the text, and correspondingly, when the model understands the text poorly, the perplexity is correspondingly higher. It can be understood that when the complexity of the sentence is higher or the logical coherence with the previous text is poorer, the perplexity is correspondingly higher, and when the complexity of the sentence is lower or the logical coherence with the previous text is better, the perplexity is correspondingly lower.

[0019] In an implementation scenario, the specific way of dividing the sentence text is not limited in the present application, for example, taking a period, a question mark, an exclamation mark, etc. as the end mark of the sentence.

[0020] In an implementation scenario, each token in the sentence text is sequentially taken as a target token, and the text composed of tokens before the target token in the to-be-processed chapter is taken as a reference text, the perplexity of the target token is obtained when the reference text is taken as the prior condition, and the perplexities of each target token in the sentence text are fused to obtain the perplexity of the sentence text. It can be understood that the token (Token) is regarded as the smallest unit of information of language, which divides the text data into smaller units, which can be words, characters or subwords. In different application scenarios, the definition of token may be different, for example, the sentence “I like artificial intelligence” will be divided into three tokens: “I”, “like”, “artificial intelligence”, and the sentence “GPT” will be divided into “G”, “P”, “T” character tokens, and the specific token segmentation method is not limited in the present application.

[0021] In a specific implementation scenario, the to-be-processed chapter is first divided into a set of single sentences For each sentence , the perplexity based on the previous text as the prior condition is calculated. The calculation formula of the perplexity is as follows:

[0022] In formula (1), represents the total number of tokens in the sentence , represents the kth token in , represents all tokens of the sentence text before in the to-be-processed chapter.

[0023] In a specific implementation scenario, the goal of a large language model is to learn the probability distribution of text, and cross-entropy provides an effective way to quantify the difference between the distribution learned by the model and the empirical distribution of sample text, which can evaluate the deviation of the model's understanding and prediction ability of the text from the actual situation. A lower cross-entropy means that the model's prediction is closer to the true text distribution, so it can more accurately capture the logical and semantic relationships in the text. Therefore, the perplexity of the large language model is measured using the cross-entropy function. Since the generation of the model's probability distribution has a lag, the size of the input data of the model is first defined as: The size of the output data obtained after the input data passes through the model is: wherein is the length of the input sentence, is the length of the word piece, , the value of the last position is deleted, and it is marked as , the size is , which limits the output that cannot include the last position. For the input data of the model, the value of the first position is deleted, and it is marked as , the size is , which limits the prediction of the first word. The cross-entropy loss of each word piece is calculated as Finally, according to the length of each sentence segmentation, the cross-entropy loss is segmented, and the average loss is taken as the perplexity of the sentence.

[0024] In a specific implementation scenario, when facing long text that exceeds the processing capacity of the model or the device, a KV cache mechanism is introduced. Specifically, the to-be-processed chapter is divided into multiple subsequences according to the word piece. During the perplexity calculation process, the GPU memory usage is continuously monitored. Once it is found that it is about to exceed the maximum context length of the server configuration or the model, the KV cache management strategy is started, and the KV pairs of the previous part of the text are appropriately removed. For example, when processing a long novel, as the calculation progresses, the memory pressure gradually increases. At this time, the system will gradually clean up the KV information of the part of the text processed earlier according to the preset rules, ensure the smooth progress of the calculation under the premise of ensuring a certain context coherence, and realize the effective processing of long text.

[0025] In one specific implementation scenario, the cache data of the previous word unit of the processing target word unit is obtained, and the cache data at least includes the key matrix and the value matrix. The cache data is taken as the reference information of the reference text. Based on the reference information, the perplexity of the target word unit is obtained. For example, the maximum character length is set as m. The to-be-processed chapter is divided into multiple subsequences according to the maximum character length. The perplexity of each subsequence is obtained by inputting the subsequence into the model. At the same time, the KV pair of the subsequence is obtained. When the next subsequence is input, the KV pair of the previous subsequence is input into the model. In the above scheme, the maximum length and the KV cache are combined to enable the pre-training model to understand the content of the text as much as possible.

[0026] In another specific implementation scenario, the perplexity of the last word unit in the sentence text is calculated. The specific perplexity calculation method can be referred to in the foregoing detailed embodiments. The perplexity of the last word unit in the sentence text is taken as the perplexity of the corresponding sentence text to reduce the data processing pressure.

[0027] Step S12: For each sentence text, the comparison result between the sentence text and the adjacent sentence text in terms of perplexity is obtained, and based on the comparison result, it is determined whether to select the sentence text as a segmentation point of the to-be-processed chapter.

[0028] In one implementation scenario, after obtaining the perplexity of each sentence text in the to-be-processed chapter, a sequence of perplexities arranged in sequence can be obtained. Based on the distribution characteristics of the perplexity values contained in the sequence, the comparison result between the sentence text and the adjacent sentence text in terms of perplexity is obtained.

[0029] In one implementation scenario, for any sentence text, the adjacent sentence texts thereof include a previous text and a next text, and the previous text is the previous sentence text in the to-be-processed chapter, and the next text is the next sentence text in the to-be-processed chapter. It is detected whether the comparison result satisfies a first condition. The first condition includes that the perplexity of the previous text and the perplexity of the next text are both higher than the perplexity of the sentence text, and the difference between the perplexity of at least one of the previous text and the next text and the perplexity of the sentence text is not less than a first threshold. In response to the comparison result representing that the first condition is satisfied, the sentence text is selected as the segmentation point of the to-be-processed chapter. In the above scheme, the perplexity reflects the logical relationship between the sentence text and the context, and measures the logical coherence of the sentence text in the to-be-processed chapter. This is conducive to obtaining the logical structure of the text semantic of the chapter, thereby improving the accuracy and flexibility of the segmentation point determination.

[0030] In another implementation scenario, it is detected whether the comparison result satisfies a second condition. The second condition includes that the perplexity of the previous text is higher than the perplexity of the sentence text, the perplexity of the next text is equal to the perplexity of the sentence text, and the difference between the perplexity of the previous text and the perplexity of the sentence text is not less than a second threshold. In response to the comparison result representing that the second condition is satisfied, the sentence text is selected as the segmentation point of the to-be-processed chapter.

[0031] In a specific implementation scenario, the first threshold value and the second threshold value can be the same or different, and the specific values of the first threshold value and the second threshold value are not limited in the present application. It should be noted that the manner of obtaining the first threshold value and the second threshold value is not limited in the present application, for example, setting a fixed value, dynamically setting based on a network model, etc.

[0032] In an implementation scenario, the order of obtaining the comparison results of the perplexity between the sentence text and the adjacent sentence texts is not limited in the present application, for example, first obtaining the comparison result of the perplexity between the sentence text and the previous text, then obtaining the comparison result of the perplexity between the sentence text and the next text, or first obtaining the comparison result of the perplexity between the sentence text and the next text, then obtaining the comparison result of the perplexity between the sentence text and the previous text, or alternatively, obtaining the comparison results of the perplexity between each sentence text and the adjacent sentence texts can be processed in parallel.

[0033] In an implementation scenario, when it is determined to select the sentence text as the segmentation point of the to-be-processed chapter, specifically, the start endpoint of the sentence text is taken as the segmentation point, that is, the selected sentence text is taken as the initial sentence text of the next text block, for example, the to-be-processed chapter contains a text of 5 sentences If is determined as the segmentation point, two text blocks are obtained: and , or the end endpoint of the sentence text is taken as the segmentation point, that is, the selected sentence text is taken as the ending sentence text of the previous text block, and still taking the foregoing embodiment as an example, two text blocks are obtained: and .

[0034] In a specific implementation scenario, obtaining the comparison result and determining the segmentation point can be processed in parallel, specifically, the to-be-processed chapter contains a text of 5 sentences , the comparison results of the perplexity between and , are obtained, and then it is determined that and , do not have segmentation points, and when the comparison result of the perplexity between and the adjacent sentence text is obtained, only the comparison result of the perplexity between and is obtained to reduce the data processing pressure. It should be noted that the foregoing implementation is only one possible embodiment, and the specific manner of obtaining the comparison result and determining the segmentation point is not limited.

[0035] Step S13: splitting the to-be-processed chapter based on each split point to obtain a plurality of text blocks.

[0036] In one implementation scenario, based on each split point, splitting the to-be-processed chapter, taking the comparison result of the perplexity between each sentence text and the adjacent sentence text thereof as the reference information for determining the split point, the perplexity can reflect the logical relationship between the sentence text and the context, measure the logical coherence between the sentence texts in the to-be-processed chapter, and be beneficial to the logical structure of the text semantic of the clear chapter, thereby improving the accuracy and flexibility of the split point determination. Therefore, the split adaptability of the chapter text can be improved, and the semantic and logical coherence of the text block obtained by splitting can be improved.

[0037] In one implementation scenario, before determining whether to select the sentence text as the to-be-processed chapter based on the comparison result, and before splitting the to-be-processed chapter based on each split point to obtain a plurality of text blocks, pre-splitting the to-be-processed chapter based on each split point to obtain a plurality of candidate blocks, generating a new split point based on the number of characters of each candidate block, and splitting the to-be-processed chapter based on each new split point to obtain a plurality of text blocks.

[0038] In one specific implementation scenario, the number of characters of each candidate block is sequentially accumulated to obtain the number of characters accumulated after each accumulation, and in response to the number of characters of a target round being not greater than a preset threshold and the number of characters of a round after the target round being greater than the preset threshold, the candidate block corresponding to the target round is selected as a new split point in the to-be-processed chapter, and the step of sequentially accumulating the number of characters of each candidate block to obtain the number of characters accumulated after each accumulation and the subsequent steps are returned to be executed based on the candidate block of the round after the target round, until the last candidate block in the to-be-processed chapter is processed, and the last candidate block in the to-be-processed chapter is selected as a new split point in the to-be-processed chapter. For example, the preset threshold is set to L, the to-be-processed chapter is pre-split based on the split point to obtain candidate blocks If And Then , , The above scheme can flexibly select the preset threshold as a suitable block granularity according to different application scenarios and chapter types. The adaptive granularity adjustment makes the text division not only ensure the integrity of the semantic and logical, but also maintain a good segmentation effect in different lengths of text.

[0039] The above scheme obtains the perplexity of each sentence text in the to-be-processed chapter, obtains a comparison result between the sentence text and the adjacent sentence text of the sentence text with respect to the perplexity, and determines whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result. The to-be-processed chapter is segmented based on each segmentation point to obtain a plurality of text blocks. Therefore, the comparison result between the sentence text and the adjacent sentence text of the sentence text with respect to the perplexity is used as reference information for determining the segmentation point. The logical relationship between the sentence text and the context can be reflected based on the perplexity, the logical coherence between the sentence texts in the to-be-processed chapter is measured, the logical structure of the text semantic of the clear chapter is beneficial, and the accuracy and flexibility of the segmentation point determination are improved. Therefore, the segmentation adaptability of the chapter text is improved, and the semantic and logical coherence of the segmented text block are improved.

[0040] Please refer to Figure 2 , Figure 2 is a flowchart of an embodiment of the problem answering method of the present application. Specifically, it can include the following steps: Step S21: Obtain a to-be-answered question and obtain a plurality of text blocks of a target chapter.

[0041] In the embodiments of the present disclosure, the plurality of text blocks are obtained by the text segmentation method described in any of the preceding embodiments based on the target chapter as a to-be-processed chapter. For brevity, the text segmentation method embodiments are not repeated here.

[0042] Step S22: Search the plurality of text blocks based on the to-be-answered question to obtain a target block.

[0043] In one implementation scenario, the to-be-answered question is converted into a target vector, and the plurality of text blocks of the target chapter are converted into a plurality of candidate vectors. The target vector is compared with the plurality of candidate vectors respectively, and at least one text block that is most matched with the to-be-answered question is found out as the target block through similarity search. It can be understood that since the plurality of text blocks of the target chapter are obtained based on the segmentation method with better effect in the preceding embodiments, the plurality of text blocks in the target chapter can preserve the logical coherence within each text block as much as possible, and therefore the accuracy and efficiency of searching the relevant text block for the to-be-answered question are improved, which helps to improve the quality of the answer content generation.

[0044] Step S23: Generate an answer content of the to-be-answered question based on the to-be-answered question and the target block.

[0045] In the embodiments of the present disclosure, the specific generation method of the answer content is not limited in the present application, and can be understood in detail by referring to the large language model and other technologies. For brevity, the details are not repeated here.

[0046] The above scheme obtains the question to be answered and obtains a plurality of text blocks of the target chapter, and the plurality of text blocks are obtained by cutting the target chapter as a to-be-processed chapter by the text cutting method in any of the preceding embodiments. The target block is obtained by searching the plurality of text blocks based on the question to be answered. The answer content of the question to be answered is generated based on the question to be answered and the target block. Since the plurality of text blocks of the target chapter are obtained by cutting based on the cutting method with better effect in the preceding embodiments, the plurality of text blocks in the target chapter can retain the logical coherence within each text block as much as possible. Therefore, the accuracy and efficiency of searching the relevant text block for the question to be answered can be improved, which helps to improve the quality of generating the answer content.

[0047] Please refer to Figure 3 , Figure 3 is a schematic diagram of the framework of an embodiment of the text cutting device 30. Specifically, the text cutting device 30 includes an acquisition module 31, a comparison module 32, and a cutting module 33. The acquisition module 31 is configured to acquire the perplexity of each sentence text in the to-be-processed chapter. The comparison module 32 is configured to, for each sentence text, acquire a comparison result between the sentence text and an adjacent sentence text in terms of perplexity, and determine whether to select the sentence text as a cutting point of the to-be-processed chapter based on the comparison result. The cutting module 33 is configured to cut the to-be-processed chapter based on each cutting point to obtain a plurality of text blocks.

[0048] The above scheme, the text cutting device 30 acquires the perplexity of each sentence text in the to-be-processed chapter, acquires a comparison result between the sentence text and an adjacent sentence text in terms of perplexity for each sentence text, and determines whether to select the sentence text as a cutting point of the to-be-processed chapter based on the comparison result. The to-be-processed chapter is cut based on each cutting point to obtain a plurality of text blocks. Therefore, the comparison result between the sentence text and the adjacent sentence text in terms of perplexity is used as reference information for determining the cutting point. The logical relationship between the sentence text and the context can be reflected based on the perplexity, the logical coherence between the sentence texts in the to-be-processed chapter is measured, and the logical structure of the text semantic of the clear chapter is facilitated. Therefore, the accuracy and flexibility of determining the cutting point are improved. The cutting adaptability of the chapter text is improved, and the semantic and logical coherence of the text block obtained by cutting are improved.

[0049] In some disclosed embodiments, for any sentence text, its adjacent sentence texts include a preceding text and a following text, and the preceding text is a preceding sentence text in the to-be-processed chapter and the following text is a following sentence text in the to-be-processed chapter; the comparison module 32 further includes a first detection module (not shown) configured to detect whether the comparison result satisfies a first condition; wherein the first condition includes that the perplexity of the preceding text and the perplexity of the following text are both higher than the perplexity of the sentence text, and the difference between the perplexity of at least one of the preceding text and the following text and the perplexity of the sentence text is not less than a first threshold; the comparison module 32 further includes a first selection module (not shown) configured to select the sentence text as a segmentation point of the to-be-processed chapter in response to the comparison result representing that the first condition is satisfied.

[0050] In some disclosed embodiments, for any sentence text, its adjacent sentence texts include a preceding text and a following text, and the preceding text is a preceding sentence text in the to-be-processed chapter and the following text is a following sentence text in the to-be-processed chapter; the comparison module 32 further includes a second detection module (not shown) configured to detect whether the comparison result satisfies a second condition; wherein the second condition includes that the perplexity of the preceding text is higher than the perplexity of the sentence text, the perplexity of the following text is equal to the perplexity of the sentence text, and the difference between the perplexity of the preceding text and the perplexity of the sentence text is not less than a second threshold; the comparison module 32 further includes a second selection module (not shown) configured to select the sentence text as a segmentation point of the to-be-processed chapter in response to the comparison result representing that the second condition is satisfied.

[0051] In some disclosed embodiments, after determining whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result, and before segmenting the to-be-processed chapter based on each segmentation point to obtain a plurality of text blocks, the text segmentation device 30 further includes a pre-segmentation module (not shown) configured to pre-segment the to-be-processed chapter based on each segmentation point to obtain a plurality of candidate blocks; the text segmentation device 30 further includes a segmentation point updating module (not shown) configured to generate new segmentation points based on the number of characters of each candidate block; the segmentation module 33 further includes a segmentation sub-module (not shown) configured to segment the to-be-processed chapter based on each new segmentation point to obtain a plurality of text blocks.

[0052] In some disclosed embodiments, the segmentation point updating module (not shown) further comprises an accumulation module (not shown) configured to sequentially accumulate the character numbers of the candidate blocks to obtain accumulated character numbers after each accumulation; the segmentation point updating module (not shown) further comprises a loop module (not shown) configured to, in response to the accumulated character number of the target round being not greater than the preset threshold and the accumulated character number of the next round after the target round being greater than the preset threshold, select the candidate block corresponding to the target round as a new segmentation point in the to-be-processed chapter, and return to execute the step of sequentially accumulating the character numbers of the candidate blocks and the subsequent steps based on the candidate block of the next round after the target round until the last candidate block in the to-be-processed chapter is processed, and select the last candidate block in the to-be-processed chapter as a new segmentation point in the to-be-processed chapter.

[0053] In some disclosed embodiments, the obtaining module 31 further comprises an obtaining sub-module (not shown) configured to sequentially take each word unit in the sentence text as a target word unit, and take the text composed of the word units before the target word unit in the to-be-processed chapter as a reference text; the obtaining module 31 further comprises a perplexity calculation module (not shown) configured to obtain the perplexity of the target word unit under the prior condition of the reference text; the obtaining module 31 further comprises a fusion module (not shown) configured to fuse based on the perplexities of each target word unit in the sentence text to obtain the perplexity of the sentence text.

[0054] In some disclosed embodiments, the obtaining sub-module (not shown) further comprises a cache data obtaining module (not shown) configured to obtain the cache data of the previous word unit of the target word unit; wherein the cache data at least includes a key matrix and a value matrix; the obtaining sub-module (not shown) further comprises a reference information module (not shown) configured to take the cache data as the reference information of the reference text; the perplexity calculation module (not shown) further comprises a calculation sub-module configured to obtain the perplexity of the target word unit based on the reference information.

[0055] Please refer to Figure 4 , Figure 4 is a schematic diagram of an embodiment of the problem answering device 40. Specifically, the problem answering device 40 comprises a processing module 41, a search module 42 and a generation module 43, the processing module 41 is configured to obtain a to-be-answered question and obtain a plurality of text blocks of a target chapter; wherein the plurality of text blocks are obtained by cutting the target chapter as a to-be-processed chapter by the text cutting method of the first aspect; the search module 42 is configured to search in the plurality of text blocks based on the to-be-answered question to obtain a target block; and the generation module 43 is configured to generate an answer content of the to-be-answered question based on the to-be-answered question and the target block.

[0056] The above scheme, the question answering device 40 obtains the question to be answered, obtains a plurality of text blocks of the target chapter, and the plurality of text blocks are obtained by cutting the target chapter as a to-be-processed chapter by the text cutting method in any of the preceding embodiments. Based on the question to be answered, the target block is searched in the plurality of text blocks, and based on the question to be answered and the target block, the answer content of the question to be answered is generated. Since the plurality of text blocks of the target chapter are cut based on the cutting method with better effect in the preceding embodiments, the plurality of text blocks in the target chapter can preserve the logical coherence inside each text block as much as possible, thereby improving the accuracy and efficiency of searching for related text blocks for the question to be answered, which helps to improve the quality of the answer content generation.

[0057] Please refer to Figure 5 , Figure 5 is a frame schematic diagram of an embodiment of the electronic device. The electronic device 50 at least includes a memory 51 and a processor 52 coupled with each other. The memory 51 at least stores program instructions. The processor 52 is configured to execute the program instructions to implement the steps in any of the text cutting method embodiments or the steps in any of the question answering method embodiments. For details, please refer to the preceding disclosed embodiments, which will not be repeated here. It should be noted that the electronic device 50 can include but is not limited to a learning machine, an office book, a smart large screen, etc. The specific type of the electronic device 50 is not limited here.

[0058] Specifically, the processor 52 is configured to control itself and the memory 51 to implement the steps in any of the text cutting method embodiments or the steps in any of the question answering method embodiments. The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 can be an integrated circuit chip with processing capability. The processor 52 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0059] According to the scheme, the electronic device 50 obtains the perplexity of each sentence text in the to-be-processed chapter, obtains, for each sentence text, a comparison result of the perplexity between the sentence text and the adjacent sentence texts, and determines whether to select the sentence text as a split point of the to-be-processed chapter based on the comparison result, and splits the to-be-processed chapter based on each split point to obtain a plurality of text blocks. Therefore, the comparison result of the perplexity between each sentence text and the adjacent sentence texts is used as reference information for determining the split point, which can reflect the logical relationship between the sentence text and the context based on the perplexity, measure the logical coherence between the sentence texts in the to-be-processed chapter, and facilitate the logical structure of the text semantic of the clear chapter, thereby improving the accuracy and flexibility of the split point determination. Therefore, the text splitting adaptability of the chapter text can be improved, and the semantic and logical coherence of the text blocks obtained by splitting can be improved. By obtaining the to-be-answered question and a plurality of text blocks of the target chapter, and obtaining the plurality of text blocks by splitting the target chapter as a to-be-processed chapter by the text splitting method in any of the preceding embodiments, searching the to-be-answered question in the plurality of text blocks to obtain a target block, and generating the answer content of the to-be-answered question based on the to-be-answered question and the target block. Since the plurality of text blocks of the target chapter are obtained by splitting based on the splitting method in the preceding embodiment with better effect, the plurality of text blocks in the target chapter can retain the logical coherence inside each text block as much as possible, and therefore the accuracy and efficiency of searching the relevant text block for the to-be-answered question can be improved, which helps to improve the quality of the answer content generation.

[0060] Please refer to Figure 6 , Figure 6 is a schematic diagram of the framework of an embodiment of the computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program instructions 61 capable of being executed by a processor, which are used to implement the steps in any of the text splitting method embodiments or the steps in any of the question answering method embodiments.

[0061] The above scheme, the computer readable storage medium 60 obtains the perplexity of each sentence text in the to-be-processed chapter, obtains the comparison result between the sentence text and the adjacent sentence text about the perplexity for each sentence text, and determines whether to select the sentence text as a segmentation point of the to-be-processed chapter based on the comparison result, and segments the to-be-processed chapter based on each segmentation point to obtain a plurality of text blocks. Therefore, the comparison result between each sentence text and the adjacent sentence text about the perplexity is used as reference information for determining the segmentation point, which can reflect the logical relationship between the sentence text and the context based on the perplexity, measure the logical coherence between each sentence text in the to-be-processed chapter, and facilitate the logical structure of the semantic of the clear chapter text, thereby improving the accuracy and flexibility of the segmentation point determination. Therefore, the segmentation adaptability of the chapter text can be improved, and the semantic and logical coherence of the segmented text block can be improved. By obtaining the to-be-answered question and obtaining a plurality of text blocks of the target chapter, and the plurality of text blocks are segmented from the target chapter as a to-be-processed chapter by the text segmentation method described in any of the preceding embodiments, searching the to-be-answered question in the plurality of text blocks to obtain a target block, and generating the answer content of the to-be-answered question based on the to-be-answered question and the target block. Since the plurality of text blocks of the target chapter are segmented based on the segmentation method with better effect in the preceding embodiments, the plurality of text blocks in the target chapter can preserve the logical coherence inside each text block as much as possible, and therefore the precision and efficiency of searching the related text block for the to-be-answered question can be improved, thereby helping to improve the quality of the answer content generation.

[0062] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0063] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0064] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the apparatus implementation described above is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0065] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0066] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0067] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present application essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0068] If the technical scheme of the present application involves personal information, the product applying the technical scheme of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical scheme of the present application involves sensitive personal information, the product applying the technical scheme of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as a camera, a clear and prominent mark is set to inform that the personal information collection range has been entered, and the personal information will be collected. If the person voluntarily enters the collection range, it is regarded as agreeing to collect the personal information. Or, on the device for processing personal information, the personal information processing rules are informed by using obvious marks / information, and the personal authorization is obtained by means of pop-up information or asking the person to upload his / her personal information. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

Claims

1. A text segmentation method, characterized in that, include: Obtain the perplexity of each sentence in the text to be processed; For each of the sentence texts, obtain the comparison result of the perplexity between the sentence text and its adjacent sentence texts, and based on the comparison result, determine whether to select the sentence text as the segmentation point of the passage to be processed; The text to be processed is segmented based on each of the aforementioned segmentation points to obtain several text blocks.

2. The method according to claim 1, characterized in that, For any given sentence text, the sentence text adjacent to it includes a preceding text and a following text, wherein the preceding text is the sentence text preceding the sentence text in the chapter to be processed, and the following text is the sentence text following the sentence text in the chapter to be processed; The step of determining whether to select the sentence text as the segmentation point of the text to be processed based on the comparison result includes: Detect whether the comparison result satisfies a first condition; wherein the first condition includes: the perplexity of both the preceding text and the following text is higher than the perplexity of the sentence text, and the difference between the perplexity of at least one of the preceding text and the following text and the perplexity of the sentence text is not lower than a first threshold. In response to the comparison result indicating that the first condition is met, the sentence text is selected as the segmentation point of the text to be processed.

3. The method according to claim 1, characterized in that, For any given sentence text, the sentence text adjacent to it includes a preceding text and a following text, wherein the preceding text is the sentence text preceding the sentence text in the chapter to be processed, and the following text is the sentence text following the sentence text in the chapter to be processed; The step of determining whether to select the sentence text as the segmentation point of the text to be processed based on the comparison result includes: The comparison result is checked to see if it meets the second condition; wherein the second condition includes: the perplexity of the preceding text is higher than the perplexity of the sentence text, the perplexity of the following text is equal to the perplexity of the sentence text, and the difference between the perplexity of the preceding text and the perplexity of the sentence text is not lower than the second threshold. In response to the comparison result indicating that the second condition is met, the sentence text is selected as the segmentation point of the text to be processed.

4. The method according to claim 1, characterized in that, After determining whether to select the sentence text as the text to be processed based on the comparison result, and before segmenting the text to be processed based on each of the segmentation points to obtain several text blocks, the method further includes: The chapter to be processed is pre-segmented based on each of the aforementioned segmentation points to obtain several candidate blocks; Based on the number of characters in each candidate block, new segmentation points are generated; The text to be processed is segmented based on each of the segmentation points to obtain several text blocks, including: The text to be processed is segmented based on each of the new segmentation points to obtain the several text blocks.

5. The method according to claim 4, characterized in that, The step of generating new segmentation points based on the number of characters in each candidate block includes: The number of characters in each candidate block is summed sequentially to obtain the cumulative number of characters after each summation. In response to the fact that the cumulative character count in the target round is not greater than a preset threshold and the cumulative character count in the round following the target round is greater than the preset threshold, the candidate block corresponding to the target round is selected as a new segmentation point in the chapter to be processed. Based on the candidate block in the round following the target round, the step of sequentially accumulating the character count of each candidate block to obtain the cumulative character count after each accumulation is returned, and subsequent steps are performed until the last candidate block in the chapter to be processed is reached, and the last candidate block in the chapter to be processed is selected as a new segmentation point in the chapter to be processed.

6. The method according to claim 1, characterized in that, The process of obtaining the perplexity of each sentence in the text to be processed includes: Each word element in the sentence text is taken as the target word element in turn, and the text composed of words elements in the chapter to be processed that are located before the target word element is taken as the reference text. Obtain the perplexity of the target word when the reference text is used as a prior condition; The perplexity of the sentence text is obtained by fusing the perplexity of each target word in the sentence text.

7. The method according to claim 6, characterized in that, The step of using the text consisting of words preceding the target word in the text to be processed as reference text includes: Obtain cached data of a word on the target word; wherein the cached data includes at least a key matrix and a value matrix; The cached data is used as reference information for the reference text; The step of obtaining the perplexity of the target lexical unit when the reference text is used as a prior condition includes: Based on the reference information, the perplexity of the target lexical is obtained.

8. A question-answering method, characterized in that, include: Obtain the question to be answered and obtain several text blocks of the target text; wherein, the several text blocks are obtained by segmenting the target text as the text to be processed using the text segmentation method according to any one of claims 1 to 7; Based on the question to be answered, a search is performed in the several text blocks to obtain the target block; Based on the question to be answered and the target block, generate the answer content for the question to be answered.

9. A text segmentation device, characterized in that, include: The acquisition module is used to obtain the perplexity of each sentence in the text to be processed; The comparison module is used to obtain a comparison result of the perplexity between each sentence text and its adjacent sentence texts, and based on the comparison result, determine whether to select the sentence text as the segmentation point of the text to be processed; The segmentation module is used to segment the text to be processed based on each of the segmentation points to obtain several text blocks.

10. A question-answering device, characterized in that, include: A processing module is used to obtain a question to be answered and to obtain several text blocks of a target text; wherein, the several text blocks are obtained by segmenting the target text as a text to be processed using the text segmentation method described in any one of claims 1 to 7; The search module is used to search within the several text blocks based on the question to be answered, and obtain the target block. The generation module is used to generate the answer content for the question to be answered based on the question to be answered and the target block.

11. An electronic device, characterized in that, It includes a memory and a processor coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the text segmentation method of any one of claims 1 to 7, or to implement the question answering method of claim 8.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the text segmentation method according to any one of claims 1 to 7, or the question answering method according to claim 8.

Images