Text segmentation method, computer device, readable storage medium and program product

Abstract

The application relates to a text segmentation method, a computer device, a readable storage medium and a program product, which are applied to the technical field of big data and include the following steps: obtaining a target text recognition interval symbol; a symbol selection step: selecting a target interval symbol with the highest symbol priority from the interval symbol; performing paragraph segmentation on the target text according to the target interval symbol to obtain a plurality of first paragraph segmentation texts; if there is a target paragraph segmentation text with a text length greater than a preset text length threshold in the plurality of first paragraph segmentation texts, updating the target paragraph segmentation text into the target text, returning to execute the symbol selection step until there is no target paragraph segmentation text with a text length greater than the preset text length threshold in the plurality of first paragraph segmentation texts, and obtaining paragraph segmentation text information; performing semantic recognition on the paragraph segmentation text information to obtain semantic recognition information, and merging each paragraph text to obtain a target segmentation text. The method can improve the accuracy of text segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular, relates to a text segmentation method, a computer device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] A knowledge base, also known as an enterprise knowledge base or intelligent knowledge base, is a system designed to store, manage, and retrieve knowledge within an organization or a specific field. It helps users efficiently find the information they need by collecting and organizing data from multiple sources. With the rapid rise of artificial intelligence, especially breakthroughs in deep learning, the form of knowledge bases is undergoing a transformation. Models (e.g., large models) are gradually becoming the core technology of the new generation of knowledge bases. However, the length of text that can be input into a model is limited, so there is an urgent need for an accurate way to segment documents. SUMMARY

[0003] Therefore, it is necessary to provide a text segmentation method, device, computer equipment, computer readable storage medium, and computer program product that can improve the accuracy of text segmentation to solve the above technical problems.

[0004] In a first aspect, the present application provides a text segmentation method, comprising:

[0005] obtaining a target text and identifying a separator symbol in the target text;

[0006] a symbol selection step: selecting a target separator symbol with the highest priority from the separator symbols in the target text;

[0007] segmenting the target text into paragraphs according to the target separator symbol to obtain a plurality of first paragraph segmentation texts;

[0008] if there is a target paragraph segmentation text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmentation texts, updating the target paragraph segmentation text to the target text and returning to execute the symbol selection step until there is no target paragraph segmentation text with a text length greater than the preset text length threshold in the plurality of first paragraph segmentation texts, obtaining paragraph segmentation text information;

[0009] respectively performing semantic recognition on each paragraph text in the paragraph segmentation text information to obtain semantic recognition information, and respectively merging each paragraph text according to the semantic recognition information to obtain a target segmentation text.

[0010] In a second aspect, the present application also provides a text segmentation device, comprising:

[0011] an acquisition module, configured to acquire a target text and identify interval symbols in the target text;

[0012] a selection module, configured to perform a symbol selection step of selecting a target interval symbol with the highest symbol priority from the interval symbols in the target text;

[0013] a segmentation module, configured to perform paragraph segmentation on the target text according to the target interval symbol to obtain a plurality of first paragraph segmented texts;

[0014] an updating module, configured to, if there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, update the target paragraph segmented text to the target text, and return to perform the symbol selection step until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, to obtain paragraph segmentation text information;

[0015] a merging module, configured to perform semantic recognition on each paragraph text in the paragraph segmentation text information respectively to obtain semantic recognition information, and perform merging on each paragraph text according to the semantic recognition information to obtain a target segmented text.

[0016] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0017] acquire a target text and identify interval symbols in the target text;

[0018] perform a symbol selection step of selecting a target interval symbol with the highest symbol priority from the interval symbols in the target text;

[0019] perform paragraph segmentation on the target text according to the target interval symbol to obtain a plurality of first paragraph segmented texts;

[0020] if there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, update the target paragraph segmented text to the target text, and return to perform the symbol selection step until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, to obtain paragraph segmentation text information;

[0021] perform semantic recognition on each paragraph text in the paragraph segmentation text information respectively to obtain semantic recognition information, and perform merging on each paragraph text according to the semantic recognition information to obtain a target segmented text.

[0022] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0023] obtaining a target text, and identifying interval symbols in the target text;

[0024] a symbol selection step of selecting a target interval symbol with the highest symbol priority from the interval symbols in the target text;

[0025] performing paragraph segmentation on the target text according to the target interval symbol to obtain a plurality of first paragraph segmented texts;

[0026] if there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, updating the target paragraph segmented text to the target text, and returning to execute the symbol selection step until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, to obtain paragraph segmented text information;

[0027] respectively performing semantic recognition on each paragraph text in the paragraph segmented text information to obtain semantic recognition information, and respectively merging each paragraph text according to the semantic recognition information to obtain a target segmented text.

[0028] In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the following steps:

[0029] obtaining a target text, and identifying interval symbols in the target text;

[0030] a symbol selection step of selecting a target interval symbol with the highest symbol priority from the interval symbols in the target text;

[0031] performing paragraph segmentation on the target text according to the target interval symbol to obtain a plurality of first paragraph segmented texts;

[0032] if there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, updating the target paragraph segmented text to the target text, and returning to execute the symbol selection step until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, to obtain paragraph segmented text information;

[0033] Respectively perform semantic recognition on each of the paragraph texts in the paragraph segmented text information, obtain semantic recognition information, and respectively merge each of the paragraph texts according to the semantic recognition information to obtain a target segmented text.

[0034] The text segmentation method, device, computer device, computer readable storage medium and computer program product described above obtain a target text and recognize a separator in the target text; a separator selection step: selecting a target separator with the highest priority from the separators in the target text; performing paragraph segmentation on the target text according to the target separator to obtain a plurality of first paragraph segmented texts; if there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, updating the target paragraph segmented text to the target text, and returning to perform the separator selection step until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, to obtain paragraph segmented text information; respectively performing semantic recognition on each of the paragraph texts in the paragraph segmented text information, obtaining semantic recognition information, and respectively merging each of the paragraph texts according to the semantic recognition information to obtain a target segmented text.

[0035] In this way, first, the target text is pre-divided into paragraphs according to the priority of the separators. Considering that the separators are determined by the will of the author of the target text, the text layout of the target text can be reflected to some extent, so the paragraph segmented text information has a certain segmentation effect. After pre-segmentation, semantic recognition is performed on each of the segmented paragraph texts, and each of the paragraph texts is merged according to the semantic recognition information, so that the merged paragraph texts have certain semantic association, thereby improving the accuracy of text segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other related drawings without creative labor based on these drawings.

[0037] Figure 1 An application environment diagram of a text segmentation method of an embodiment;

[0038] Figure 2 A flowchart of a text segmentation method of an embodiment;

[0039] Figure 3A flowchart of a step of obtaining target segmented text by merging each paragraph text according to semantic recognition information in one embodiment is shown in FIG. 1;

[0040] Figure 4 A structure block diagram of a text segmentation device in one embodiment is shown in FIG. 2;

[0041] Figure 5 An internal structure diagram of a computer device in one embodiment is shown in FIG. 3. DETAILED DESCRIPTION

[0042] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0043] It should be noted that the information and data involved in the present application (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or authorized by all parties, and the acquisition, transmission, storage, use and processing of the relevant data comply with the relevant provisions of the national laws and regulations. The content pushed to the user (for example, the first paragraph segmented text, semantic recognition information, target segmented text, etc.) can be refused or can be conveniently refused by the user. In the embodiments of the present application, some software, components, models and other industry solutions may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0044] The text segmentation method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The target text is obtained through the server 104, and the interval symbol in the target text is identified; The symbol selection step: selecting the target interval symbol with the highest symbol priority from the interval symbol in the target text; According to the target interval symbol, the target text is segmented into paragraphs to obtain a plurality of first paragraph segmented texts; If there is a target paragraph segmented text with a text length greater than the preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, the target paragraph segmented text is updated to the target text, and the symbol selection step is executed until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, and the paragraph segmentation text information is obtained; The semantic recognition information is obtained by respectively performing semantic recognition on each paragraph text in the paragraph segmentation text information, and each paragraph text is merged according to the semantic recognition information to obtain the target segmented text. The server 104 can push at least one of the first paragraph segmented text, the semantic recognition information and the target segmented text to the terminal 102. Among them, the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, projection devices, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0045] In one exemplary embodiment, as shown in Figure 2 , a text segmentation method is provided, and the method is applied to Figure 1 the server 104 in the example, and is described in the form of omitting the subject, including the following steps 202 to step 210. Among them:

[0046] Step 202, obtaining a target text and identifying interval symbols in the target text.

[0047] The target text in step 202 is the text waiting for text segmentation, which can be the text waiting to be input into the model, or the text waiting to be laid out in a preset position, which can be a newspaper column block or a webpage layout position or other positions that need to limit the length of the text. The interval symbol is a connecting symbol between texts, including a connecting symbol between texts and a connecting symbol between sentences. The connecting symbol between texts includes, but is not limited to, a line break symbol and a continuous line break symbol, etc. The connecting symbol between sentences includes, but is not limited to, a space, a period, a semicolon, a comma, and a colon.

[0048] Therefore, considering that the text processing of the model usually has a limit on the length of the input text, the text waiting to be input into the model is taken as the target text for subsequent text segmentation, so that the target segmented text obtained by segmentation does not exceed the limit on the length of the input text of the model, thereby ensuring that the input text received by the model each time is complete. Considering that the length of the text in the preset position such as the newspaper column block and the webpage layout position needs to be limited to ensure the reading experience of the user, the text waiting to be laid out in the preset position is taken as the target text for subsequent text segmentation, so that the target segmented text obtained by segmentation can conform to the conventional reading mode after being laid out.

[0049] As an embodiment, the interval symbol in the target text is identified, including: identifying the symbols in the target text, and screening the interval symbols from the symbols in the target text.

[0050] In step 204, the target interval symbol with the highest symbol priority is selected from the interval symbols in the target text.

[0051] In step 204, the symbol priority of the connecting symbol between texts is higher than that of the connecting symbol between sentences. The symbol priorities of the connecting symbol between texts and the connecting symbol between sentences can be set by the user as needed, or can be determined in the following manner: the higher the text correlation of the two parts of the text connected by the connecting symbol, the lower the symbol priority of the connecting symbol. The determination of the text correlation includes, but is not limited to, Pearson correlation coefficient, Spearman rank correlation coefficient, cosine similarity, and Jaccard similarity.

[0052] As an example, before step 204, the method further includes: obtaining the symbol priority of each interval symbol in the target text, and performing step 204.

[0053] Further, as an embodiment, the symbol priority of each interval symbol in the target text is obtained, including: obtaining the symbol priority of each interval symbol in the target text set by the user.

[0054] As another example, the symbol priority of each interval symbol in the target text is obtained, including: for each interval symbol, obtaining the text correlation between the two parts of text corresponding to the interval symbol, and generating the symbol priority of the interval symbol according to the text correlation between the two parts of text corresponding to the interval symbol.

[0055] In this way, by shallow text correlation analysis, the symbol priority of the interval symbol corresponding to the higher text correlation between the two parts of text is set to be lower, and the two parts of text with certain correlation are not divided as much as possible.

[0056] Step 206, according to the target interval symbol, the target text is segmented into paragraphs to obtain a plurality of first paragraph segmented texts.

[0057] For example, step 206 includes: according to the position of the target interval symbol, the target text is segmented into paragraphs to obtain a plurality of first paragraph segmented texts.

[0058] If there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, step 208 is executed to update the target paragraph segmented text to the target text.

[0059] The preset text length threshold is adapted to the object corresponding to the input of the target text (including but not limited to the model and the preset position described above), and specifically, the longer the input limit text length of the object corresponding to the input of the target text, the longer the preset text length threshold, and the preset text length threshold is less than the input limit text length of the object corresponding to the input of the target text.

[0060] In this way, since the preset text length threshold is used to limit the length of the pre-segmented paragraph text, and the longer the input limit text length of the object corresponding to the input of the target text, the stronger the accommodation capacity of the object corresponding to the input of the target text, then the preset text length threshold can also be set longer to ensure that the displayable information matches the accommodation capacity of the object corresponding to the input of the target text; and since text merging needs to be performed subsequently, the preset text length threshold needs to be set to be less than the input limit text length of the object corresponding to the input of the target text, so as to ensure that the target segmented text obtained finally does not exceed the accommodation capacity of the object corresponding to the input of the target text.

[0061] Return to step 204 until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, and obtain the paragraph segmented text information.

[0062] In this way, the target text is segmented according to the symbol priority in sequence, and then segmented according to the preset text length threshold, so that each first segmented text in the segmented text information meets the length requirement and ensures segmentation accuracy.

[0063] In step 210, semantic recognition is performed on each paragraph text in the segmented text information to obtain semantic recognition information, and each paragraph text is merged according to the semantic recognition information to obtain the target segmented text.

[0064] The semantic recognition in step 210 includes but is not limited to a rule-based semantic recognition method, such as a dictionary matching method, a pattern matching method, and a syntax analysis tree method, a machine learning-based method, such as a word embedding model, a language model, and a graph neural network.

[0065] As an embodiment, according to the semantic recognition information, each paragraph text is merged to obtain the target segmented text, including: according to the semantic recognition information, the association degree between each paragraph text is evaluated to obtain association information; and according to each association information, each paragraph text is merged to obtain the target segmented text.

[0066] As an embodiment, after each paragraph text is merged according to the semantic recognition information to obtain the target segmented text, the above method further includes: determining target paragraph segmentation information according to the target segmented text; and constructing target metadata according to the target paragraph segmentation information and the target text.

[0067] As an embodiment, the target metadata is constructed according to the target paragraph segmentation information and the target text, including: constructing initial metadata according to the target text; and adding the target paragraph segmentation information in the initial metadata to obtain the target metadata.

[0068] In the above text segmentation method, the target text is first segmented according to the priority of the interval symbol, and considering that the interval symbol is determined by the will of the author of the target text, the text layout of the target text can be reflected to some extent, so that the segmented text information has a certain segmentation effect. After pre-segmentation, the semantic recognition is performed on each segmented text, and the segmented text is merged according to the semantic recognition information, so that the segmented text has a certain semantic association, thereby improving the accuracy of text segmentation.

[0069] In an exemplary embodiment, as shown in FIG. 1, the text segmentation method includes the following steps. Figure 3As shown, a manner of accurately merging the paragraph texts is provided, and the target segmented text is obtained by merging each paragraph text according to semantic recognition information, including steps 302 to 310. Among them:

[0070] In step 302, each paragraph text is divided into a plurality of paragraph text groups.

[0071] Exemplarily, dividing each paragraph text into a plurality of paragraph text groups includes: determining text information of each paragraph text, wherein the text information is used to represent at least one of a segmentation condition and a text complexity of each paragraph text, and dividing each paragraph text into a plurality of paragraph text groups according to the text information of each paragraph text.

[0072] Further, determining the text information of each paragraph text includes: the text information includes the text complexity, obtaining the text length and text character information of the paragraph text, determining the text character repetition rate of the paragraph text according to the text character information, wherein the text character repetition rate is the average value of the character repetition rate of the characters in the paragraph text; and determining the text complexity of the paragraph text according to the text character repetition rate and the text length of the paragraph text, wherein the lower the text character repetition rate of the paragraph text, the higher the determined text complexity of the paragraph text, and the longer the text length of the paragraph text, the higher the determined text complexity of the paragraph text.

[0073] As an embodiment, the text information includes the total number of paragraphs; and dividing each paragraph text into a plurality of paragraph text groups according to the text information of each paragraph text includes: determining any integer divisor of the total number of paragraphs of each paragraph text as the number of text groups of each paragraph text, or determining the maximum integer divisor or the median integer divisor of the total number of paragraphs of each paragraph text as the number of text groups of each paragraph text, wherein the integer divisor of the total number of paragraphs means that the total number of paragraphs is divisible by the integer divisor without a remainder; and dividing each paragraph text into a plurality of paragraph text groups according to the number of text groups.

[0074] In this way, it can be ensured that the number of paragraphs in each group divided according to the number of text groups is consistent for all paragraph texts.

[0075] It can be understood that only the number of paragraphs in each group divided according to the number of text groups is consistent, and the following situation may exist: when the text length of each paragraph text is unevenly distributed, but consistent text group paragraph numbers are used for division, it is easy to appear that the total text length of each paragraph text contained in one paragraph text group is longer, or the total text length of each paragraph text contained in another paragraph text group is shorter, resulting in the technical defect of uneven paragraph text division.

[0076] To overcome the technical defect of uneven division of the paragraph texts in the above paragraph, as another embodiment, the text information comprises text complexity; the complexity difference between the text total complexity of the paragraph texts contained in each of the divided paragraph text groups is within a preset difference range, which can be set by the user as needed or be an empirical value.

[0077] Specifically, according to the text information of each of the paragraph texts, the paragraph texts are divided into a plurality of paragraph text groups, including: obtaining a pre-grouping number, wherein the pre-grouping number can be set by the user as needed or be determined according to the text total complexity of all the paragraph texts, specifically, the higher the text total complexity of all the paragraph texts, the larger the pre-grouping number; the ratio between the text total complexity of all the paragraph texts and the pre-grouping number is determined as the text total complexity of each of the paragraph texts contained in each group; for the first paragraph text, the first paragraph text is divided into the first paragraph text group; for each non-first paragraph text, if the sum of the text complexity of the non-first paragraph text and the text complexity of the last paragraph text group is not greater than the text total complexity of each of the paragraph texts contained in each group, the non-first paragraph text is divided into the last paragraph text group, and if the sum of the text complexity of the non-first paragraph text and the text complexity of the last paragraph text group is not greater than the text total complexity of each of the paragraph texts contained in each group, the non-first paragraph text is divided into the current paragraph text group.

[0078] For example, the paragraph texts include paragraphs a, b, c, d and e in order, the paragraph a is the first paragraph text, the paragraph a is divided into the first paragraph text group, the sum of the text complexity of the paragraph b and the paragraph a is not greater than the text total complexity of each of the paragraph texts contained in each group, and the paragraph b is divided into the first paragraph text group; the sum of the text complexity of the paragraph c, the paragraph a and the paragraph b is greater than the text total complexity of each of the paragraph texts contained in each group, and the paragraph c is the second paragraph text group; the sum of the text complexity of the paragraph d and the paragraph c is not greater than the text total complexity of each of the paragraph texts contained in each group, and the paragraph d is divided into the second paragraph text group; the sum of the text complexity of the paragraph e, the paragraph d and the paragraph c is not greater than the text total complexity of each of the paragraph texts contained in each group, and the paragraph e is divided into the second paragraph text group.

[0079] In this way, the text total complexity of each of the divided paragraph text groups can be ensured to be relatively uniform.

[0080] It can be understood that, only based on the text information, the division of the paragraph text groups is performed, after the paragraph text groups are divided, the paragraph texts in the paragraph text groups may need to be merged, and then all the paragraph texts in the paragraph text groups may need to be merged, without considering the length limit of the division text of the paragraph text groups, the final merged target segmentation text may not conform to the length limit of the input text corresponding to the target text, resulting in the technical defect of inaccurate text segmentation.

[0081] To overcome the technical defect of inaccurate text segmentation caused by the final merged target segmentation text not conforming to the length limit of the input text corresponding to the target text, as an embodiment, the division of the paragraph texts into the paragraph text groups comprises: obtaining the length limit of the text corresponding to the target text, and determining the text information of each paragraph text, wherein the text information is used to represent at least one of the segmentation status and the text complexity of each paragraph text; and dividing the paragraph texts into the paragraph text groups according to the text information of each paragraph text and the length limit of the text.

[0082] In this way, considering that all the paragraph texts in the paragraph text groups may need to be merged, the text information of the paragraph texts and the length limit of the text are used as the basis for dividing the paragraph text groups, so that the multiple paragraph text groups obtained by the division will not exceed the length limit of the input text corresponding to the target text, and then it can be ensured that the final merged target segmentation text will not exceed the length limit of the input text corresponding to the target text, thereby improving the accuracy of text segmentation.

[0083] The length limit of the text is adapted to the object corresponding to the input of the target text, specifically, the longer the input length limit of the text of the object corresponding to the input of the target text, the longer the length limit of the text, and the length limit of the text is greater than a preset text length threshold.

[0084] Further, as an embodiment, the text information comprises the text complexity of each paragraph text; and the division of the paragraph texts into the paragraph text groups according to the text information of each paragraph text and the length limit of the text comprises: determining the number of pre-grouped paragraphs according to the text complexity of each paragraph text; a pre-division step of pre-dividing each paragraph text according to the number of pre-grouped paragraphs; determining the pre-grouped text length of the multiple pre-division text groups obtained by the pre-division according to the text length of each paragraph text; if each pre-grouped text length is not less than or equal to the length limit of the text, reducing the number of pre-grouped paragraphs, and returning to execute the pre-division step until each pre-grouped text length is less than or equal to the length limit of the text, and determining the multiple pre-division text groups as the multiple paragraph text groups.

[0085] The number of pre-grouped paragraphs is the total number of paragraphs included in the pre-division text group.

[0086] Optionally, the specific implementation of determining the pre-grouping paragraph number according to the text complexity of each paragraph text can refer to the specific implementation of dividing each paragraph text into a plurality of paragraph text groups according to the text information of each paragraph text, which will not be repeated here.

[0087] In this way, it is ensured that the plurality of paragraph text groups obtained by division not only match the text complexity of each paragraph text, but also do not exceed the length limit of the input text corresponding to the target text.

[0088] As an example, the step of adjusting the pre-grouping paragraph number includes: in the case where the pre-grouping paragraph number includes a plurality of pre-grouping paragraph numbers, at least one pre-grouping paragraph number can be adjusted.

[0089] For each paragraph text group, step 304 is performed to evaluate the degree of association between each pair of paragraph texts included in the paragraph text group according to the semantic recognition information, to obtain association information.

[0090] The association information can be used to represent at least one of the degree of association between each pair of adjacent paragraph texts included in the paragraph text group and the degree of association between each pair of non-adjacent paragraph texts.

[0091] For example, step 304 includes: for each paragraph text group, extracting text semantic features of each paragraph text included in the paragraph text group according to the semantic recognition information, wherein the text semantic features are used to represent at least one of the theme features, the viewpoint features, the term features, the event features and the concept features of the paragraph text; and evaluating the degree of association between each pair of paragraph texts included in the paragraph text group according to the text semantic features of each paragraph text included in the paragraph text group, to obtain the association information.

[0092] As an embodiment, the relevance between each pair of the passage texts in the passage text group is evaluated according to the text semantic features of the passage texts in the passage text group, and the relevance information is obtained, including: for each pair of passage texts composed of two passage texts, the relevance between the pair of passage texts is evaluated from each semantic feature dimension according to the text semantic features of the two passage texts in the pair of passage texts, and the semantic dimension relevance information corresponding to each semantic feature dimension is obtained, wherein the semantic feature dimension includes but is not limited to the theme feature, the viewpoint feature, the term feature, the event feature and the concept feature; the semantic dimension relevance information corresponding to each semantic feature dimension is weighted according to the semantic dimension weight corresponding to each semantic feature dimension, and the semantic weighted relevance information corresponding to each semantic feature dimension is obtained, wherein the semantic dimension weight can be set by the user as needed, and the semantic dimension weight includes the weight of at least one of the theme feature, the viewpoint feature, the term feature, the event feature and the concept feature of the passage text on the influence of the relevance evaluation between the passages; the semantic weighted relevance information corresponding to each semantic feature dimension is fused to obtain the relevance information.

[0093] If the target pair of passage texts in the passage text group has relevance information that meets the preset relevance condition, step 306 is performed to merge the target pair of passage texts in the passage text group.

[0094] If the relevance degree represented by the relevance information is greater than the preset degree threshold, it is determined that the relevance information meets the preset relevance condition, and the preset degree threshold can be set by the user as needed or can be an empirical value, which is not limited here.

[0095] It can be understood that the above-mentioned relevance information may represent the relevance degree between two passage texts that are not adjacent, therefore, the two passage texts included in the target pair of passage texts may not be adjacent passages, at this time, if the target pair of passage texts is directly merged, the merged text may have a description that is not smooth or a logical error, resulting in the technical defect of low accuracy of passage merging.

[0096] To overcome the technical defect of low accuracy of passage merging due to the possibility that the merged text may have a description that is not smooth or a logical error, before merging the target pair of passage texts in the passage text group, the above method further includes: if the two passage texts included in the target pair of passage texts are adjacent passages, the step of merging the target pair of passage texts in the passage text group is performed; if the two passage texts included in the target pair of passage texts are not adjacent passages, the text fluency between the two passage texts included in the target pair of passage texts is determined, and the step of merging the target pair of passage texts in the passage text group is performed if the text fluency meets the preset fluency condition.

[0097] Further, the text fluency between the two passage texts contained in the target passage text pair is determined, including: extracting the last text part of the earlier passage text in the target passage text pair, and the first text part of the later passage text in the target passage text pair; evaluating the coherence fluency and logical fluency between the last text part and the first text part; and determining the text fluency between the two passage texts contained in the target passage text pair according to the coherence fluency and the logical fluency between the last text part and the first text part, wherein the higher the coherence fluency, the higher the text fluency; and the higher the logical fluency, the higher the text fluency.

[0098] The coherence fluency and the logical fluency between the last text part and the first text part can be achieved by natural language processing (NLP) technology.

[0099] If there is no target passage text pair in the passage text group that meets the preset relevance condition, step 308 is performed to keep the passage text group unchanged.

[0100] When all the passage text groups are processed, step 310 is performed to combine the passage text groups to obtain the target segmented text.

[0101] In this embodiment, the passage texts are divided into multiple passage text groups, so that the relevance degree of each passage text group is evaluated, and the text pairs in the passage text group are merged. Considering that the passages with relevance usually do not have a long interval, it is not necessary to evaluate the relevance degree between all the passage texts. The relevance degree is evaluated by grouping, which ensures the efficiency of the passage text merging.

[0102] As a detailed embodiment, the target text is obtained, and a separator symbol in the target text is identified; a symbol selection step: selecting a target separator symbol with the highest symbol priority from the separator symbols in the target text; according to the target separator symbol, performing paragraph segmentation on the target text to obtain a plurality of first paragraph segmented texts; if there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, updating the target paragraph segmented text to the target text, and returning to execute the symbol selection step until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, to obtain paragraph segmented text information; respectively performing semantic recognition on each paragraph text in the paragraph segmented text information to obtain semantic recognition information, and obtaining a limited text length corresponding to the target text, and determining text information of each paragraph text, wherein the text information is used to represent at least one of a segmentation condition and a text complexity of each paragraph text; according to the text complexity of each paragraph text, determining a pre-grouping paragraph number; a pre-division step: according to the pre-grouping paragraph number, pre-dividing each paragraph text; according to the text length of each paragraph text, determining a pre-grouping text length of a plurality of pre-division text groups obtained by pre-division; if each pre-grouping text length is not all less than or equal to the limited text length, reducing the pre-grouping paragraph number, and returning to execute the pre-division step until each pre-grouping text length is less than or equal to the limited text length, and determining the plurality of pre-division text groups as a plurality of paragraph text groups.

[0103] Further, for each paragraph text group, according to the semantic recognition information, text semantic features of each paragraph text contained in the paragraph text group are extracted, wherein the text semantic features are used to represent at least one of theme features, viewpoint features, term features, event features and concept features of the paragraph text; according to the text semantic features of each paragraph text contained in the paragraph text group, the relevance between each pair of paragraph texts contained in the paragraph text group is evaluated to obtain relevance information; if there is a target paragraph text pair in the paragraph text group that satisfies the preset relevance condition, in the case that the two paragraph texts contained in the target paragraph text pair are adjacent paragraphs, the step of merging the target paragraph text pair in the paragraph text group is performed; in the case that the two paragraph texts contained in the target paragraph text pair are not adjacent paragraphs, the text fluency between the two paragraph texts contained in the target paragraph text pair is determined, and in the case that the text fluency satisfies the preset fluency condition, the step of merging the target paragraph text pair in the paragraph text group is performed; if there is no target paragraph text pair in the paragraph text group that satisfies the preset relevance condition, the paragraph text group remains unchanged; when all the paragraph text groups are processed, the paragraph text groups are combined to obtain a target segmented text; according to the target segmented text, target paragraph segmentation information is determined; and according to the target paragraph segmentation information and the target text, target metadata is constructed.

[0104] In this way, first, the target text is pre-divided into paragraphs according to the priority of the interval symbol, and considering that the interval symbol is determined by the will of the author of the target text, it can reflect the text layout of the target text to some extent, so the paragraph segmentation text information has a certain segmentation effect; and after pre-segmentation, the semantic recognition of each paragraph text is performed, and the paragraph texts are merged according to the semantic recognition information, so that the paragraph texts after merging have certain semantic association, thereby improving the accuracy of text segmentation.

[0105] Further, the paragraph texts are divided into multiple paragraph text groups, so that the relevance of each paragraph text group is evaluated, and the merging of the text pairs in the paragraph text group is realized, and considering that the paragraphs with relevance will not be far apart, it is not necessary to evaluate the relevance between all the paragraph texts, and the relevance of each group is evaluated, thereby ensuring the efficiency of the paragraph text merging.

[0106] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time but can be executed at different times, and the execution of the steps or stages is not necessarily sequential but can be performed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiments of the present application also provide a text segmentation device for implementing the above-mentioned text segmentation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more text segmentation device embodiments provided below can refer to the limitations of the text segmentation method in the above text, which will not be repeated here.

[0108] In one exemplary embodiment, as shown in Figure 4 A text segmentation device 400 is provided, comprising: an acquisition module 402, a selection module 404, a segmentation module 406, an update module 408 and a merging module 410, wherein:

[0109] The acquisition module 402 is configured to acquire a target text and identify a separator in the target text;

[0110] The selection module 404 is configured to perform a symbol selection step of selecting a target separator with the highest symbol priority from the separators in the target text;

[0111] The segmentation module 406 is configured to perform paragraph segmentation on the target text according to the target separator to obtain a plurality of first paragraph segmented texts;

[0112] The update module 408 is configured to update a target paragraph segmented text to the target text if there is a target paragraph segmented text with a text length greater than a preset text length threshold corresponding to the target text in the plurality of first paragraph segmented texts, and return to perform the symbol selection step until there is no target paragraph segmented text with a text length greater than the preset text length threshold in the plurality of first paragraph segmented texts, to obtain paragraph segmentation text information;

[0113] The merging module 410 is configured to perform semantic recognition on each paragraph text in the paragraph segmentation text information respectively to obtain semantic recognition information, and merge each paragraph text according to the semantic recognition information to obtain a target segmented text.

[0114] In one of the embodiments, the merging module 410 is further configured to divide the paragraph texts into a plurality of paragraph text groups; for each of the paragraph text groups, evaluate the relevance degree between each pair of the paragraph texts contained in the paragraph text group according to the semantic recognition information, and obtain relevance information; if there is a target paragraph text pair in the paragraph text group that satisfies a preset relevance condition, merge the target paragraph text pair in the paragraph text group; if there is no target paragraph text pair in the paragraph text group that satisfies the preset relevance condition, keep the paragraph text group unchanged; and when all the paragraph text groups are processed, combine the paragraph text groups to obtain the target segmented text.

[0115] In one of the embodiments, the merging module 410 is further configured to obtain a limited text length corresponding to the target text, and determine text information of each of the paragraph texts, wherein the text information is used to represent at least one of a segmentation state and a text complexity of each of the paragraph texts; and divide the paragraph texts into the plurality of paragraph text groups according to the text information of each of the paragraph texts and the limited text length.

[0116] In one of the embodiments, the text information includes the text complexity of each of the paragraph texts; and the merging module 410 is further configured to determine a pre-grouping paragraph number according to the text complexity of each of the paragraph texts; and perform a pre-division step according to the pre-grouping paragraph number, determine a pre-grouping text length of a plurality of pre-division text groups obtained by the pre-division according to the text length of each of the paragraph texts, and if each of the pre-grouping text lengths is greater than or equal to the limited text length, reduce the pre-grouping paragraph number and return to perform the pre-division step until each of the pre-grouping text lengths is less than or equal to the limited text length, and determine the plurality of pre-division text groups as the plurality of paragraph text groups.

[0117] In one of the embodiments, before merging the target paragraph text pair in the paragraph text group, the merging module 410 is further configured to, if the two paragraph texts contained in the target paragraph text pair are adjacent paragraphs, perform the step of merging the target paragraph text pair in the paragraph text group; and if the two paragraph texts contained in the target paragraph text pair are not adjacent paragraphs, determine a text fluency between the two paragraph texts contained in the target paragraph text pair, and perform the step of merging the target paragraph text pair in the paragraph text group if the text fluency satisfies a preset fluency condition.

[0118] In one of the embodiments, the merging module 410 is further configured to extract, for each paragraph text group, text semantic features of the paragraph texts included in the paragraph text group according to the semantic recognition information, wherein the text semantic features are used to represent at least one of theme features, viewpoint features, term features, event features and concept features of the paragraph texts; and evaluate the correlation degrees between the paragraph texts included in the paragraph text group according to the text semantic features of the paragraph texts included in the paragraph text group, to obtain the correlation information.

[0119] In one of the embodiments, after the merging of the paragraph texts according to the semantic recognition information, the device further comprises a constructing module configured to determine target paragraph segmentation information according to the target segmented texts; and construct target metadata according to the target paragraph segmentation information and the target texts.

[0120] The modules in the above text segmentation device can be realized by software, hardware or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to the modules.

[0121] In one of the embodiments, a computer device is provided, which can be a terminal. The internal structure diagram of the computer device can be as shown in Figure 5The computer device shown in the figure includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (NFC) or other technologies. The computer program is executed by the processor to realize a text segmentation method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0122] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0124] In one embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0125] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0127] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0128] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A text segmentation method, characterized in that, The method includes: Obtain the target text and identify the spacing symbols in the target text; Symbol selection step: Select the target spacer symbol with the highest symbol priority from the spacer symbols in the target text; Based on the target interval symbol, the target text is segmented into paragraphs to obtain multiple first paragraph segmented texts; If there is a target paragraph segment text among the plurality of first paragraph segment texts whose text length is greater than the preset text length threshold corresponding to the target text, then the target paragraph segment text is updated to the target text, and the symbol selection step is returned to be executed until there is no target paragraph segment text among the plurality of first paragraph segment texts whose text length is greater than the preset text length threshold, and paragraph segment text information is obtained; Semantic recognition is performed on each paragraph text in the segmented text information to obtain semantic recognition information, and the restricted text length corresponding to the target text is obtained. The text information of each paragraph text is determined, wherein the text information is used to characterize at least one of the segmentation status and text complexity of each paragraph text. Based on the text complexity of each paragraph text, the number of pre-grouped paragraphs is determined, wherein the text complexity is determined by the text length and the text character repetition rate; the higher the text complexity of all paragraph texts, the larger the number of pre-grouped paragraphs is determined. The pre-segmentation step is as follows: For the first paragraph text, it is divided into the first paragraph text group; for each non-first paragraph text, if the sum of the text complexity of the non-first paragraph text and the text complexity of the previous paragraph text group is not greater than the total text complexity of all paragraph texts contained in each group, then the non-first paragraph text is divided into the previous paragraph text group; if the sum of the text complexity of the non-first paragraph text and the text complexity of the previous paragraph text group is greater than the total text complexity of all paragraph texts contained in each group, then the non-first paragraph text is divided into the previous paragraph text group. If the non-first paragraph text is assigned to the current paragraph text group, then the pre-group text length of the multiple pre-divided text groups is determined based on the text length of each paragraph text. If the length of each pre-group text is not less than or equal to the limit text length, then the number of pre-group paragraphs is reduced, and the pre-division step is returned to be executed until the length of each pre-group text is less than or equal to the limit text length, and the multiple pre-divided text groups are determined as multiple paragraph text groups. For each paragraph text group, the degree of correlation between each pair of paragraph texts contained in the paragraph text group is evaluated based on the semantic recognition information to obtain correlation information. If there are target paragraph text pairs in the paragraph text group whose correlation information meets the preset correlation conditions, then the target paragraph text pairs in the paragraph text group are merged. If there are no target paragraph text pairs in the paragraph text group whose correlation information meets the preset correlation conditions, then the paragraph text group remains unchanged. When all paragraph text groups have been processed, the paragraph text groups are combined to obtain the target segmented text.

2. The method according to claim 1, characterized in that, Before merging the target paragraph text pairs in the paragraph text group, the method further includes: If the two paragraphs in the target paragraph text pair are adjacent paragraphs, then the step of merging the target paragraph text pairs in the paragraph text group is performed; If the two paragraphs in the target paragraph text pair are not adjacent paragraphs, then the text fluency between the two paragraphs in the target paragraph text pair is determined, and if the text fluency meets the preset fluency condition, the step of merging the target paragraph text pairs in the paragraph text group is executed.

3. The method according to claim 1 or 2, characterized in that, The step of evaluating the degree of correlation between each pair of paragraph texts in the paragraph text group based on the semantic recognition information to obtain correlation information includes: For each paragraph text group, based on the semantic recognition information, the text semantic features of each paragraph text contained in the paragraph text group are extracted respectively, wherein the text semantic features are used to characterize at least one of the topic features, viewpoint features, terminology features, event features and concept features of the paragraph text; Based on the semantic features of each paragraph in the paragraph text group, the degree of correlation between each pair of paragraph texts in the paragraph text group is evaluated to obtain correlation information.

4. The method according to claim 1, characterized in that, After merging the paragraph texts according to the semantic recognition information to obtain the target segmented text, the method further includes: Based on the target segmented text, determine the target paragraph segmentation information; Based on the target paragraph segmentation information and the target text, construct the target metadata.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images