Long text analysis method and device, storage medium and terminal

Abstract

The invention discloses a long text analysis method and device, a storage medium and a terminal. Identifying at least one title in the to-be-analyzed long text, and generating directory information of the to-be-analyzed long text based on each title; dividing the long text to be analyzed into a plurality of text paragraphs according to the titles and the input length requirement of the vector model; determining a target superior directory corresponding to each text paragraph from the directory information, and respectively integrating each text paragraph and the target superior directory corresponding to each text paragraph; and inputting the integrated text paragraphs into a vector model to obtain vector features output by the vector model for the text paragraphs. Due to the fact that the whole document is divided into the text paragraphs through the title of the long text, the text paragraphs are divided more reasonably. And continuously integrating the long text information according to the context information contained in the superior directory, so that the semantic information of the text paragraphs is richer, and the model can more accurately understand and process the text paragraphs on this basis.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a long text parsing method, device, storage medium and terminal. Background Art

[0002] In the face of a knowledge base that includes various documents, in order to improve the efficiency and accuracy of information retrieval, building an efficient and accurate indexing system has become an indispensable part. In this process, the accuracy of document parsing affects the accuracy of document indexing. Generally speaking, document parsing follows a set of carefully designed rules and patterns to cut continuous text streams into pieces that are easy to process and understand. However, this direct parsing method gradually reveals its limitations when dealing with long texts. The internal structure of long texts is complex and changeable, and the associations between contexts often span multiple paragraphs or even chapters. Therefore, the traditional fixed-pattern parsing method has the problem of easily breaking the coherence and logic of long texts. Therefore, when parsing long texts, more flexible methods are needed to adapt to the characteristics of long texts. Summary of the invention

[0003] The present application provides a long text parsing method, device, storage medium and terminal, which can solve the technical problem in the related art that it is difficult to parse long texts completely and accurately.

[0004] In a first aspect, an embodiment of the present application provides a long text parsing method, the method comprising: Identify at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title; Divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; Determine the target parent directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target parent directory corresponding to each text paragraph respectively; The integrated text paragraphs are input into the vector model to obtain the vector features output by the vector model for each text paragraph.

[0005] In a second aspect, an embodiment of the present application provides a long text parsing device, the device comprising: A title recognition module, used to recognize at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title; The paragraph splitting module is used to divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; A semantic integration module is used to determine the target upper directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target upper directory corresponding to each text paragraph respectively; The vector generation module is used to input the integrated text paragraphs into the vector model to obtain the vector features output by the vector model for each text paragraph.

[0006] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0008] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least: The present application provides a long text parsing method, which identifies at least one title in the long text to be parsed, generates directory information of the long text to be parsed based on each title; divides the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; determines the target parent directory corresponding to each text paragraph from the directory information, and integrates each text paragraph and the target parent directory corresponding to each text paragraph respectively; inputs each integrated text paragraph into the vector model to obtain the vector features output by the vector model for each text paragraph. Since the title is an accurate division of the document hierarchy, the title contains the contextual association relationship of the document, then the directory information is generated by the title of the long text, and then the entire document is split into text paragraphs that meet the length requirements of the model input based on the directories at all levels, so as to achieve a more reasonable text paragraph division. Further, the long text information is integrated according to the context information contained in its parent directory, so that the original semantics of the text and the semantics of the directory title are combined, so that the semantic information of the text paragraph is richer; then each text paragraph with rich semantic information is input into the vector model, so that the model can understand and process it more accurately, thereby obtaining a vector sequence rich in semantic information output by the vector model for each text paragraph. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 An exemplary system architecture diagram of a long text parsing method provided in an embodiment of the present application; Figure 2 A flowchart of a long text parsing method provided in an embodiment of the present application; Figure 3 A flowchart of a long text parsing method provided in an embodiment of the present application; Figure 4 A logic flow chart of a long text parsing method provided in an embodiment of the present application; Figure 5 A structural block diagram of a long text parsing device provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0012] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. And in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0013] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0014] In the knowledge base, there are various documents of different lengths. Generally speaking, the richer the content of the document, the longer its text length. When it is necessary to index these documents so that users can retrieve them through queries, the commonly used document parsing method often follows a set of specific rules and patterns to deconstruct the text. This process mainly implements the splitting and fragmentation of the text based on the basic units of the text, such as punctuation marks, specific symbols, etc. However, this disassembly method, although simple and fast to implement, also has certain limitations. This limitation is particularly significant when processing long texts.

[0015] The reason is that related contexts in long texts are often far apart, and two related contents may exist at the beginning and end of the article. Traditional splitting methods based on punctuation or fixed patterns have difficulty capturing these contextual associations across long distances, resulting in information fragmentation and separation. This loss of information and the break in contextual relationships will affect the performance and effects of downstream tasks that have extremely high requirements for text understanding accuracy and depth.

[0016] Therefore, an embodiment of the present application provides a long text parsing method to solve the above-mentioned technical problem that it is difficult to parse a long text completely and accurately.

[0017] See also Figure 1 , Figure 1 An exemplary system architecture diagram of a long text parsing method provided in an embodiment of the present application.

[0018] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.

[0019] The terminal 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103, or the terminal 101 can interact with the server 103 through the network 102 to receive messages or data sent by other users to the server 103. The terminal 101 can be hardware or software. When the terminal 101 is hardware, it can be various electronic devices, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers and desktop computers. When the terminal 101 is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0020] In an embodiment of the present application, first, the terminal 101 identifies at least one title in the long text to be parsed, and generates directory information of the long text to be parsed based on each title; based on the title information, the terminal 101 can divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; further, the terminal 101 determines the target parent directory corresponding to each text paragraph from the directory information, and integrates each text paragraph and the target parent directory corresponding to each text paragraph respectively; after enriching the original semantic information of the text paragraph, the terminal 101 inputs the integrated text paragraphs into the vector model to obtain the vector features output by the vector model for each text paragraph.

[0021] The server 103 may be a business server that provides various services. It should be noted that the server 103 may be hardware or software. When the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 103 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or as a single software or software module, which is not specifically limited here.

[0022] Alternatively, the system architecture may not include the server 103. In other words, the server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification may be applied to a system structure that only includes the terminal 101, and the embodiments of this application do not limit this.

[0023] It should be understood that Figure 1 The number of terminals, networks and servers in the figure is only for illustration and any number of terminals, networks and servers may be used according to implementation requirements.

[0024] See also Figure 2 , Figure 2A flowchart of a long text parsing method provided for an embodiment of the present application. The execution subject of the embodiment of the present application can be a terminal that executes long text parsing, or a processor in the terminal that executes the long text parsing method, or a long text parsing service in the terminal that executes the long text parsing method. For the convenience of description, the specific execution process of the long text parsing method is introduced below by taking the execution subject being a processor in a terminal as an example.

[0025] like Figure 2 As shown, the long text parsing method may at least include: S202: Identify at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title.

[0026] Alternatively, since the context distance of a long text is relatively far, if the long text is directly parsed as a whole, the model will find it difficult to understand the relationship between the contexts, resulting in inaccurate semantic information in the vector, and generating a vector based on the entire text will also result in a large amount of information loss in the original text. Therefore, in order to ensure that the vector of a long text has complete and accurate semantic information, it is necessary to first split the long text, and then perform semantic parsing on each text paragraph and generate a vector, so as to achieve effective parsing of the original text.

[0027] Alternatively, for long texts, if key information is missed or the original logical relationship is distorted due to improper splitting during the text splitting stage, the accuracy of the entire parsing result will be affected in the end. Therefore, when splitting the text, it is necessary to consider reasonable splitting according to the hierarchical structure of the text. Understandably, in long texts, in order to facilitate reading and understanding, the hierarchical structure of the document is often divided by adding titles. In addition to clarifying the structure of the document, the title also has the function of refining and summarizing the content of the document, so that the title can construct and highlight the hierarchical structure and contextual relationship within the document.

[0028] Optionally, based on the rich contextual information contained in the title, the long text can be reasonably split by the title. Therefore, in the process of parsing the long text, it is first necessary to identify the various titles in the long text to be parsed. The specific process is to traverse the text and identify at least one representative and distinctive title. This is not just a simple extraction of the title, but also a deep analysis and reconstruction of the internal logical structure of the text. Subsequently, after these titles are identified, based on the hierarchical relationship and order of these titles, detailed and accurate directory information can be further dynamically generated.

[0029] Furthermore, in an embodiment of the present application, an efficient and intuitive form is chosen to reflect the directory information - the directory tree. The structure of the directory tree is essentially a tree data structure (Tree Data Structure), which is particularly suitable for representing data with hierarchical or nested relationships. In the directory tree, each title (such as chapter name, paragraph title, etc.) in the long text to be parsed is regarded as a node in the tree, and the hierarchical relationship between these titles (such as chapters, subsections, subsections, etc.) is used as the edge connecting these nodes. In the directory tree, this hierarchical relationship is usually vertical, that is, there can be multiple subtitles (child nodes) under a title (parent node), forming a top-down hierarchical structure. The edge not only defines the position of the node, but also determines the access path of the text paragraph. In summary, the directory information, as a directory tree with a tree data structure, clearly displays the logical relationship between text paragraphs through the combination of nodes and edges, effectively organizes and displays information with hierarchical relationships, and provides great convenience for the storage, management and use of directory information.

[0030] S204: Divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model.

[0031] Optionally, parsing long texts is usually done by converting the semantic information in the text into vectors to obtain the vector features corresponding to the text. In this process, the vector features can be output through the vector model. Due to the different parameters and algorithms of different vector models (such as Word2Vec, BERT, etc.), there are different restrictions or recommended values ​​for the length of the input text. Therefore, in order to facilitate the conversion of vector features, when dividing the text paragraphs of long texts, in addition to dividing according to the title, it is also necessary to consider the input rules of the downstream model.

[0032] Optionally, in the above-mentioned embodiment steps, by traversing and scanning the long text, all possible chapter titles, subsection titles or naturally formed paragraph topic sentences have been identified, and these titles or topic sentences can summarize the core idea of ​​the content of this part. At the same time, it is also necessary to evaluate the input length requirement of the vector model. Therefore, when dividing paragraphs, it is necessary to divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model, ensuring that the length of each paragraph does not exceed the processing capacity of the model, and can retain sufficient context information so that the model can accurately capture the semantic features of the paragraph.

[0033] Furthermore, after the division is completed, the entire text can be reviewed again to ensure that the connection between paragraphs is natural and smooth and the content logic is clear. At the same time, the test data can be used to verify whether the division results meet the processing requirements of the vector model and make fine adjustments if necessary. This division process fully considers the topic boundaries indicated by the title and enables the model to smoothly receive and process all the divided text paragraphs, thereby achieving a more refined and reasonable organization of the text content.

[0034] S206: Determine the target upper-level directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target upper-level directory corresponding to each text paragraph.

[0035] Optionally, in order to fully explore and utilize the semantic connection between the text paragraph and its title, the semantic information of each text paragraph can be deeply integrated with the semantic information in its parent title, so as to enrich the information in the text paragraph. This process not only involves semantic analysis at the lexical level, but also covers the identification of logical relationships between phrases, sentences and even paragraphs, aiming to build a more comprehensive and rich semantic network.

[0036] Optionally, the tree structure of the directory tree supports multiple traversal methods (such as depth-first search, breadth-first search) and search algorithms (such as binary search, which is not directly applicable to ordinary tree structures but can be used to balance trees or search trees). Through these operations, the target parent directory corresponding to each text paragraph can be efficiently determined from the directory information.

[0037] Furthermore, a series of natural language processing (NLP) technologies are adopted to achieve deep fusion and enhancement of semantic information between text paragraphs and target parent directories. In the long text to be parsed, the identification of logical relationships between paragraphs and between paragraphs and titles is the core of semantic information integration. This includes identifying logical relationships such as cause and effect, transition, parallelism, and progression between text paragraphs, as well as how these paragraphs form overall support, overview, or detailed elaboration relationships with parent titles. By integrating semantic information at all levels, text paragraphs with richer semantics can be obtained.

[0038] S208 , inputting each integrated text paragraph into a vector model to obtain vector features output by the vector model for each text paragraph.

[0039] Optionally, these semantically rich text paragraphs are used as input and sent to a pre-trained vector model. The model can capture and learn complex features in the text through a deep learning algorithm, and then generate a high-dimensional vector representation for each text paragraph. These vector features not only retain the core content of the text paragraph, but also contain deep association information with the entire long text and other paragraphs. Finally, thanks to the accurate representation of the text paragraph by the vector model, the system can understand and process the long text in a more efficient and intelligent way. This makes it possible to achieve more accurate and comprehensive result output based on the method of the embodiment of the present application, whether it is for long texts in information retrieval and summary generation tasks, or for long texts in advanced tasks such as sentiment analysis and semantic matching in intelligent processing scenarios of office presentation files. This process not only improves the efficiency of text processing, but also enhances the interpretability and applicability of text content.

[0040] In an embodiment of the present application, a long text parsing method is provided, which identifies at least one title in the long text to be parsed, generates directory information of the long text to be parsed based on each title; divides the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; determines the target parent directory corresponding to each text paragraph from the directory information, and integrates each text paragraph and the target parent directory corresponding to each text paragraph respectively; inputs each integrated text paragraph into the vector model to obtain the vector features output by the vector model for each text paragraph. Since the title is an accurate division of the document hierarchy, the title contains the contextual association relationship of the document, then the directory information is generated by the title of the long text, and then the entire document is split into text paragraphs that meet the length requirements of the model input based on the directories at all levels, so as to achieve a more reasonable text paragraph division. Further, the long text information is integrated according to the context information contained in its parent directory, so that the original semantics of the text and the semantics of the directory title are combined, so that the semantic information of the text paragraph is richer; then each text paragraph with rich semantic information is input into the vector model, so that the model can understand and process it more accurately, thereby obtaining a vector sequence rich in semantic information output by the vector model for each text paragraph.

[0041] See also Figure 3 , Figure 3 A flowchart of a long text parsing method provided in an embodiment of the present application.

[0042] like Figure 3 As shown, the long text parsing method may at least include: S302: Identify at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title.

[0043] Regarding step S302, please refer to the detailed description in step S202, which will not be repeated here.

[0044] S304: Split the long text to be parsed into multiple text segments according to each title.

[0045] Optionally, see Figure 4 , Figure 4 The following is a logic flow chart of a long text parsing method provided in an embodiment of the present application. Figure 4 As shown, when parsing a long text, firstly, all titles in the long text are identified through the title recognition module, and based on the titles and the hierarchy between the titles, a directory tree corresponding to the long text is generated. On this basis, the long text to be parsed continues to be processed in paragraphs. When splitting the long text to be parsed, the long text to be parsed is first split into multiple text fragments according to the titles. In this process, the division can be carried out according to specific fragment separation conditions. For example, in the fragment separation conditions, it is stipulated that the long text is split according to the minimum level of titles, then it is necessary to first determine the title level that meets the fragment separation conditions, that is, all the minimum level titles, and split the long text to be parsed into multiple text fragments according to these titles.

[0046] S306: Determine whether the length of each text segment meets the input length requirement of the vector model. If there is a target text segment that does not meet the input length requirement, split the target text segment into multiple text paragraphs that meet the input length requirement.

[0047] Optionally, after dividing the text segments, it is necessary to make necessary adjustments to the text segments initially divided based on the titles according to the input length requirements of the vector model, and further process these text segments into text paragraphs whose lengths meet the model requirements. For segments that are too long, further segmentation can be considered based on content logic or keywords; for segments that are too short, if their content is closely related to adjacent segments, merging can be considered.

[0048] For details, please continue to see Figure 4 First, determine whether the length of each text segment meets the input length requirement of the vector model. For text segments that already meet the input length requirement, they can be directly used as text paragraphs. For target text segments that do not meet the input length requirement, the target text segment needs to be split into multiple text paragraphs that meet the input length requirement.

[0049] Optionally, in order to perform efficient splitting, text delimiters (such as periods, semicolons, commas, etc.) and the overlap length between text paragraphs can be preset. When splitting the target text segment, the target text segment is first split into multiple sentences according to the preset text delimiters, and then the multiple sentences are spliced ​​according to the preset paragraph overlap length to obtain multiple text paragraphs that meet the input length requirements.

[0050] S308, determining the target parent directory corresponding to each text paragraph from the directory information, obtaining the context information of each text paragraph according to the target parent directory corresponding to each text paragraph, and integrating the context information of each text paragraph into the semantic information of each text paragraph.

[0051] Alternatively, if Figure 4 As shown, the directory tree records all the title information in the long text to be parsed. Based on the directory tree and the long text to be parsed, the parent title of each text paragraph can be traced back to determine the target parent directory corresponding to each text paragraph, and the paragraph can be accurately associated with the corresponding directory item. After determining the mapping relationship between each text paragraph and its target parent directory, this mapping relationship can be used to extract the context information of each text paragraph. The context information includes not only the semantics of the directly adjacent paragraph content, but may also include the semantics of other paragraphs under the target parent directory and its sibling directory items, and even the semantics of higher-level directory content, depending on the scope and depth of the context.

[0052] Furthermore, after obtaining the context information of each text paragraph according to the target parent directory, the extracted context information is integrated and fused with the original semantic information of each text paragraph, thereby enriching the semantics protected in each text paragraph. When fusing the context information, the text paragraph can be first subjected to an in-depth semantic analysis to understand its intrinsic meaning, emotional tendency, key concepts, etc. After the original semantic information of the text paragraph is extracted, the context information based on the directory is combined with the original semantic information of the text paragraph through weight allocation, information screening, semantic fusion algorithm and other technologies. In this way, the original semantics of the text and the semantics of the directory title are combined to enrich the semantic information of the text paragraph. In this way, the text paragraphs with rich semantic information are subsequently input into the vector model, so that the model can understand and process them more accurately, thereby obtaining a vector sequence richer in semantic information.

[0053] S310 , inputting each integrated text paragraph into a vector model to obtain vector features output by the vector model for each text paragraph.

[0054] Regarding step S310, please refer to the detailed description in step S208, which will not be repeated here.

[0055] S312: Establish a vector index relationship between the vector features of each text paragraph and the long text to be parsed, and store the vector index relationship in a retrieval database.

[0056] Optionally, after obtaining the vector features of the long text to be parsed, an accurate and efficient vector index relationship can be further established. Specifically, an index is established for the vector of the long text to be parsed in the vector space so that the similarity between vectors can be quickly calculated. These indexes may be constructed based on strategies such as space partitioning (such as KD tree, ball tree, etc.), hashing technology, or approximate nearest neighbor search algorithm, aiming to improve the efficiency of vector retrieval. In this way, documents and queries are mapped to the vector space and retrieved using vector indexes, which can accurately capture the deep semantic connections between texts.

[0057] It is understandable that for a long text with vector features of multiple text paragraphs, there are two ways to construct its index. One is to establish a vector index relationship between each text segment and its own vector feature, and then establish a connection relationship between the blocks of the text segment and the original document. The other is to directly establish an index relationship between all vector features of all text segments and their original documents. This embodiment of the application is not limited to this.

[0058] In an embodiment of the present application, a long text parsing method is provided, which splits the long text to be parsed into multiple text segments according to each title, and splits the target text segments that do not meet the input length requirements into multiple text paragraphs that meet the input length requirements by presetting text delimiters and overlapping lengths, thereby achieving effective and reasonable segmentation of the long text. The target parent directory corresponding to each text paragraph is determined from the directory information, and the context information of each text paragraph is obtained according to the target parent directory corresponding to each text paragraph. The context information of each text paragraph is respectively integrated into the semantic information of each text paragraph, and the semantic information of the text paragraph itself and the semantic information of its corresponding directory tree are integrated. And after generating the vector, a vector index relationship between the vector features of each text paragraph and the long text to be parsed is established, and the vector index relationship is stored in a retrieval database to support the user's retrieval activities for the long text to be parsed.

[0059] See also Figure 5 , Figure 5 This is a structural block diagram of a long text parsing device provided in an embodiment of the present application. Figure 5 As shown, the long text parsing device 500 includes: A title identification module 510, used to identify at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title; A paragraph splitting module 520 is used to divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; The semantic integration module 530 is used to determine the target upper directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target upper directory corresponding to each text paragraph respectively; The vector generation module 540 is used to input the integrated text paragraphs into the vector model to obtain the vector features output by the vector model for each text paragraph.

[0060] Optionally, the paragraph splitting module 520 is also used to split the long text to be parsed into multiple text segments according to each title; determine whether the length of each text segment meets the input length requirement of the vector model; if there is a target text segment that does not meet the input length requirement, split the target text segment into multiple text paragraphs that meet the input length requirement.

[0061] Optionally, the paragraph splitting module 520 is further used to determine a title level that meets the segment separation condition, and split the long text to be parsed into multiple text segments according to each title belonging to the title level.

[0062] Optionally, the paragraph splitting module 520 is further used to split the target text segment into multiple sentences according to preset text delimiters; and to splice the multiple sentences according to a preset paragraph overlap length to obtain multiple text paragraphs that meet the input length requirement.

[0063] Optionally, the directory information is a directory tree with a tree data structure, in which each title is used as a node and the level relationship between the titles is used as an edge connecting the nodes.

[0064] Optionally, the semantic integration module 530 is further used to obtain context information of each text paragraph according to the target parent directory corresponding to each text paragraph, and fuse the context information of each text paragraph into the semantic information of each text paragraph.

[0065] Optionally, the long text parsing device 500 further includes: an index building module, which is used to establish a vector index relationship between the vector features of each text paragraph and the long text to be parsed, and store the vector index relationship in a retrieval database.

[0066] In an embodiment of the present application, a long text parsing device is provided. Among them, a title recognition module is used to identify at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title; a paragraph splitting module is used to divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; a semantic integration module is used to determine the target parent directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target parent directory corresponding to each text paragraph respectively; a vector generation module is used to input each integrated text paragraph into the vector model to obtain the vector features output by the vector model for each text paragraph. Since the title is an accurate division of the document hierarchy, the contextual association relationship of the document is contained in the title, so the directory information is generated through the title of the long text, and then the entire document is split into text paragraphs that meet the length requirements of the model input based on the directories at all levels, thereby achieving a more reasonable text paragraph division. Furthermore, the long text information is integrated according to the contextual information contained in its parent directory, which combines the original semantics of the text and the semantics of the directory title to enrich the semantic information of the text paragraph; then each text paragraph with rich semantic information is input into the vector model, so that the model can understand and process it more accurately, thereby obtaining a vector sequence rich in semantic information output by the vector model for each text paragraph.

[0067] An embodiment of the present application further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.

[0068] See also Figure 6 , Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 6 As shown, the terminal 600 may include: at least one terminal processor 601 , at least one network interface 604 , a user interface 603 , a memory 605 , and at least one communication bus 602 .

[0069] The communication bus 602 is used to realize the connection and communication between these components.

[0070] The user interface 603 may include a display screen (Display) and a camera (Camera). The optional user interface 603 may also include a standard wired interface and a wireless interface.

[0071] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0072] Among them, the terminal processor 601 may include one or more processing cores. The terminal processor 601 uses various interfaces and lines to connect various parts in the entire terminal 600, and executes various functions and processes data of the terminal 600 by running or executing instructions, programs, code sets or instruction sets stored in the memory 605, and calling data stored in the memory 605. Optionally, the terminal processor 601 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The terminal processor 601 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the terminal processor 601, and it can be implemented separately through a chip.

[0073] Among them, the memory 605 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 605 includes a non-transitory computer-readable storage medium. The memory 605 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 605 may also be optionally at least one storage device located away from the aforementioned terminal processor 601. As Figure 6 As shown, the memory 605 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a long text parsing program.

[0074] exist Figure 6In the terminal 600 shown, the user interface 603 is mainly used to provide an input interface for the user and obtain the data input by the user; and the terminal processor 601 can be used to call the long text parsing program stored in the memory 605 and specifically perform the following operations: Identify at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title; Divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; Determine the target parent directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target parent directory corresponding to each text paragraph respectively; The integrated text paragraphs are input into the vector model to obtain the vector features output by the vector model for each text paragraph.

[0075] In some embodiments, when the terminal processor 601 divides the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model, it specifically performs the following steps: splits the long text to be parsed into multiple text segments according to each title; determines whether the length of each text segment meets the input length requirements of the vector model; if there is a target text segment that does not meet the input length requirements, splits the target text segment into multiple text paragraphs that meet the input length requirements.

[0076] In some embodiments, when the terminal processor 601 splits the long text to be parsed into multiple text segments according to each title, it specifically performs the following steps: determining the title level that meets the segment separation condition, and splitting the long text to be parsed into multiple text segments according to each title belonging to the title level.

[0077] In some embodiments, when the terminal processor 601 executes the process of splitting the target text segment into multiple text paragraphs that meet the input length requirements, it specifically performs the following steps: splitting the target text segment into multiple sentences according to a preset text separator; splicing the multiple sentences according to a preset paragraph overlap length to obtain multiple text paragraphs that meet the input length requirements.

[0078] In some embodiments, the directory information is a directory tree with a tree data structure, in which each title is used as a node and the level relationship between the titles is used as an edge connecting the nodes.

[0079] In some embodiments, when the terminal processor 601 executes the integration of each text paragraph and the target parent directory corresponding to each text paragraph, it specifically performs the following steps: obtain the context information of each text paragraph according to the target parent directory corresponding to each text paragraph, and integrate the context information of each text paragraph into the semantic information of each text paragraph.

[0080] In some embodiments, after executing to obtain the vector features output by the vector model for each text paragraph, the terminal processor 601 further specifically performs the following steps: establishing a vector index relationship between the vector features of each text paragraph and the long text to be parsed, and storing the vector index relationship in a retrieval database.

[0081] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0082] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the above process or function according to the embodiment of this specification is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above computer instructions can be stored in a computer-readable storage medium or transmitted through the above computer-readable storage medium. The above computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The above computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, digital versatile discs (DVD)), or semiconductor media (for example, solid state drives (SSD)), etc.

[0084] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0085] In addition, it should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the long texts and documents involved in this application are all obtained with full authorization.

[0086] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0088] The above is a description of a long text parsing method, device, storage medium and terminal provided in the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A long text parsing method, It is characterized in that The method comprises: Identify at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title; Dividing the long text to be parsed into a plurality of text paragraphs according to the input length requirements of each title and the vector model; Determine the target parent directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target parent directory corresponding to each text paragraph respectively; The integrated text paragraphs are input into the vector model to obtain vector features output by the vector model for each text paragraph.

2. The method according to claim 1, It is characterized in that The step of dividing the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model includes: Splitting the long text to be parsed into multiple text segments according to each title; Determine whether the length of each text segment meets the input length requirement of the vector model. If there is a target text segment that does not meet the input length requirement, split the target text segment into multiple text paragraphs that meet the input length requirement.

3. The method according to claim 2, It is characterized in that The step of splitting the long text to be parsed into multiple text segments according to each title includes: A title level that satisfies a segment separation condition is determined, and the long text to be parsed is split into a plurality of text segments according to each title belonging to the title level.

4. The method according to claim 2, It is characterized in that The step of splitting the target text segment into a plurality of text paragraphs meeting the input length requirement includes: Splitting the target text segment into multiple sentences according to preset text delimiters; The multiple sentences are spliced ​​according to a preset paragraph overlap length to obtain multiple text paragraphs that meet the input length requirement.

5. The method according to claim 1, It is characterized in that The directory information is a directory tree with a tree data structure, in which each title is used as a node and the level relationship between the titles is used as an edge connecting the nodes.

6. The method according to claim 1, It is characterized in that The step of respectively integrating each text paragraph and the target parent directory corresponding to each text paragraph includes: The context information of each text paragraph is obtained according to the target parent directory corresponding to each text paragraph, and the context information of each text paragraph is respectively integrated into the semantic information of each text paragraph.

7. The method according to claim 1, It is characterized in that After obtaining the vector features output by the vector model for each text paragraph, the method further includes: A vector index relationship between the vector features of each text paragraph and the long text to be parsed is established, and the vector index relationship is stored in a retrieval database.

8. A long text parsing device, It is characterized in that The device comprises: A title recognition module, used to recognize at least one title in the long text to be parsed, and generate directory information of the long text to be parsed based on each title; A paragraph splitting module, used to divide the long text to be parsed into multiple text paragraphs according to the input length requirements of each title and the vector model; A semantic integration module, used to determine the target upper-level directory corresponding to each text paragraph from the directory information, and integrate each text paragraph and the target upper-level directory corresponding to each text paragraph respectively; The vector generation module is used to input the integrated text paragraphs into the vector model to obtain the vector features output by the vector model for each text paragraph.

9. A computer storage medium, It is characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method as claimed in any one of claims 1 to 7.

10. A terminal, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

Images