Method and device for paragraph segmentation based on content awareness

Abstract

The invention discloses a paragraph segmentation method and device based on content awareness, and relates to the field of document processing. In the method, an original file in an original format is converted into a target file in a target format, and a text and a table are identified and separated from the target file; traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing paragraph contents according to the outline levels to form structured text data; traversing the table, extracting header data and data of each cell, and constructing key value pairs for the header data and the data of each cell to form table data; identifying a plain text from the text, segmenting the plain text into lexical units, and processing the lexical units by using a preset rule to form array data; and combining and outputting the text data, the table data and the array data. By implementing the technical scheme provided by the invention, the bid inviting word file is automatically and accurately segmented into logically independent paragraphs.

Description

Technical Field

[0001] The present application relates to the technical field of document processing, and in particular to a method and device for paragraph segmentation based on content awareness. Background Art

[0002] With the development of information technology, the application scope of paragraph segmentation technology has gradually expanded to many fields, such as electronic document management, automated office, etc., becoming one of the key technologies in data processing.

[0003] Existing paragraph segmentation techniques mainly identify line breaks in documents to mark paragraph boundaries and then achieve preliminary segmentation of content. Although processing power grid bidding documents docx files based on line break segmentation has some unique advantages, it also has some obvious shortcomings, including: (1) Format inconsistency: Line breaks may be represented differently in different operating systems or editors (such as \n, \r\n or \r), which may lead to inconsistent paragraph segmentation when opening documents in different environments; (2) Inability to distinguish between paragraphs and format line breaks: Line breaks may be used to separate paragraphs or to format line breaks within the text (such as poems, addresses, etc.). Segmentation based on line breaks cannot distinguish between these two situations; (3) Ignoring document structure: This method does not consider the hierarchy and style of the document, which may result in segmentation results that are inconsistent with the actual structure of the document.

[0004] Therefore, a method is needed to automatically and accurately divide a bidding word document into logically independent paragraphs. Summary of the invention

[0005] The present application provides a method and device for paragraph segmentation based on content awareness, which can automatically and accurately segment a bidding word document into logically independent paragraphs.

[0006] In a first aspect of the present application, a method for paragraph segmentation based on content awareness is provided, which is applied to a paragraph segmentation platform. The method comprises: converting a raw file in a raw format into a target file in a target format, identifying and separating text and tables from the target file; Traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing the paragraph contents according to the outline levels to form structured text data; Traversing the table, extracting table header data and each cell data, and constructing a key-value pair of the table header data and each cell data to form table data; Identifying plain text from the text, segmenting the plain text into word units, and processing the word units using preset rules to form array data; The text data, the table data and the array data are combined and then output.

[0007] By adopting the above technical solution, the original file is converted into the target format, the file format is standardized, and the subsequent data processing is facilitated. This conversion ensures that files from different sources can be processed in a unified format, improving processing efficiency and compatibility. Accurately identify and separate text and tables from the target file, so that subsequent data processing can be optimized for text and tables respectively, improving the pertinence and efficiency of data processing. Traverse the text to identify paragraphs and their corresponding outline levels, and splice the paragraph content according to the outline level. This process not only helps to understand the structure of the text, but also generates structured text data, which is convenient for subsequent indexing, retrieval and analysis. Identify plain text from the text, and perform word segmentation and preset rule processing. This process helps to extract key information from the text and form array data, providing a basis for subsequent text analysis, sentiment analysis, topic extraction and other tasks. Traverse the table, extract the header data and each cell data, and construct key-value pairs to form table data. This processing method makes the table data more structured, which is convenient for subsequent data analysis, report generation and other tasks. At the same time, the construction of key-value pairs also facilitates data storage and query. The processed text data, table data and array data are combined and output. This process integrates different types of data and provides users with a comprehensive and structured data view. Users can make more in-depth analysis and decisions based on this data.

[0008] Optionally, traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing the paragraph contents according to the outline levels includes: Determine a paragraph object of the text according to the line break character, and read text content from the paragraph object; Extracting a preset character string from the paragraph object, and obtaining an outline level corresponding to the paragraph object from the preset character string; The read text content is spliced ​​according to the outline level.

[0009] By adopting the above technical solution, the line breaks (such as paragraph marks, line breaks, etc.) in the text can be identified, and the text can be accurately divided into multiple paragraph objects. This method is simple and direct, and is applicable to most document formats, ensuring the accuracy and efficiency of paragraph recognition. Preset strings (such as title tags, outline level tags, etc.) are extracted from the paragraph objects. These preset strings are usually used to indicate the importance of the paragraph or the outline level to which it belongs. By identifying these strings, the outline level corresponding to each paragraph object can be quickly and accurately obtained. The outline level information is extracted from the preset string. This process realizes the conversion from text content to structured information. The clarity of the outline level makes subsequent data processing and analysis more orderly and efficient. According to the extracted outline level, the read text content is spliced. This process not only retains the integrity of the text content, but also organizes it in a structured manner according to the outline level. This structured organization method helps users better understand the text content, and also facilitates subsequent indexing, retrieval, analysis and other operations. Through paragraph recognition and outline level extraction, the structural framework of the document can be clearly presented. This structured presentation method makes the document easier to read and understand, and also facilitates users to quickly locate the content of interest. For documents containing a large number of paragraphs and complex structures, the workload of manual document organization can be greatly reduced through automated paragraph recognition and outline-level extraction.

[0010] Optionally, traversing the table, extracting the header data and each cell data, and constructing a key-value pair with the header data and each cell data includes: Obtaining a table object of the table according to the index of the table data tag, obtaining the data of the first row from the table object as table header data, and parsing the data except the first row in the table object to obtain each cell data; For each row, semantically combine the header data with the corresponding cell data to construct data in a key-value pair format; Integrate all rows of key-value pair data into a target list for output.

[0011] By adopting the above technical solution, the table object can be quickly located and obtained through the index of the table data tag. This method reduces the time of data search and improves the efficiency of data extraction. The header data and cell data are accurately distinguished from the table object. The header data is used as the key (Key) of the subsequent key-value pair, and the cell data is used as the value (Value). This separation provides a clear boundary for subsequent data processing. The header data is semantically combined with the corresponding cell data to construct data in the key-value pair format. This structured data representation method makes the data clearer and easier to understand, and also facilitates subsequent data processing, analysis and storage. The key-value pair format data of all rows are integrated into a target list output. This collection output method presents the data of the entire table in a unified and orderly form, which is convenient for users to view and operate as a whole. When constructing key-value pairs, flexible semantic combinations can be performed according to actual needs to adapt to different data processing scenarios. For example, cell data can be formatted, type converted, or additional metadata can be added. The data structure in the key-value pair format has good scalability, and new keys can be easily added or existing values ​​can be modified to adapt to data changes or new needs. The structured key-value pair data provides a basis for subsequent data visualization, and can easily generate visual displays such as charts and reports to help users better understand the information and trends behind the data.

[0012] Optionally, segmenting the plain text into word units includes: Identify preset punctuation marks, split the sentences before and after the preset punctuation marks, delete the preset punctuation marks and stop words, and retain the sentences before and after the decimal point. The preset punctuation marks include Chinese and English semicolons, Chinese and English periods, Chinese and English exclamation marks, and Chinese and English ellipsis marks.

[0013] By adopting the above technical solutions, preset punctuation marks such as Chinese and English semicolons, full stops, exclamation marks, ellipsis, etc. can be accurately recognized, enabling effective segmentation of the text. These punctuation marks play a crucial role in sentence division, and the accuracy of their recognition directly affects the quality of subsequent token segmentation. When processing text containing numbers, the sentences before and after the decimal point are specifically retained. This processing avoids text segmentation errors caused by the decimal point being misrecognized as the sentence end marker, ensuring the integrity of the numbers and their related information. Stop words refer to those words that frequently appear in the text but contribute little to the meaning of the text, such as "de" (的), "le" (了), "zai" (在), etc. By deleting these stop words, the redundancy of the text data can be reduced, improving the efficiency and accuracy of subsequent text processing (such as text classification, sentiment analysis, topic extraction, etc.). The accurate segmentation of preset punctuation marks and the deletion of stop words help reduce noise interference in the text, making the text data cleaner and more valuable. After splitting the plain text into tokens, more refined text analysis can be performed based on the tokens. For example, word frequencies can be counted, word co-occurrence relationships can be analyzed, and lexical networks can be constructed, etc., so as to deeply understand the content and structure of the text. Since the differences between Chinese and English punctuation marks are considered, this method can support cross-language text processing. This is of great significance for processing text data mixed with multiple languages. By deleting stop words and unnecessary punctuation marks, the readability of the text is optimized. For users who need to read a large amount of text, this improvement can significantly enhance the reading experience and efficiency. Precise text segmentation and token extraction help users obtain key information in the text faster, improving the efficiency and accuracy of information acquisition.

[0014] Optionally, the processing of the tokens using the preset rules includes: Adding a target word library for the power grid tendering Q&A scenario to the part-of-speech tagger, and assigning corresponding part-of-speech tags to each of the tokens according to the target word library, where the part-of-speech tags include nouns, verbs, and adjectives.

[0015] By adopting the above technical solution, adding a target word library for the power grid bidding question-answering scenario to the part-of-speech tagger, the system can more accurately identify the words and their parts of speech that are unique to this field. These words may not be common in daily language, but they appear frequently in power grid bidding questions and answers, such as "transformer", "tender", "bid", etc. The introduction of the target word library reduces the misjudgment rate of the part-of-speech tagger for these specific words and improves the accuracy of the entire part-of-speech tagging process. Accurate part-of-speech tagging helps the question-answering system better understand the user's questions. In the power grid bidding question-answering scenario, the system needs to accurately identify the user's intention, such as querying bidding information, submitting bidding documents, etc. The accuracy of part-of-speech tagging directly affects the system's ability to parse user questions. With the development of the power grid bidding field and the emergence of new words, the target word library can be easily updated and expanded. This flexibility enables the system to adapt to changes in the field and maintain its performance and accuracy. By introducing the target word library, the system reduces its dependence on the general part-of-speech tagger and reduces the maintenance cost caused by the update or change of the general part-of-speech tagger. Accurate part-of-speech tagging is the basis of many natural language processing tasks, such as syntactic analysis, semantic role tagging, information extraction, etc. In the power grid bidding question-answering scenario, these tasks are of great significance for deeply understanding user questions and extracting key information.

[0016] Optionally, the processing the word element using a preset rule further includes: Analyze the dependency syntactic structure in the text using a dependency syntactic parser to determine the semantic connection between sentence components, and combine the part-of-speech tags to concatenate the target noun and the context of the target noun to form a word unit; Named entity recognition is used to identify named entities in text and classify the named entities into predefined categories, including names of people, places, organizations, dates, times, currencies, and percentages.

[0017] By adopting the above technical solution, the dependency syntactic structure in the text is analyzed by the dependency syntactic parser, and the semantic connection between the sentence components, such as the subject-predicate, object-verb and other relationships, can be clarified. This analysis helps to understand the meaning of the sentence more deeply. Especially for complex sentences or long paragraphs, dependency syntactic parsing can reveal hidden semantic structures and provide strong support for subsequent text processing tasks. Combining part-of-speech tags and dependency syntactic structures, the target noun and its context are spliced ​​to form word units. This process is actually building a higher-level text representation. This representation not only contains the lexical information of the word unit, but also incorporates the semantic relationship and contextual information between word units, thereby improving the richness and accuracy of the text representation. Named entity recognition technology can automatically identify named entities in text, such as names, places, organizations, dates, times, currencies, percentages, etc., and classify them into predefined categories. This ability is crucial for information extraction tasks because it allows the system to quickly and accurately extract key information from the text, such as character information, geographic location, time events, etc. Through named entity recognition, the system can generate structured data representations that are not only easy to understand and use, but can also be used for various subsequent data processing and analysis tasks, such as data mining, information retrieval, question-answering systems, etc. The introduction of dependency parsing and named entity recognition technology enables the text processing system to automatically complete complex text analysis tasks, reducing the need for manual intervention and improving processing efficiency and accuracy. Dependency parsing and named entity recognition technology have a certain degree of versatility and can be applied to text processing tasks in different fields. At the same time, by adjusting the predefined categories and rules, the system can also adapt to the specific needs of different fields, thereby realizing cross-domain text processing.

[0018] Optionally, the processing the word element using a preset rule further includes: Processing the word-gram to obtain a sub-array consisting of a plurality of the word-grams, dividing the plurality of the sub-arrays into a plurality of blocks according to sentence start attributes of the word-grams, each of the blocks including at least one of the sub-arrays; The sub-arrays in each block are concatenated to form a target object, and an output array is constructed according to a plurality of the target objects.

[0019] By adopting the above technical solution, by organizing the word elements into sub-arrays and dividing them into blocks according to the sentence start attributes, this process establishes a clearer logical structure and hierarchical relationship for the text content. This structured organization makes the text information more orderly and easy to understand, which facilitates subsequent processing. Each block contains one or more sub-arrays, which are logically closely connected and together constitute an independent part of the text (such as a sentence, a paragraph or a logical unit). Block-level processing allows the system to perform more fine-grained analysis and operations on the text, improving the flexibility and accuracy of processing. The process of splicing the sub-arrays in the block into target objects is actually integrating and reconstructing the text information. Through splicing, the originally scattered word elements are organized into more meaningful text units, which are more complete and coherent in expression. Multiple target objects are further constructed into output arrays, which are convenient for subsequent processing and analysis. The output array not only retains the integrity and order of the text content, but also realizes the effective organization and presentation of text information through the division and splicing of target objects. The above processing process has certain scalability and can be adjusted and optimized according to different text processing requirements. For example, the division rules of sub-arrays, the division criteria of blocks, and the splicing methods of target objects can be adjusted to adapt to text processing tasks in different fields. Since the text is divided into multiple blocks, and the sub-arrays within the blocks can be processed in parallel, this helps to improve processing efficiency. At the same time, block-level processing also reduces the error propagation and cumulative effects in the processing process, thereby improving the accuracy of processing. In the process of constructing the output array, redundant information can be reduced through reasonable splicing and combination, making the output results more concise and clear. This helps the smooth progress of subsequent processing and analysis.

[0020] In a second aspect of the present application, a system for paragraph segmentation based on content awareness is provided, comprising a conversion module, a text module, a table module, a content module and an output module, wherein: a conversion module configured to convert an original file in an original format into a target file in a target format, and to identify and separate text and tables from the target file; A text module configured to traverse the text to identify paragraphs and outline levels corresponding to the paragraphs, and to splice the contents of the paragraphs according to the outline levels to form structured text data; A table module is configured to traverse the table, extract table header data and each cell data, and construct a key-value pair with the table header data and each cell data to form table data; A content module configured to identify plain text from the text, segment the plain text into word units, and process the word units using preset rules to form array data; An output module is configured to combine the text data, the table data and the array data and then output them.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By converting the original file into the target format and automatically identifying and separating text and tables, the need for manual intervention is reduced, and the efficiency of text processing is greatly improved. Paragraph content is spliced ​​according to the outline level, and data is extracted from the table to construct key-value pairs, generating structured text and table data for subsequent analysis and processing; 2. Through in-depth understanding and analysis of text content, paragraphs and their outline levels are identified, thus achieving more accurate and flexible paragraph segmentation. It can handle files of different formats and types, including nested paragraphs and tables in complex documents, and has strong adaptability and versatility; 3. Combine text data, table data and array data processed by preset rules for output, providing a diverse data source for subsequent text mining, information extraction, data analysis and other tasks. Through the careful processing of text and tables, the information in the original file can be fully retained and utilized, improving the utilization rate and value of information. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of a method for paragraph segmentation based on content awareness disclosed in an embodiment of the present application; Figure 2 is a schematic diagram of a natural language processing method for power grid bidding documents disclosed in an embodiment of the present application; Figure 3 It is a schematic diagram of the recognition effect before and after the word segmenter is adjusted according to the embodiment of the present application; Figure 4 It is a schematic diagram of the recognition effect before and after the part-of-speech tagger is adjusted according to the embodiment of the present application; Figure 5 is a schematic diagram of a doc array disclosed in an embodiment of the present application; Figure 6 It is a schematic diagram of the effect of paragraph segmentation disclosed in the embodiment of the present application; Figure 7 It is a module schematic diagram of a system for performing paragraph segmentation based on content awareness disclosed in an embodiment of the present application; Figure 8 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0025] Explanation of the accompanying drawings: 701, conversion module; 702, text module; 703, table module; 704, content module; 705, output module; 801, processor; 802, communication bus; 803, user interface; 804, network interface; 805, memory. DETAILED DESCRIPTION

[0026] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0028] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0029] This embodiment discloses a method for paragraph segmentation based on content perception, which is applied to a paragraph segmentation platform. Figure 1 is a flow chart of a method for paragraph segmentation based on content awareness disclosed in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: S110, converting an original file in an original format into a target file in a target format, and identifying and separating text and tables from the target file; The original format in the embodiment of the present application is docx format, and the target format is xml format. The .save() method of the python-docx library can be used to convert the docx format file into an xml format file. In the identification xml structure document, determine whether the current tag ends with "p", if so, it is a text paragraph data tag; determine whether the current tag ends with "tbl", if so, it is a table data tag. Therefore, text and tables can be identified and separated from the target file based on the tags.

[0030] S120, traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing the paragraph contents according to the outline levels to form structured text data; Identify independent paragraphs from the text. In most document formats, paragraphs are usually separated by specific characters (such as line breaks, paragraph marks, etc.). By traversing the text and segmenting it according to these separators, you can get a series of paragraphs. In many documents (especially Word documents), paragraphs may have different outline levels to indicate the hierarchy of the document structure. For example, Heading 1, Heading 2, Heading 3, etc. correspond to different outline levels, which are clearly displayed in the document outline view. Identifying the outline level corresponding to a paragraph usually involves parsing the document's style information or specific tags (such as style names in Word documents, <h1>、< / h1> <h2>Tags such as . Once paragraphs and their corresponding outline levels are identified, the paragraph contents can be organized or concatenated according to these outline levels to form structured text data. This usually means arranging paragraphs according to a hierarchy of outline levels, so that paragraphs with lower outline levels (such as headings) serve as parent nodes or containers for paragraphs with higher outline levels (such as body paragraphs). The way to construct structured text data depends on specific needs. In some cases, it may be necessary to convert these paragraph and outline level information into a tree structure (such as XML, JSON, or a specific object model), where each node represents a paragraph and contains the paragraph's text content, outline level, and possible child paragraphs (if any). In other cases, it may only be necessary to rearrange the paragraphs in the order of outline levels and insert chapter breaks or heading tags at appropriate locations.

[0031] Optionally, traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing the paragraph contents according to the outline levels includes: Determine a paragraph object of the text according to the line break character, and read text content from the paragraph object; Extracting a preset character string from the paragraph object, and obtaining an outline level corresponding to the paragraph object from the preset character string; The read text content is spliced ​​according to the outline level.

[0032] Traverse the entire text content and identify different paragraphs in the text by identifying line breaks (such as \n). Every time a line break is encountered, it means the beginning of a new paragraph, thus dividing the text into multiple paragraph objects. For each paragraph object, the program reads the text content it contains. This text content contains specific information about the paragraph, such as text descriptions, titles, etc. While reading the paragraph text content, the program also checks whether the paragraph contains a specific preset string indicating the outline level (such as <w:outlinelvl w:val="number" / > ). If such a string exists, the program will extract the value of number from it, which represents the outline level of the current paragraph. According to the preset rules, different numerical values ​​correspond to different outline levels (such as 0 for the first-level title, 1 for the second-level title, 2 for the third-level title, and none or the absence of the string represents ordinary text). After determining all paragraphs and their corresponding outline levels, the program will splice the paragraph contents according to the order of outline levels. The general rule is to first splice the title paragraphs in the ascending order of outline levels (from small to large) to ensure the logic of the structure. When encountering a new title with a higher outline level, start a new combination and continue to splice subsequent paragraphs in this combination in the order of outline levels until a title with a higher outline level is encountered or the text ends. Ordinary text (i.e., a paragraph without an outline level) is usually appended to the end of the combination of its previous title paragraph. Generate one or more structured text combinations according to the above rules and output these combinations as results.

[0033] For example, the parsed paragraph outline is as follows: 0: A 1: B 2: C none: normal 2: D none: text 2: E The output result is ["ABC Normal", "ABD Text", "ABE"], which means that the text is successfully spliced ​​according to the outline level and the relationship between paragraphs is correctly organized.

[0034] By identifying line breaks, the text is divided into paragraphs, and the outline level of each paragraph is further extracted, so that the preliminary structuring of the text is achieved. The paragraph content is spliced ​​according to the outline level, so that the logical structure and content level of the text are clearer. This organization method helps readers quickly grasp the main points and key points of the text, improve reading efficiency and information absorption ability. The structured text data is easier to be processed and analyzed by computer programs. For example, in a document management system, a content analysis platform, or an automated report generation system, outline level information can be used to automatically extract key information, generate directories, optimize layouts, etc. In a document management scenario, organizing and classifying documents according to outline levels can greatly improve the efficiency of document retrieval, editing, and archiving. Users can quickly locate specific parts in the document as needed to read or modify them. Compared with traditional manual sorting and classification methods, the embodiments of the present application reduce the need for manual intervention, reduce the possibility of human errors, and improve the accuracy and consistency of text processing.

[0035] S130, traversing the table, extracting table header data and each cell data, and constructing a key-value pair with the table header data and each cell data to form table data; The header data is usually located in the first row (or first column, but more commonly the first row) of the table, and contains the names or identifiers of each column (or row, but columns are used as an example here). When traversing to the header row, extract the text or value of each cell as the key of the subsequent key-value pair. In addition to the header row, each cell in the table contains certain data. In the process of traversing the table, you need to extract the data of these cells one by one, and these data will be used as the value of the key-value pair. A key-value pair is a structure that links two related data items together, where one data item is used as a key to uniquely identify the other data item (value). In the context of tabular data processing, the key is usually the header data (column name) and the value is the data of each cell in the corresponding column. For each row in the table (except the header row), traverse all its cells. For each cell, use the header data (column name) of its column as the key and the data in the cell as the value to construct a key-value pair. Then, add this key-value pair to a collection (such as a list, a list of dictionaries, a Pandas DataFrame, etc.) for subsequent use. Through the above process, we will eventually get a set containing multiple key-value pairs. These key-value pairs are organized according to the original structure of the table to form a structured table data set.

[0036] Optionally, traversing the table, extracting the header data and each cell data, and constructing a key-value pair with the header data and each cell data includes: Obtaining a table object of the table according to the index of the table data tag, obtaining the data of the first row from the table object as table header data, and parsing the data except the first row in the table object to obtain each cell data; For each row, semantically combine the header data with the corresponding cell data to construct data in a key-value pair format; Integrate all rows of key-value pair data into a target list for output.

[0037] In the XML structure document, tags ending with "tbl" are identified by traversing the tags, which are considered as table data tags. When table data tags are identified, the corresponding table objects are located in the document according to the indexes (or position information) of these tags. This table object contains the data structure of the entire table, including headers, rows, and cells. From the table object, in particular, the header data is extracted from the first row of data. The header data usually contains the name or identifier of each column of data, which is crucial for subsequent data parsing and semantic combination. Next, iterate each row in the table object (except the first row, because the first row is the header), and parse each cell in each row to obtain its value. These values ​​will be used to match and combine with the header data. For each row in the table (starting from the second row), the header data is semantically combined with the data of each cell in the row. This combination usually takes the header name as the key (Key) and the corresponding cell value as the value (Value), thereby constructing data in the key-value pair format. For example, if the table headers have "column name 1" and "column name 2", and the cell values ​​of a row are "value 1" and "value 2", the key-value pairs constructed are {"column name 1": "value 1", "column name 2": "value 2"}. Integrate the key-value pair format data of all rows into a list (or other suitable data structure) for output. Each element in this list is a dictionary, representing a row of data in the table. The key of the dictionary is the table header name, and the value is the corresponding cell data. The final output format is as follows: [{column name 1: first row column value 1, column name 2: first row column value 2...}, {Column name 1: second row column value 1, column name 2: second row column value 2…}].

[0038] By traversing the table, extracting the header data and each cell data, and constructing them into a key-value pair format, the structured processing of the table data is achieved. This structured processing makes the information originally scattered in the table orderly, easy to understand and operate, which facilitates the subsequent data analysis, processing or storage. Combining the header data and the cell data into a key-value pair format makes each row of data clearly represent the name and corresponding value of each field. This representation method improves the readability of the data, allowing users to intuitively understand the meaning of each row of data. The structured data is easier to be processed by computer programs. For example, structures such as loops and conditional judgments in programming languages ​​can be used to traverse, filter, sort or modify data. In addition, structured data is also easier to interact with storage systems such as databases and data warehouses to achieve persistent storage and fast retrieval of data. Traditional table data processing often requires a lot of manual intervention, such as manual data extraction and formatting. By automatically traversing the table, extracting data and constructing a key-value pair format, the need for manual intervention can be greatly reduced, and the efficiency and accuracy of data processing can be improved.

[0039] S140, identifying plain text from the text, dividing the plain text into word units, and processing the word units using preset rules to form array data; Identify the plain text content from a given text. This process may involve removing non-text elements from the text, such as HTML tags, XML tags, comments, special characters (such as tabs, line breaks, etc., unless these characters have special uses in subsequent processing), metadata tags, etc. Plain text usually refers to text content without these tags and formats, and only contains human-readable language text. After identifying the plain text, the next step is to split the text into smaller units, namely tokens. Tokens can be words, phrases, punctuation marks, or other meaningful text fragments. This process is usually called tokenization or lemmatization. The tokenization method depends on the language and context used, and may involve splitting the text according to spaces, punctuation marks, specific separators, or algorithmic rules. After tokenization, the resulting token set needs to be further processed by preset rules to meet specific requirements or formats. Finally, the processed tokens are organized into array data. An array is a data structure used to store data of the same type continuously in computer memory. In this context, each element of the array is a processed token.

[0040] Figure 2 is a schematic diagram of a natural language processing method for power grid bidding documents disclosed in an embodiment of the present application, such as Figure 2 As shown, Text can be processed by nlp (natural language processing method), specifically, it can be preliminarily processed by tokenizer, and then processed by tagger (part-of-speech tagger), parser (dependency syntax parser) and ner (named entity recognition) in sequence. The present application embodiment uses spaCy library (a natural language processing (NLP) library), loads zh_core_web_lg-3.7.0 (Chinese processing language model), and creates a natural language processing method nlp for power grid bidding documents, as follows: nlp = spacy.load("zh_core_web_lg-3.7.0").

[0041] Optionally, segmenting the plain text into word units includes: Identify preset punctuation marks, split the sentences before and after the preset punctuation marks, delete the preset punctuation marks and stop words, and retain the sentences before and after the decimal point. The preset punctuation marks include Chinese and English semicolons, Chinese and English periods, Chinese and English exclamation marks, and Chinese and English ellipsis marks.

[0042] The embodiment of the present application optimizes and adjusts the word segmentation rules of the word segmenter. Figure 3 is a schematic diagram of the recognition effect before and after the word segmenter is adjusted according to the embodiment of the present application. Figure 3 As shown, the decimal point (.) is optimized to recognize its role as part of a number, rather than a simple separator. Therefore, when processing numbers containing a decimal point (such as "0.47 billion"), the decimal point and the numbers around it are retained as a whole, rather than being separated. This helps maintain the integrity and accuracy of the digital information.

[0043] In the embodiments of the present application, the preset punctuation marks are explicitly defined to include Chinese and English semicolons (;), Chinese and English full stops (.), Chinese and English exclamation marks (!), and Chinese and English ellipses (……). These punctuation marks are regarded as important boundaries for text segmentation because they often indicate the end of a sentence or paragraph or a longer pause. When any of the above-mentioned preset punctuation marks is recognized, the program will regard it as a separator and split the text before and after the punctuation mark into independent units (i.e., sentences or phrases). This step helps to perform more detailed analysis on the text later, such as sentiment analysis, theme extraction, etc. While splitting the text, these preset punctuation marks used as separators will be deleted. This is to ensure that subsequent processing (such as word frequency statistics, text classification, etc.) will not be interfered by these punctuation marks. Stop words (such as "de", "le", "shi", etc.) are those words that appear frequently in the text but contribute little to the theme or meaning of the text. At the final stage of text segmentation and preprocessing, these stop words will be deleted to reduce noise and redundant information in the subsequent processing. The noun_chunks attribute of the spaCy library is used to detect noun phrases and may involve splicing of key nouns and their contexts (such as adjectives, verbs, etc.). This is actually a processing step after text segmentation, used to extract key information in the text or construct a more complex text representation By identifying the preset punctuation marks and splitting the sentences or phrases before and after these punctuation marks, the structure of the text can be divided more precisely. While splitting the text, the decimal point and the numbers around it are retained, avoiding incorrect splitting of numerical information. This is particularly important for processing texts containing numerical information such as financial and scientific data, ensuring the integrity and accuracy of numerical information. By deleting the preset punctuation marks and stop words, the noise and redundant information in the text can be reduced. Stop words are those high-frequency words that contribute little to the theme or meaning of the text. Deleting them can reduce the complexity of text processing and the computational workload in subsequent tasks. After the above preprocessing steps, the text becomes clearer and more concise, which is beneficial for subsequent natural language processing tasks. For example, in tasks such as text classification and information extraction, the preprocessed text is easier to be understood and processed by the model. Since the preset punctuation marks include Chinese and English semicolons, full stops, exclamation marks and ellipses, this technical effect is not only applicable to Chinese texts, but also to English texts. This helps to process text data in a multilingual environment and improve the generality and practicality of the system. By optimizing the text segmentation rules and improving the accuracy and efficiency of text processing, the performance of the entire text processing system or application can be indirectly improved. This includes aspects such as reducing processing time, increasing accuracy, and reducing resource consumption.

[0044] Optionally, the processing of the token using the preset rule includes: A target word library for the power grid bidding question-answering scenario is added to the part-of-speech tagger, and a corresponding part-of-speech tag is assigned to each word element according to the target word library, wherein the part-of-speech tags include nouns, verbs and adjectives.

[0045] According to the actual needs of the power grid bidding question and answer, a series of proper nouns and terms closely related to the scenario are collected and sorted out, such as "bidding company", "bidder", "tenderer", etc. These words appear frequently in the power grid bidding question and answer, and have specific meanings and usages. The constructed target vocabulary is integrated into the part-of-speech tagger. This usually involves modifying the internal dictionary or configuration file of the tagger (part-of-speech tagger) to ensure that these specific words can be recognized and correctly tagged when processing text. In part-of-speech tagging, part-of-speech tags are used to identify the grammatical role of words in sentences. Common part-of-speech tags include nouns (NOUN), verbs (VERB), adjectives (ADJ), etc. For the power grid bidding question and answer scenario, special attention is paid to noun tags, especially proper nouns (PROPN), because they can accurately represent noun phrases with unique meanings. When the tagger processes the text, it traverses each token (i.e., word or phrase) in the text and tries to match it with a vocabulary in the vocabulary. If the token matches a vocabulary in the target vocabulary, it is annotated with the corresponding part-of-speech tag according to the definition of the vocabulary in the vocabulary. In the embodiment of the present application, words such as "bidding company", "bidder", and "tenderer" will be marked as proper nouns (PROPN). By increasing the target vocabulary and adjusting the part-of-speech tagging rules, the tagger can more accurately identify and mark the proper nouns and terms in the power grid bidding question and answer text. This helps subsequent natural language processing tasks (such as information extraction, question-answering systems, etc.) to better understand the text content and improve the overall processing effect and efficiency. Figure 4 is a schematic diagram of the recognition effect before and after the part-of-speech tagger is adjusted according to the embodiment of the present application, such as Figure 4 As shown, before the adjustment, "Bidding Company" will be identified as two words, and after the adjustment, it will be identified as one word.

[0046] By adding a target vocabulary for the power grid bidding question-and-answer scenario to the part-of-speech tagger, the system can more accurately identify and tag proper nouns and terms in the text. These proper nouns and terms have important semantic information in the power grid bidding question-and-answer scenario. Accurately tagging them helps subsequent natural language processing tasks (such as information extraction, question-and-answer systems, etc.) better understand the text content. Traditional part-of-speech taggers may not be well adapted to text processing needs in specific fields or scenarios. By adding a target vocabulary and adjusting the part-of-speech tagging rules based on the vocabulary, the part-of-speech tagger can be made more adaptable to the specific scenario of power grid bidding question-and-answer. This improved adaptability helps to process more diverse text data and improve the generalization ability of the system. Accurate part-of-speech tagging is the basis of many text processing tasks, such as syntactic analysis and semantic role labeling. By improving the accuracy of part-of-speech tagging, errors and redundancy in subsequent processing can be reduced, thereby improving the efficiency of the entire text processing process. In the power grid bidding question-and-answer scenario, efficient text processing can extract key information faster and meet users' real-time needs. For users of the power grid bidding question-answering system, accurate part-of-speech tagging means that the system can more accurately understand the user's query intent and return more relevant and accurate answers. This improved user experience helps to increase user satisfaction and trust in the system, thereby promoting the widespread application and promotion of the system.

[0047] Optionally, the processing the word element using a preset rule further includes: Analyze the dependency syntactic structure in the text using a dependency syntactic parser to determine the semantic connection between sentence components, and combine the part-of-speech tags to concatenate the target noun and the context of the target noun to form a word unit; Named entity recognition is used to identify named entities in text and classify the named entities into predefined categories, including names of people, places, organizations, dates, times, currencies, and percentages.

[0048] In text processing tasks, the dependency syntax parser plays a key role. It can analyze the dependency syntax structure in the text and reveal the semantic connection between the sentence components. In order to better meet the needs of specific tasks (such as power grid bidding questions and answers), the embodiment of the present application adjusts the parser and combines the part-of-speech tag and noun_chunks attribute to optimize the processing process of the word unit. First, the text is parsed using the dependency syntax parser to identify the subject, predicate, object and other sentence components in the sentence, and determine the dependency relationship between them. This process helps to understand the semantic structure inside the sentence. On the basis of dependency syntax analysis, the nature of the sentence components is further analyzed by combining part-of-speech tags (such as nouns, verbs, adjectives, etc.). Part-of-speech tags provide important information about the function and role of word units in sentences. The noun_chunks attribute in tokens is used to detect noun phrases in the text. Noun_chunks usually contains one or more nouns and related modifying components (such as adjectives, determiners, etc.), which together constitute a meaningful semantic unit. After a noun phrase is detected, the text before and after the noun phrase is further retrieved, especially chunks of parts of speech such as adjectives and verbs related to it. These chunks may provide important contextual information about the noun phrase. Then, according to preset rules, these related chunks are concatenated with the noun phrase to form a new token with richer semantic information. Chunk in the above context means "a lot, most of", especially when describing text data or information. For example, when processing a large amount of text data, it may be divided into smaller chunks for easy processing block by block. In addition to dependency parsing and context concatenation, named entity recognition (NER) technology is also used to identify named entities in the text. NER can identify entities with specific meanings in the text, such as names, places, organizations, dates, times, currencies, percentages, etc. The identified named entities are classified into predefined categories. These categories are defined according to task requirements, which helps subsequent processing tasks (such as information extraction, question-answering systems, etc.) to better understand and utilize these entity information.

[0049] Using a dependency parser to analyze the dependency syntactic structure in a text can reveal the semantic connections between sentence components. This understanding of semantic connections is crucial for subsequent natural language processing tasks because it helps the system grasp the logical relationships and information structure in the text more accurately. The target noun and its context are concatenated in combination with part-of-speech tags to form more meaningful tokens, further enhancing the system's ability to understand the text semantics. This concatenation operation not only retains the information of the noun itself, but also incorporates the semantic information of its surroundings, making the processing results more comprehensive and accurate. The application of named entity recognition technology enables the system to automatically identify named entities in the text, such as names, places, organizations, etc. These named entities are often key information points in the text and are of great significance for tasks such as information extraction and question-answering systems. Classifying named entities into predefined categories not only facilitates the organization and use of information in subsequent processing tasks, but also improves the efficiency of information extraction. The system can quickly locate and extract named entities of a specific category, providing users with more accurate and useful information. Accurate dependency syntactic analysis and named entity recognition provide a solid foundation for more complex natural language processing tasks. For example, in tasks such as sentiment analysis, text summarization, and machine translation, the system needs to deeply understand the semantic and structural information in the text. Through the optimization of the above technical means, the system can better handle these complex tasks and improve the overall natural language processing level.

[0050] Optionally, the processing the word element using a preset rule further includes: Processing the word-gram to obtain a sub-array consisting of a plurality of the word-grams, dividing the plurality of the sub-arrays into a plurality of blocks according to sentence start attributes of the word-grams, each of the blocks including at least one of the sub-arrays; The sub-arrays in each block are concatenated to form a target object, and an output array is constructed according to a plurality of the target objects.

[0051] The plain text data is processed by the NLP method, and the text is divided into multiple tokens. Each token may contain a variety of information, such as text content (.text attribute), part-of-speech tag, sentence start attribute (sent_start), etc. Figure 5 is a schematic diagram of the doc array disclosed in the embodiment of this application, such as Figure 5 As shown in the figure, the text is divided into multiple tokens, each of which represents a unit in the text (such as a word, punctuation mark, etc.). According to the sentence start attribute (sent_start) of the token, all tokens are divided into multiple subarrays. Here, each subarray contains all tokens from the beginning of a sentence to the end of the sentence. Specifically, if the sent_start attribute of a token is True, it means that the token is the first token of the sentence, and it and all tokens before the next token whose sent_start is True are divided into the same subarray. For each block (that is, each sentence composed of subarrays), all subarrays in the block are concatenated. Since only the use of the .text attribute is mentioned here, the concatenation operation is actually to connect the .text attributes of the tokens in each subarray to form a complete sentence text. The subarrays in each block are concatenated to form a target object (span), which contains a collection of tokens, part-of-speech tags, dependencies, etc. The span (here is a string or text block) formed by concatenating each block (that is, each sentence) is added to the output array (doc array). Ultimately, the doc array will contain the concatenation of all the sentences in the text, with each element corresponding to the text content of a sentence.

[0052] Through the sentence start attribute of the word unit (such as sent_start in spaCy), the system can automatically identify the sentence boundaries in the text. This automatic identification reduces the need for manual intervention and improves the automation of text processing. The text is divided into multiple blocks according to sentences (each block represents a sentence), and each block is further organized into sub-arrays (each sub-array contains the word units in the sentence). This structured processing makes the text information more orderly and easy to manage, which facilitates subsequent analysis or processing. Since the text is divided into multiple independent sentence blocks, the system can process these blocks in parallel or one by one without processing the entire text at one time. This processing method reduces redundant calculations and improves processing efficiency. Processing text at the sentence level can more accurately extract key information in the sentence, such as subject, predicate, object, etc. This accuracy improvement is particularly important for tasks such as information extraction, sentiment analysis, and question-answering systems. Structured text blocks enable the system to support more complex text analysis tasks, such as dependency syntactic analysis, semantic role labeling, etc. These advanced analysis tasks require accurate division and organization of text at the sentence level. Dividing text into sentence blocks and displaying them in a structured form can enhance the readability of the text. Users or systems can more easily understand the structure and content of the text, allowing for more effective interaction or analysis. Structured text blocks also support custom display methods, such as displaying text content in chronological order, by subject classification, etc. This flexibility makes text processing results more in line with user needs or application scenarios.

[0053] S150, combining the text data, the table data and the array data and outputting the combined data.

[0054] All the extracted text data, tables, and plain text array data are combined and output in the form of [ "1 General Principles: 1.2 Sources and implementation of funds for bidding projects: 1.2.1 Sources and proportions of funds: See the table in front of the Bidder's Instructions.\n 1.2.2 Funding status: see the table in front of the Bidder's Instructions"------fixed structure format, "Time: June 2024, Tenderer: Jiangsu Bingxin Technology"------Table format, "Bidding Company Analysis Project Proposal" ---- Data in plain text format ………………………… ] Figure 6 Schematic diagram of the effect of paragraph segmentation disclosed in the embodiment of the present application, such as Figure 6 As shown, the original text is divided into multiple fragments.

[0055] This embodiment also discloses a system for performing paragraph segmentation based on content perception. Figure 7 is a schematic diagram of a module of a system for performing paragraph segmentation based on content perception disclosed in an embodiment of the present application, such as Figure 7 As shown, the system includes a conversion module 701, a text module 702, a table module 703, a content module 704 and an output module 705, wherein: A conversion module 701 is configured to convert an original file in an original format into a target file in a target format, and identify and separate text and tables from the target file; A text module 702 is configured to traverse the text to identify paragraphs and outline levels corresponding to the paragraphs, and to splice the paragraph contents according to the outline levels to form structured text data; The table module 703 is configured to traverse the table, extract the header data and each cell data, and construct a key-value pair with the header data and each cell data to form table data; A content module 704 is configured to identify plain text from the text, segment the plain text into word units, and process the word units using a preset rule to form array data; The output module 705 is configured to combine the text data, the table data and the array data and then output them.

[0056] Optionally, the text module 702 is configured to: Determine a paragraph object of the text according to the line break character, and read text content from the paragraph object; Extracting a preset character string from the paragraph object, and obtaining an outline level corresponding to the paragraph object from the preset character string; The read text content is spliced ​​according to the outline level.

[0057] Optionally, the table module 703 is configured to: Obtaining a table object of the table according to the index of the table data tag, obtaining the data of the first row from the table object as table header data, and parsing the data except the first row in the table object to obtain each cell data; For each row, semantically combine the header data with the corresponding cell data to construct data in a key-value pair format; Integrate all rows of key-value pair data into a target list for output.

[0058] Optionally, the content module 704 is configured to: Identify preset punctuation marks, split the sentences before and after the preset punctuation marks, delete the preset punctuation marks and stop words, and retain the sentences before and after the decimal point. The preset punctuation marks include Chinese and English semicolons, Chinese and English periods, Chinese and English exclamation marks, and Chinese and English ellipsis marks.

[0059] Optionally, the content module 704 is configured to: A target word library for the power grid bidding question-answering scenario is added to the part-of-speech tagger, and a corresponding part-of-speech tag is assigned to each word element according to the target word library, wherein the part-of-speech tags include nouns, verbs and adjectives.

[0060] Optionally, the content module 704 is configured to: Analyze the dependency syntactic structure in the text using a dependency syntactic parser to determine the semantic connection between sentence components, and combine the part-of-speech tags to concatenate the target noun and the context of the target noun to form a word unit; Named entity recognition is used to identify named entities in text and classify the named entities into predefined categories, including names of people, places, organizations, dates, times, currencies, and percentages.

[0061] Optionally, the content module 704 is configured to: Processing the word-gram to obtain a sub-array consisting of a plurality of the word-grams, dividing the plurality of the sub-arrays into a plurality of blocks according to sentence start attributes of the word-grams, each of the blocks including at least one of the sub-arrays; The sub-arrays in each block are concatenated to form a target object, and an output array is constructed according to a plurality of the target objects.

[0062] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0063] This embodiment also discloses an electronic device, referring to Figure 8 The electronic device may include: at least one processor 801 , at least one communication bus 802 , a user interface 803 , a network interface 804 , and at least one memory 805 .

[0064] The communication bus 802 is used to realize the connection and communication between these components.

[0065] The user interface 803 may include a display screen (Display) and a camera (Camera). The optional user interface 803 may also include a standard wired interface and a wireless interface.

[0066] The network interface 804 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0067] Among them, the processor 801 may include one or more processing cores. The processor 801 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 805, and calling data stored in the memory 805. Optionally, the processor 801 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 801 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 801, and it can be implemented separately through a chip.

[0068] Among them, the memory 805 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 805 includes a non-transitory computer-readable storage medium. The memory 805 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 805 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 805 may also be optionally at least one storage device located away from the aforementioned processor 801. As Figure 8 As shown, the memory 805 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a method for performing paragraph segmentation based on content awareness.

[0069] exist Figure 8 In the electronic device shown, the user interface 803 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 801 can be used to call the application program for the method of paragraph segmentation based on content perception stored in the memory 805. When executed by one or more processors 801, the electronic device executes one or more methods in the above-mentioned embodiments.

[0070] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0071] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0073] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0075] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory 805, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory 805 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0076] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.< / h2>

Claims

1. A method for paragraph segmentation based on content awareness, characterized in that: Applied to a paragraph segmentation platform, the method comprises: converting a raw file in a raw format into a target file in a target format, identifying and separating text and tables from the target file; Traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing the paragraph contents according to the outline levels to form structured text data; Traversing the table, extracting table header data and each cell data, and constructing a key-value pair of the table header data and each cell data to form table data; Identifying plain text from the text, segmenting the plain text into word units, and processing the word units using preset rules to form array data; The text data, the table data and the array data are combined and then output.

2. The method for paragraph segmentation based on content awareness according to claim 1, characterized in that: The traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing the paragraph contents according to the outline levels comprises: Determine a paragraph object of the text according to the line break character, and read text content from the paragraph object; Extracting a preset character string from the paragraph object, and obtaining an outline level corresponding to the paragraph object from the preset character string; The read text content is spliced ​​according to the outline level.

3. The method for paragraph segmentation based on content awareness according to claim 1, characterized in that: The traversing the table, extracting the header data and each cell data, and constructing a key-value pair with the header data and each cell data comprises: Obtaining a table object of the table according to the index of the table data tag, obtaining the data of the first row from the table object as table header data, and parsing the data except the first row in the table object to obtain each cell data; For each row, semantically combine the header data with the corresponding cell data to construct data in a key-value pair format; Integrate all rows of key-value pair data into a target list for output.

4. The method for paragraph segmentation based on content awareness according to claim 1, characterized in that: The segmenting of the plain text into word units comprises: Identify preset punctuation marks, split the sentences before and after the preset punctuation marks, delete the preset punctuation marks and stop words, and retain the sentences before and after the decimal point. The preset punctuation marks include Chinese and English semicolons, Chinese and English periods, Chinese and English exclamation marks, and Chinese and English ellipsis marks.

5. The method for paragraph segmentation based on content awareness according to claim 1, characterized in that: The processing of the word element using the preset rule includes: A target word library for the power grid bidding question-answering scenario is added to the part-of-speech tagger, and a corresponding part-of-speech tag is assigned to each word element according to the target word library, wherein the part-of-speech tags include nouns, verbs and adjectives.

6. The method for paragraph segmentation based on content awareness according to claim 5, characterized in that: The processing of the word element using the preset rule also includes: Analyze the dependency syntactic structure in the text using a dependency syntactic parser to determine the semantic connection between sentence components, and combine the part-of-speech tags to concatenate the target noun and the context of the target noun to form a word unit; Named entity recognition is used to identify named entities in text and classify the named entities into predefined categories, including names of people, places, organizations, dates, times, currencies, and percentages.

7. The method for paragraph segmentation based on content awareness according to claim 6, characterized in that: The processing of the word element using the preset rule also includes: Processing the word-gram to obtain a sub-array consisting of a plurality of the word-grams, dividing the plurality of the sub-arrays into a plurality of blocks according to sentence start attributes of the word-grams, each of the blocks including at least one of the sub-arrays; The sub-arrays in each block are concatenated to form a target object, and an output array is constructed according to a plurality of the target objects.

8. A system for paragraph segmentation based on content awareness, characterized in that: It includes conversion module, text module, table module, content module and output module, among which: a conversion module configured to convert an original file in an original format into a target file in a target format, and to identify and separate text and tables from the target file; A text module configured to traverse the text to identify paragraphs and outline levels corresponding to the paragraphs, and to splice the contents of the paragraphs according to the outline levels to form structured text data; A table module is configured to traverse the table, extract table header data and each cell data, and construct a key-value pair with the table header data and each cell data to form table data; A content module configured to identify plain text from the text, segment the plain text into word units, and process the word units using preset rules to form array data; An output module is configured to combine the text data, the table data and the array data and then output them.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images