Text extraction method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a text extraction method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that a target identifier is retrieved in a first to-be-processed character string of a streaming text, and the target identifier is a character used for representing the format type of the first to-be-processed character string; after the target identifier is retrieved, a label is retrieved in the first to-be-processed character string according to the format type by taking the target identifier as an initial position, and the label is used for identifying a starting point or an end point of the character string belonging to the format type; and extracting a target character string from the first to-be-processed character string according to the identifier, and storing the target character string in a storage space corresponding to the format type, the target character string being a character string between the tags. According to the method and the device, the technical problem that the accuracy of segmentation extraction is relatively low due to the fact that specific types of various texts cannot be accurately distinguished in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing, and more specifically, to a text extraction method, apparatus, electronic device, and non-volatile storage medium. Background Technology

[0002] In related technologies, when classifying and extracting text with multiple format types, it is impossible to accurately distinguish the specific types of each type of text, resulting in a low accuracy rate of segmentation and extraction.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a text extraction method, apparatus, electronic device, and non-volatile storage medium to at least solve the technical problem of low accuracy in segmentation and extraction caused by the inability to accurately distinguish the specific types of various texts in related technologies.

[0005] According to one aspect of the embodiments of this application, a text extraction method is provided, comprising: retrieving a target identifier in a first string to be processed of a stream of text, wherein the target identifier is a character used to represent the format type of the first string to be processed; after retrieving the target identifier, retrieving tags in the first string to be processed based on the format type, with the target identifier as the starting position, wherein the tags are used to identify the start or end point of the string belonging to the format type; extracting a target string from the first string to be processed based on the identifier, and storing the target string in the storage space corresponding to the format type, wherein the target string is the string between tags.

[0006] Optionally, retrieving tags from the first string to be processed, starting from the target identifier and according to the format type, includes: establishing a tag stack; recording the retrieved unclosed tags in the tag stack, wherein the unclosed tags include starting tags for which no corresponding ending tags were found, and ending tags for which no corresponding starting tags were found.

[0007] Optionally, storing the target string in the storage space corresponding to the format type includes: after determining the start label or end label corresponding to the unclosed label, determining that the unclosed label and its corresponding start label or end label form a closed label pair, and popping the label at the top of the stack; if it is determined that all labels in the label stack have been popped, determining that the target string is a closed complete statement, and storing the target string in the storage space corresponding to the format type.

[0008] Optionally, after retrieving the target identifier, starting from the target identifier, the tag is retrieved in the first string to be processed according to the format type, including: if a start tag for marking the start point is retrieved, but no end tag for marking the end point is retrieved, the target string is determined to be a multi-line statement, and the start tag is stored; after receiving the second string to be processed, the end tag is retrieved in the second string to be processed, and a semantic coherence judgment is performed on the second string to be processed. Based on the judgment result of the semantic coherence judgment, it is determined whether the characters before the end tag in the second string to be processed are characters in the same statement as the characters after the start tag in the first string to be processed, wherein the generation time of the second string to be processed is later than the generation time of the first string to be processed.

[0009] Optionally, extracting the target string from the first string to be processed based on the identifier and storing the target string in the storage space corresponding to the format type includes: determining a feature matching value based on the target identifier, wherein if the format type indicated by the target identifier is a preset format type, the feature matching value is determined to be a first value, otherwise a second value, and the first value is greater than the second value; determining a syntax closure value based on the tag detection result, wherein if the detected tags include a start tag for marking the start point and an end tag for marking the end point, the syntax closure value is determined to be a third value, otherwise a fourth value, and the third value is greater than the fourth value; determining a comprehensive evaluation score based on the feature matching value and the syntax closure value, and if the comprehensive evaluation score is not less than a preset threshold, extracting the target string from the first string to be processed based on the identifier and storing the target string in the storage space corresponding to the format type; if the comprehensive evaluation score is less than the preset threshold, retaining the first string to be processed in the cache, and after receiving the third string to be processed, retrieving the target identifier or tag again, wherein the generation time of the third string to be processed is later than the generation time of the first string to be processed.

[0010] Optionally, determining the comprehensive evaluation score based on the feature matching value and the grammatical closure value includes: determining the first weight coefficient corresponding to the feature matching value and the second weight coefficient corresponding to the grammatical closure value; determining the first product between the feature matching value and the first weight coefficient, and the second product between the grammatical closure value and the second weight coefficient; and determining the sum of the first product and the second product as the comprehensive evaluation score.

[0011] Optionally, after storing the target string in the storage space corresponding to the format type, the method further includes: determining the rendering method corresponding to the storage space; rendering the target string using the rendering method; and displaying the rendered target string.

[0012] According to another aspect of the embodiments of this application, a text extraction apparatus is also provided, comprising: a first processing module, configured to retrieve a target identifier in a first string to be processed of a stream of text, wherein the target identifier is a character used to represent the format type of the first string to be processed; a second processing module, configured to, after retrieving the target identifier, retrieve tags in the first string to be processed based on the format type, with the target identifier as the starting position, wherein the tags are used to identify the start or end point of a string belonging to the format type; and a third processing module, configured to extract a target string from the first string to be processed based on the identifier and store the target string in a storage space corresponding to the format type, wherein the target string is a string between tags.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, in which a program is stored, wherein the program executes a text extraction method when it runs.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes a text extraction method when it runs.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that implements a text extraction method when executed by a processor.

[0016] In this embodiment, a target identifier is retrieved from the first string to be processed in the streaming text. The target identifier is a character used to represent the format type of the first string to be processed. After the target identifier is retrieved, tags are retrieved from the first string to be processed based on the format type, starting from the target identifier. The tags are used to identify the start or end point of the string belonging to the format type. The target string is extracted from the first string to be processed based on the identifier and stored in the storage space corresponding to the format type. The target string is a string between tags. By determining the format type based on the target identifier and then retrieving tags from the string to be processed based on the format type to determine the target string, the purpose of accurately determining the text type and start and end positions is achieved. This improves the segmentation accuracy and solves the technical problem of low segmentation and extraction accuracy caused by the inability to accurately distinguish the specific types of text in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram of the structure of a computer terminal (mobile terminal) according to an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a text extraction method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a rendering effect provided according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of a text extraction device provided according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0025] Bidirectional semantic localization algorithm: a text segmentation method that combines forward feature matching and backward syntax verification, and determines the optimal segmentation point through dynamic weight calculation.

[0026] Multimodal classification storage architecture: A hierarchical storage structure that isolates and stores SSE (Server-Sent Events StreamText) stream text by type and supports dynamic expansion to new formats.

[0027] Formatting pollution: The phenomenon of mixing and storing content of different formats due to classification errors (such as SQL statements being incorrectly classified into Markdown containers).

[0028] With the widespread application of large-scale artificial intelligence models, SSE (Streaming Text overlay) has become common in real-time interactive scenarios. Existing technologies have the following drawbacks when processing mixed-format SSE streaming text:

[0029] 1. Simple splitting methods that rely on fixed delimiters are susceptible to interference from nested formats.

[0030] 2. Dynamic classification has low accuracy and cannot effectively distinguish between structured text such as SQL / XML and ordinary text.

[0031] 3. The lack of intelligent adaptation mechanism in front-end rendering leads to a poor user experience.

[0032] To address the aforementioned issues, this application provides a solution to achieve high-precision classification of SSE stream text containing mixed formats such as plain text, Markdown, SQL, and XML, thus resolving the misclassification problem caused by reliance on fixed delimiters in related technologies. Detailed explanation follows.

[0033] According to an embodiment of this application, a method embodiment for text extraction is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] The method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a text extraction method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the text extraction method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned text extraction method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0038] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0039] Under the above operating environment, this application embodiment provides a text extraction method, such as... Figure 2 As shown, the method includes the following steps:

[0040] Step S202: Retrieve the target identifier in the first string to be processed in the stream text, wherein the target identifier is a character used to represent the format type of the first string to be processed;

[0041] In the technical solution provided in step S202, the streaming text includes various types of SSE streaming text output by the large model. Regular expressions can be used to scan the first string to be processed to identify target identifiers, such as SQL keywords (e.g., ```sql```\n) and XML tags (e.g., identifying...). <tag>It includes formatting features such as start tags and Markdown syntax (e.g., capturing combinations of symbols like #, *, and []. Additionally, during the scanning process, it can scan the first string to be processed character by character and maintain a pointer to the current stream position.

[0042] Furthermore, upon detecting specific formatting features (such as SQL keywords, XML tags, or Markdown symbols), the current stream position pointer can immediately mark that location as the starting point for subsequent validation or classification. This allows for forward and backward scanning based on this position to confirm the integrity and type of text units, thus achieving precise text segmentation and classification. Moreover, when handling complex formats such as nested structures, the current stream position pointer essentially defines a dynamic search window. This window moves according to the pointer's movement, enabling the algorithm to quickly locate the start or end point of the previously unparsed semantic unit when new data arrives, and then complete its subsequent processing. For example, for multi-line SQL statements or unclosed XML tags, the pointer helps the algorithm maintain a search within the correct context.

[0043] Furthermore, by maintaining a pointer to the current stream position, unnecessary repetitive processing can be avoided. For example, once a piece of text is identified and classified, the pointer jumps to the beginning of the next piece of text, thus avoiding the need to rescan previously processed content, significantly improving processing efficiency, especially in scenarios with large data volumes and high real-time requirements.

[0044] In some embodiments of this application, the current stream position pointer serves to record the algorithm's reading progress in real time. Because the SSE stream text is continuously entering the processing system, the existence of the pointer allows the algorithm to know which part of the stream has been read, which is crucial for dynamically performing text classification, feature detection, and format verification.

[0045] Step S204: After retrieving the target identifier, starting from the target identifier, retrieve tags in the first string to be processed according to the format type, wherein the tags are used to identify the start or end point of the string belonging to the format type.

[0046] In the technical solution provided in step S204, the tags retrieved are different for different format types. For example, for XML format, the nearest unclosed tag start position can be retrieved backward from the location of the target identifier (also referred to as start_point in this application), and the nearest unclosed tag end position can be retrieved backward.

[0047] For SQL format, it can detect the nearest semicolon or statement terminator near `start_point`. For Markdown format, it can detect the newline character (`\n`) at the end of `markdown`.

[0048] In some embodiments of this application, the processing flow for multi-line statements includes the following steps:

[0049] The first step is to cache the starting characteristics of unclosed statements;

[0050] Optionally, once the caching mechanism is enabled, it can capture start signals with special formats in the first string to be processed. These signals could be, for example, the SELECT keyword in an SQL query or the beginning of an XML node. <tag>Or, in Markdown syntax, the bold text is immediately recorded, along with its exact coordinates in the character stream, forming a pending status marker, awaiting further closure or confirmation.

[0051] The second step is to continue feature association matching in subsequent character streams;

[0052] Optionally, each new character following the first string to be processed can be analyzed in detail to find a terminal character that matches the initial signal. For SQL, this is the semicolon at the end of the statement;. For XML, a corresponding closing semicolon is required.< / tag> Markdown bold text expects a termination **. In some embodiments, retrieval can be performed dynamically within the character stream.

[0053] The third step is to determine whether the current statement continues based on semantic coherence.

[0054] Optionally, if no matching endpoint identifier is found at the end of the first string to be processed, the algorithm assumes that the paragraph is not yet complete, marks it as a potential multi-line statement, and saves the start tag and its characteristic information in a temporary buffer for subsequent processing. This marks the temporary end of parsing the first string to be processed, but the starting point information is retained, awaiting the arrival of new data packets.

[0055] Upon receiving the second string to be processed, the process begins scanning from the beginning, focusing on retrieving previously unseen endpoint tags. This operation is not merely a simple character comparison, but a deep parsing process that takes into account the format-specific syntax rules and possible nested structures. If the expected endpoint identifier is identified in the second string to be processed, such as in SQL (;) or XML (;), then the process continues.< / tag> If it is a Markdown '**', then further operations are initiated to verify the semantic coherence.

[0056] It's important to note that semantic coherence assessment is a deep check of the connectivity between the second and first strings. During this process, it evaluates whether all content before the end marker in the second string and the information after the start marker in the first string form a coherent sentence structure. For example, for SQL, the algorithm ensures that all necessary keywords, such as FROM and WHERE, are present in the second string and that the overall sentence logic is complete. For XML, it verifies that all open tags are properly closed in the second string, without any nesting issues. Markdown syntax consistency is also rigorously checked to ensure correct formatting.

[0057] Based on the semantic coherence judgment, the algorithm makes a decision to confirm whether the characters before the end marker in the second string to be processed form a unified statement entity with the characters after the start marker in the first string to be processed. If the judgment result is positive, meaning that the semantics and syntax are seamlessly connected, the algorithm merges the two strings to form a complete statement, categorizes and stores it, and marks it as processed. Conversely, if the judgment result is negative, meaning that the statement has not yet reached a complete state, the algorithm will continue to search in the subsequent character stream until it finds the correct closing point.

[0058] Optionally, if the target string may be a multi-line statement, after retrieving the target identifier, starting from the target identifier, tags are retrieved in the first string to be processed according to the format type, including: if a start tag for marking the start point is retrieved, but no end tag for marking the end point is retrieved, the target string is determined to be a multi-line statement, and the start tag is stored; after receiving the second string to be processed, the end tag is retrieved in the second string to be processed, and a semantic coherence judgment is performed on the second string to be processed. Based on the judgment result of the semantic coherence judgment, it is determined whether the characters before the end tag in the second string to be processed are characters in the same statement as the characters after the start tag in the first string to be processed, wherein the generation time of the second string to be processed is later than the generation time of the first string to be processed.

[0059] Step S206: Extract the target string from the first string to be processed based on the identifier, and store the target string in the storage space corresponding to the format type. The target string is the string between tags.

[0060] In the technical solution provided in step S206, nested tags in formats such as XML can be processed using the following procedure:

[0061] The first step is to maintain the tag stack to record unclosed tags;

[0062] In some embodiments of this application, maintaining the tag stack record of unclosed tags includes: whenever a start tag is detected in the XML stream text or HTML stream text, such as 、 、 Or something more complicated<custom-componentattr="value"> This tag is then pushed onto the tag stack. This action ensures that every opening statement is traceable and that every nesting level is properly managed. Regardless of its depth, the state of each unclosed tag can be clearly tracked through the tag stack.

[0063] The second step is to pop the top element of the stack each time a closed label is detected;

[0064] Optionally, each time a closed label is detected, such as , The system can immediately check the match between the first tag pushed onto the top element of the tag stack. This matching process relies on the property of the tag stack: the last pushed starting tag should match the currently detected closing tag. When a match is confirmed, the top starting tag is popped off the stack, signifying that a complete tag pair has been identified and closed.

[0065] The third step is to confirm the complete closure of the stack once it is empty.

[0066] In some embodiments of this application, an empty stack signifies that all previously pushed start tags have found matching closing tags, and the tag stack is restored to an empty state. This state indicates that the statement is complete, meaning that all tag nesting in the entire XML or HTML stream text has been closed, and no unclosed tags remain to be processed. In this case, it can be determined that the target string in the stream text has been identified and extracted. For example, for a complex HTML stream text, if it contains multiple nested tags... , Tagging, through this mechanism, ensures that each There are corresponding ones Each There are corresponding ones This ensures that the integrity of the structure is maintained during the analysis process.

[0067] This tag-stack-based parsing strategy is particularly important for processing XML or HTML streaming text. Since these two formats are commonly used for web page layout and data presentation, their structures are often complex and varied, potentially containing multiple levels of nesting and various custom tags. The algorithm must be able to accurately track the lifecycle of each tag, from its birth (the appearance of the start tag) to its demise (the appearance of the corresponding closing tag), ensuring the integrity of each nesting and the correctness of the syntax. By maintaining a tag stack, each new start tag is pushed onto the stack, and each time a closing tag is encountered, its match with the top element of the stack is checked. Once a match is found, the top element is immediately popped from the stack. This mechanism provides a clear path for processing multi-line statements.

[0068] In practice, this process effectively avoids common format pollution problems, namely, incorrectly nesting an unrelated tag within a tag container, leading to parsing and rendering chaos. For example, a Tags might be used to wrap text and images, but if in An independent one was mistakenly inserted before it was fully closed. Tags can disrupt the original structure. Dynamic management of the tag stack allows for timely detection of such errors, ensuring that each tag closes in the appropriate position and maintaining the logical and visual clarity of the entire document.

[0069] Furthermore, this mechanism supports real-time processing of streaming text, meaning that tags can be identified and closed instantly as the text stream continues to arrive, rather than waiting until all the text has been received. This is crucial for the response speed of SSE (Server Send Event) streaming text, ensuring that each tag unit is processed in the shortest possible time, providing users with near-instantaneous feedback, and enhancing application interactivity and user experience.

[0070] Optionally, extracting the target string from the first string to be processed based on the identifier and storing the target string in the storage space corresponding to the format type includes: determining a feature matching value based on the target identifier, wherein if the format type indicated by the target identifier is a preset format type, the feature matching value is determined to be a first value, otherwise a second value, and the first value is greater than the second value; determining a syntax closure value based on the tag detection result, wherein if the detected tags include a start tag for marking the start point and an end tag for marking the end point, the syntax closure value is determined to be a third value, otherwise a fourth value, and the third value is greater than the fourth value; determining a comprehensive evaluation score based on the feature matching value and the syntax closure value, and if the comprehensive evaluation score is not less than a preset threshold, extracting the target string from the first string to be processed based on the identifier and storing the target string in the storage space corresponding to the format type; if the comprehensive evaluation score is less than the preset threshold, retaining the first string to be processed in the cache, and after receiving the third string to be processed, retrieving the target identifier or tag again, wherein the generation time of the third string to be processed is later than the generation time of the first string to be processed.

[0071] As an optional implementation, determining the comprehensive evaluation score based on the feature matching value and the syntax closure value includes: determining a first weight coefficient corresponding to the feature matching value and a second weight coefficient corresponding to the syntax closure value; determining a first product between the feature matching value and the first weight coefficient, and a second product between the syntax closure value and the second weight coefficient; and determining the sum of the first product and the second product as the comprehensive evaluation score.

[0072] In some embodiments of this application, the formula for calculating the comprehensive evaluation score is as follows:

[0073] Score = α × Feature Matching Value + β × Syntax Closure Value

[0074] The values ​​of α and β can be set by the user, and their sum is a fixed value, such as 1. For example, α = 0.6, β = 0.4. The first and third values ​​can be 1, and the second and fourth values ​​can be 0 or other numbers less than 1. The preset threshold corresponding to the comprehensive evaluation score can be 1 or other set values ​​when the first and third values ​​are 1. The preset threshold can be determined based on the first and third values.

[0075] As an optional implementation, for nested structures, the step of retrieving tags in the first string to be processed based on the target identifier and the format type includes: establishing a tag stack; recording the retrieved unclosed tags in the tag stack, wherein the unclosed tags include the starting tag where no corresponding ending tag was found, and the ending tag where no corresponding starting tag was found.

[0076] In some embodiments of this application, storing the target string in the storage space corresponding to the format type includes: after determining the start tag or end tag corresponding to the unclosed tag, determining that the unclosed tag and its corresponding start tag or end tag form a closed tag pair, and popping the closed tag pair from the tag stack; if it is determined that all tags in the tag stack have been popped, determining that the target string is a closed complete statement, and storing the target string in the storage space corresponding to the format type.

[0077] In some embodiments of this application, it is assumed that the input stream fragment is a string with the following nested XML structure: <root> <data> <info> text< / info> < / data> < / root> The specific processing procedure is as follows:

[0078] The first step, forward scanning, detected at position 0. <root>Label

[0079] The second step was to continue scanning until location 15 was found. <data>Label

[0080] Third step, update the tag stack to [ <root> , <data>]

[0081] Step 4, location 28 was detected. <info>Tags, stack becomes [ <root>, <data>, <info>]

[0082] Step 5, location 42 was detected.< / info> Pop the top element from the stack

[0083] Step 6, location 53 was detected.< / data> Pop the top of the stack again

[0084] Step 7, location 64 was detected.< / root> Stack empty confirms complete XML

[0085] In some embodiments of this application, after storing the target string in the storage space corresponding to the format type, the method further includes: determining the rendering method corresponding to the storage space; rendering the target string using the rendering method; and displaying the rendered target string.

[0086] Optionally, assuming there are four types of strings, a four-level categorization container can be created to store different types of strings or text streams composed of strings in independent spaces (`plainText`, `formattedText`, `queryLang`, `markupLang`), avoiding formatting contamination. Furthermore, as... Figure 3 As shown, it can be rendered according to the preset rendering methods corresponding to different formats, and finally presented to the user as follows. Figure 3 The text shown contains various formats.

[0087] In some embodiments of this application, new format types (such as JSON and YAML) can also be supported through predefined interfaces. The new formats can be made compatible simply by extending the `componentMap` mapping rules.

[0088] By retrieving a target identifier from the first string to be processed in the streaming text, where the target identifier is a character representing the format type of the first string to be processed; after retrieving the target identifier, starting from the target identifier, searching for tags in the first string to be processed according to the format type, where tags are used to identify the start or end point of strings belonging to the format type; extracting the target string from the first string to be processed based on the identifier, and storing the target string in the storage space corresponding to the format type, where the target string is a string between tags, the format type is determined based on the target identifier, and then the target string is determined by searching for tags in the string to be processed according to the format type. This achieves the goal of accurately determining the text type and start and end positions, thereby improving the technical effect of segmentation accuracy and solving the technical problem of low segmentation and extraction accuracy caused by the inability to accurately distinguish the specific types of various texts in related technologies.

[0089] Furthermore, the method provided in this application embodiment can effectively improve the accuracy of text and string classification, increasing the classification accuracy from 65% in related technologies to over 95%, and reducing the false positive rate by 80% when processing nested structures (such as XML parent-child tags) compared to related technologies. Moreover, the method provided in this application embodiment has a single streaming processing time of <50ms, which is 4 times more efficient than the multi-pass scanning method in related technologies.

[0090] This application provides a text extraction device. Figure 4 This is a schematic diagram of the device. From Figure 4 As can be seen from the diagram, the device includes: a first processing module 40, used to retrieve a target identifier in a first string to be processed of the streaming text, wherein the target identifier is a character used to represent the format type of the first string to be processed; a second processing module 42, used to retrieve tags in the first string to be processed based on the format type, starting from the target identifier, after retrieving the target identifier, wherein the tags are used to identify the start or end point of the string belonging to the format type; and a third processing module 44, used to extract a target string from the first string to be processed based on the identifier and store the target string in the storage space corresponding to the format type, wherein the target string is the string between tags.

[0091] In some embodiments of this application, after the second processing module 42 retrieves the target identifier, the step of retrieving a tag in the first string to be processed based on the target identifier as the starting position and the format type includes: if a start tag for marking the start point is retrieved, but no end tag for marking the end point is retrieved, the target string is determined to be a multi-line statement, and the start tag is stored; after receiving the second string to be processed, the end tag is retrieved in the second string to be processed, and the semantic coherence of the second string to be processed is judged, and based on the judgment result of the semantic coherence judgment, it is determined whether the characters before the end tag in the second string to be processed are characters in the same statement as the characters after the start tag in the first string to be processed, wherein the generation time of the second string to be processed is later than the generation time of the first string to be processed.

[0092] In some embodiments of this application, the second processing module 42 retrieves tags in the first string to be processed based on the target identifier as the starting position and the format type, including: establishing a tag stack; recording the retrieved unclosed tags in the tag stack, wherein the unclosed tags include starting tags for which no corresponding ending tags have been found, and ending tags for which no corresponding starting tags have been found.

[0093] In some embodiments of this application, the step of the third processing module 44 storing the target string into the storage space corresponding to the format type includes: after determining the start label or end label corresponding to the unclosed label, determining that the unclosed label and its corresponding start label or end label form a closed label pair, and popping the label at the top of the stack; if it is determined that all labels in the label stack have been popped, determining that the target string is a closed complete statement, and storing the target string into the storage space corresponding to the format type.

[0094] In some embodiments of this application, the third processing module 44 extracts the target string from the first string to be processed based on the identifier and stores the target string in the storage space corresponding to the format type, including: determining a feature matching value based on the target identifier, wherein, if the format type indicated by the target identifier is a preset format type, the feature matching value is determined to be a first value, otherwise a second value, and the first value is greater than the second value; determining a syntax closure value based on the tag detection result, wherein, if the detected tags include a start tag for marking the start point and an end tag for marking the end point, the syntax closure value is determined to be a third value, otherwise a fourth value, and the third value is greater than the fourth value; determining a comprehensive evaluation score based on the feature matching value and the syntax closure value, and if the comprehensive evaluation score is not less than a preset threshold, extracting the target string from the first string to be processed based on the identifier and storing the target string in the storage space corresponding to the format type; if the comprehensive evaluation score is less than the preset threshold, retaining the first string to be processed in the cache, and after receiving the third string to be processed, retrieving the target identifier or tag again, wherein the generation time of the third string to be processed is later than the generation time of the first string to be processed.

[0095] In some embodiments of this application, the step of the third processing module 44 in determining the comprehensive evaluation score based on the feature matching value and the syntax closure value includes: determining the first weight coefficient corresponding to the feature matching value and the second weight coefficient corresponding to the syntax closure value; determining the first product between the feature matching value and the first weight coefficient, and the second product between the syntax closure value and the second weight coefficient; and determining the sum of the first product and the second product as the comprehensive evaluation score.

[0096] In some embodiments of this application, after storing the target string in the storage space corresponding to the format type, the third processing module 44 is further used to: determine the rendering method corresponding to the storage space; render the target string using the rendering method, and display the rendered target string.

[0097] It should be noted that each module in the above-mentioned text extraction device can be a program module (for example, a set of program instructions that implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0098] According to an embodiment of this application, a non-volatile storage medium is also provided, which stores a program. When the program runs, it executes the following text extraction method: retrieving a target identifier from a first string to be processed in the stream text, wherein the target identifier is a character used to represent the format type of the first string to be processed; after retrieving the target identifier, retrieving tags from the first string to be processed based on the format type, starting from the target identifier, wherein the tags are used to identify the start or end point of strings belonging to the format type; extracting a target string from the first string to be processed based on the identifier, and storing the target string in the storage space corresponding to the format type, wherein the target string is the string between tags.

[0099] According to an embodiment of this application, an electronic device is also provided, including a memory and a processor. The processor is used to run a program stored in the memory, wherein the program executes the following text extraction method: retrieving a target identifier in a first string to be processed of the streaming text, wherein the target identifier is a character used to represent the format type of the first string to be processed; after retrieving the target identifier, retrieving tags in the first string to be processed based on the format type, with the target identifier as the starting position, wherein the tags are used to identify the start or end point of the string belonging to the format type; extracting a target string from the first string to be processed based on the identifier, and storing the target string in the storage space corresponding to the format type, wherein the target string is the string between tags.

[0100] According to an embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the following text extraction method: retrieving a target identifier in a first string to be processed from the streaming text, wherein the target identifier is a character used to represent the format type of the first string to be processed; after retrieving the target identifier, retrieving tags in the first string to be processed based on the format type, with the target identifier as the starting position, wherein the tags are used to identify the start or end point of the string belonging to the format type; extracting a target string from the first string to be processed based on the identifier, and storing the target string in the storage space corresponding to the format type, wherein the target string is the string between tags.

[0101] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0106] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / info> < / data> < / root> < / data> < / root>

Claims

1. A text extraction method, characterized in that, include: Retrieve a target identifier in a first string to be processed from the stream text, wherein the target identifier is a character used to represent the format type of the first string to be processed; After the target identifier is retrieved, a tag is retrieved in the first string to be processed, starting from the target identifier and according to the format type. The tag is used to identify the start or end point of the string belonging to the format type. The target string is extracted from the first string to be processed based on the identifier, and the target string is stored in the storage space corresponding to the format type, wherein the target string is the string between the tags.

2. The text extraction method according to claim 1, characterized in that, Starting from the target identifier, retrieving tags from the first string to be processed according to the format type includes: Establish a tag stack; The retrieved unclosed tags are recorded in the tag stack, wherein the unclosed tags include the starting tags for which no corresponding ending tags were found, and the ending tags for which no corresponding starting tags were found.

3. The text extraction method according to claim 2, characterized in that, Storing the target string into the storage space corresponding to the format type includes: After determining the start label or end label corresponding to the unclosed label, determine that the unclosed label and its corresponding start label or end label form a closed label pair, and pop the label at the top of the stack; If all tags in the tag stack are popped, the target string is determined to be a closed complete statement, and the target string is stored in the storage space corresponding to the format type.

4. The text extraction method according to claim 1, characterized in that, After retrieving the target identifier, starting from the target identifier, the tag retrieval in the first string to be processed according to the format type includes: If a start tag for marking the start point is found, but no end tag for marking the end point is found, the target string is determined to be a multi-line statement, and the start tag is stored. After receiving the second string to be processed, the endpoint tag is retrieved in the second string to be processed, and a semantic coherence judgment is performed on the second string to be processed. Based on the judgment result of the semantic coherence judgment, it is determined whether the characters before the endpoint tag in the second string to be processed are characters in the same statement as the characters after the starting tag in the first string to be processed. The second string to be processed is generated later than the first string to be processed.

5. The text extraction method according to claim 1, characterized in that, Extracting the target string from the first string to be processed based on the identifier and storing the target string in the storage space corresponding to the format type includes: A feature matching value is determined based on the target identifier, wherein if the format type indicated by the target identifier is a preset format type, the feature matching value is determined to be a first value; otherwise, it is a second value, and the first value is greater than the second value. The syntax closure value is determined based on the tag detection results. If the detected tags include a start tag for marking the start point and an end tag for marking the end point, the syntax closure value is determined to be a third value; otherwise, it is a fourth value. The third value is greater than the fourth value. A comprehensive evaluation score is determined based on the feature matching value and the syntax closure value. If the comprehensive evaluation score is not less than a preset threshold, a target string is extracted from the first string to be processed based on the identifier, and the target string is stored in the storage space corresponding to the format type. If the overall evaluation score is less than a preset threshold, the first string to be processed is stored in the cache, and after receiving the third string to be processed, the target identifier or the tag is retrieved again, wherein the generation time of the third string to be processed is later than the generation time of the first string to be processed.

6. The text extraction method according to claim 5, characterized in that, The comprehensive evaluation score is determined based on the feature matching value and the syntax closure value, including: Determine the first weight coefficient corresponding to the feature matching value and the second weight coefficient corresponding to the syntax closure value; Determine the first product between the feature matching value and the first weight coefficient, and the second product between the syntax closure value and the second weight coefficient; The sum of the first product and the second product is determined as the comprehensive evaluation score.

7. The text extraction method according to claim 1, characterized in that, After storing the target string into the storage space corresponding to the format type, the method further includes: Determine the rendering method corresponding to the storage space; The target string is rendered using the aforementioned rendering method, and the rendered target string is then displayed.

8. A text extraction device, characterized in that, include: A first processing module is configured to retrieve a target identifier in a first string to be processed of the stream text, wherein the target identifier is a character used to represent the format type of the first string to be processed; The second processing module is used to retrieve tags in the first string to be processed based on the format type, starting from the target identifier after the target identifier is retrieved. The tags are used to identify the start or end point of the string belonging to the format type. The third processing module is used to extract the target string from the first string to be processed based on the identifier, and store the target string in the storage space corresponding to the format type, wherein the target string is the string between the tags.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to perform the text extraction method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the text extraction method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the text extraction method according to any one of claims 1 to 7.

Images