Data extraction method and device, electronic equipment, storage medium and program product

Abstract

The data extraction method, device, electronic equipment, storage medium and program product provided by the application, the method comprises the following steps: sequentially writing the streaming text into a global buffer, and storing the current parsed key and value into a key buffer and a value buffer respectively; when a key end identifier is detected in the global buffer, it is judged whether the key expected to be extracted currently needs to be corrected; if yes, the corrected key is determined, and the corrected key is used to replace the key expected to be extracted currently, otherwise, no processing is performed; when a value end identifier is detected in the global buffer, the value in the current value buffer is bound and output with the corrected key, and the current key buffer and the current value buffer are emptied until a streaming output termination identifier is detected. The application can accurately extract specific field values in structured content in real time during the streaming data output process, and provide support for subsequent data analysis and application.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data extraction method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of intelligent technologies, large language models have become one of the core technologies in the field of natural language processing. Chat products based on large language models are widely used in scenarios such as intelligent customer service, virtual assistants, and live streaming, providing users with an intelligent interactive experience. To optimize user experience, mainstream products currently adopt streaming output technology. This technology presents the responses generated by the large model to the user progressively, word by word, rather than the traditional one-time complete output. This technology significantly reduces user waiting time, effectively alleviates the latency problem caused by the high computational complexity of large models, and greatly improves the smoothness of interaction and user experience.

[0003] During streaming output, the model generates content that is continuous and dynamically changing, with only a portion of the content being output at each time segment until a complete response is formed. While this output method improves the perceived response speed on the user's end, it also introduces new technical challenges to backend processing. Especially in scenarios where structured data fields need to be extracted in real-time during streaming output, traditional parsing methods based on complete data face fundamental limitations. Streaming output results in data fragments being incomplete during transmission, making it impossible for conventional parsers to directly identify and extract target fields.

[0004] Therefore, existing technologies struggle to extract specific field values ​​from structured content in real time and accurately during streaming data output, necessitating a structured data processing method that can adapt to the characteristics of streaming output. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a data extraction method, streaming data extraction device, electronic device, storage medium, and program product, which can extract specific field values ​​from structured content in real time and accurately during streaming data output, providing support for subsequent data analysis and applications. To achieve the above objective, the technical solution adopted by this invention is as follows: In a first aspect, the present invention provides a data extraction method, the method comprising: sequentially writing streaming text into a global buffer, and storing the currently parsed key and value into a key buffer and a value buffer respectively; when a key end marker is detected in the global buffer, determining whether the key expected to be extracted needs to be corrected; if so, determining the corrected key based on the key in the current buffer, and replacing the key expected to be extracted with the corrected key; otherwise, no processing is performed; when a value end marker is detected in the global buffer, binding the value of the current value buffer with the corrected key for output, and clearing the current key buffer and the current value buffer until a streaming output termination marker is detected.

[0006] Secondly, the present invention provides a streaming data extraction device, comprising a parsing module, a judgment module, a correction module, and an extraction module; the parsing module is used to sequentially write the streaming text into a global buffer and store the currently parsed key and value into a key buffer and a value buffer, respectively; the judgment module is used to determine whether the key to be extracted needs to be corrected when a key end marker is detected in the global buffer; the correction module is used to determine the corrected key based on the key in the current buffer if the result of the judgment module is yes, and replace the key to be extracted with the corrected key, otherwise no processing is performed; the extraction module is used to bind the value in the current value buffer with the corrected key and output it when a value end marker is detected in the global buffer, and clear the current key buffer and the current value buffer until a streaming output termination marker is detected.

[0007] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the method described in any of the foregoing embodiments.

[0008] Fourthly, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data extraction method as described in any of the foregoing embodiments.

[0009] Fifthly, the present invention provides a program product comprising a computer program, wherein the processor is capable of executing the computer program to implement the data extraction method described in any of the foregoing embodiments.

[0010] The data extraction method, streaming data extraction device, electronic device, storage medium, and program product provided in this invention sequentially write streaming text into a global buffer, while temporarily storing the parsed keys and values ​​in a key buffer and a value buffer, respectively, providing a temporary storage mechanism for subsequent field extraction. When a key end marker is detected in the global buffer, a key completion judgment process is immediately triggered to ensure the accuracy of the extracted keys. Then, when a value end marker is detected, the content of the current value buffer is bound to the corrected key and output to form a structured key-value pair. At the same time, the key buffer and value buffer are cleared to prepare for receiving the next set of key-value content. This process continues until a streaming output termination marker is detected, thereby achieving real-time and accurate extraction of structured fields while the streaming text is still being output, providing support for subsequent data analysis and applications.

[0011] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of the data extraction method provided in this embodiment of the invention; Figure 2 A schematic flowchart illustrating the data extraction method provided in this embodiment of the invention; Figure 3 A functional block diagram of a data extraction device provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0015] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0016] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0017] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the data extraction method provided in an embodiment of the present invention. Figure 1 In this system, a large language model is deployed on server 10, and intelligent applications can be deployed on terminal 20. During user interaction with the intelligent application, user input is sent to the large model deployed on server 10. The large model performs inference based on the input and generates response content. In streaming output mode, the model sends the generated content in chunks to terminal 20 step by step. Terminal 20 displays the content word by word or sentence by sentence in real time, thereby improving the immediacy and user experience of the interaction.

[0018] Server 10 can be any of a single server, multiple servers, a cloud computing platform, or a virtualization center. Furthermore, server 10 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services (such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms).

[0019] There is no limit to the number of terminals 20; there can be one or more. Terminal 20 can be any of the following devices: smartphone, tablet, smart speaker, smartwatch, laptop, handheld smart device, portable gaming device, or desktop computer; the type of terminal 20 is not limited to these. Each terminal 20 runs a corresponding smart application. Users can interact with the large model through these applications.

[0020] Taking a real-world application scenario as an example, suppose the intelligent application is an AI customer service system. During the process of answering user questions, the large model outputs structured data (such as JSON format) as follows: {"name": "Zhang San", "age": "25", "city": "Beijing"}. During the streaming output, the data received by terminal 20 arrives progressively in streaming chunks. For example, the first chunk is "{", the second chunk is "name", the third chunk is ":", the fourth chunk is "Zhang San", and so on, until the output ends with "}".

[0021] Considering that in streaming output scenarios, real-time and accurate extraction of specific fields from structured data can support subsequent data analysis and applications, but traditional one-time data parsing methods are not suitable for streaming text output. Therefore, this invention provides a data extraction method for streaming text output, which can accurately extract structured field content even before the streaming data is completely output.

[0022] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating the data extraction method provided in an embodiment of the present invention. The execution subject of this method can be an electronic device (such as...). Figure 1 The method (including steps S201 to S204) is described below: S201: Write the streaming text sequentially to the global buffer, and store the currently parsed key and value into the key buffer and value buffer respectively; S202: When a key end marker is detected in the global buffer, determine whether the key that should be extracted needs to be corrected. S203: If yes, then determine the corrected key based on the key in the current cache and replace the key that should be retrieved with the corrected key; otherwise, do nothing. S204: When a value end marker is detected in the global buffer, the value of the current value buffer is bound to the corrected key for output, and the current key buffer and the current value buffer are cleared until a streaming output termination marker is detected.

[0023] The data extraction method provided in this invention writes streaming text sequentially into a global buffer, while temporarily storing the parsed keys and values ​​in a key buffer and a value buffer, respectively, providing a temporary storage mechanism for subsequent field extraction. When a key end marker is detected in the global buffer, a key completion judgment process is immediately triggered to ensure the accuracy of the extracted keys. Then, when a value end marker is detected, the content of the current value buffer is bound to the corrected key and output to form a structured key-value pair. At the same time, the key buffer and value buffer are cleared to prepare for receiving the next set of key-value content. This process continues until a streaming output termination marker is detected, thereby achieving real-time and accurate extraction of structured fields while the streaming text is still being output, providing support for subsequent data analysis and applications.

[0024] To better understand these effects, consider this example: Suppose a user asks the intelligent customer service, "Help me find a 25-year-old friend in Guangzhou." When the agent outputs structured data, once it extracts "city": "Guangzhou" and "age": "25", it can immediately call the backend service to start the search, instead of waiting for the entire JSON data to be output, thus improving interaction efficiency. As another example, if the intelligent customer service is generating JSON data containing order information, once it extracts "order_id", the system can immediately start querying the order status, instead of waiting for the entire JSON to be output, enhancing system responsiveness.

[0025] Next, the embodiments of the present invention will provide a detailed description of the above steps S201 to S204.

[0026] In step S201, this embodiment of the invention first establishes a three-level buffer mechanism. In a streaming output scenario, structured data (such as JSON) is generated and received incrementally, and the data is not complete. This embodiment of the invention designs a three-level buffer mechanism to manage global streaming content, key content, and value content, respectively, to achieve fine-grained management of streaming text. Specifically: the global buffer (denoted as buffer) accumulates all received streaming text as a global content reference; the key buffer (denoted as key_buffer) specifically stores the currently parsed key text; and the value buffer (denoted as value_buffer) specifically stores the value text corresponding to the currently parsed key.

[0027] Based on the above three-level buffer setup, in step S201 of this embodiment, the streaming text is sequentially written to the global buffer. By continuously parsing the content in the buffer, it can be determined that the currently parsed key and value are stored in the key buffer and value buffer, respectively. To accurately determine whether the currently parsed content is a key or a value, this embodiment can identify it in the following way: Step a1: identifying current content to be parsed by detecting structure separators in the global buffer; Step a2: when the structure separator is a predefined key separator, confirming that the current parsed content is a key, and writing the key into a key buffer; Step a3: when the structure separator is a predefined value separator, confirming that the current parsed content is a value, and writing the value into a value buffer.

[0028] In the embodiment of the present invention, the starting position of a key or a value can be determined by detecting whether a structure separator appears in the streaming text. Wherein, the key separators comprise a streaming output start symbol and a value end identifier. The value separators comprise a key end identifier. For example, the streaming text output by the model is: {"name": "Zhang San", "age": "25", "city": "Guangzhou"}. Wherein, "{" is the start of a JSON object, that is, the streaming output start symbol, and "name" immediately following the streaming output start symbol is the key; "," is the value end identifier, and the key "city" immediately follows it. Similarly, the value separator comprises the key end identifier, for example, ":" in the above example, which is followed by the value "Zhang San". That is, when "{" or "," is detected in the buffer, it indicates that the next content received will be a key, and when ":" is detected, it indicates that the key parsing is completed, and the next content received will be a value.

[0029] For intuitive understanding of the above step S201, an example is given below. Assume that the model is outputting the following structured data content in a streaming manner: {"name": "Zhang San","age": "25","city": "Guangzhou"}. This structured data content is output in chunks step by step. For example, the model outputs "{" first, then outputs "name", then outputs ":", and so on. The operation of the three buffers is introduced by taking the parsing process of one key-value pair "name":"Zhang San" as an example.

[0030] In the initial stage (just received "{"), the model outputs the first character "{", and the system writes this character into the buffer, indicating that this is the start of the entire streaming text. Key and value parsing has not started yet, so both the key_buffer and the value_buffer are empty. At this time, "{" as a streaming output start indicates that the next content is a key.

[0031] Next, the model continues to output "name", which the system writes to the buffer. The buffer now contains {"name"}. The system detects that this is the beginning of a key and writes "name" to the key_buffer. At this point, the value parsing stage has not yet begun, so the value_buffer is still empty.

[0032] The model continues to output a colon (:), indicating that the current key has ended. The content in the buffer is updated to {"name":}. The key_buffer still retains "name" because it is the currently being processed key. The value_buffer is in the state of receiving values, but there is no value data yet. This serves as a key end marker, indicating that the next content will be a value. The system begins to detect that the colon (:) is the beginning of a value.

[0033] The model continues to output "Zhang San", which the system writes to the value_buffer. The buffer is now updated to {"name": "Zhang San", which is the value corresponding to "name". At this point, "name":"Zhang San" can be extracted.

[0034] When the model outputs a comma (,), it indicates that the current value has been parsed. The buffer is updated to {"name": "Zhang San", recording all the current content. At this point, the key_buffer can be cleared to prepare for the next key "age", and the value_buffer can be cleared to wait for the parsing of the next value.

[0035] The above example demonstrates that the buffer continuously records all received streaming text content for structure determination and subsequent extraction and analysis; the key_buffer stores the key currently being parsed until the value corresponding to that key is parsed; and the value_buffer stores the value corresponding to the current key, clearing it once the end-of-value marker appears and preparing for the parsing of the next value. These three components work together to achieve real-time and accurate extraction of structured data from streaming output.

[0036] Next, the embodiments of the present invention can extract key-value pairs based on the contents stored in each buffer, see step S202.

[0037] In step S202, when a key end marker is detected in the global buffer, it is determined whether the key that should be extracted needs to be corrected.

[0038] In this embodiment of the invention, as can be seen from the example above, the key end marker can be represented by ":". When the system detects the key end marker, it determines that a complete key has been generated and stored in the key_buffer. At this point, to prevent the key obtained during the key parsing stage from being parsed repeatedly, this embodiment of the invention needs to first determine whether the key expected to be extracted needs to be corrected.

[0039] Before performing key correction, this embodiment of the invention first sets up a predefined key list (denoted as dict_key_list) according to business requirements. This table is a list that maintains the output order of predefined keys, and the desired extracted keys and their order can be controlled through this table. For example: dict_key_list = ["name","age", "city"], that is, the system will match the keys output by the model in this order to ensure that the extracted fields are consistent with the business logic.

[0040] It should be understood that `dict_key_list` can be set by developers or the system according to business needs before the parsing process starts. Each time parsing begins, the system can load the corresponding `dict_key_list` based on the current task. The content in `key_buffer` is the keys actually extracted during the model's streaming output, while `dict_key_list` contains the keys expected to be extracted. The data extraction method in this embodiment is particularly suitable for situations where the order of the keys output by the model matches the predefined key output order.

[0041] Furthermore, in this embodiment of the invention, a list of parsed keys (denoted as parsed_key_list) is pre-set. This list stores the keys that have been parsed and can be used as a basis for subsequent determination of whether the currently parsed key is a duplicate parsing.

[0042] Based on the pre-set dict_key_list and parsed_key_list, in this embodiment of the invention, the method for determining whether the currently expected keys to be extracted need to be corrected is as shown in steps b1 to b2, and is explained below: Step b1: Determine the key to be extracted based on the predefined key list; In this embodiment of the invention, the number of times the key end marker appears in the global buffer corresponds to the order of the predefined keys in the predefined key list. For example, the first appearance of the key end marker corresponds to the predefined key in the first position of the predefined key list, the second appearance corresponds to the predefined key in the second position, and so on.

[0043] Therefore, in step b1, the number of times the currently detected key end marker appears can be determined; in the predefined key list, the predefined key corresponding to the number of occurrences is taken as the key that should be extracted now, thereby quickly identifying the key that should be extracted now, which serves as the basis for subsequent extraction decisions.

[0044] Step b2: If the key that should be extracted is in the list of decrypted keys, then the key that should be extracted needs to be corrected; otherwise, no correction is needed.

[0045] To better understand the correction process described above, let's illustrate with an example. Suppose: `dict_key_list = ["A","B","C"]`. When the end-of-key marker is detected for the first time in the buffer, the expected key to be extracted is "A". If "A" is already in `parsed_key_list`, it means "A" is being parsed repeatedly, and the currently extracted key needs to be corrected. Conversely, if "A" has not yet been parsed, then the extracted key does not need to be corrected.

[0046] Next, in this embodiment of the invention, a key correction mechanism based on edit distance is provided to address issues such as key spelling errors that may occur during streaming output, thereby improving the accuracy of structured data parsing. Specifically, steps c1 to c2 are described below: Step c1: Determine the edit distance between the keys in the current key buffer and each predefined key in the predefined key list.

[0047] In this step, upon detecting a key separator (such as a colon ":"), the system extracts the keys stored in the current key buffer (key_buffer) and compares them one by one with each key in the predefined key list (dict_key_list), calculating the edit distance between each key. The edit distance measures the degree of difference between two strings and can be determined by the minimum number of operations required to insert, delete, or replace characters.

[0048] Step c2: Use the predefined key with the smallest edit distance as the corrected key, replacing the key that should be extracted.

[0049] In this step, the key with the smallest distance from the current key buffer content editing is selected from the predefined key list as the corrected key.

[0050] After the above correction mechanism, in step S204, when a value end marker is detected in the global buffer, the value of the current value buffer is bound to the corrected key for output, and the current key buffer and the current value buffer are cleared until a streaming output termination marker is detected.

[0051] In this embodiment of the invention, after the key correction is completed, the value extraction stage begins. Specifically, when a value end marker is detected in the buffer, it is determined that the current value content has been completely received. Then, the value content stored in the current value buffer is bound to the determined key (i.e., the key in the current key buffer) to form a structured key-value pair (such as "name": "Zhang San"), and this key-value pair is output as the current parsing result.

[0052] After completing the key-value binding and output operations, the current key buffer and value buffer can be cleared to reserve space for parsing subsequent key-value pairs. At the same time, continue to listen to the streaming output content and repeat the above process until the termination marker of the streaming output (such as the complete JSON structure closing symbol "}") is detected. At this time, it is determined that the parsing of the current structured data block is complete.

[0053] Through the above steps, the embodiments of the present invention can realize real-time parsing and extraction of structured data in a streaming output environment, thereby extracting specific field values ​​in structured content in real time and accurately during the generation of streaming data streams, providing data support for subsequent applications.

[0054] In one embodiment of the present invention, after extracting a key-value pair, the corrected key can be recorded in the parsed key list, and the value of the current value buffer can be recorded in the predefined key list at the position of the predefined key with the smallest edit distance, so as to correct subsequent key-value pairs.

[0055] In this embodiment of the invention, an effective processing mechanism is provided for situations where the key order in the model output is inconsistent with the predefined key order. It should be noted that "inconsistent order" here specifically refers to a change in the position of the entire key-value pair in the output sequence. For example, in the predefined order, the first output key is expected to be "A", but in the actual model output, "A" appears as the second key. In this case, although the order in which "A" appears is inconsistent with expectations, its corresponding value is still the correct value. A solution could be: Step 1: When a key end marker is detected, extract the current key from the key buffer, calculate the edit distance between the current key and all predefined keys, and determine the predefined key with the smallest edit distance as the current most similar key.

[0056] Step 2: Determine if the most similar key matches the expected key to be extracted; if they match, proceed to Step 3; if they do not match, proceed to Step 4.

[0057] Step 3: Bind the most similar key to the value in the current value buffer and mark the key as resolved; Step 4: Check if there is a key in the list of keys waiting to be parsed that has the smallest edit distance to the current most similar key; if not, bind the most similar key to the value in the current value buffer and add the key that should be extracted to the list of keys waiting to be parsed; if it exists, remove the key from the list of keys waiting to be parsed, bind the most similar key to the value in the current value buffer, and then add the key that should be extracted to the list of keys waiting to be parsed. Step 5: Continue parsing until all keys have been parsed.

[0058] In this embodiment of the invention, after each parsing is completed, the predefined key list and the list of keys waiting to be parsed are updated according to the current state; if all predefined keys have been parsed, the process ends; if there are still unparsed keys, the streaming output content is parsed again, and attempts are made to extract the unparsed keys.

[0059] For example, the predefined key order is [A, B, C, D], while the model's actual output key order is [D, B, A, C]. During parsing, when the first key end marker is detected, the actual key "Dd" is extracted from the key buffer. The most similar predefined key is determined to be "D" by calculating the edit distance. At this point, the expected key to be extracted is "A," which is inconsistent with the identified most similar key "D." Therefore, the extraction of "A" is skipped, "D" is bound to the value in the current value buffer, and "A" is added to the list of keys waiting to be parsed, p = [A], indicating that this key has not yet been extracted and may be extracted later. At this point, "D" is marked as parsed in the predefined key list. Continuing to parse the next key, the model outputs "B." After calculating the edit distance, the most similar key is confirmed to still be "B," consistent with the expected key to be extracted. Therefore, "B" can be directly bound to its corresponding value, and the list of keys waiting to be parsed remains p = [A]. At this point, "B" is marked as parsed in the predefined key list.

[0060] Continuing to parse the next key, the model outputs "A", while the expected key to be extracted is "C", which is inconsistent. At this point, the key with the smallest edit distance to "A" is searched in the list of keys waiting to be parsed, p. It is found that "A" itself exists in p. Therefore, "A" is removed from the list of keys waiting to be parsed, and "A" is bound to the current value in the output. Simultaneously, the expected key to be extracted, "C", is added to the list of keys waiting to be parsed. Now p = [C], and "A" is marked as parsed in the predefined key list.

[0061] Continuing to parse the next key, the model outputs "C". The expected key to be extracted is "D" (but it has already been parsed). "C" is found in the list of keys waiting to be parsed, indicating that this key was previously skipped. Therefore, "C" is bound to the current value in the output and removed from the list of keys waiting to be parsed. At this point, p = [], and "C" is marked as parsed in the predefined key list.

[0062] Ultimately, all predefined keys are successfully parsed. Once the parsing key list is empty, the field extraction process is complete, and the output maintains the same order as the predefined keys. In summary, the data extraction method provided by this invention has the following advantages: First, this invention can simultaneously extract structured content during streaming output without waiting for complete data generation, thus significantly reducing data processing response latency and improving system real-time performance and interaction efficiency. Second, this invention introduces a real-time error correction mechanism based on edit distance. By calculating the edit distance between the keys output by the model and each key in the predefined key list, and combining this with the expected order of the keys for matching, it can effectively identify and correct spelling errors or key order discrepancies that may occur in the model output in a streaming environment, improving parsing accuracy. Furthermore, this invention significantly improves the extraction accuracy of structured data fields through a three-level caching mechanism, accurately distinguishing the value content corresponding to different fields and effectively avoiding data confusion. Finally, this invention has low computational complexity and good scalability. The overall parsing algorithm has a time complexity of O(n), supports linear processing of infinitely long streaming data, and is easily extended to more complex structured data formats.

[0063] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of the data extraction device 30 is given below. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a functional block diagram of a data extraction device provided in an embodiment of the present invention. It should be noted that the data extraction device 30 provided in this embodiment includes: a parsing module 301, a judgment module 302, a correction module 303, and an extraction module 304.

[0064] Parsing module 301 is used to write the streaming text sequentially into the global buffer, and store the currently parsed key and value into the key buffer and value buffer respectively; The judgment module 302 is used to determine whether the key that should be extracted needs to be corrected when a key end marker is detected in the global buffer. The correction module 303 is used to determine the corrected key based on the key in the current cache if the result of the judgment module 302 is yes, and replace the key that should be extracted with the corrected key; otherwise, no processing is performed. The extraction module 304 is used to bind the value of the current value buffer with the corrected key and output it when a value end marker is detected in the global buffer, and to clear the current key buffer and the current value buffer until a streaming output termination marker is detected.

[0065] It is understandable that the parsing module 301, the judgment module 302, the correction module 303, and the extraction module 304 can be executed collaboratively. Figure 2 Each step in the process is used to achieve the corresponding technical effect.

[0066] It should be noted that the data extraction device 30 provided in this embodiment of the invention can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this embodiment of the invention are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0067] Optionally, the above modules can be stored in the form of software or firmware. Figure 4 The memory shown is in the operating system (OS) of the electronic device 40, and can be... Figure 4 The processor executes the commands. Meanwhile, the data and program code required to execute these modules can be stored in memory.

[0068] Please see Figure 4 , Figure 4 The structural block diagram of the electronic device provided in the embodiment of the present invention includes a memory 401, a processor 402, and a communication interface 403. The memory 401, processor 402, and communication interface 403 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0069] Optionally, the bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0070] In this embodiment of the invention, the processor 402 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly manifested as execution by the hardware processor, or execution by a combination of hardware and software modules within the processor. The software modules may reside in the memory 401, and the processor 402 reads the program instructions from the memory 401 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0071] In this embodiment of the invention, the memory 401 can be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or it can be volatile memory, such as RAM. The memory can also be any other medium capable of carrying or storing desired executable program code having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory in this embodiment of the invention can also be a circuit or any other device capable of implementing a storage function for storing instructions and / or data.

[0072] The memory 401 can be used to store software programs and modules, such as the instructions / modules of the data extraction device 30 provided in this embodiment of the invention. These can be stored in the memory 401 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device 40. The processor 402 executes various functional applications and data processing by executing the software programs and modules stored in the memory 401. The communication interface 403 can be used for signaling or data communication with other node devices.

[0073] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device 4 may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0074] Based on the above embodiments, the present invention also provides a readable storage medium storing a computer program. When the computer program is executed by a computer, it causes the computer to perform the data extraction method provided in the above embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0075] Based on the above embodiments, the present invention also provides a program product, which includes a computer program. The processor can execute the computer program to implement the data extraction method provided in the embodiments of the present invention. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0076] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0077] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs.

[0078] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0079] It should be noted that if the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code.

Claims

1. A data extraction method, characterized in that, The method includes: The streaming text is written sequentially to the global buffer, and the currently parsed keys and values ​​are stored in the key buffer and value buffer, respectively. When a key end marker is detected in the global buffer, it is determined whether the key expected to be extracted needs to be corrected. The key expected to be extracted is obtained by: determining the number of times the key end marker is detected; and selecting the predefined key in the predefined key list that corresponds to the number of occurrences as the key expected to be extracted. If so, determine the corrected key based on the key in the current key cache and replace the key that should be retrieved with the corrected key; otherwise, do nothing. When a value end marker is detected in the global buffer, the value in the current value buffer is bound to the corrected key and output, and the current key buffer and the current value buffer are cleared until a streaming output termination marker is detected.

2. The data extraction method according to claim 1, characterized in that, Determine whether the currently expected keys to be extracted need to be corrected, including: If the key that is expected to be extracted is in the list of parsed keys, then the key that is expected to be extracted needs to be corrected; otherwise, no correction is needed.

3. The data extraction method according to claim 1, characterized in that, Determine the corrected key based on the key in the current key cache, and replace the key that should be retrieved with the corrected key, including: Determine the edit distance between the keys in the current key buffer and each predefined key in the predefined key list; The predefined key with the smallest edit distance is used as the corrected key, replacing the key that should be extracted.

4. The data extraction method according to claim 2, characterized in that, Also includes: The corrected key is recorded in the list of parsed keys.

5. The data extraction method according to any one of claims 1-4, characterized in that, The streamed output text is written to the global buffer sequentially, and the currently parsed keys and values ​​are stored in the key buffer and value buffer respectively, including: The current parsing stage is identified by detecting the structure delimiters in the global buffer; When the structure delimiter is a predefined key delimiter, the current parsed content is confirmed to be a key, and the key is written to the key buffer; When the structure delimiter is a predefined value delimiter, the current parsed content is confirmed to be a value, and the value is written to the value buffer.

6. A data extraction device, characterized in that, The module consists of a parsing module, a judgment module, a correction module, and an extraction module. The parsing module is used to write the streaming text sequentially to the global buffer and store the currently parsed keys and values ​​into the key buffer and value buffer, respectively. The judgment module is used to determine whether the key expected to be extracted needs to be corrected when a key end marker is detected in the global buffer; wherein, the key expected to be extracted is obtained by: determining the number of times the key end marker is detected; and selecting the predefined key corresponding to the number of occurrences from the predefined key list as the key expected to be extracted. The correction module is used to determine the corrected key based on the key in the current key cache if the result of the judgment module is yes, and replace the key that should be extracted with the corrected key; otherwise, no processing is performed. The extraction module is used to bind the value of the current value buffer with the corrected key and output it when a value end marker is detected in the global buffer, and to clear the current key buffer and the current value buffer until a streaming output termination marker is detected.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the data extraction method according to any one of claims 1-5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data extraction method as described in any one of claims 1-5.

9. A program product, characterized in that, The program product includes a computer program, which a processor can execute to implement the data extraction method according to any one of claims 1-5.

Images