Data processing methods, apparatus, equipment and storage media
By segmenting structured text data and counting the number of nesting levels, and using a state stack mechanism for character-by-character parsing, the parsing anomaly problem of complex nested text data is solved, achieving efficient and accurate field segmentation and data processing.
Patent Information
- Application Number
- CN202610901958.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-22
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies are prone to errors in parsing structured text data, especially complex nested text data, such as incorrect field segmentation and cascading parsing anomalies across fragments, resulting in low parsing accuracy and efficiency.
The structured text data is divided into multiple data fragments with line boundaries aligned. The ending level of each fragment is determined by counting the nesting levels of quotation marks. The ending level of the previous fragment is configured as the starting level of the next fragment. A state stack mechanism is used to perform character-by-character parsing and field segmentation in the nodes to ensure the continuity and accuracy of the parsing state.
It improves the parsing accuracy of structured text data in distributed processing scenarios, avoids field misalignment and data truncation, and enhances the efficiency and quality of data processing.
Smart Images

Figure CN123115126A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology
[0002] Structured text data, such as CSV (Comma-Separated Values) format data, is used as a primary carrier for data exchange and data warehousing due to its simple structure and wide compatibility. As business data volume continues to grow, efficient and accurate parsing and field extraction of massive amounts of structured text data has become a fundamental requirement in the field of big data processing.
[0003] Currently, either a single-machine parsing method or a distributed parallel parsing method is commonly used. The single-machine parsing method mainly relies on Boolean state inversion logic, using a single variable to record whether the current state is enclosed in quotes. When a quote character is encountered, it is inverted to distinguish between characters within a field and field separators. The distributed parallel parsing method, on the other hand, divides large files into several independent physical fragments, which are read and executed by multiple nodes in the cluster. Finally, the parsing results from each node are concatenated and summarized to overcome the computation and transmission bottlenecks of single-machine processing.
[0004] However, current parsing methods have limitations when processing data containing complex nested text. For example, parsing logic based on Boolean state inversion only represents the current parsing context with a simple binary state. When faced with multi-level nested quotes, it cannot distinguish the boundaries of different levels of quotes, and is prone to misinterpreting the closure of an inner quote as the termination of the entire field, resulting in the complete field being incorrectly cut in the middle. In addition, in the existing distributed sharded parsing architecture, the parsing processes of each node are isolated from each other, and their parsing context exists only as a transient state within the process. When a data shard happens to be truncated inside an unclosed nested quote, the shard node that receives the subsequent text cannot perceive the nesting depth of the preceding text and can only forcibly start parsing with the default initial state. This causes the text originally wrapped in quotes to be incorrectly cut according to the conventional delimiter logic, thus triggering a chain of parsing anomalies across shards. Summary of the Invention
[0005] This invention provides a data processing method, apparatus, device, and storage medium to solve the technical problem that complex nested text is prone to field misslicing and cascading parsing anomalies across fragments when parsing structured text data.
[0006] This invention provides a data processing method, comprising the following steps: The structured text data is divided into multiple data segments in sequence, and the boundaries of each data segment are aligned with the line boundaries. Each data segment is scanned and the nesting level of quotation marks is counted to obtain the ending level of each data segment. According to the order of the data shards, the ending level of the (N-1)th data shard is configured as the starting level of the Nth data shard, and each data shard and its corresponding starting level are assigned to the corresponding node in the cluster, where N is an integer greater than 1 and the starting level of the first data shard is configured as the initial value. In each node, the starting level of the data fragment corresponding to the node is loaded into the state stack, and the data fragment is parsed character by character and segmented field by field based on the current level to obtain the parsing result of the data fragment; wherein, the state stack is pushed onto the stack and the current level is incremented when the current parsed character is an open quotation mark, and popped from the stack and the current level is decremented when the current parsed character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsed character is a field separator; The parsing results of each data fragment are merged according to the order of the data fragments.
[0007] According to the data processing method provided by the present invention, the step of parsing the data fragments character by character and segmenting them field by field based on the current level to obtain the parsing result of the data fragments includes: During the character-by-character parsing process, if the currently parsed character is a preset invalid character, the invalid character is discarded and the parsing positions of other characters are kept unchanged. When the current level is the initial value and the current parsed character is a field separator, the current buffered character stream is segmented into fields to obtain the segmented fields; The cutting fields are sequentially cleaned in a preset priority order, performing first-level cleaning, second-level cleaning, and third-level cleaning to obtain the parsing results. The first level of cleaning includes at least one of null value filling and newline character replacement; the second level of cleaning includes at least one of space trimming and case conversion based on the processing result of the first level of cleaning; and the third level of cleaning includes at least one of data format standardization, data anonymization, and field length truncation based on the processing result of the second level of cleaning.
[0008] According to the data processing method provided by the present invention, the step of scanning each of the data fragments and counting the nesting levels of quotation marks to obtain the ending level of each data fragment includes: When scanning each of the data segments, if the currently scanned character is an open quotation mark, the scanning level is incremented. If the currently scanned character is a closed quotation mark, the scanning level is decreased, and the nested quotation mark state is exited when the scanning level is decreased to the initial value. The scanning level at which the data fragment ends scanning is determined to be the end level of the data fragment.
[0009] According to the data processing method provided by the present invention, the step of allocating each data shard and its corresponding starting level to the corresponding node in the cluster includes: For each data shard, extract the data volume of the data shard and determine the level difference between the end level and the start level of the data shard; Based on the data volume and the hierarchical difference, the task processing volume corresponding to the data shard is determined; Determine the current available resources for each node in the cluster; Based on the task processing volume and the current available resources, each data shard and its corresponding starting level are allocated to the corresponding node through a target allocation strategy. The target allocation strategy is to allocate resources based on the positive correlation between the task processing volume and the current available resources, so that the expected load difference of each node after allocation is less than a preset load threshold.
[0010] According to the data processing method provided by the present invention, after configuring the end level of the (N-1)th data fragment as the start level of the Nth data fragment according to the order of the data fragments, the method further includes: Based on the identifiers of each data fragment, the starting level, and the ending level, a global state chain is constructed, and based on the dependency relationships between adjacent data fragments in the global state chain, a directed acyclic dependency graph is constructed. During the process of parsing and field-by-field segmentation of the data fragments based on the current level, if a target data fragment that fails to be parsed is detected, the downstream data fragments of the target data fragments are determined based on the directed acyclic dependency graph. The target data shard is redistributed to the corresponding node in the cluster, and the process returns to load the starting level of the data shard corresponding to the node into the state stack, and performs character-by-character parsing and field splitting on the data shard based on the current level. After the target data fragment is parsed, the downstream data fragments of the target data fragment are scanned, and the nesting level of quotation marks is counted to update the end level and the corresponding start level of the downstream data fragment.
[0011] According to the data processing method provided by the present invention, the step of parsing the data fragments character by character and segmenting them field by field based on the current level to obtain the parsing result of the data fragments includes: Extract the target row data from the data slices sequentially according to the row boundaries; The current level of the previous row of data at the end position is loaded into the state stack, and the target row of data is parsed character by character and segmented based on the state stack; If the target row data triggers a preset exception condition during the parsing process, an exception identifier corresponding to the target row data is generated, and the target row data is isolated and stored. The preset exception condition includes that when parsing to the end of the target row data, the current level has not been restored to the initial value, or the number of fields after field splitting is inconsistent with the preset field number threshold. The parsing results of the data shards are obtained based on the parsing results of the target row data that is not marked with the anomaly identifier.
[0012] According to the data processing method provided by the present invention, merging the parsing results of each of the data fragments in the order of the data fragments includes: According to the order of the data fragments, extract the target end level of the N-1th data fragment when the last line of data is parsed, and the target start level of the Nth data fragment when the first line of data is parsed. If the target end level and the target start level are the same level, then the parsing result of the Nth data segment is merged into the parsing result of the (N-1)th data segment; If the target end level and the target start level are different levels, then a continuity check anomaly identifier corresponding to the Nth data fragment is generated.
[0013] The present invention also provides a data processing apparatus, comprising: The segmentation scanning module is used to divide structured text data into multiple data segments in sequence, with the boundaries of each data segment aligned with the line boundaries, and to scan each data segment separately, counting the nesting levels of quotation marks to obtain the ending level of each data segment. The data distribution module is used to configure the end level of the (N-1)th data fragment as the start level of the Nth data fragment according to the order of the data fragments, and to allocate each data fragment and its corresponding start level to the corresponding node in the cluster, where N is an integer greater than 1 and the start level of the first data fragment is configured as an initial value. The parsing module is used to load the starting level of the data fragment corresponding to each node into the state stack, and to perform character-by-character parsing and field segmentation on the data fragment based on the current level to obtain the parsing result of the data fragment; wherein, the state stack performs push operation and increments the current level when the current parsing character is an open quotation mark, and pops the current level and decrements the current level when the current parsing character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsing character is a field separator; The merging module is used to merge the parsing results of each of the data fragments according to the order of the data fragments.
[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the data processing method as described above.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above.
[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data processing methods described above.
[0017] The data processing method, apparatus, device, and storage medium provided by this invention divides structured text data into multiple data fragments with line boundaries aligned. After division, each data fragment is scanned to count the nesting level of quotation marks, thereby obtaining the ending level of each data fragment. Then, according to the fragmentation order, the ending level of the previous data fragment is configured as the starting level of the next data fragment, and the starting level is sent to each node along with the data fragments. During parsing, each node loads the starting level into the state stack as the initial parsing state. The current level is maintained through the push and pop mechanism of the state stack, and field splitting is triggered based on whether the current level is the initial value. This avoids field splitting misalignment and data truncation, and improves the parsing accuracy of non-standard structured text data in distributed processing scenarios. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the present invention or the conventional ones, the drawings used in the embodiments or conventional descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0019] Figure 1This is a flowchart illustrating the data processing method provided in an embodiment of the present invention.
[0020] Figure 2 This is a schematic diagram of the structure of the data processing device provided in an embodiment of the present invention.
[0021] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0023] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0024] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0025] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of the present invention, as shown below. Figure 1As shown, the method includes the following steps 110, 120, 130 and 140.
[0026] Step 110: Divide the structured text data into multiple data segments in sequence, with the boundaries of each data segment aligned with the line boundaries. Scan each data segment separately and count the nesting levels of quotation marks to obtain the ending level of each data segment.
[0027] The source of the structured text data can be a distributed file system, a local file system, an object storage service, an FTP (File Transfer Protocol) service, etc., and the data format can be CSV or other text formats that organize fields with specific delimiters, without any restrictions.
[0028] In this embodiment, the structured text data file is cut into multiple data fragments according to certain rules, and each data fragment is assigned to each node in the cluster for parallel processing, thereby achieving efficient processing of large-scale data.
[0029] In one example, the size of the data shard can be determined based on the number of cluster nodes, the computing resources of each node, and the total amount of input data; in another example, the size of the data shard can also be dynamically adjusted within a preset range.
[0030] In this embodiment, when the structured text data is divided into multiple data segments in sequence, the boundaries of each data segment need to be aligned with the line boundaries, that is, the starting position of each data segment is the beginning of a line of data, and the ending position is the end of a line of data.
[0031] Specifically, the position of the newline character can be found by streaming the byte stream, and the end position of the fragment can be corrected based on the newline character, thereby ensuring that a single line of data is not split into two different data fragments.
[0032] After the data is partitioned, each data partition is scanned to count the nesting level of quotation marks, so as to obtain the ending level of each data partition.
[0033] It should be noted that in structured text data, field content may contain nested quotation marks. When structured text data is split into multiple data fragments, the end of a data fragment may be in a state of unclosed nested quotation marks, that is, the number of nested quotation marks in the last line of the fragment is not zero. If the current nested quotation mark state is not identified during subsequent parsing of the fragment, it will cause the parsing state to break, which will lead to field splitting misalignment.
[0034] In one example, a scan level counter is started during the scanning process, with an initial value of 0, indicating that there is no nested quotation mark state at present. The data fragment is scanned character by character. If the currently scanned character is an open quotation mark, the scan level counter is incremented; if the currently scanned character is a closed quotation mark, the scan level counter is decremented. When the scan level counter decrements to the initial value, it indicates that the current nested quotation mark state has been completely closed. When the data fragment scanning is completed, the current value of the scan level counter is the end level of the data fragment.
[0035] Step 120: According to the order of the data shards, configure the end level of the (N-1)th data shard as the start level of the Nth data shard, and assign each data shard and its corresponding start level to the corresponding node in the cluster, where N is an integer greater than 1 and the start level of the first data shard is configured as the initial value.
[0036] In this embodiment, based on the end level of each data shard, the starting level is assigned to each data shard in the order in which the data shards are arranged in the original data. The starting level indicates the number of quote nesting state levels that the data shard should inherit when it starts parsing, so that when the nodes in the cluster parse the data shard, they can start from the correct quote nesting state and realize the continuous transmission of parsing state across shards.
[0037] Specifically, the end level of the (N-1)th data fragment is configured as the start level of the Nth data fragment. That is, if the nesting level of the quotes at the end position of the (N-1)th data fragment is L, then when the Nth data fragment starts parsing, its nesting level should continue from L. In addition, for the first data fragment, since there are no other data fragments before it, its start level is configured as an initial value, such as 0, indicating that the parsing starts from the initial state without any nested quotes.
[0038] For example, structured text data is divided into four data fragments: fragment 1, fragment 2, fragment 3, and fragment 4. The ending level of fragment 1 is 0, the ending level of fragment 2 is 2, the ending level of fragment 3 is 1, and the ending level of fragment 4 is 0. Therefore, the starting level of fragment 1 is 0, which is the initial value; the starting level of fragment 2 is 0, which is the ending level of fragment 1; the starting level of fragment 3 is 2, which is the ending level of fragment 2; and the starting level of fragment 4 is 1, which is the ending level of fragment 3.
[0039] After configuring the starting level of each data shard, each data shard and its corresponding starting level are assigned to the corresponding nodes in the cluster. A node is a computing unit in the cluster that undertakes specific data processing tasks. Each node can be a physical machine or a virtual computing instance in the cluster.
[0040] In one example, data can be sharded and scheduled to nodes with relatively low loads based on the current resource usage of each node, in order to achieve overall load balancing of the cluster.
[0041] Step 130: In each node, the starting level of the data fragment corresponding to the node is loaded into the state stack, and the data fragment is parsed character by character and segmented field by field based on the current level to obtain the parsing result of the data fragment; wherein, the state stack is pushed onto the stack and the current level is incremented when the current parsed character is an open quotation mark, and popped from the stack and the current level is decremented when the current parsed character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsed character is a field separator.
[0042] After receiving the allocated data fragment and its corresponding starting level, each node loads the starting level into the state stack as the initial state of the current parsing state, and then performs character-by-character parsing and field segmentation on the data fragment.
[0043] It should be noted that the state stack is an explicit data structure used to maintain the nested quotation marks state during the parsing process. Its stack elements record information such as the current parsing level; the current level refers to the number of nested quotation marks corresponding to the current top of the state stack, reflecting the depth of quotation mark nesting at the current parsing position.
[0044] During character-by-character parsing, if the currently parsed character is an open quote, the state stack is pushed onto the stack, and the current level is incremented, indicating that a deeper level of quote nesting has been entered. If the currently parsed character is a closed quote, the state stack is popped from the stack, and the current level is decremented, indicating that the current level of quote nesting has been exited to the previous level. If the current level is the initial value and the currently parsed character is a field separator, a field splitting operation is performed, and the currently buffered character sequence is output as an independent field. If the current level is not the initial value, it indicates that the current parsing position is inside the quote nesting. In this case, even if a field separator is parsed, field splitting is not triggered, and the field separator is treated as part of the field content.
[0045] In this embodiment, through the aforementioned state stack mechanism, the node can accurately distinguish between field separators and field separators based on inheriting the previous data sharding parsing state. Thus, even in the presence of nested quotation marks and field commas, it can achieve precise field segmentation and obtain the data sharding parsing results.
[0046] For example, if the initial level of a data shard is 2, the node sets the current level in the stack to 2 when initializing the state stack, and then begins to process the byte stream in the data shard character by character. If an open quotation mark is encountered, it is pushed onto the stack and the current level is updated to 3. If a closing quotation mark is subsequently encountered, it is popped from the stack and the current level is updated to 2. Processing continues, and if another closing quotation mark is encountered, it is popped from the stack and the current level is updated to 1. If another closing quotation mark is encountered, it is popped from the stack and the current level is updated to 0. At this point, if a comma separator is encountered, the current level is 0, which is the initial value, triggering a field splitting operation, and the buffered character sequence is output as a complete field.
[0047] Step 140: Merge the parsing results of each data fragment according to the order of the data fragments.
[0048] In this embodiment, after each node completes the parsing of its corresponding data fragment, the parsing results of each data fragment are merged sequentially according to the order in which the data fragments are arranged in the original data, thereby obtaining a complete parsing result that is consistent with the overall order of the original structured text data.
[0049] In one example, the merged parsing results can retain the identification information of each data shard and its corresponding row to support data traceability requirements. In addition, the merged parsing results can be output as a data file in a preset format, such as CSV, Parquet, ORC, etc., and written to the target data warehouse for subsequent data analysis.
[0050] The data processing method provided in this invention divides structured text data into multiple data fragments with line boundaries aligned. After division, each data fragment is scanned to count the nesting levels of quotation marks, thereby obtaining the ending level of each data fragment. Then, according to the fragmentation order, the ending level of the previous data fragment is configured as the starting level of the next data fragment, and the starting level is sent to each node along with the data fragments. During parsing, each node loads the starting level into the state stack as the initial parsing state. The current level is maintained through the push and pop mechanism of the state stack, and field splitting is triggered based on whether the current level is the initial value. This avoids field splitting misalignment and data truncation, and improves the parsing accuracy of non-standard structured text data in distributed processing scenarios.
[0051] In some embodiments, the step of parsing the data fragments character by character and segmenting them field by field based on the current level to obtain the parsing results of the data fragments includes: During the character-by-character parsing process, if the currently parsed character is a preset invalid character, the invalid character is discarded and the parsing positions of other characters are kept unchanged. When the current level is the initial value and the current parsed character is a field separator, the current buffered character stream is segmented into fields to obtain the segmented fields; The cutting fields are sequentially cleaned in a preset priority order, performing first-level cleaning, second-level cleaning, and third-level cleaning to obtain the parsing results. The first level of cleaning includes at least one of null value filling and newline character replacement; the second level of cleaning includes at least one of space trimming and case conversion based on the processing result of the first level of cleaning; and the third level of cleaning includes at least one of data format standardization, data anonymization, and field length truncation based on the processing result of the second level of cleaning.
[0052] Here, invalid characters refer to characters that do not have valid semantics for the data content and may interfere with subsequent processing, such as invisible characters with ASCII code values less than 32 and not spaces, zero-width characters, and invalid UTF-8 encoded characters caused by encoding anomalies.
[0053] In practical applications, the types of invalid characters to be filtered can be pre-configured in the configuration file.
[0054] It should be noted that when a node determines that the currently parsed character is invalid, it discards this invalid character, that is, it skips it in the character stream without replacing or modifying it. This ensures that the parsing position of the remaining characters in the data fragment, that is, the offset of the parsing pointer, will not change due to the character operation, thereby ensuring the accuracy of the current level in the state stack and the correctness of the field cutting position.
[0055] After completing character-level filtering, the node continues to maintain the state stack and track the current level. If and only if the current level is the initial value, that is, the current parsing position is not in any nested quotation marks and the current parsing character is a field separator, the character sequence accumulated in the current character buffer is output as a complete cutting field, and the buffer is cleared to prepare to receive the character content of the next field. If the current level is not the initial value, the current parsing position is inside nested quotation marks, the field separator is treated as part of the field content and written to the buffer, and no field cutting operation is triggered.
[0056] After obtaining the cutting fields, the node performs three levels of cleaning operations in sequence according to the preset priority order. The cleaning rules of the cleaning operations can be defined through the configuration file, and the effective scope of each cleaning rule can be specified by field index to meet the differentiated processing needs of different fields.
[0057] Specifically, the first-level cleaning performs basic processing on the cut field that does not depend on the state of other fields, including but not limited to: Null value filling: For fields where the content is empty, i.e., the field content is an empty string or missing, fill with a preset default value, such as using \N as a null value placeholder; Newline character replacement: Replaces newline characters in the content of the split field with preset characters, such as replacing newline characters with spaces.
[0058] The second-level cleaning is performed based on the results of the first-level cleaning, and includes, but is not limited to: Space trimming: Removes extra spaces at the beginning and end of field content to standardize field boundaries; Case conversion: Performs case conversion on the content of a specified field, such as converting the content of the region field to uppercase.
[0059] The third-level cleaning is performed based on the results of the second-level cleaning, including but not limited to: Data format standardization: Perform format conversion operations on the content of the segmented fields, such as converting multiple mixed formats in the date field into a unified target standard format to ensure that the data is consistent in format after being entered into the warehouse; Data masking: Perform masking processing on fields containing sensitive information, such as masking the middle four digits of a mobile phone number field to meet data security compliance requirements; Field length truncation: Performs a truncation operation on field content that exceeds a preset character length threshold, and adds a preset suffix to the end of the truncated content to avoid excessively long fields affecting write operations in the target data warehouse.
[0060] In this embodiment, the three-level cleaning operations are all performed after the field segmentation is completed, without modifying the original position and order of the characters. Therefore, they will not cause any reverse interference to the field segmentation results in the parsing stage. After each level of cleaning operation is completed, the set of fields that have undergone cleaning is the parsing result of the row of data, which is further summarized into the parsing result of data sharding.
[0061] The data processing method provided in this embodiment of the invention ensures that the parsing pointer does not deviate due to character filtering operations by discarding invalid characters during the character-by-character parsing process; after the field is segmented, the layered cleaning operation is performed according to a preset priority order, thereby decoupling parsing and cleaning. While ensuring the accuracy of field segmentation, it also achieves the standardization of non-standard field content, thus improving the data quality of the final parsing result.
[0062] In some embodiments, scanning each of the data fragments and counting the nesting levels of quotation marks to obtain the ending level of each data fragment includes: When scanning each of the data segments, if the currently scanned character is an open quotation mark, the scanning level is incremented. If the currently scanned character is a closed quotation mark, the scanning level is decreased, and the nested quotation mark state is exited when the scanning level is decreased to the initial value. The scanning level at which the data fragment ends scanning is determined to be the end level of the data fragment.
[0063] In this embodiment, each data slice is pre-scanned first, and the character content in the data slice is traversed byte by byte. A scan level counter is maintained to track the depth of quotation mark nesting at the current scan position in real time. The initial value of the scan level can be set to 0, indicating that no quotation mark nesting state has been entered at the moment.
[0064] During the scanning process, an open quote indicates that the current scanned character has entered a new level of quote nesting. If the current scanned character is determined to be an open quote, the scan level counter is incremented, increasing the scan level by 1 from the current value, indicating that the current scanned position has entered a deeper level of quote nesting.
[0065] Similarly, a closed quotation mark refers to the quotation character that ends the current nested quotation range, that is, the closing symbol that matches the open quotation mark. If the currently scanned character is determined to be a closed quotation mark, the scan level counter will be decremented, reducing the scan level by 1 from the current value, indicating that the current scan position has exited the current nested quotation level.
[0066] It should be noted that in this embodiment, the nested quotation mark state is exited only when the scanning level decreases to the initial value, which can improve the accuracy of level judgment in multi-level nested quotation mark scenarios.
[0067] It should be understood that when determining whether a character is within quotation marks based on a Boolean flag, the state of the Boolean flag is reversed every time a quotation mark is read. It can only indicate whether a character is within quotation marks, but cannot record how many levels of quotation marks are nested, making it impossible to distinguish between single-level and multi-level nested quotation marks. In contrast, this embodiment uses a scanning level counter. Only when the count value of the scanning level counter drops to the initial value is it considered that all levels of quotation marks have been closed. This allows for accurate differentiation of different depths of quotation mark nesting, ensuring the accuracy of level statistics.
[0068] After scanning all characters of a data fragment, the current value of the scan level counter is the end level of that data fragment. If the end level is the initial value, it indicates that the end of the data fragment is in a state without nested quotes, and the next data fragment will be parsed starting from the initial state of nested quotes. If the end level is greater than the initial value, it indicates that the end of the data fragment is in a state of nested quotes at some level, such as an unclosed quote across fragments, and the next data fragment needs to continue parsing from that level.
[0069] In one example, the pre-scan results of each data shard can also be persistently stored in a temporary directory of the distributed file system to support recovery from the breakpoint when the scan task is interrupted, avoiding repeated scan operations on data shards that have already been scanned.
[0070] The data processing method provided in this embodiment of the invention improves the accuracy of identifying the end level of each data shard by maintaining a scan level counter and exiting the quote nesting state only when the level drops to the initial value. This provides an accurate data foundation for the subsequent configuration of the start level across shards and ensures the accuracy of quote nesting state transmission throughout the entire distributed parsing process.
[0071] In some embodiments, the step of allocating each data shard and its corresponding starting level to the corresponding node in the cluster includes: For each data shard, extract the data volume of the data shard and determine the level difference between the end level and the start level of the data shard; Based on the data volume and the hierarchical difference, the task processing volume corresponding to the data shard is determined; Determine the current available resources for each node in the cluster; Based on the task processing volume and the current available resources, each data shard and its corresponding starting level are allocated to the corresponding node through a target allocation strategy. The target allocation strategy is to allocate resources based on the positive correlation between the task processing volume and the current available resources, so that the expected load difference of each node after allocation is less than a preset load threshold.
[0072] After completing the initial level configuration for each data shard, extract the data volume and level difference corresponding to each data shard to evaluate the processing complexity of that data shard.
[0073] The data volume refers to the number of bytes or characters contained in the data slice. The larger the data volume, the more computing resources and time are required to process the data slice. The data volume of the data slice can be obtained from the metadata of the data slice.
[0074] The level difference refers to the difference between the ending level and the starting level of the data partition. The level difference reflects the complexity of the nested quotation structure inside the data partition. The larger the level difference, the more nested quotation changes there are inside the data partition. The more frequent the push and pop operations of the state stack are during parsing, the higher the consumption of computing resources. The smaller the level difference or even zero, the simpler the nested quotation structure inside the data partition, and the lower the parsing overhead.
[0075] In this embodiment, the task processing volume corresponding to each data segment is determined based on the data volume and hierarchical difference of the data segments. The task processing volume is a quantitative estimate of the comprehensive computing resources required to process the data segment.
[0076] In one example, the task processing volume can be calculated by weighting the data volume and the hierarchical difference with different weights. In addition, when determining the task processing volume of data shards, other evaluation indicators can also be considered, such as the number of invalid characters filtered in the pre-scanning stage, without any restrictions.
[0077] In addition, this embodiment will also collect the current available resources of each node in the cluster in real time, including but not limited to: CPU utilization, memory utilization, storage IO bandwidth and other resource indicators of each node, in order to evaluate the computing power of each node that can currently undertake new tasks.
[0078] Based on this, the node allocation for data sharding is completed by combining the processing volume of each data sharding task and the current available resources of each node through a target allocation strategy.
[0079] Specifically, the target allocation strategy prioritizes allocating data shards with high task processing volume to nodes with more available resources, while data shards with low task processing volume can be allocated to nodes with relatively limited resources or to fill the idle resources of nodes.
[0080] When implementing the target allocation strategy, the optimization objective is to ensure that the expected load difference between nodes after task allocation is less than a preset load threshold. The allocation scheme is iteratively adjusted to make the load of each node in the cluster tend to be balanced. In some embodiments, the preset load threshold can be 15%, which requires that the load difference between any two nodes in the cluster after allocation does not exceed 15%.
[0081] The data processing method provided in this invention quantifies the task processing volume of each data fragment by comprehensively considering the data volume and hierarchical difference of the data fragments, and combines the current available resources of the cluster nodes to schedule tasks according to the target allocation strategy of positive correlation between task processing volume and available resources. This keeps the load difference of each node in the cluster within a preset load threshold range, thereby improving the overall processing efficiency of distributed parsing tasks.
[0082] In some embodiments, after configuring the end level of the (N-1)th data fragment as the start level of the Nth data fragment according to the order of the data fragments, the method further includes: Based on the identifiers of each data fragment, the starting level, and the ending level, a global state chain is constructed, and based on the dependency relationships between adjacent data fragments in the global state chain, a directed acyclic dependency graph is constructed. During the process of parsing and field-by-field segmentation of the data fragments based on the current level, if a target data fragment that fails to be parsed is detected, the downstream data fragments of the target data fragments are determined based on the directed acyclic dependency graph. The target data shard is redistributed to the corresponding node in the cluster, and the process returns to load the starting level of the data shard corresponding to the node into the state stack, and performs character-by-character parsing and field splitting on the data shard based on the current level. After the target data fragment is parsed, the downstream data fragments of the target data fragment are scanned, and the nesting level of quotation marks is counted to update the end level and the corresponding start level of the downstream data fragment.
[0083] In this embodiment, the identifiers, start levels, and end levels of each data fragment are integrated according to the order of the data fragments to construct a global state chain. The global state chain is an ordered data structure, in which each record corresponds to a data fragment and includes the identifier, start level, end level, and the identifier of the preceding data fragment that it depends on.
[0084] In one example, a checksum, such as an MD5 checksum, can be calculated for each record in the global state chain, and the global state chain can be persistently stored in a distributed file system to support subsequent state chain integrity verification and task failure recovery.
[0085] Specifically, after constructing the global state chain, a directed acyclic dependency graph is constructed based on the dependency relationship between adjacent data fragments in the global state chain. Each node in the directed acyclic dependency graph corresponds to a data fragment, and the directed edges represent the state dependency relationship between fragments.
[0086] During the data shard parsing task on each node, the task execution status of each node is continuously monitored. If a node fails while parsing a data shard, such as a node crash, parsing timeout, or an unrecoverable error during the parsing process, the data shard is marked as the target data shard, i.e., the data shard that failed to be parsed.
[0087] After determining the target data shard, query the directed acyclic dependency graph. Starting from the target data shard, traverse along the directed edges to determine the set of all downstream data shards that are located after the target data shard in terms of dependency relationship, and whose starting level directly or indirectly depends on the ending level of the target data shard.
[0088] The target data shard is marked as pending reallocation. After confirming that there are no incomplete preceding dependent shards in the directed acyclic dependency graph, the target data shard is reallocated to an available node in the cluster. During reallocation, based on the target data shard's starting level data already stored in the global state chain, the starting level is sent to the newly allocated node along with the sharding task. The new node loads the starting level onto the state stack based on this and continues to perform character-by-character parsing and field splitting operations.
[0089] In one example, for a target data shard that fails to be parsed in a single attempt, a limited number of retries can be performed automatically, such as a maximum of 3 retries. If the parsing still fails after the number of retries has been exhausted, the data shard is marked as an abnormal shard and a fault log is recorded for subsequent manual troubleshooting.
[0090] Furthermore, after the target data shard is re-parsed on the newly allocated node, the scanning operation is re-executed for each downstream data shard in the affected downstream data shard set based on the directed acyclic dependency graph, so as to obtain the updated end level of each downstream data shard.
[0091] After updating the end level of each downstream data shard, the start and end level data of the corresponding records in the global state chain are updated synchronously to ensure the consistency and accuracy of the global state chain. The updated start level is then redistributed to the corresponding node to continue the parsing task.
[0092] The data processing method provided in this invention constructs a global state chain and a directed acyclic dependency graph to accurately describe the state dependencies between data shards. When a target data shard that has failed to be parsed is detected, the range of affected downstream shards is determined using the directed acyclic dependency graph. After the target data shard is re-parsed, only the affected downstream shards are updated. This achieves stability and recoverability of large-scale distributed data parsing tasks when local node failures occur, avoiding interruption or full recalculation of the entire parsing task due to a single point of failure.
[0093] In some embodiments, the step of parsing the data fragments character by character and segmenting them field by field based on the current level to obtain the parsing results of the data fragments includes: Extract the target row data from the data slices sequentially according to the row boundaries; The current level of the previous row of data at the end position is loaded into the state stack, and the target row of data is parsed character by character and segmented based on the state stack; If the target row data triggers a preset exception condition during the parsing process, an exception identifier corresponding to the target row data is generated, and the target row data is isolated and stored. The preset exception condition includes that when parsing to the end of the target row data, the current level has not been restored to the initial value, or the number of fields after field splitting is inconsistent with the preset field number threshold. The parsing results of the data shards are obtained based on the parsing results of the target row data that is not marked with the anomaly identifier.
[0094] In this embodiment, when a node starts parsing a data shard, it extracts the target row data to be parsed from the data shard line by line, using the row boundaries as separators. Here, the target row data refers to the complete row content to be parsed and processed in the current round of the data shard. Since the data shards are aligned with the row boundaries when they are divided, each row in the data shard can be extracted independently.
[0095] Before a node begins parsing each target row of data, it loads the current level of the previous row at its end position into the state stack as the initial level state for parsing this row. For the first row of data in a data shard, its parsing starting level is the starting level of that data shard. For each subsequent row of data in a data shard, its parsing starting level is the current level value in the state stack when the previous row of data has been parsed.
[0096] When parsing each target row of data, the node simultaneously performs anomaly detection. Preset anomaly conditions include, but are not limited to: When parsing to the end of the target line of data, the current level has not been restored to the initial value: that is, after the node has finished parsing all the characters of the line, the current level value in the state stack is still greater than the initial value. This situation indicates that there are unclosed nested quotation marks in the line of data, such as unmatched quotation marks. The number of fields after field splitting is inconsistent with the preset field number threshold: that is, after the data in this row is split, the number of fields obtained does not match the preset field number threshold. For example, the preset number of fields in the target dataset is 6, but the actual number of fields obtained after splitting is 5 or 7, indicating that there are missing or redundant fields in this row of data.
[0097] When the target row data meets the above-mentioned preset abnormal conditions, the node generates a corresponding abnormal identifier for the row data. The abnormal identifier records the abnormal type of the row data, such as unclosed nested quotation marks or mismatched field numbers. In addition, it also records the identifier of the data shard to which it belongs, the row number in the data shard, and the original content of the row, in order to support subsequent manual verification and data repair.
[0098] After the anomaly identifier is generated, the node isolates and stores the data in that row, that is, it diverts the abnormal row data to an independent dirty data storage path and does not include it in the normal parsing result output process.
[0099] After a node completes the parsing and anomaly detection of all rows in a data shard, it only summarizes the parsing results of rows without anomaly markers as the final parsing result for that data shard. Rows marked with anomaly markers have been isolated and stored and do not participate in the merging of data shard parsing results, thus ensuring that the parsing results output to subsequent merging steps are all qualified data that has been filtered for anomalies.
[0100] It should be noted that the isolated storage operation of abnormal rows in this embodiment only applies to the row data that triggers the preset abnormal conditions, and does not affect the parsing progress of other normal row data in the same data shard. That is, a single row abnormality does not interrupt the execution of the entire data shard or even the entire distributed parsing task.
[0101] The data processing method provided in this embodiment of the invention extracts target row data line by line according to row boundaries, takes the level at the end of the previous row as the starting level of the next row, and generates an anomaly identifier for row data that meets preset anomaly conditions and stores it in isolation. This achieves row-level anomaly isolation, ensuring that single-row data anomalies do not interrupt the parsing task of the entire data shard while ensuring the continuous transmission of cross-row parsing status.
[0102] In some embodiments, merging the parsing results of each data fragment according to the order of the data fragments includes: According to the order of the data fragments, extract the target end level of the N-1th data fragment when the last line of data is parsed, and the target start level of the Nth data fragment when the first line of data is parsed. If the target end level and the target start level are the same level, then the parsing result of the Nth data segment is merged into the parsing result of the (N-1)th data segment; If the target end level and the target start level are different levels, then a continuity check anomaly identifier corresponding to the Nth data fragment is generated.
[0103] In this embodiment, before merging the parsing results of each data fragment in sequence, the current level value in the state stack of the (N-1)th data fragment when its last line of data is parsed is extracted, i.e., the target end level, and the initial level value in the state stack of the Nth data fragment when its first line of data is parsed is extracted, i.e., the target start level.
[0104] It should be understood that the target end level is the actual level of the state stack when the last line of data in the (N-1)th data slice is parsed; the target start level is the start level inherited from the global state chain when the Nth data slice begins parsing. Since the start level of the Nth data slice is configured based on the end level obtained from the pre-scan of the (N-1)th data slice, the state level of the (N-1)th data slice after actual parsing should be consistent with the end level obtained from the pre-scan, assuming correct parsing.
[0105] When the target end level of the (N-1)th data fragment and the target start level of the Nth data fragment are the same level, i.e. their values are equal, indicating that the cross-fragment nested state transmission is accurate, it is determined that the parsing results of these two adjacent data fragments are continuous and consistent at the state level. The parsing result of the Nth data fragment is appended and merged into the parsing result of the (N-1)th data fragment in row order to complete the local merging.
[0106] When the target end level of the (N-1)th data fragment is not equal to the target start level of the Nth data fragment, that is, when their level values are different, it indicates that the start level inherited by the Nth data fragment during the parsing process is inconsistent with the level state at the actual end of the parsing of the (N-1)th data fragment. This indicates that there is a problem of discontinuous cross-fragment quotation mark nesting state between two adjacent data fragments.
[0107] At this point, a continuity verification anomaly identifier is generated for the Nth data shard. Information such as the data shard identifier, the actual value of the target end level, and the configuration value of the target start level corresponding to the anomaly is recorded. The anomaly identifier is written to the system log or anomaly record storage for subsequent manual verification or triggering of the automatic repair process.
[0108] In one example, after completing the continuity check of all adjacent data shards, multi-dimensional data checks can be performed on the merged complete parsing result set, including field number consistency checks, field length checks, and data type checks, to further ensure the integrity and standardization of the incoming data.
[0109] The data processing method provided in this embodiment of the invention improves the accuracy of inbound data by extracting the target end level and the configured target start level between adjacent data fragments during the result merging stage and comparing their consistency. When they are consistent, the results are merged; when they are inconsistent, a continuity verification anomaly flag is generated.
[0110] The data processing apparatus provided in the embodiments of the present invention will be described below. The data processing apparatus described below and the data processing method described above can be referred to in correspondence.
[0111] The data processing apparatus provided in the embodiments of the present invention, such as Figure 2 As shown, it includes the following modules: The segmentation scanning module 210 is used to divide structured text data into multiple data segments in sequence, with the boundaries of each data segment aligned with the line boundaries, and to scan each data segment separately, counting the nesting levels of quotation marks to obtain the ending level of each data segment. The data distribution module 220 is configured to configure the end level of the (N-1)th data fragment as the start level of the Nth data fragment according to the order of the data fragments, and to allocate each data fragment and its corresponding start level to the corresponding node in the cluster, wherein N is an integer greater than 1 and the start level of the first data fragment is configured as an initial value. The parsing module 230 is used to load the starting level of the data fragment corresponding to each node into the state stack, and perform character-by-character parsing and field segmentation on the data fragment based on the current level to obtain the parsing result of the data fragment; wherein, the state stack performs push operation and increments the current level when the current parsing character is an open quotation mark, and pop operation and decrements the current level when the current parsing character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsing character is a field separator; The merging module 240 is used to merge the parsing results of each of the data fragments according to the order of the data fragments.
[0112] The data processing apparatus provided in this embodiment of the invention divides structured text data into multiple data fragments with line boundaries aligned. After division, each data fragment is scanned to count the nesting level of quotation marks, thereby obtaining the ending level of each data fragment. Then, according to the fragmentation order, the ending level of the previous data fragment is configured as the starting level of the next data fragment, and the starting level is sent to each node along with the data fragments. During parsing, each node loads the starting level into the state stack as the initial parsing state. The current level is maintained through the push and pop mechanism of the state stack, and field splitting is triggered based on whether the current level is the initial value. This avoids field splitting misalignment and data truncation, and improves the parsing accuracy of non-standard structured text data in distributed processing scenarios.
[0113] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communications bus 340. The processor 310 can call logical instructions in the memory 330 to execute a data processing method, which includes: The structured text data is divided into multiple data segments in sequence, and the boundaries of each data segment are aligned with the line boundaries. Each data segment is scanned and the nesting level of quotation marks is counted to obtain the ending level of each data segment. According to the order of the data shards, the ending level of the (N-1)th data shard is configured as the starting level of the Nth data shard, and each data shard and its corresponding starting level are assigned to the corresponding node in the cluster, where N is an integer greater than 1 and the starting level of the first data shard is configured as the initial value. In each node, the starting level of the data fragment corresponding to the node is loaded into the state stack, and the data fragment is parsed character by character and segmented field by field based on the current level to obtain the parsing result of the data fragment; wherein, the state stack is pushed onto the stack and the current level is incremented when the current parsed character is an open quotation mark, and popped from the stack and the current level is decremented when the current parsed character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsed character is a field separator; The parsing results of each data fragment are merged according to the order of the data fragments.
[0114] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the conventional, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), disk, or optical disk, etc., any medium capable of storing program code.
[0115] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the data processing method provided in each of the above embodiments, the method including: The structured text data is divided into multiple data segments in sequence, and the boundaries of each data segment are aligned with the line boundaries. Each data segment is scanned and the nesting level of quotation marks is counted to obtain the ending level of each data segment. According to the order of the data shards, the ending level of the (N-1)th data shard is configured as the starting level of the Nth data shard, and each data shard and its corresponding starting level are assigned to the corresponding node in the cluster, where N is an integer greater than 1 and the starting level of the first data shard is configured as the initial value. In each node, the starting level of the data fragment corresponding to the node is loaded into the state stack, and the data fragment is parsed character by character and segmented field by field based on the current level to obtain the parsing result of the data fragment; wherein, the state stack is pushed onto the stack and the current level is incremented when the current parsed character is an open quotation mark, and popped from the stack and the current level is decremented when the current parsed character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsed character is a field separator; The parsing results of each data fragment are merged according to the order of the data fragments.
[0116] In another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the data processing method provided in each of the above embodiments, the method comprising: The structured text data is divided into multiple data segments in sequence, and the boundaries of each data segment are aligned with the line boundaries. Each data segment is scanned and the nesting level of quotation marks is counted to obtain the ending level of each data segment. According to the order of the data shards, the ending level of the (N-1)th data shard is configured as the starting level of the Nth data shard, and each data shard and its corresponding starting level are assigned to the corresponding node in the cluster, where N is an integer greater than 1 and the starting level of the first data shard is configured as the initial value. In each node, the starting level of the data fragment corresponding to the node is loaded into the state stack, and the data fragment is parsed character by character and segmented field by field based on the current level to obtain the parsing result of the data fragment; wherein, the state stack is pushed onto the stack and the current level is incremented when the current parsed character is an open quotation mark, and popped from the stack and the current level is decremented when the current parsed character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsed character is a field separator; The parsing results of each data fragment are merged according to the order of the data fragments.
[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, or of course, by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the conventional, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in each of the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.
Claims
1. A data processing method, characterized by, include: The structured text data is divided into multiple data segments in sequence, and the boundaries of each data segment are aligned with the line boundaries. Each data segment is scanned and the nesting level of quotation marks is counted to obtain the ending level of each data segment. According to the order of the data shards, the ending level of the (N-1)th data shard is configured as the starting level of the Nth data shard, and each data shard and its corresponding starting level are assigned to the corresponding node in the cluster, where N is an integer greater than 1 and the starting level of the first data shard is configured as the initial value. In each node, the starting level of the data fragment corresponding to the node is loaded into the state stack, and the data fragment is parsed character by character and segmented field by field based on the current level to obtain the parsing result of the data fragment; wherein, the state stack is pushed onto the stack and the current level is incremented when the current parsed character is an open quotation mark, and popped from the stack and the current level is decremented when the current parsed character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsed character is a field separator; The parsing results of each data fragment are merged according to the order of the data fragments.
2. The data processing method according to claim 1, characterized in that, The step of parsing the data fragments character by character and splitting them field by field based on the current level to obtain the parsing results of the data fragments includes: During the character-by-character parsing process, if the currently parsed character is a preset invalid character, the invalid character is discarded and the parsing positions of other characters are kept unchanged. When the current level is the initial value and the current parsed character is a field separator, the current buffered character stream is segmented into fields to obtain the segmented fields; The cutting fields are sequentially cleaned in a preset priority order, performing first-level cleaning, second-level cleaning, and third-level cleaning to obtain the parsing results; The first level of cleaning includes at least one of null value filling and newline character replacement; the second level of cleaning includes at least one of space trimming and case conversion based on the processing result of the first level of cleaning; and the third level of cleaning includes at least one of data format standardization, data anonymization, and field length truncation based on the processing result of the second level of cleaning.
3. The data processing method of claim 1, wherein, The step of scanning each of the data fragments and counting the nesting levels of quotation marks to obtain the ending level of each data fragment includes: When scanning each of the data segments, if the currently scanned character is an open quotation mark, the scanning level is incremented. If the currently scanned character is a closed quotation mark, the scanning level is decreased, and the nested quotation mark state is exited when the scanning level is decreased to the initial value. The scanning level at which the data fragment ends scanning is determined to be the end level of the data fragment.
4. The data processing method of claim 1, wherein, The step of allocating each data shard and its corresponding starting level to the corresponding node in the cluster includes: For each data shard, extract the data volume of the data shard and determine the level difference between the end level and the start level of the data shard; Based on the data volume and the hierarchical difference, determine the task processing volume corresponding to the data shard; Determine the current available resources for each node in the cluster; Based on the task processing volume and the current available resources, each data shard and its corresponding starting level are allocated to the corresponding node through a target allocation strategy. The target allocation strategy is to allocate resources based on the positive correlation between the task processing volume and the current available resources, so that the expected load difference of each node after allocation is less than a preset load threshold.
5. The data processing method of claim 1, wherein, After configuring the end level of the (N-1)th data fragment as the start level of the Nth data fragment according to the order of the data fragments, the method further includes: Based on the identifiers of each data fragment, the starting level, and the ending level, a global state chain is constructed, and based on the dependency relationships between adjacent data fragments in the global state chain, a directed acyclic dependency graph is constructed. During the process of parsing and field-by-field segmentation of the data fragments based on the current level, if a target data fragment that fails to be parsed is detected, the downstream data fragments of the target data fragments are determined based on the directed acyclic dependency graph. The target data shard is redistributed to the corresponding node in the cluster, and the process returns to load the starting level of the data shard corresponding to the node into the state stack, and performs character-by-character parsing and field splitting on the data shard based on the current level. After the target data fragment is parsed, the downstream data fragments of the target data fragment are scanned, and the nesting level of quotation marks is counted to update the end level and the corresponding start level of the downstream data fragment.
6. The data processing method of claim 1, wherein, The step of parsing the data fragments character by character and splitting them field by field based on the current level to obtain the parsing results of the data fragments includes: Extract the target row data from the data slices sequentially according to the row boundaries; The current level of the previous row of data at the end position is loaded into the state stack, and the target row of data is parsed character by character and segmented based on the state stack; If the target row data triggers a preset exception condition during the parsing process, an exception identifier corresponding to the target row data is generated, and the target row data is isolated and stored. The preset exception condition includes that when parsing to the end of the target row data, the current level has not been restored to the initial value, or the number of fields after field splitting is inconsistent with the preset field number threshold. The parsing results of the data shards are obtained based on the parsing results of the target row data that is not marked with the anomaly identifier.
7. The data processing method of claim 1, wherein, The step of merging the parsing results of each data fragment according to the order of the data fragments includes: According to the order of the data fragments, extract the target end level of the N-1th data fragment when the last line of data is parsed, and the target start level of the Nth data fragment when the first line of data is parsed. If the target end level and the target start level are the same level, then the parsing result of the Nth data segment is merged into the parsing result of the (N-1)th data segment; If the target end level and the target start level are different levels, then a continuity check anomaly identifier corresponding to the Nth data fragment is generated.
8. A data processing apparatus, characterized by, include: The segmentation scanning module is used to divide structured text data into multiple data segments in sequence, with the boundaries of each data segment aligned with the line boundaries, and to scan each data segment separately, counting the nesting levels of quotation marks to obtain the ending level of each data segment. The data distribution module is used to configure the end level of the (N-1)th data fragment as the start level of the Nth data fragment according to the order of the data fragments, and to allocate each data fragment and its corresponding start level to the corresponding node in the cluster, where N is an integer greater than 1 and the start level of the first data fragment is configured as an initial value. The parsing module is used to load the starting level of the data fragment corresponding to each node into the state stack, and perform character-by-character parsing and field segmentation on the data fragment based on the current level to obtain the parsing result of the data fragment; wherein, the state stack performs push operation and increments the current level when the current parsing character is an open quotation mark, and pops the current level and decrements the current level when the current parsing character is a closed quotation mark; the field segmentation is performed when the current level is the initial value and the current parsing character is a field separator; The merging module is used to merge the parsing results of each of the data fragments according to the order of the data fragments.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 7.
