Data standardization method and related device
By identifying and recording data formats and using corresponding parsers for phased parsing, the problem of low standardization efficiency caused by high human intervention in existing technologies is solved, achieving automated and efficient data standardization.
Patent Information
- Application Number
- CN202510943910.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-17
AI Technical Summary
Existing data standardization solutions have the problem of high level of manual participation, resulting in low efficiency of standardization processing.
By identifying the main data format of the original data, the first parser is used for the first stage of parsing, the embedding position of the mixed data is recorded, and the second parser is used for the second stage of parsing based on the embedding position, thus automatically completing the standardization process of the mixed data format.
It reduces human intervention, improves the efficiency of data standardization processing, and enables automated processing of various data formats.
Smart Images
Figure CN120804196A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and in particular, to a data standardization method and related device. BACKGROUND
[0002] With the development of big data technology, the data sources involved in the server system are increasingly diversified, and data of different sources or different types may have different data formats. Such data heterogeneity brings great challenges to data integration and application, so the server needs to perform data standardization processing on the received data to convert data of different formats into an internal unified representation. However, the inventors have found that the existing data standardization scheme has the problem of high degree of manual participation, which in turn leads to low standardization processing efficiency. SUMMARY
[0003] The present application aims to at least solve one of the above technical defects, in particular, the technical defect of low standardization processing efficiency in the prior art.
[0004] In a first aspect, an embodiment of the present application provides a data standardization method, comprising:
[0005] obtaining original data and identifying a main data format of the original data;
[0006] performing first-stage parsing on the original data using a first parser, and in the first-stage parsing process, if mixed data embedded in the original data is detected, recording an embedding position of the mixed data in the original data; wherein the first parser is a parser corresponding to the main data format, and the data format of the mixed data is different from the main data format;
[0007] performing second-stage parsing on the mixed data using a second parser according to the embedding position; wherein the second parser is a parser corresponding to the data format of the mixed data;
[0008] obtaining standardized data based on the results of the first-stage parsing and the second-stage parsing.
[0009] In one embodiment, the performing second-stage parsing on the mixed data using a second parser according to the embedding position comprises:
[0010] identifying the data format of the mixed data according to the embedding position;
[0011] if the identification is successful, determining the second parser according to the data format of the mixed data, and performing second-stage parsing on the mixed data using the second parser;
[0012] If the identification is unsuccessful, a plurality of predetermined parsers are taken as a plurality of second parsers, and each of the second parsers is used to perform second-stage parsing on the mixed data to obtain a plurality of initial parsing results, and an initial parsing result with the most valid information is selected as a result of the second-stage parsing.
[0013] In one of the embodiments, the method further comprises:
[0014] If a syntax error or a format error is detected in the first-stage parsing process, data repair is performed on the original data according to a preset error repair database, and the first-stage parsing is performed again on the repaired original data using the first parser;
[0015] If a syntax error or a format error is detected in the second-stage parsing process, data repair is performed on the mixed data according to the error repair database, and the second-stage parsing is performed again on the repaired mixed data using the second parser.
[0016] In one of the embodiments, the method further comprises:
[0017] If the number of failures of the first-stage parsing reaches a preset threshold, a preset regular expression is used to match the original data to identify a structured segment of the original data, and the identified structured segment is taken as a result of the first-stage parsing;
[0018] If the number of failures of the second-stage parsing reaches the preset threshold, the regular expression is used to match the mixed data to identify a structured segment of the mixed data, and the identified structured segment is taken as a result of the second-stage parsing.
[0019] In one of the embodiments, if the main data format is a JSON format, the first-stage parsing of the original data using the first parser comprises:
[0020] The first parser is used to perform recursive parsing on the original data until a current parsing depth is greater than or equal to a preset parsing depth threshold;
[0021] If the mixed data is JSON format data, the second-stage parsing of the mixed data using the second parser comprises:
[0022] The second parser is used to perform recursive parsing on the mixed data until the current parsing depth is greater than or equal to the parsing depth threshold.
[0023] In one of the embodiments, after obtaining the standardized data based on the results of the first-stage parsing and the results of the second-stage parsing, the method further comprises:
[0024] According to the preset matching ID name, naming format and / or type mark, the main data entities are identified from the results of the first-stage parsing and the results of the second-stage parsing;
[0025] According to the attributes of each of the main data entities, the foreign key relationship is identified;
[0026] The data relationship graph is created or updated according to the foreign key relationship; in the data relationship graph, one node represents one main data entity, and the edge is used to represent the reference relationship between the nodes, the reference relationship including at least one of the reference type, the reference strength and the reference direction.
[0027] In one of the embodiments, after obtaining the standardized data based on the results of the first-stage parsing and the results of the second-stage parsing, the method further comprises:
[0028] In response to the data acquisition request sent by the client device, the requested target standardized data is determined;
[0029] The output format corresponding to the client device is determined, and the response data is generated according to the output format and the target standardized data;
[0030] The response data is sent to the client device.
[0031] In a second aspect, the embodiments of the present application provide a data standardization device, comprising:
[0032] A first format identification module is configured to obtain original data and identify the main data format of the original data;
[0033] A first parsing module is configured to perform first-stage parsing on the original data by using a first parser; during the first-stage parsing, if mixed data embedded in the original data is detected, the embedding position of the mixed data in the original data is recorded; the first parser is a parser corresponding to the main data format, and the data format of the mixed data is different from the main data format;
[0034] A second parsing module is configured to perform second-stage parsing on the mixed data by using a second parser according to the embedding position; the second parser is a parser corresponding to the data format of the mixed data;
[0035] A standardization conversion module is configured to obtain standardized data based on the results of the first-stage parsing and the results of the second-stage parsing.
[0036] In a third aspect, an embodiment of the present application provides a storage medium, the storage medium storing computer readable instructions, the computer readable instructions being executed by one or more processors to cause the one or more processors to perform the steps of the data standardization method according to any of the embodiments described above.
[0037] In a fourth aspect, an embodiment of the present application provides a computer device, the computer device comprising: one or more processors, and a memory;
[0038] The memory stores computer readable instructions, the computer readable instructions being executed by the one or more processors to perform the steps of the data standardization method according to any of the embodiments described above.
[0039] In some embodiments of the data standardization method and the related apparatus provided by the present application, the original data can be obtained first, and the main data format of the original data can be identified. Then, the original data can be parsed in a first stage by using a first parser corresponding to the main data format. During the first stage parsing, if it is detected that mixed data different from the main data format is embedded in the original data, the data format of the mixed data and the embedding position of the mixed data in the original data can be recorded. After the parsing of the main data format is completed, the mixed data can be parsed in a second stage by using a second parser corresponding to the mixed data according to the embedding position, and the standardized data can be obtained based on the results of the first stage parsing and the second stage parsing. In this way, the original data mixed with multiple data formats can be automatically standardized by the present application, so that the degree of human participation can be reduced, and the efficiency of data standardization processing can be improved. BRIEF DESCRIPTION OF DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0041] Figure 1 For some embodiments, the application environment diagram of the data standardization method;
[0042] Figure 2 For some embodiments, the flowchart of the data standardization method;
[0043] Figure 3 For some embodiments, the structure diagram of the data standardization apparatus;
[0044] Figure 4For some embodiments, the internal structure diagram of the computer device. DETAILED DESCRIPTION
[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.
[0046] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of data information involved in the present application comply with the relevant legal regulations and do not violate public order and good customs.
[0047] In some embodiments, the data standardization method provided by the present application can be applied to Figure 1 The application environment is shown. The collection device 102 refers to a device with data collection function and capable of transmitting the collected data to the server 104. It can be understood that the collection device 102 can be, but is not limited to, various personal computers, notebook computers, laptop computers, tablet computers, smart phones, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc., and the portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc.
[0048] The server 104 refers to a device capable of receiving and storing collected data. Before storing the data, the server 104 can perform data standardization processing on the received data to convert data of different formats into an internal unified representation. In the present application, the server 104 can be a single server or a server cluster. The client device 106 refers to a device requesting to obtain collected data from the server 104, which can be, but is not limited to, various personal computers, notebook computers, laptop computers, tablet computers, smart phones, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc., and the portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc.
[0049] It can be understood that the raw data involved in the present application can be any type of data, which is not limited in the present application. In some examples, for the convenience of description, the present application takes browser data as an example for description. It should be understood that in addition to browser data, the data involved in the present application can also be other types of data, or include more types of data.
[0050] Specifically, the collection device 102 can collect raw browser data and transmit the raw browser data to the server 104. The server 104 can receive the raw browser data and store the raw browser data after standardization processing. In some examples, the server 104 can also provide any one or any combination of data processing functions such as data truncation, temporary storage of data, data analysis, etc. to the raw browser data.
[0051] When the client device 106 needs to obtain browser data, the client device 106 can send a data acquisition request to the server 104 to request to obtain target browser data. The server 104, upon receiving the data acquisition request, can determine target standardized data corresponding to the data acquisition request and return the target standardized data as response data to the client device 106.
[0052] In some embodiments, the present application provides a data standardization method, and the following embodiments are described with the server of the present application applied to Figure 1 as an example. As shown in Figure 2 , the data standardization method of the present application specifically includes the following steps:
[0053] S202: Obtain raw data and identify the main data format of the raw data.
[0054] The raw data refers to data that needs to be processed by the data standardization method, for example, the raw browser data in the example. The main data format refers to the top-level data format of the raw data, which can be determined according to the first M characters of the raw data. Figure 1
[0055] S204: Perform first-stage parsing of the raw data using a first parser. During the first-stage parsing, if mixed data embedded in the raw data is detected, the embedded position of the mixed data in the raw data is recorded. The first parser is a parser corresponding to the main data format, and the data format of the mixed data is different from the main data format.
[0056] In this step, after identifying the main data format of the raw data, the parser corresponding to the main data format (i.e., the first parser) can be used to perform first-stage parsing of the raw data to process the top-level structure of the raw data. During the first-stage parsing, if mixed data different from the main data format is detected to be embedded in the raw data, it indicates that the raw data is mixed format data, and in this case, the present application can record the position of the mixed data in the raw data to mark the area of the mixed data in the raw data.
[0057] For example, if the original data is in JSON format and contains escaped HTML, the application can use a JSON parser to perform the first-stage data parsing, and record the location of the escaped HTML data in the original data to mark the escaped HTML in the original data.
[0058] In some examples, if the main data format is JSON, the application can use a depth-first recursive parsing algorithm to sequentially traverse each field of the JSON object and each element of the array. For large objects (e.g., objects with more than a threshold number of fields, which can be 100 fields), the server can use a lazy parsing strategy to parse the content under a specific path only when the path is accessed. The application can perform a structure flattening conversion to convert a complex hierarchical structure into a flat key-value pair with a path separator, such as combining user.address.city into a single key user_address_city.
[0059] In some examples, if the main data format is a large array, the application can use a chunk processing strategy to perform the first-stage parsing. Specifically, the server can split the large data into multiple chunks according to a pre-set chunk size (e.g., 1000 elements), and load one chunk into memory at a time during processing. The server can use an iterator pattern to sequentially access the array elements in the chunk through a streaming processing mechanism. In this way, the entire array can be avoided to be loaded at once. This approach allows the server to release its memory immediately after completing the processing of the current element, significantly reducing memory usage.
[0060] In some examples, if the main data format is an array, the application can perform array homogeneity detection on the original data by analyzing the structural characteristics of the first N elements (e.g., the first 50 elements) of the array to determine whether the array contains homogeneous objects. For homogeneous arrays, the application can generate a unified parsing template based on sampling and apply the parsing template to the entire array to improve parsing efficiency. For heterogeneous arrays (containing elements of different types or structures), the application can determine a parsing strategy for each element individually. When the array elements are simple types (strings, numbers, booleans), the server can apply batch processing optimization to process multiple elements simultaneously. When the array elements are complex objects, the server can extract key fields for summarization, such as retaining only the ID and name fields of the object and skipping detailed attributes.
[0061] In some examples, if the main data format is XML (Extensible Markup Language) document or HTML (HyperText Markup Language) document, the application can choose different parsing strategies according to the document size. For example, for small documents with a size less than 1MB, the application can use DOM (Document Object Model) based complete parsing to build a complete in-memory document object model, allowing random access and query of any node. This method is suitable for scenarios that require multiple iterations or complex queries.
[0062] For example, for large documents with a size greater than 1MB, the application can use SAX based streaming parsing to sequentially process XML / HTML tags and trigger corresponding start tag, end tag and text content events. SAX parser only maintains the context of the current processing position, greatly reducing memory usage, and is suitable for single iteration scenarios. For specific types of XML / HTML documents, the server loads specially configured parsers, such as SOAP (Simple Object Access Protocol Message) parsers for SOAP (Simple Object Access Protocol Message), Feed parsers for RSS (Really Simple Syndication), and the like. These special parsers are optimized for specific formats, providing more efficient and more semantic processing.
[0063] Further, in the process of parsing XML or HTML, the application can use selective parsing strategy to pre-screen the nodes of interest through XPath or CSS selector, and only parse the matching part, ignoring other content. For XML containing mixed content model (text and child elements mixed nodes), the computer retains the complete content structure of the node, correctly handles the CDATA section and special character escape.
[0064] S206: According to the embedded position, a second parser corresponding to the data format of the mixed data is used to perform a second stage parsing on the mixed data.
[0065] In this step, after the parsing of the main data format is completed, the application can enter the second stage of parsing. In the process of the second stage of parsing, the server can use a second parser to parse each mixed data according to the embedding position, and obtain the result of the second stage of parsing. For example, for the HTML string field in JSON, the server first parses JSON to obtain the string value, and then uses the HTML parser to process the string. The parsing process of the second stage of parsing can refer to the related description of the first stage of parsing, which will not be repeated here.
[0066] S208: Obtain standardized data based on the result of the first stage of parsing and the result of the second stage of parsing.
[0067] In this step, the server can construct a unified memory representation according to the parsing results of the two stages, maintain the reference relationship between different format data, and thus obtain standardized data. Through the standardized data, the server or the client device can consistently access the mixed data.
[0068] In some examples, the standardized data can be JSON Schema-based data.
[0069] In the application, the original data can be obtained first, and the main data format of the original data can be identified. Then, the first parser corresponding to the main data format is used to perform the first stage of parsing on the original data. In the process of the first stage of parsing, if it is detected that the mixed data different from the main data format is embedded in the original data, the data format of the mixed data and the embedding position of the mixed data in the original data are recorded. After the parsing of the main data format is completed, the second parser corresponding to the mixed data can be used to perform the second stage of parsing on the mixed data according to the embedding position, and the standardized data can be obtained based on the result of the first stage of parsing and the result of the second stage of parsing. In this way, the original data mixed with multiple data formats can be automatically standardized, so that the degree of human participation can be reduced, and the efficiency of data standardization can be improved.
[0070] In some embodiments, the second stage of parsing on the mixed data according to the embedding position using the second parser includes:
[0071] According to the embedding position, the data format of the mixed data is identified;
[0072] If the identification is successful, the second parser is determined according to the data format of the mixed data, and the second stage of parsing on the mixed data is performed using the second parser;
[0073] If the identification is unsuccessful, a plurality of predetermined parsers are taken as a plurality of second parsers, and each second parser is used to perform second-stage parsing on the mixed data to obtain a plurality of initial parsing results, and the initial parsing result with the most valid information is selected as the result of the second-stage parsing.
[0074] In the embodiment, in the process of the second-stage parsing, the application can first identify the data format of the mixed data. If the identification is successful, it indicates that the data format of the mixed data is a determined format. In this case, the corresponding parser can be determined according to the identified data format of the mixed data, and the second parser is obtained, and the second parser is used to perform the second-stage parsing on the mixed data.
[0075] If the identification is unsuccessful, it indicates that the mixed data is data in an undetermined format. In this case, the server can use a plurality of parsers to parse the mixed data, and obtain an initial parsing result corresponding to each parser. Among the plurality of initial parsing results, the server can select the initial parsing result with the most valid information as the best choice, as the result of the second-stage parsing. The most valid information can be the case with the most valid results and the least errors.
[0076] In the embodiment, for data in an undetermined format, the application can attempt to use a plurality of parsers to parse the data, and select the parsing result with the most valid information from a plurality of initial parsing results as the final result of the second-stage parsing. In this way, the degree of human participation can be further reduced, and the efficiency of data standardization processing can be further improved.
[0077] In some embodiments, the data standardization method of the application further includes the following steps:
[0078] If it is detected in the first-stage parsing process that the original data has a syntax error or a format error, the original data is repaired according to a preset error repair database, and the repaired original data is parsed again by the first parser in the first-stage parsing;
[0079] If it is detected in the second-stage parsing process that the mixed data has a syntax error or a format error, the mixed data is repaired according to the error repair database, and the repaired mixed data is parsed again by the second parser in the second-stage parsing.
[0080] In the embodiment, the server can automatically correct common syntax errors through a data repair mechanism. In the data parsing process, the server can enable a tolerant mode in the parser to accept formats that do not completely conform to the specifications. If the server detects that a certain position of the data has a syntax error such as unmatched parentheses or unclosed labels, the server will not stop parsing immediately, but will try to repair the syntax error of the data through a preset error repair database, and continue parsing after parsing.
[0081] For example, for JSON parsing, the server can accept out-of-specification format variants, such as trailing commas, comments, field names with unmatched quotes, etc. The server automatically fixes common syntax errors, such as adding missing quotes, removing extra commas, balancing parentheses, etc. When the server detects an approximately valid JSON like {'key':'value'} (using single quotes instead of double quotes), it is re-parsed after conversion to standard format.
[0082] For example, for XML / HTML, the server accepts common errors such as unclosed self-closing tags, attributes missing quotes, overlapping tags, etc., and internally normalizes these structures. The server maintains a database of common error patterns and corresponding fix-up strategies, and applies predefined fix-up steps when a matching error pattern is encountered.
[0083] In some examples, for JSON parsing, if the server detects an invalid token at position X, it can skip the current attribute or element and look for the next valid syntactic boundary (such as a comma or closing parenthesis) to resume parsing.
[0084] In some examples, for XML parsing, when the server encounters an unclosed tag, it can automatically insert the missing closing tag or jump to the next sibling tag to continue parsing. Further, the application can implement a region isolation strategy to separate successfully parsed content from the part with errors, ensuring that errors do not contaminate correctly parsed data.
[0085] For format errors in data, the application can automatically correct common format errors through a data fix-up mechanism and continue parsing after correction. For example, for obvious spelling errors or format deformations, the server can apply heuristic corrections, such as correcting flase to false and TRUE to true. For example, when the server detects stage data (e.g., incomplete JSON due to data size limitations), it can attempt to close open structures, add missing closing parentheses or quotes. For example, when facing encoding problems, the application can detect and fix UTF-8 half-byte sequences, escape character errors, or illegal Unicode characters. Further, when encountering mixed encoding text, the application can attempt to identify the encoding of different segments, convert them separately, and then recombine them. For date and time format errors, the server can attempt multiple common format patterns to find the most matching parsing method, such as recognizing 2023-13-45 as a day-month order error and correcting it to 2023-12-04 or extracting the possibly valid part.
[0086] In this embodiment, by automatically repairing syntax errors and format errors, the degree of human involvement can be further reduced, and the efficiency of data standardization processing can be further improved.
[0087] In some embodiments, the data standardization method of the present application further comprises the following steps:
[0088] If the number of failures in the first stage parsing reaches a preset threshold, a preset regular expression is used to match the original data to identify structured fragments of the original data, and the identified structured fragments are taken as the result of the first stage parsing.
[0089] If the number of failures in the second stage parsing reaches a preset threshold, a regular expression is used to match the mixed data to identify structured fragments of the mixed data, and the identified structured fragments are taken as the result of the second stage parsing.
[0090] In this embodiment, if the parser cannot parse valid information from the data, the effective part can be extracted from the data with an incorrect format through a partial extraction strategy. For data whose accurate format cannot be determined, the present application can be degraded to simple text processing. By applying basic row division, segmentation, and keyword extraction strategies, as much original information as possible can be extracted from the data.
[0091] Specifically, in the data parsing process, if the number of parsing failures reaches a preset threshold, the server can use a preset regular expression to match structured fragments in the data. For example, a JSON object in the form of "{...}" or an XML element in the form of " can be matched by a regular expression. <tag> ...< / tag> In the case where the server successfully matches structured fragments, the server can extract them as independent fragments and parse them separately, bypassing the surrounding invalid content. For tabular data, even if some rows are incorrectly formatted, the server can parse the correctly formatted row data through a preset regular expression, treating each row as an independent record.
[0092] When the server cannot parse the complete complex structure, the present application can attempt to extract the key path or field, such as extracting obvious key-value pairs from a damaged JSON or extracting the content of certain specific tags from a malformed XML. The server also attempts to identify data boundaries through contextual clues, such as using indentation levels, line separators, or specific markers to infer logical units.
[0093] In some embodiments, if the main data format is JSON format, a first parser is used to perform first stage parsing on the original data, including:
[0094] The first parser is used to perform recursive parsing on the original data until the current parsing depth is greater than or equal to a preset parsing depth threshold.
[0095] If the mixed data is in JSON format, the second parser is used to perform second-stage parsing on the mixed data, including:
[0096] The second parser is used to perform recursive parsing on the mixed data until the current parsing depth is greater than or equal to the parsing depth threshold.
[0097] In this embodiment, for the JSON structure data with deep nesting, the preset parsing depth threshold can be used as the maximum parsing depth to prevent infinite recursion. During the parsing of the JSON data, if the current parsing depth exceeds the preset parsing depth threshold, further recursion is stopped, and the content of the excessively deep level can be marked as [Deep Structure]. In this way, the program can be prevented from falling into infinite recursion, thereby improving the data processing efficiency.
[0098] In some embodiments, after obtaining the standardized data based on the results of the first-stage parsing and the second-stage parsing, the following steps are further included:
[0099] According to the preset matching ID name, naming format, and / or type mark, the main data entities are identified from the results of the first-stage parsing and the second-stage parsing.
[0100] According to the attributes of each main data entity, the foreign key relationship is identified.
[0101] According to the foreign key relationship, a data relationship graph is created or updated. In the data relationship graph, one node represents one main data entity, and the edges are used to represent the reference relationship between the nodes, including at least one of the reference type, the reference strength, and the reference direction.
[0102] In this embodiment, the server can construct an entity relationship mapping to represent the logical connection relationship between the data entities. Specifically, the server can first identify the main data entities from the results of the two-stage parsing by matching the ID name, the preset naming format specification, and / or the type mark. The main data entities can include user data, product data, order data, configuration data, and the like.
[0103] The server can analyze the attributes of each primary data entity, from which it identifies foreign key relationships. For example, the userId field in the order object references the user entity, so there is a foreign key relationship between the userId field and the user entity. Upon identifying foreign key relationships, the server can create or update a data relationship graph accordingly. In the data relationship graph, nodes represent entities and edges represent reference relationships between entities. For each reference relationship, the server records the reference type (e.g., “belongs to,” “has many,” “references”), the reference strength (strong or weak), and the reference direction (one-way or two-way). The server detects circular reference patterns, such as A references B, B references C, and C references A, and appropriately marks these cycles to prevent infinite recursion. When the server finds a new entity in the data, it updates the relationship graph, adding new nodes and edges, and maintains a dynamic representation of entity relationships.
[0104] In some embodiments, after obtaining the standardized data based on the results of the first-stage parsing and the results of the second-stage parsing, the method further includes:
[0105] In response to a data acquisition request sent by the client device, determining target standardized data requested;
[0106] Determining an output format corresponding to the client device, and generating response data according to the output format and the target standardized data;
[0107] Sending the response data to the client device.
[0108] In the embodiment, the server can implement multi-format output matching. When the client device requests to acquire data, the application can convert the standardized data into the most suitable output format according to the needs of the client device, thereby obtaining response data and returning the response data to the client device. Further, in some examples, the server can only be a progressive and incremental data transmission mechanism.
[0109] In some embodiments, for non-standard format data, the application can use a heuristic pattern matching strategy. Specifically, the server can pre-load a series of predefined regular expression patterns for identifying common non-standard format features, such as specific separator patterns, key-value pair formats, or markup syntax. When the data matches a certain pattern, the application applies the corresponding extraction rule.
[0110] Further, the application can construct an adaptive parsing template library to record patterns that successfully parse non-standard formats. When encountering new non-standard data, the application attempts to match it with known templates, selecting the template with the highest similarity as a starting point.
[0111] In some embodiments, the present application implements progressive format learning, analyzes multiple similar data samples, identifies common structural features and delimiters, and gradually improves the parsing template. For text data that can contain various delimiters, the present application tests multiple common delimiters (such as comma, tab, pipe), and selects the delimiter that produces the most consistent row-column structure. When the present application cannot determine the precise format rule, a fault-tolerant extraction strategy is used to identify and extract useful parts of the data, such as date, URL, email address, and other structured information, even if the complete format cannot be parsed.
[0112] The data standardization apparatus provided by the embodiments of the present application is described below. The data standardization apparatus described below can be referred to in conjunction with the data standardization method described above.
[0113] In some embodiments, as shown in FIG. 3, Figure 3 The data standardization apparatus 300 provided by the present application includes:
[0114] A first format identification module 302 is configured to obtain raw data and identify the main data format of the raw data.
[0115] A first parsing module 304 is configured to perform first-stage parsing on the raw data using a first parser. During the first-stage parsing, if mixed data embedded in the raw data is detected, the embedding position of the mixed data in the raw data is recorded. The first parser is a parser corresponding to the main data format, and the data format of the mixed data is different from the main data format.
[0116] A second parsing module 306 is configured to perform second-stage parsing on the mixed data using a second parser according to the embedding position. The second parser is a parser corresponding to the data format of the mixed data.
[0117] A standardization conversion module 308 is configured to obtain standardized data based on the results of the first-stage parsing and the second-stage parsing.
[0118] In some embodiments, the second parsing module 306 of the present application includes:
[0119] A format identification unit is configured to identify the data format of the mixed data according to the embedding position.
[0120] A first parsing unit is configured to determine the second parser according to the data format of the mixed data if the identification is successful, and perform second-stage parsing on the mixed data using the second parser.
[0121] The second analysis unit is configured to, if the identification is unsuccessful, take a plurality of predetermined parsers as a plurality of second parsers, and use each of the second parsers to perform second-stage analysis on the mixed data to obtain a plurality of initial analysis results, and select an initial analysis result with the most valid information as a result of the second-stage analysis.
[0122] In some embodiments, the data standardization apparatus 300 further comprises:
[0123] The first repair module is configured to, if a syntax error or a format error is detected in the first-stage analysis process, perform data repair on the original data according to a preset error repair database, and perform first-stage analysis again on the repaired original data using the first parser.
[0124] The second repair module is configured to, if a syntax error or a format error is detected in the second-stage analysis process, perform data repair on the mixed data according to the error repair database, and perform second-stage analysis again on the repaired mixed data using the second parser.
[0125] In some embodiments, the data standardization apparatus 300 further comprises:
[0126] The first structured fragment extraction module is configured to, if the number of failures of the first-stage analysis reaches a preset threshold, use a preset regular expression to match the original data to identify a structured fragment of the original data, and take the identified structured fragment as a result of the first-stage analysis.
[0127] The second structured fragment extraction module is configured to, if the number of failures of the second-stage analysis reaches the preset threshold, use the regular expression to match the mixed data to identify a structured fragment of the mixed data, and take the identified structured fragment as a result of the second-stage analysis.
[0128] In some embodiments, if the main data format is a JSON format, the first analysis module 304 comprises:
[0129] The first recursive analysis unit is configured to use the first parser to perform recursive analysis on the original data until a current analysis depth is greater than or equal to a preset analysis depth threshold.
[0130] If the mixed data is JSON format data, the second analysis module 306 comprises:
[0131] The second recursive analysis unit is configured to use the second parser to perform recursive analysis on the mixed data until a current analysis depth is greater than or equal to the analysis depth threshold.
[0132] In some embodiments, the data standardization apparatus 300 of the present application further comprises:
[0133] an entity recognition module, configured to recognize main data entities from the results of the first-stage parsing and the results of the second-stage parsing according to preset matching ID names, naming formats and / or type labels;
[0134] a foreign key relationship recognition module, configured to recognize foreign key relationships according to attributes of each of the main data entities;
[0135] a relationship graph updating module, configured to create or update a data relationship graph according to the foreign key relationships; wherein in the data relationship graph, one node represents one main data entity, and an edge is used to represent a reference relationship between nodes, the reference relationship including at least one of a reference type, a reference strength and a reference direction.
[0136] In some embodiments, the data standardization apparatus 300 of the present application further comprises:
[0137] a request receiving module, configured to determine target standardized data requested in response to a data acquisition request sent by a client device;
[0138] a response data generating module, configured to determine an output format corresponding to the client device, and generate response data in the output format and the target standardized data;
[0139] a response data sending module, configured to send the response data to the client device.
[0140] In one embodiment, the present application further provides a storage medium having computer readable instructions stored therein, and the computer readable instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the data standardization method in any embodiment.
[0141] In one embodiment, the present application further provides a computer device having computer readable instructions stored therein, and the computer readable instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the data standardization method in any embodiment.
[0142] Schematically, Figure 4 An internal structure schematic diagram of a computer device provided by an embodiment of the present application is shown in one example, which can be a server. Referring to Figure 4The computer device 900 includes a processing component 902, which is further composed of one or more processors, and a memory resource represented by the memory 901 for storing instructions, such as an application program, executable by the processing component 902. The application program stored in the memory 901 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 902 is configured to execute the instructions to perform the steps of the data standardization method described in any of the embodiments above.
[0143] The computer device 900 can further include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 can operate based on an operating system stored in the memory 901, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.
[0144] Those skilled in the art can understand that the internal structure of the computer device shown in the present application is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0145] Finally, it should also be noted that, in this document, relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitation, an element preceded by "comprises... a" does not, without more limitations, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the recited element. In this document, "a", "an", "the", and "this" are also used to include plural references unless the context clearly dictates otherwise. A plurality of items is indicated by the term "a plurality of", for example, two, three, five, or eight, etc. "And / or" includes any and all combinations of one or more of the associated listed items.
[0146] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.
[0147] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data standardization method, characterized in that: include: Acquire original data and identify the main data format of the original data; Performing a first-stage parsing of the original data using a first parser, and during the first-stage parsing, if mixed data is detected to be embedded in the original data, recording the embedded position of the mixed data in the original data; wherein the first parser is a parser corresponding to the primary data format, and the data format of the mixed data is different from the primary data format; According to the embedding position, a second parser is used to perform a second-stage parsing on the mixed data; wherein the second parser is a parser corresponding to the data format of the mixed data; Based on the results of the first-stage analysis and the results of the second-stage analysis, standardized data is obtained.
2. The method according to claim 1, characterized in that The second parser is used to perform a second-stage parsing of the mixed data according to the embedding position, including: identifying a data format of the mixed data according to the embedding position; If the recognition is successful, determining the second parser according to the data format of the mixed data, and performing a second-stage parsing of the mixed data using the second parser; If the recognition is unsuccessful, a plurality of predetermined parsers are used as the plurality of second parsers, and each of the second parsers is used to perform the second stage parsing on the mixed data to obtain a plurality of initial parsing results, and the initial parsing result with the most valid information is selected as the result of the second stage parsing.
3. The method according to claim 1, characterized in that The method further comprises: If a syntax error or format error is detected in the original data during the first-stage parsing process, the original data is repaired according to a preset error repair database, and the first-stage parsing is performed again on the repaired original data using the first parser; If a syntax error or format error is detected in the mixed data during the second-stage parsing process, data repair is performed on the mixed data according to the error repair database, and the second-stage parsing is performed again on the repaired mixed data using the second parser.
4. The method according to claim 3, characterized in that The method further comprises: If the number of failures in the first-stage parsing reaches a preset threshold, a preset regular expression is used to match the original data to identify structured segments of the original data, and the identified structured segments are used as the results of the first-stage parsing; If the number of failures of the second-stage parsing reaches the preset threshold, the regular expression is used to match the mixed data to identify structured segments of the mixed data, and the identified structured segments are used as the results of the second-stage parsing.
5. The method according to claim 1, wherein If the primary data format is JSON, the first parser is used to perform a first-stage parsing of the original data, including: Recursively parsing the original data using the first parser until a current parsing depth is greater than or equal to a preset parsing depth threshold; If the mixed data is in JSON format, the second parser is used to perform a second-stage parsing on the mixed data, including: The mixed data is recursively parsed using the second parser until a current parsing depth is greater than or equal to the parsing depth threshold.
6. The method according to any one of claims 1 to 5, characterized in that After obtaining the standardized data based on the results of the first-stage analysis and the results of the second-stage analysis, the method further includes: Identify primary data entities in the results of the first-stage parsing and the results of the second-stage parsing according to preset matching ID names, naming formats, and / or type tags; identifying foreign key relationships based on attributes of each of the primary data entities; A data relationship graph is created or updated based on the foreign key relationship; wherein, in the data relationship graph, a node represents a main data entity, and an edge is used to represent a reference relationship between nodes, and the reference relationship includes at least one of a reference type, a reference strength, and a reference direction.
7. The method according to any one of claims 1 to 5, characterized in that After obtaining the standardized data based on the results of the first-stage analysis and the results of the second-stage analysis, the method further includes: In response to a data acquisition request sent by a client device, determining the requested target standardized data; Determining an output format corresponding to the client device, and generating response data according to the output format and the target standardized data; The response data is sent to the client device.
8. A data standardization device, characterized in that: include: A first format recognition module is used to obtain original data and identify a main data format of the original data; a first parsing module, configured to perform a first-stage parsing of the original data using a first parser, and during the first-stage parsing, if mixed data is detected to be embedded in the original data, record the embedded position of the mixed data in the original data; wherein the first parser is a parser corresponding to the primary data format, and the data format of the mixed data is different from the primary data format; A second parsing module, configured to perform a second-stage parsing of the mixed data using a second parser according to the embedding position; wherein the second parser is a parser corresponding to the data format of the mixed data; The standardized conversion module is used to obtain standardized data based on the results of the first-stage analysis and the results of the second-stage analysis.
9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the data normalization method according to any one of claims 1 to 7.
10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the data normalization method according to any one of claims 1 to 7.