Configuration-based structured data analysis and processing method, equipment, and electronic device

Abstract

The present invention provides a method, equipment and electronic device for parsing and processing structured data based on configuration. By reading a predefined configuration file, configuration information is obtained and parsed and loaded; the configuration information is configured data processing information; the data processing information includes field mapping relationships and conversion rules; receiving and reading a data source, and parsing the data source according to the field mapping relationship; according to the conversion rules, converting the parsed data to obtain converted target format data; outputting the target format data. Compared with the prior art, by reading the configuration file, the structure and field information of the data source can be automatically identified, and data can be extracted, converted and processed according to the configuration rules, which can adapt to various complex data sources and data formats, improve the flexibility and scalability of data parsing, and improve development efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of structured data processing, and in particular to a method, equipment and electronic device for analyzing and processing structured data based on configuration. Background Art

[0002] With the rapid development of the information age, the parsing and processing of various structured data (such as XML, JSON, CSV, etc.) has become an indispensable part of daily work. However, traditional data parsing methods usually require writing specific parsing codes for each data structure, which is not only inefficient but also difficult to adapt to the needs of frequent changes in data structures. Therefore, it is of great practical significance to develop a structured data parsing engine that can flexibly adapt to different data structures and is based on configuration.

[0003] The existing technology uses hard coding to parse data for specific businesses and data sources. There are the following technical defects: (1) Lack of flexibility: Hard-coded logic makes it difficult for the code to adapt to changes in data structure. If the data format changes, the code also needs to be modified accordingly, which increases maintenance costs. (2) Poor code reusability: Since the parsing logic is tightly bound to the data structure, the same parsing logic cannot be easily applied to different data structures, resulting in low code reusability. (3) Poor scalability: The hard-coded method is difficult to handle new data types or structures that may be introduced in the future. Adding new parsing logic usually requires a lot of modifications to the existing code. (4) High maintenance cost: Whenever the data structure changes, developers must recheck and modify the code, which may lead to error introduction and additional testing costs. (5) Difficult to debug and test: Hard-coded logic makes the code more complex and difficult to understand and debug; unit testing may also become more difficult because the test needs to cover changes in various data structures. (6) Low code readability: Hard-coded parsing logic is usually complex and difficult to read and understand, especially for new developers who are not familiar with the code. (7) Lack of modularity: Hard coding makes the code not modular enough, and it is difficult to separate the parsing logic from the business logic. This violates the single responsibility principle in software design. (8) Security risks: If the hard-coded data parsing logic involves external input (such as user input or network data), it may be more vulnerable to attacks such as SQL injection or XML injection. (9) No support for configuration: Hard coding does not support adjusting the parsing logic through configuration files or external input, which limits the flexibility and adaptability of the system. (10) Not conducive to version control: Changes in data structure often lead to a large number of code changes, which makes version control more complicated and difficult to track and merge code changes. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a configuration-based structured data parsing and processing solution to enhance the scalability of the code, reduce maintenance costs, improve software development efficiency, and shorten the development cycle.

[0005] A first aspect of the present invention provides a configuration-based structured data parsing and processing method, the method comprising:

[0006] Read a predefined configuration file, obtain configuration information, parse and load it; the configuration information is configured data processing information; the data processing information includes field mapping relationship and conversion rules;

[0007] Receive and read the data source, and perform a parsing operation on the data source according to the field mapping relationship;

[0008] According to the conversion rules, the parsed data is converted to obtain the converted target format data; wherein each field type corresponds to a conversion rule;

[0009] Output the target format data.

[0010] Furthermore, the data processing information also includes a data source structure, a target data structure and data processing logic;

[0011] The reading of a predefined configuration file, obtaining configuration information and parsing and loading the configuration information includes: reading the data source structure and the target data structure through configuration; the data source structure and the target data structure both contain field names and field types of structured data.

[0012] Further, the receiving and reading of the data source, and performing a parsing operation on the data source according to the field mapping relationship, includes:

[0013] The data format read from the data source determines the field mapping relationship corresponding to the data source;

[0014] If there is a corresponding field mapping relationship, extract the value of each field, and map each field to a corresponding position in the target data structure according to the field mapping relationship;

[0015] If there is no corresponding field mapping relationship, the field is ignored and no mapping is performed.

[0016] Furthermore, the method further comprises: receiving a user configuration change request; the configuration change request comprises adding, modifying or deleting a configuration rule; and modifying the configuration file according to the configuration modification request.

[0017] Further, the conversion rules include: converting a string type into a numeric type, and / or normalizing a predetermined field;

[0018] The target format data includes JSON, XML or other custom formats.

[0019] In order to solve the above technical problems, in a second aspect, the embodiment of the present application provides a configurable structured data parsing and processing device, which adopts the following technical solution:

[0020] The device comprises a reading module, a parsing module, a conversion module and an output module, wherein:

[0021] A reading module is used to read a predefined configuration file, obtain configuration information, parse and load it; the configuration information is the configured data processing information; the data processing information includes field mapping relationships and conversion rules;

[0022] A parsing module, used to receive and read a data source, and perform a parsing operation on the data source according to the field mapping relationship;

[0023] A conversion module, used to perform a conversion operation on the parsed data according to the conversion rule to obtain converted target format data; wherein each field type corresponds to a conversion rule;

[0024] An output module is used to output the target format data.

[0025] Furthermore, the data processing information also includes a data source structure, a target data structure and data processing logic;

[0026] The reading module is also used to read the data source structure and the target data structure through configuration; the data source structure and the target data structure both contain field names and field types of structured data.

[0027] Furthermore, the parsing module is further used to determine the field mapping relationship corresponding to the data source based on the data format read from the data source;

[0028] If there is a corresponding field mapping relationship, extract the value of each field, and map each field to a corresponding position in the target data structure according to the field mapping relationship;

[0029] If there is no corresponding field mapping relationship, the field is ignored and no mapping is performed.

[0030] In addition, the third aspect of the present invention provides an electronic device, which includes: one or more processors, and a memory, wherein the memory is used to store one or more computer programs; the computer program is configured to be executed by the one or more processors, and the program includes steps for executing the configurable structured data parsing and processing method as described in the first aspect.

[0031] In addition, the fourth aspect of the present invention provides a storage medium storing a computer program; the program is loaded and executed by a processor to implement the steps of the structured data parsing processing method based on configuration as described in the first aspect above.

[0032] In the scheme of the present invention, by reading a predefined configuration file, configuration information is obtained and parsed and loaded; the configuration information is configured data processing information; the data processing information includes field mapping relationships and conversion rules; receiving and reading a data source, and performing a parsing operation on the data source according to the field mapping relationship; performing a conversion operation on the parsed data according to the conversion rules to obtain converted target format data; wherein each field type corresponds to a conversion rule; and outputting the target format data. Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: by reading the configuration file, the structure and field information of the data source can be automatically identified, and data can be extracted, converted and processed according to the configuration rules, and various complex data sources and data formats can be adapted, and maintenance costs can be reduced. Data parsing rules can be easily managed and adjusted through the configuration file, which improves the flexibility and scalability of data parsing and improves development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a flowchart of a configuration-based structured data analysis and processing method disclosed in an embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of the structure of the engine system disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] It should be noted that the “plurality” mentioned in this article refers to two or more.

[0039] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0040] In the first aspect of this embodiment, a method for parsing and processing structured data based on configuration is implemented. Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for analyzing and processing structured data based on configuration disclosed in an embodiment of the present invention. Figure 1 As shown, the configuration-based structured data parsing and processing method of the embodiment of the present invention includes:

[0041] Step S101, read a predefined configuration file, obtain configuration information, parse and load it; the configuration information is configured data processing information; the data processing information includes field mapping relationship and conversion rules.

[0042] Furthermore, the data processing information also includes a data source structure, a target data structure and data processing logic;

[0043] In the step S101, a predefined configuration file is read to obtain configuration information and parse and load it, including: reading the data source structure and target data structure through configuration; the data source structure and target data structure both contain field names and field types of structured data.

[0044] Specifically, in this embodiment, the corresponding method is executed based on the data engine. After the engine is started, it first reads a predefined configuration file, which contains detailed information such as the data source structure, field mapping relationship, conversion rules, and processing logic. The configuration information is parsed and loaded to make it a specific rule that the engine can recognize and execute. The engine reads the source data structure model and the destination data structure model defined by the user through the configuration reader, and the structure model contains the field name and type of the structured data.

[0045] The annotation format is as follows:

[0046] Structure definition:

[0047]

[0048] In this example, the engine identifies structured data through schema and data types through type.

[0049] In some embodiments, the data structure can be described using an engine-defined language, for example:

[0050]

[0051] or

[0052] Target data structure

[0053]

[0054] You can also use a common language such as JSON to describe, for example:

[0055] Data source structure:

[0056]

[0057]

[0058] or

[0059] Target data structure

[0060]

[0061] This engine supports general languages, such as JSON-written configurations, and also defines a set of configuration writing languages ​​that can be applied to the context of this engine. By comparing the above examples, it can be found that the configuration file writing method provided by this engine is more concise and more expressive. However, for the sake of compatibility and learning cost, this engine also provides support for general languages.

[0062] Step S102: receiving and reading a data source, and performing a parsing operation on the data source according to the field mapping relationship.

[0063] Furthermore, in step S102, receiving and reading a data source, and performing a parsing operation on the data source according to the field mapping relationship, includes:

[0064] The data format read from the data source determines the field mapping relationship corresponding to the data source;

[0065] If there is a corresponding field mapping relationship, extract the value of each field, and map each field to a corresponding position in the target data structure according to the field mapping relationship;

[0066] If there is no corresponding field mapping relationship, the field is ignored and no mapping is performed.

[0067] Specifically, in this embodiment, step S102 can be divided into a data source access stage and a data pre-analysis stage.

[0068] Data source access phase: When a structured data source is input, the engine identifies and receives the data source; verifies whether the format and content of the data source meet expectations, and ensures the integrity and accuracy of the data.

[0069] In some embodiments, the system can define a Reader interface for the data source. The Reader interface includes a read method signature. Users can customize interface implementations that conform to the Reader interface for different data sources. The parsing engine reads the original data by calling the read method of the Reader interface.

[0070] Data pre-parsing phase: According to the rules defined in the configuration file, the engine begins to parse the data source line by line or field by field. It extracts the value of each field and maps it to the corresponding position in the target data structure according to the mapping relationship.

[0071] In some embodiments, assume the following parsing rules:

[0072] Original structure:

[0073]

[0074] Pre-parsed structure:

[0075]

[0076] Field mapping relationship:

[0077]

[0078]

[0079] Custom Types:

[0080]

[0081] Specifically, in this embodiment, the parsing engine can apply the mapping relationship to the data in the sourceSchema format read from the data source according to the field mapping relationship in fieldMap1, generate pre-parsed data in the preSchema format, and provide data support for the next step of data conversion. If the mapping relationship is not configured, the field is ignored by default. From the above example, it can be seen that this embodiment does not configure a mapping relationship for the field field2 in the original structure, so the pre-parsed data does not include this field.

[0082] Step S103, performing a conversion operation on the parsed data according to the conversion rule to obtain converted target format data; wherein each field type corresponds to a conversion rule.

[0083] Specifically, in this embodiment, according to the conversion rules set in the configuration, the parsed data is subjected to type conversion, data cleaning, encoding conversion and other operations. For example, a string type is converted to a numeric type, or a specific field is normalized.

[0084] Assume the following field conversion rules:

[0085] Target structure format:

[0086]

[0087]

[0088] Field conversion configuration:

[0089]

[0090] Rule definition:

[0091]

[0092]

[0093] Among them, rule1Func, rule2Func, rule3Func, and rule4Func are user-defined rules, and custom logic is implemented through code.

[0094] Based on this, in this embodiment, the parsing engine executes the corresponding conversion rules on each field after pre-parsing, and finally generates data in the target format defined in the si nkSchema.

[0095] Step S104: output the target format data.

[0096] Furthermore, the conversion rules include: converting a string type into a numeric type, and / or normalizing a predetermined field; the target format data includes JSON, XML or other custom formats.

[0097] Specifically, in this embodiment, the data after parsing, conversion and rule processing are organized into the final output result. The output can be in a specific format, such as JSON, XML or other custom formats. The system defines a Writer interface for output. The Writer interface contains the Writer method signature. For different output targets, users can customize the interface implementation that conforms to the Writer interface. The parsing engine outputs data by calling the Writer method of the Writer interface.

[0098] Furthermore, the method further comprises: receiving a user configuration change request; the configuration change request comprises adding, modifying or deleting a configuration rule; and modifying the configuration file according to the configuration modification request.

[0099] Throughout the entire workflow, the engine can monitor and handle possible abnormal situations in real time, such as data source errors, configuration errors, etc., and promptly feedback the corresponding error information for correction and adjustment. At the same time, the engine has good scalability, allowing users to add, modify or delete configuration rules at any time according to actual needs to adapt to different business scenarios and data processing requirements.

[0100] In order to implement the above solution, an engine system is designed in this case. Figure 2 The diagram is a schematic diagram of the engine system structure of this embodiment. The engine system mainly includes a configuration parser (including an XML configuration parser, a JSON configuration parser, etc.) and a parsing engine. The parsing engine includes a data source reader, a rule executor, and a data receiver.

[0101] In summary, in the scheme of this embodiment, by reading a predefined configuration file, configuration information is obtained and parsed and loaded; the configuration information is the configured data processing information; the data processing information includes field mapping relationships and conversion rules; the data source is received and read, and the data source is parsed according to the field mapping relationship; according to the conversion rules, the parsed data is converted to obtain the converted target format data; wherein each field type corresponds to a conversion rule; and the target format data is output. Compared with the prior art, the embodiment of the present application has the following main beneficial effects: by reading the configuration file, the structure and field information of the data source can be automatically identified, and data can be extracted, converted and processed according to the configuration rules, and it can adapt to various complex data sources and data formats, reducing maintenance costs, and the data parsing rules can be easily managed and adjusted through the configuration file, which improves the flexibility and scalability of data parsing and improves development efficiency.

[0102] In order to solve the above technical problems, in a second aspect, the embodiment of the present application provides a configurable structured data parsing and processing device, which adopts the following technical solution:

[0103] The device comprises a reading module, a parsing module, a conversion module and an output module, wherein:

[0104] A reading module is used to read a predefined configuration file, obtain configuration information, parse and load it; the configuration information is the configured data processing information; the data processing information includes field mapping relationships and conversion rules;

[0105] A parsing module, used to receive and read a data source, and perform a parsing operation on the data source according to the field mapping relationship;

[0106] A conversion module, used to perform a conversion operation on the parsed data according to the conversion rule to obtain converted target format data; wherein each field type corresponds to a conversion rule;

[0107] An output module is used to output the target format data.

[0108] Furthermore, the data processing information also includes a data source structure, a target data structure and data processing logic;

[0109] The reading module is also used to read the data source structure and the target data structure through configuration; the data source structure and the target data structure both contain field names and field types of structured data.

[0110] Furthermore, the parsing module is further used to determine the field mapping relationship corresponding to the data source based on the data format read from the data source;

[0111] If there is a corresponding field mapping relationship, extract the value of each field, and map each field to a corresponding position in the target data structure according to the field mapping relationship;

[0112] If there is no corresponding field mapping relationship, the field is ignored and no mapping is performed.

[0113] The conversion rules include: converting a string type into a numeric type, and / or normalizing a predetermined field; the target format data includes JSON, XML or other custom formats.

[0114] Furthermore, the device also includes a change module, which is used to receive a user configuration change request; the configuration change request includes adding, modifying or deleting a configuration rule; and changing the configuration file according to the configuration modification request.

[0115] In addition, an embodiment of the present application also discloses an electronic device, which includes: one or more processors, and a memory, wherein the memory is used to store one or more computer programs; characterized in that the computer program is configured to be executed by the one or more processors, and the program includes steps for executing the configurable structured data parsing and processing method described in the first aspect above.

[0116] In addition, an embodiment of the present application further provides a storage medium storing a computer program; the program is loaded and executed by a processor to implement the steps of the structured data parsing processing method based on configuration as described in the first aspect above.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0119] The units described as separate components may or may not be physically separated. As a unit, it can be appreciated by those skilled in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0120] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a grid device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-On lyMemory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0122] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for analyzing and processing structured data based on configuration, characterized in that: The method comprises: Read a predefined configuration file, obtain configuration information, parse and load it; the configuration information is configured data processing information; the data processing information includes field mapping relationship and conversion rules; Receive and read the data source, and perform a parsing operation on the data source according to the field mapping relationship; According to the conversion rules, the parsed data is converted to obtain the converted target format data; wherein each field type corresponds to a conversion rule; Output the target format data.

2. The method for analyzing and processing structured data based on configuration according to claim 1, characterized in that: The data processing information also includes a data source structure, a target data structure and data processing logic; The reading of the predefined configuration file, obtaining the configuration information and parsing and loading the configuration information includes: reading the data source structure and the target data structure; The data source structure and the target data structure both contain field names and field types of structured data.

3. The method for analyzing and processing structured data based on configuration according to claim 2, characterized in that: The receiving and reading of the data source, and performing a parsing operation on the data source according to the field mapping relationship, includes: The data format read from the data source determines the field mapping relationship corresponding to the data source; If there is a corresponding field mapping relationship, extract the value of each field, and map each field to a corresponding position in the target data structure according to the field mapping relationship; If there is no corresponding field mapping relationship, the field is ignored and no mapping is performed.

4. The method for analyzing and processing structured data based on configuration according to claim 3, characterized in that: The method further includes: receiving a user configuration change request; the configuration change request includes adding, modifying or deleting a configuration rule; and modifying the configuration file according to the configuration modification request.

5. The method for analyzing and processing structured data based on configuration according to claim 4, characterized in that: The conversion rules include: converting a string type into a numeric type, and / or normalizing a predetermined field; The target format data includes JSON, XML or other custom formats.

6. A configuration-based structured data analysis and processing device, characterized in that: The device includes a reading module, a parsing module, a conversion module and an output module, wherein: A reading module is used to read a predefined configuration file, obtain configuration information, parse and load it; the configuration information is the configured data processing information; the data processing information includes field mapping relationships and conversion rules; A parsing module, used to receive and read a data source, and perform a parsing operation on the data source according to the field mapping relationship; A conversion module, used to perform a conversion operation on the parsed data according to the conversion rule to obtain converted target format data; wherein each field type corresponds to a conversion rule; An output module is used to output the target format data.

7. The configurable structured data analysis and processing device according to claim 6, characterized in that: The data processing information also includes a data source structure, a target data structure and data processing logic; The reading module is also used to read the data source structure and the target data structure through configuration; the data source structure and the target data structure both contain field names and field types of structured data.

8. The configurable structured data analysis and processing device according to claim 7, characterized in that: The parsing module is further used to read the data format from the data source and determine the field mapping relationship corresponding to the data source; If there is a corresponding field mapping relationship, extract the value of each field, and map each field to a corresponding position in the target data structure according to the field mapping relationship; If there is no corresponding field mapping relationship, the field is ignored and no mapping is performed.

9. An electronic device, comprising: One or more processors, a memory, wherein the memory is used to store one or more computer programs; characterized in that the computer program is configured to be executed by the one or more processors, and the program includes steps for executing the configurable structured data parsing and processing method as described in any one of claims 1-5.

10. A storage medium storing a computer program; the program is loaded and executed by a processor to implement the steps of the configuration-based structured data parsing and processing method as described in any one of claims 1 to 5.

Images