Data import system and data import method using multi-stage structural transformation.

Abstract

This invention provides a data import system and method that allows for the flexible and efficient import of unspecified structured data with diverse structures, without depending on a specific data format, simply by changing settings. [Solution] The method comprises a document structure parser that analyzes the structure of input data and converts it into formatted data having a predetermined intermediate data structure; a template storage means that stores a template describing the processing content to be applied to the formatted data; a template processing means that assigns meaning to the values ​​contained in the formatted data based on the formatted data generated by the document structure parser and the template stored in the template storage means, and generates evaluated data having a structure that conforms to the input requirements required by a subsequent data processing means; and a data processing means that performs data registration or update processing to a data store based on the evaluated data generated by the template processing means.

Description

Technical Field

[0005] ,

[0004] , , , , ,

[0003]

[0001] The present invention relates to a data import system and a data import method for importing text data generated by an external system or the like into a data store of the own system. In particular, it relates to a technique for importing unspecified structured data whose input data structure is not specified or has various structures.

Background Art

[0002] In information systems of enterprises and the like, a so-called data import function for importing data from external systems or files and registering it in a database or the like of the own system is widely used. For example, when constructing a new EC site (E Commerce site), a case where the output result (such as a CSV file) of the order management data handled by the old EC site is imported into the sales management system of the new EC site corresponds to this.

[0003] Generally, such a data import function is implemented as a dedicated processing logic after specifying the format of the input data (order of columns, delimiter characters, etc.) and the corresponding processing content (which column to store in which database column, etc.).

[0004] In relatively high-functional import processing, there are also those that support so-called mapping processing such as swapping columns of input data and ignoring specific columns. However, with these conventional methods, there is a problem that they cannot handle complex structure conversions in cases where the structure of the input data itself is significantly different from the premise, for example, when expressing one data across multiple rows, when column splitting / merging or value calculation is required.

[0005] Therefore, every time it became necessary to support a new input data format, it was necessary to develop an import process tailored to that format, resulting in significant development costs and time. Furthermore, because the data structure analysis and semantic interpretation (what each value represents) were tightly coupled, the system lacked flexibility and made it difficult to adapt to specification changes. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2025-34258 [Overview of the project] [Problems that the invention aims to solve]

[0007] This invention has been made in view of the problems of the prior art described above, and its purpose is to provide a data import system and a data import method that can flexibly and efficiently import unspecified structured data with diverse structures, without depending on a specific data format, simply by changing settings. [Means for solving the problem]

[0008] This problem concerns a data import system that takes input data with a predetermined structure, received from an external source, and imports it into a predetermined data store. A document structure parser analyzes the structure of the input data and converts it into formatted data with a predetermined intermediate data structure, A template storage means that stores a template containing a description of the processing content to be applied to the formatted data, A template processing means that assigns meaning to the values ​​contained in the formatted data based on the formatted data generated by the document structure parser and the template stored in the template storage means, and generates evaluated data having a structure that conforms to the input requirements of a subsequent processing module. A data processing means that performs data registration or update processing to the data store based on the evaluated data generated by the template processing means, This is solved by a data import system characterized by having the following features.

[0009] Furthermore, it is advantageous for the document structure parser to not be involved in the semantic content of the values ​​contained in the input data, but to perform only structural transformation.

[0010] Furthermore, the template processing means may also generate the evaluated data by combining or splitting the multiple values ​​contained in the formatted data, or by performing value transformations involving calculations or format conversions.

[0011] Here, the template includes processing instruction information for controlling the operation of the data processing means, It is advantageous for the template processing means to dynamically generate the processing instruction information according to the status of the values ​​of the formatted data and include it in the evaluated data.

[0012] The challenge is further to provide a data import method for importing input data with a predetermined structure, which is received from an external source, into a predetermined data store. The document structure parser analyzes the structure of the input data and converts it into formatted data with a predetermined intermediate data structure. The template processing means assigns meaning to the values ​​contained in the formatted data based on the formatted data and a pre-stored template, and generates evaluated data having a structure that conforms to the input requirements of a subsequent data processing means. The data processing means performs a step of registering or updating data in the data store based on the evaluated data, This is resolved by a data import method characterized by including [a specific feature]. [Effects of the Invention]

[0013] According to the present invention, since the "structural analysis" and "semantic annotation" of data are separated, the following remarkable effects can be achieved. (1) Even for input data in an unknown format, by combining a document structure parser that interprets its structure and a template that defines the processing content, it becomes possible to flexibly capture data without developing a new program. (2) By describing the template, it is possible to easily realize advanced data conversions such as not only simple column mappings, but also combination and splitting of values across rows and columns, value conversions such as arithmetic operations, and dynamic instructions for processing according to the data situation (e.g., allowing overwriting updates when the ID matches, etc.). (3) As a result, the costs and time associated with the development and maintenance of the data import function can be significantly reduced.

Brief Description of the Drawings

[0014] [Figure 1] Conceptual diagram showing the configuration and processing flow of the data import system according to the present invention [Figure 2] Sequence diagram showing the processing sequence according to an embodiment of the present invention [Figure 3] Diagram of variations in data structure [Figure 4] It is a diagram showing input data in a specific example of the shipment achievement import processing. [Figure 5] It is a diagram showing a template in a specific example of the shipment achievement import processing. [Figure 6] It is a diagram showing output data (evaluated data) in a specific example of the shipment achievement import processing. [Figure 7] It is a diagram showing the input specification of the processing module in a specific example of the shipment achievement import processing. [Figure 8] Diagram of an example of the implementation of the system according to the present invention <000009s><00s0091>Diagram of an example of the implementation of the system according to the present invention [Figure 10] Diagram of an example of the implementation of the system according to the present invention [Figure 11] Conceptual diagram showing the flow of conventional data import processing

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a conceptual diagram showing the overall configuration and processing flow of a data import system according to an embodiment of the present invention. This system operates on a computer and mainly has a document configuration parser, template processing means, template storage means, and data processing means (processing modules).

[0016] [Overview of the processing flow] The processing flow of this system is roughly divided into three phases, namely, an analysis phase, an evaluation phase, and a data processing phase. The first phase is the analysis phase. Input data (for example, a text file such as CSV) provided to the system by a user (a human or an automated process) is analyzed by a document structure parser. This document structure parser is not concerned with the semantic content of the input data at all, interprets only its structure (such as row and column delimiters), and converts it into formatted data having a prescribed single structure (for example, JSON format) that is easy to handle in subsequent processing.

[0017] Here, the input data is the entire string in the format used by the user for input, and is data that has not been processed or converted. As the input data, text is used. This input data is a set of multiple data with a set of values regarded as one data, and has data delimiters and value delimiters according to some rules.

[0018] In the analysis phase, in addition to providing input data, the user specifies a document structure parser, a template (described later), and a data processing means (processing module) (described later). This ensures that an analysis module is prepared in the analysis phase that corresponds to the structure of the input text data. The analysis module assumes the presence of multiple data points within multiple input texts and is configured to identify each data point and then format the individual values ​​within each data point into a two-dimensional array. As a result, regardless of the underlying structure, the input data becomes data in a predetermined format with the same data delimiters and value delimiters, i.e., "formatted data."

[0019] The second phase is the evaluation phase. Here, a template processing means (e.g., a template engine) evaluates the formatted data generated in the first phase by combining it with a template specified in advance by the user. The template contains rules that define what business meaning (e.g., shipping code) a particular part of the formatted data (e.g., a specific row and column in an array) has. The template processing means extracts values ​​from the formatted data according to these rules, performs transformations such as calculations, concatenation, and splitting, and then adds processing instructions to the data processing means (processing module) described later to generate the final "evaluated data." This evaluated data is in a format that the data processing means (processing module) can directly interpret (e.g., JSON format with specific keys).

[0020] The third phase is the data processing phase. The evaluated data generated in the second phase is handed over to the data processing means (processing module). The data processing means (processing module) performs specific processing, such as registering or updating records, on a data store such as a database, according to the data body and processing method instructions contained in the evaluated data.

[0021] [Comparison with conventional technology] In the conventional import process shown in Figure 7, the format of the input data and the implementation of the import process are tightly coupled, and only simple mapping such as selecting and rearranging items can be performed. In contrast, the system shown in Figure 1 separates structural analysis and semantic assignment, and by inserting flexible transformations using templates, it can achieve complex structural transformations that were difficult to handle with conventional techniques, such as splitting and joining in the column direction, splitting and joining in the row direction, and transforming values, as shown in Figure 2.

[0022] [Sequence] Figure 3 shows the processing sequence when this system is implemented as a web application. 1. In the first step, the client (web browser, etc.) specifies the input data, the document structure parser, template, data processing means (processing module), etc. to be used, and uploads them to the main application. 2. The main application launches the specified document structure parser, parses the input data, and generates formatted data with a generalized structure (2). 3. Next, the main application starts a template processing means (template engine) and has it evaluate the individual formatted data and the specified template to generate evaluated data (3). This process is repeated for each data identified in (2). 4. Finally, the main application activates the specified data processing means (processing module), passes the generated evaluated data to it, and executes the registration process to the data store (4). This process is also repeated for each individual data item.

[0023] [Example: Importing shipment data] Next, an example of importing shipment performance data will be described based on Figures 4 to 7. Figure 4 shows an example of input data. In this example, the input data is in CSV format and contains a shipment code to identify the shipment, a shipment date, and a shipment tracking number for inquiry. The header row is not required in this invention and is therefore skipped. In this example, since there are multiple shipment tracking numbers for a single shipment code ("T0001A", "T0001B"), the data spans two lines.

[0024] The structural analysis parser parses this CSV file. Here, for example, a parser configured to recognize "one data entry up to the change in the first column (shipping code)" is used. As a result, two rows of data with "S0001" in the first column are passed to the template engine as a single formatted data entry in the form of a two-dimensional array (tabular data with rows and columns).

[0025] The user describes, as a template, how to convert this formatted data into data for data processing tools (processing modules) (Figure 5). Within the template, the formatted data can be referenced using the variable data.Data, and each value can be accessed in the format data.Data[row][column].

[0026] In this example, Assign the first column of the first row (data.Data[0][0]) to "ShipmentCode". Assign the second column of the first row (data.Data[0][1]) to "ShippedDate". • The "TrackingNumber" is assigned as an array by aggregating the values ​​of the third column of the first row (data.Data[0][2]) and the third column of the second row (data.Data[1][2]). The processing instruction key "AssignTrackingNumberIfShipped" is assigned a fixed value of true.

[0027] In this way, using templates makes it easy to aggregate data spanning multiple rows to generate arrays, or to add processing instructions for fixed values ​​that are not present in the input data.

[0028] When the template engine applies the above template, "evaluated data" as shown in Figure 6 is generated. This data is in a format that can be directly interpreted by subsequent processing modules.

[0029] The processing module is pre-implemented with input specifications as shown in Figure 7. This module receives the evaluated data from Figure 6, searches for data using "ShipmentCode," and updates "TrackingNumber" and "ShippedDate." It also changes its behavior depending on the value of processing instruction keys such as "AssignTrackingNumberIfShipped." Importantly, this processing module itself does not need to know anything about the format of the original input data in CSV format.

[0030] Figures 8 to 11 show examples of the system's implementation screen. Users can flexibly combine and configure the data processing means (processing modules, Figure 9) and document structure parser (character encoding, file structure, data delimiter method, etc., Figure 10) for each import process. This makes it possible to handle completely different input data formats while using the same processing module, or conversely, process the same data format with different logic, all without coding.

[0031] As described above, according to the present invention, by separating the data import process into three phases—an analysis phase, an evaluation phase, and a data processing phase—and by providing a flexible conversion layer using templates, particularly between the analysis phase and the evaluation phase, it is possible to construct a data import system that is extremely versatile and scalable.

Claims

1. A data import system that takes input data with a predetermined structure, received from an external source, and imports it into a predetermined data store. A document structure parser analyzes the structure of the input data and converts it into formatted data with a predetermined intermediate data structure, A template storage means that stores a template containing a description of the processing content to be applied to the formatted data, A template processing means that assigns meaning to the values ​​contained in the formatted data based on the formatted data generated by the document structure parser and the template stored in the template storage means, and generates evaluated data having a structure that conforms to the input requirements of a subsequent data processing means. A data processing means that performs data registration or update processing to the data store based on the evaluated data generated by the template processing means, Equipped with, The system is configured to allow specification of document structure parsers, templates, and data processing means (processing modules). The template processing means combines or splits multiple values ​​in the formatted data, or performs value transformations involving calculations or format conversions, to generate the evaluated data. The template includes processing instruction information for controlling the operation of the data processing means, The template processing means dynamically generates the processing instruction information according to the status of the values ​​of the formatted data and includes it in the evaluated data. A data import system characterized by having [this feature].

2. The data import system according to claim 1, characterized in that the document structure parser does not concern itself with the semantic content of the values ​​contained in the input data, but only performs structural transformation.

3. A data import method for importing input data having a predetermined structure, which is input from an external source, into a predetermined data store, The steps include specifying the document structure parser, template, and data processing means (processing module), The document structure parser analyzes the structure of the input data and converts it into formatted data with a predetermined intermediate data structure. The template processing means performs the steps of assigning meaning to the values ​​contained in the formatted data based on the formatted data and a pre-stored template, and generating evaluated data having a structure that conforms to the input requirements of a subsequent processing module, The data processing means performs a step of registering or updating data in the data store based on the evaluated data, Includes, The template processing means combines or splits multiple values ​​in the formatted data, or performs value transformations involving calculations or format conversions, to generate the evaluated data. The template includes processing instruction information for controlling the operation of the data processing means, The template processing means dynamically generates the processing instruction information according to the status of the values ​​of the formatted data and includes it in the evaluated data. A data import method characterized by the following.

Images