Data processing method and device, terminal equipment and computer readable storage medium

Abstract

The application is suitable for the technical field of data processing, and provides a data processing method and device, terminal equipment and computer readable storage medium, comprising: obtaining semi-structured data to be processed; identifying the data type of the semi-structured data according to a trained recognition model; generating a table building statement according to the identified type; building a table in a preset database according to the table building statement to obtain a database table; and importing the semi-structured data into the database table. The application uses machine learning to analyze and process semi-structured table data, is suitable for all kinds of table data, has strong versatility, and greatly increases the processing efficiency of table data.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, terminal and computer-readable storage medium. Background Technology

[0002] The information age has brought about a massive amount of data files, including structured data, semi-structured data, and unstructured data. Semi-structured data files include core data assets of governments and enterprises, which have high application value. Therefore, parsing and managing this semi-structured data has become an urgent technical problem to be solved.

[0003] However, traditional technologies can only process and analyze structured data, and cannot directly process semi-structured and unstructured data. They also have low parsing efficiency for tabular data, lack targeted optimization, require a large amount of development work, have complex processes, and do not support tabular data such as Excel, resulting in poor versatility. Summary of the Invention

[0004] This application provides a data processing method, apparatus, terminal device, and computer-readable storage medium that can automatically parse and store various types of semi-structured tabular data.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Acquire the semi-structured data to be processed;

[0007] The data type of the semi-structured data is identified based on the trained recognition model;

[0008] Generate table creation statements based on the identified type;

[0009] Create a table in the preset database according to the table creation statement to obtain the database table;

[0010] Import the semi-structured data into the database table.

[0011] In this embodiment, the data type of semi-structured data is identified based on the trained recognition model, and a table creation statement is generated according to the recognition type. The table creation statement is then used to create a table in a preset database to obtain a database table. Finally, the semi-structured data is imported into the database table. In other words, the initial stage of the transformation from semi-structured data to structured data involves using a machine learning-trained recognition model to quickly identify and classify the data. Through the above method, it is not necessary to develop a large number of components in the process of processing semi-structured data. It is only necessary to use the recognition model to identify and process the data, and then complete other transformation work from semi-structured data to structured data based on the recognition information. This method simplifies the complex process and improves the processing of semi-structured data, enabling more efficient automatic parsing and database entry of various types of semi-structured data.

[0012] In one possible implementation of the first aspect, the data processing method further includes:

[0013] Obtain sample data, which is semi-structured data;

[0014] The recognition model is trained based on the feature information to obtain the trained recognition model.

[0015] In one possible implementation of the first aspect, acquiring sample data includes:

[0016] Obtain the raw data to be trained;

[0017] The sample data is obtained by extracting feature information from the original data according to the preset feature attributes.

[0018] In one possible implementation of the first aspect, generating the table creation statement based on the data type includes:

[0019] Obtain the preset template corresponding to the data type;

[0020] Extract attribute information from the semi-structured data according to the preset template;

[0021] The table creation statement is generated by concatenating the attribute information.

[0022] In one possible implementation of the first aspect, the step of generating the table creation statement by concatenating the attribute information includes:

[0023] The attribute information is converted into converted data in a preset format, which matches the preset database.

[0024] The table creation statement is generated by concatenating the transformed data.

[0025] In one possible implementation of the first aspect, importing the semi-structured data into the database table includes:

[0026] If the data type of the semi-structured data is the first type, then the semi-structured data is imported into the database table according to the preset script;

[0027] If the data type of the semi-structured data is the second type, then the semi-structured data is imported into the database table using a preset engine, wherein the data structure of the second type of data is more complex than that of the first type of data.

[0028] In one possible implementation of the first aspect, the data format of the semi-structured data includes any one of the following formats: ordinary format, multi-row header format, crosstab format, multi-row data fusion format, etc.

[0029] Secondly, embodiments of this application provide a data processing apparatus, including:

[0030] The acquisition unit acquires the semi-structured data to be processed.

[0031] The identification unit identifies the data type of the semi-structured data based on the trained identification model;

[0032] The generation unit generates a table creation statement based on the identified type;

[0033] The table creation unit creates tables in a preset database according to the table creation statement, thus obtaining the database tables;

[0034] The import unit imports the semi-structured data into the database table.

[0035] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above data processing methods.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0037] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the data processing method described in any one of the first aspects.

[0038] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a system schematic diagram of a data processing method provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of the table creation statement generation process provided in an embodiment of this application;

[0042] Figure 3 This is a structural diagram of a data processing apparatus provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0044] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0045] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0046] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0047] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0048] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0049] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0050] Machine learning involves using a portion of data to learn from and then using that data to make predictions and judgments. Structured data refers to data that can be represented and stored using relational databases, and is presented in a two-dimensional format. Semi-structured tabular data refers to data stored in tables, but these tables are not standard two-dimensional tables; their format is not standardized, and they may contain multiple headers or multiple two-dimensional tables stored within a single table.

[0051] The information age has brought about a massive amount of data files, including structured, semi-structured, and unstructured data. Traditional technologies can only process and analyze structured data, and cannot directly process semi-structured and unstructured data. Semi-structured data files include core data assets of governments and enterprises, and have high application value. How to parse and manage this semi-structured data has become an urgent technical problem to be solved.

[0052] Some implementations identify the type and version of semi-structured data files through an upload component, parse the semi-structured data files to generate structured data through a parsing component, and then present, store, and output the structured data through a data object component. Because the script code snippets in the parsing component are modifiable, it offers high flexibility and can be applied to parsing different types of semi-structured data. However, this method of parsing semi-structured data using custom components requires customized development for each type of semi-structured data, resulting in a large development workload, a complex process, and poor versatility as it does not support tabular data such as Excel. Other related technologies for processing semi-structured data are less efficient at parsing tabular data and lack targeted optimization.

[0053] To address the aforementioned issues, embodiments of this application provide a data processing method. This method eliminates the need to develop numerous components for processing semi-structured data. Instead, it utilizes a recognition model to process the data and then uses the recognition information to complete other transformations from semi-structured to structured data. This simplifies complex processes, improves the processing of semi-structured data, and more efficiently enables automatic parsing and storage of various types of semi-structured data.

[0054] The data processing method provided in this application is applicable to various types of semi-structured data. See also... Figure 1 This is a schematic diagram of the system flow of the data processing method provided in the embodiments of this application. It is intended as an example and not a limitation. The method includes the following steps:

[0055] Step S101: Obtain the semi-structured data to be processed.

[0056] In this embodiment, semi-structured data refers to data stored in tables, but the tables are not standard two-dimensional tables. The format is not standardized and may contain multiple headers or multiple two-dimensional tables stored in one table. Common data formats include ordinary format, multi-row header format, cross-tab format, multi-segment merging format, multi-row data merging format, and large-scale data import format. The generation method is to use a script to generate the semi-structured data to be processed based on characteristic attributes such as file size, number of file lines, whether there is a header row, number of header rows, number of headers, and number of rows in the header position.

[0057] Step S102: Identify the data type of the semi-structured data based on the trained recognition model.

[0058] In this embodiment, the trained recognition model is primarily trained using machine learning. Machine learning, a branch of artificial intelligence and computer science, focuses on using data and algorithms to mimic human learning methods and gradually improve its accuracy. Machine learning is an important component of the ever-growing field of data science. By using statistical methods, algorithms are trained to classify or predict data, revealing key insights in data mining projects. The three types of machine learning algorithms include unsupervised learning, supervised learning, and reinforcement learning. In this embodiment, supervised learning algorithms are used to train the data to obtain a training model. As the semi-structured data samples increase, the model is continuously improved. When new semi-structured data needs to be added to the database, the training model automatically identifies and parses the feature values ​​of the data and classifies it using the model. The classification information is based on the data's feature information, which includes the data file size, number of lines, whether there is a title, number of title lines, number of table headers, and number of rows in the header position.

[0059] Step S103: Generate a table creation statement based on the identified type.

[0060] In this embodiment, to convert semi-structured data into structured data, a specific relational database and specific table creation statements within that database are required. Structured data refers to data that can be represented and stored using relational data structures, exhibiting a two-dimensional format. A relational database is a database that uses a relational model to organize data, storing it in rows and columns for user understanding. These rows and columns are called tables, and a set of tables constitutes a database. Users retrieve data from the database through queries, which are executable codes used to define specific areas within the database. A relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them. Semi-structured data tables are not standard two-dimensional tables and have irregular formats. Therefore, before converting semi-structured data into structured data and storing it in a relational database, it is necessary to obtain the table creation statements corresponding to standard two-dimensional tables in the relational database. For example, MySQL is a relational database management system. Relational databases store data in different tables instead of putting all data in one large repository, thus increasing speed and flexibility. The table creation language used is SQL, the most commonly used standardized language for accessing databases. MySQL software adopts a dual-licensing policy, with a community edition and a commercial edition. Due to its small size, high speed, low total cost of ownership, and especially its open-source nature, MySQL is generally chosen as the database for the development of small, medium and large websites.

[0061] In one embodiment, see Figure 2This is a schematic diagram of the table creation statement generation process provided in an embodiment of this application, such as... Figure 2 As shown, step S103 further includes:

[0062] Step S201: Obtain the preset template corresponding to the data type;

[0063] Step S202: Extract attribute information from the semi-structured data according to the preset template.

[0064] Step S203: Generate the table creation statement by concatenating the attribute information.

[0065] In this embodiment, the table creation statements are preset according to categories, with one table creation statement generator for each category, which is currently implemented manually. Before generating the table creation statements, a preset template is set in advance based on the database. This preset template contains all the feature information of the sample data. The preset template provides the feature information required for the document format for generating the table creation statements. For example, if the document of semi-structured data is in a normal format, the corresponding attribute information of this document, such as table name, table description, table field names, and Chinese names of the table, needs to be extracted first according to the preset template. Then, the placeholders of each useful field are replaced and concatenated to form the table creation statement required for table creation.

[0066] In one embodiment, step S203 further includes:

[0067] The attribute information is converted into converted data in a preset format, which is matched with the preset database.

[0068] The table creation statement is generated by concatenating the transformed data.

[0069] In this embodiment, after obtaining the data information corresponding to the preset template from the above document parsing results, it is necessary to perform necessary conversions based on the output format of the document and the preset format in the preset database. For example, semi-structured documents of the ordinary format and multi-line header format type can be converted into the pinyin abbreviation or English format required by the preset format. Finally, the converted data is concatenated to form the table creation statement required for table creation in the preset data.

[0070] Step S104: Create a table in the preset database according to the table creation statement to obtain the database table.

[0071] In this embodiment, the relational database is a database table that can obtain data in a two-dimensional form using table creation statements, that is, a database table that can transform semi-structured data into structured data.

[0072] Step S105: Import the semi-structured data into the database table.

[0073] By using data acquisition tools, semi-structured books were collected into a two-dimensional database table built in a relational database using the table creation statements mentioned above, thus realizing the transformation of semi-structured tabular data into structured data.

[0074] In one embodiment, step S105 further includes:

[0075] If the data type of the semi-structured data is the first type, then the semi-structured data is imported into the database table according to the preset script.

[0076] If the data type of the semi-structured data is the second type, then the semi-structured data is imported into the database table using a preset engine, wherein the data structure of the second type of data is more complex than that of the first type of data.

[0077] Machine learning is used to learn from semi-structured data, which can be categorized into two types: the first type is a standard format, while the second type includes formats such as multi-row headers, cross-headers, and merged multi-row data. When the document format is of the first type, a preset script can be used to parse and import the semi-structured data into the database table. However, for complex data tables (the second type), the data structure is more complex than the first type. Therefore, when importing this data into the database, an integrated data acquisition engine is needed. Examples of acquisition engines include Kettle and NiFi; we commonly use Kettle.

[0078] In one embodiment, the data processing method further includes:

[0079] Obtain sample data, which is semi-structured data.

[0080] The recognition model is trained based on the feature information to obtain the trained recognition model.

[0081] Sample data is a commonly used concept in probability calculations. For example, consider a batch of 10,000 rounds of ammunition, where 90% are required to be of excellent performance. It would be meaningless to fire all 10,000 rounds. Instead, 100 rounds can be randomly selected, and the number of high-performance rounds can be calculated to determine the success rate. Assuming a success rate of 94%, since it's random and these 100 rounds represent a certain proportion of the 10,000 rounds, we can say that the success rate of this batch of ammunition is 94%, meeting the requirement. The value calculated from these 100 rounds is called sample data, meaning sampling. Since semi-structured data is data between structured and unstructured data, it has wide applications in everyday life, including email systems, web clusters, mathematical resource libraries, data mining systems, and archival systems. These applications have basic storage needs such as data storage, data backup, data sharing, and data archiving. Therefore, due to the large range of semi-structured data types, sample data is needed when learning from semi-structured data for training. In this embodiment, the sample data is the semi-structured data generated using the script mentioned above.

[0082] In one embodiment, the above steps for obtaining sample data further include:

[0083] Obtain the raw data to be trained.

[0084] The sample data is obtained by extracting feature information from the original data according to the preset feature attributes.

[0085] In a database, raw data, also known as user data, is data that has not been processed or simplified. In this embodiment, the raw data to be trained is semi-structured data obtained through a script.

[0086] Before training the model, it is necessary to obtain the feature information in the original data as sample data for training based on the preset feature attributes. The feature attributes generally include: file size, number of lines in the file, whether there is a header line, number of header lines, number of table headers, and number of rows in the header position. These feature attributes are used to construct the training and test sets.

[0087] In artificial intelligence, facing a large amount of user-input data or materials, accurately and easily recognizing and outputting the desired image or speech from messy content is not so easy. Therefore, algorithms are particularly important; an algorithm is essentially a model. Besides the core recognition engine, an algorithm also includes various configuration parameters. For example, a semi-structured data recognition model can recognize multiple parameters such as file size, number of lines, presence of header rows, and number of table headers. A mature recognition engine generally doesn't change its core content frequently. To achieve the goal of "successful recognition," we need to adjust the configuration parameters. For different inputs, we configure different parameter values. Finally, we statistically analyze the results and select a set of parameter values ​​that are relatively balanced across all aspects and have a high recognition rate. This set of parameter values ​​is the result we obtain after training. This is the training process, also called model training. In this embodiment, a Support Vector Machine (SVM) is used for supervised learning of the sample data. It is a type of classification algorithm that mainly acquires the feature information of Excel files, such as file size, number of lines, presence of header rows, number of header rows, and number of rows in the header position, to construct training and testing sets, thereby obtaining the trained model.

[0088] Corresponding to the data processing method described in the above embodiments, Figure 3 This is a structural block diagram of the data processing apparatus provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0089] Reference Figure 3 The device includes:

[0090] Unit 31 is used to acquire the semi-structured data to be processed.

[0091] The identification unit 32 identifies the data type of the semi-structured data based on the trained identification model.

[0092] The generation unit 33 generates a table creation statement based on the identified type.

[0093] Table creation unit 34 creates a table in a preset database according to the table creation statement, thereby obtaining a database table.

[0094] Import unit 35 imports the semi-structured data into the database table.

[0095] Accordingly, the identification unit 32 is also used for:

[0096] Obtain sample data, which is semi-structured data;

[0097] The recognition model is trained based on the feature information to obtain the trained recognition model.

[0098] Optionally, the generating unit 33 is also used for:

[0099] Obtain the preset template corresponding to the data type;

[0100] Extract attribute information from the semi-structured data according to the preset template;

[0101] The table creation statement is generated by concatenating the attribute information.

[0102] Optionally, the generating unit 33 is also used for:

[0103] The attribute information is converted into converted data in a preset format, which matches the preset database.

[0104] The table creation statement is generated by concatenating the transformed data.

[0105] Optionally, the import unit 35 is also used for:

[0106] If the data type of the semi-structured data is the first type, then the semi-structured data is imported into the database table according to the preset script;

[0107] If the data type of the semi-structured data is the second type, then the semi-structured data is imported into the database table using a preset engine, wherein the data structure of the second type of data is more complex than that of the first type of data.

[0108] in addition, Figure 3 The data processing device shown can be a software unit, hardware unit, or a combination of software and hardware built into an existing terminal device, or it can be integrated into the terminal device as an independent component, or it can exist as an independent terminal device.

[0109] Figure 4 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 4 As shown, the terminal device 4 in this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown) a processor, a memory 41, and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, which, when executing the computer program 42, implements the steps in any of the above-described data processing method embodiments.

[0110] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4This is merely an example of terminal device 4 and does not constitute a limitation on terminal device 4. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0111] The processor 40 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0112] In some embodiments, the memory 41 may be an internal storage unit of the terminal device 4, such as a hard disk or memory of the terminal device 4. In other embodiments, the memory 41 may be an external storage device of the terminal device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 4. Furthermore, the memory 41 may include both internal and external storage units of the terminal device 4. The memory 41 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0113] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0114] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Acquire the semi-structured data to be processed; The data format of the semi-structured data includes any one of the following: ordinary format, multi-row header format, crosstab format, and multi-row data merging format; The semi-structured data is identified based on the trained recognition model. The semi-structured data has two formats: a first type is a standard format, and a second type includes multi-row header formats, crosstab formats, and multi-row merged data formats. The recognition model is trained based on the feature information of sample data, which is semi-structured data. The feature information includes file size, number of lines, presence of a header line, number of header lines, number of header rows, and number of rows in the header position. Generating a table creation statement based on the data type includes: obtaining a preset template corresponding to the data type; extracting attribute information from the semi-structured data based on the preset template; and concatenating the attribute information to generate the table creation statement. The preset template provides the document format features required for generating the table creation statement. When the document of the semi-structured data is in a standard format, the attribute information corresponding to the document is first extracted based on the preset template. The attribute information includes the table name, table description, table field names, and Chinese table names. Then, placeholders are replaced in each useful field, and the data is concatenated to form the table creation statement required for table creation. Create a table in the preset database according to the table creation statement to obtain the database table; Importing the semi-structured data into the database table includes: if the data type of the semi-structured data is a first type, then importing the semi-structured data into the database table according to a preset script; if the data type of the semi-structured data is a second type, then importing the semi-structured data into the database table using a preset engine.

2. The data processing method as described in claim 1, characterized in that, The method further includes: Obtain sample data; The recognition model is trained based on the feature information to obtain the trained recognition model.

3. The data processing method as described in claim 2, characterized in that, The acquisition of sample data includes: Obtain the raw data to be trained; The sample data is obtained by extracting feature information from the original data according to the preset feature attributes.

4. The data processing method as described in claim 1, characterized in that, The step of generating the table creation statement based on the attribute information includes: The attribute information is converted into converted data in a preset format, which matches the preset database. The table creation statement is generated by concatenating the transformed data.

5. A data processing apparatus, characterized in that, include: The acquisition unit acquires the semi-structured data to be processed. The data format of the semi-structured data includes any one of the following: ordinary format, multi-row header format, crosstab format, and multi-row data merging format; The recognition unit identifies the data type of the semi-structured data based on the trained recognition model. The semi-structured data has two data formats: a first type is a standard format, and a second type includes multi-row header formats, crosstab formats, and multi-row merged data formats. The recognition model is trained based on the feature information of sample data, which is semi-structured data. The feature information includes file size, number of lines, presence of a header line, number of header lines, number of header rows, and number of rows in the header position. The generation unit generates table creation statements based on the data type. The table creation unit creates tables in a preset database according to the table creation statement, thus obtaining the database tables; The import unit imports the semi-structured data into the database table; The generation unit is also used for: Obtain a preset template corresponding to the data type; extract attribute information from the semi-structured data according to the preset template; concatenate the attribute information to generate the table creation statement; wherein, the preset template provides feature information required for the document format for generating the table creation statement. When the document of the semi-structured data is in a normal format, first extract the attribute information corresponding to the document according to the preset template. The attribute information includes table name, table description, table field names, and Chinese names in the table; then, replace the placeholders in each useful field and concatenate them to form the table creation statement required for table creation. The import unit is also used for: If the data type of the semi-structured data is a first type, then the semi-structured data is imported into the database table according to a preset script; if the data type of the semi-structured data is a second type, then the semi-structured data is imported into the database table using a preset engine.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images