Data format conversion method, system and equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a data format conversion method, system and device and a storage medium, and the method comprises the steps: reading and storing content data in a target document by using a third-party library; acquiring database connection information and a field mapping rule; splicing the content data into a structured statement which can be identified by the database based on a field mapping rule; establishing communication connection with the database by utilizing database connection information, and storing the structured statement to the database; and accessing the database, and querying the structured statement to verify the storage validity of the structured statement. According to the invention, automatic conversion of document formats is realized, and the method has high flexibility and accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a data format conversion method, system, device and storage medium. Background Art

[0002] In today's highly information-based era, data processing and analysis have become an indispensable core link in all walks of life. In this field, Excel, Word and other format documents generally play an important role in data storage and transmission. However, if the data in these documents are to be directly integrated into the database system for further operation, they face many obstacles and need to go through a series of tedious preprocessing processes such as data cleaning and format conversion. In the past, most of these tasks relied on manual execution, which was not only inefficient but also prone to errors, greatly limiting the speed and quality of work. Although there are already several data conversion tools on the market, they often have relatively single functions and lack sufficient flexibility, making it difficult to fully meet the complex and changing data processing needs, which has brought many inconveniences to practical applications. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a data format conversion method, system, device and storage medium to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a data format conversion method, comprising: Use third-party libraries to read and store content data in the target document; Get database connection information and field mapping rules; splicing the content data into structured statements recognizable by the database based on field mapping rules; Establishing a communication connection with the database using the database connection information, and storing the structured statement in the database; The database is accessed to query the structured statement to verify the storage validity of the structured statement.

[0005] In an optional implementation, using a third-party library to read and store content data in a target document includes: Acquire the format of the target document, which includes Excel and Word; A third-party library corresponding to the format is called to read the table data of the target document.

[0006] In an optional implementation, calling a third-party library corresponding to the format to read the table data of the target document includes: If the target document is in Excel format, the read_excel function in the pandas library is used to read the worksheet data in the Excel document and the worksheet data in a specific area; If the target document is in Word format, the python-docx library is used to read the table data in the Word document and convert the table data into a pandas DataFrame format.

[0007] In an optional implementation, obtaining database connection information and field mapping rules includes: Perform syntax check and type validation on user configuration files; Confirm that the user configuration file has passed the verification, and parse the database connection information, table structure information, and field mapping rules in the user configuration file; The database connection information, table structure information, and field mapping rules are stored in a data structure in memory.

[0008] In an optional implementation, accessing the database and querying the structured statement to verify the storage validity of the structured statement includes: Accessing the database and executing the structured statement; The feedback information of the database is obtained. If the feedback information indicates that the database connection fails or the structured statement has a grammatical error, the structured statement is determined to be invalid.

[0009] In an optional embodiment, the method further comprises: Save the documents uploaded by the user to the specified path; A plurality of documents are read from the specified path, and format conversion processing is performed on the plurality of documents.

[0010] In an optional embodiment, the method further comprises: The process data of format conversion of the target document and the verification results are written into the log.

[0011] In a second aspect, the present invention provides a data format conversion system, comprising: A data reading module is used to read and store content data in a target document using a third-party library; Configuration acquisition module, used to obtain database connection information and field mapping rules; A data conversion module, used for splicing the content data into structured statements recognizable by the database based on field mapping rules; A data storage module, used to establish a communication connection with the database using the database connection information, and store the structured statement in the database; The storage verification module is used to access the database and query the structured statement to verify the storage validity of the structured statement.

[0012] In a third aspect, a device is provided, comprising: A memory, used for storing a data format conversion program; A processor is used to implement the steps of the data format conversion method provided in the first aspect when executing the data format conversion program.

[0013] In a fourth aspect, a computer-readable storage medium is provided, on which a data format conversion program is stored. When the data format conversion program is executed by a processor, the steps of the data format conversion method provided in the first aspect are implemented.

[0014] The beneficial effect of the present invention is that the data format conversion method, system, device and storage medium provided by the present invention realize automatic conversion of document formats through automatic reading of document formats and parsing of custom rules, and have high flexibility and accuracy.

[0015] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0018] Figure 2 It is a schematic flow chart of a document parsing process of a method according to an embodiment of the present invention.

[0019] Figure 3 is a schematic block diagram of a system according to an embodiment of the present invention.

[0020] Figure 4 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0023] The key terms appearing in the present invention are explained below.

[0024] Python is a widely used high-level programming language, known for its concise and readable syntax, powerful library support, and efficient interpreted execution features. Python's design philosophy emphasizes code readability and concise syntax (especially the use of space indentation to divide code blocks instead of braces or keywords), which makes Python the preferred language for many programmers and beginners.

[0025] Features of Python: Easy to learn: Python has concise and clear syntax, and its syntax design helps programmers write readable and maintainable code.

[0026] Wide range of application areas: Python can be used in many fields such as Web development, data analysis, artificial intelligence, machine learning, automated operation and maintenance, game development, etc.

[0027] Powerful standard library and third-party libraries: Python has a huge standard library and third-party libraries, covering almost all common programming tasks, such as network requests, file processing, database operations, image processing, etc.

[0028] Cross-platform compatibility: Python can run on multiple operating systems such as Windows, Linux, macOS, etc., and has good cross-platform compatibility.

[0029] Interpreted language: Python is an interpreted language, which means that it does not need to convert the code into machine code at the compilation stage, but is interpreted line by line by the interpreter at runtime. This helps in rapid development and debugging of code.

[0030] Dynamic typing: Python is a dynamically typed language, and the type of a variable is determined at runtime, which provides greater flexibility but also requires programmers to pay more attention to type safety issues when writing code.

[0031] Object-oriented: Python supports object-oriented programming (OOP), allowing programmers to define classes, objects, and methods, as well as implement OOP features such as inheritance and polymorphism.

[0032] Pandas, openpyxl and python-docx are all very popular libraries in Python, and they each play an important role in different fields. The following is a detailed introduction to these three libraries: Pandas: Overview: Pandas is a Python-based data analysis library that provides a wealth of data processing and analysis capabilities. It was originally developed by AQR Capital Management in 2008 and open sourced at the end of 2009. Pandas is designed for data analysis and processing, and it provides two main data structures: Series (a one-dimensional labeled array) and DataFrame (a two-dimensional labeled tabular data structure).

[0033] Key features: Data cleaning: processing missing values, duplicate values, etc.

[0034] Data transformation: reshape, merge, aggregate data.

[0035] Data visualization: Combine with libraries such as Matplotlib to generate charts.

[0036] Time series analysis: Process time series data and perform date operations.

[0037] The Pandas library is one of the important factors that makes Python a powerful and efficient data analysis environment.

[0038] Openpyxl: Overview: Openpyxl is a Python library for reading and writing Excel files (.xlsx). It supports the file formats of Excel 2010 and higher, and provides a convenient way to operate Excel files.

[0039] Key features: Read and write Excel files: You can read existing Excel files, get data, modify data, and save it to a new file.

[0040] Operation cells: Allows reading and writing of data by row, column or specific cell.

[0041] Create and modify worksheets: You can create new worksheets, copy and delete existing worksheets, set worksheet properties, etc.

[0042] Style settings: You can set the cell's font, color, border and other styles.

[0043] Charts and formulas: You can create charts, add formulas, etc. through Openpyxl.

[0044] Support for number and date formats: Ability to correctly handle number and date formats, ensuring that the correct format is displayed in Excel.

[0045] Python-docx: Overview: Python-docx is a Python library for creating, modifying, and manipulating Word documents. It provides rich functions for creating and formatting documents, including adding titles, paragraphs, tables, images, etc.

[0046] Key features: Create and save documents: You can create a new Word document, add titles, paragraphs, and other content, and then save it as a .docx file.

[0047] Format text: You can set the text's font, size, color, bold, italic and other styles.

[0048] Table operations: You can add tables to documents, set table styles, fill in table data, etc.

[0049] Image insertion: You can insert pictures into the document and set the position and size of the pictures.

[0050] In short, Pandas, openpyxl, and python-docx are all very useful libraries in Python, and they play an important role in different fields. Pandas focuses on data analysis and processing, openpyxl focuses on reading, writing, and manipulating Excel files, and python-docx focuses on creating, modifying, and manipulating Word documents.

[0051] The data format conversion method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the data format conversion system runs in the computer device.

[0052] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a data format conversion system. According to different requirements, the order of the steps in the flow chart may be changed, and some may be omitted.

[0053] like Figure 1 As shown, the method includes: S1. Use a third-party library to read and store content data in the target document.

[0054] Use third-party libraries in the Python programming language (such as pandas for reading Excel or CSV files, openpyxl for reading and writing Excel files, python-docx for processing Word documents, etc.) to read target documents (such as files in Excel, Word, CSV, etc.). These libraries can parse the structure of the document, extract the content data, and store it in memory in a program-processable form (such as a list, dictionary, or Pandas' DataFrame, etc.) to prepare for subsequent processing and conversion.

[0055] S2. Obtain database connection information and field mapping rules.

[0056] After reading and storing the content data of the target document, you need to obtain the database connection information (such as database type, server address, port number, database name, user name and password, etc.) and field mapping rules from sources such as configuration files, environment variables or user input. Field mapping rules define the correspondence between fields in the target document and fields in the database table, which is the key to converting content data into structured statements that can be recognized by the database.

[0057] S3. Based on field mapping rules, the content data is spliced ​​into structured statements that can be recognized by the database.

[0058] According to the acquired field mapping rules, the content data stored in the memory (such as data read from Excel files) is converted into structured statements recognizable by the database (such as SQL INSERT statements). This step may involve operations such as data cleaning, conversion, and formatting to ensure that the data conforms to the field type and constraints of the database table. By concatenating the field name, field value, and necessary SQL syntax, a structured statement that can be inserted into the database is generated.

[0059] S4. Establishing a communication connection with the database using the database connection information, and storing the structured statement in the database.

[0060] After the structured statement is generated, a communication connection is established with the database using the database connection information (such as a database connection string). Then, the structured statement (such as an INSERT statement) is sent to the database by executing an SQL statement (such as using Python's sqlite3, pymysql, psycopg2, and other database driver libraries) so that the data is stored in the corresponding table. This step may involve error handling and transaction management to ensure data consistency and integrity.

[0061] S5. Access the database and query the structured statement to verify the storage validity of the structured statement.

[0062] In order to verify whether the structured statement has been successfully stored in the database and the data is correct, you need to access the database and execute a query statement (such as a SELECT statement). By querying the records corresponding to the data just inserted, you can check the accuracy, completeness, and consistency of the data. If any problems or inconsistencies are found, error handling and data repair need to be carried out in a timely manner. This step is an important part of ensuring data quality and data integrity.

[0063] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0064] S101. Obtain the format of the target document, which includes Excel and Word.

[0065] Read the target document's file name and identify the format based on the file name's suffix.

[0066] S102. Retrieve a third-party library corresponding to the format and read the table data of the target document.

[0067] Depending on the format of the target document, the corresponding third-party library will be selected to read the table data in the document.

[0068] If the target document is in Excel format: In this case, the read_excel function in the pandas library will be used to read the worksheet data in the Excel document and the worksheet data in a specific area. Pandas is a powerful data processing and analysis library that provides the read_excel function to easily read Excel files. With this function, you can specify parameters such as the worksheet name, row and column index range to be read to obtain the data portion of interest. The read data will be stored in a pandas DataFrame object, which is a two-dimensional labeled data structure that is very suitable for subsequent data processing and analysis.

[0069] If the target document is in Word format: If the target document is in Word format, the python-docx library will be used to read the table data in the document. python-docx is a Python library for creating and modifying Microsoft Word (.docx) files. With it, you can traverse the tables in the document and extract the row and column data in the table. In order to be compatible with the data processing functions of the pandas library, the data extracted from the Word table will be converted to the pandas DataFrame format. This usually involves converting the tabular data into a Python native data structure such as a list or dictionary, and then converting it into a DataFrame object using the pandas DataFrame constructor.

[0070] In another embodiment of the present invention, batch processing of documents, the specific method is as follows Figure 2 As shown, including: 1. Read the file to be processed First, you need to write a function or script to read all the files to be processed in the specified directory or path. This can be achieved by traversing the directory and reading the file name and path.

[0071] 2. Determine whether the file is a compressed package Next, for each read file, you need to determine whether it is a compressed package. Common compressed package formats include ZIP, RAR, TAR, etc. In Python, you can use the zipfile module to process compressed packages in ZIP format, use rarfile (which needs to be installed separately) to process compressed packages in RAR format, and use the tarfile module to process compressed packages in TAR format.

[0072] A simple way to determine whether a file is a compressed package is to try to open the file using these modules. If it is successfully opened, it means that the file is a compressed package; if it fails (an exception is thrown), it means that the file is not a compressed package.

[0073] 3. Batch read files in compressed packages If the file is identified as a compressed file, the files in the compressed file need to be further processed. This usually involves decompressing the compressed file (it can be decompressed in memory or to a temporary directory on disk), then traversing the decompressed files and performing subsequent processing on each file (such as reading table data).

[0074] When processing files in a compressed package, you need to pay attention to the file type and format. If the compressed package contains multiple types of files (such as Excel, Word, PDF, etc.), you may need to write different processing logic to process these files separately.

[0075] 4. Continue processing a single document If the file is not a compressed file but a single document (such as an Excel or Word file), you can directly perform subsequent processing on the document. This includes using appropriate third-party libraries (such as pandas and python-docx) to read the table data in the document and convert it into a data structure suitable for subsequent processing (such as pandas DataFrame).

[0076] Process integrated into the original solution: Integrating the above batch file recognition steps into the referenced solution (S101 and S102), the following complete processing flow can be obtained: Read the list of files to be processed.

[0077] Judge each file: If it is a compressed package, decompress it and traverse the files in it, and perform subsequent processing on each file (such as reading table data).

[0078] If it is a single document, subsequent processing is performed directly (such as using pandas and python-docx to read table data).

[0079] Depending on the format of the file (Excel or Word), choose the appropriate third-party library and function to read the table data.

[0080] The read data is stored in a pandas DataFrame object for subsequent data processing and analysis.

[0081] This processing flow ensures that no matter the input is a single document or multiple files in a compressed package, it can be processed and read efficiently.

[0082] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0083] S201. Syntax checking and type validation First, the user configuration file needs to be syntax checked to ensure that it conforms to the expected format and structure. This may include verifying that the file uses the correct encoding, contains the necessary fields, uses the correct delimiters between fields, etc.

[0084] Next, perform type verification. You need to check whether each field in the configuration file contains data of the expected type. For example, the username and password in the database connection information should be string types, the port number should be integer type, the field name in the table structure information should be a string list, and the field mapping rule should be a key-value dictionary, etc.

[0085] To implement these validations, you can write a validation function that reads the contents of the configuration file and uses conditional statements and type checking functions (such as isinstance in Python) to verify the data type of each field.

[0086] S202. Confirm that the configuration file has passed verification Once the configuration file passes the syntax check and type validation, it can be confirmed that it is valid and continue with the subsequent processing steps. If the configuration file fails the validation, the error information needs to be recorded and the error prompt needs to be returned to the user so that they can make appropriate corrections.

[0087] S203. Parsing information in the configuration file After confirming that the configuration file is valid, you need to parse the key information in it, including database connection information, table structure information, and field mapping rules. This information is usually organized into a specific structure (such as JSON, YAML, or INI format), so you need to use the corresponding parsing library (such as json, PyYAML, or configparser in Python) to read and parse this information.

[0088] Database connection information: may include database type, server address, port number, database name, user name and password, etc.

[0089] Table structure information: describes the structure of the table in the database, including table name, field name, field type, etc.

[0090] Field mapping rules: define the correspondence between fields in the configuration file and fields in the database table.

[0091] S204. Stored in a data structure in memory After parsing the information in the configuration file, you need to store this information in a data structure in memory for use in subsequent processing steps. These data structures can be basic data types in Python (such as dictionaries, lists, tuples, etc.) or more complex custom data structures.

[0092] For example, you can use a dictionary to store database connection information, where the key is the name of the connection parameter (such as host, port, dbname, etc.) and the value is the corresponding parameter value. A similar dictionary structure can also be used to store table structure information and field mapping rules.

[0093] By performing syntax checking and type verification on the user configuration file, the accuracy and validity of the configuration file can be ensured. Once the configuration file passes the verification, the key information can be parsed and stored in the data structure in the memory for use in subsequent processing steps. This processing flow helps to improve the robustness and reliability of the system.

[0094] In an embodiment of the present invention, based on step S3, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0095] According to the database rules and specific configuration items, it is responsible for intelligently splicing the read data into SQL statements such as INSERT, UPDATE, and DELETE that the database can recognize. Specifically, it will meticulously convert the data structure stored in the memory into the corresponding SQL statement form based on the table structure information and field mapping rules defined in the configuration items. In the process of splicing these SQL statements, it will automatically perform necessary type conversion and field mapping operations based on the data type (such as integer, floating point number, string, etc.) and field mapping rules, thereby ensuring that the final generated SQL statement fully complies with the database syntax specifications.

[0096] In addition, it also has powerful data preprocessing capabilities. It can automatically identify and process empty values ​​(NULL values) and special characters (such as single quotes, double quotes, backslashes, etc.) in the data, and ensure that the generated SQL statements will not cause syntax errors or security issues due to these special situations through appropriate escape or replacement strategies. This function is crucial to maintaining the integrity and security of the database.

[0097] Furthermore, it can intelligently match the primary key or unique identifier field in the data. When performing UPDATE or DELETE operations, it can ensure that these operations can accurately locate the target record, thereby avoiding misoperation or data inconsistency. By accurately matching the primary key or unique identifier field, the accuracy and efficiency of data operations can be significantly improved.

[0098] In summary, it not only has the ability to convert memory data structures into SQL statements, but also has a series of advanced functions such as data preprocessing, type conversion, field mapping, and primary key matching, which can ensure that the generated SQL statements not only comply with the syntax specifications of the database, but also can complete data operation tasks efficiently and accurately.

[0099] In an embodiment of the present invention, based on step S4, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0100] The connection and communication with the database is realized by calling Python database connection library (such as pymysql, cx_Oracle, etc.). The content data of the target document is imported into the database through the spliced ​​SQL statements.

[0101] In an embodiment of the present invention, based on step S5, a possible embodiment is given below to illustrate its specific implementation in a non-limiting manner.

[0102] In order to fully verify the storage effect of structured statements, a series of rigorous measures will be taken. First, the database will be accessed, which is the first step in the verification process. Through the database connection information, a stable and secure communication connection can be established with the database system.

[0103] Next, to verify the accuracy of the data, a SELECT query is executed. This query is designed based on the previously stored structured statements and is intended to retrieve rows from the database that match these statements. The writing of the SELECT query requires a high degree of precision because it will directly affect the accuracy of the verification results.

[0104] After executing a SELECT query, the database system returns a set of results. These results contain the rows in the table that match the structured statement. Next, these results need to be carefully analyzed and compared. By comparing the query results with the expected data one by one, you can verify whether the structured statement is correctly stored in the database.

[0105] In addition, it will check whether the database correctly recognizes and handles the data. This includes the type and format of the data and any possible constraints. For example, if the structured statement contains data of type date, it will verify whether the database stores this data in the correct date format. Similarly, if the data contains specific constraints (such as unique constraints), it will verify whether the database correctly enforces these constraints.

[0106] During the verification process, if any mismatch or anomaly is found, error troubleshooting and repair will be carried out immediately. This may involve checking data input, field mapping rules, and database connection information. Through continuous debugging and optimization, the accuracy and integrity of the data can be ensured, thereby improving the efficiency and quality of the entire data processing and storage process.

[0107] In short, verifying the storage validity of structured statements is an indispensable part of the data processing and storage process. By executing SELECT queries, analyzing query results, and checking data constraints, the accuracy and integrity of the data can be ensured, providing a solid foundation for subsequent data analysis and decision-making.

[0108] In addition, in order to further improve the efficiency and accuracy of data processing, the technical solution of the present invention also includes the following optimization measures: (1) Batch processing: The toolkit supports batch processing of multiple documents. It can read data from multiple Excel, Word and other documents at one time and generate corresponding SQL statements. This can not only improve the efficiency of data processing, but also reduce the errors caused by multiple operations.

[0109] (2) Log Recording: In order to facilitate users to track the progress and results of data processing, the toolkit of the present invention uses Python's logging library to implement the log recording function during the data processing process and generate detailed log records, including the time of reading data, the time of splicing SQL statements, the time of executing SQL statements, etc. These log records can help users better understand the progress and results of data processing and discover and solve problems in a timely manner.

[0110] (3) Customized rules: In order to meet the user's processing requirements for different data formats and database rules, the toolkit of the present invention allows the user to customize field mapping rules and type conversion rules. Users can flexibly configure the field mapping method and data type conversion method according to their own needs. This can not only improve the flexibility of data processing, but also meet the user's processing requirements for different data formats and database rules.

[0111] (4) Visual interface: To facilitate user use, the toolkit can also provide a visual interface. Users can configure database rules, select documents to be processed, view log records, etc. through a graphical interface. This can not only reduce the user's learning cost, but also improve usage efficiency. At the same time, the visual interface can also provide a more intuitive and friendly user experience.

[0112] In some embodiments, the data format conversion system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the data format conversion system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Function of data format conversion.

[0113] In this embodiment, the data format conversion system can be divided into multiple functional modules according to the functions it performs, such as Figure 3As shown. The functional modules of the system 300 may include: a data reading module 310, a configuration acquisition module 320, a data conversion module 330, a data storage module 340 and a storage verification module 350. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0114] A data reading module is used to read and store content data in a target document using a third-party library; Configuration acquisition module, used to obtain database connection information and field mapping rules; A data conversion module, used for splicing the content data into structured statements recognizable by the database based on field mapping rules; A data storage module, used to establish a communication connection with the database using the database connection information, and store the structured statement in the database; The storage verification module is used to access the database and query the structured statement to verify the storage validity of the structured statement.

[0115] Optionally, as an embodiment of the present invention, using a third-party library to read and store content data in a target document includes: Acquire the format of the target document, which includes Excel and Word; A third-party library corresponding to the format is called to read the table data of the target document.

[0116] Optionally, as an embodiment of the present invention, calling a third-party library corresponding to the format to read the table data of the target document includes: If the target document is in Excel format, the read_excel function in the pandas library is used to read the worksheet data in the Excel document and the worksheet data in a specific area; If the target document is in Word format, the python-docx library is used to read the table data in the Word document and convert the table data into a pandas DataFrame format.

[0117] Optionally, as an embodiment of the present invention, obtaining database connection information and field mapping rules includes: Perform syntax check and type validation on user configuration files; Confirm that the user configuration file has passed the verification, and parse the database connection information, table structure information, and field mapping rules in the user configuration file; The database connection information, table structure information, and field mapping rules are stored in a data structure in memory.

[0118] Optionally, as an embodiment of the present invention, accessing the database and querying the structured statement to verify the storage validity of the structured statement includes: Accessing the database and executing the structured statement; The feedback information of the database is obtained. If the feedback information indicates that the database connection fails or the structured statement has a grammatical error, the structured statement is determined to be invalid.

[0119] Optionally, as an embodiment of the present invention, it further includes: Save the documents uploaded by the user to the specified path; A plurality of documents are read from the specified path, and format conversion processing is performed on the plurality of documents.

[0120] Optionally, as an embodiment of the present invention, it further includes: The process data of format conversion of the target document and the verification results are written into the log.

[0121] Figure 4 The data format conversion method provided for the embodiment of the present application can be applied to equipment. It will be appreciated by those skilled in the art that the equipment structure involved in the embodiment of the present invention does not constitute a limitation on the equipment, and the equipment may include more or less components than shown, or combine certain components, or different component arrangements. In an embodiment of the present invention, the equipment includes but is not limited to laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The equipment may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the embodiments of the present application described herein and / or required.

[0122] The device 400 may include: a processor 410, a memory 420 and a communication unit 430. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0123] The memory 420 may be used to store the execution instructions of the processor 410, and the memory 420 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 420 are executed by the processor 410, the device 400 is enabled to perform some or all of the steps in the following method embodiments.

[0124] The processor 410 is the control center of the storage device, and uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 420, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 410 can include only a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0125] The communication unit 430 is used to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0126] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0127] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0128] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0129] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0130] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0132] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A data format conversion method, characterized in that: include: Use third-party libraries to read and store content data in the target document; Get database connection information and field mapping rules; splicing the content data into structured statements recognizable by the database based on field mapping rules; Establishing a communication connection with the database using the database connection information, and storing the structured statement in the database; The database is accessed to query the structured statement to verify the storage validity of the structured statement.

2. The method according to claim 1, characterized in that Use third-party libraries to read and store content data in the target document, including: Acquire the format of the target document, which includes Excel and Word; A third-party library corresponding to the format is called to read the table data of the target document.

3. The method according to claim 2, characterized in that Retrieve the third-party library corresponding to the format and read the table data of the target document, including: If the target document is in Excel format, the read_excel function in the pandas library is used to read the worksheet data in the Excel document and the worksheet data in a specific area; If the target document is in Word format, the python-docx library is used to read the table data in the Word document and convert the table data into a pandas DataFrame format.

4. The method according to claim 1, characterized in that: Get database connection information and field mapping rules, including: Perform syntax check and type validation on user configuration files; Confirm that the user configuration file has passed the verification, and parse the database connection information, table structure information, and field mapping rules in the user configuration file; The database connection information, table structure information, and field mapping rules are stored in a data structure in memory.

5. The method according to claim 1, characterized in that Accessing the database and querying the structured statement to verify the storage validity of the structured statement includes: Accessing the database and executing the structured statement; The feedback information of the database is obtained. If the feedback information indicates that the database connection fails or the structured statement has a grammatical error, the structured statement is determined to be invalid.

6. The method according to claim 1, characterized in that The method further comprises: Save the documents uploaded by the user to the specified path; A plurality of documents are read from the specified path, and format conversion processing is performed on the plurality of documents.

7. The method according to claim 1, characterized in that The method further comprises: The process data of format conversion of the target document and the verification results are written into the log.

8. A data format conversion system, characterized in that: include: A data reading module is used to read and store content data in a target document using a third-party library; Configuration acquisition module, used to obtain database connection information and field mapping rules; A data conversion module, used for splicing the content data into structured statements recognizable by the database based on field mapping rules; A data storage module, used to establish a communication connection with the database using the database connection information, and store the structured statement in the database; The storage verification module is used to access the database and query the structured statement to verify the storage validity of the structured statement.

9. A device, characterized in that: include: A memory, used for storing a data format conversion program; A processor, used to implement the steps of the data format conversion method as described in any one of claims 1 to 7 when executing the data format conversion program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a data format conversion program, and when the data format conversion program is executed by the processor, the steps of the data format conversion method according to any one of claims 1 to 7 are implemented.

Images