Method and device for importing file data into database, storage medium and equipment

Abstract

The invention discloses a method and device for importing file data into a database, a storage medium and equipment, relates to the technical field of databases, is suitable for the technical field of finance, and mainly aims at solving the problems of memory overflow, poor program reusability and low derivative flexibility when existing data files are imported into the database in a financial scene. Comprising the steps that an import file configuration table containing multiple derivative tasks is obtained, and the derivative tasks contain multiple universal fields; task name information and derivative state information corresponding to the task name field and the derivative state field are obtained from the universal field, and a target derivative task is determined based on the task name information and the derivative state information; obtaining a file path corresponding to the target derivative task from the imported file configuration table, and obtaining a to-be-imported file through the SFTP server based on the file path; analyzing the to-be-imported file to obtain a derivative script; and executing the derivative script to complete database import of the to-be-imported file.

Description

Technical Field

[0001] The present invention relates to the field of database technology, applicable to the field of financial technology, and in particular to a method and device, storage medium, and equipment for importing file data into a database. Background Art

[0002] In the process of developing new insurance products, it is necessary to calculate the reserve for the new products. In the process of calculating the reserve, it is necessary to actuate the factor data calculated in advance. Therefore, the factor data needs to be imported into the database before the reserve calculation can be performed. However, in some insurance scenarios, the reserve factor data of new products can reach tens of millions or even hundreds of millions, and the data files can reach several GB.

[0003] The existing method of importing data files into a database requires uploading the file to the application system first, and then using an import program developed based on a specific business scenario to perform the database import operation. However, for data files with large data volumes, the system often reports memory overflow errors. In addition, as business scenarios change, the import program needs to be modified accordingly, which makes the import program less reusable and reduces the flexibility of file data import. Summary of the invention

[0004] In view of this, the present invention provides a method and device, storage medium, and equipment for importing file data into a database, the main purpose of which is to solve the problems of memory overflow, poor program reusability, and low derivative flexibility that occur when importing existing data files into a database.

[0005] According to one aspect of the present invention, a method for importing file data into a database is provided, comprising:

[0006] Obtain an import file configuration table, wherein the import file configuration table includes multiple derivative tasks, and the derivative tasks include multiple common fields;

[0007] Acquire task name information and derivative status information corresponding to the task name field and the derivative status field from the general field, and determine the target derivative task based on the task name information and the derivative status information;

[0008] Obtaining a file path corresponding to the target derivative task from the import file configuration table, and obtaining a file to be imported through an SFTP server based on the file path;

[0009] The file to be imported is parsed to obtain a derivative script; and the derivative script is executed to complete the database import of the file to be imported.

[0010] Furthermore, before obtaining the import file configuration table, the method further includes:

[0011] Setting a task name of the derivative task according to the business scope, wherein the task name corresponds to at least one derivative task;

[0012] The task name, table name, file path, derivative status, and number of file lines are set as the general fields to create the import file configuration table.

[0013] Further, the determining the target derivative task based on the task name information and the derivative state information includes:

[0014] receiving a file data import application from a user, the file data import application carrying business scope information; and determining a target task name from the task name information based on the business scope information;

[0015] Determine at least one derivative task to be analyzed from the import file configuration table based on the target task name;

[0016] The target derivative task is determined based on the derivative state information corresponding to the derivative task to be analyzed.

[0017] Furthermore, the step of parsing the file to be imported to obtain a derivative script includes:

[0018] Parse the file to be imported, and obtain the content of the title row and the content of the data rows other than the title row of the file to be imported;

[0019] Determine the content of the header row as a table field name, and determine the content of the data row as a table field value corresponding to the table field name;

[0020] The derivative script is generated based on the table field name and the table field value.

[0021] Furthermore, the general fields in the import file configuration table include a field for deleting historical data and a field for deleting historical data script;

[0022] Before parsing the file to be imported to obtain the derivative script, the method further includes:

[0023] Obtaining a keyword in the field of whether to delete historical data, and determining whether to delete the historical data based on the keyword;

[0024] If the historical data is to be deleted, a personalized deletion script corresponding to the script field for deleting historical data is obtained; and the personalized deletion script is executed to delete the historical data.

[0025] Furthermore, the general fields in the import file configuration table also include a statistical result data field, a statistical result data script field, and an import database row number field;

[0026] After executing the derivative script to complete the database import of the file to be imported, the method further includes:

[0027] Obtaining a keyword in the statistical result data field, and determining whether to perform data statistical processing on the table after the imported data is completed based on the keyword;

[0028] If data statistics processing is performed on the table after the data import is completed, the personalized statistical script corresponding to the statistical result data script field is obtained; and the personalized statistical script is executed to obtain the row number statistical result of the table after the data import is completed;

[0029] The import database row number field is updated based on the row number statistics result.

[0030] Furthermore, after updating the import database row number field based on the row number statistics result, the method further includes:

[0031] Obtain the first row value corresponding to the number of file rows in the general field; and obtain the second row value corresponding to the updated number of imported database rows field;

[0032] Compare the first row value with the second row value to obtain a row comparison result;

[0033] The row number comparison result is fed back to the relevant personnel of the target derivative task.

[0034] According to another aspect of the present invention, there is provided a device for importing file data into a database, comprising:

[0035] A configuration table acquisition module is used to acquire an import file configuration table, wherein the import file configuration table includes multiple derivative tasks, and the derivative tasks include multiple common fields;

[0036] A task determination module, used to obtain task name information and derivative status information corresponding to the task name field and the derivative status field from the general field, and determine the target derivative task based on the task name information and the derivative status information;

[0037] A file acquisition module, used to acquire a file path corresponding to the target derivative task from the import file configuration table, and acquire a file to be imported through an SFTP server based on the file path;

[0038] The derivative module is used to parse the file to be imported to obtain a derivative script; and execute the derivative script to complete the database import of the file to be imported.

[0039] Furthermore, the device also includes a configuration table creation module, which is used to:

[0040] Setting a task name of the derivative task according to the business scope, wherein the task name corresponds to at least one derivative task;

[0041] The task name, table name, file path, derivative status, and number of file lines are set as the general fields to create the import file configuration table.

[0042] Furthermore, the task determination module is also used to:

[0043] receiving a file data import application from a user, the file data import application carrying business scope information; and determining a target task name from the task name information based on the business scope information;

[0044] Determine at least one derivative task to be analyzed from the import file configuration table based on the target task name;

[0045] The target derivative task is determined based on the derivative state information corresponding to the derivative task to be analyzed.

[0046] Furthermore, the derivative module is also used for:

[0047] Parse the file to be imported, and obtain the content of the title row and the content of the data rows other than the title row of the file to be imported;

[0048] Determine the content of the header row as a table field name, and determine the content of the data row as a table field value corresponding to the table field name;

[0049] The derivative script is generated based on the table field name and the table field value.

[0050] Furthermore, the general fields in the import file configuration table include a field for deleting historical data and a field for deleting historical data script; the device also includes a data deletion module for:

[0051] Obtaining a keyword in the field of whether to delete historical data, and determining whether to delete the historical data based on the keyword;

[0052] If the historical data is to be deleted, a personalized deletion script corresponding to the script field for deleting historical data is obtained; and the personalized deletion script is executed to delete the historical data.

[0053] Furthermore, the general fields in the import file configuration table further include a statistical result data field, a statistical result data script field, and an import database row number field; the device further includes a statistical module for:

[0054] Obtaining a keyword in the statistical result data field, and determining whether to perform data statistical processing on the table after the imported data is completed based on the keyword;

[0055] If data statistics processing is performed on the table after the data import is completed, the personalized statistical script corresponding to the statistical result data script field is obtained; and the personalized statistical script is executed to obtain the row number statistical result of the table after the data import is completed;

[0056] The import database row number field is updated based on the row number statistics result.

[0057] Furthermore, the device also includes a comparison module, which is used to:

[0058] Obtain the first row value corresponding to the number of file rows in the general field; and obtain the second row value corresponding to the updated number of imported database rows field;

[0059] Compare the first row value with the second row value to obtain a row comparison result;

[0060] The row number comparison result is fed back to the relevant personnel of the target derivative task.

[0061] According to another aspect of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned method of importing file data into a database.

[0062] According to another aspect of the present invention, there is provided a device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0063] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the method for importing file data into a database as described above.

[0064] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0065] The present invention provides a method, device, storage medium and equipment for importing file data into a database. Compared with the prior art, the present invention configures the file data import requirements by importing a file configuration table, which can improve the flexibility of importing file data into a database, avoid the development of a corresponding import program in the prior art, improve the efficiency of importing file data into the data derivative, and save the time of the derivative. The present invention also sets a task name field in the import file configuration table, so that the method of importing file data into a database of the present invention is applicable to a variety of business scenarios, which greatly increases the scope of use of the method of the present invention. The present invention also obtains the file to be imported through an SFTP server, which can support the import of large files in the financial field, is safe and stable, and there is no need to worry about the problem of server crash and memory overflow due to the increase in file size.

[0066] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0068] Figure 1 A schematic flow chart of a method for importing file data into a database provided by an embodiment of the present invention is shown;

[0069] Figure 2 A schematic flow chart of another method for importing file data into a database provided by an embodiment of the present invention is shown;

[0070] Figure 3 A schematic flow chart of another method for importing file data into a database provided by an embodiment of the present invention is shown;

[0071] Figure 4 A schematic flow chart of another method for importing file data into a database provided by an embodiment of the present invention is shown;

[0072] Figure 5 A schematic diagram showing the structure of a device for importing file data into a database provided by an embodiment of the present invention;

[0073] Figure 6 A schematic diagram showing the structure of another device for importing file data into a database provided by an embodiment of the present invention;

[0074] Figure 7 A schematic structural diagram of a device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0075] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0076] The embodiment of the present invention provides a method for importing file data into a database, such as Figure 1 As shown, the method includes:

[0077] 101. Obtain an import file configuration table, wherein the import file configuration table includes a plurality of derivative tasks, and the derivative tasks include a plurality of general fields;

[0078] In an embodiment of the present invention, the current execution end obtains an import file configuration table, which contains multiple derivative tasks, and each derivative task contains multiple general fields. Among them, the import file configuration table is a general configuration table pre-set according to the derivative needs. Among them, the derivative task represents the task corresponding to each business scope that needs to import the data in the business-related files into the database. For example, within the business scope of the insurance new product development scenario in the financial field, the factor data in the relevant files needs to be imported into the database; in addition, the data in the customer-related files within the insurance business scope also needs to be imported into the database, and the data in the product transaction-related files also needs to be imported into the database, etc., which are not specifically limited in the embodiment of the present invention. Among them, the general fields can be set according to the derivative requirements, and can include the task name field of each derivative task, the table name field of the specific data table that needs to import the relevant files into the database, the file path field for obtaining the relevant files, the derivative status field for state control of the derivative task, the file line number field in the relevant files, etc., which are not specifically limited in the embodiment of the present invention.

[0079] 102. Acquire task name information and derivative status information corresponding to the task name field and the derivative status field from the general field, and determine a target derivative task based on the task name information and the derivative status information;

[0080] In an embodiment of the present invention, the current execution end obtains the task name information and derivative status information corresponding to the task name field and the derivative status field from the general field. Among them, the task name can be named according to the financial product, such as setting the task name information to "insurance product 1", "insurance product 2", "financial product 1", "financial product 2", "structured savings product 1", "structured savings product 2", etc., which is not specifically limited in the embodiment of the present invention. Among them, the derivative status information is used to control the process of file import, including the pending import state, the derivative success state and the derivative failure state. In the initial stage, the derivative status information can be set to the pending import state. When it is detected that all the data in the relevant files are imported into the database, the derivative status needs to be updated to the derivative success state; if the database import of the relevant files is not completed during the derivative process, the derivative status is updated to the derivative failure state, which is not specifically limited in the embodiment of the present invention. In an embodiment of the present invention, the current execution end also determines the target derivative task based on the task name information and derivative status information obtained above, that is, determines the derivative task to be executed currently.

[0081] 103. Obtain a file path corresponding to the target derivative task from the import file configuration table, and obtain a file to be imported through an SFTP server based on the file path;

[0082] In an embodiment of the present invention, the current execution end obtains the file path corresponding to the target derivative task from the import file configuration table. Wherein, the file path is used to indicate the specific storage location of the relevant file. In an embodiment of the present invention, the file path is a file acquisition path for extracting the file to be imported from the disk of the machine equipped with the SFTP server. The current execution end obtains the file to be imported through the SFTP server installed on the machine based on the file path. It should be noted that the SFTP server is a server implementation of the Secure File Transfer Protocol. SFTP is a part of SSH (SecureShell), which uses the SSH2 protocol to provide security for file transfer. The SFTP server ensures security during the file transfer process by encrypting data transmission. In addition, since the SFTP server needs to split the large file first when transmitting a large file to obtain small file blocks, and then reassemble the small file blocks on the server side. Therefore, using the disk of the machine equipped with the SFTP server to store the file to be imported can effectively avoid the problem of system memory overflow.

[0083] 104. Parse the file to be imported to obtain a derivative script; and execute the derivative script to complete the database import of the file to be imported.

[0084] In the embodiment of the present invention, the current execution end parses the file to be imported determined in step 103, obtains the title and data in the file to be imported, and maps the title and data to the target data table of the database to obtain a derivative script. After obtaining the derivative script, the current execution end executes the derivative script to complete the data import of the file to be imported.

[0085] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to make the method of the present invention applicable to a wider range of scenarios, avoid repeated modifications to the data import program, and save database import time, another method for importing file data into a database is provided, such as Figure 2 As shown, before the step of obtaining the import file configuration table, the method further includes:

[0086] 201. Setting a task name of a derivative task according to a business scope, wherein the task name corresponds to at least one derivative task;

[0087] In an embodiment of the present invention, the current execution end sets the task name of the derivative task according to the business scope. For example, it can be named according to the financial product, and the task name information is set to "insurance product 1", "insurance product 2", "wealth management product 1", "wealth management product 2", "structured savings product 1", "structured savings product 2", etc., which is not specifically limited in the embodiment of the present invention. It can also be named according to the customer group served by the financial product, such as setting the task name information to "student accident insurance", "aviation accident insurance", etc., which is not specifically limited in the embodiment of the present invention. It should be noted that the above task name corresponds to at least one derivative task. In the case of complex business, one task name can have multiple derivative tasks.

[0088] 202. Set the task name, table name, file path, derivative status, and file line number as the general fields to create the import file configuration table.

[0089] In an embodiment of the present invention, the current execution end sets the task name of the derivative task, the table name of the specific data table in the file import database, the file path for obtaining the file, the derivative status of the derivative program for controlling the derivative, the number of file lines of the file to be imported, etc. as the general fields to create an import file configuration table, and the embodiment of the present invention does not make specific limitations.

[0090] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to clarify the tasks to be executed from multiple derivative tasks, another method for importing file data into a database is provided, such as Figure 3 As shown, the step of determining the target derivative task based on the task name information and the derivative state information includes:

[0091] 301. Receive a file data import application from a user, the file data import application carrying business scope information; and determine a target task name from the task name information based on the business scope information;

[0092] In an embodiment of the present invention, the current execution end accepts a user's file data import application, and the file data import application carries business scope information. The business scope information may be task name information, or may be financial product name information when determining the task name information, or financial product service customer group information, etc., which is not specifically limited in the embodiment of the present invention. After obtaining the above-mentioned business scope information, the current execution end determines the target task name, i.e., the task name to be executed, from multiple task name information based on the obtained business scope information.

[0093] 302. Determine at least one derivative task to be analyzed from the import file configuration table based on the target task name;

[0094] In an embodiment of the present invention, the current execution end determines at least one derivative task to be analyzed from the import file configuration table based on the target task name, that is, the derivative task matching the target task name in the import file configuration table is determined as the derivative task to be analyzed.

[0095] 303. Determine the target derivative task based on the derivative state information corresponding to the derivative task to be analyzed.

[0096] In an embodiment of the present invention, since there may be multiple derivative tasks to be analyzed determined in step 302, and there may also be completed derivative tasks, the current execution end is required to perform analysis based on the derivative status corresponding to the derivative tasks to be analyzed, and only the derivative tasks to be analyzed whose derivative status is the to-be-imported status can be determined as the target derivative tasks.

[0097] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to meet the personalized database historical data deletion requirements, another method for importing file data into a database is provided, wherein the step of parsing the file to be imported to obtain a derivative script includes:

[0098] Parse the file to be imported, and obtain the content of the title row and the content of the data rows other than the title row of the file to be imported;

[0099] Determine the content of the header row as a table field name, and determine the content of the data row as a table field value corresponding to the table field name;

[0100] The derivative script is generated based on the table field name and the table field value.

[0101] In the embodiment of the present invention, since the file to be imported is generally in the form of an electronic table, the first row of the electronic table is generally the title row of the data, which represents the meaning of each column of data. The current execution end parses the file to be imported and obtains the content of the title row of the file to be imported. And the content of the title row is determined as the table field name, that is, the derivative is performed according to the real field name of the obtained table, so that it is known which fields of the file to be imported are to be derived, and the program will read the title row as the table field of the derivative script. Then, the current execution end determines the content of the data row as the table field value corresponding to the above-mentioned table field name, that is, the table field value in the original file to be imported is determined as the import value of the corresponding table field. When the field name and the corresponding import value are determined, the corresponding derivative script is generated. Spark, as a fast, general, and scalable big data analysis engine, is a big data parallel computing framework based on memory computing, which improves the real-time performance of data processing in a big data environment, while ensuring high fault tolerance and high scalability. Therefore, in the embodiment of the present invention, the Spark language framework is preferably used to read the file and generate a derivative script.

[0102] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to meet the personalized demand for deleting historical data, another method for importing file data into a database is provided, in which the general fields in the imported file configuration table include a field for deleting historical data and a field for deleting historical data script; before the step of parsing the file to be imported to obtain a derivative script, the method further includes:

[0103] Obtaining a keyword in the field of whether to delete historical data, and determining whether to delete the historical data based on the keyword;

[0104] If the historical data is to be deleted, a personalized deletion script corresponding to the script field for deleting historical data is obtained; and the personalized deletion script is executed to delete the historical data.

[0105] In an embodiment of the present invention, the current execution end obtains a keyword in the field of whether to delete historical data. If the historical data needs to be deleted, the keyword "yes" is set in the field of whether to delete historical data. If the historical data does not need to be deleted, the keyword "no" is set in the field of whether to delete historical data. The current execution end determines whether to delete the historical data based on the keywords set above. If the keyword obtained by the current execution end is "yes", it is necessary to obtain a personalized deletion script corresponding to the script field for deleting historical data. Among them, the personalized deletion script is set according to personalized deletion requirements. After the current execution end obtains the personalized deletion script, it executes the obtained personalized deletion script to clean up the old data. If the keyword obtained by the current execution end is "no", there is no need to delete the old data, and the data can be directly appended on the basis of the original database.

[0106] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to meet personalized statistical and verification requirements, another method for importing file data into a database is provided, in which the general fields in the imported file configuration table also include a statistical result data field, a statistical result data script field, and an imported database row number field. Figure 4 As shown in the step, after the derivative script is executed to complete the database import of the file to be imported, the method further includes:

[0107] 401, obtaining a keyword in the statistical result data field, and determining whether to perform data statistical processing on the table after the imported data is completed based on the keyword;

[0108] In the embodiment of the present invention, the current execution end obtains the keyword in the field of whether to perform statistical processing on the result data. If statistical processing is required for the result data, the keyword "yes" is set in the field of whether to perform statistical processing on the result data. If statistical processing is not required for the result data, the keyword "no" is set in the field of whether to perform statistical processing on the result data. The current execution end determines whether to perform statistical processing on the result data based on the keyword set above.

[0109] 402. If data statistics processing is performed on the table after the data is imported, a personalized statistical script corresponding to the statistical result data script field is obtained; and the personalized statistical script is executed to obtain the row number statistical result of the table after the data is imported;

[0110] 403. Update the import database row number field based on the row number statistics result.

[0111] In an embodiment of the present invention, if the keyword obtained by the current execution end is "yes", it is necessary to obtain a personalized statistical script corresponding to the statistical result data script field. Among them, the personalized statistical script is a script for counting the number of rows in the data table in the database after the import is completed, and can be configured accordingly according to the statistical requirements. After the current execution end obtains the personalized statistical script, it executes the obtained personalized statistical script, and obtains the statistical result of the number of rows of the table after the import is completed through statistics by the personalized statistical script, and updates the obtained row number statistical result to the import database row number field in the import file configuration table, so that relevant personnel can check and process based on the row value of the updated import database row number field.

[0112] In order to check the imported table, after the step of updating the imported database row number field based on the row number statistics result, the method further includes:

[0113] 404. Obtain the first row value corresponding to the number of file rows in the general field; and obtain the second row value corresponding to the updated number of imported database rows field;

[0114] 405. Compare the first row value and the second row value to obtain a row comparison result;

[0115] 406. Feedback the row number comparison result to relevant personnel of the target derivative task.

[0116] In an embodiment of the present invention, the current execution end obtains the first row value corresponding to the number of rows in the file in the general field, wherein the first row value represents the number of rows of data in the file to be imported. And, the current execution end obtains the second row value corresponding to the updated imported database row number field, wherein the second row value represents the row value of the table imported into the database. The current execution end compares the first row value and the second row value obtained in step 401 to obtain a row number comparison result. The comparison result can represent the relationship between the change in the number of rows in the table in the database and the number of rows in the file to be imported. Then, the current execution end feeds back the row number comparison result obtained in step 405 to the relevant personnel of the target derivative task, and the relevant personnel perform verification processing based on the row number comparison result, and can quickly obtain the change in the data table after the derivative task is executed.

[0117] The embodiment of the present invention provides a method for importing file data into a database. Compared with the prior art, the present invention configures the file data import requirements by importing a file configuration table, which can improve the flexibility of importing file data into a database, avoid the development of a corresponding import program in the prior art, improve the efficiency of importing file data into the data derivative, and save the derivative time. The present invention also sets a task name field in the import file configuration table, so that the method for importing file data into a database of the present invention is applicable to a variety of business scenarios, which greatly increases the scope of use of the method of the present invention. The present invention also obtains the file to be imported through the SFTP server, which can support the import of large files in the financial field, is safe and stable, and there is no need to worry about the problem of server crash and memory overflow due to the increase in file size.

[0118] As the above Figure 1 The implementation of the method shown in the figure, the embodiment of the present invention provides a device for importing file data into a database, such as Figure 5 As shown, the device comprises:

[0119] A configuration table acquisition module 51 is used to acquire an import file configuration table, wherein the import file configuration table includes a plurality of derivative tasks, and the derivative tasks include a plurality of general fields;

[0120] A task determination module 52, used to obtain task name information and derivative status information corresponding to the task name field and the derivative status field from the general field, and determine the target derivative task based on the task name information and the derivative status information;

[0121] A file acquisition module 53, used to acquire a file path corresponding to the target derivative task from the import file configuration table, and acquire a file to be imported through an SFTP server based on the file path;

[0122] The derivative module 54 is used to parse the file to be imported to obtain a derivative script; and execute the derivative script to complete the database import of the file to be imported.

[0123] Furthermore, if Figure 6 As shown, the device also includes a configuration table creation module 55, which is used to:

[0124] Setting a task name of the derivative task according to the business scope, wherein the task name corresponds to at least one derivative task;

[0125] The task name, table name, file path, derivative status, and number of file lines are set as the general fields to create the import file configuration table.

[0126] Furthermore, the task determination module 52 is also used for:

[0127] receiving a file data import application from a user, the file data import application carrying business scope information; and determining a target task name from the task name information based on the business scope information;

[0128] Determine at least one derivative task to be analyzed from the import file configuration table based on the target task name;

[0129] The target derivative task is determined based on the derivative state information corresponding to the derivative task to be analyzed.

[0130] Furthermore, the derivative module 54 is also used for:

[0131] Parse the file to be imported, and obtain the content of the title row and the content of the data rows other than the title row of the file to be imported;

[0132] Determine the content of the header row as a table field name, and determine the content of the data row as a table field value corresponding to the table field name;

[0133] The derivative script is generated based on the table field name and the table field value.

[0134] Furthermore, the general fields in the import file configuration table include whether to delete historical data fields and delete historical data script fields; Figure 6 As shown, the device further includes a data deletion module 56, which is used to:

[0135] Obtaining a keyword in the field of whether to delete historical data, and determining whether to delete the historical data based on the keyword;

[0136] If the historical data is to be deleted, a personalized deletion script corresponding to the script field for deleting historical data is obtained; and the personalized deletion script is executed to delete the historical data.

[0137] Furthermore, the general fields in the import file configuration table also include a statistical result data field, a statistical result data script field, and an import database row number field; Figure 6 As shown, the device further includes a statistical module 57, which is used to:

[0138] Obtaining a keyword in the statistical result data field, and determining whether to perform data statistical processing on the table after the imported data is completed based on the keyword;

[0139] If data statistics processing is performed on the table after the data import is completed, the personalized statistical script corresponding to the statistical result data script field is obtained; and the personalized statistical script is executed to obtain the row number statistical result of the table after the data import is completed;

[0140] The import database row number field is updated based on the row number statistics result.

[0141] Furthermore, if Figure 6 As shown, the device further comprises a comparison module 58, which is used to:

[0142] Obtain the first row value corresponding to the number of file rows in the general field; and obtain the second row value corresponding to the updated number of imported database rows field;

[0143] Compare the first row value with the second row value to obtain a row comparison result;

[0144] The row number comparison result is fed back to the relevant personnel of the target derivative task.

[0145] The embodiment of the present invention provides a device for importing file data into a database. Compared with the prior art, the present invention configures the file data import requirements by importing a file configuration table, which can improve the flexibility of importing file data into a database, avoid the development of a corresponding import program in the prior art, improve the efficiency of importing file data into the data, and save the time of importing. The present invention also sets a task name field in the import file configuration table, so that the method of importing file data into a database of the present invention is applicable to a variety of business scenarios, which greatly increases the scope of use of the method of the present invention. The present invention also obtains the file to be imported through the SFTP server, which can support the import of large files in the financial field, is safe and stable, and there is no need to worry about the problem of server crash and memory overflow due to the increase in file size.

[0146] According to an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the method of importing file data into a database in any of the above method embodiments.

[0147] Figure 7 A schematic diagram of the structure of a device provided according to an embodiment of the present invention is shown, and the specific embodiment of the present invention does not limit the specific implementation of the device.

[0148] like Figure 7 As shown, the device may include: a processor (processor) 602 , a communication interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0149] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0150] The communication interface 604 is used to communicate with other devices such as clients or other servers.

[0151] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps of the method for importing file data into the database.

[0152] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0153] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0154] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0155] The program 610 may be specifically configured to enable the processor 602 to perform the following operations:

[0156] Obtain an import file configuration table, wherein the import file configuration table includes multiple derivative tasks, and the derivative tasks include multiple common fields;

[0157] Acquire task name information and derivative status information corresponding to the task name field and the derivative status field from the general field, and determine the target derivative task based on the task name information and the derivative status information;

[0158] Obtaining a file path corresponding to the target derivative task from the import file configuration table, and obtaining a file to be imported through an SFTP server based on the file path;

[0159] The file to be imported is parsed to obtain a derivative script; and the derivative script is executed to complete the database import of the file to be imported.

[0160] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. The non-Company software tools or components appearing in the embodiments of the present invention are only examples and do not represent actual use.

Claims

1. A method for importing file data into a database, characterized in that: include: Obtain an import file configuration table, wherein the import file configuration table includes multiple derivative tasks, and the derivative tasks include multiple common fields; Acquire task name information and derivative status information corresponding to the task name field and the derivative status field from the general field, and determine the target derivative task based on the task name information and the derivative status information; Obtaining a file path corresponding to the target derivative task from the import file configuration table, and obtaining a file to be imported through an SFTP server based on the file path; The file to be imported is parsed to obtain a derivative script; and the derivative script is executed to complete the database import of the file to be imported.

2. The method according to claim 1, characterized in that Before obtaining the import file configuration table, the method further includes: Setting a task name of the derivative task according to the business scope, wherein the task name corresponds to at least one derivative task; The task name, table name, file path, derivative status, and number of file lines are set as the general fields to create the import file configuration table.

3. The method according to claim 1, characterized in that The determining of the target derivative task based on the task name information and the derivative state information comprises: receiving a file data import application from a user, the file data import application carrying business scope information; and determining a target task name from the task name information based on the business scope information; Determine at least one derivative task to be analyzed from the import file configuration table based on the target task name; The target derivative task is determined based on the derivative state information corresponding to the derivative task to be analyzed.

4. The method according to claim 1, characterized in that The step of parsing the file to be imported to obtain a derivative script includes: Parse the file to be imported, and obtain the content of the title row and the content of the data rows other than the title row of the file to be imported; Determine the content of the header row as a table field name, and determine the content of the data row as a table field value corresponding to the table field name; The derivative script is generated based on the table field name and the table field value.

5. The method according to claim 1, characterized in that The general fields in the import file configuration table include a field on whether to delete historical data and a field on deleting historical data script; Before parsing the file to be imported to obtain the derivative script, the method further includes: Obtaining a keyword in the field of whether to delete historical data, and determining whether to delete the historical data based on the keyword; If the historical data is to be deleted, a personalized deletion script corresponding to the script field for deleting historical data is obtained; and the personalized deletion script is executed to delete the historical data.

6. The method according to claim 1, characterized in that The general fields in the import file configuration table also include a statistical result data field, a statistical result data script field and an import database row number field; After executing the derivative script to complete the database import of the file to be imported, the method further includes: Obtaining a keyword in the statistical result data field, and determining whether to perform data statistical processing on the table after the imported data is completed based on the keyword; If data statistics processing is performed on the table after the data import is completed, the personalized statistical script corresponding to the statistical result data script field is obtained; and the personalized statistical script is executed to obtain the row number statistical result of the table after the data import is completed; The import database row number field is updated based on the row number statistics result.

7. The method according to claim 6, characterized in that After updating the import database row number field based on the row number statistics result, the method further includes: Obtain the first row value corresponding to the number of file rows in the general field; and obtain the second row value corresponding to the updated number of imported database rows field; Compare the first row value with the second row value to obtain a row comparison result; The row number comparison result is fed back to the relevant personnel of the target derivative task.

8. A device for importing file data into a database, characterized in that: include: A configuration table acquisition module is used to acquire an import file configuration table, wherein the import file configuration table includes multiple derivative tasks, and the derivative tasks include multiple common fields; A task determination module, used to obtain task name information and derivative status information corresponding to the task name field and the derivative status field from the general field, and determine the target derivative task based on the task name information and the derivative status information; A file acquisition module, used to acquire a file path corresponding to the target derivative task from the import file configuration table, and acquire a file to be imported through an SFTP server based on the file path; The derivative module is used to parse the file to be imported to obtain a derivative script; and execute the derivative script to complete the database import of the file to be imported.

9. A storage medium storing at least one executable instruction, wherein the executable instruction executes an operation corresponding to the method for importing file data into a database according to any one of claims 1 to 7.

10. A device comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method for importing file data into a database according to any one of claims 1 to 7.

Images