Large file uploading method and device based on relational database, equipment and medium

Abstract

The invention discloses a large file uploading method, device and equipment based on a relational database and a medium, and relates to the technical field of big data processing.The method comprises the steps that a Spark task is started, and all records stored in a configuration table under a specified path of an SFTP server are read; wherein the specified path of the SFTP server further stores a file uploaded by the user side, and the file corresponds to the configuration table; performing data verification before uploading according to each read record, performing conversion operation on the file after successful verification to generate a file meeting a format requirement, and uploading the converted file to a relational database; and after the uploading is completed, checking the uploaded data, and recording a checking result. According to the method, the problem that a large file with a large-scale data set is unsuccessfully uploaded to the relational database is solved, meanwhile, the uploading efficiency is increased by more than 70%, the uploading time is more flexible and is not limited by time, and emergency requirements of services can be met in time.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a large file uploading method, device, equipment and medium based on a relational database. Background Art

[0002] Due to the extremely strict management of the production environment, there are strict restrictions on direct manual operations on relational databases, so it is impossible to directly operate the production relational database. However, due to business needs, a large amount of configuration factor data needs to be uploaded to certain configuration tables of the relational database. The amount of data is very large, sometimes up to hundreds of millions.

[0003] In the prior art, the traditional DB deployment DML script upload method is used, that is, when processing large-scale data sets, DML statements are used to upload data to a relational database. Due to the characteristics of traditional databases, when the amount of data written by DML statements exceeds 10,000, the performance of this method will be significantly reduced, and when the amount of data reaches millions, tens of millions, or even hundreds of millions, the system may crash or fail to operate, resulting in unsuccessful file upload. In addition, DML deployment operations usually need to be synchronized with version releases, and the time is relatively fixed, which makes it unable to meet some urgent business needs. Summary of the invention

[0004] The purpose of the present invention is to provide a large file uploading method, device, equipment and medium based on a relational database, aiming to solve the problem of unsuccessful uploading of large files to a relational database in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a large file uploading method based on a relational database, comprising:

[0006] Start the Spark task to read all records in the configuration table stored in the specified path of the SFTP server; wherein the specified path of the SFTP server also stores files uploaded by the user, and the files correspond to the configuration table;

[0007] Performing data verification before uploading according to each record read, and converting the file to generate a file that meets the format requirements after the verification is successful, and uploading the converted file to the relational database;

[0008] After uploading is completed, the uploaded data is verified and the verification results are recorded.

[0009] In a second aspect, an embodiment of the present invention further provides a large file uploading device based on a relational database, comprising: a starting unit, used to start a Spark task, and read all records in a configuration table stored in a specified path of an SFTP server; wherein the specified path of the SFTP server also stores files uploaded by a user terminal, and the files correspond to the configuration table;

[0010] The uploading unit is used to perform data verification before uploading according to each record read, and after the verification is successful, perform a conversion operation on the file to generate a file that meets the format requirements, and upload the converted file to the relational database;

[0011] The verification unit is used to verify the uploaded data after the upload is completed and record the verification result.

[0012] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the large file uploading method based on a relational database as described in the first aspect above is implemented.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the large file uploading method based on a relational database as described in the first aspect above.

[0014] The embodiment of the present invention provides a large file uploading method, device, equipment and medium based on a relational database, and the method includes: starting a Spark task, reading all records in a configuration table stored in a specified path of an SFTP server; wherein the specified path of the SFTP server also stores files uploaded by a user, and the files correspond to the configuration table; performing data verification before uploading according to each record read, and converting the file to generate a file that meets the format requirements after the verification is successful, and uploading the converted file to a relational database; performing post-upload data verification after the upload is completed, and recording the verification result. The method solves the problem of unsuccessful uploading of large files with large-scale data sets to a relational database, and at the same time speeds up the upload efficiency by more than 70%, and the upload time is more flexible, not limited by time, and can meet the urgent needs of the business in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0016] Figure 1 Schematic diagram of an application environment of a large file uploading method based on a relational database provided in an embodiment of the present invention;

[0017] Figure 2 It is a flow chart of a large file uploading method based on a relational database provided in an embodiment of the present invention;

[0018] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S102;

[0019] Figure 4 yes Figure 3 A schematic flow chart of a specific implementation of step S202;

[0020] Figure 5 yes Figure 2 Another specific implementation flow diagram of step S102;

[0021] Figure 6 yes Figure 2 A flowchart of another specific implementation of step S102;

[0022] Figure 7 yes Figure 2 A schematic flow chart of a specific implementation of step S103;

[0023] Figure 8 It is a structural diagram of a large file uploading device based on a relational database provided in an embodiment of the present invention;

[0024] Fig. 9 is a schematic diagram of a structure of a computer device provided in an embodiment of the present invention;

[0025] Fig.10 2 is another structural diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0028] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0030] The large file uploading method based on relational database provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, wherein the client communicates with the program end through a network, the program end can start a Spark task through the client, and read all records in the configuration table stored in the specified path of the SFTP server; wherein the specified path of the SFTP server also stores files uploaded by the client, and the files correspond to the configuration table; data verification is performed before uploading according to each record read, and after the verification is successful, the file is converted to generate a file that meets the format requirements, and the converted file is uploaded to a relational database; after the upload is completed, the data verification after uploading is performed, and the verification result is recorded; in the present invention, for complex insurance entities such as insurance business, by starting a Spark task, reading the configuration table, performing data verification, file conversion, and finally uploading to the relational database and recording the verification results, the problem of unsuccessful uploading of large files with large-scale data sets to the relational database is solved, and the upload efficiency is accelerated by more than 70%, and the upload time is more flexible, not limited by time, and can meet the urgent needs of the business in a timely manner. Among them, the client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The program end can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 , Figure 2 A flowchart of a large file uploading method based on a relational database provided in an embodiment of the present invention includes steps S101 to S103:

[0032] S101, start the Spark task, read all records in the configuration table stored in the specified path of the SFTP server; wherein the specified path of the SFTP server also stores files uploaded by the user, and the files correspond to the configuration table;

[0033] In this embodiment, Spark is an open source distributed computing system that provides powerful data processing capabilities. Starting a Spark task means initializing the Spark application and allocating resources and execution environment for the next file data processing task. SFTP (Secure File Transfer Protocol) is a secure file transfer protocol used to transfer files between a local computer and a remote server. The Spark task connects to the SFTP server and reads the configuration table stored in the specified path. This configuration table contains information about the data files to be processed, such as the upload flag up_flag. Setting the upload flag up_flag to W [waiting for upload status] indicates that the file is ready for upload processing.

[0034] Furthermore, each record in the configuration table is associated with one or more files stored on the SFTP server, and this correspondence can be established through the configuration information in the configuration table. By reading the configuration table, the Spark task can determine which files need to be processed and how to process these files.

[0035] It should be noted that before starting the Spark task, the user will put the factor files to be uploaded (files uploaded by the user) into the specified path of the SFTP server. These files may contain business data, such as policy information, customer information, etc.

[0036] S102, performing data verification before uploading according to each record read, and converting the file to generate a file that meets the format requirements after the verification is successful, and uploading the converted file to the relational database;

[0037] In this embodiment, the Spark task reads each record in the configuration table from the SFTP server. Each record contains configuration information about the file to be uploaded. The Spark task selects the records whose up_flag is W [waiting for upload status]. The files corresponding to these records need to be processed later. For each selected record, the Spark task first performs data verification. After the verification is successful, the Spark task uploads the converted data to the relational database.

[0038] It should be noted that before uploading to the relational database, a database connection can be established by using appropriate database connection parameters (such as URL, user name, password, etc.) to establish a connection with the relational database.

[0039] Among them, Figure 3 As shown, in step S102, data verification is performed before uploading according to each record read, including the following steps S201-S202:

[0040] S201, obtaining files and configuration tables from the SFTP server, and uploading the files and configuration tables to the Hadoop cluster using an upload tool;

[0041] S202: Read configuration information of each record in the configuration table, and perform data verification before uploading according to the configuration information.

[0042] For steps S201 to S202, the user terminal first needs to connect to the SFTP server. You can use an SFTP client tool (such as an SSH client, WinSCP, an SFTP command line tool, etc.) to connect to the SFTP server, and provide necessary authentication information (such as a user name, password, or key) during the connection process to access the server. In the SFTP client, navigate to the specified path where files and configuration tables are stored. Use the download function of the SFTP client to download the files and configuration tables uploaded by the user terminal to the local storage (which may be the machine running the upload tool). Next, prepare the Hadoop upload tool and ensure that the Hadoop command line tool (such as hadoop fs) or a third-party Hadoop data upload tool (such as DistCp, hdfs dfs, etc.) has been correctly installed and configured. Then, upload the files and configuration tables to the Hadoop cluster. You can use the Hadoop upload tool to upload the files and configuration tables downloaded to the local to the specified directory of HDFS of the Hadoop cluster, which usually involves using commands such as hdfsdfs-put <local-file-path>

[0043] <hdfs-destination-path>(or similar commands, depending on the Hadoop version and tools used).

[0044] It should be noted that before uploading files and configuration tables to the Hadoop cluster, you need to ensure that the HDFS service of the Hadoop cluster is running and has sufficient storage space. In addition, verify whether the uploaded files and configuration tables are complete and correct in the HDFS web interface or by using command line tools.

[0045] Furthermore, the configuration table stored on HDFS is accessed using Hadoop command line tools or big data processing frameworks such as Hive and Spark. Each record in the configuration table is read to obtain relevant configuration information, and the file to be uploaded is verified according to the configuration information, that is, the data is verified before uploading according to the configuration information.

[0046] Among them, Figure 4 As shown, in step S202, the configuration information of each record in the configuration table is read, and data verification before uploading is performed according to the configuration information, including the following steps S301-S302:

[0047] S301. According to the target table name in the record, detect whether there is a target table corresponding to the target table name in the target database. If the target table name does not exist, update the upload mark of the corresponding record in the configuration table to the first mark, and record error information;

[0048] S302. If the target table name exists, check whether the data type contained in the file is consistent with the data type in the configuration table. If they are consistent, the verification is successful. If they are inconsistent, update the upload mark of the corresponding record in the configuration table to the first mark and record error information.

[0049] In this embodiment, the information in the configuration table includes the target table name tabl_name and data type for the file upload configured by the user in the specified configuration table. The target table name tabl_name is the table name in the relational database to which the factor file needs to be uploaded. The data type includes the file name file_name, which is the name of the factor file stored on the SFTP server; and the insurance type plan_code contained in the file, which is used to distinguish different types of insurance policies or insurance products.

[0050] For step S301, the target table name of each record is read from the configuration table, and an appropriate database connection tool or library (such as JDBC, ODBC, Python database connection library, etc.) is used to connect to the target database. An SQL query statement is executed to check whether a table corresponding to the read target table name exists in the target database, which usually involves using SELECT * FROM information_schema.tables WHERE table_name = ' <target-table-name>' (or similar query, depending on the database type) to check whether the table exists. If the target table does not exist, the upload mark of the corresponding record in the configuration table is updated to the first mark (such as "failed", "N", etc.), and then the error information is recorded, including detailed information such as the target table name, database name, error time, etc. If the target table exists, proceed to the next step (S302).

[0051] For step S302, read the file to be uploaded, and parse the file header or metadata to obtain the data type information contained in the file. At the same time, read the data type information corresponding to the current record from the configuration table. Compare the data type in the file with the data type in the configuration table column by column. Check whether the data type matches, including numeric type, string type, date type, etc. If the data type is consistent, it is considered that the verification is successful and the subsequent upload process can continue. If the data type is inconsistent, the upload mark of the corresponding record in the configuration table is updated to the first mark (the same as the mark in S301), and then record the error information, including detailed information such as file name, column name, data type in the configuration table, data type in the file, error time, etc.

[0052] Among them, Figure 5 As shown, in step S102, that is, converting the file to generate a file that meets the format requirements, and uploading the converted file to the relational database, the following steps S401 to S403 are included:

[0053] S401, obtaining the field value of the record deletion field in the configuration table;

[0054] S402: If the field value is yes, a delete statement is automatically constructed according to the target table name and data type, and the data of the target table in the target database is deleted according to the delete statement;

[0055] S403: If the field value is no, convert the file to generate a file that meets the format requirements of the target table, and upload the converted file to a relational database.

[0056] In this embodiment, the configuration information also includes a field for deleting records, and a flag for deleting historical records is_del_his_data [Y: delete historical records, N: do not delete historical records] is set as needed.

[0057] For steps S401 to S403, the system needs to read the configuration table and obtain the value of the field named "Whether to delete records". The value of this field determines whether the subsequent operation is to delete data or convert and upload files. If the value of the "Whether to delete records" field is "Yes", the system will automatically build an SQL delete statement based on the target table name and data type. This delete statement is used to delete all data in the corresponding target table in the target database (target library). The "target table name" and "data type" here may be pre-defined or read from the configuration table. If the field value is no, the file is converted to generate a file that meets the format requirements of the target table, and the converted file is uploaded to the relational database.

[0058] Specifically, this conversion operation is to make the format of the file conform to the requirements of the target table. After the conversion is completed, the system will upload the converted file to the relational database.

[0059] It should be noted that the entire process determines whether to perform a deletion operation or a file conversion and upload operation by reading the field value in the configuration table. This design makes the data processing process more flexible and configurable. In actual applications, such a process can be used in various scenarios such as data cleaning, data migration, and data synchronization.

[0060] Among them, Figure 6 As shown, in step S102, the converted file is uploaded to the relational database, including the following steps S501-S502:

[0061] S501, repartitioning the converted file to control the parallelism of uploading;

[0062] S502: Upload the file batches after the repartitioning operation to the relational database by using a multi-threaded concurrent method.

[0063] For step S501, the system will repartition the files that have been converted. It should be understood that repartitioning usually refers to dividing the data into multiple smaller parts or partitions according to certain rules (such as hash values, ranges, etc.). The purpose of doing this is mainly to control the degree of parallelism when uploading data to a relational database. Through repartitioning, a file that may have been very large can be divided into multiple small files or data blocks, and each small file or data block can be uploaded simultaneously by different threads or processes, which can significantly improve the efficiency of the upload operation and reduce the upload time.

[0064] It should be noted that the specific repartitioning strategy will be determined based on the characteristics of the data and the capabilities of the target database to which it is uploaded. For example, if the target database supports batch inserts, the data can be split into sizes suitable for batch inserts.

[0065] For step S502, after completing the file repartitioning operation, the system will use a multi-threaded concurrent method to upload these partitioned files or data blocks. Multi-threaded concurrency means that multiple threads are performing upload operations at the same time, and each thread is responsible for uploading one or more partitioned data. Multi-threaded concurrent uploading can make full use of the processing power of the database, thereby speeding up the data upload speed. At the same time, since the data has been divided into multiple small parts, even if a thread encounters a problem during the upload process (such as network interruption, database connection failure, etc.), it can be more easily recovered and retried without affecting the entire upload process.

[0066] It should be noted that multi-threaded concurrent uploading also requires reasonable control of concurrency to avoid excessive pressure on the database or data conflicts. Therefore, in actual applications, it is necessary to set the appropriate number of threads based on the performance and concurrent processing capabilities of the target database.

[0067] S103: After uploading is completed, verify the uploaded data and record the verification result.

[0068] In this embodiment, by performing data verification after uploading and recording the verification result, it is ensured that the uploaded file data is not lost. It should be noted that when extracting and verifying data, the security and privacy of the data need to be ensured.

[0069] Among them, Figure 7 As shown, in step S103, that is, after the upload is completed, the uploaded data is verified and the verification result is recorded, including the following steps S601 to S603:

[0070] S601, obtaining the uploaded data of the file and the successfully uploaded data in the target table, and comparing them;

[0071] S602, if they are consistent, updating the upload mark of the corresponding record in the configuration table to the second mark, and recording success information;

[0072] S603: If they are inconsistent, update the upload mark of the corresponding record in the configuration table to the first mark, and record error information.

[0073] For steps S601 to S603, data is extracted from the uploaded file, which is usually data that is prepared to be uploaded to a relational database after conversion and repartitioning. The successfully uploaded data is obtained from the target table. This step requires a query operation to retrieve data records related to the uploaded file in the target table. The two sets of data are compared to verify whether they are consistent. The verification can be performed by comparing row by row, calculating hash values, or performing other forms of verification.

[0074] Furthermore, if it is determined after comparison that the uploaded data is consistent with the data in the target table, the system needs to perform the following operations: update the records related to the current upload task in the configuration table, set its upload mark to the second mark (usually indicating "success" or "completed"), and then record the success information, including the uploaded file name, target table name, upload time, verification results and other detailed information, which can be used for subsequent data audits or troubleshooting.

[0075] Furthermore, if it is determined after comparison that the uploaded data is inconsistent with the data in the target table, the system needs to perform the following operations: update the records related to the current upload task in the configuration table, set its upload mark to the first mark (usually indicating "failure" or "incomplete"), and then record the error information. The error information includes the uploaded data volume of the file and the successfully uploaded data volume in the target table, difference description, uploaded file name, target table name, upload time and other detailed information. This information is crucial for subsequent problem location and repair.

[0076] In a specific embodiment, if they are consistent, the file upload is successful. The Up_flag field of the record in the configuration table is set to E["upload successful"], and the field log is set to [file upload successful, upload record data is: [the amount of successfully uploaded data in the target table]]; if they are inconsistent, the file upload fails. The Up_flag field of the record in the configuration table is set to F["upload failed"], and the field log is set to [file upload failed, upload file record data is: [the amount of uploaded data of the file], the target table successful upload record data is [the amount of successfully uploaded data in the target table], and the two are inconsistent]].

[0077] It can be seen that in the above solution, for complex insurance entities under insurance business, by starting Spark tasks, reading configuration tables, performing data verification, file conversion, and finally uploading to relational databases and recording verification results, the problem of unsuccessful uploading of large files with large-scale data sets to relational databases is solved. At the same time, the upload efficiency is increased by more than 70%, and the upload time is more flexible and not restricted by time, which can meet the urgent needs of the business in a timely manner.

[0078] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0079] The embodiment of the present invention also provides a large file upload device based on a relational database, and the large file upload device based on a relational database corresponds one-to-one to the large file upload method based on a relational database in the above embodiment. Figure 8 As shown, the large file uploading device based on relational database includes a starting unit 701, an uploading unit 702, and a verification unit 703. Each functional unit is described in detail as follows:

[0080] The starting unit 701 is used to start the Spark task and read all records in the configuration table stored in the specified path of the SFTP server; wherein the specified path of the SFTP server also stores files uploaded by the user end, and the files correspond to the configuration table;

[0081] The uploading unit 702 is used to perform data verification before uploading according to each record read, and after the verification is successful, perform a conversion operation on the file to generate a file that meets the format requirements, and upload the converted file to the relational database;

[0082] The verification unit 703 is used to verify the uploaded data after the upload is completed and record the verification result.

[0083] In one embodiment, the uploading unit 702 is specifically configured to:

[0084] Obtain files and configuration tables from the SFTP server, and upload the files and configuration tables to the Hadoop cluster using an upload tool;

[0085] The configuration information of each record in the configuration table is read, and data verification is performed before uploading according to the configuration information.

[0086] In one embodiment, the uploading unit 702 is specifically configured to:

[0087] According to the target table name in the record, detect whether there is a target table corresponding to the target table name in the target database; if the target table name does not exist, update the upload mark of the corresponding record in the configuration table to the first mark, and record error information;

[0088] If the target table name exists, check whether the data type contained in the file is consistent with the data type in the configuration table. If they are consistent, the verification is successful. If they are inconsistent, update the upload mark of the corresponding record in the configuration table to the first mark and record the error information.

[0089] In one embodiment, the uploading unit 702 is further configured to:

[0090] Obtain the field value of the record deletion field in the configuration table;

[0091] If the field value is yes, a delete statement is automatically constructed based on the target table name and data type, and the data of the target table in the target database is deleted based on the delete statement;

[0092] If the field value is no, a conversion operation is performed on the file to generate a file that meets the format requirements of the target table, and the converted file is uploaded to the relational database.

[0093] In one embodiment, the uploading unit 702 is further configured to:

[0094] Repartitioning the converted files to control the parallelism of uploading;

[0095] The file batches after the repartitioning operation are uploaded to the relational database using multi-threaded concurrent mode.

[0096] In one embodiment, the verification unit 703 is specifically configured to:

[0097] Get the uploaded data of the file and the successfully uploaded data in the target table, and compare them;

[0098] If they are consistent, the upload mark of the corresponding record in the configuration table is updated to the second mark, and success information is recorded;

[0099] If they are inconsistent, the upload mark of the corresponding record in the configuration table is updated to the first mark, and an error message is recorded.

[0100] The present invention provides a large file uploading device based on a relational database. The device solves the problem of unsuccessful uploading of large files with large-scale data sets to a relational database by starting a Spark task, reading a configuration table, performing data verification, file conversion, and finally uploading to a relational database and recording verification results. At the same time, the upload efficiency is increased by more than 70%, and the upload time is more flexible and not limited by time, so that the urgent needs of the business can be met in time.

[0101] For the specific limitations of the large file upload device based on a relational database, please refer to the limitations of the large file upload method based on a relational database above, which will not be repeated here. Each module in the above-mentioned large file upload device based on a relational database can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0102] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a large file upload method based on a relational database are implemented.

[0103] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a large file upload method based on a relational database are implemented.

[0104] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0105] Receive user questions, and perform question recall processing from a preset standard question library according to the user questions to obtain multiple candidate questions;

[0106] Start the Spark task to read all records in the configuration table stored in the specified path of the SFTP server; wherein the specified path of the SFTP server also stores files uploaded by the user, and the files correspond to the configuration table;

[0107] Performing data verification before uploading according to each record read, and converting the file to generate a file that meets the format requirements after the verification is successful, and uploading the converted file to the relational database;

[0108] After uploading is completed, the uploaded data is verified and the verification results are recorded.

[0109] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0110] Start the Spark task to read all records in the configuration table stored in the specified path of the SFTP server; wherein the specified path of the SFTP server also stores files uploaded by the user, and the files correspond to the configuration table;

[0111] Performing data verification before uploading according to each record read, and converting the file to generate a file that meets the format requirements after the verification is successful, and uploading the converted file to the relational database;

[0112] After uploading is completed, the uploaded data is verified and the verification results are recorded.

[0113] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0116] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention. < / local-file-path>

Claims

1. A large file uploading method based on relational database, characterized in that: include: Start the Spark task to read all records in the configuration table stored in the specified path of the SFTP server; wherein the specified path of the SFTP server also stores files uploaded by the user, and the files correspond to the configuration table; Performing data verification before uploading according to each record read, and converting the file to generate a file that meets the format requirements after the verification is successful, and uploading the converted file to the relational database; After uploading is completed, the uploaded data is verified and the verification results are recorded.

2. The large file uploading method based on relational database according to claim 1 is characterized in that: The data verification before uploading according to each record read includes: Obtain files and configuration tables from the SFTP server, and upload the files and configuration tables to the Hadoop cluster using an upload tool; The configuration information of each record in the configuration table is read, and data verification is performed before uploading according to the configuration information.

3. The large file uploading method based on relational database according to claim 2 is characterized in that: The configuration information includes a target table name and a data type, and the reading of the configuration information of each record in the configuration table and performing data verification before uploading according to the configuration information include: According to the target table name in the record, detect whether there is a target table corresponding to the target table name in the target database; if the target table name does not exist, update the upload mark of the corresponding record in the configuration table to the first mark, and record error information; If the target table name exists, check whether the data type contained in the file is consistent with the data type in the configuration table. If they are consistent, the verification is successful. If they are inconsistent, update the upload mark of the corresponding record in the configuration table to the first mark and record the error information.

4. The large file uploading method based on relational database according to claim 3 is characterized in that: The configuration information also includes whether to delete the record field, and the converting operation on the file to generate a file that meets the format requirements, and uploading the converted file to the relational database includes: Obtain the field value of the record deletion field in the configuration table; If the field value is yes, a delete statement is automatically constructed based on the target table name and data type, and the data of the target table in the target database is deleted based on the delete statement; If the field value is no, a conversion operation is performed on the file to generate a file that meets the format requirements of the target table, and the converted file is uploaded to the relational database.

5. The large file uploading method based on relational database according to claim 4 is characterized in that: The step of uploading the converted file to a relational database includes: Repartitioning the converted files to control the parallelism of uploading; The file batches after the repartitioning operation are uploaded to the relational database using multi-threaded concurrent mode.

6. The large file uploading method based on relational database according to claim 1 is characterized in that: After the upload is completed, the uploaded data is verified and the verification result is recorded, including: Get the uploaded data of the file and the successfully uploaded data in the target table, and compare them; If they are consistent, the upload mark of the corresponding record in the configuration table is updated to the second mark, and success information is recorded; If they are inconsistent, the upload mark of the corresponding record in the configuration table is updated to the first mark, and an error message is recorded.

7. The large file uploading method based on relational database according to claim 6 is characterized in that: The error information includes the uploaded data volume of the file and the successfully uploaded data volume in the target table.

8. A large file uploading device based on a relational database, characterized in that: include: A starting unit, used to start a Spark task, and read all records in a configuration table stored in a specified path of an SFTP server; wherein the specified path of the SFTP server also stores files uploaded by a user, and the files correspond to the configuration table; The uploading unit is used to perform data verification before uploading according to each record read, and after the verification is successful, perform a conversion operation on the file to generate a file that meets the format requirements, and upload the converted file to the relational database; The verification unit is used to verify the uploaded data after the upload is completed and record the verification result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the large file uploading method based on a relational database as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the large file uploading method based on a relational database as described in any one of claims 1 to 7.

Images