FTP (File Transfer Protocol) data storage method, device and equipment and readable storage medium
By presetting the mapping relationship between persistence class and file identifier, batch storage of FTP server data to the database, solving the problem of low FTP data storage efficiency in the prior art, and achieving efficient data storage and processing.
Patent Information
- Application Number
- CN202310780991.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-28
- Publication Date
- 2024-12-31
AI Technical Summary
In the prior art, FTP data storage efficiency is low, and data table configuration is required by file-by-file and cannot be batch-saved, resulting in inefficiency.
By presetting a persistent class that represents the data table mapping relationship corresponding to the file identifier, a persistent class entity object is constructed, the row data of the file to be downloaded in the FTP server is stored in the entity object, and the entity object is used storing row data into the database data table in the form of storing row data tables without configuring the data tables file by file.
It greatly improves the efficiency of FTP data storage, and can configure the persistence classes corresponding to multiple files according to the file identity, and save rows of data in different files into the data table in batches, reducing the number of database operations and improving storage efficiency.
Smart Images
Figure CN119226283A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to data storage technology, and in particular to a FTP data storage method, device, equipment and readable storage medium. Background Art
[0002] Operators will upload the massive alarm logs, fault work orders and other files they receive to the File Transfer Protocol (FTP) server for storage. However, files stored on the FTP server are not convenient for data statistics, query and other operations. Therefore, the file data stored on the FTP server needs to be stored in the database for subsequent data processing.
[0003] At present, the method of storing the file data of the FTP server in the database is mainly to first download the required files from the FTP server to the local computer, and then store the file data in the database. However, this method is inefficient in storing FTP data. Summary of the invention
[0004] The present application provides an FTP data storage method, apparatus, device and readable storage medium to solve the problem of low FTP data storage efficiency.
[0005] In a first aspect, the present application provides an FTP data storage method, comprising:
[0006] Using the storage information of the file to be downloaded to the database, a connection is established with the FTP server, and a candidate file list is determined from the FTP server;
[0007] Determine the file to be downloaded to the database from the candidate file list according to the file identifier in the download configuration parameter;
[0008] According to the persistent class indicated in the download configuration parameter, an entity object of the persistent class is constructed, and the row data of the file is stored in a data table corresponding to the persistent class in the database, wherein the entity object corresponds to a row in the data table.
[0009] Optionally, constructing an entity object of the persistent class according to the persistent class indicated in the download configuration parameter, and storing the row data of the file in a data table corresponding to the persistent class in the database includes:
[0010] Obtaining a column name corresponding to each column of data in the row of data, and dividing the row of data using a data separator in the download configuration parameter to obtain each column of data in the row of data;
[0011] Assign each column data and the corresponding column name to the value and key of the intermediate object respectively to obtain the intermediate object;
[0012] Assigning the attributes of the intermediate object to the entity object of the persistent class;
[0013] The entity object is stored in the data table, so as to store the row data in the database table.
[0014] Optionally, the download configuration parameter further includes an entity type object of a preset class, the persistent class is the entity type object of the preset class, and the assigning the attribute of the intermediate object to the entity object of the persistent class includes:
[0015] Based on Java reflection technology, construct an entity object of the entity type object of the preset class to obtain the entity object of the persistent class;
[0016] Assign the attributes of the intermediate object to the entity object.
[0017] Optionally, storing the entity object in the database table includes:
[0018] Add the entity object to the entity object list of the data table;
[0019] When the number of entity objects stored in the entity object list reaches a preset value, the entity object list is saved in the data table.
[0020] Optionally, the method further comprises: constructing an entity object of the persistent class according to the persistent class indicated in the download configuration parameter, and storing the row data of the file in a data table corresponding to the persistent class in the database;
[0021] Read row data from the file in sequence, and store the read row data in a row list;
[0022] Determining whether the number of row data in the row list reaches the number of row data batches indicated in the storage information of the file to be downloaded to the database;
[0023] If the number of row data in the row list reaches the batch number, or the number of row data in the row list does not reach the batch number and there is no unread row data in the file, then the row data in the row list is stored in the data table corresponding to the persistent class in the database by using the entity object of the persistent class indicated in the download configuration parameter.
[0024] Optionally, the method further includes:
[0025] Configuration information is received, where the configuration information is used to configure the storage information and / or the download configuration parameters.
[0026] Optionally, the storage information also includes: the host name, user name, password, and file storage path of the FTP server;
[0027] The download configuration parameters also include: file parsing encoding format and database interface object.
[0028] In a second aspect, the present application provides an FTP data storage device, comprising:
[0029] A processing module, used to establish a connection with an FTP server using storage information of files to be downloaded to a database, and to determine a candidate file list from the FTP server;
[0030] A determination module, used to determine the file to be downloaded to the database from the candidate file list according to the file identifier in the download configuration parameter;
[0031] The storage module is used to construct an entity object of the persistent class according to the persistent class indicated in the download configuration parameters, and store the row data of the file in a data table corresponding to the persistent class in the database, wherein the entity object corresponds to a row in the data table.
[0032] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;
[0033] The memory stores computer-executable instructions;
[0034] The processor executes the computer-executable instructions stored in the memory to implement the FTP data storage method as described in any one of the first aspects.
[0035] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the FTP data storage method as described in any one of the first aspects.
[0036] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the FTP data storage method as described in any one of the first aspects.
[0037] In a sixth aspect, the present application provides a chip having a computer program stored thereon, wherein when the computer program is executed by the chip, the FTP data storage method as described in any one of the first aspects is implemented.
[0038] The FTP data storage method, device, equipment and readable storage medium provided by the present application pre-set a persistent class corresponding to a file identifier and representing a data table mapping relationship, and by constructing an entity object of the persistent class, the row data of the file to be downloaded to the database in the FTP server is stored in the entity object, and the row data of the file to be downloaded to the database is stored in the data table corresponding to the persistent class in the form of the entity object. The method does not need to configure the data table file by file, but only needs to configure a persistent class corresponding to the file identifier according to the file identifier, and all files corresponding to the file identifier in the storage path can be stored in the data table, which greatly improves the efficiency of FTP data storage. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0040] Figure 1 A schematic diagram of an application scenario of an FTP data storage method provided in this application;
[0041] Figure 2 A flowchart of an FTP data storage method provided by this application;
[0042] Figure 3 A flowchart of another FTP data storage method provided by this application;
[0043] Figure 4 A flowchart of another FTP data storage method provided by this application;
[0044] Figure 5 A schematic diagram of the structure of an FTP data storage device provided in this application;
[0045] Figure 6 A schematic diagram of the structure of an electronic device provided in this application.
[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0048] First, the terms involved in this application are explained:
[0049] Persistent class: In an application, a class used to implement business problem entities, whose instances (objects) will be persistently stored in the database.
[0050] The operator's data monitoring system will receive a large number of alarm logs, fault work orders and other files. The data monitoring system will upload the various files received to the FTP server for storage. However, storing them in the FTP server is not convenient for performing operations such as data statistics and analysis. Therefore, the files stored in the FTP server need to be stored in the database to facilitate subsequent data processing and analysis.
[0051] At present, the method of first downloading the file from the FTP server to the local computer and then importing the data in the file into the database for storage is usually adopted. For example, the following two methods can be adopted:
[0052] (1) After downloading the file from the FTP server to the local computer, use the database import software to configure the data table through human-computer interaction. The database import software then stores each row of data in the file into the data table line by line.
[0053] However, this method requires the user to configure a corresponding data table for each file, and cannot implement batch saving of files, that is, the efficiency of FTP data storage is low.
[0054] (2) After downloading the files from the FTP server to the local computer, develop database software based on the configuration requirements of the data table to save files with the same data table configuration requirements.
[0055] However, this method requires the development of different database software for different configuration requirements, and has low reusability and low efficiency.
[0056] In summary, the efficiency of the FTP data storage method in the prior art is low.
[0057] In view of this, the present application provides an FTP data storage method, which classifies files based on file identifiers, configures a persistence class for a class of files that establishes a mapping relationship with a data table, and stores the row data in the file in an entity object of the persistence class so as to store the row data in the data table. There is no need to configure the data table for each file. Only the persistence classes corresponding to multiple file identifiers need to be configured to save the row data in different files into the data table, thereby improving the efficiency of FTP data storage.
[0058] Figure 1 This is a schematic diagram of an application scenario of an FTP data storage method provided in this application. Figure 1 As shown, it includes FTP server, FTP client and database.
[0059] The FTP server is used to store files. It should be noted that this application does not limit the type and source of files stored in the FTP server. The files may be uploaded to the server by the aforementioned data monitoring system, acquired by the server through communication with other systems, or imported by external devices.
[0060] The FTP client is software for exchanging files between an electronic device and an FTP server. The FTP client may be a client of the World Wide Web (web) version or an application (app) version, which is not limited in this application. The FTP client is deployed in the electronic device, which may be a computer, a mobile phone, a tablet computer, etc., which is not limited in this application.
[0061] The above database is used to store data according to a preset data structure. It should be noted that the application does not limit the type of database.
[0062] The executor of this application is the above-mentioned electronic device deployed with an FTP client.
[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0064] Figure 2 A flowchart of an FTP data storage method provided by this application is as follows: Figure 2 As shown, the method includes:
[0065] S101, using the storage information of the file to be downloaded to the database, establishing a connection with an FTP server, and determining a candidate file list from the FTP server.
[0066] The above-mentioned files to be downloaded to the database refer to files stored in the FTP server but need to be downloaded to the database.
[0067] Optionally, the above storage information may include the following parameters: the host name, user name, password, file storage path, and the number of row data batches of the FTP server. The number of row data batches refers to the maximum number of rows processed at one time. It should be noted that the parameters included in the above storage information are only a schematic representation and may include some or all of the above parameters, which is specifically related to the setting of the storage information. When the above storage information includes some of the above parameters, other parameters may be preset by the above FTP client, such as the number of row data batches preset by the client.
[0068] The candidate file list refers to the file names of all files stored in the file storage path of the FTP server.
[0069] Optionally, configuration information is received, and the configuration information is used to configure the above storage information, for example, the host name, user name, password, file storage path, and number of row data batches of the above FTP server. The configuration information can be received by the FTP client through human-computer interaction, receiving the above configuration information input by the user, or by the FTP client receiving the configuration information from other electronic devices through communication.
[0070] The FTP client can establish a connection with the FTP server according to the host name, user name, password and other information of the FTP server in the storage information, and obtain all file names in the file storage path of the FTP server according to the file storage path in the storage information to obtain the candidate file list.
[0071] S102: Determine the file to be downloaded to the database from the candidate file list according to the file identifier in the download configuration parameter.
[0072] Optionally, the above-mentioned download configuration parameters may include the following parameters: file identifier, persistence class, data separator, file parsing encoding format, and database interface object. It should be noted that the parameters included in the above-mentioned download configuration parameters are only a schematic representation and may include some or all of the above-mentioned parameters, which is specifically related to the setting of the download configuration parameters. When the above-mentioned download configuration parameters include some of the above-mentioned parameters, other parameters may be preset by the above-mentioned FTP client, such as the data separator preset by the client.
[0073] Exemplarily, the download configuration parameters may be as shown in Table 1:
[0074] Table 1 Download configuration parameters
[0075] Download Configuration Parameters Numeric File Identification SZ_lan Persistence Class TpnmGridSzlan.class Database interface object TpnmGridSzlanRepository Data separator "&” File parsing encoding format "UTF-8"
[0076] The above file identifier is used to characterize the type of the file, for example, it can be a prefix of the file name, and files with the same prefix belong to the same type of file. It should be noted that files of the same type have the same file attributes, such as file parsing encoding format, data separator, and data format. The data format can be, for example, a column name setting of the data.
[0077] The above persistence class is a class that is preset in advance to establish a mapping relationship with a data table in the database.
[0078] The above data separator is used to split a row of data to obtain data in each column of the row, such as "&". It should be noted that this application does not limit this.
[0079] The above file parsing encoding format refers to the encoding method when the characters in the file are stored, such as American Standard Code for Information Interchange (ASCII), 8-bit (Universal Character Set / Unicode Transformation Format, UTF-8), etc. It should be noted that this application does not make any limitation here.
[0080] The above database interface object is an object of the database interface class in Java, which is used to represent the interface of the database where the file to be downloaded is stored. It should be noted that the interface has established a database connection with the FTP client in advance. The database interface class in Java can refer to the existing technology, and this application does not limit it here.
[0081] Optionally, receive configuration information, the configuration information is used to configure the above download configuration parameters, for example, may include the above file identifier, persistence class, data separator, file parsing encoding format, database interface object. The configuration information can be received by the FTP client through human-computer interaction, receiving the above configuration information input by the user, or the FTP client can receive the configuration information from other electronic devices through communication.
[0082] According to the above file identifier, the file to be downloaded to the database can be determined from the above candidate file list. A possible implementation method is to determine one by one whether the files in the candidate file list have the same file identifier. If they have the same file identifier, the file will be used as the file to be downloaded to the database.
[0083] Another possible implementation method is to filter the file identifier to obtain a sublist of the candidate file list, which includes all files with the same file identifier in the candidate file list and arranges them in chronological order, and determines the first file as the file to be downloaded to the database.
[0084] Optionally, a download configuration parameter list may be preconfigured, and the download configuration parameter list includes multiple groups of download configuration parameters. It should be noted that the file identifiers in each group of download configuration parameters are different and correspond to different persistence classes. Other parameters except the file identifier and persistence class may be the same or different, and this application does not limit this. Exemplarily, the above download configuration parameter list is shown in Table 2:
[0085] Table 2 Download configuration parameter list
[0086] Serial number / download configuration parameters Download configuration parameters 1 Download configuration parameters 2 File Identification "SZ_lan" "SZ_olt" Persistence Class TpnmGridSzlan.class TpnmGridSzOlt.class Database interface object TpnmGridSzlanRepository tpnmGridSzOltRepository Data separator "&” "&” File parsing encoding format "UTF-8" "UTF-8"
[0087] According to the above-mentioned download configuration parameter list, for each file in the above-mentioned candidate file list, determine whether the file has the file identifier included in the above-mentioned download configuration parameter list. If so, it means that the file has the file identifier included in the above-mentioned download configuration parameter list, and the file is determined as a file to be downloaded to the database; if not, it means that the file does not have the file identifier included in the above-mentioned download configuration parameter list, and continue to determine the next file in the above-mentioned candidate file list.
[0088] S103: According to the persistent class indicated in the above download configuration parameters, construct an entity object of the persistent class, and store the row data of the file in a data table corresponding to the persistent class in the database.
[0089] The entity object of the above-mentioned persistent class corresponds to a row in the data table. Regarding how to construct the entity object of the persistent class, one possible implementation method is to directly use the new method to construct the entity object of the persistent class. Another possible implementation method is that the above-mentioned download configuration parameters also include the entity type object of the preset class, and the persistent class is the entity type object of the preset class, and then the entity object of the entity type object can be constructed based on Java reflection technology to obtain the entity object of the persistent class.
[0090] How to store the row data of the file in the data table corresponding to the persistent class in the database can be implemented as follows:
[0091] A possible implementation method is to first use the data separator in the above download configuration parameters to split the row data to obtain the column data in the row data; then assign each column data and the column name corresponding to each column data to the value and key of the entity object respectively, so as to store the row data of the file in the data table corresponding to the persistent class in the database.
[0092] Another possible implementation method is to first use the data separator in the above download configuration parameters to split the row data to obtain the column data in the row data; then assign each column data and the column name corresponding to each column data to the value and key of the intermediate object, and use the intermediate object to store the column data in the row data and the corresponding column name; finally, assign the attributes of the intermediate object to the entity object to store the row data of the file in the data table corresponding to the persistence class in the database.
[0093] It should be understood that, as mentioned above, the entity object of the above-mentioned persistence class corresponds to a row in the data table, and assigning the column data of the row data and the corresponding column name to the entity object is equivalent to storing the column data of the row data and the corresponding column name to the corresponding position in the data table.
[0094] It should be noted that the above method can be used to store each row of data in the file to be downloaded to the database row by row in the above data table.
[0095] It should be understood that, as mentioned above, the file identifier in the above download configuration parameters has a one-to-one correspondence with the persistence class, that is, the file identifier has a one-to-one correspondence with the data table, and each row of data of the file can be stored in the corresponding data table according to the file identifier.
[0096] The FTP data storage method provided by the present application presets a persistent class corresponding to a file identifier and representing a data table mapping relationship, and by constructing an entity object of the persistent class, stores the row data of the file to be downloaded to the database in the FTP server in the entity object, and uses the entity object to store the row data of the file to be downloaded to the database in the data table corresponding to the persistent class. The method does not need to configure the data table file by file, but only needs to configure a persistent class corresponding to the file identifier according to the file identifier, and all files corresponding to the file identifier in the storage path can be stored in the data table, which greatly improves the efficiency of FTP data storage.
[0097] The following describes how, after determining the file to be downloaded to the database, an entity object of the persistent class is constructed according to the persistent class indicated in the above download configuration parameters, and the row data of the file is stored in the data table corresponding to the persistent class in the database.
[0098] Figure 3 A flowchart of another FTP data storage method provided by this application. Figure 3 As shown, the method includes:
[0099] S201, reading a row of data in the file to be downloaded to the database.
[0100] It should be noted that the above row of data is data other than the first row of data of the file, because the first row of data of a file is usually the column name.
[0101] A possible implementation method is to first download the file to be downloaded to the database to the local computer through an FTP client, and then read a line of data in the file. Regarding how to download the file to the local computer through an FTP client, reference can be made to the prior art, and this application does not limit this. Regarding the reading method, for example, it can be the readLine method, and this application does not limit this.
[0102] Another possible implementation is to save each line of data in the file to be downloaded to the database into a buffer preset by the FTP client, and read a line of data from the buffer.
[0103] Exemplarily, an input stream can be constructed, and the byte stream data in the above file can be converted into character stream data using the InputStreamReader method and the file parsing encoding format in the above download configuration parameters, and saved in a preset buffer, and then a line of data in the buffer can be read using BufferedReader.
[0104] In this way, the FTP data can be read and subsequently stored without downloading the files in the FTP server to the local computer, thereby improving the efficiency of FTP data storage.
[0105] S202: Store the read row data into a row list.
[0106] S203: Determine whether the number of row data in the row list reaches the number of row data batches indicated in the storage information of the file to be downloaded to the database.
[0107] If yes, it means that the number of row data in the row list reaches the batch number, and step S204 is executed; if no, it means that the number of row data in the row list does not reach the batch number, and step S201 is returned to execute; if the number of row data in the row list does not reach the batch number, and there is no unread row data in the file, it means that the number of households in the file has been read, but the batch number in the row list has not been reached, and step S204 is executed. It should be understood that there is no unread row data in the file can be no unread data in the file, or there is no unread data in the buffer.
[0108] It should be noted that the above steps S202-S203 are optional steps. In this way, the row data is stored and processed in batches, which can reduce the number of subsequent communications with the database, relieve the pressure on the database, and further reduce the number of disk reads and writes, thereby increasing the service life of the disk.
[0109] S204: Obtain the column name corresponding to each column of data in the row of data, and use the data separator in the download configuration parameter to split the row of data to obtain each column of data in the row of data.
[0110] It should be noted that the column names corresponding to the column data in the above row data are preset in the persistence class in the above download configuration parameters and can be obtained from the persistence class.
[0111] Exemplarily, the row of data can be segmented using the data separator in the download configuration parameters to obtain an array including the data in each column of the row, so as to obtain the data in each column of the row, wherein each element in the array corresponds one-to-one to each column of the row.
[0112] It should be noted that if step S204 is executed after step S203, the row data obtained in step S204 is the row data in the above row list.
[0113] S205 . Assign each column of data and the corresponding column name to the value and key of the intermediate object respectively, to obtain the intermediate object.
[0114] The above intermediate object is used to store each column of data and its column name in a row of data in the form of key-value pairs, for example, it can be a map object.
[0115] A column of data is assigned to the value of the intermediate object, a column name corresponding to the column of data is assigned to the key of the intermediate object, and the column of data and its column name are stored in the intermediate object in the form of a key-value pair.
[0116] S206: Assign the attributes of the intermediate object to the entity object of the persistent class.
[0117] As mentioned above, the download configuration parameters also include an entity type object of a preset class, and the persistent class is an entity type object of the preset class. For example, the preset class is a class class, and the persistent class is assigned to an object of the class class, that is, an entity type object, in the download configuration parameters. Exemplarily, the download configuration parameters may be as shown in Table 3:
[0118] Table 3 Download configuration parameters
[0119] Download Configuration Parameters Numeric File Identification SZ_lan Entity Type Object TpnmGridSzlan.class Database interface object TpnmGridSzlanRepository Data separator "&” File parsing encoding format "UTF-8"
[0120] When the persistent class is subsequently called, there is no need to write code based on the persistent class to call the method of the persistent class, etc. It is only necessary to construct the entity object of the entity type object based on Java reflection technology to obtain the entity object of the persistent class. For example, the object of the entity type object is constructed by the getDeclaredConstructor().newInstance() method to obtain the object of the persistent class.
[0121] This method is equivalent to establishing a template for the persistent class (the entity type object of the class class). There is no need to modify the calling method in the code according to the different persistent classes. You only need to write a call to the entity type object once based on Java reflection technology to call various persistent classes.
[0122] Optionally, the object of the entity type object constructed by the getDeclaredConstructor().newInstance() method is a bean object, and the properties of the intermediate object are directly copied to the bean object through the BeanUtils.populate() method, thereby assigning the properties of the intermediate object to the entity object of the persistent class. This method can copy all the key-value pair information of the intermediate object to the bean object at one time, improving efficiency while ensuring that the key-value pair information will not be confused.
[0123] S207: Store the entity object in the data table to store the row of data in the database table.
[0124] Optionally, the entity object is first saved in an entity object list corresponding to the data table, the entity object list being used to store entity objects in the data table. When the number of entity objects stored in the entity object list reaches a preset value, the entity object list is stored in the data table.
[0125] In this way, row data can be saved in batches to the data table, which can reduce the number of communications with the database, relieve the pressure on the database, and thus reduce the number of disk reads and writes, thereby increasing the service life of the disk.
[0126] The FTP data storage method provided by the present application utilizes an intermediate object to store the column data and column names in the row data of an FTP file, and then directly copies the attributes of the intermediate object to the entity object of the persistence class, thereby improving the efficiency of row data acquisition and storage.
[0127] Figure 4 A flowchart of another FTP data storage method provided by this application. Figure 4 As shown, the method includes:
[0128] S301, using the storage information of the file to be downloaded to the database, establish a connection with an FTP server, and determine a candidate file list from the FTP server.
[0129] S302: Determine the file to be downloaded to the database from the candidate file list according to the file identifier in the download configuration parameter.
[0130] Specifically, a sublist of the candidate file list can be obtained according to the file identifier, the sublist including all files with the same file identifier in the candidate file list and arranged in chronological order, and the first file is determined as the file to be downloaded to the database.
[0131] S303: Save each line of data in the file to be downloaded to the database into a buffer preset by the FTP client, and read a line of data from the buffer.
[0132] Specifically, the byte stream data in the above file is converted into character stream data by constructing an input stream, using the InputStreamReader method and the file parsing encoding format in the above download configuration parameters, and saved in a preset buffer, and then a line of data in the buffer is read using BufferedReader.
[0133] S304: Store the read row data into a row list.
[0134] S305: Determine whether the number of row data in the row list reaches the number of row data batches indicated in the storage information of the file to be downloaded to the database.
[0135] If so, execute step S306; if not, return to execute step S303; if the number of row data in the row list does not reach the batch number, and there is no unread row data in the buffer, it means that the data in the file has been read, but the batch number in the row list has not been reached, then execute step S306.
[0136] S306: Obtain the column name corresponding to each column of data in a row of data in the row list, and use the data separator in the download configuration parameter to split the row of data to obtain each column of data in the row of data.
[0137] Specifically, the row of data is segmented using the data separator in the download configuration parameter to obtain an array including the data in each column of the row, so as to obtain the data in each column of the row of data, wherein each element in the array corresponds one-to-one to each column of the row.
[0138] S307. Assign each column of data and the corresponding column name to the value and key of the map object respectively to obtain a map object.
[0139] S308. Construct a bean object of the entity type object based on Java reflection technology.
[0140] Specifically, the bean object of the entity type object is constructed through getDeclaredConstructor().newInstance().
[0141] S309. Assign the attributes of the map object to the bean object.
[0142] Specifically, use BeanUtils.populate() to assign the properties of the map object to the bean object.
[0143] S310. Save the bean object to the entity object list corresponding to the data table.
[0144] S311 . When the number of entity objects stored in the entity object list reaches a preset value, the entity object list is stored in the data table.
[0145] S312, clear the above row list.
[0146] Return to step S303 until all the row data in the file are read and saved. When it is determined in step S305 that the number of row data in the row list does not reach the batch number, and there is no unread row data in the buffer, it means that the file has been read, and after executing the subsequent steps S306-S312, return to step S302 to continue to determine the next file to be downloaded to the database, so as to save the row data of the file into the data table.
[0147] The FTP data storage method provided by the present application uses a buffer to read FTP file data, and can realize the storage of subsequent file row data without downloading the file to the local computer, thereby improving the efficiency of FTP data storage, and uses a map object to store the data of each column and its column name in the row data. Based on Java reflection technology, the attributes of the map object are assigned to the bean object of the persistence class, thereby improving the efficiency of row data storage; at the same time, by using the settings of the row list and the entity object list, the row data is saved in batches to the data table, thereby reducing the number of communications with the database, alleviating the pressure on the database, and further reducing the number of disk reads and writes, thereby increasing the service life of the disk; finally, the method is based on the highly portable Java language, thereby improving the portability of the method and reducing the dependence on the development environment.
[0148] Figure 5 A schematic diagram of the structure of an FTP data storage device provided by this application. Figure 5 As shown, the device comprises:
[0149] The processing module 11 is used to establish a connection with the FTP server using the storage information of the file to be downloaded to the database, and determine a candidate file list from the FTP server;
[0150] A determination module 12, configured to determine a file to be downloaded to the database from the candidate file list according to the file identifier in the download configuration parameter;
[0151] The storage module 13 is used to construct an entity object of the persistent class according to the persistent class indicated in the download configuration parameters, and store the row data of the file in a data table corresponding to the persistent class in the database, wherein the entity object corresponds to a row in the data table.
[0152] In a possible implementation method, the storage module 13 is specifically used to obtain the column name corresponding to each column of data in the row data, and use the data delimiter in the download configuration parameter to split the row data to obtain each column of data in the row data; assign each column of data and the corresponding column name to the value and key of the intermediate object respectively to obtain the intermediate object; assign the attributes of the intermediate object to the entity object of the persistent class; store the entity object in the data table to store the row data in the database table.
[0153] In a possible implementation method, the download configuration parameters also include an entity type object of a preset class, and the persistent class is the entity type object of the preset class. The above-mentioned storage module 13 is specifically used to construct an entity object of the entity type object of the preset class based on Java reflection technology to obtain the entity object of the persistent class; and assign the attributes of the intermediate object to the entity object.
[0154] In a possible implementation, the storage module 13 is specifically configured to add the entity object to the entity object list of the data table; when the number of entity objects stored in the entity object list reaches a preset value, the entity object list is saved in the data table.
[0155] In a possible implementation, the storage module 13 is specifically used to read row data from the file in sequence and store the read row data in a row list; determine whether the number of row data in the row list reaches the number of row data batches indicated in the storage information of the file to be downloaded to the database;
[0156] If the number of row data in the row list reaches the batch number, or the number of row data in the row list does not reach the batch number and there is no unread row data in the file, then the row data in the row list is stored in the data table corresponding to the persistent class in the database by using the entity object of the persistent class indicated in the download configuration parameter.
[0157] In a possible implementation, the receiving module 14 is used to receive configuration information, where the configuration information is used to configure the storage information and / or the download configuration parameters.
[0158] In a possible implementation, the storage information further includes: the host name, user name, password, and file storage path of the FTP server; the download configuration parameters further include: file parsing encoding format and database interface object.
[0159] The functional modules of the above-mentioned FTP data storage device can cooperate to execute the FTP data storage method described in the above-mentioned embodiment, and its implementation principle and technical effect are similar, which will not be described in detail.
[0160] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 6 As shown, the electronic device 400 may include: at least one processor 401 and a memory 402. The electronic device may be a terminal, a server, a computer device, or other device with processing capabilities.
[0161] The memory 402 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions.
[0162] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0163] The processor 401 is used to execute the computer execution instructions stored in the memory 402 to implement the FTP data storage method described in the above method embodiment. The processor 401 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0164] The electronic device 400 may further include a communication interface 403, so as to communicate and interact with an external device through the communication interface 403. The external device may be, for example, a computer, a tablet, and the like.
[0165] In specific implementation, if the communication interface 403, the memory 402 and the processor 401 are implemented independently, the communication interface 403, the memory 402 and the processor 401 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.
[0166] Optionally, in a specific implementation, if the communication interface 403, the memory 402 and the processor 401 are integrated on a chip, the communication interface 403, the memory 402 and the processor 401 can communicate through an internal interface.
[0167] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used in the FTP data storage method in the above embodiment.
[0168] The present application also provides a computer program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of the electronic device 400 can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the electronic device 400 implements the methods provided in the various embodiments described above.
[0169] The present application also provides a chip having a computer program stored thereon, and when the computer program is executed by the chip, the methods provided in various implementation modes are implemented.
[0170] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0171] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A FTP data storage method, characterized in that: The method comprises: Using the storage information of the file to be downloaded to the database, a connection is established with the FTP server, and a candidate file list is determined from the FTP server; Determine the file to be downloaded to the database from the candidate file list according to the file identifier in the download configuration parameter; According to the persistent class indicated in the download configuration parameter, an entity object of the persistent class is constructed, and the row data of the file is stored in a data table corresponding to the persistent class in the database, wherein the entity object corresponds to a row in the data table.
2. The method according to claim 1, characterized in that The step of constructing an entity object of the persistent class according to the persistent class indicated in the download configuration parameter, and storing the row data of the file in a data table corresponding to the persistent class in the database includes: Obtaining a column name corresponding to each column of data in the row of data, and dividing the row of data using a data separator in the download configuration parameter to obtain each column of data in the row of data; Assign each column data and the corresponding column name to the value and key of the intermediate object respectively to obtain the intermediate object; Assigning the attributes of the intermediate object to the entity object of the persistent class; The entity object is stored in the data table, so as to store the row data in the database table.
3. The method according to claim 2, characterized in that The download configuration parameter also includes an entity type object of a preset class, the persistent class is the entity type object of the preset class, and the assigning of the attribute of the intermediate object to the entity object of the persistent class includes: Based on Java reflection technology, construct an entity object of the entity type object of the preset class to obtain the entity object of the persistent class; Assign the attributes of the intermediate object to the entity object.
4. The method according to claim 2, characterized in that: The storing the entity object in the database table comprises: Add the entity object to the entity object list of the data table; When the number of entity objects stored in the entity object list reaches a preset value, the entity object list is saved in the data table.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: constructing an entity object of the persistent class according to the persistent class indicated in the download configuration parameter, and storing the row data of the file in a data table corresponding to the persistent class in the database; Read row data from the file in sequence, and store the read row data in a row list; Determine whether the number of row data in the row list reaches the number of row data batches indicated in the storage information of the file to be downloaded to the database; If the number of row data in the row list reaches the batch number, or the number of row data in the row list does not reach the batch number and there is no unread row data in the file, then the row data in the row list is stored in the data table corresponding to the persistent class in the database by using the entity object of the persistent class indicated in the download configuration parameter.
6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Configuration information is received, where the configuration information is used to configure the storage information and / or the download configuration parameters.
7. The method according to claim 6, characterized in that The storage information also includes: the host name, user name, password, and file storage path of the FTP server; The download configuration parameters also include: file parsing encoding format and database interface object.
8. An FTP data storage device, characterized in that: The device comprises: A processing module, used to establish a connection with an FTP server using storage information of files to be downloaded to a database, and to determine a candidate file list from the FTP server; A determination module, used to determine the file to be downloaded to the database from the candidate file list according to the file identifier in the download configuration parameter; The storage module is used to construct an entity object of the persistent class according to the persistent class indicated in the download configuration parameters, and store the row data of the file in a data table corresponding to the persistent class in the database, wherein the entity object corresponds to a row in the data table.
9. An electronic device, characterized in that: The electronic device comprises: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the FTP data storage method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the FTP data storage method according to any one of claims 1 to 7 when executed by a processor.