Data import processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data import method and device, equipment, a storage medium and a program product, which are used for guaranteeing the consistency, stability and efficiency in a data import process so as to realize efficient and accurate data import. The data import method comprises the steps that in response to a pre-request from a client side, the pre-request is verified, and the pre-request comprises metadata of a data file to be imported; under the condition that the pre-request passes verification, a file import request sent by the client is received, and the file import request comprises a data file to be imported; dividing the data file into a plurality of data units based on the metadata; and sending the plurality of data units to message-oriented middleware, and consuming the message-oriented middleware so as to import the data units in the message-oriented middleware into a database.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] In the process of business system migration towards intelligence and onlineization, it is often necessary to import long-term accumulated historical data from the local to the database. How to achieve efficient and accurate data import becomes a key issue in the migration process. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data import method, device, equipment, storage medium and program product, which are used to ensure consistency, stability and efficiency in the data import process, thereby achieving efficient and accurate data import.

[0004] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a data import method, the method comprising:

[0006] In response to a pre-request from a client, verifying the pre-request, the pre-request including metadata of a data file to be imported;

[0007] If the pre-request passes the verification, receiving a file import request sent by the client, wherein the file import request includes a data file to be imported;

[0008] Based on the metadata, dividing the data file into a plurality of data units;

[0009] The multiple data units are sent to a message middleware, and the message middleware is consumed to import the data units in the message middleware into a database.

[0010] In a second aspect, an embodiment of the present application provides a data import method, the method comprising:

[0011] Obtaining a data file to be imported and metadata of the data file;

[0012] Sending a pre-request to a server, wherein the pre-request includes the metadata;

[0013] When the server passes the pre-request verification, a file import request is sent to the server, wherein the file import request includes the data file, and the file import request is used by the server to divide the data file into multiple data units based on the metadata, and import the multiple data units into the database through the message middleware.

[0014] In a third aspect, an embodiment of the present application provides a data import device, the device comprising:

[0015] A receiving module, configured to verify a pre-request in response to a pre-request from a client, wherein the pre-request includes metadata of a data file to be imported;

[0016] The receiving module is further configured to receive, when the pre-request passes verification, a file import request sent by the client, wherein the file import request includes a data file to be imported;

[0017] A division module, used for dividing the data file into a plurality of data units based on the metadata;

[0018] The import module is used to send the multiple data units to the message middleware and consume the message middleware to import the data units in the message middleware into the database.

[0019] In a fourth aspect, an embodiment of the present application provides a data import device, the device comprising:

[0020] An acquisition module, used for acquiring a data file to be imported and metadata of the data file;

[0021] A sending module, used for sending a pre-request to a server, wherein the pre-request includes the metadata;

[0022] The sending module is also used to send a file import request to the server when the server passes the pre-request verification. The file import request includes the data file. The file import request is used by the server to divide the data file into multiple data units based on the metadata and import them into the database.

[0023] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0024] processor;

[0025] a memory for storing instructions executable by the processor;

[0026] The processor is configured to execute the instructions to implement the data import method provided in the first aspect; or, the processor is configured to execute the instructions to implement the data import method provided in the second aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the data import method provided in the first aspect; or, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the data import method provided in the second aspect.

[0028] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the data import method provided in the first aspect or the data import method provided in the second aspect.

[0029] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0030] Before the client uploads the data file to be imported to the server, the client first sends a pre-request to the server. The pre-request contains the metadata of the data file. The server verifies the pre-request. If the pre-request passes the verification, the client encapsulates the local data file to be imported into a file import request and sends the file import request to the server for import processing. In this way, the server receives the file import request sent by the client after the pre-request verification passes. This can not only effectively prevent malicious and forged illegal upload behaviors, ensure that only verified data files can enter the server's import process, and improve the security barrier of data import, thereby ensuring the security and legality of cross-domain access to the server, but also filter illegal upload behaviors in advance, reduce unnecessary server bandwidth usage, and reduce server load. In this way, the security and efficiency of the file import process are improved. On this basis, since the metadata reflects the attributes of the data records in the data file, the server divides the data file into multiple data units based on the metadata, so that the import transaction for the data file is reasonably broken down into multiple smaller sub-transactions; further, the message middleware is introduced on the server side, and multiple data units are sent to the message middleware to queue for processing, and then the orderliness of the message middleware is utilized to consume the message middleware to import the data units in the message middleware into the database, thereby realizing the phased execution of multiple smaller sub-transactions, avoiding the database from bearing the pressure of writing all data at one time, not only eliminating the lock contention during concurrent writing of the database, but also significantly reducing the order of magnitude of database transactions, thereby effectively avoiding the performance bottleneck caused by large transactions, reducing the risk of downtime, thereby ensuring the consistency and stability of data import, and improving the efficiency and accuracy of data import. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 A schematic diagram of an implementation environment applicable to the data import method provided in an embodiment of the present application;

[0033] Figure 2 A flow chart of a data import method applied to a server provided in one embodiment of the present application;

[0034] Figure 3 A schematic diagram of a flow chart of a data import method applied to a client provided in one embodiment of the present application;

[0035] Figure 4 A flowchart of a data import method provided for one embodiment of the present application;

[0036] Figure 5 A schematic diagram of the structure of a data import device applied to a server provided in one embodiment of the present application;

[0037] Figure 6 A schematic diagram of the structure of a data import device applied to a client provided by an embodiment of the present application;

[0038] Figure 7 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0040] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0041] As mentioned above, during the migration of business systems, it is often necessary to import long-term accumulated historical data from the local to the database. Considering that during the data import process, the large size of data will block the operation of the database, thus affecting the efficiency and accuracy of data import.

[0042] In view of this, the embodiment of the present application proposes a data import method, before the client uploads the data file to be imported to the server, the client first sends a pre-request to the server, the pre-request contains the metadata of the data file, the server verifies the pre-request, and if the pre-request passes the verification, the client then encapsulates the local data file to be imported into a file import request, and sends the file import request to the server for import processing. In this way, the server receives the file import request sent by the client when the pre-request verification passes, which can not only effectively prevent malicious and forged illegal upload behaviors, ensure that only verified data files can enter the server's import process, improve the security barrier of data import, thereby ensuring the security and legality of cross-domain access to the server, but also filter illegal upload behaviors in advance, reduce unnecessary server bandwidth occupancy, and reduce server load. In this way, the security and efficiency of the file import process are improved. On this basis, since the metadata reflects the attributes of the data records in the data file, the server divides the data file into multiple data units based on the metadata, so that the import transaction for the data file is reasonably broken down into multiple smaller sub-transactions; further, the message middleware is introduced on the server side, and multiple data units are sent to the message middleware to queue for processing, and then the orderliness of the message middleware is utilized to consume the message middleware to import the data units in the message middleware into the database, thereby realizing the phased execution of multiple smaller sub-transactions, avoiding the database from bearing the pressure of writing all data at one time, not only eliminating the lock contention during concurrent writing of the database, but also significantly reducing the order of magnitude of database transactions, thereby effectively avoiding the performance bottleneck caused by large transactions, reducing the risk of downtime, thereby ensuring the consistency and stability of data import, and improving the efficiency and accuracy of data import.

[0043] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0044] In order to facilitate the understanding of the technical solution provided in the embodiment of the present application, firstly, Figure 1 , the implementation environment to which the technical solution provided in the embodiment of the present application is applicable is described. It should be understood that the technical solution provided in the embodiment of the present application is applicable to Figure 1 The implementation environment shown is only an exemplary description and should not be understood as a limitation on the implementation environment to which the technical solution is applicable.

[0045] like Figure 1As shown, an implementation environment applicable to the technical solution provided in the embodiment of the present application includes: a client 1, a server 2 and a database 3.

[0046] The client 1 may be a terminal device that provides a data import function, such as but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a vehicle terminal, an aircraft, etc. Alternatively, the client 1 may also be software running in a terminal device. The data import function provided by the client 1 allows business personnel to select the data file to be imported.

[0047] Server 2 is a device with data processing and importing functions, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Server 2 can provide import services for the data files of client 1, that is, import the data files requested to be imported by client 1 into database 3.

[0048] Please refer to Figure 2 , is a flow chart of a data import method provided by an embodiment of the present application, the method is applied to the above-mentioned server, and the method comprises the following steps:

[0049] S202: In response to the pre-request from the client, verify the pre-request.

[0050] The pre-request includes metadata of the data file to be imported. Exemplarily, the request header of the pre-request includes metadata of the data file.

[0051] In one implementation, the above S202 includes the following steps: parsing the pre-request to obtain a request header of the pre-request; and verifying the request header of the pre-request based on metadata of the data file.

[0052] Specifically, the request header of the pre-request is obtained by encapsulating the metadata based on RESTFul by the client. The server verifies the pre-request, including: performing integrity check on the request header; if the request header passes the integrity check, obtaining the metadata of the data file requested to be imported by the client from the request header; and performing legality check on the metadata.

[0053] Exemplarily, the pre-request is an OPTIONS request. The server first checks whether the request header of the pre-request contains the Cst-File, Cst-Excel-HD, and Cst-Other fields. These fields are customized by the client and are used to describe the metadata of the data file. Among them, the Cst-File field is used to describe metadata such as the file type, maximum number of bytes, maximum number of rows, and maximum number of columns of the data file, and Cst-Excel-HD is used to describe the header information of the data file, such as the field name and field type of the fields contained in the header. Cst-Other is used to describe other metadata of the data file, such as the person who modified the data file, the modification timestamp, etc. If the request header contains the above fields, it is determined that the request header passes the integrity check.

[0054] Then, the metadata of the data file is obtained from the above fields, and the content of the metadata is verified to see whether it meets the expected requirements, etc., thereby obtaining the metadata legitimacy verification result. If the metadata passes the legitimacy verification, it is determined that the pre-request passes the verification, and then the file import request sent by the client is received. If the pre-request does not pass the verification, a second error prompt message is returned to the client. The second error prompt message is used to prompt the client to modify the metadata in the request header.

[0055] For example, the content of a pre-request header is shown below:

[0056] Cst-File:file_type=excel&file_max_size=5000000&file_max_row=-1&file_max_col=-1

[0057] Cst-Excel-HD: Attendance date, user id, user name, attendance type

[0058] Cst-Other:operator=1×tamp=16919192399123

[0059] Since the above request header contains the Cst-File, Cst-Excel-HD, and Cst-Other fields, and the metadata obtained from these fields meets the expected requirements, it is determined that the pre-request passes the verification.

[0060] In the above implementation, the server first verifies the pre-request sent by the client, and then receives the file import request sent by the client if the pre-request verification passes. This can not only effectively prevent malicious and forged illegal uploads, but also ensure that only verified data files can enter the server's import process, thereby improving the security barrier of data import, thereby ensuring the security and legality of cross-domain access to the server, and can also filter illegal uploads in advance, reduce unnecessary server bandwidth usage, and reduce server load. In this way, the security and efficiency of the file import process are improved.

[0061] S204: When the pre-request passes the verification, receive a file import request sent by the client.

[0062] The file import request includes the data file to be imported. Optionally, the file import request also includes metadata of the data file. Exemplarily, the file import request includes a request header and a request body, the request header includes the metadata of the data file, and the request body includes the data file. The file import request is used to request to import the data file into the database based on the metadata. The type of the file import request is set by the client according to actual needs, such as a POST request, etc., and this embodiment of the present application does not limit this.

[0063] Metadata refers to data used to describe the properties of a data file, and may include, but is not limited to, at least one of the following data: the file type of the data file, the maximum number of bytes of the data file, the maximum number of rows of the data file, the field names and field types of the fields contained in the data records in the data file, etc. The metadata may be encapsulated by the client in the request header of the file import request according to the RESTFul specification.

[0064] S206: Divide the data file into multiple data units based on the metadata.

[0065] Since metadata describes the properties of data files, partitioning data files based on metadata can ensure the rationality of the partitioning results, thereby ensuring that the import transaction of the data file is reasonably broken down into smaller sub-transactions.

[0066] In the above S206, the data file may be divided into a plurality of data units in various appropriate ways, which may be selected according to actual needs.

[0067] In one embodiment, the above S206 includes the following steps: S261, based on metadata, determining the association relationship between data records in the data file; S262, based on the association relationship, grouping the data records in the data file to obtain multiple groups of data records; S263, converting the multiple groups of data records into multiple data units.

[0068] Exemplarily, in the above S261, the fields contained in the data records in the data file are determined based on the metadata, and the data records with the same values ​​in the specified fields are determined as associated data records, or the data records with values ​​in the specified fields that meet a preset value range are determined as associated data records, etc.

[0069] In the above S262, the associated data records are divided into the same group.

[0070] For example, the data file is an agent schedule, in which each data record includes the values ​​of multiple fields such as attendance date, user ID, user name, and attendance type, and each data record represents the schedule data of an agent. The designated fields include attendance date and user ID. In this case, data records with attendance dates in the same month and the same user ID are determined to be associated data records, and these data records are divided into the same group, thereby obtaining multiple groups of data records, which are then converted into multiple data units. In this way, the agent schedule is reasonably divided into multiple groups of data records, each group of data records includes the schedule data of the same agent in each month.

[0071] In the above S263, for each group of data records, the group of data records is converted into a data format in the database and serialized into a json string to obtain a data unit corresponding to the group of data records.

[0072] For example, a set of data records includes the following data record:

[0073] [{"Agent name":"user1","Agent id":"122","Attendance date":"2024-03-28","Attendance type":"All day"}]

[0074] According to the data format and field name of the database, the data record is serialized into a JSON string for easy storage and transmission, and the following result is obtained:

[0075] [{"work_date":"2024-03-28","user_id":"122","user_name":"user1","attendance_type":"ALL_DAY"}]

[0076] In the above implementation, based on the association relationship between data records in the data file, the data file is divided into multiple data units, which can ensure that related data records are imported into the database together, improve the rationality of the division, and facilitate access to related data records in the database in subsequent business processes, reduce the frequency of access to the database, and improve the efficiency of access to the database.

[0077] The embodiments of the present application illustrate some implementation methods for dividing data files. Of course, it should be understood that the division of data files can also be implemented in other ways, and the embodiments of the present application are not limited to this. For example, in some other implementation methods, the fields contained in the data records in the data file are determined based on metadata, and then the data records in the data file are randomly divided into multiple groups in units of data records (i.e., in units of lines); then, the multiple groups of data records are converted into multiple data units.

[0078] In the above S206, a suitable time may be selected to divide the data file into a plurality of data units.

[0079] In one implementation, after receiving the file import request, the data file may be directly divided into a plurality of data units.

[0080] In another embodiment, the data records in the data file are first checked one by one, and the data records that pass the check are divided into multiple data units. Specifically, based on the metadata, the data records in the data file are read one by one, and the currently read data records are checked; the data records in the data file that pass the check are written to the memory of the server; based on the metadata, the data records stored in the memory are divided into multiple data units.

[0081] Furthermore, if the currently read data record fails to pass the verification, the data record in the data file and the currently read data record are stopped, and a first error prompt message is returned to the client. The first error prompt message is used to prompt the client to modify the data record that fails the verification and resend it to the server.

[0082] Exemplarily, if the type of the data file is determined to be excel based on the metadata, the data file is written into ZipInputStream, and according to the characteristics of the excel file, the bytes in the data file are read one by one and the binary is decompressed; then, a data parsing tool (such as Apache POI, etc.) is used to parse the decompressed binary to obtain the header information of the data file, and the data records in the data file are located using the header information, and the data records are read one by one. Each time a data record is read, the data record is verified, such as whether the content of the data record is legal, whether it conforms to the business logic, etc.; if the data record passes the verification, the data record is temporarily stored in the memory of the server, and the next data record is read; the above process is repeated until the data records in the data file are read or the read data record fails the verification; finally, the data records in the memory are divided into multiple data units.

[0083] In the above implementation, data records in the data file are read and verified one by one. Once it is found that they do not meet the requirements, the reading operation on the data file is stopped, and a prompt message is returned for the client to modify it. For the data records that pass the verification, they are first temporarily stored in the memory and then uniformly divided into multiple data units. This can timely and comprehensively find the non-compliant data records in the data file, which can not only avoid invalid or erroneous data records from occupying network bandwidth and reduce the bandwidth peak pressure caused by the one-time loading of too large data files, but also ensure that the data records imported into the database can be used for subsequent business processing, thereby ensuring the reliability and stability of subsequent business.

[0084] S208, sending the multiple data units to the message middleware, and consuming the message middleware to import the data units in the message middleware into the database.

[0085] The message middleware can use various message queues commonly used in the field, such as distributed message queues such as Kafka. The message middleware includes multiple topics, each of which corresponds to different businesses. According to the different businesses to which the data files belong, multiple data units are sent to the corresponding topics in sequence, and then the corresponding topics are consumed, and the data units are taken out in sequence and imported into the database.

[0086] In this way, the orderliness of the message middleware can be used to reasonably decompose the import transaction for the data file into sub-transactions for multiple data units, and these sub-transactions are executed in order. Since the amount of data in each data unit is significantly reduced compared to the data file, that is, the amount of data written in each sub-transaction is significantly reduced compared to the original import transaction, it can avoid the database from bearing the pressure of writing all the data at once, which not only eliminates the lock competition during concurrent writing of the database, but also significantly reduces the order of magnitude of database transactions, thereby effectively avoiding the performance bottleneck caused by large transactions and reducing the risk of downtime, thereby ensuring the consistency and stability of data import and improving the efficiency and accuracy of data import.

[0087] It is worth noting that if a subtransaction for each data unit is rolled back, no commit instruction will be sent to the message middleware, that is, the next data unit will not be sent to the message middleware. Instead, the next data unit will be sent to the message middleware after the data unit corresponding to the currently rolled back subtransaction is successfully stored in the database. In this way, even if an abnormal situation occurs during the import process, the data integrity can be ensured through the transaction rollback mechanism, which enhances the stability of the import process and data reliability.

[0088] After the data is imported, the server provides real-time query and in-depth analysis of these data records. As an example, the server can generate various reports, statistical graphics and analysis results in response to the instructions entered by the administrator, which is not only conducive to business personnel to review past business processing situations, but also conducive to optimizing business resource allocation and accurately evaluating business processing results, and also provides detailed and reliable data support for internal business testing.

[0089] Please refer to Figure 3 , is a flow chart of a data import method provided by an embodiment of the present application, the method is applied to a client, and the method comprises the following steps:

[0090] S302: Acquire the data file to be imported and the metadata of the data file.

[0091] The client may obtain the data file to be imported in various ways, which may be selected according to actual needs, and the embodiments of the present application do not limit this.

[0092] In one embodiment, the data file to be imported is obtained by: displaying a file selection interface, wherein the file selection interface includes at least one candidate data file; in response to the selection operation of at least one candidate data file, performing a normative check on the selected candidate data file; if the selected candidate data file passes the check, the selected candidate data file is used as the data file to be imported. The normative check may include, for example, but is not limited to, at least one of the following check operations: checking whether the extension of the candidate data file is in accordance with expectations, checking whether the size of the candidate data file does not exceed a preset size threshold, checking whether the file structure of the candidate data file conforms to the allowed file structure, etc.

[0093] Exemplarily, the client stores rule data pre-set by the administrator for compliance verification, including, for example, but not limited to, the file size, file type, maximum number of bytes, maximum number of rows, and maximum number of columns allowed for import. In response to the file import function being triggered, the client displays a file selection interface, which includes candidate data files 1 to 5. In response to the selection operation of candidate data file 1, a compliance verification is performed on candidate data file 1 based on the pre-set rule data. If candidate data file 1 passes the compliance verification, candidate data file 1 is used as the data file to be imported; if candidate data file 1 fails the compliance verification, a prompt message is output to prompt the user to select other candidate data files.

[0094] The above implementation is applicable to the case where the data files already existing in the client are imported into the database. This method is simple to implement. The user only needs to select in the file selection interface to realize the one-click import of the data file into the database. In addition, in this implementation, following the principle of front-end and back-end separation, the client at the front end first performs a preliminary check on the selected candidate data files.

[0095] In another embodiment, the data file to be imported is obtained in the following manner: displaying a file editing interface, wherein the file editing interface is used to input data fields and value ranges of the data fields contained in the data file; in response to input operations in the file editing interface, obtaining multiple business objects from a business system based on the data fields; based on the value conditions of the data fields, assigning values ​​of the multiple business objects in the data fields to obtain multiple data records; and generating the data file to be imported based on the multiple data records.

[0096] For example, the file editing interface provides an online scheduling function. The data fields entered by the administrator through the file editing interface include attendance date, user ID, user name, and attendance type. The attributes and value ranges of the data fields are shown in the following table.

[0097]

[0098] In this case, multiple business objects are obtained from the business system, each of which represents a seat, including the user ID and user name of the seat. Then, based on the value range of the attendance date, the value of the attendance date is assigned to each business object, and based on the value range of the attendance type, the value of the attendance type is assigned to each business object, and multiple data records are obtained. Each data record corresponds to an agent, including the values ​​of the four fields of the agent's attendance date, user ID, user name, and attendance type. Finally, these data records are filled into the preset table structure template to obtain a data table as the data file to be imported.

[0099] In the above scheduling scenario, in response to the publishing operation on the file editing interface, each agent can be automatically notified of the scheduling status of each agent through email, SMS, internal message, etc. In addition, in response to the export operation on the file editing interface, the data file can be exported to a file in a specified format, such as an Excel file, which is not only convenient for offline viewing, printing and distribution, but also convenient for sharing scheduling information with other non-system users, or for backup and archiving.

[0100] The above implementation method provides an intuitive and easy-to-use file editing interface. Relevant personnel only need to input the data fields contained in the data file and the value range of the data fields to realize automatic and flexible editing of the data file. Compared with the traditional manual maintenance of data files, it can effectively avoid human errors and is extremely convenient for data retrieval, statistics and analysis, especially when faced with a large number of data records, the processing efficiency can be greatly improved.

[0101] The metadata of the data file can be obtained by parsing the data file by the client, or can be configured by relevant personnel through the front-end configuration component provided by the client.

[0102] S304: Send a pre-request to the server.

[0103] The pre-request contains the metadata of the data file to be imported. In order to improve the security barrier of data import, thereby ensuring the security and legality of the client's cross-domain access to the server, and assisting the server to filter illegal uploads in advance, reduce unnecessary server bandwidth usage, and reduce server load, you can first send a pre-request to the server, and then send a file import request to the server after the server successfully verifies the pre-request.

[0104] S306: When the server passes the pre-request verification, a file import request is sent to the server.

[0105] The file import request includes the data file to be imported. Exemplarily, the file import request includes a request header and a request body, the request header includes metadata of the data file, and the request body includes the data file.

[0106] The file import request is used by the server to divide the data file into multiple data units based on metadata and import them into the database.

[0107] In one embodiment, the client generates a custom request header based on metadata, generates a request body based on the data file, generates a file import request based on the request header and the request body, such as a POST request, and sends the file import request to the server.

[0108] In another embodiment, in the above S304, the client generates a request header based on the metadata, and sends a pre-request including the request header to the server, and the pre-request is used for the server to verify based on the request header. Accordingly, in the above S306, if the server returns a response message indicating that the pre-request has passed the verification, a file import request is sent to the server based on the request header and the data file.

[0109] Exemplarily, the metadata of a data file is as follows:

[0110] File type: excel;

[0111] Maximum number of bytes of a file: 5000000;

[0112] Maximum number of lines in a file: -1;

[0113] Maximum number of columns in a file: -1;

[0114] The data fields included in the file header are: attendance date, user ID, user name, and attendance type;

[0115] The person who modified the data file: 1;

[0116] The modification timestamp of the data file: 16919192399123.

[0117] Based on the above metadata, the client assigns values ​​to the custom request header fields, and then encapsulates the values ​​of the request header fields into a request header according to the RESTFul specification, as shown below:

[0118] Cst-File:file_type=excel&file_max_size=5000000&file_max_row=-1&file_max_col=-1

[0119] Cst-Excel-HD: Attendance date, user id, user name, attendance type

[0120] Cst-Other:operator=1×tamp=16919192399123

[0121] The Cst-File field is used to describe metadata such as the file type, maximum file byte count, maximum file row count, and maximum file column count of the data file. Cst-Excel-HD is used to describe the header information of the data file, such as the field name and field type of the fields contained in the header. Cst-Other is used to describe other metadata of the data file, such as the person who modified the data file and the modification timestamp. If the request header contains the above fields, it is determined that the request header passes the integrity check.

[0122] Then, the client encrypts the request header based on an encryption algorithm, such as the AES encryption algorithm, to generate an OPTIONS pre-request, and sends the OPTIONS pre-request to the server for verification; if the OPTIONS pre-request passes the verification, a POST request is generated using the request header, and a POST-type file import request is sent to the server to request that the data file be imported into the database.

[0123] One or more embodiments of the present application provide a data import method, in which before the client uploads the data file to be imported to the server, the client first sends a pre-request to the server, the pre-request contains the metadata of the data file, the server verifies the pre-request, and if the pre-request passes the verification, the client then encapsulates the local data file to be imported into a file import request, and sends the file import request to the server for import processing. In this way, the server receives the file import request sent by the client when the pre-request verification passes, which can not only effectively prevent malicious and forged illegal upload behaviors, ensure that only verified data files can enter the server's import process, improve the security barrier of data import, thereby ensuring the security and legality of cross-domain access to the server, but also filter illegal upload behaviors in advance, reduce unnecessary server bandwidth usage, and reduce server load. In this way, the security and efficiency of the file import process are improved. On this basis, since the metadata reflects the attributes of the data records in the data file, the server divides the data file into multiple data units based on the metadata, so that the import transaction for the data file is reasonably broken down into multiple smaller sub-transactions; further, the message middleware is introduced on the server side, and multiple data units are sent to the message middleware to queue for processing, and then the orderliness of the message middleware is utilized to consume the message middleware to import the data units in the message middleware into the database, thereby realizing the phased execution of multiple smaller sub-transactions, avoiding the database from bearing the pressure of writing all data at one time, not only eliminating the lock contention during concurrent writing of the database, but also significantly reducing the order of magnitude of database transactions, thereby effectively avoiding the performance bottleneck caused by large transactions, reducing the risk of downtime, thereby ensuring the consistency and stability of data import, and improving the efficiency and accuracy of data import.

[0124] In order to facilitate the understanding of the interaction process between the client and the server in this application, Figure 4 and Figure 5 , the data import method provided in the embodiment of the present application is described.

[0125] Please refer to Figure 4 , is a flow chart of a data import method provided by another embodiment of the present application, which describes a specific implementation method of interaction between a client and a server. Figure 4 As shown, the method comprises the following steps:

[0126] S401: The client obtains the data file to be imported and the metadata of the data file.

[0127] S402: The client generates a request header based on the metadata.

[0128] S403, the client sends a pre-request including a request header to the server.

[0129] S404: The server verifies the pre-request based on the request header of the pre-request.

[0130] S405: If the pre-request fails the verification, the client prompts the user to make changes.

[0131] S406: If the pre-request passes the verification, the client sends a file import request to the server based on the request header and the data file.

[0132] S407: The server parses the file import request to obtain the data file and metadata of the data file.

[0133] S408: The server reads data records in the data file one by one based on the metadata, and verifies the data records currently read.

[0134] S409: If the currently read data record fails to pass the verification, the server stops reading the data file and returns a first error prompt message to the client.

[0135] S410: If the currently read data record passes the verification, the server writes the data record into the memory.

[0136] S411, after the data file is read, the server divides the data records in the memory into multiple data units based on the metadata.

[0137] S412, the server sends the multiple data units to the message middleware in sequence.

[0138] S413: The server consumes the message middleware to import the data units in the message middleware into the database.

[0139] If the data unit in the message middleware fails to be imported, the above S413 is executed again.

[0140] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0141] Based on the same inventive concept, the present application embodiment also provides a data import device. Figure 5, is a structural diagram of a data import device 500 provided in an embodiment of the present application, the device 500 is applied to a server, and the device 500 includes: a receiving module 510, a dividing module 520 and an importing module 530. The receiving module 510 is used to respond to a pre-request from a client and verify the pre-request, wherein the pre-request includes metadata of a data file to be imported.

[0142] The receiving module 510 is further configured to receive a file import request sent by the client when the pre-request passes the verification, wherein the file import request includes a data file to be imported.

[0143] The division module 520 is used to divide the data file into multiple data units based on the metadata.

[0144] The import module 530 is used to send the multiple data units to the message middleware, and consume the message middleware to import the data units in the message middleware into the database.

[0145] In another embodiment, the partitioning module is used to:

[0146] Based on the metadata, determining association relationships between data records in the data file;

[0147] Based on the association relationship, grouping the data records in the data file to obtain multiple groups of data records;

[0148] The plurality of groups of data records are converted into the plurality of data units.

[0149] In another embodiment, the partitioning module is used to:

[0150] Based on the metadata, read the data records in the data file one by one, and verify the data records currently read;

[0151] Writing the verified data records in the data file into the memory of the server;

[0152] Based on the metadata, the data records stored in the memory are divided into a plurality of data units.

[0153] In another embodiment, the partitioning module is further configured to:

[0154] If the data record currently being read fails the verification, then stop reading the data record in the data file;

[0155] Based on the data record currently read, first error prompt information is returned to the client.

[0156] In another embodiment, the receiving module is used to:

[0157] Parsing the pre-request to obtain a request header of the pre-request, wherein the request header includes metadata of the data file;

[0158] Based on the metadata, the request header of the pre-request is verified.

[0159] In another embodiment, the receiving module performs the following steps when verifying the request header of the pre-request based on the metadata:

[0160] Performing integrity check on the request header;

[0161] If the request header passes the integrity check, obtaining metadata of the data file from the request header;

[0162] The metadata is verified for legality.

[0163] Obviously, the data import device 500 provided in the embodiment of the present application can be used as the above Figure 2 The execution subject of the data import method shown in the figure can realize the data import device in Figure 2 The functions realized are as follows. Since the principles are the same, they will not be described in detail.

[0164] Based on the same inventive concept, the present application embodiment also provides a data import device. Figure 6 , is a structural diagram of a data import device 600 provided in an embodiment of the present application. The device 600 is applied to a client, and the device 600 includes: an acquisition module 610 and a sending module 620.

[0165] The acquisition module 610 is used to acquire the data file to be imported and the metadata of the data file.

[0166] The sending module 620 is used to send a pre-request to the server.

[0167] The pre-request includes the metadata.

[0168] The sending module 620 is also used to send a file import request to the server when the server passes the pre-request verification. The file import request includes the data file. The file import request is used by the server to divide the data file into multiple data units based on the metadata and import them into the database.

[0169] In another embodiment, the sending module is used to:

[0170] Based on the metadata, generate a request header;

[0171] Sending a pre-request including the request header to the server;

[0172] If the server returns a response message indicating that the pre-request has passed the verification, a file import request is sent to the server based on the request header and the data file.

[0173] In another embodiment, the data file to be imported is obtained by:

[0174] Displaying a file selection interface, wherein the file selection interface includes at least one candidate data file;

[0175] In response to the selection operation of the at least one candidate data file, performing a normative check on the selected candidate data file;

[0176] If the selected candidate data file passes the verification, the selected candidate data file is used as the data file to be imported.

[0177] In another embodiment, the data file to be imported is obtained by:

[0178] Displaying a file editing interface, wherein the file editing interface is used to configure data fields contained in the data file and value ranges of the data fields;

[0179] In response to an input operation in the file editing interface, acquiring a plurality of business objects from a business system based on the data field;

[0180] Based on the value condition of the data field, assigning values ​​of the multiple business objects in the data field to obtain multiple data records;

[0181] The data file to be imported is generated based on the multiple data records.

[0182] Obviously, the data import device 600 provided in the embodiment of the present application can be used as the above Figure 3 The execution subject of the data import method shown in the figure can realize the data import device in Figure 3 The functions realized are as follows. Since the principles are the same, they will not be described in detail.

[0183] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 7At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0184] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0185] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0186] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data import device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0187] In response to a pre-request from a client, verifying the pre-request, the pre-request including metadata of a data file to be imported;

[0188] If the pre-request passes the verification, receiving a file import request sent by the client, wherein the file import request includes a data file to be imported;

[0189] Based on the metadata, dividing the data file into a plurality of data units;

[0190] The multiple data units are sent to a message middleware, and the message middleware is consumed to import the data units in the message middleware into a database.

[0191] Alternatively, the processor executes the program stored in the memory and is specifically configured to perform the following operations:

[0192] Obtaining a data file to be imported and metadata of the data file;

[0193] Sending a pre-request to a server, wherein the pre-request includes the metadata;

[0194] When the server passes the pre-request verification, a file import request is sent to the server, wherein the file import request includes the data file, and the file import request is used by the server to divide the data file into multiple data units based on the metadata, and import the multiple data units into the database through the message middleware.

[0195] The above application Figure 2 or Figure 3 The method performed by the data import device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0196] The electronic device may also perform Figure 2 The method and the data import device are implemented in FIG. Figure 2 Alternatively, the electronic device may also perform the functions of the embodiments shown. Figure 3 The method is to realize the data import device in Figure 3The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0197] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0198] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 2 The method of the embodiment shown is specifically used to perform the following operations:

[0199] In response to a pre-request from a client, verifying the pre-request, the pre-request including metadata of a data file to be imported;

[0200] If the pre-request passes the verification, receiving a file import request sent by the client, wherein the file import request includes a data file to be imported;

[0201] Based on the metadata, dividing the data file into a plurality of data units;

[0202] The multiple data units are sent to a message middleware, and the message middleware is consumed to import the data units in the message middleware into a database.

[0203] Alternatively, when the instruction is executed by a portable electronic device including a plurality of application programs, the portable electronic device can execute Figure 3 The method of the embodiment shown is specifically used to perform the following operations:

[0204] Obtaining a data file to be imported and metadata of the data file;

[0205] Sending a pre-request to a server, wherein the pre-request includes the metadata;

[0206] When the server passes the pre-request verification, a file import request is sent to the server, wherein the file import request includes the data file, and the file import request is used by the server to divide the data file into multiple data units based on the metadata, and import the multiple data units into the database through the message middleware.

[0207] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute a data import method as provided in the embodiment of the present application or part or all of the steps in the data import method as provided in the embodiment of the present application.

[0208] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0209] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0210] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0211] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0212] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A data import method, characterized in that: The method comprises: In response to a pre-request from a client, verifying the pre-request, the pre-request including metadata of a data file to be imported; If the pre-request passes the verification, receiving a file import request sent by the client, wherein the file import request includes a data file to be imported; Based on the metadata, dividing the data file into a plurality of data units; The multiple data units are sent to a message middleware, the message middleware is consumed, and the data units in the message middleware are imported into a database.

2. The method according to claim 1, characterized in that The step of dividing the data file into a plurality of data units based on the metadata includes: Based on the metadata, determining association relationships between data records in the data file; Based on the association relationship, grouping the data records in the data file to obtain multiple groups of data records; The plurality of groups of data records are converted into the plurality of data units.

3. The method according to claim 1, characterized in that The step of dividing the data file into a plurality of data units based on the metadata includes: Based on the metadata, read the data records in the data file one by one, and verify the data records currently read; Writing the verified data records in the data file into the memory of the server; Based on the metadata, the data records stored in the memory are divided into a plurality of data units.

4. The method according to claim 3, characterized in that After the currently read data record is verified, the method further includes: If the data record currently being read fails the verification, then stop reading the data record in the data file; Based on the data record currently read, first error prompt information is returned to the client.

5. The method according to claim 1, characterized in that The verifying the pre-request includes: Parsing the pre-request to obtain a request header of the pre-request, wherein the request header includes metadata of the data file; Based on the metadata, the request header of the pre-request is verified.

6. The method according to claim 5, characterized in that The verifying the request header of the pre-request based on the metadata includes: Performing integrity check on the request header; If the request header passes the integrity check, obtaining metadata of the data file from the request header; The metadata is verified for legality.

7. A data import method, characterized in that: The method comprises: Obtaining a data file to be imported and metadata of the data file; Sending a pre-request to a server, wherein the pre-request includes the metadata; When the server passes the pre-request verification, a file import request is sent to the server, wherein the file import request includes the data file, and the file import request is used by the server to divide the data file into multiple data units based on the metadata, and import the multiple data units into the database through the message middleware.

8. The method according to claim 7, characterized in that The sending of a pre-request to the server comprises: Based on the metadata, generate a request header; Sending a pre-request including the request header to the server; When the server passes the pre-request verification, sending the file import request to the server includes: If the server returns a response message indicating that the pre-request has passed the verification, a file import request is sent to the server based on the request header and the data file.

9. The method according to claim 7, characterized in that: The data file to be imported is obtained in the following manner: Displaying a file selection interface, wherein the file selection interface includes at least one candidate data file; In response to the selection operation of the at least one candidate data file, performing a normative check on the selected candidate data file; If the selected candidate data file passes the verification, the selected candidate data file is used as the data file to be imported.

10. The method according to claim 7, characterized in that The data file to be imported is obtained in the following manner: Displaying a file editing interface, wherein the file editing interface is used to configure data fields contained in the data file and value ranges of the data fields; In response to an input operation in the file editing interface, acquiring a plurality of business objects from a business system based on the data field; Based on the value condition of the data field, assigning values ​​of the multiple business objects in the data field to obtain multiple data records; The data file to be imported is generated based on the multiple data records.

11. A data import device, characterized in that: The device comprises: A receiving module, configured to verify a pre-request in response to a pre-request from a client, wherein the pre-request includes metadata of a data file to be imported; The receiving module is further configured to receive, when the pre-request passes verification, a file import request sent by the client, wherein the file import request includes a data file to be imported; A division module, used for dividing the data file into a plurality of data units based on the metadata; The import module is used to send the multiple data units to the message middleware and consume the message middleware to import the data units in the message middleware into the database.

12. A data import device, characterized in that: The device comprises: An acquisition module, used for acquiring a data file to be imported and metadata of the data file; A sending module, used for sending a pre-request to a server, wherein the pre-request includes the metadata; The sending module is also used to send a file import request to the server when the server passes the pre-request verification. The file import request includes the data file. The file import request is used by the server to divide the data file into multiple data units based on the metadata and import them into the database.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data import method as described in any one of claims 1 to 6; or, the processor is configured to execute the instructions to implement the data import method as described in any one of claims 7 to 10.

14. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the data import method as described in any one of claims 1 to 6; or, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the data import method as described in any one of claims 7 to 10.

15. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the data import method according to any one of claims 1 to 6 or the data import method according to any one of claims 7 to 10.

Images