Data processing method and device, equipment and medium

Abstract

The invention discloses a data processing method and device, equipment and a medium, and the method comprises the steps: when a monitored target data table name in a write operation request sent by an application server to a database is matched with a data table name in a set configuration file, and the target data table is in a table locking state, the target data table is in a table locking state; if yes, generating a temporary table corresponding to the write operation request based on an operation type in the write operation request and an SQL statement in the write operation request; when it is monitored that the target data table is not in the table locking state, determining each temporary table corresponding to the target data table; and based on the creation time of each temporary table corresponding to the target data table, performing data processing on the target data table by using each temporary table in sequence according to a set sequence. According to the method, the performance consumption can be reduced in a large-batch data processing process, and the data processing efficiency and the user experience are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular, to a data processing method, apparatus, device and medium. Background Art

[0002] In the current system, many systems call each other intertwinedly, and large batches of data are often processed. When executing a batch operation request to process data in a data table in the database, the data table in the database is usually locked (protection mechanism), allowing only the current operation request to be executed, and the lock will be released only after the batch operation request is completed.

[0003] Currently, during the table lock period, the user's write operation request to the data table is stored in the transaction table, and the write operation request in the transaction table is continuously retried using a scheduled task until the execution is successful or the maximum retry threshold is reached, which will cause great pressure on the database and cause high performance consumption. In addition, if the number of retries of a write operation request in the transaction table exceeds the maximum retry threshold, the write operation request will no longer be executed, affecting data processing efficiency, thereby making the user experience poor. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and medium for reducing performance consumption, improving data processing efficiency and user experience during large-scale data processing.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] When the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is in a locked table state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the structured query language SQL statement in the write operation request;

[0007] When it is monitored that the target data table is not in a locked state, determining each temporary table corresponding to the target data table;

[0008] Based on the creation time of each temporary table corresponding to the target data table, each temporary table is used in turn to process data on the target data table in a set order.

[0009] In an embodiment of the present application, when the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is in a locked table state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the SQL statement in the write operation request, thereby realizing that during the period when the target data table is locked, a temporary table corresponding to each write operation request to the target data table is generated. In addition, when the present application monitors that the target data table is not in a locked table state, it is determined that the lock of the target data table has been released, thereby determining each temporary table corresponding to the target data table, that is, during the period when the target data table is locked, the temporary table corresponding to each write operation request to the target data table is generated. Based on the creation time of each temporary table corresponding to the target data table, the present application sequentially uses each temporary table to process data on the target data table in accordance with the set order, without using a timed task to continuously retry the write operation request, avoiding the situation where the write operation request is no longer executed because the number of retries of the write operation request exceeds the maximum retry number threshold, thereby reducing the pressure on the database, thereby reducing performance consumption, and improving data processing efficiency. Therefore, the present application can reduce performance consumption during large-scale data processing, improve data processing efficiency and user experience.

[0010] In a possible design, generating a temporary table corresponding to the write operation request based on the operation type in the write operation request and the SQL statement in the write operation request includes:

[0011] If the operation type in the write operation request is an insert type, the target data table is copied to obtain a temporary table to be processed, and the SQL statement in the write operation request is used to process data in the temporary table to be processed, and the processed temporary table to be processed is used as the temporary table; or

[0012] If the operation type in the write operation request is an update type or a delete type, the temporary table is generated based on the SQL statement in the write operation request.

[0013] In the embodiment of the present application, since there are many insert-type write operation requests in the batch data processing scenario, in order to reduce the pressure on the database, that is, the database frequently executes the SQL statement in the insert-type write operation request, the present application copies the target data table to obtain a temporary table to be processed, uses the SQL statement in the write operation request to process the data of the temporary table to be processed, and uses the processed temporary table to be processed as a temporary table, thereby reducing performance consumption and improving data processing efficiency. However, there are fewer update-type or delete-type write operation requests, so based on the SQL statement in the write operation request, the generation of the temporary table corresponding to the write operation request is realized.

[0014] In a possible design, using each temporary table to perform data processing on the target data table includes:

[0015] If the operation type in the write operation request corresponding to the temporary table is an insert type, copy the data in the temporary table and use the copied data to update the data in the target data table; or

[0016] If the operation type in the write operation request corresponding to the temporary table is an update type or a delete type, the SQL statement in the temporary table is used to perform data processing on the target data table.

[0017] In the embodiment of the present application, since there are many insert-type write operation requests in the batch data processing scenario, the data in the temporary table is copied and the copied data is used to update the data in the target data table, which reduces the time it takes for the database to execute the SQL statement in the write operation request, reduces performance consumption, and improves data processing efficiency. However, there are fewer update-type or delete-type write operation requests, so the SQL statement in the temporary table is executed to implement data processing on the target data table.

[0018] In one possible design, the method further includes:

[0019] If an abnormal event is detected during the process of processing data on the target data table using any temporary table corresponding to the target data table, the data of the any temporary table is rolled back and the unused temporary table corresponding to the target data table is suspended.

[0020] In an embodiment of the present application, in the process of processing data of the target data table using any temporary table corresponding to the target data table, if an abnormal event is detected, the data of any temporary table is rolled back to restore any temporary table to a state when it is not in use, thereby ensuring the accuracy of any temporary table, and the unused temporary table corresponding to the target data table is suspended to ensure the accuracy of the unused temporary table corresponding to the target data table, so that after the abnormal event is handled, the target data table can continue to be processed using any temporary table and the unused temporary table corresponding to the target data table, thereby improving the data processing efficiency.

[0021] In one possible design, the method further includes:

[0022] If the target data table name in the write operation request sent by the monitored application server to the database does not match the data table name in the set configuration file, the SQL statement in the write operation request is used to process data on the target data table.

[0023] In an embodiment of the present application, when the target data table name in the write operation request sent by the monitored application server to the database does not match the data table name in the configuration file, it indicates that there is no need to monitor the target data table. Therefore, the SQL statement in the write operation request is directly used to process the data of the target data table, thereby improving data processing efficiency.

[0024] In one possible design, the method further includes:

[0025] If the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is not in a locked state, the SQL statement in the write operation request is used to process data on the target data table.

[0026] In an embodiment of the present application, the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the configuration file, and the target data table is not in a locked state, indicating that the target data table is not locked, allowing the SQL statement in the write operation request to be used to process data on the target data table, thereby improving data processing efficiency.

[0027] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0028] A generation module, configured to generate a temporary table corresponding to a write operation request based on an operation type in the write operation request and a structured query language SQL statement in the write operation request when a target data table name in a write operation request sent by a monitored application server to a database matches a data table name in a set configuration file and the target data table is in a locked table state;

[0029] A determination module, configured to determine each temporary table corresponding to the target data table when it is detected that the target data table is not in a locked state;

[0030] The processing module is used to process the data of the target data table in turn by using each temporary table according to a set order based on the creation time of each temporary table corresponding to the target data table.

[0031] In one possible design, the generating module is used to:

[0032] If the operation type in the write operation request is an insert type, the target data table is copied to obtain a temporary table to be processed, and the SQL statement in the write operation request is used to process data in the temporary table to be processed, and the processed temporary table to be processed is used as the temporary table; or

[0033] If the operation type in the write operation request is an update type or a delete type, the temporary table is generated based on the SQL statement in the write operation request.

[0034] In one possible design, the processing module is used to:

[0035] If the operation type in the write operation request corresponding to the temporary table is an insert type, copy the data in the temporary table and use the copied data to update the data in the target data table; or

[0036] If the operation type in the write operation request corresponding to the temporary table is an update type or a delete type, the SQL statement in the temporary table is used to perform data processing on the target data table.

[0037] In one possible design, the device further includes:

[0038] The detection module is used to roll back the data of any temporary table corresponding to the target data table and suspend the unused temporary table corresponding to the target data table if an abnormal event is detected during the process of processing the data of the target data table using any temporary table corresponding to the target data table.

[0039] In a possible design, the generating module is further used for:

[0040] The monitoring module is used to use the SQL statement in the write operation request to process data on the target data table if the target data table name in the write operation request sent by the monitored application server to the database does not match the data table name in the set configuration file.

[0041] In a possible design, the generating module is further used for:

[0042] If the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is not in a locked state, the SQL statement in the write operation request is used to process data on the target data table.

[0043] In a third aspect, the present application provides an electronic device, including:

[0044] A memory for storing program instructions;

[0045] The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods in the first aspect according to the obtained program instructions.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods described in the first aspect.

[0047] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect.

[0048] The technical effects brought about by the second to fifth aspects and any one of their designs can refer to the technical effects brought about by the corresponding designs in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0050] Figure 2 A schematic diagram of another application scenario provided for an embodiment of the present application;

[0051] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0052] Figure 4 A detailed flow chart of a data processing method provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0056] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0057] In the technical solution of this application, the collection, dissemination, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0058] Before introducing a data processing method provided by an embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0059] In the current system, many systems call each other intertwinedly, and large batches of data are often processed. When executing a batch operation request to process data in a data table in the database, the data table in the database is usually locked (protection mechanism), allowing only the current operation request to be executed, and the lock will be released only after the batch operation request is completed.

[0060] Currently, during the table lock period, the user's write operation request to the data table is stored in the transaction table, and the write operation request in the transaction table is continuously retried using a scheduled task until the execution is successful or the maximum retry threshold is reached, which puts a lot of pressure on the database and consumes a lot of performance. In addition, if the number of retries of a write operation request in the transaction table exceeds the maximum retry threshold, the write operation request will no longer be executed, affecting data processing efficiency, resulting in a poor user experience.

[0061] In order to solve the above problems, the present application proposes a data processing method, device, equipment and medium to reduce performance consumption, improve data processing efficiency and user experience during large-scale data processing.

[0062] First reference Figure 1, which is a schematic diagram of an application scenario of an embodiment of the present application, including a database 11 and an application server 12, wherein the database 11 and the application server 12 are communicatively connected via a network, and the network may be a local area network, a wide area network, etc. The application server 12 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0063] In an embodiment of the present application, as an optional implementation, when the target data table name in the write operation request sent by the application server 12 to the database 11 matches the data table name in the set configuration file, and the target data table is in a locked table state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the structured query language (SQL) statement in the write operation request; when it is monitored that the target data table is not in a locked table state, each temporary table corresponding to the target data table is determined; based on the creation time of each temporary table corresponding to the target data table, each temporary table is used in turn in a set order to process data on the target data table.

[0064] refer to Figure 2 , which is another application scenario schematic diagram of the embodiment of the present application, including a database 21, a monitoring device 22 and an application server 23, wherein the database 21 and the monitoring device 22 are connected to each other through a network, which may be a local area network, a wide area network, etc., the monitoring device 22 and the application server 23 are connected to each other through a network, which may be a local area network, a wide area network, etc., the application server 23 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The monitoring device 22 may be a device that has the function of monitoring the application server 23 sending a write operation request to the database 21, and whether each data table in the database 21 is in a locked table state.

[0065] In an embodiment of the present application, as an optional implementation, when the target data table name in the write operation request sent by the application server 23 monitored by the monitoring device 22 to the database 21 matches the data table name in the set configuration file, and the target data table is in a locked table state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the SQL statement in the write operation request; when it is monitored that the target data table is not in a locked table state, each temporary table corresponding to the target data table is determined; based on the creation time of each temporary table corresponding to the target data table, each temporary table is used in turn in the set order to process data on the target data table.

[0066] Reference below Figure 3 A flowchart of a data processing method is shown to illustrate the technical solution provided by the embodiment of the present application:

[0067] Step 301: When the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is in a locked state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the SQL statement in the write operation request;

[0068] The write operation request sent by the application server to the database includes information such as the target data table name, operation type, SQL statement, etc. The operation type includes insert type, update type, and delete type. The set configuration file includes the data table name of each data table that needs to be monitored.

[0069] Optionally, the method further comprises:

[0070] If the target data table name in the write operation request sent by the monitored application server to the database does not match the data table name in the set configuration file, the SQL statement in the write operation request is used to process data on the target data table.

[0071] Specifically, if the target data table name in the write operation request does not match the data table name in the set configuration file, it indicates that the target data table is a data table that does not need to be monitored. Therefore, the SQL statement in the write operation request is used to directly process data on the target data table.

[0072] Optionally, the method further comprises:

[0073] If the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is not in a locked table state, the SQL statement in the write operation request is used to process data on the target data table.

[0074] Specifically, if the target data table name in the write operation request matches the data table name in the set configuration file, and the target data table is not in a locked table state, it indicates that the target data table is a data table that needs to be monitored, but the target data table is not in a locked table state. During the period when the table is not locked, the write operation request is allowed to execute. Therefore, the SQL statement in the write operation request is used to directly process data on the target data table.

[0075] Step 302, when it is monitored that the target data table is not in a locked state, determine each temporary table corresponding to the target data table;

[0076] Since the target data table releases the lock, the state of the target data table changes from being in the locked table state to being in the unlocked table state. Therefore, when it is monitored that the target data table is not in the locked table state, each temporary table corresponding to the target data table is determined. The number of temporary tables is the number of write operation requests to the target data table monitored between the target data table locks, which can be 1 or more.

[0077] Step 303 : Based on the creation time of each temporary table corresponding to the target data table, and in accordance with a set order, each temporary table is used in turn to process data on the target data table.

[0078] Among them, the setting order can be set according to actual conditions. For example, the setting order can be from early to late. In the embodiment of the present application, based on the creation time of each temporary table corresponding to the target data table, each temporary table is used in turn in order from early to late to process data of the target data table.

[0079] In an embodiment of the present application, when the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is in a locked table state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the SQL statement in the write operation request, thereby realizing that during the period when the target data table is locked, a temporary table corresponding to each write operation request to the target data table is generated. In addition, when the present application monitors that the target data table is not in a locked table state, it is determined that the lock of the target data table has been released, thereby determining each temporary table corresponding to the target data table, that is, during the period when the target data table is locked, the temporary table corresponding to each write operation request to the target data table is generated. Based on the creation time of each temporary table corresponding to the target data table, the present application sequentially uses each temporary table to process data on the target data table in accordance with the set order, without using a timed task to continuously retry the write operation request, avoiding the situation where the write operation request is no longer executed because the number of retries of the write operation request exceeds the maximum retry number threshold, thereby reducing the pressure on the database, thereby reducing performance consumption, and improving data processing efficiency. Therefore, the present application can reduce performance consumption during large-scale data processing, improve data processing efficiency and user experience.

[0080] The specific steps of the data processing method provided above are described in detail below. Figure 4 As shown:

[0081] Step 401: monitor a write operation request sent by an application server to a database.

[0082] In the embodiment of the present application, if other operation requests, such as write operation requests, sent by the application server to the database are monitored, the target data table is processed accordingly using the other operation requests.

[0083] Step 402 , determining whether the target data table name in the write operation request matches the data table name in the configuration file, if so, executing step 403 , otherwise, executing step 407 .

[0084] Step 403 , determining whether the target data table is in a locked state, if so, executing step 404 , otherwise, executing step 407 .

[0085] Specifically, request information for requesting the state of the target data table is sent to the database, and response information of the target data table sent by the database is received, wherein the response information includes information on the state of the target data table.

[0086] The information about the state of the target data table may be a state code corresponding to the state of the target data table, that is, different state codes correspond to different states of the target data table. The state code may be set according to actual conditions, for example, a first state code and a second state code may be set, the first state code indicating that the target data table is in a locked state, and the second state code indicating that the target data table is not in a locked state.

[0087] Step 404: Generate a temporary table corresponding to the write operation request based on the operation type in the write operation request and the SQL statement in the write operation request.

[0088] Due to different operation types in write operation requests, the generation methods of temporary tables corresponding to write operation requests are different. Therefore, based on the operation type in the write operation request and the SQL statement in the write operation request, the temporary table corresponding to the write operation request is generated. There are three situations:

[0089] In the first case, if the operation type in the write operation request is an insert type, the target data table is copied to obtain a pending temporary table, and the SQL statement in the write operation request is used to process data in the pending temporary table, and the processed pending temporary table is used as a temporary table.

[0090] Specifically, if the operation type in the write operation request is insert type, the target data table tableA is copied to obtain a pending temporary table named tableA_timestamp_insert, the SQL statement in the write operation request is executed, and data is processed on the pending temporary table, that is, data is inserted into the pending temporary table to obtain the processed pending temporary table, and the processed pending temporary table is used as the temporary table.

[0091] In the second case, if the operation type in the write operation request is an update type, a temporary table is generated based on the SQL statement in the write operation request.

[0092] Specifically, if the operation type in the write operation request is the update type, and the name of the target data table is tableA, a temporary table named tableA_timestamp_update is created, and the SQL statement in the write operation request is written into the temporary table.

[0093] In a third case, if the operation type in the write operation request is a delete type, the temporary table is generated based on the SQL statement in the write operation request.

[0094] Specifically, if the operation type in the write operation request is the delete type, and the name of the target data table is tableA, a temporary table named tableA_timestamp_delete is created, and the SQL statement in the write operation request is written into the temporary table.

[0095] Step 405: when it is detected that the target data table is not in a locked state, each temporary table corresponding to the target data table is determined.

[0096] Step 406 , based on the creation time of each temporary table corresponding to the target data table, and in accordance with a set order, each temporary table is used in turn to process data on the target data table.

[0097] Specifically, based on the creation time of each temporary table corresponding to the target data table, each temporary table is used in turn to process data on the target data table in order from earliest to latest.

[0098] In the embodiment of the present application, in the process of sequentially using each temporary table to process data on the target data table, the used temporary table corresponding to the target data table is deleted.

[0099] For example, the four temporary tables corresponding to the target data table are temporary table 1, temporary table 2, temporary table 3 and temporary table 4. The creation time of temporary table 1 is earlier than the creation time of temporary table 2, the creation time of temporary table 2 is earlier than the creation time of temporary table 3, and the creation time of temporary table 3 is earlier than the creation time of temporary table 4. Therefore, first use temporary table 1 to process the data of the target data table, delete temporary table 1 after the data processing is completed, then use temporary table 2 to process the data of the target data table, delete temporary table 2 after the data processing is completed, then use temporary table 3 to process the data of the target data table, delete temporary table 3 after the data processing is completed, and finally use temporary table 4 to process the data of the target data table, delete temporary table 4 after the data processing is completed.

[0100] Due to the different operation types in the write operation requests corresponding to the temporary tables, the methods of using the temporary tables to process data on the target data table are different. Therefore, when using each temporary table to process data on the target data table, there are the following three situations:

[0101] In the first case, if the operation type in the write operation request corresponding to the temporary table is an insert type, the data in the temporary table is copied, and the copied data is used to update the data in the target data table.

[0102] For example, if the operation type in the write operation request corresponding to the temporary table is insert, the name of the target data table is tableA, and the name of the temporary table is tableA_timestamp_insert, execute the following SQL statement: insertinto tableAselect*from tableA_timestamp_insert to copy the data in the temporary table and update the data in the target data table with the copied data.

[0103] In the second case, if the operation type in the write operation request corresponding to the temporary table is an update type, the SQL statement in the temporary table is used to process data on the target data table.

[0104] For example, if the operation type in the write operation request corresponding to the temporary table is update type, the name of the target data table is tableA, and the name of the temporary table is tableA_timestamp_update, then the SQL statement in the temporary table tableA_timestamp_update is executed to process data on the target data table.

[0105] In the third case, if the operation type in the write operation request corresponding to the temporary table is a delete type, the SQL statement in the temporary table is used to process data on the target data table.

[0106] For example, if the operation type in the write operation request corresponding to the temporary table is the delete type, the name of the target data table is tableA, and the name of the temporary table is tableA_timestamp_delete, then the SQL statement in the temporary table tableA_timestamp_delete is executed to process data on the target data table.

[0107] Since abnormal events may occur in the process of using each temporary table to process the target data table in turn, in order to ensure the efficiency and accuracy of data processing, this application uses the following method to handle abnormal events:

[0108] If an abnormal event is detected during the process of processing data on the target data table using any temporary table corresponding to the target data table, the data of any temporary table will be rolled back and the unused temporary table corresponding to the target data table will be suspended.

[0109] Among them, an abnormal event refers to the process of processing data on the target data table using any temporary table corresponding to the target data table, due to the abnormality of any temporary table, which leads to the interruption of the process of processing data on the target data table using any temporary table. Data rollback refers to restoring the state of a database or system to a previous correct or known state to correct data inconsistencies caused by incorrect operations or system failures. Data rollback is usually used to handle program or data processing errors to ensure data integrity and consistency. In the embodiment of the present application, rolling back data on any temporary table refers to restoring any temporary table to an unused state.

[0110] In an embodiment of the present application, after the unused temporary table corresponding to the target data table is suspended, an alarm is triggered to notify the operation and maintenance personnel to handle it in a timely manner. After the processing is completed, based on the creation time of any temporary table corresponding to the target data table and the creation time of the unused temporary table, in accordance with the set order, data processing is performed on the target data table in sequence using any temporary table corresponding to the target data table and the unused temporary table.

[0111] For example, the four temporary tables corresponding to the target data table are temporary table A, temporary table B, temporary table C, and temporary table D. The creation time of temporary table A is earlier than the creation time of temporary table B, the creation time of temporary table B is earlier than the creation time of temporary table C, and the creation time of temporary table C is earlier than the creation time of temporary table D. Based on the creation time of the four temporary tables corresponding to the target data table, each temporary table is used in turn in the order from earliest to latest to process the data of the target data table. If an abnormal event is detected during the process of using temporary table B to process the data of the target data table, the data of temporary table B is rolled back, and temporary tables C and D are suspended, triggering an alarm to notify the operation and maintenance personnel to handle it in time. After the operation and maintenance personnel have finished the processing, based on the creation time of temporary tables B, temporary tables C, and temporary tables D, each temporary table is used in turn in the order from earliest to latest to process the data of the target data table.

[0112] Step 407: Process data in the target data table using the SQL statement in the write operation request.

[0113] Based on the same technical concept, the present application exemplarily provides a data processing device, such as Figure 5 As shown, the device comprises:

[0114] The generating module 501 is used for generating a temporary table corresponding to the write operation request based on the operation type in the write operation request and the structured query language SQL statement in the write operation request when the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file and the target data table is in a locked table state;

[0115] A determination module 502 is used to determine each temporary table corresponding to the target data table when it is monitored that the target data table is not in a locked state;

[0116] The processing module 503 is used to process the data of the target data table using each temporary table in turn according to a set order based on the creation time of each temporary table corresponding to the target data table.

[0117] Optionally, the generating module 501 is used to:

[0118] If the operation type in the write operation request is an insert type, the target data table is copied to obtain a temporary table to be processed, and the SQL statement in the write operation request is used to process data in the temporary table to be processed, and the processed temporary table to be processed is used as the temporary table; or

[0119] If the operation type in the write operation request is an update type or a delete type, the temporary table is generated based on the SQL statement in the write operation request.

[0120] Optionally, the processing module 503 is used to:

[0121] If the operation type in the write operation request corresponding to the temporary table is an insert type, copy the data in the temporary table and use the copied data to update the data in the target data table; or

[0122] If the operation type in the write operation request corresponding to the temporary table is an update type or a delete type, the SQL statement in the temporary table is used to perform data processing on the target data table.

[0123] Optionally, the device further comprises:

[0124] The detection module is used to roll back the data of any temporary table corresponding to the target data table and suspend the unused temporary table corresponding to the target data table if an abnormal event is detected during the process of processing the data of the target data table using any temporary table corresponding to the target data table.

[0125] Optionally, the generating module 501 is further used for:

[0126] The monitoring module is used to use the SQL statement in the write operation request to process data on the target data table if the target data table name in the write operation request sent by the monitored application server to the database does not match the data table name in the set configuration file.

[0127] Optionally, the generating module 501 is further used for:

[0128] If the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is not in a locked state, the SQL statement in the write operation request is used to process data on the target data table.

[0129] Based on the same inventive concept, the present application embodiment provides an electronic device, which can realize the functions of the data processing device discussed above. Please refer to Figure 6 , the device comprises a processor 601 and a memory 602, wherein the memory 602 is used to store program instructions;

[0130] The processor 601 calls the program instructions stored in the memory and executes the program instructions to implement the following steps:

[0131] When the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is in a locked table state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the structured query language SQL statement in the write operation request;

[0132] When it is monitored that the target data table is not in a locked state, determining each temporary table corresponding to the target data table;

[0133] Based on the creation time of each temporary table corresponding to the target data table, each temporary table is used in turn to process data on the target data table in a set order.

[0134] Optionally, generating a temporary table corresponding to the write operation request based on the operation type in the write operation request and an SQL statement in the write operation request includes:

[0135] If the operation type in the write operation request is an insert type, the target data table is copied to obtain a temporary table to be processed, and the SQL statement in the write operation request is used to process data in the temporary table to be processed, and the processed temporary table to be processed is used as the temporary table; or

[0136] If the operation type in the write operation request is an update type or a delete type, the temporary table is generated based on the SQL statement in the write operation request.

[0137] Optionally, the using each temporary table to perform data processing on the target data table includes:

[0138] If the operation type in the write operation request corresponding to the temporary table is an insert type, copy the data in the temporary table and use the copied data to update the data in the target data table; or

[0139] If the operation type in the write operation request corresponding to the temporary table is an update type or a delete type, the SQL statement in the temporary table is used to perform data processing on the target data table.

[0140] Optionally, the processor 601 is further configured to execute:

[0141] If an abnormal event is detected during the process of processing data on the target data table using any temporary table corresponding to the target data table, the data of the any temporary table is rolled back and the unused temporary table corresponding to the target data table is suspended.

[0142] Optionally, the processor 601 is further configured to execute:

[0143] If the target data table name in the write operation request sent by the monitored application server to the database does not match the data table name in the set configuration file, the SQL statement in the write operation request is used to process data on the target data table.

[0144] Optionally, the processor 601 is further configured to execute:

[0145] If the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is not in a locked state, the SQL statement in the write operation request is used to process data on the target data table.

[0146] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium, and the computer program product includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the data processing method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0147] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data processing method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: include: When the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is in a locked table state, a temporary table corresponding to the write operation request is generated based on the operation type in the write operation request and the structured query language SQL statement in the write operation request; When it is monitored that the target data table is not in a locked state, determining each temporary table corresponding to the target data table; Based on the creation time of each temporary table corresponding to the target data table, each temporary table is used in turn to process data on the target data table in a set order.

2. The method according to claim 1, characterized in that The generating a temporary table corresponding to the write operation request based on the operation type in the write operation request and the SQL statement in the write operation request includes: If the operation type in the write operation request is an insert type, the target data table is copied to obtain a temporary table to be processed, and the SQL statement in the write operation request is used to process data in the temporary table to be processed, and the processed temporary table to be processed is used as the temporary table; or If the operation type in the write operation request is an update type or a delete type, the temporary table is generated based on the SQL statement in the write operation request.

3. The method according to claim 1, characterized in that The using each temporary table to process data on the target data table includes: If the operation type in the write operation request corresponding to the temporary table is an insert type, copy the data in the temporary table and use the copied data to update the data in the target data table; or If the operation type in the write operation request corresponding to the temporary table is an update type or a delete type, the SQL statement in the temporary table is used to perform data processing on the target data table.

4. The method according to claim 1, characterized in that The method further comprises: If an abnormal event is detected during the process of processing data on the target data table using any temporary table corresponding to the target data table, the data of the any temporary table is rolled back and the unused temporary table corresponding to the target data table is suspended.

5. The method according to claim 1, characterized in that The method further comprises: If the target data table name in the write operation request sent by the monitored application server to the database does not match the data table name in the set configuration file, the SQL statement in the write operation request is used to process data on the target data table.

6. The method according to claim 1, characterized in that The method further comprises: If the target data table name in the write operation request sent by the monitored application server to the database matches the data table name in the set configuration file, and the target data table is not in a locked state, the SQL statement in the write operation request is used to process data on the target data table.

7. A data processing device, characterized in that: include: A generation module, configured to generate a temporary table corresponding to a write operation request based on an operation type in the write operation request and a structured query language SQL statement in the write operation request when a target data table name in a write operation request sent by a monitored application server to a database matches a data table name in a set configuration file and the target data table is in a locked table state; A determination module, configured to determine each temporary table corresponding to the target data table when it is detected that the target data table is not in a locked state; The processing module is used to process the data of the target data table in turn by using each temporary table according to a set order based on the creation time of each temporary table corresponding to the target data table.

8. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of claims 1-6 according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images