Data processing method, device, electronic device and computer storage medium

Abstract

An embodiment of the present application provides a data processing method, device, electronic device and computer storage medium, the method comprising: intercepting multiple data operation requests received by a server within a current time period; the data operation requests are used to perform write operations or delete operations on operation data; counting the data inflow rate corresponding to the multiple operation data completed within the current time period; the data inflow rate includes at least one of a data write rate and a data deletion rate; based on the data inflow rate, determining the release time of a lock corresponding to each target data operation request received within the next time period; in this way, elastic scaling of the lock release time is achieved according to the data inflow rate, thereby improving data processing efficiency in high-concurrency scenarios.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device, and computer storage medium. Background Art

[0002] Data storage often presents the problem of maintaining consistency in data deletion and writing. Relational databases, such as relational database management systems (MySQL, MySQL Structured Query Language), Structured Query Language Server (SqlSever), and Oracle, can solve this problem through key-association operations. However, when users have strict requirements on data read and write speeds, non-relational databases are needed, and the first database that comes to mind is Redis.

[0003] The Redis database typically uses a locking mechanism, known as a Redis lock. When a thread operates on a particular object, it acquires the lock. If another thread cannot obtain the lock, it waits for execution. While this mechanism ensures consistency when deleting and writing hierarchically related data, it introduces issues like thread waiting, resulting in low data processing efficiency in high-concurrency scenarios. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, and computer storage medium, which can improve data processing efficiency in high-concurrency scenarios.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a data processing method, the method comprising:

[0007] Intercept multiple data operation requests received by the server in the current time period; the data operation requests are used to write or delete the operation data;

[0008] Counting the data inflow rate corresponding to the multiple operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deletion rate;

[0009] Determining, based on the data inflow rate, a lock release time corresponding to each target data operation request received in a next time period;

[0010] Based on the determined lock release time, an operation corresponding to each target data operation request is executed.

[0011] In some embodiments, when the data inflow rate includes a data write rate and a data delete rate, determining, based on the data inflow rate, a lock release time corresponding to each target data operation request received in a next time period includes:

[0012] Obtain the write release time and delete release time of the lock in the current time period;

[0013] Determining a first ratio of the data write rate to a write release time of a lock in the current time period, and a second ratio of the data delete rate to a delete release time of a lock in the current time period;

[0014] Based on the first ratio and the second ratio, a lock release time corresponding to each target data operation request received within the next time period is determined.

[0015] In some embodiments, determining, based on the first ratio and the second ratio, a lock release time corresponding to each target data operation request received within the next time period includes:

[0016] Adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal;

[0017] Based on the adjusted first ratio and the second ratio, a lock release time corresponding to each target data operation request received in the next time period is determined.

[0018] In some embodiments, executing the operation corresponding to each target data operation request based on the determined lock release time includes:

[0019] For each target data operation request, acquiring a lock of the target operation data; the target data operation request is used to perform a write operation or a delete operation on the target operation data;

[0020] Based on the determined lock release time, a locking operation is performed on the target operation data, and a corresponding operation is performed on the target operation data according to the target data operation request.

[0021] In some embodiments, acquiring a lock on target operation data includes:

[0022] Obtain a lock for the target operation data from a Redis database; the lock for the target operation data is a key-value lock, the key value is an identifier for the target operation data, and the value value is a release time for the lock for the target operation data.

[0023] In some embodiments, the method further comprises:

[0024] The execution process corresponding to each target data operation request is monitored until the monitoring duration reaches the determined lock release time, and the lock of the target operation data is released.

[0025] In some embodiments, the method further comprises:

[0026] When the monitoring time reaches the determined lock release time and it is detected that the lock of the target operation data is not released, a rollback operation is performed on the execution of the target operation data.

[0027] The embodiment of the present application provides a data processing device, which includes an interception module, a statistics module, a determination module, and an execution module, wherein:

[0028] An interception module, configured to intercept multiple data operation requests received by the server within a current time period; the data operation requests are used to write or delete the operation data;

[0029] A statistics module, configured to count data inflow rates corresponding to a plurality of operation data completed within the current time period; the data inflow rate includes at least one of a data writing rate and a data deletion rate;

[0030] a determination module, configured to determine, based on the data inflow rate, a lock release time corresponding to each target data operation request received within a next time period;

[0031] An execution module is configured to execute an operation corresponding to each target data operation request based on the determined lock release time.

[0032] An embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the data processing method provided by one or more of the aforementioned technical solutions is implemented.

[0033] An embodiment of the present application provides a computer storage medium storing a computer program; after the computer program is executed, it can implement the data processing method provided by one or more of the aforementioned technical solutions.

[0034] An embodiment of the present application provides a data processing method, device, electronic device and computer storage medium, the method comprising: intercepting multiple data operation requests received by a server within a current time period; the data operation requests are used to perform write operations or delete operations on operation data; counting the data inflow rate corresponding to the multiple operation data completed within the current time period; the data inflow rate includes at least one of a data write rate and a data deletion rate; based on the data inflow rate, determining a lock release time corresponding to each target data operation request received within a next time period; and based on the determined lock release time, executing an operation corresponding to each target data operation request.

[0035] It can be seen that in the embodiment of the present application, by intercepting each data operation request in the current time period to count the data inflow rate in the time period, and then dynamically adjusting the lock release time corresponding to each target data operation request in the next time period, that is, the elastic scaling of the lock release time can be achieved; for high concurrency scenarios, because the lock release time is dynamically adjusted according to the data inflow rate, that is, different data inflow rates correspond to different lock release times, it is possible to ensure that the data deletion and writing operations are completed as much as possible within the lock release time, ensuring the consistency of data deletion and writing operations while optimizing the thread waiting problem, thereby improving data processing efficiency in high concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a process for writing and deleting operation data provided in an embodiment of the present application;

[0038] Figure 3 A flowchart of another data processing method provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in this application will be described clearly and completely below in conjunction with the accompanying drawings in this application.

[0042] The present application will be further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely intended to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions described in the present application may be implemented in any combination.

[0043] It should be noted that, in this application, the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus comprising a series of elements includes not only the elements explicitly stated, but also other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a ..." does not exclude the presence of other related elements (such as steps in the method or units in the apparatus, for example, a unit may be part of a processor, part of a program or software, etc.) in the method or apparatus comprising the element.

[0044] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "C and / or D" can mean: C exists alone, C and D exist simultaneously, and D exists alone. Furthermore, the term "at least one" herein means any combination of at least two of any one or more of a plurality of items. For example, "at least one of C, D, and E" can mean any one or more elements selected from the set consisting of C, D, and E.

[0045] For example, the data processing method provided in this application includes a series of steps, but the data processing method provided in this application is not limited to the recorded steps. Similarly, the data processing device provided in this application includes a series of modules, but the data processing device provided in this application is not limited to including the modules explicitly recorded, and may also include modules that need to be set up to obtain relevant information or perform processing based on information.

[0046] In the related art, the meaning of a lock is simply that each object data stored by the user in the Redis database has a lock, and this lock can only be acquired and consumed by a single thread; when the lock has been acquired, other threads can only suspend and wait. Usually, the lock release time is fixed, and in order to ensure the accuracy of each thread's task operation, the lock release time is set to be relatively long. In this way, when there are many object data to be processed, there will be a problem of thread waiting, resulting in low data processing efficiency in high-concurrency scenarios; in addition, when there are fewer object data to be processed in the first time period, and more object data to be processed in the second time period, if the lock release time set in the first time period is continued to be used in the second time period, some object data will not be operated on after the lock release time has expired, and there is a risk of data loss, which makes it difficult to meet the data processing requirements in high-concurrency scenarios.

[0047] The implementation of Redis locks is related to the atomicity of Redis itself. Redis is single-threaded. A Redis lock is equivalent to a tag for object data, typically represented by a key-value pair. The key is usually the object's universally unique identifier (UUID), and the value is typically 1 to indicate that the Redis lock for the object data has been acquired, and 0 to indicate that the Redis lock for the object data can be acquired. This type of Redis lock provides the advantage of perfectly guaranteeing the atomicity of single-data operations. However, when storing multiple objects with hierarchical or dependent data, this method has difficulty ensuring consistency in data deletion and writing.

[0048] In order to solve the above problems, the following embodiments are proposed.

[0049] In some embodiments of the present application, the data processing method can be implemented using a processor in a data processing device, and the above-mentioned processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0050] Figure 1 A flow chart of a data processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the process may include:

[0051] Step 100: intercept multiple data operation requests received by the server within the current time period.

[0052] For example, a user can initiate a data operation request on the client, and the client sends the data operation request initiated by the user to the server, and the server can receive the data operation request sent by the client; here, for each data operation request received by the server, an interceptor can be used to perform an interception operation, thereby intercepting multiple data operation requests received by the server in the current time period.

[0053] Here, the current time period can be the first time period for the interceptor to perform the interception operation; the length of the current time period can be determined according to the actual scenario, and the embodiment of the present application does not limit this; for example, in a high concurrency scenario, the length of the current time period is usually less than 1 second.

[0054] In an embodiment of the present application, a data operation request is used to perform a write operation or a delete operation on the operation data; wherein the operation data may be data with a hierarchical association relationship, and the operation data has corresponding identification information, which may be the uuid of the operation data.

[0055] Exemplarily, writing operations or deleting operations on operation data are both operations on the database; that is, writing operation data into a certain database, or deleting operation data in a certain database; here, there is no limitation on the type of database, for example, it can be a Redis database or other types of databases.

[0056] Here, taking Redis database as an example, combined with Figure 2To explain the meaning of operation data with a hierarchical association relationship, suppose that an operation data b needs to be written to the Redis database. The writing of this data is divided into two parts: the first part is to write an operation data b to the Redis database (corresponding to step (2) in the figure); the second part is to put the identifier of the operation data b (for example, uuid) into the object data a (corresponding to step (1) in the figure, object data a is a list that can contain multiple uuids similar to the operation data b). Similarly, suppose that an operation data b needs to be deleted from the Redis database. The deletion of this data is also divided into two parts: the first part is to delete an operation data b from the Redis database (corresponding to step (4) in the figure); the second part is to delete the uuid of the operation data b from the object data a (corresponding to step (3) in the figure). At this time, the operation data b and the object data a have a hierarchical association relationship, that is, the operation data b is data with a hierarchical association relationship.

[0057] It should be noted that multiple data operation requests correspond to multiple operation data, and the operation data corresponding to different data operation requests can be the same or different. For example, if a data operation request is used to write operation data b, another data operation request can be used to delete operation data b or to delete operation data c.

[0058] Step 101: Count the data inflow rates corresponding to multiple operation data completed in the current time period.

[0059] In an embodiment of the present application, after intercepting multiple data operation requests for multiple operation data within the current time period according to the above steps, the data inflow rate corresponding to the multiple operation data completed within the time period is further counted; here, the data inflow rate may include at least one of the data write rate and the data deletion rate.

[0060] Exemplarily, the intercepted multiple data operation requests may include the following three situations: 1) multiple data operation requests are all requests for writing operations on multiple operation data; 2) multiple data operation requests are all requests for deleting operations on multiple operation data; 3) multiple data operation requests include both requests for writing operations on several operation data and requests for deleting operations on several operation data.

[0061] It can be understood that when statistics are collected on multiple operation data corresponding to the above-mentioned first situation, the data inflow rate obtained only includes the data write rate; when statistics are collected on multiple operation data corresponding to the above-mentioned second situation, the data inflow rate obtained only includes the data deletion rate; when statistics are collected on multiple operation data corresponding to the above-mentioned third situation, the data inflow rate obtained includes the data write rate and the data deletion rate.

[0062] Step 102: Based on the data inflow rate, determine the lock release time corresponding to each target data operation request received in the next time period.

[0063] Here, the next time period refers to a time period after the current time period, and the time period has the same length as the current time period.

[0064] Exemplarily, the target data operation request is used to perform a write operation or a delete operation on the target operation data, and the target operation data is data with a hierarchical association relationship; it should be noted that the target data operation request and the above-mentioned data operation request are both requests received by the server, and the two have the same function. The only difference is that the time period in which the server receives them is different. The target operation data is also similar to the above-mentioned operation data, and will not be repeated here.

[0065] In an embodiment of the present application, the release time of the lock is also called the locking time of the lock, which may include at least one of the write release time and the delete release time of the lock; it can be understood that the content included in the lock release time is determined according to the content included in the data inflow rate.

[0066] For example, if the data inflow rate only includes the data write rate, then based on the data write rate, the write release time of the lock corresponding to each target data operation request for performing a write operation received in the next time period can be determined. If the data inflow rate only includes the data deletion rate, then based on the data deletion rate, the delete release time of the lock corresponding to each target data operation request for performing a delete operation received in the next time period can be determined. If the data inflow rate includes the data write rate and the data deletion rate, then based on the data write rate and the data deletion rate, the write release time of the lock corresponding to each target data operation request for performing a write operation received in the next time period and the delete release time of the lock corresponding to each target data operation request for performing a delete operation can be determined. Because in the case where the data inflow rate includes the data write rate and the data deletion rate, the write release time and the delete release time of the lock corresponding to the next time period can be determined simultaneously; below, taking this case as an example, the method for determining the write release time and the delete release time of the lock is explained.

[0067] In one embodiment, when the data inflow rate includes a data write rate and a data deletion rate, determining the release time of the lock corresponding to each target data operation request received in the next time period based on the data inflow rate may include: obtaining the write release time and the delete release time of the lock in the current time period; determining a first ratio of the data write rate and the write release time of the lock in the current time period, and a second ratio of the data deletion rate and the delete release time of the lock in the current time period; and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio.

[0068] Here, the write release time and delete release time of the lock in the current time period can be pre-set according to actual conditions, and the values ​​of the two can be the same or different; the embodiment of the present application is not limited to this. For example, the values ​​of both can be set to 5 minutes, or one can be set to 3 minutes and the other can be set to 5 minutes.

[0069] For example, the write release time t of the lock within the current time period is obtained write and delete release time t delete After that, we further determine the data writing rate v write And the write release time t of the lock in the current time period delete The first ratio and the data deletion rate v delete And the lock deletion release time t in the current time period delete The second ratio Then, based on the two ratios, a lock release time corresponding to each target data operation request received in the next time period is determined.

[0070] In some embodiments, determining the release time of the lock corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio may include: adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal; and determining the release time of the lock corresponding to each target data operation request received in the next time period based on the adjusted first ratio and the second ratio.

[0071] In the embodiment of the present application, the first ratio and the second ratio are adjusted so that the first ratio and the second ratio are equal, that is, the relationship shown in formula (1) is satisfied:

[0072]

[0073] Here, the data writing rate v write and data deletion rate v delete The unit is kilobytes per second (KB / s); the write release time of the lock is t writeand delete release time t delete The unit is seconds (s).

[0074] Exemplarily, during the adjustment process of formula (1), the following conditions must be met: the values ​​of the first ratio and the second ratio are both greater than 1, and the adjusted lock write release time and lock delete release time are both greater than the set value; here, the value of the set value can be set according to actual conditions, for example, it can be set to 30s.

[0075] For example, according to formula (1), if the data writing rate v in the current time period is write Increases, the write release time t of the lock write Increase proportionally, that is, the adjusted lock write release time can be obtained; similarly, if the data deletion rate v in the current time period delete Increases, the lock deletion release time t delete The lock release time is increased proportionally, that is, the adjusted lock deletion release time can be obtained; then, the adjusted lock write release time is determined as the lock write release time corresponding to each target data operation request for write operation received in the next time period, and at the same time, the adjusted lock deletion release time is determined as the lock deletion release time corresponding to each target data operation request for delete operation received in the next time period.

[0076] It should be noted that after intercepting multiple target data operation requests in the next time period, the data inflow rate corresponding to the multiple target operation data continues to be counted to determine the release time of the lock corresponding to the data operation request in the next time period, and so on, until the server no longer receives data operation requests sent by the client.

[0077] It can be seen that the embodiment of the present application controls the dynamic scaling of the lock release time corresponding to each target data operation request in the next time period of the time period by counting the data inflow rate in a certain time period; specifically, when the data inflow rate in a certain time period is relatively high, by adjusting the lock release time in the time period, the lock release time in the next time period is relatively extended, ensuring that each operation data can complete the write or delete operation within the lock release time, thereby greatly reducing the risk of data loss and meeting the data processing requirements in high concurrency scenarios. Conversely, when the data inflow rate in a certain time period is relatively low, by adjusting the lock release time in the time period, the lock release time in the next time period is relatively shortened. In this way, not only can it be ensured that each operation data can complete the write or delete operation within the lock release time as much as possible, but it can also solve the thread waiting problem and improve the data processing efficiency in high concurrency scenarios.

[0078] Step 103: Based on the determined lock release time, execute the operation corresponding to each target data operation request.

[0079] In some embodiments, based on the determined lock release time, performing an operation corresponding to each target data operation request may include: for each target data operation request, obtaining a lock on the target operation data; based on the determined lock release time, performing a locking operation on the target operation data, and performing corresponding operations on the target operation data according to each target data operation request.

[0080] Exemplarily, obtaining a lock on target operation data may include: obtaining a lock on target operation data from a Redis database; it should be noted that the locks involved in the embodiments of the present application are all key-value locks; wherein, for the lock on operation data, the key value is the identifier of the operation data, and the value value is the release time of the lock on the operation data; for the lock on target operation data, the key value is the identifier of the target operation data, and the value value is the release time of the lock on the target operation data.

[0081] Exemplarily, after obtaining the lock of the target operation data, the target operation data can be locked according to the lock release time determined in the above steps, and the corresponding write or delete operation can be performed on the target operation data according to the target data operation request within the lock release time.

[0082] In some embodiments, the above method further includes: monitoring the execution process corresponding to each target data operation request until the monitoring duration reaches the determined lock release time, and releasing the lock of the target operation data.

[0083] For example, the execution process corresponding to each target data operation request can be monitored through a monitoring function or a monitoring program. When the monitoring time reaches the determined lock release time, the lock of the target operation data will be released; generally speaking, if there is no abnormality or interruption in the execution process of the target operation data, the lock will be actively released.

[0084] Here, after the lock of the target operation data is released, the released lock will be returned to the Redis database.

[0085] In some embodiments, the above method may further include: when the monitoring time reaches the determined lock release time and it is detected that the lock of the target operation data is not released, rolling back the execution of the target operation data.

[0086] For example, if the monitoring duration reaches the determined lock release time and the lock on the target operation data is not released, it indicates that there may be some abnormality in the execution process of the target operation data. In this case, the execution of the target operation data can be rolled back. That is, the execution of the target operation data is rolled back to the state before execution and then the relevant operations are executed again to ensure the accuracy of the execution process. Here, the rollback operation can be implemented by adding a transaction operation to the data execution process, or it can be implemented by other methods.

[0087] It should be noted that if after the rollback operation, it is still detected that the lock of the target operation data has not been released, then it is necessary to capture the exception and print a log to remind the user to handle the exception in time.

[0088] The embodiment of the present application provides a data processing method, the method comprising: intercepting multiple data operation requests received by a server in a current time period; the data operation requests are used to perform write operations or delete operations on the operation data; counting the data inflow rate corresponding to the multiple operation data completed in the current time period; the data inflow rate includes at least one of a data write rate and a data delete rate; based on the data inflow rate, determining the lock release time corresponding to each target data operation request received in the next time period; and performing the operation corresponding to each target data operation request based on the determined lock release time. It can be seen that in the embodiment of the present application, by intercepting each data operation request in the current time period to count the data inflow rate in the time period, and then dynamically adjusting the lock release time corresponding to each target data operation request in the next time period, that is, elastic scaling of the lock release time can be achieved; for high concurrency scenarios, because the lock release time is dynamically adjusted according to the data inflow rate, that is, different data inflow rates correspond to different lock release times, it can ensure that the data deletion and writing operations are completed as much as possible within the lock release time, ensuring the consistency of the data deletion and writing operations while optimizing the thread waiting problem, thereby improving the data processing efficiency in high concurrency scenarios.

[0089] In order to better reflect the purpose of this application, the embodiments of this application are further explained on the basis of the above embodiments.

[0090] In a high-concurrency scenario, a large number of access requests (corresponding to the above-mentioned data operation requests or target data operation requests) are usually received, and data writing and deletion operations need to be performed continuously on the database; here, based on the above-mentioned Figure 2 Take the following as an example to illustrate: Figure 2There may be multiple execution orders of the steps (1)(2)(3)(4) shown. Ideally, when a write operation and a delete operation for the operation data are received at the same time (assuming that the execution of step (1) takes precedence over step (2), and the execution of step (3) takes precedence over step (4)), the optimal step execution order is (1)(2)(3)(4) or (3)(4)(1)(2), so that when the user performs a read operation during the execution of any step, no dirty data will be seen; however, in actual scenarios, such step execution orders may appear as (1)(3)(4)(2) or (3)(1)(2)(4), etc.; it can be seen that in these cases, abnormal situations may occur, such as the operation data to be written is deleted, or the operation data to be deleted is not deleted, and these situations are difficult to check and modify online, and often cause accidents in the live network.

[0091] In order to overcome the above problems, the embodiment of the present application uses an interceptor to obtain multiple data operation requests received by the server in the current time period, and optimizes the Redis lock mechanism in the related technology based on these intercepted data operation requests.

[0092] For example, the optimization process for the Redis lock mechanism is as follows. Similarly, a key-value pair is used to maintain locks on object data (corresponding to the aforementioned operation data or target operation data). Unlike related technologies, the value stored is not 0 or 1, but rather the release time of the object data lock. A major risk with conventional Redis locks is that the lock may not be released, making the object data permanently unavailable. The purpose of optimizing Redis locks is to prevent the lock from being released inadvertently.

[0093] For example, the following two situations may occur when the lock is not released actively: one is that an exception occurs during the execution of the intermediate steps (1)(2)(3)(4) but is not captured, resulting in the lock release program not being executed; the other is that the execution of the intermediate steps (1)(2)(3)(4) takes too long, resulting in a large number of tasks piling up and overflowing the memory. Therefore, time is used here to achieve the automatic release of the lock of the object data, also known as passive release. Figure 3 Describes the process of releasing a lock.

[0094] Figure 3 A flow chart of another data processing method provided in an embodiment of the present application is shown as follows: Figure 3As shown, the figure includes an interceptor and a Redis database. The process may include: whenever the server receives a delete operation or a write operation on an object data, it will automatically read the lock of the object data from the Redis database. If the lock of the object data is obtained, the object data will be locked, that is, the object data will be locked, and the value of the lock of the object data will be set to System.currentTimeMillis()+t, where System.currentTimeMillis() represents the current time and t represents the release time of the lock; when the lock release time t is set to 5 minutes, it means that the lock will be automatically released after 5 minutes. If the lock is not obtained after 5 minutes, wait for t time and request to obtain the lock again. When steps (1)(2)(3)(4) are successfully executed, the system will release the lock, that is, actively release the current lock. Generally speaking, if there is no exception or interruption in the program execution process, the lock will be actively released and will return to the Redis database after being released.

[0095] For example, for the above two situations where the locks cannot be released actively, the following targeted processing can be performed. For the first situation, the exception can be captured and the log can be printed; for the second situation, the duration of the intermediate execution of steps (1)(2)(3)(4) is monitored. If the lock release time t is exceeded, an exception is thrown and captured. Here, steps (1)(2) and (3)(4) can also be added to the transaction operation at the same time. When an exception occurs, the write and delete operations on the object data are rolled back.

[0096] As can be seen, the data processing method provided by the embodiments of this application optimizes parameter settings, dynamically adjusts lock release times, saves the cost of manual intervention, and reduces redundant parameter adjustment work. Furthermore, the dynamic lock-based storage process for Redis hierarchical data can provide a universal storage solution for a wider range of business scenarios and has considerable potential for application.

[0097] Figure 4 A structural diagram of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device includes: an interception module 400, a statistics module 401, a determination module 402 and an execution module 403, wherein:

[0098] The interception module 400 is used to intercept multiple data operation requests received by the server in the current time period; the data operation requests are used to write or delete the operation data;

[0099] A statistics module 401 is configured to count data inflow rates corresponding to a plurality of operation data completed within the current time period; the data inflow rate includes at least one of a data write rate and a data delete rate;

[0100] A determination module 402 is configured to determine, based on the data inflow rate, a lock release time corresponding to each target data operation request received within a next time period;

[0101] The execution module 403 is configured to execute an operation corresponding to each target data operation request based on the determined lock release time.

[0102] In some embodiments, when the data inflow rate includes a data write rate and a data delete rate, the determining module 402 is configured to determine, based on the data inflow rate, a lock release time corresponding to each target data operation request received in a next time period, including:

[0103] Obtain the write release time and delete release time of the lock in the current time period;

[0104] Determining a first ratio of the data write rate to a write release time of a lock in the current time period, and a second ratio of the data delete rate to a delete release time of a lock in the current time period;

[0105] Based on the first ratio and the second ratio, a lock release time corresponding to each target data operation request received within the next time period is determined.

[0106] In some embodiments, the determining module 402 is configured to determine, based on the first ratio and the second ratio, a lock release time corresponding to each target data operation request received within the next time period, including:

[0107] Adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal;

[0108] Based on the adjusted first ratio and the second ratio, a lock release time corresponding to each target data operation request received in the next time period is determined.

[0109] In some embodiments, the execution module 403 is configured to execute an operation corresponding to each target data operation request based on the determined lock release time, including:

[0110] For each target data operation request, acquiring a lock of the target operation data; the target data operation request is used to perform a write operation or a delete operation on the target operation data;

[0111] Based on the determined lock release time, a locking operation is performed on the target operation data, and a corresponding operation is performed on the target operation data according to the target data operation request.

[0112] In some embodiments, the execution module 403 is configured to acquire a lock on target operation data, including:

[0113] Obtain a lock for the target operation data from a Redis database; the lock for the target operation data is a key-value lock, the key value is an identifier for the target operation data, and the value value is a release time for the lock for the target operation data.

[0114] In some embodiments, the apparatus further includes a monitoring module, wherein the monitoring module is configured to:

[0115] The execution process corresponding to each target data operation request is monitored until the monitoring duration reaches the determined lock release time, and the lock of the target operation data is released.

[0116] In some embodiments, the apparatus further includes a rollback module, the rollback module being configured to:

[0117] When the monitoring time reaches the determined lock release time and it is detected that the lock of the target operation data is not released, a rollback operation is performed on the execution of the target operation data.

[0118] In actual applications, the above-mentioned interception module 400, statistics module 401, determination module 402, execution module 403, monitoring module and rollback module can all be implemented by a processor located in an electronic device, and the processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0119] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0120] If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the part that contributes to the relevant technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program code.

[0121] Specifically, the computer program instructions corresponding to a data processing method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the computer program instructions corresponding to a data processing method in the storage medium are read or executed by an electronic device, any data processing method in the aforementioned embodiments is implemented.

[0122] Based on the same technical concept as the above embodiment, see Figure 5 , which shows an electronic device 500 provided in an embodiment of the present application, which may include: a memory 501 and a processor 502; wherein,

[0123] Memory 501, used to store computer programs and data;

[0124] The processor 502 is configured to execute the computer program stored in the memory to implement any one of the data processing methods in the aforementioned embodiments.

[0125] In practical applications, the memory 501 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 502.

[0126] The processor 502 may be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor. It is understood that for different data processing systems, the electronic device used to implement the processor functions may also be other, and this embodiment of the application does not specifically limit this.

[0127] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0128] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0129] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0130] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0131] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0135] The above are merely preferred embodiments of the present application and are not intended to limit the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Intercept multiple data operation requests received by the server in the current time period; the data operation requests are used to write or delete the operation data; Counting the data inflow rate corresponding to the multiple operation data completed in the current time period; the data inflow rate includes at least one of a data writing rate and a data deletion rate; Determining, based on the data inflow rate, a lock release time corresponding to each target data operation request received in a next time period; executing an operation corresponding to each target data operation request based on the determined lock release time; In a case where the data inflow rate includes a data write rate and a data delete rate, determining, based on the data inflow rate, a lock release time corresponding to each target data operation request received in a next time period, includes: Obtain the write release time and delete release time of the lock in the current time period; Determining a first ratio of the data write rate to a write release time of a lock in the current time period, and a second ratio of the data delete rate to a delete release time of a lock in the current time period; Based on the first ratio and the second ratio, a lock release time corresponding to each target data operation request received within the next time period is determined.

2. The method according to claim 1, wherein determining the lock release time corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio comprises: Adjusting the first ratio and the second ratio so that the first ratio and the second ratio are equal; Based on the adjusted first ratio and the second ratio, a lock release time corresponding to each target data operation request received in the next time period is determined.

3. The method according to claim 1, characterized in that The performing of an operation corresponding to each target data operation request based on the determined lock release time includes: For each target data operation request, acquiring a lock of the target operation data; the target data operation request is used to perform a write operation or a delete operation on the target operation data; Based on the determined lock release time, a locking operation is performed on the target operation data, and a corresponding operation is performed on the target operation data according to the target data operation request.

4. The method according to claim 3, characterized in that The acquiring of a lock for target operation data includes: Obtain a lock for the target operation data from the Redis database; the lock for the target operation data is a key-value lock, where the key value is the identifier of the target operation data and the value value is the release time of the lock for the target operation data.

5. The method according to claim 3, characterized in that The method further comprises: The execution process corresponding to each target data operation request is monitored until the monitoring duration reaches the determined lock release time, and the lock of the target operation data is released.

6. The method according to claim 5, characterized in that The method further comprises: When the monitoring time reaches the determined lock release time and it is detected that the lock of the target operation data is not released, a rollback operation is performed on the execution of the target operation data.

7. A data processing device, characterized in that: The device comprises: An interception module, configured to intercept multiple data operation requests received by the server within a current time period; the data operation requests are used to write or delete the operation data; A statistics module, configured to count data inflow rates corresponding to a plurality of operation data completed within the current time period; the data inflow rate includes at least one of a data writing rate and a data deletion rate; a determination module, configured to determine, based on the data inflow rate, a lock release time corresponding to each target data operation request received within a next time period; an execution module, configured to execute an operation corresponding to each target data operation request based on the determined lock release time; The determination module is further used to obtain the write release time and delete release time of the lock in the current time period when the data inflow rate includes the data write rate and the data deletion rate; determine a first ratio of the data write rate and the write release time of the lock in the current time period, and a second ratio of the data deletion rate and the delete release time of the lock in the current time period; and determine the lock release time corresponding to each target data operation request received in the next time period based on the first ratio and the second ratio.

8. An electronic device, characterized in that: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any one of claims 1 to 6 when executing the program.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images