Data processing method, system and apparatus, computer device, readable storage medium, and program product

Abstract

A data processing method, comprising: acquiring a database operation log of a first database; determining a data update operation of the first database on the basis of the database operation log of the first database; and when the data update operation comprises an archiving flagging operation and a deletion operation on data to be updated, retaining said data in a second database, wherein the archiving flagging operation and the deletion operation are operations triggered in a target situation, and the target situation comprises a situation of archiving and deleting data, satisfying an archiving rule, in the first database.

Description

Data processing method, system, device, computer device, readable storage medium and program product

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 2024108240702, filed on June 24, 2024, and entitled "Data processing method, system, computer device, readable storage medium and program product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of data processing, and in particular to a data processing method, system, device, computer device, computer readable storage medium and computer program product. BACKGROUND

[0004] In the logistics industry, a large amount of logistics data is generated along with the development of logistics business. Logistics institutions usually need to store a large amount of logistics data for users to query and use.

[0005] Among them, due to the high concurrency of a large amount of data and the storage of the data, the stability is poor, and therefore in the traditional technology, a data double-writing method is usually used to store the logistics data into different databases, and different databases are used to meet the different data needs of users. However, this method is difficult to ensure the consistency between the databases, and the synchronous response is time-consuming, which is difficult to meet the needs of the logistics data calling and storage. SUMMARY

[0006] According to various embodiments of the present application, a data processing method, system, device, computer device, computer readable storage medium and computer program product are provided.

[0007] In a first aspect, the present application provides a data processing method, comprising:

[0008] obtaining a database operation log of a first database;

[0009] determining a data update operation of the first database according to the database operation log of the first database;

[0010] when the data update operation includes an archiving marking operation and a deletion operation on to-be-updated data, retaining the to-be-updated data in a second database; wherein the archiving marking operation and the deletion operation are operations triggered under a target condition, and the target condition includes a case of archiving and deleting data in the first database that meets an archiving rule.

[0011] In one of the embodiments, the archiving rule comprises: a creation time length of the data is greater than a first time length threshold.

[0012] In one of the embodiments, the method further comprises: determining a creation time length range of the data to be queried according to a data query request; when a maximum creation time length in the creation time length range is not greater than a second time length threshold, performing a first query operation on the data to be queried in the first database; wherein the second time length threshold is not greater than the first time length threshold; when the maximum creation time length in the creation time length range is greater than the second time length threshold, performing a second query operation on the data to be queried in the second database.

[0013] In one of the embodiments, the method further comprises: monitoring a capacity of the first database; when the capacity of the first database exceeds a capacity threshold, reducing the second time length threshold.

[0014] In one of the embodiments, the determining the data update operation of the first database according to the database operation log of the first database comprises: capturing a change content of the database operation log; determining the data update operation according to the change content; the method further comprises: when the data update operation comprises a deletion operation on the data to be updated, judging whether the data update operation comprises an archiving marking operation on the data to be updated.

[0015] In one of the embodiments, the first database comprises a relational database, and the second database comprises a non-relational database.

[0016] In a second aspect, the application further provides a server, a first database and a second database; wherein,

[0017] The server is configured to perform a data change operation on the first database according to a data change request.

[0018] The first database is configured to store data written by the server performing the data change operation.

[0019] The server is further configured to perform an archiving marking operation and a deletion operation on the data in the first database under a target condition; wherein the target condition comprises: an archiving deletion condition of data in the first database meeting an archiving rule.

[0020] The server is further configured to acquire a database operation log of the first database; determine a data update operation of the first database according to the database operation log of the first database; when the data update operation comprises an archiving marking operation and a deletion operation on the data to be updated, retain the data to be updated in the second database.

[0021] The second database is configured to store all data stored in the first database.

[0022] In a third aspect, the present application provides a data processing apparatus, comprising:

[0023] an obtaining module configured to obtain a database operation log of a first database;

[0024] a determining module configured to determine a data update operation of the first database according to the database operation log of the first database;

[0025] an operating module configured to, when the data update operation comprises an archiving marking operation and a deletion operation on to-be-updated data, retain the to-be-updated data in a second database; wherein the archiving marking operation and the deletion operation are operations triggered under a target condition, and the target condition comprises a condition that data in the first database that meets an archiving rule is deleted by archiving.

[0026] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following operations when executing the computer program:

[0027] obtaining a database operation log of a first database;

[0028] determining a data update operation of the first database according to the database operation log of the first database;

[0029] when the data update operation comprises an archiving marking operation and a deletion operation on to-be-updated data, retaining the to-be-updated data in a second database; wherein the archiving marking operation and the deletion operation are operations triggered under a target condition, and the target condition comprises a condition that data in the first database that meets an archiving rule is deleted by archiving.

[0030] In a fifth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following operations:

[0031] obtaining a database operation log of a first database;

[0032] determining a data update operation of the first database according to the database operation log of the first database;

[0033] when the data update operation comprises an archiving marking operation and a deletion operation on the to-be-updated data, retaining the to-be-updated data in the second database; wherein the archiving marking operation and the deletion operation are operations triggered in a target case, and the target case comprises a case of archiving and deleting data in the first database that meets an archiving rule.

[0034] In a sixth aspect, the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the following operations:

[0035] obtaining a database operation log of the first database;

[0036] determining a data update operation of the first database according to the database operation log of the first database;

[0037] when the data update operation comprises an archiving marking operation and a deletion operation on the to-be-updated data, retaining the to-be-updated data in the second database; wherein the archiving marking operation and the deletion operation are operations triggered in a target case, and the target case comprises a case of archiving and deleting data in the first database that meets an archiving rule.

[0038] The above data processing method, system, device, computer equipment, computer readable storage medium and computer program product first obtain a database operation log of the first database, and then determine a data update operation of the first database according to the database operation log of the first database. When the data update operation comprises an archiving marking operation and a deletion operation on the to-be-updated data, the to-be-updated data in the second database is retained. In this scheme, by obtaining the database operation log of the first database and determining the data update operation thereof, and then retaining the to-be-updated data in the second database when the data update operation comprises an archiving marking operation and a deletion operation on the to-be-updated data, the data deletion caused by data archiving in the first database can be distinguished from the data deletion actively performed by the user in the target case of archiving and deleting data in the first database that meets an archiving rule. Thus, by determining the data update operation of the first database from the database operation log thereof, and performing subsequent data processing on the second database according to the data update operation, the full data of the first database can be synchronized to the second database, and consistency can be ensured in a low time consumption case and archiving data can be prevented from being mistakenly deleted in the second database, effectively ensuring data security. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on the disclosed drawings are within the scope of the present application.

[0040] Fig. 1 is an application environment diagram of a data processing method in an embodiment;

[0041] Fig. 2 is a flow diagram of a data processing method in an embodiment;

[0042] Fig. 3 is an application environment diagram of a data processing method in another embodiment;

[0043] Fig. 4 is a flow diagram of data query in an embodiment;

[0044] Fig. 5 is a structural diagram of a data processing system in an embodiment;

[0045] Fig. 6 is a structural diagram of a data processing system in another embodiment;

[0046] Fig. 7 is a structural block diagram of a data processing apparatus in an embodiment;

[0047] Fig. 8 is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of the present application.

[0049] The data processing method provided by the embodiments of the present application can be applied in the application environment as shown in Fig. 1. The data processing system can include a server, a first database and a second database.

[0050] The first database can be a hot database and can be used to store recently created data, and the second database can be a full database and can be used to store all data stored in the first database. The server can be used to perform data change operations on the first database according to a data change request, and the first database can add, modify, or delete corresponding data according to the data change operations performed by the server. For example, the data change request can be sent by a user to the server through a terminal. The data change request can include adding, modifying, deleting, or the like, and the server can perform adding, modifying, deleting, or the like according to the received data change request. The server can also update the data change in the first database to the second database by performing a data update operation, so that the data consistency between the two databases is ensured.

[0051] In an example embodiment, as shown in FIG. 2, a data processing method is provided, which is applied to the server in FIG. 1 as an example, and includes the following operations:

[0052] Operation S201, obtaining a database operation log of a first database.

[0053] The first database can be a database used to store recent logistics data generated in the process of logistics business development. It can store, for example, recent logistics orders and the like. The database operation log of the first database is a log generated after the data of the first database is changed, which can record, for example, adding operations, modifying operations, deleting operations, archiving marking operations, and the like. These data change operations can include operations performed by the server according to received data change requests of users, or operations performed by the server under a target condition.

[0054] For example, the server can trigger archiving marking operations and deleting operations on the data in the first server under a target condition. The target condition can include archiving and deleting data in the first database that meets the archiving rule.

[0055] Specifically, the server can determine data that meets the archiving rule in the first database according to the archiving rule. Then, archiving marking operations can be performed on these data, and then deleting operations can be performed. For example, the server can repeatedly perform this process according to a preset execution period through a timing task.

[0056] The archiving rule can include a rule about a data creation duration, for example, the data creation duration is greater than a certain duration threshold, and the data meets the archiving rule. The archiving rule can also include a rule about data access frequency, for example, the data access frequency is less than a certain frequency threshold, and the data meets the archiving rule. The archiving rule can also be a rule formulated according to other properties of the data.

[0057] The archiving marking operation performed on the data can be marking the data to indicate that the data belongs to the data to be archived. For example, an archiving marking bit can be set for each item of data in the first database, and the default state of the archiving marking bit is set to 0. Then, when the archiving marking operation is performed on a certain item of data, the archiving marking bit corresponding to the item of data is updated to 1, thereby marking the item of data.

[0058] In operation S202, a data update operation of the first database is determined according to the database operation log of the first database.

[0059] The obtained database operation log of the first database can be compared with the previously obtained database operation log, and the newly added part is used to determine the data update operation of the first database.

[0060] It can be understood that after the data update operation of the first database is determined, the data synchronization from the first database to the second database can be realized by performing a corresponding data update operation on the second database.

[0061] For example, when the data update operation of the first database includes an addition operation on the to-be-updated data, the addition operation on the to-be-updated data can be performed in the second database correspondingly. When the data update operation of the first database includes a modification operation on the to-be-updated data, the modification operation on the to-be-updated data can be performed in the second database correspondingly.

[0062] When the data update operation of the first database includes a deletion operation on the to-be-updated data, it can be further determined whether the data update operation includes an archiving marking operation on the to-be-updated data, so as to distinguish whether the to-be-updated data is deleted due to being archived under the triggering of the target condition or is deleted by the user actively. When the data update operation includes the archiving marking operation and the deletion operation, operation S203 can be performed.

[0063] In an example embodiment, the operation of determining the data update operation of the first database according to the database operation log of the first database can include capturing the change content of the database operation log, and determining the data update operation according to the change content. The method can further include determining whether the data update operation includes a deletion operation on the to-be-updated data, and determining whether the data update operation includes an archiving marking operation on the to-be-updated data.

[0064] In the embodiment, the data capture technology can be used to capture the change content in real time when the database operation log is changed, and determine the corresponding data update operation according to the change content, and then the data update operation can be processed in real time. Then, in the process of processing the data update operation, when it is determined that the data update operation includes a delete operation on the to-be-updated data, it can be further determined whether it includes an archiving mark operation on the to-be-updated data, so as to distinguish different data deletion situations.

[0065] In the above process, by capturing the change content of the database operation log in real time and determining the data update operation according to the change content, the change of the first database can be sensed in time and the second database can be updated in time, so as to ensure the consistency of the second database and the first database. After it is determined that the data update operation includes a delete operation, it is further determined whether it includes an archiving mark operation, so as to effectively distinguish different data deletion situations in the first database and avoid mistakenly deleting the archived data in the second database.

[0066] In operation S203, when the data update operation includes an archiving mark operation and a delete operation on the to-be-updated data, the to-be-updated data in the second database is retained.

[0067] When the data update operation includes an archiving mark operation and a delete operation on the to-be-updated data, it can be determined that the data update operation is triggered in the target situation, that is, the to-be-updated data belongs to the data that is archived and deleted according to the archiving rule. Based on this, the to-be-updated data can be retained in the second database in the operation.

[0068] It can be understood that when the data update operation includes a delete operation on the to-be-updated data and does not include an archiving mark operation, it can be considered that the data update operation is not triggered in the target situation, that is, the to-be-updated data does not belong to the archived data, but is the data deleted by the user. Based on this, the server can execute a corresponding delete operation on the to-be-updated data in the second database, so as to ensure the consistency of the second database and the first database.

[0069] In the data processing method, the database operation log of the first database is acquired, the data update operation of the first database is determined according to the database operation log of the first database, and in the case that the data update operation includes the archiving mark operation and the deletion operation on the to-be-updated data, the to-be-updated data in the second database is retained. In this scheme, by acquiring the database operation log of the first database and determining the data update operation thereof, and in the case that the data update operation includes the archiving mark operation and the deletion operation on the to-be-updated data, the to-be-updated data in the second database is retained, the data deletion caused by data archiving in the first database can be distinguished from the data deletion actively performed by the user in the case that the target data including the data meeting the archiving rule in the first database is deleted for archiving. Therefore, by determining the data update operation of the first database from the database operation log thereof and performing subsequent data processing on the second database according to the data update operation, the full data of the first database can be synchronized to the second database, consistency can be ensured in a low time consumption, and the misdeletion of the archived data in the second database can be avoided, thereby effectively ensuring data security.

[0070] In an example embodiment, the archiving rule can include that the creation duration of the data is greater than a first duration threshold.

[0071] In this embodiment, the archiving rule can include that the creation duration of the data is greater than a first duration threshold. The first duration threshold can be a preset time length, for example, 60 days.

[0072] Based on the archiving rule, in this embodiment, whether there is data with a creation duration greater than the first duration threshold can be determined according to the creation duration of each item of data in the first database. If there is data with a creation duration greater than the first duration threshold, the corresponding data can be confirmed as to-be-archived data meeting the archiving rule. Then, for each item of to-be-archived data, an archiving mark operation and a deletion operation can be respectively performed thereon to complete the archiving processing of the item of to-be-archived data.

[0073] In this embodiment, by performing the archiving mark operation and the deletion operation on the data in the first database according to the archiving rule that the creation duration is greater than the first duration threshold, the older data can be deleted and the newer data can be retained in the first database, so that the new data that is more likely to be used can be retained in the first database while reducing the capacity of the first database, thereby facilitating the user to change the data.

[0074] In an example embodiment, as shown in FIG. 3, in the data processing system, the server can further receive a data query request from the user, and perform a first query operation in the first database or perform a second query operation in the second database.

[0075] In the embodiment of the present application, the data processing method can further include the following operations, as shown in FIG. 4.

[0076] In operation S401, the creation time length range of the data to be queried is determined according to the data query request.

[0077] The data query request can include parameters for data query, which can be submitted by the user through interaction with the terminal and sent to the server by the terminal.

[0078] For example, the parameters included in the data query request can include, but are not limited to, the creation time length range of the data to be queried, etc. In this operation, the creation time length range of the data to be queried can be directly obtained according to the parameters included in the data query request.

[0079] In other embodiments, the parameters included in the data query request can not include the parameter of the creation time length range, but include parameters related to the creation time. Based on this, the corresponding creation time length range can be calculated according to the distance between the creation time and the current time. For example, when the data query request includes the earliest creation time and the latest creation time of the data to be queried, the maximum creation time length and the minimum creation time length of the data to be queried can be calculated accordingly, so as to determine the creation time length range.

[0080] Based on the creation time length range obtained in the foregoing process, the maximum creation time length corresponding to the data to be queried can be further determined. For example, when the creation time length range is one year, the maximum creation time length is one year; when the creation time length range is between x days and y days (x>y), the maximum creation time length is x days. Then, the maximum creation time length of the data to be queried can be compared with the preset second time length threshold, and the operation S402 or the operation S403 can be performed according to the comparison result. The second time length threshold can be equal to the first time length threshold, or less than the first time length threshold.

[0081] In operation S402, when the maximum creation time length in the creation time length range is not greater than the second time length threshold, a first query operation on the data to be queried is performed in the first database.

[0082] When the maximum creation time length in the creation time length range is not greater than the second time length threshold, it can be determined that the maximum creation time length of the data to be queried is also not greater than the first time length threshold, so the data to be queried still exists in the first database and has not been archived and deleted. Based on this, the first query operation can be performed in the first database to obtain the data to be queried.

[0083] Operation S403, when the maximum creation time length in the creation time length range is greater than the second time length threshold, performing a second query operation on the to-be-queried data in the second database.

[0084] Wherein, when the maximum creation time length in the creation time length range is greater than the second time length threshold, the to-be-queried data can have been archived and does not exist in the first database. Based on this, the second query operation can be performed in the second database storing full data to obtain the to-be-queried data in the operation.

[0085] In the embodiment, for the data query request of the user, the creation time length range of the to-be-queried data is determined first, and then the query operation is determined to be performed in the first database or the second database according to the size relationship between the maximum creation time length in the creation time length range and the second time length threshold, which can divert the data query of the user and improve the overall efficiency.

[0086] In an exemplary embodiment, the above method can further include monitoring the capacity of the first database; and when the capacity of the first database exceeds a capacity threshold, reducing the second time length threshold.

[0087] Specifically, although the archiving marking operation and the deleting operation on the data in the first database that meets the archiving rule in the above method can effectively reduce the capacity of the first database to maintain its performance, the first database can still have a problem of capacity being too large to affect the performance due to, for example, a large amount of data being written in a short time.

[0088] For this case, in the embodiment, the capacity threshold of the first database can be set according to the relationship between the capacity and the performance of the first database, and the capacity of the first database can be monitored in real time. Wherein, when the capacity of the first database is monitored to exceed the capacity threshold, it can be determined that the performance of the first database can be reduced. Based on this, the data query request of the user can be more converted into the second data query operation performed in the second database by reducing the second time length threshold, and the first data query operation performed in the first database can be reduced.

[0089] In the embodiment, by monitoring the capacity of the first database and reducing the second time length threshold when the capacity of the first database exceeds the capacity threshold, part of the query originally performed in the first database can be transferred to the second database, so that the operation required to be performed by the first database can be reduced in the case of the performance of the first database being affected, and the response efficiency of the first database can be guaranteed.

[0090] In an exemplary embodiment, the first database can include a relational database, and the second database can include a non-relational database.

[0091] The first database and the second database can be different types of databases. For example, the first database can be a relational database, and the second database can be a non-relational database such as a file-type database.

[0092] In this embodiment, the relational database is used as the first database, and the characteristics of the relational database, such as easy to use and maintain, can be used to conveniently meet the user's change requirements for the logistics business data. The non-relational database is used as the second database, and the characteristics of the non-relational database, such as high scalability and high query efficiency, can be used to meet the storage and query requirements of the full data. Therefore, the combination of different types of databases in this embodiment can meet the high-concurrency calling and storage of a large amount of data as a whole, thereby ensuring the stable operation of the system and improving the user's experience.

[0093] To further illustrate the data processing method of the present application, the following will be described in detail by embodiments.

[0094] Exemplarily, the application environment of the method of this embodiment can be as shown in FIG. 3. The data processing system can include a first database, a second database and a server. The first database can be a relational database, and the second database can be a non-relational database. The first database and the server can be located in a cloud platform that provides logistics data services to users, and the second database can be located in a big data platform.

[0095] The data processing method in this embodiment can be applied to the server of the data processing system.

[0096] The server can receive a data change request of a user, and perform a corresponding data change operation on the first database according to the data change request. The data change request of the user can include adding, modifying, deleting, etc., and the data change operation performed by the server can include adding operation, modifying operation, deleting operation, etc. The first database can generate a database operation log according to the data change operation.

[0097] The data change operation performed by the server can also include an archiving marking operation and a deleting operation on the data in the first database under the triggering of a target condition.

[0098] Specifically, the target case can include a case of archiving and deleting data in the first database that meets an archiving rule. The server can perform timed archiving and deletion of data in the first database that meets the archiving rule by configuring a task execution timing manner, that is, performing an archiving marking operation and a deletion operation on the data. In this embodiment, the archiving rule can be that the creation duration of the data is greater than a first duration threshold (for example, 60 days). The server can determine data in the first database whose creation duration is greater than 60 days, and perform an archiving marking operation on the data first, update the state of the archiving marking bit of the data to 1, and then perform a deletion operation on the data in the first database. It can be understood that the first database can also generate a database operation log according to the archiving marking operation and the deletion operation.

[0099] The server can also use data capture technology to consume the database operation log generated by the first database, capture the change content of the database operation log in real time, obtain the data update operation corresponding to the change content, and perform a corresponding operation in the second database.

[0100] For example, when the data update operation includes an addition operation on the data to be changed, the server can perform an addition operation on the data to be changed in the second database. When the data update operation includes a modification operation on the data to be changed, the server can perform a modification operation on the data to be changed in the second database.

[0101] When the data update operation includes a deletion operation on the data to be changed, the server can further perform logical judgment to determine how to process the data to be changed in the second database by judging whether the data update operation includes an archiving marking operation on the data to be changed. If it is judged that the data update operation includes an archiving marking operation, the server can not process the data to be changed in the second database, so that the data to be changed is retained in the second database. If it is judged that the data update operation does not include an archiving marking operation, the server can perform a deletion operation on the data to be changed in the second database, so as to maintain the synchronization of the second database and the first database.

[0102] The server can also receive a data query request of a user, and judge the creation duration range of the data to be queried by the user. Then, the maximum creation duration in the creation duration range can be compared with a set second duration threshold, and the data query can be routed to different databases according to the comparison result.

[0103] Exemplarily, the second time length threshold in this embodiment can be 60 days. When the maximum creation time length corresponding to the data query request does not exceed 60 days, the data query can be routed to the first database by default, and the to-be-queried data can be obtained by performing the first query operation in the first database. When the maximum creation time length corresponding to the data query request exceeds 60 days, the data query can be routed to the second database, and the to-be-queried data can be obtained by performing the second query operation in the second database.

[0104] Further, the server can also monitor the capacity of the first database, and when the capacity exceeds a preset capacity threshold, the second time length threshold can be reduced, so that part of the queries originally performed in the first database can be migrated to the second database.

[0105] In this embodiment, by performing the archiving marking operation and the deletion operation on the data in the first database that meets the archiving rule under the trigger of the target condition of archiving and deleting the data in the first database that meets the archiving rule, and determining whether to delete the to-be-updated data in the second database according to whether the data updating operation contains the archiving marking operation and the deletion operation on the to-be-updated data, the data actively deleted by the user and the data deleted in the archiving process can be distinguished, and the accidental deletion of the archiving data in the second database can be effectively avoided. Moreover, in this embodiment, the second database is updated according to the database operation log of the first database, so that the operation on the first database and the update on the second database can be decoupled, and the operation on the two databases does not need to be performed at the same time, so that the response time of the system can be reduced and the efficiency can be improved while ensuring the consistency of the two databases. Moreover, this embodiment also distinguishes the to-be-queried data according to the maximum creation time length of the to-be-queried data according to the data query request of the user, and performs the query of the cold data or the hot data in different databases, and dynamically adjusts the boundary of distinguishing the cold data and the hot data according to the capacity of the first database, so that the query isolation of the cold data and the hot data can be implemented without the user's awareness, the performance influence on the first database can be reduced, the query efficiency of the user can be improved, and the user experience can be improved.

[0106] It should be understood that, although the operations in the flowcharts related to the embodiments as described above are shown in a sequential order, these operations are not necessarily performed in the order shown by the arrows. Unless explicitly stated otherwise herein, the operations can be performed in other orders. Moreover, at least some of the operations involved in the flowcharts related to the embodiments as described above can include multiple operations or stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the operations or stages can not be sequential, but can be round-robin or alternating with at least some of the operations or stages of other operations or stages.

[0107] Based on the same inventive concept, the embodiments of the present application also provide a data processing system for implementing the above-mentioned data processing method. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more data processing system embodiments provided below can refer to the limitations of the data processing method described above, which will not be repeated here.

[0108] In an exemplary embodiment, as shown in FIG. 5, a data processing system is provided, comprising: a server, a first database, a second database; wherein,

[0109] The server is configured to perform a data change operation on the first database according to a data change request;

[0110] The first database is configured to store data written by the server performing the data change operation;

[0111] The server is further configured to perform an archiving marking operation and a deletion operation on the data in the first database under a target condition; wherein the target condition includes the case of archiving and deleting the data in the first database that meets the archiving rule;

[0112] The server is further configured to obtain a database operation log of the first database, determine a data update operation of the first database according to the database operation log of the first database, and retain the to-be-updated data in the second database when the data update operation includes an archiving marking operation and a deletion operation on the to-be-updated data;

[0113] The second database is configured to store all the data stored in the first data.

[0114] In an exemplary embodiment, the archiving rule includes that the creation duration of the data is greater than a first duration threshold.

[0115] In an example embodiment, as shown in FIG. 6, the server is further configured to: determine a creation time range of the data to be queried according to the data query request; when a maximum creation time in the creation time range is not greater than a second time threshold, perform a first query operation on the data to be queried in the first database; wherein the second time threshold is not greater than the first time threshold; when the maximum creation time in the creation time range is greater than the second time threshold, perform a second query operation on the data to be queried in the second database.

[0116] In an example embodiment, the server is further configured to: monitor a capacity of the first database; when the capacity of the first database exceeds a capacity threshold, reduce the second time threshold.

[0117] In an example embodiment, the server is further configured to: capture a change content of the database operation log; determine the data update operation according to the change content; when the data update operation includes a deletion operation on the data to be updated, determine whether the data update operation includes an archiving marking operation on the data to be updated.

[0118] In an example embodiment, the first database comprises a relational database, and the second database comprises a non-relational database.

[0119] Based on the same inventive concept, the embodiments of the present application also provide a data processing apparatus for implementing the data processing method described above. The implementation scheme of the problem solving provided by the apparatus is similar to the implementation scheme described in the above method, so the specific limitations in one or more data processing apparatus embodiments provided below can refer to the limitations of the data processing method described above, which will not be repeated here.

[0120] In an example embodiment, as shown in FIG. 7, a data processing apparatus is provided, comprising:

[0121] The acquisition module 701 is configured to acquire a database operation log of a first database.

[0122] The determination module 702 is configured to determine a data update operation of the first database according to the database operation log of the first database.

[0123] The operation module 703 is configured to, when the data update operation includes an archiving marking operation and a deletion operation on the data to be updated, retain the data to be updated in the second database; wherein the archiving marking operation and the deletion operation are operations triggered in a target condition, and the target condition includes a case of archiving and deleting data in the first database that meets an archiving rule.

[0124] In an example embodiment, the archiving rule comprises: a creation time length of the data is greater than a first time length threshold.

[0125] In an example embodiment, the apparatus further comprises: a range determining module, configured to determine a creation time length range of the data to be queried according to the data query request; a first query module, configured to perform a first query operation on the data to be queried in the first database when a maximum creation time length in the creation time length range is not greater than a second time length threshold; wherein the second time length threshold is not greater than the first time length threshold; and a second query module, configured to perform a second query operation on the data to be queried in the second database when the maximum creation time length in the creation time length range is greater than the second time length threshold.

[0126] In an example embodiment, the apparatus further comprises: a capacity monitoring module, configured to monitor a capacity of the first database; and an adjusting module, configured to reduce the second time length threshold when the capacity of the first database exceeds a capacity threshold.

[0127] In an example embodiment, the determining module 702 is further configured to: capture a change content of the database operation log; and determine the data update operation according to the change content; and the apparatus further comprises: a judging module, configured to judge whether the data update operation comprises an archiving mark operation on the data to be updated when the data update operation comprises a deletion operation on the data to be updated.

[0128] In an example embodiment, the first database comprises a relational database, and the second database comprises a non-relational database.

[0129] Each module in the data processing apparatus can be realized by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0130] In an example embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 8. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data such as archiving rules. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement a data processing method.

[0131] Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In an example embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the operations in the above method embodiments.

[0133] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the operations in the above method embodiments.

[0134] In an example embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the operations in the above method embodiments.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant regulations.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0137] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0138] The above embodiments only express several implementation ways of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve the database operation logs of the first database; Based on the database operation log of the first database, determine the data update operation of the first database; and, When the data update operation includes an archive marking operation and a deletion operation for the data to be updated, the data to be updated in the second database is retained; wherein, the archive marking operation and the deletion operation are operations triggered under a target condition, the target condition includes: the case where data in the first database that meets the archive rules is archived and deleted.

2. The method according to claim 1, characterized in that, The archiving rules include: The data creation time exceeds the first duration threshold.

3. The method according to claim 2, characterized in that, The method further includes: Based on the data query request, determine the creation time range of the data to be queried; When the maximum creation time within the creation time range is not greater than the second time threshold, a first query operation on the data to be queried is performed in the first database; wherein, the second time threshold is not greater than the first time threshold; and, When the maximum creation time within the creation time range exceeds the second time threshold, a second query operation on the data to be queried is performed in the second database.

4. The method according to claim 3, characterized in that, The method further includes: Monitor the capacity of the first database; and, When the capacity of the first database exceeds the capacity threshold, the second duration threshold is reduced.

5. The method according to claim 1, characterized in that, The step of determining the data update operation of the first database based on the database operation log of the first database includes: Capture the changes in the database operation log; Based on the changes, determine the data update operation; and, The method further includes: When the data update operation includes a deletion operation of the data to be updated, determine whether the data update operation includes an archive marking operation of the data to be updated.

6. The method according to any one of claims 1 to 5, characterized in that, The first database includes a relational database, and the second database includes a non-relational database.

7. A data processing system, characterized in that, The system includes: a server, a first database, and a second database; wherein, The server is configured to perform data change operations on the first database according to the data change request; The first database is used to store data written by the server performing the data change operation; The server is also configured to perform archiving and deletion operations on data in the first database when triggered by a target condition; wherein the target condition includes: archiving and deleting data in the first database that conforms to the archiving rules; The server is further configured to obtain the database operation log of the first database; determine the data update operation of the first database based on the database operation log of the first database; and retain the data to be updated in the second database when the data update operation includes an archive marking operation and a deletion operation for the data to be updated. The second database is used to store all the data stored in the first database.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the database operation logs of the first database; The determination module is used to determine the data update operation of the first database based on the database operation log of the first database; An operation module is configured to retain the data to be updated in the second database when the data update operation includes an archive marking operation and a deletion operation for the data to be updated; wherein the archive marking operation and the deletion operation are operations triggered under a target condition, the target condition including: the case of archive deleting data in the first database that meets the archive rules.

9. The apparatus according to claim 8, characterized in that, The archiving rules include: The data creation time exceeds the first duration threshold.

10. The apparatus according to claim 9, characterized in that, The device further includes: The range determination module is used to determine the creation time range of the data to be queried based on the data query request; The first query module is used to perform a first query operation on the data to be queried in the first database when the maximum creation time within the creation time range is not greater than a second time threshold; wherein the second time threshold is not greater than the first time threshold. The second query module is used to perform a second query operation on the data to be queried in the second database when the maximum creation time in the creation time range is greater than the second time threshold.

11. The apparatus according to claim 10, characterized in that, The device further includes: The capacity monitoring module is used to monitor the capacity of the first database; An adjustment module is used to reduce the second duration threshold when the capacity of the first database exceeds the capacity threshold.

12. The apparatus according to claim 8, characterized in that, The determining module is further configured to capture changes in the database operation log; and determine the data update operation based on the changes. The device further includes a determination module, used to determine whether the data update operation includes an archiving mark operation for the data to be updated when the data update operation includes a deletion operation for the data to be updated.

13. The apparatus according to any one of claims 8 to 12, characterized in that, The first database includes a relational database, and the second database includes a non-relational database.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it performs the operation of the method according to any one of claims 1 to 6.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the operation of the method according to any one of claims 1 to 6.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it performs the operation of the method according to any one of claims 1 to 6.

Images