Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of determining a processing strategy of a field according to a data change operation type and a change data field contained in a change log of a database, and performing change processing; and determining a target data field associated with the changed data field based on the topological dependency relationship between the data fields, and realizing the change processing of the target data field based on the processing strategy and the changed data field after the change processing. According to the embodiment, intelligent modification of the associated data of the changed data is realized based on the data change log, the accuracy of data synchronous change is improved, accurate determination of the associated data is realized through a topological algorithm, and the accuracy of data synchronous change processing is further guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data processing. Background Art

[0002] At present, with the continuous development of Internet technology, the scale of business services provided by e-commerce is getting larger and larger, and the data storage capacity of the corresponding database is showing an exponential growth trend. For example, in logistics, the number of waybill data in the waybill system of large enterprises has rapidly increased from an average of one million per day to an average of ten million per day. In such a large data volume scenario, the change operations of the data tables in the database may be extremely frequent, including adding / deleting fields, adding data, modifying fields, etc. This type of operation has an important impact on improving the accuracy and stability of data in the production system. In the existing scenario, the enterprise's big data platform is obviously aware of the above operations, and it is unable to process the corresponding change operations in a timely manner, and it is also difficult to notify the downstream data users of the platform in a timely manner.

[0003] The current method for the above problem is that when a change operation that affects the table structure and data structure occurs in the database of the big data platform, the downstream user will be informed in the form of a system work order or offline synchronization, so that the downstream user can process the data. However, the system work order or offline synchronization often has a certain message delay, and it is impossible to determine which data of the downstream user is affected, which can easily cause data loss and inaccurate data. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and apparatus for data processing, which can accurately determine a target data field associated with a changed data field and intelligently change the target data field, thereby improving the accuracy of data processing.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0006] Obtain data change logs by monitoring the database log system;

[0007] Parse the data change log to determine the data change operation type and the changed data field;

[0008] Based on the data change operation type, determine the processing strategy for the changed data field, and perform change processing on the changed data field based on the processing strategy;

[0009] Determine, based on the topological dependency between the changed data field and the data fields, a target data field in the target data table associated with the changed data field;

[0010] Based on the processing strategy and the changed data fields after the change processing, data change processing is performed on the target data fields in the target data table to obtain the target data.

[0011] Optionally, determining a target data field associated with the changed data field based on the topological dependency between the changed data field and the data fields includes:

[0012] Building a data field topology graph based on the changed data fields and the topological dependency between the data fields;

[0013] Based on the data field topology map, a target data field associated with the changed data field is determined.

[0014] Optionally, based on the data change operation type, a processing strategy for the changed data field is determined, including:

[0015] In response to the data change operation type being a field type operation, based on the data change operation type, modifying the task field corresponding to the changed data field;

[0016] Determine the processing strategy for the changed data fields based on the task fields.

[0017] Optionally, performing data change processing on a target data field in a target data table based on the processing strategy and the changed data field after the change processing to obtain the target data includes:

[0018] Perform consistency check on the changed data fields after the change processing and the target data fields in the target data table;

[0019] In response to the judgment result being consistent, data change processing is performed on the target data field in the target data table based on the processing strategy and the changed data field after the change processing.

[0020] Optionally, based on the data change operation type, a processing strategy for the changed data field is determined, including:

[0021] In response to the data class change operation type being a data deletion operation, a processing strategy for the changed data field is determined to be storing deleted data corresponding to the changed data field.

[0022] Optionally, data change processing is performed on the target data field in the target data table based on the processing strategy and the changed data field after the change processing, including:

[0023] Match the deleted data corresponding to the processing strategy with the data in the target data table;

[0024] In response to first data in the target data table being matched to the deleted data, a deletion process is performed on the target data field in the target data table based on the changed data field after the change process.

[0025] According to another aspect of an embodiment of the present invention, there is provided a data processing device, including:

[0026] The acquisition module is used to obtain data change logs by monitoring the database log system;

[0027] The parsing module is used to parse the data change log and determine the data change operation type and the changed data field;

[0028] A determination module is used to determine a processing strategy for the changed data field based on the data change operation type, and perform change processing on the changed data field based on the processing strategy; based on the topological dependency relationship between the changed data field and the data field, determine a target data field in a target data table associated with the changed data field;

[0029] The processing module is used to perform data change processing on the target data field in the target data table based on the processing strategy and the changed data field after the change processing to obtain the target data.

[0030] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by an embodiment of the present invention.

[0031] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method provided by an embodiment of the present invention is implemented.

[0032] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data processing method provided by an embodiment of the present invention is implemented.

[0033] An embodiment of the above invention has the following advantages or beneficial effects: it is able to obtain data change logs through a monitoring log system, parse the data change logs to obtain data change operation types and changed data fields, accurately determine the changed data and the changed operations, and generate a processing strategy. Furthermore, based on the topological dependency, the target data field of the downstream table associated with the changed data field is determined, that is, the downstream data affected by the changed data is determined, and the data scope that needs to be changed is accurately determined. Based on the processing strategy and the changed changed data field, the target data field in the target data table is automatically changed to obtain the target data, and the synchronous change of the associated data based on the dependency is realized, the accuracy of data change processing is improved, and the reliability of the data in the upstream and downstream data tables is maintained.

[0034] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0036] Figure 1 is a schematic diagram of main steps of a method for data processing according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of table field dependency in an embodiment of the present invention;

[0038] Figure 3 is a flow chart of a data processing method according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0040] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0044] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0045] Figure 1 is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention, such as Figure 1 As shown, including:

[0046] Step S101, monitoring the log system of the database to obtain data change logs.

[0047] The log system of the production database is monitored and analyzed in real time through the preset monitoring program to detect whether there are any change activities that affect the accuracy of the data, such as the addition and deletion of data, the addition and deletion of fields, etc. If a change activity is detected, the data change log of this part is obtained. Among them, the monitored log system monitors the log data of the upstream production system table.

[0048] Optionally, changes to the production system table of the production database include: adding or deleting fields in the production system, changing field types in the production system, physically deleting invalid records in the production system, modifying database indexes in the production system, dropping (deleting) tables in the production database, etc.

[0049] Step S102: parsing the data change log to determine the data change operation type and the changed data field.

[0050] By parsing the acquired data change log, it is possible to obtain specific information about the data change operation, including the change operation type and the changed data field. The data change operation type includes, for example, adding / deleting a field and adding / deleting data.

[0051] Taking the widely used MySQL (relational database management system) as an example, through the MySQL binlog log (binary log), you can view the database operation statements, find the deleted record fields and field values, and based on the log, you can accurately determine the scope of deleted data.

[0052] Step S103: determining a processing strategy for the changed data field based on the data change operation type, and performing change processing on the changed data field based on the processing strategy.

[0053] According to the determined data change operation type, a processing strategy for the changed data field is intelligently generated, that is, the data change operation information for the changed data field. For example, if the determined data change operation type is a newly added field, and the changed data field is field a, a processing strategy for adding field a to the data table of the database is generated. Based on the generated processing strategy, corresponding change processing is performed in the data table of the database.

[0054] Step S104: determining a target data field in a target data table associated with the changed data field based on the topological dependency relationship between the changed data field and the data fields.

[0055] According to the preset topological dependency between data fields, the impact scope of the changed data field is determined, that is, the target data field in the target data table directly or indirectly associated with the changed data field is determined. Specifically, as shown in Table 1, an example of a production table dependency relationship is shown. The upstream table (upstream production data table) fields include a1, c1, d1, and the downstream table fields include b1, c1, d1, e1, and f1. The dependency relationship includes b1 depending on a1, c1 depending on a1, etc.; if the changed data field is a1, the determined target data fields include b1, c1, d1, e1, and f1.

[0056]

[0057] Step S105 , performing data change processing on the target data field in the target data table based on the processing strategy and the changed data field after the change processing, to obtain the target data.

[0058] According to the determined processing strategy and the changed data fields after the change processing, the target data fields are processed with corresponding data changes, including field addition / deletion and data addition / deletion, to obtain the target data of the processed target data table.

[0059] Optionally, the change processing depends on the processing strategy. If the processing strategy is to delete field b, and the field does not affect the business logic of downstream fields, a corresponding field deletion operation is automatically performed in the downstream data table.

[0060] Optionally, if the processing strategy is to delete field c, which affects the business logic of the downstream field, such as the value-taking rule, then the processing strategy and the changed data field need to be sent to the terminal of the target user so that the target user can replace the value-taking rule of the downstream field. Therefore, before performing the change processing, the processing strategy needs to be identified to determine whether it needs to be sent to the terminal corresponding to the target user.

[0061] Optionally, the target data field change operation is implemented by establishing a change task, and the task operation status is verified in real time through the monitoring system. If an abnormal task operation is monitored, an alarm is immediately triggered and manual intervention is requested to solve the problem.

[0062] According to the data processing method provided by the embodiment of the present invention, it is possible to obtain a data change log through a monitoring log system, parse the data change log to obtain a data change operation type and a changed data field, accurately determine the changed data and the changed operation, and generate a processing strategy. Furthermore, based on the topological dependency, the target data field of the downstream table associated with the changed data field is determined, that is, the downstream data affected by the changed data is determined, and the data scope that needs to be changed is accurately determined. Based on the processing strategy and the changed changed data field, the target data field in the target data table is automatically changed to obtain the target data, and the synchronous change of the associated data based on the dependency is realized, thereby improving the accuracy of data change processing and maintaining the reliability of the data in the upstream and downstream data tables.

[0063] Optionally, based on the topological dependency between the changed data field and the data field, the target data field associated with the changed data field is determined, including: constructing a data field topological graph based on the topological dependency between the changed data field and the data field; determining the target data field associated with the changed data field based on the data field topological graph. When determining the target data field based on the topological dependency, the topological sorting result can be determined by constructing a topological graph, and the target data field can be determined based on the result. Specifically, the monitored change table and change items are used as the starting node; the downstream dependent tables and fields are identified based on the topological sorting graph algorithm, and scanned layer by layer, so that all affected tables and fields can be identified.

[0064] Topological sorting is to sort a directed acyclic graph G (i.e.: DAG) topologically. It is to arrange all the vertices in G into a linear sequence, so that for any pair of vertices u and v in the graph, if the edge (u, v)∈E(G), then u appears before v in the linear sequence. Usually, such a linear sequence is called a sequence that satisfies the topological order, or topological sequence for short. The order of sorting is based on the pointing relationship of the nodes. Each node has an in-degree (how many points lead to it, that is, how many prerequisites it has to start) and an out-degree (how many points it leads to, that is, how many other nodes it is the prerequisite for starting). There are two search methods for topological sorting: breadth-first and depth-first. The following is the idea of ​​​​breadth-first search: find node A with an in-degree of 0 from the DAG graph (that is, a node with no arrow pointing to it), and put it into the result set of the topological sequence; at the same time, delete all edges starting from node A; repeat the above two steps in the remaining DAG graph until all nodes can be deleted and added to the result set, which means that the DAG graph can be topologically sorted. Taking the data in Table 1 as an example, draw the table field dependency relationship as follows Figure 2As shown in the figure, assuming that the table field a1 is changed, the topological sorting algorithm can be used to detect that the data dependency sequence is [a1, d1, f1, b1, c1, e1]. According to the sequence given by the topological sorting, a1-d1-f1-b1-c1-e1 can be gradually changed accordingly.

[0065] Optionally, the target data fields corresponding to the changed data fields may be determined one by one according to the topological dependency relationship. That is, if the changed data field is a1, its downstream data field b1 is determined, and then the downstream field of b1 is determined.

[0066] The data processing method provided by the embodiment of the present invention can determine the scope of the fields in the downstream data table that are affected after the fields in the upstream data table are changed, that is, the target data fields, based on the topological relationship between the fields in the upstream and downstream data tables, and perform synchronous data change operations on the target data fields, thereby improving the accuracy of data change processing.

[0067] Optionally, based on the data change operation type, determining a processing strategy for the changed data field includes: in response to the data change operation type being a field-type operation, based on the data change operation type, modifying the task field corresponding to the changed data field; and determining the processing strategy for the changed data field based on the task field. Before generating the processing strategy, first determine the data change operation type. If it is a field-type operation, that is, adding / deleting / changing a certain field, then modify the task field of the changed data field to perform the corresponding change operation, and determine the processing strategy as adding / deleting / changing a certain field based on the modified task field.

[0068] Optionally, data change processing is performed on the target data field in the target data table based on the processing strategy and the changed data field after the change processing to obtain the target data, including: judging the consistency of the changed data field after the change processing and the target data field in the target data table; in response to the judgment result being consistent, data change processing is performed on the target data field in the target data table based on the processing strategy and the changed data field after the change processing.

[0069] Obtain the data after the change processing of the changed data field of the upstream table, and perform consistency judgment with the target data field in the target data table to obtain the judgment result. For example, if a new field a is added to the upstream table, the newly added field a is judged for consistency with the target data field in the target data table to determine whether the values ​​corresponding to the fields correspond one to one and obtain the judgment result. When the judgment result is consistent, the target data field in the target data table is processed based on the processing strategy and the changed data field after the change processing.

[0070] Optionally, when the judgment result is inconsistent, a notification is sent to a preset user terminal to notify the user to perform manual intervention processing.

[0071] Optionally, based on the data change operation type, determining a processing strategy for the changed data field includes: in response to the data change operation type being a data deletion operation, determining that the processing strategy for the changed data field is to store the deleted data corresponding to the changed data field. If it is determined that the data change operation type is a data deletion operation, the processing strategy is to store the deleted data for subsequent comparison and analysis.

[0072] Optionally, data change processing is performed on the target data field in the target data table based on the processing strategy and the changed data field after the change processing, including: matching the deleted data corresponding to the processing strategy with the data in the target data table; in response to matching the first data consistent with the deleted data in the target data table, deleting the target data field in the target data table based on the changed data field after the change processing. When data change processing is performed on the downstream data table, the stored deleted data is matched with the downstream data table to determine whether there is data identical to the deleted data in the downstream data table. If the identical data is matched, the data in the downstream data table is deleted; if the identical data is not matched, a notification is sent to a preset user terminal to notify the user to perform manual intervention processing.

[0073] According to the data processing method provided by the embodiment of the present invention, the data change log obtained by the monitoring log system can be parsed to obtain the data change operation type and the changed data field, and a processing strategy can be generated. Furthermore, based on the topological dependency, a topological map can be constructed specifically according to the topological dependency, and the target data field of the downstream table associated with the changed data field is determined based on the topological map, that is, the downstream data affected by the changed data is determined, and the associated fields affected by the changed data field are accurately determined. Based on the processing strategy and the changed changed data field, the target data field in the target data table is automatically changed to obtain the target data, thereby realizing accurate synchronous change of the associated data, ensuring the reliability of the data in the upstream and downstream data tables, and realizing efficient data change operations.

[0074] like Figure 3The figure shows a flow chart of a data processing method. The production database table is monitored by a log monitoring program to determine whether there is a data change in the log. If not, offline data extraction is performed on the production database table. If yes, the change operation type is determined, including: whether to add / delete fields, whether to change field types, and whether to physically delete data. If it is to add / delete fields, the add / delete field operation is performed; if it is to change field types, the field type change operation is performed; if it is to physically delete data, the deleted data is recorded and stored. After the above execution operation is completed, the modification impact range is determined based on topological sorting, and a change task is established for the changed data field in the modification range. The downstream data table change task is synchronized, and the task can be synchronized to the terminal of the person in charge of the downstream data table, and offline data extraction is performed based on the change task. Data consistency comparison is performed based on offline data (downstream data table data) and the changed data field after the change processing. If they are consistent, a standard data layer result (data processing result) is generated based on the changed data field after the change processing and the offline data. If they are inconsistent, a notification is sent to the preset user terminal to notify manual processing.

[0075] Based on the data processing method of the present invention, the following results can be obtained by simulation experiments: in the scenario of adding a vehicle model field to the power of attorney table, the traditional method requires 2 hours of processing time, while the present solution does not require separate processing and can directly add fields, replacing the traditional work order form with automatic identification changes; in the scenario of changing the primary key type of the logistics transfer order, the traditional method requires data research and development to determine the scope of impact based on the work order, and cannot effectively evaluate downstream usage, while the present solution does not require separate processing and can directly change the field and intelligently identify without manual participation; in the scenario of deleting logistics routing waybill data, the traditional method cannot perform incremental extraction due to the extremely large PB-level data of the logistics routing waybill, resulting in dirty data in the table when deleting, while the present solution can directly identify and delete data. The average daily data in the logistics routing table is about 1.8 billion. Due to configuration abnormalities, the number of data items affected when back-calculating and deleting data reaches millions, improving the accuracy of 0.05% of the data in the table.

[0076] Figure 4 4 is a schematic diagram of a data processing device 400 provided in an embodiment of the present invention, comprising:

[0077] The acquisition module 401 is used to acquire data change logs by monitoring the log system of the database;

[0078] Parsing module 402, used to parse the data change log to determine the data change operation type and the changed data field;

[0079] The determination module 403 is used to determine a processing strategy for the changed data field based on the data change operation type, and perform change processing on the changed data field based on the processing strategy; determine a target data field in a target data table associated with the changed data field based on a topological dependency relationship between the changed data field and the data field;

[0080] The processing module 404 is used to perform data change processing on the target data field in the target data table based on the processing strategy and the changed data field after the change processing to obtain the target data.

[0081] The data processing device provided in accordance with the embodiment of the present invention can obtain the data change log through the monitoring log system, parse the data change log to obtain the data change operation type and the changed data field, accurately determine the changed data and the changed operation, and generate a processing strategy. Furthermore, based on the topological dependency, the target data field of the downstream table associated with the changed data field is determined, that is, the downstream data affected by the changed data is determined, and the data scope that needs to be changed is accurately determined. Based on the processing strategy and the changed changed data field, the target data field in the target data table is automatically changed to obtain the target data, and the synchronous change of the associated data based on the dependency is realized, thereby improving the accuracy of data change processing and maintaining the reliability of the data in the upstream and downstream data tables.

[0082] Optionally, the determination module 403 is further used to: construct a data field topology map based on the changed data field and the topological dependency relationship between the data fields;

[0083] Based on the data field topology map, a target data field associated with the changed data field is determined.

[0084] Optionally, the determination module 403 is further configured to: in response to the data change operation type being a field type operation, modify the task field corresponding to the changed data field based on the data change operation type;

[0085] Determine the processing strategy for the changed data fields based on the task fields.

[0086] Optionally, the processing module 404 is further used to: perform consistency judgment on the changed data field after the change processing and the target data field in the target data table;

[0087] In response to the judgment result being consistent, data change processing is performed on the target data field in the target data table based on the processing strategy and the changed data field after the change processing.

[0088] Optionally, the determination module 403 is further configured to: in response to the data change operation type being a data deletion operation, determine that the processing strategy for the changed data field is to store deleted data corresponding to the changed data field.

[0089] Optionally, the processing module 404 is further used to: match the deleted data corresponding to the processing strategy with the data in the target data table;

[0090] In response to first data in the target data table being matched to the deleted data, a deletion process is performed on the target data field in the target data table based on the changed data field after the change process.

[0091] According to the data processing device provided by the embodiment of the present invention, the data change log obtained by the monitoring log system can be parsed to obtain the data change operation type and the changed data field, and a processing strategy can be generated. Furthermore, based on the topological dependency, a topological map can be constructed specifically according to the topological dependency, and the target data field of the downstream table associated with the changed data field is determined based on the topological map, that is, the downstream data affected by the changed data is determined, and the associated fields affected by the changed data field are accurately determined. Based on the processing strategy and the changed changed data field, the target data field in the target data table is automatically changed to obtain the target data, thereby realizing accurate synchronous change of the associated data, ensuring the reliability of the data in the upstream and downstream data tables, and realizing efficient data change operations.

[0092] Figure 5 An exemplary system architecture 500 is shown to which the data processing method or data processing apparatus according to the embodiment of the present invention can be applied.

[0093] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0094] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0095] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0096] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received processing request and other data, and feed back the processing result (such as the processing result - only an example) to the terminal device.

[0097] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data processing device is generally set in the server 505 .

[0098] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0099] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 6 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0100] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0101] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0102] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0103] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be provided in a processor. For example, they may be described as follows: a processor includes an acquisition module, a parsing module, a determination module, and a processing module. The names of these modules do not, in certain cases, constitute limitations on the units or modules themselves. For example, the acquisition module may also be described as a "module for acquiring data change logs by monitoring the log system of a database."

[0106] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0107] Obtain data change logs by monitoring the database log system;

[0108] Parse the data change log to determine the data change operation type and the changed data field;

[0109] Based on the data change operation type, determine the processing strategy for the changed data field, and perform change processing on the changed data field based on the processing strategy;

[0110] Determine, based on the topological dependency between the changed data field and the data fields, a target data field in the target data table associated with the changed data field;

[0111] Based on the processing strategy and the changed data fields after the change processing, data change processing is performed on the target data fields in the target data table to obtain the target data.

[0112] According to the technical solution of the embodiment of the present invention, it is possible to obtain data change logs through a monitoring log system, parse the data change logs to obtain data change operation types and changed data fields, accurately determine the changed data and the changed operations, and generate a processing strategy. Furthermore, based on the topological dependency, the target data field of the downstream table associated with the changed data field is determined, that is, the downstream data affected by the changed data is determined, and the data scope that needs to be changed is accurately determined. Based on the processing strategy and the changed changed data field, the target data field in the target data table is automatically changed to obtain the target data, and the synchronous change of the associated data based on the dependency is realized, which improves the accuracy of data change processing and maintains the reliability of the data in the upstream and downstream data tables.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data processing, characterized in that: include: Obtain data change logs by monitoring the database log system; Parsing the data change log to determine the data change operation type and the changed data field; Determining a processing strategy for the changed data field based on the data change operation type, and performing change processing on the changed data field based on the processing strategy; Determine, based on the topological dependency relationship between the changed data field and the data field, a target data field in the target data table associated with the changed data field; Based on the processing strategy and the changed data field after the change processing, data change processing is performed on the target data field in the target data table to obtain the target data.

2. The method according to claim 1, characterized in that The determining, based on the topological dependency between the changed data field and the data field, a target data field associated with the changed data field comprises: Constructing a data field topology graph based on the changed data field and the topological dependency relationship between the data fields; Based on the data field topology map, a target data field associated with the changed data field is determined.

3. The method according to claim 1, characterized in that: The determining, based on the data change operation type, a processing strategy for the changed data field includes: In response to the data change operation type being a field type operation, modifying the task field corresponding to the changed data field based on the data change operation type; A processing strategy for the changed data field is determined based on the task field.

4. The method according to claim 3, characterized in that The step of performing data change processing on the target data field in the target data table based on the processing strategy and the changed data field after the change processing to obtain the target data includes: Performing consistency judgment on the changed data field after the change processing and the target data field in the target data table; In response to the judgment result being consistent, data change processing is performed on the target data field in the target data table based on the processing strategy and the changed data field after the change processing.

5. The method according to claim 1, characterized in that The determining, based on the data change operation type, a processing strategy for the changed data field includes: In response to the data change operation type being a data deletion operation, determining a processing strategy for the changed data field as storing deleted data corresponding to the changed data field.

6. The method according to claim 5, characterized in that The performing data change processing on the target data field in the target data table based on the processing strategy and the changed data field after the change processing includes: Matching the deleted data corresponding to the processing strategy with the data in the target data table; In response to first data in the target data table being matched to the deleted data, a deletion process is performed on the target data field in the target data table based on the changed data field after the change process.

7. A data processing device, characterized in that: include: The acquisition module is used to obtain data change logs by monitoring the database log system; A parsing module, used to parse the data change log to determine the data change operation type and the changed data field; A determination module, configured to determine a processing strategy for the changed data field based on the data change operation type, and perform change processing on the changed data field based on the processing strategy; Determine, based on the topological dependency relationship between the changed data field and the data field, a target data field in the target data table associated with the changed data field; The processing module is used to perform data change processing on the target data field in the target data table based on the processing strategy and the changed data field after the change processing to obtain the target data.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images