Data processing method and device, computer equipment, readable storage medium and program product
By generating and merging event structure information and periodically updating the target database, the problem of excessive resource consumption in large-scale data synchronization is solved, and fast and accurate data synchronization is achieved.
Patent Information
- Application Number
- CN202511318183.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-16
- Publication Date
- 2025-12-26
Smart Images
Figure CN121210171A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device, computer equipment, a computer readable storage medium and a computer program product. BACKGROUND
[0002] In a large-scale data processing scenario, in the process of synchronizing data from a source system such as a database to a target system such as a search engine, due to high update frequency, full update, unreasonable component link design, and other reasons, the problem of write amplification may occur, that is, the actual amount of data written or resource consumption is much larger than the necessary range.
[0003] In related technologies, data synchronization can be performed through log recording combined with a stream processing framework. However, in this way, the architecture is complex, multiple components need to be maintained, and there are problems of data consistency. SUMMARY
[0004] Therefore, it is necessary to provide a data processing method, device, computer equipment, computer readable storage medium and computer program product capable of improving data synchronization efficiency and reducing resource consumption in view of the above technical problems.
[0005] In a first aspect, the present application provides a data processing method, comprising:
[0006] In response to a trigger operation of data modification for a document in a source database, obtaining a data modification log corresponding to the trigger operation, and generating event structure information corresponding to the data modification according to the data modification log, the event structure information being obtained after being encapsulated based on a preset standard event structure;
[0007] storing the event structure information into a target message queue, and in the case that there is initial event structure information matching the event structure information in the target message queue, merging the event structure information and the initial event structure information;
[0008] periodically obtaining event structure information corresponding to the source database in the target message queue according to a preset period, and updating a corresponding document in a target database according to the event structure information.
[0009] In one of the embodiments, in the case that there is initial event structure information matching the event structure information in the target message queue, merging the event structure information and the initial event structure information comprises:
[0010] determining a target document corresponding to the event structure information;
[0011] In a case where the initial event structure information for the target document exists in the target message queue, an initial data change field corresponding to the initial event structure information is acquired;
[0012] According to the initial data change field and a data change field corresponding to the event structure information, a target data change field is generated, and merged event structure information is obtained.
[0013] In an embodiment, in a case where the initial event structure information for the target document exists in the target message queue, the initial data change field corresponding to the initial event structure information is acquired, including:
[0014] Candidate event structure information in a preset time period in the target message queue is acquired.
[0015] According to the candidate event structure information, a document identification field corresponding to the candidate structure information is determined.
[0016] In a case where the document identification field matches the target document, the candidate event structure information is determined as initial event structure information, and an initial data change field corresponding to the initial event structure information is acquired.
[0017] In an embodiment, the event structure information corresponding to the data modification is generated according to the data modification log, including:
[0018] The data modification log is parsed to determine a modification type corresponding to the data modification, target field information in a document corresponding to the data modification, and before-and-after values of the target field;
[0019] The modification type, the target field information, and the before-and-after values of the target field are encapsulated based on a preset standard event structure to obtain event structure information.
[0020] In an embodiment, the corresponding document in the target database is updated according to the event structure information, including:
[0021] According to target field information corresponding to the event structure information, a to-be-updated field in a document in the target database that matches the event structure information is determined;
[0022] The to-be-updated field is updated based on a modification type corresponding to the event structure information and the before-and-after values.
[0023] In an embodiment, the preset period is determined in the following manner:
[0024] A data amount of data update corresponding to the target database in a preset historical time period is acquired.
[0025] determining a preset period according to the data amount.
[0026] In a second aspect, the present application provides a data processing apparatus, comprising:
[0027] The acquisition module is configured to, in response to a trigger operation of data modification on a document in a source database, acquire a data modification log corresponding to the trigger operation, and generate event structure information corresponding to the data modification according to the data modification log, the event structure information being obtained by encapsulating based on a preset standard event structure.
[0028] The merging module is configured to store the event structure information into a target message queue, and in a case where there is initial event structure information matching the event structure information in the target message queue, merge the event structure information and the initial event structure information.
[0029] The updating module is configured to periodically acquire event structure information corresponding to the source database in the target message queue according to a preset period, and update a corresponding document in a target database according to the event structure information.
[0030] In one of the embodiments, the merging module is further configured to:
[0031] determine a target document corresponding to the event structure information.
[0032] in a case where there is initial event structure information for the target document in the target message queue, acquire an initial data change field corresponding to the initial event structure information.
[0033] generate a target data change field according to the initial data change field and a data change field corresponding to the event structure information, to obtain merged event structure information.
[0034] In one of the embodiments, the merging module is further configured to:
[0035] acquire candidate event structure information in a preset time period in the target message queue.
[0036] determine a document identification field corresponding to the candidate event structure information according to the candidate event structure information.
[0037] in a case where the document identification field matches the target document, determine the candidate event structure information as initial event structure information, and acquire an initial data change field corresponding to the initial event structure information.
[0038] In one of the embodiments, the acquisition module is further configured to:
[0039] The data modification log is parsed to determine a modification type corresponding to the data modification, target field information in a document corresponding to the data modification, and a value of the target field before and after the change;
[0040] The modification type, the target field information, and the value before and after the change are encapsulated based on a preset standard event structure to obtain event structure information.
[0041] In one of the embodiments, the updating module is further configured to:
[0042] According to the target field information corresponding to the event structure information, a to-be-updated field in a document in the target database that matches the event structure information is determined.
[0043] According to the modification type corresponding to the event structure information and the value before and after the change, the to-be-updated field is updated.
[0044] In one of the embodiments, the apparatus further includes a preset period determination module configured to:
[0045] Obtain a data volume of data update corresponding to the target database in a preset historical period;
[0046] According to the data volume, a preset period is determined.
[0047] In a third aspect, the embodiments of the present disclosure further provide a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in any one of the embodiments of the present disclosure when executing the computer program.
[0048] In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the steps of the method in any one of the embodiments of the present disclosure when executed by a processor.
[0049] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product. The computer program product includes a computer program, and the computer program implements the steps of the method in any one of the embodiments of the present disclosure when executed by a processor.
[0050] The data processing method, device, computer device, computer readable storage medium and computer program product, when performing data synchronization, in response to a trigger operation of data modification of a document in a source database, obtain a data modification log, generate event structure information, store the event structure information into a target message queue, and in a case where there is matching initial event structure information in the target message queue, merge the event structure information and the initial event structure information, periodically obtain the event structure information, and update a corresponding document in a target database, thereby realizing fast and accurate synchronization of data from the source database to the target database. The event structure information is encapsulated based on the data modification log, the core information of data modification is extracted, and the data processing workload is reduced. When stored in the target message queue, the event structure information is merged, the problem of repeated synchronization during data synchronization is reduced, and the problem of data synchronization amplification is avoided. Through periodic data synchronization and batch data processing, the data synchronization write request is reduced, the data synchronization efficiency is improved, resource consumption is effectively reduced, and the method is suitable for more application scenarios. BRIEF DESCRIPTION OF DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor based on these drawings.
[0052] Figure 1 Flowchart of a data processing method in an embodiment;
[0053] Figure 2 Flowchart of a data processing method in another embodiment;
[0054] Figure 3 Optimization schematic diagram of a stream processing layer in an embodiment;
[0055] Figure 4 Flowchart of a data processing method in an embodiment;
[0056] Figure 5 Optimization schematic diagram of a data write layer in an embodiment;
[0057] Figure 6 Internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION
[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.
[0059] In one embodiment, as shown in Figure 1 A data processing method is provided, and the embodiment is exemplarily described by taking the method applied to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction of the terminal and the server. In the embodiment, the method includes the following steps.
[0060] In step S110, in response to a trigger operation of data modification on a document in a source database, a data modification log corresponding to the trigger operation is obtained, and event structure information corresponding to the data modification is generated according to the data modification log, the event structure information being obtained by encapsulating based on a preset standard event structure.
[0061] Exemplarily, in response to a trigger operation of data modification on a document in a source database, a data modification log corresponding to the trigger operation is obtained. In some examples, the source database can be a database that needs to be synchronized, for example, the source database can be determined according to an actual application scenario, such as a MySQL database (an open source relational database management system) and the like.
[0062] Optionally, the data modification event can be captured by a listening component. After obtaining the data modification log, the event structure information corresponding to the data modification is generated according to the data modification log. The event structure information can include table name, primary key, and change field information, etc. The event structure information is obtained by encapsulating based on a preset standard event structure, wherein the preset standard event structure can be determined according to an actual application scenario. In one example, the event structure information can include core data directly associated with data change in the data modification log, and redundant data is filtered out.
[0063] In step S120, the event structure information is stored in a target message queue, and in the case that there is initial event structure information matching the event structure information in the target message queue, the event structure information and the initial event structure information are merged.
[0064] Exemplarily, after generating the event structure information, the event structure information is stored in a target message queue, and in the case that there is initial event structure information matching the event structure information in the target message queue, the event structure information and the initial event structure information are merged. In some examples, when the event structure information and the initial event structure information are information corresponding to data modification on the same document, it is determined that they match.
[0065] Optionally, the target message queue is used to store event structure information corresponding to data modification of the document in the source database. In some examples, the target message queue stores event structure information corresponding to data update from the last time of data update synchronization to the current time.
[0066] In step S130, the event structure information corresponding to the source database in the target message queue is periodically obtained according to a preset period, and the corresponding document in the target database is updated according to the event structure information.
[0067] For example, the event structure information corresponding to the source database in the target message queue is periodically obtained according to a preset period. The preset period can be determined in advance according to the actual application scenario, and in some examples, the preset period can be determined according to the data update frequency in the historical period in the actual application scenario.
[0068] Optionally, the obtained event structure information can be event structure information corresponding to the current period. According to the event structure information, the change of the corresponding document in the target database can be determined, and the corresponding document in the database can be updated. The target database can be determined according to the actual application scenario, for example, the target database can be a preset search engine, and the present disclosure does not limit this.
[0069] In the embodiment of the present disclosure, when data synchronization is performed, in response to the trigger operation of data modification of the document in the source database, the data modification log is obtained, the event structure information is generated, the event structure information is stored in the target message queue, and in the case that there is matching initial event structure information in the target message queue, the event structure information and the initial event structure information are merged, the event structure information is periodically obtained, and the corresponding document in the target database is updated, thereby realizing the fast and accurate synchronization of data from the source database to the target database. Based on the data modification log, the event structure information is encapsulated to realize the extraction of the core information of the data modification, and the data processing workload is reduced. When stored in the target message queue, the event structure information is merged to reduce the problem of repeated synchronization during data synchronization, and the problem of data synchronization amplification is avoided. By periodically performing data synchronization and batch processing data, the request of data synchronization writing is reduced, the data synchronization efficiency is improved, the resource consumption is effectively reduced, and the method is suitable for more application scenarios.
[0070] In one embodiment, in the case that there is initial event structure information matching the event structure information in the target message queue, the event structure information and the initial event structure information are merged, including:
[0071] determining the target document corresponding to the event structure information;
[0072] In a case where the initial event structure information for the target document exists in the target message queue, an initial data change field corresponding to the initial event structure information is acquired;
[0073] According to the initial data change field and a data change field corresponding to the event structure information, a target data change field is generated, and merged event structure information is obtained.
[0074] Exemplarily, a target document corresponding to the event structure information is determined. In some examples, identification information of a target document corresponding to data modification is included in the event structure information, and the target document can be determined according to the identification information. The target document is the document to be modified corresponding to the event structure information.
[0075] Optionally, it is determined whether initial event structure information for the target document exists in the target message queue. If the initial event structure information exists, it indicates that modification of the target document has occurred in the current period. At this time, an initial data change field corresponding to the initial event structure information is determined, and a target data change field is generated according to the initial data change field and a data change field corresponding to the event structure information. In some examples, the initial event structure information and modification of the target document by the event structure information are merged according to the initial data change field and the data change field to generate the target data change field, and merged event structure information is obtained. When modification is performed, only one update synchronization needs to be performed according to the merged event structure information, and multiple update synchronizations are not needed.
[0076] According to the embodiments of the present disclosure, the event structure information is quickly and accurately merged based on the document corresponding to the event structure information, thereby effectively reducing repeated synchronization of modification of the same document in the period and reducing resource consumption.
[0077] In one embodiment, in a case where the initial event structure information for the target document exists in the target message queue, the initial data change field corresponding to the initial event structure information is acquired, including:
[0078] Candidate event structure information in a preset time period in the target message queue is acquired.
[0079] According to the candidate event structure information, a document identification field corresponding to the candidate structure information is determined.
[0080] In a case where the document identification field matches the target document, the candidate event structure information is determined as initial event structure information, and an initial data change field corresponding to the initial event structure information is acquired.
[0081] Exemplarily, candidate event structure information in a preset time period in the target message queue is acquired, where the preset time period can be determined according to an actual application scenario. In an example, the preset time period can be from a last update time to a current update time, which matches the preset period. The candidate event structure information can be event structure information corresponding to data changes occurring in the current update period.
[0082] Optionally, the candidate event structure information has a document identifier field corresponding to the document to which the candidate event structure information is directed. In some examples, the document identifier field can include a document ID (Identifier) or the like. In a case where the document identifier field matches the target document, the current candidate event structure information can be considered as initial event structure information corresponding to data changes of the target document, and an initial data change field corresponding to the initial event structure information is acquired.
[0083] According to the embodiments of the present disclosure, candidate event structure information in a preset time period is acquired, and an initial data change field is screened according to a document identifier field, so that initial event structure information of the target document in the preset time period can be quickly and accurately determined, a subsequent merging operation is facilitated, resource consumption is reduced, and the embodiments are applicable to more application scenarios.
[0084] In an embodiment, the generating of the event structure information corresponding to the data modification based on the data modification log comprises:
[0085] The data modification log is parsed to determine a modification type corresponding to the data modification, target field information in a document corresponding to the data modification, and before-and-after values of the target field;
[0086] The modification type, the target field information, and the before-and-after values of the target field are encapsulated based on a preset standard event structure to obtain the event structure information.
[0087] Exemplarily, the data modification log is parsed. In some examples, the data modification log can be a binary log. When the data modification log is parsed, the data modification log is converted into a structured intermediate format by a preset parsing tool, including but not limited to JSON (JavaScript Object Notation, a lightweight data exchange format), Avro (an open source data serialization system), etc., which can be determined according to an actual application scenario; the core fields required for a general event are extracted from the parsed information, including a modification type, target field information in a document corresponding to the data modification, and before-and-after values of the target field, and in some examples, the core fields further include a document identifier field corresponding to the document, which can be determined according to an actual application scenario; the modification type, the target field information, and the before-and-after values are packaged based on a preset standard event structure to obtain corresponding event structure information, which can be determined according to an actual application scenario.
[0088] In the embodiments of the present disclosure, the data modification log is parsed, the related information corresponding to the data modification is determined, and the event structure information is obtained after packaging, which realizes the extraction and packaging of the core data associated with the data modification in the data modification log, reduces the data amount of subsequent data processing and data synchronization, improves the data synchronization efficiency, and is suitable for more application scenarios.
[0089] In one embodiment, the updating of the corresponding document in the target database according to the event structure information includes:
[0090] According to the target field information corresponding to the event structure information, a to-be-updated field in a document in the target database that matches the event structure information is determined.
[0091] According to the modification type and the before-and-after values corresponding to the event structure information, the to-be-updated field is updated.
[0092] Exemplarily, the to-be-updated field is determined based on the target field information corresponding to the event structure information, where the to-be-updated field is a specific field of the data modification. In some examples, the target field information can include a position of the to-be-updated field in the document, a field identifier corresponding to the to-be-updated field, etc., which can be determined according to an actual application scenario.
[0093] Optionally, the event structure information further includes the modification type and the before-and-after values, and the to-be-updated field is updated according to the modification type and the before-and-after values to obtain an updated field.
[0094] According to the event structure information, the embodiments of the present disclosure can accurately locate the field to be updated, and update the field to be updated during data synchronization, thereby realizing field-level data synchronization and update, enabling incremental update during data synchronization, avoiding the problem of synchronization amplification caused by full-document update synchronization, saving data processing resources, and improving data synchronization efficiency.
[0095] In one embodiment, the determination manner of the preset period comprises:
[0096] obtaining a data amount of data update corresponding to the target database in a preset historical period;
[0097] determining a preset period according to the data amount.
[0098] Exemplarily, the preset period is determined according to historical data. In some examples, a data amount of data update corresponding to the target database in a preset historical period is obtained, and the preset historical period can be determined according to an actual application scenario. In some examples, a data amount of data update of the target database according to the source database in a preset period is obtained.
[0099] Optionally, a preset period is determined according to the data amount. Generally, the larger the data amount, the higher the data update frequency, and the shorter the preset period, so as to timely and accurately perform data synchronization update and avoid data update delay. The smaller the data amount, the lower the data update frequency, and the longer the preset period, so as to avoid unnecessary resource waste and save resources.
[0100] According to the embodiments of the present disclosure, the preset period is determined according to the data amount of historical update, the flexibility of the preset period is ensured, the preset period can be adjusted according to an actual application scenario, the efficiency of data synchronization is taken into account, resource consumption is reduced, and the embodiments are suitable for more application scenarios.
[0101] Figure 2 For a flowchart of a data processing method according to an exemplary embodiment, refer to Figure 2As shown, the Flink (an open source stream processing framework)-based stream processing architecture uses Canal (a database change capture tool) or a self-developed CDC (Change Data Capture) component to capture binlog (binary log) change events (i.e., data modification logs) of MySQL and synchronizes the data to Elasticsearch (an open source distributed search engine) in real time. In some examples, a metadata management configuration center is also provided to manage metadata in the entire process, such as database table structure, field meaning, processing rules, and other configuration information. A monitoring and alarm system is used to monitor the running state of the entire process, such as whether data transmission is delayed and whether processing is incorrect. Once there is a problem, an alarm is sent to facilitate the timely processing of the operation and maintenance personnel.
[0102] In some examples, the data capture layer, the stream processing layer, and the data writing layer are optimized respectively. Specifically, in the data capture layer, a binlog monitoring mechanism is provided, and Canal or a self-developed CDC component is used to capture binlog change events of MySQL. A change event standardization is also provided to parse the original binlog into a standard FieldChange object (field change object) containing table name, primary key, and change field information. A message queue buffer is also provided to realize data buffering and decoupling through a Kafka (a distributed message queue system) / RocketMQ (a distributed message queue system) message queue.
[0103] Figure 3 For an optimization schematic diagram of the stream processing layer in one embodiment, refer to Figure 3 As shown, the change events are collected and grouped, the time window is slid to pull data, and the data update amount in a unit of time is reduced. Batch aggregation and merging processing are performed on document changes, and the latest stable change is taken for the update of the same document in the time window to reduce repeated updates. For field-level extraction, the binlog is preprocessed, the changed fields are transmitted to the downstream through a Flink state operator, and after the asynchronous query returns data, a local update strategy is performed on the document to reduce the byte number of a single document and realize incremental update optimization.
[0104] Figure 4 For an optimization schematic diagram of the data writing layer in one embodiment, refer to Figure 4As shown, the optimization strategy includes changing the aggregation strategy, where a time window aggregation is set, a reasonable time window (such as 10 seconds) is set to aggregate multiple changes in a short time; a count trigger is set, which triggers processing when the cumulative number of documents reaches a threshold (such as 2000); a grouping aggregation is set, which groups by document ID to ensure that multiple changes to the same document can be effectively merged. The optimization strategy also includes field-level update optimization, where a change field tracking is set to accurately record the field information involved in each change; a field filtering extraction is set to extract and process only the fields that have changed, rather than the entire document; a field merging logic is set to retain only the latest value for the same field in multiple changes to the same document; a version control process is set to use optimistic locking or version numbers to avoid concurrent update conflicts. The optimization strategy also includes an Elasticsearch write strategy, where a bulk write API is set to use the ES Bulk API to batch process document write requests; a partial update mode is set to use the _update API of ES to implement document partial field updates; a refresh interval optimization is set to dynamically adjust the refresh_interval parameter according to the write load; a transaction log configuration is set to optimize translog settings to reduce I / O operations; an index setting optimization is set to reasonably set parameters such as the number of shards and the number of replicas to improve write performance.
[0105] In the embodiments of the present disclosure, by batch processing, the proportion of write requests is effectively reduced, the network transmission amount is reduced, the efficiency of data synchronization is improved, energy is saved, and the method is suitable for more application scenarios.
[0106] It should be understood that, although each step in the flowchart involved in each of the above-described embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above-described embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.
[0107] Based on the same inventive concept, the embodiments of the present application also provide a data processing device for implementing the above-mentioned data processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data processing device embodiments provided below can refer to the limitations of the data processing method described above, and will not be repeated here.
[0108] In one example embodiment, as shown in Figure 5 a data processing apparatus 500 is provided, comprising:
[0109] an obtaining module 510, configured to, in response to a trigger operation of data modification on a document in a source database, obtain a data modification log corresponding to the trigger operation, and generate event structure information corresponding to the data modification according to the data modification log, the event structure information being obtained after being encapsulated based on a preset standard event structure;
[0110] a merging module 520, configured to store the event structure information into a target message queue, and in a case where there is initial event structure information matching the event structure information in the target message queue, merge the event structure information and the initial event structure information;
[0111] an updating module 530, configured to periodically obtain event structure information corresponding to the source database in the target message queue according to a preset period, and update a corresponding document in a target database according to the event structure information.
[0112] In one example embodiment, the merging module is further configured to:
[0113] determine a target document corresponding to the event structure information;
[0114] in a case where there is initial event structure information for the target document in the target message queue, obtain an initial data change field corresponding to the initial event structure information;
[0115] generate a target data change field according to the initial data change field and a data change field corresponding to the event structure information, to obtain merged event structure information.
[0116] In one example embodiment, the merging module is further configured to:
[0117] obtain candidate event structure information in a preset time period in the target message queue;
[0118] determine a document identification field corresponding to the candidate event structure information according to the candidate event structure information;
[0119] in a case where the document identification field matches the target document, determine the candidate event structure information as initial event structure information, and obtain an initial data change field corresponding to the initial event structure information.
[0120] In one example embodiment, the obtaining module is further configured to:
[0121] analyzing the data modification log, determining a modification type corresponding to the data modification, target field information in a document corresponding to the data modification, and before-and-after values of the target field;
[0122] based on a preset standard event structure, encapsulating the modification type, the target field information, and the before-and-after values to obtain event structure information.
[0123] In one of the embodiments, the updating module is further configured to:
[0124] determine, according to the target field information corresponding to the event structure information, a field to be updated in a document in the target database that matches the event structure information;
[0125] update the field to be updated based on the modification type corresponding to the event structure information and the before-and-after values.
[0126] In one of the embodiments, the apparatus further includes a preset period determination module configured to:
[0127] obtain a data volume of data updates corresponding to the target database in a preset historical period;
[0128] determine a preset period according to the data volume.
[0129] The modules in the data processing apparatus described above can be realized by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the modules.
[0130] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 6As shown in the figure. The computer device includes a processor, a memory, an Input / Output (I / O) interface, and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data modification logs and other data involved in the methods described in this embodiment. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a data processing method.
[0131] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.
[0132] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the method embodiments described above.
[0133] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments described above.
[0134] In one embodiment, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments described above.
[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant regulations.
[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.
[0137] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.
[0138] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.
Claims
1. A data processing method, characterized by, The method comprises: In response to a trigger operation of data modification on a document in a source database, obtaining a data modification log corresponding to the trigger operation, and generating event structure information corresponding to the data modification according to the data modification log, the event structure information being obtained after being encapsulated based on a preset standard event structure; Storing the event structure information into a target message queue, and in the case that there is initial event structure information matching the event structure information in the target message queue, merging the event structure information and the initial event structure information; Periodically obtaining event structure information corresponding to the source database in the target message queue according to a preset period, and updating a corresponding document in a target database according to the event structure information.
2. The method of claim 1, wherein, In the case that there is initial event structure information matching the event structure information in the target message queue, the event structure information and the initial event structure information are merged, comprising: Determine the target document corresponding to the event structure information; In the case that there is initial event structure information corresponding to the target document in the target message queue, obtain the initial data change field corresponding to the initial event structure information; According to the initial data change field and the data change field corresponding to the event structure information, generate a target data change field to obtain the merged event structure information.
3. The method of claim 2, wherein, In the case that there is initial event structure information corresponding to the target document in the target message queue, the initial data change field corresponding to the initial event structure information is obtained, comprising: Obtain candidate event structure information in a preset period in the target message queue; According to the candidate event structure information, determine the document identification field corresponding to the candidate structure information; In the case that the document identification field matches the target document, determine that the candidate event structure information is initial event structure information, and obtain the initial data change field corresponding to the initial event structure information.
4. The method of claim 1, wherein, The event structure information corresponding to the data modification is generated according to the data modification log, comprising: Parsing the data modification log to determine the modification type corresponding to the data modification, the target field information in the document corresponding to the data modification, and the change before and after value of the target field; Based on the preset standard event structure, the modification type, the target field information, and the change before and after value are encapsulated to obtain event structure information.
5. The method of claim 4, wherein, According to the event structure information, the corresponding document in the target database is updated, comprising: According to the target field information corresponding to the event structure information, determine the to-be-updated field in the document in the target database matching the event structure information; Based on the modification type corresponding to the event structure information and the change before and after value, update the to-be-updated field.
6. The method of claim 1, wherein, The determination method of the preset period comprises: Obtain the data amount of data update corresponding to the target database in a preset historical period; According to the data amount, determine a preset period.
7. A data processing apparatus, characterized by, The device comprises: An acquisition module is configured to, in response to a trigger operation of data modification on a document in a source database, acquire a data modification log corresponding to the trigger operation, and generate event structure information corresponding to the data modification according to the data modification log, the event structure information being obtained by encapsulating based on a preset standard event structure; A merging module is configured to store the event structure information into a target message queue, and in a case where initial event structure information matching the event structure information exists in the target message queue, merge the event structure information and the initial event structure information. An updating module is configured to periodically acquire event structure information corresponding to the source database in the target message queue according to a preset period, and update a corresponding document in a target database according to the event structure information.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.