Message processing method and apparatus, electronic device, and storage medium

Abstract

The application provides a message processing method and device, electronic equipment and storage medium, and the generation of a synchronization message is completed by using a synchronization plug-in constructed based on a callback function, the original code of a database is avoided from being modified, and the amount of code modification in the synchronization message generation process is reduced; the synchronization message is generated by using the synchronization plug-in or a synchronization component, the synchronization message interruption problem caused by the working exception of a single synchronization message generation data link is solved, and therefore the stability of the synchronization message generation is improved; the synchronization message is processed according to a password, the repeated synchronization message is marked by using the processing state of the synchronization message in the password, the repeated synchronization message is identified, and therefore the idempotency of the service when the synchronization message is repeated is ensured.

Description

Technical Field

[0001] This application relates to the field of database control technology, and in particular to a message processing method, apparatus, electronic device and storage medium. Background Technology

[0002] CDC (Change Data Capture) is the process of identifying and capturing changes made to data in a database, such as the insertion, updating, and deletion of data or tables, then recording these changes in the order they occurred and transmitting them to downstream processes or systems in real time via message middleware.

[0003] Typically, Databus (an open-source change data synchronization component) is used to retrieve database change data. Databus decouples data production and usage in the database, improving the scalability of the overall database application architecture. For subsequent change data processing, the message user only needs to handle Databus synchronization messages. However, since the normal operation of the message user depends on the working state of Databus, any malfunction in Databus will cause malfunctions in the message user. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a message processing method, apparatus, electronic device and storage medium to solve or partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the first aspect of this application provides a message processing method, comprising:

[0006] Retrieve the data operation set from the database;

[0007] A callback function is constructed based on the data operation set, and the callback function is used to generate the synchronization message output by the synchronization plugin;

[0008] The response data of the callback function is used as the synchronization message.

[0009] A second aspect of this application provides another message processing method, the method comprising:

[0010] Synchronously retrieve changed data from the database;

[0011] Generate a synchronization message based on the changed data using a synchronization plugin and / or synchronization component;

[0012] The synchronization message is sent to the synchronization message queue for processing by the message user.

[0013] A third aspect of this application provides an apparatus for building a synchronization plugin, comprising:

[0014] The acquisition module is configured to acquire the data operation set from the database;

[0015] The building module is configured to build a callback function based on the data operation set, the callback function being used to generate a synchronization message output by the synchronization plugin;

[0016] The messaging module is configured to use the response data of the callback function as the synchronization message.

[0017] A fourth aspect of this application provides an apparatus for generating synchronization messages, comprising:

[0018] The synchronization module is configured to synchronously retrieve changed data from the database.

[0019] The generation module is configured to generate a synchronization message based on the changed data using a synchronization plugin and / or a synchronization component.

[0020] The sending module is configured to send the synchronization message to a synchronization message queue for processing by the message user.

[0021] The fifth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in any of the above.

[0022] A sixth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in any of the above-mentioned methods.

[0023] As can be seen from the above, the message processing method, apparatus, electronic device, and storage medium provided in this application utilize a synchronization plugin built based on callback functions to generate synchronization messages, avoiding modifications to the original database code and reducing the amount of code modification during the synchronization message generation process. By generating synchronization messages through a synchronization plugin or synchronization component, the problem of synchronization message interruption caused by the abnormal operation of a single synchronization message generation data chain is solved, thereby improving the stability of synchronization message generation. By processing synchronization messages according to a password and marking duplicate synchronization messages based on the processing status of the synchronization messages in the password, duplicate synchronization messages are identified, thus ensuring the idempotency of the service when synchronization messages are repeated. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the process of generating synchronous messages in related technologies;

[0026] Figure 2 This is a schematic diagram illustrating the steps of a message processing method according to an embodiment of this application;

[0027] Figure 3a This is a schematic diagram illustrating the steps of another message processing method according to an embodiment of this application;

[0028] Figure 3b This is a flowchart illustrating another message processing method according to an embodiment of this application;

[0029] Figure 3c This is a schematic diagram of a process for generating synchronization messages according to an embodiment of this application.

[0030] Figure 3d This is a schematic diagram of the process for obtaining the working status of the synchronization component according to an embodiment of this application;

[0031] Figure 3e This is a schematic diagram illustrating the steps of message processing by a message user in an embodiment of this application.

[0032] Figure 3f This is a schematic diagram of a process for processing synchronization messages according to an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of the structure of a message processing device according to an embodiment of this application;

[0034] Figure 5 This is a schematic diagram of another message processing device according to an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0037] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0038] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0039] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this application's technical solution, based on the prompt message.

[0040] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation of this application. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0042] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0044] The principles and spirit of this application will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0045] According to embodiments of this application, a message processing method, apparatus, electronic device, and storage medium are proposed.

[0046] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0047] The principles and spirit of this application will be explained in detail below with reference to several representative embodiments.

[0048] In related technologies, databases need an architecture capable of capturing changes to the data within the database, so that these changes can be used by other components. For example, ... Figure 1 As shown, in the heterogeneous database system 100, each organization has numerous subordinate units, and each subordinate unit maintains its own relatively independent database 101. However, administrators need to be concerned with global, consistent, and complete change data. This change data needs to be extracted, cleaned, and transformed from the databases 101 maintained by each subordinate unit, and finally loaded into a unified standard structure (such as...). Figure 1 In the message queue 103 shown. However, when the synchronization component 102 used to synchronize changed data in the database malfunctions, it will affect the generation of the synchronization message corresponding to the changed data, and thus affect the subsequent processing of the synchronization message by the message user 104.

[0049] To address the aforementioned problems, this application provides a message processing method, apparatus, electronic device, and storage medium. In the construction process of the synchronization plugin in this application embodiment, a callback function is constructed based on the database's data operation set, and the response data of the callback function is used as the synchronization message generated by the synchronization plugin. This approach of constructing the database plugin based on callback functions avoids modification of the original database code, reduces the amount of code modification during synchronization message generation, and the database plugin has the advantages of simple deployment, non-intrusiveness, flexible customization, and free expansion. The synchronization plugin constructed using the above scheme also possesses these advantages. In the synchronization message generation process in this application embodiment, synchronization messages are generated based on changed data through a synchronization plugin or synchronization component, solving the problem of synchronization message interruption caused by malfunctions in the single synchronization message generation data chain, thereby improving the stability of synchronization message generation. In the synchronization message processing process in this application embodiment, synchronization messages are processed according to a password, and duplicate synchronization messages are marked based on the processing status of the synchronization message in the password, achieving the identification of duplicate synchronization messages and thus ensuring the idempotency of the service when synchronization messages are repeated.

[0050] refer to Figure 2 A message processing method according to this embodiment may include the following steps:

[0051] Step 201: Obtain the data operation set from the database.

[0052] In this step, the construction process of the synchronization plugin requires the plugin's injection point. In this embodiment, the injection point can be the database's data operation area. The database refers to a system capable of providing data services. In this embodiment, the database can be a management system connected to the synchronization component or plugin, capable of providing data services. For example, the database can be MySQL (a relational database management system). The data operation set refers to the set of operations corresponding to changes in the database. In this embodiment, the data operation set can be a set of operations that modify data, such as Create, Update, and Delete. This provides the operational basis for the subsequent construction of the synchronization plugin's callback functions.

[0053] Step 202: Construct a callback function based on the data operation set. The callback function is used to generate the synchronization message output by the synchronization plugin.

[0054] In this step, the callback function refers to a function that can generate synchronization messages based on data operations. In this embodiment, the callback function can be injected as a plugin, such as the Callback function in Go's database connection and interaction framework. This provides a functional basis for the subsequent generation of synchronization messages, avoiding modifications to the original database code and reducing the amount of code modification during the synchronization message generation process.

[0055] Step 203: Use the response data of the callback function as the synchronization message.

[0056] In this step, response data refers to the output data when the callback function is called. In this embodiment, the response data can be the output data of the callback function injected as a plugin when it is called. Synchronization message refers to a message that indicates a database change. In this embodiment, the synchronization message can be a processable message indicating a database change. Synchronization plugin refers to a plugin that can acquire synchronization messages. In this embodiment, the synchronization plugin can be a plugin connected to the database that can acquire synchronization messages. Thus, the generated synchronization message can provide a data foundation for subsequent message processing by the user.

[0057] The above approach utilizes a callback function-based synchronization plugin to generate synchronization messages, avoiding modifications to the original database code and reducing the amount of code modification required during the synchronization message generation process.

[0058] In some embodiments, step 202 specifically includes:

[0059] Step 2021: Obtain the callback container corresponding to each data operation in the data operation set to obtain the callback container set.

[0060] Step 2022: Select at least one callback container from the callback container set as the initial callback function.

[0061] Step 2023: Set the injection timing of the initial callback function using a Cartesian product.

[0062] Step 2024: Register the initial callback function as the callback function.

[0063] In the above scheme, the callback container set refers to a set of program processes that can run independently. In this embodiment, the callback container set can be a set of program processes that can run independently, corresponding to callback functions. For example, the callback container list of *gorm.Db corresponding to data operations in Grom. The injection timing refers to the time point at which the initial callback function can be injected. In this embodiment, the injection timing can be the time point at which the initial callback function can be injected, corresponding to data operations. For example, the set of multiple injection timings is A, and the set of data operations is B. The injection timing can be before or after the data operation, and the data operation can be Create, Update, or Delete. The scheme C for setting the initial callback function through the Cartesian product can be obtained according to the following formula: C = A × ... × A × B, where × represents the Cartesian product. Db represents the set of callback containers corresponding to step 2021, Callback represents the initial callback function corresponding to step 2022, Gorm operation and After / Before represent the injection timing setting process corresponding to step 2023, and Register represents the initial callback function registration process corresponding to step 2024.

[0064] The above approach provides a functional foundation for the generation of subsequent synchronization messages, avoiding modifications to the original database code and reducing the amount of code modification required during the synchronization message generation process.

[0065] refer to Figure 3a Another message processing method in this embodiment may include the following steps:

[0066] Step 301: Synchronously obtain the changed data from the database.

[0067] In this step, "changed data" refers to data that can represent changes in the database. In this embodiment, the changed data can be data corresponding to data operations that can represent changes in the database. This provides a data foundation for subsequent synchronization plugins or synchronization components to generate synchronization messages.

[0068] Step 302: Generate a synchronization message based on the changed data using a synchronization plugin and / or synchronization component.

[0069] In this step, the synchronization component refers to a component capable of acquiring synchronization messages. In this embodiment, the synchronization component can be a component connected to the database that can acquire synchronization messages. For example, the synchronization plugin can be a plugin built based on an ORM (Object Relational Mapping) framework and callback functions. The ORM framework can solve the incompatibility between object-oriented programming and relational databases. The synchronization component can be Databus (an open-source data synchronization component). The synchronization component decouples data generation and usage, improving the scalability of the overall architecture. Message users only need to process Databus data for subsequent data processing, and it can also be used in an event-driven architecture. In this way, generating synchronization messages through a synchronization plugin or synchronization component solves the problem of synchronization message interruption caused by the abnormal operation of a single synchronization message generation data chain, thereby improving the stability of synchronization message generation.

[0070] It is understandable that synchronization plugins or components can connect to the database simultaneously or separately to retrieve the corresponding synchronization messages. For example... Figure 3b As shown, synchronization plugin A and synchronization component B connect to the database to obtain synchronization messages, such as... Figure 3c As shown, synchronization plugin A connects to the database to obtain synchronization messages.

[0071] Step 303: Send the synchronization message to the synchronization message queue for processing by the message user.

[0072] In this step, the synchronization message queue refers to middleware capable of receiving synchronization messages generated by synchronization plugins or components. In this embodiment, the synchronization message queue can be middleware capable of pushing synchronization messages to message users and receiving synchronization messages generated by synchronization plugins or components. This provides a data foundation for subsequent processing of synchronization messages by message users.

[0073] The above solution, which generates synchronization messages through a synchronization plugin or component, solves the problem of synchronization message interruption caused by the malfunction of the data chain for generating a single synchronization message, thereby improving the stability of synchronization message generation.

[0074] In some embodiments, prior to step 302, the processing method may further include:

[0075] The synchronization plugin sends a request to the synchronization component.

[0076] The synchronization component obtains its working status according to the acquisition request and sends the working status to the synchronization plugin, wherein the working status includes a normal state and an abnormal state.

[0077] In the above scheme, the "acquisition request" refers to request data that can obtain the working status of the synchronization component. In this embodiment, the acquisition request can be sent by the synchronization plugin and can obtain request data that can obtain the working status of the synchronization component. The working status refers to data that can indicate whether the synchronization component is working normally or not. In this embodiment, the working status can be data sent to the synchronization plugin that can indicate whether the synchronization component is working normally or not. For example, an SLA (Service Level Agreement) can be selected as the working status of the synchronization component.

[0078] The above approach provides a data foundation for subsequent switching between synchronization plugins and synchronization components.

[0079] In some embodiments, sending an acquisition request to the synchronization component via the synchronization plugin includes:

[0080] The synchronization plugin sends the acquisition request to the synchronization component at a pre-set first time interval.

[0081] In the above scheme, the first time interval refers to the time for obtaining the working status of the synchronization component. In this embodiment, the first time interval may be the time for obtaining the working status of the synchronization component in the synchronization plugin.

[0082] Specifically, such as Figure 3d As shown, after the synchronization component and the synchronization plugin obtain the changed data, the synchronization plugin sends an acquisition request to the synchronization component at a first time interval. The synchronization component sends its working status to the synchronization plugin according to the acquisition request so that the synchronization plugin can generate a synchronization message based on the working status and the changed data.

[0083] The above approach provides a data foundation for subsequent switching between synchronization plugins and synchronization components.

[0084] In some embodiments, step 302 specifically includes:

[0085] In response to determining that the working state is normal, the synchronization component generates a synchronization message based on the changed data.

[0086] In response to determining that the working state is an abnormal state, the synchronization plugin generates a synchronization message based on the changed data.

[0087] In the above scheme, the normal state refers to the state when the synchronization component is working properly. In this embodiment, the normal state can be the state when the message user is processing the synchronization message normally and the synchronization component is working properly. For example, a working state with an SLA ≥ 99.99% is a normal state. The abnormal state refers to the state when the synchronization component is malfunctioning. In this embodiment, the abnormal state can be the state when the message user cannot process the synchronization message normally and the synchronization component is malfunctioning. For example, a working state with an SLA < 99.99% is an abnormal state.

[0088] The above approach provides an architectural foundation for generating synchronous messages using synchronous plugins or components in the future.

[0089] In some embodiments, generating a synchronization message based on the changed data using the synchronization plugin includes:

[0090] Step 3021: Classify the callback functions in the synchronization plugin according to data operations to obtain classification results;

[0091] Step 3022: Select at least one callback function from the classification results based on the data operation corresponding to the changed data;

[0092] Step 3023: Obtain the response data of the callback function according to the injection timing sequence corresponding to the selected callback function;

[0093] Step 3024: The response data is merged into the synchronization message.

[0094] In the above scheme, since changing data can correspond to multiple data operations, classifying the callback functions in the synchronization plugin according to data operations provides a sorting basis for subsequent callback function invocation based on the injection timing. Callback function 1, Callback function 2… represent the callback functions in the synchronization plugin in step 3021; ​​Classify represents the classification process of callback functions in step 3021; ​​Create, Callback function 1… represent the callback function classification corresponding to the Create operation in step 3022; before:all represents the response data of callback function 1 in step 3022; before:3after:1 represents the response data of callback function 2 in step 3022; after:1 represents the response data of callback function 3 in step 3022; and after:2 represents the response data of callback function 4 in step 3022.

[0095] The above approach improves the granularity of the synchronous message generation process by calling the callback function according to the injection timing when calling the callback function based on the changed data.

[0096] In some embodiments, after generating a synchronization message based on the changed data using the synchronization plugin, the method further includes:

[0097] In response to determining that the working state has changed from the abnormal state to the normal state, the synchronization plugin and the synchronization component generate a password based on the synchronization message, and send the password to the message queue so that the message user can determine the processing status of the synchronization message.

[0098] In the above scheme, the password refers to data that can mark the status of synchronous message processing. In this embodiment, the password can be generated by a synchronous plugin or synchronous component, which can mark the status of synchronous message processing. For example, the password corresponding to the synchronous message can be generated by using the MD5 (Message-Digest 5, a cryptographic hash function) encryption method.

[0099] The above scheme provides a data foundation for subsequent message users to process synchronized messages based on passwords.

[0100] In some embodiments, reference Figure 3e After step 303, another message processing method also includes the following steps:

[0101] Step 304: The message user receives the synchronization message and the password corresponding to the synchronization message.

[0102] In this step, to ensure the idempotency of the synchronization messages, that is, to avoid the situation of repeated processing of synchronization messages, the password corresponding to the synchronization message is received to provide a verification basis for subsequent processing of the synchronization message based on the password.

[0103] Step 305: The message user processes the synchronization message according to the password.

[0104] In this step, processing refers to the behavior of using synchronous messages. In this embodiment, processing can be the behavior of the message user using synchronous messages in the synchronous message queue. For example, the approval message user uses the updated approval list data, updated reminder data and updated reminder data in the database.

[0105] The above scheme processes synchronization messages based on passwords and marks duplicate synchronization messages by tracking their processing status within the password. This enables the identification of duplicate synchronization messages and ensures the idempotency of the service when synchronization messages are repeated.

[0106] In some embodiments, step 305 specifically includes:

[0107] The user of the message determines the processing status of the synchronization message based on the password status, and obtains the processing result.

[0108] The message user processes the synchronization message based on the processing result.

[0109] In the above scheme, the password status refers to the data that can represent different states of the password. In this embodiment, the password status can be data that is pre-set by the message user and can represent different states of the password. For example, the password label can be set using the MD5 encryption method, and the password status can be the hash value generated by MD5.

[0110] Specifically, the message user compares the password status with the password status corresponding to the pre-set processing status, and uses the comparison result as the processing result.

[0111] The above scheme processes synchronization messages based on passwords and marks duplicate synchronization messages by tracking their processing status within the password. This enables the identification of duplicate synchronization messages and ensures the idempotency of the service when synchronization messages are repeated.

[0112] In some embodiments, the step of having the message user process the synchronization message according to the processing result includes:

[0113] In response to determining that the processing result is unprocessed, the message user performs business processing according to the synchronization message, sets the password status corresponding to the synchronization message to processed, and stores the password status in the password database. The data in the password database is synchronized to the database periodically according to a pre-set second time interval.

[0114] In response to determining that the processing result has been processed, the message user abandons processing the synchronization message.

[0115] In the above scheme, the second time interval refers to the time for obtaining the password status. In this embodiment, the second time interval can be the time for the database to obtain the password status from the synchronization message list. The password database refers to a database that can store and manage passwords. In this embodiment, the password database can be a database that is synchronized with the synchronization message queue and can store and manage passwords. For example, the password database can be Redis (a key-value database).

[0116] Specifically, such as Figure 3f As shown, when the processing result is unprocessed, the message user performs business processing based on the synchronous message. That is, the message user processes the synchronous message, and then the message user generates a processed password status based on the synchronous message and stores the processed password status in the password database for updating the password status in the synchronous message list.

[0117] The above scheme identifies duplicate synchronization messages by using the password status, thereby ensuring the idempotency of the service when synchronization messages are repeated.

[0118] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0119] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] Based on the same inventive concept, and on the basis of the corresponding implementation schemes of the above-mentioned embodiments, the following specific implementations are possible.

[0121] Step 1: In Gorm (corresponding to the synchronization plugin in this application embodiment), a callback function is injected in the form of a plugin at the point where data modification is performed. The callback function will be called after the data is modified.

[0122] Step 2: Install a switch in the callback function. This switch determines whether the Databus (corresponding to the synchronization component in this embodiment) degradation scheme is enabled. The switch mainly consists of a monitor that periodically checks the Databus system's operation. When a problem is detected, the Databus degradation scheme is enabled, and the degradation scheme (corresponding to the process of generating synchronization messages using the synchronization plugin in this application) is executed. Monitoring continues, and the degradation scheme is disabled when the Databus system is running normally. The monitor will perform monitoring at fixed time intervals, d.

[0123] Step 3: When executing the callback function, check the switch status. If the switch is off, it means the Databus degradation scheme is not enabled, and the callback function ends and will not continue execution. Conversely, if the switch is on, meaning the Databus degradation scheme is enabled, the callback function will continue execution, and the data change information will be sent to the corresponding message queue for downstream service processing. In this embodiment, there is a time period that requires special attention: when the Databus system recovers from an anomaly and resumes normal operation, the switch closure will be delayed due to the monitor's monitoring interval (worst case d). This means that both schemes are running simultaneously, leading to duplicate messages in the message queue and duplicate message pushes from downstream services. In this situation, it is necessary to ensure the idempotency of the service. Idempotency can be achieved using a token mechanism. When the producer sends each piece of data, it generates a token based on the data modification operation information. When the downstream service processes the data, it verifies whether the token has been processed. If it hasn't been processed, business processing is performed. After processing, the token is stored in Redis with an expiration time set and its status set to processed. Then, the Redis data is periodically synchronized to the database for persistent storage. If the data has already been processed, no further processing is performed.

[0124] In addition, this embodiment also supports more detailed plugin configuration, such as setting the time point where the content of the information sent by the callback function is located: before a certain Gorm operation, after a certain Gorm operation, and the Gorm operation can be subdivided into: Create, Update, Delete, and other operations. For ease of description, let the set of different time points be A, the set of Gorm operations be B, then all the combined operation schemes C are the Cartesian product of A (A can be multiple times) and B: C = A × … × A × B (1) The process of function injection in the form of plugin in this embodiment is as follows: (a) Obtain the *gorm.Db corresponding to the database operation, and then obtain the callback container list of the corresponding *gorm.Db; (b) Next, create the callback container to be injected; (c) Next, set the timing of function injection. The timing here can be the timing of multiple constraints (corresponding to multiple A in formula (1)), for example, after event Event1, before Event2, Event3, etc.; (d) Finally, register the user-defined function in the callback container. When calling callback functions after a Gorm operation is completed, the calls need to be made according to these timing constraints. This is done by categorizing these callback functions based on their corresponding Gorm operations. Then, the callback functions within each Gorm operation class are topologically sorted to obtain a corresponding execution order list. It can be seen that the synchronization plugin is flexible and customizable, meaning that data changes can be captured flexibly. Furthermore, this embodiment supports independent deployment, meaning it can be used independently without using a traditional CDC implementation. This approach offers the advantages of simple deployment and non-intrusiveness.

[0125] Although this embodiment is implemented using the Gorm framework in the Golang language, it is theoretically generalizable. Assuming the ORM framework supports injecting callback functions as plugins, then the synchronous plugin solution in this embodiment can be used.

[0126] The message processing method of the above embodiments has the same beneficial effects as the message processing method embodiments in any of the foregoing embodiments, and will not be repeated here.

[0127] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a device for constructing a synchronization plugin.

[0128] refer to Figure 4 The message processing device includes:

[0129] Module 401 is configured to retrieve the data operation set from the database.

[0130] Module 402 is configured to construct a callback function based on the data operation set, the callback function being used to generate a synchronization message output by the synchronization plugin;

[0131] The message module 403 is configured to use the response data of the callback function as the synchronization message.

[0132] In some embodiments, the construction module 402 specifically includes:

[0133] Obtain the callback container corresponding to each data operation in the data operation set to obtain a callback container set;

[0134] Select at least one callback container from the set of callback containers as the initial callback function;

[0135] The timing of the initial callback function injection is set by the Cartesian product;

[0136] Register the initial callback function as the callback function.

[0137] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a device for generating synchronization messages.

[0138] refer to Figure 5 The other message processing device includes:

[0139] Synchronization module 501 is configured to synchronously acquire changed data from the database;

[0140] Generation module 502 is configured to generate a synchronization message based on the changed data using a synchronization plugin and / or synchronization component;

[0141] The sending module 503 is configured to send the synchronization message to the synchronization message queue for processing by the message user.

[0142] In some embodiments, the generating apparatus further includes:

[0143] The monitoring and sending module is configured to send an acquisition request to the synchronization component through the synchronization plugin;

[0144] The status acquisition module is configured to use the synchronization component to acquire the working status of the synchronization component according to the acquisition request, and send the working status to the synchronization plugin, wherein the working status includes a normal status and an abnormal status.

[0145] In some embodiments, the monitoring sending module is specifically configured as follows:

[0146] The synchronization plugin sends the acquisition request to the synchronization component at a pre-set first time interval.

[0147] In some embodiments, the generation module 502 specifically includes:

[0148] The component unit is configured to generate a synchronization message based on the change data using the synchronization component in response to determining that the working state is a normal state;

[0149] The plug-in unit is configured to generate a synchronization message based on the changed data in response to determining that the working state is an abnormal state.

[0150] In some embodiments, the plug-in unit is specifically configured as follows:

[0151] The callback functions in the synchronization plugin are categorized according to data operations to obtain the categorization results;

[0152] Select at least one callback function from the classification results based on the data operation corresponding to the changed data;

[0153] The response data of the selected callback function is obtained according to the injection timing sequence.

[0154] The response data is merged into the synchronization message.

[0155] In some embodiments, the plug-in unit is further configured to:

[0156] In response to determining that the working state has changed from the abnormal state to the normal state, the synchronization plugin and the synchronization component generate a password based on the synchronization message, and send the password to the message queue so that the message user can determine the processing status of the synchronization message.

[0157] In some embodiments, the other message processing apparatus further includes:

[0158] The receiving module 504 is configured to receive a synchronization message and the password corresponding to the synchronization message using the message user;

[0159] The processing module 505 is configured to process the synchronization message according to the password by the message user.

[0160] In some embodiments, the processing module 505 specifically includes:

[0161] The processing status unit is configured to determine the processing status of the synchronization message based on the password status of the password by the message user, and obtain the processing result.

[0162] The processing result unit is configured to process the synchronization message according to the processing result by the message user.

[0163] In some embodiments, the processing result unit is specifically configured as follows:

[0164] In response to determining that the processing result is unprocessed, the message user performs business processing according to the synchronization message, sets the password status corresponding to the synchronization message to processed, and stores the password status in the password database. The data in the password database is synchronized to the database periodically according to a pre-set second time interval.

[0165] In response to determining that the processing result has been processed, the message user abandons processing the synchronization message.

[0166] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0167] The apparatus described above is used to implement the corresponding message processing method in the embodiment shown in FIG3, and has the beneficial effects of the corresponding method embodiment, which will not be described again here.

[0168] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the message processing method described in any of the above embodiments.

[0169] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0170] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0171] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0172] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0173] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.).

[0174] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0175] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0176] The electronic devices described above are used to implement the corresponding message processing methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0177] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the message processing method as described in any of the above embodiments.

[0178] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0179] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the message processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0180] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0181] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0182] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0183] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A message processing method characterized by, include: Retrieve the data operation set from the database; A callback function is constructed based on the data operation set, and the callback function is used to generate the synchronization message output by the synchronization plugin; The callback function is obtained by registering the initial callback function; the initial callback function is obtained by selecting at least one callback container from the callback container set; the injection timing of the initial callback function is set according to the data operation set; the callback container set is obtained by obtaining the callback container corresponding to each data operation in the data operation set; the callback container is an independently running program process corresponding to the callback function. The response data of the callback function is used as the synchronization message.

2. A message processing method characterized by, The method includes: Synchronously retrieve changed data from the database; A request is sent to the synchronization component via a synchronization plugin; the synchronization plugin is a plugin built based on object-relational mapping and callback functions; the synchronization component is an open-source change data synchronization component. The synchronization component obtains its working status according to the acquisition request and sends the working status to the synchronization plugin, so that the synchronization plugin can determine whether to generate a synchronization message based on the changed data. The working status includes normal status and abnormal status. The synchronization plugin and / or the synchronization component are used to generate a synchronization message based on the changed data; The synchronization message is sent to the synchronization message queue for processing by the message user.

3. The treatment method according to claim 2, characterized in that, The step of sending an acquisition request to the synchronization component through the synchronization plugin includes: The synchronization plugin sends the acquisition request to the synchronization component at a pre-set first time interval.

4. The treatment method of claim 2, wherein The step of generating a synchronization message based on the changed data using a synchronization plugin and / or synchronization component includes: In response to determining that the working state is normal, the synchronization component generates a synchronization message based on the changed data. In response to determining that the working state is an abnormal state, the synchronization plugin generates a synchronization message based on the changed data.

5. The treatment method according to claim 4, characterized in that, The step of generating a synchronization message based on the changed data using the synchronization plugin includes: The callback functions in the synchronization plugin are categorized according to data operations to obtain the categorization results; Select at least one callback function from the classification results based on the data operation corresponding to the changed data; The response data of the selected callback function is obtained according to the injection timing sequence. The response data is merged into the synchronization message.

6. The treatment method of claim 4, wherein After generating a synchronization message based on the changed data using the synchronization plugin, the process also includes: In response to determining that the working state has changed from the abnormal state to the normal state, the synchronization plugin and the synchronization component generate a password based on the synchronization message, and send the password to the message queue so that the message user can determine the processing status of the synchronization message.

7. The treatment method according to claim 6, characterized in that, After sending the synchronization message to the synchronization message queue for processing by the message user, the method further includes: The message user receives the synchronization message and the password corresponding to the synchronization message; The message user processes the synchronization message according to the password.

8. The processing method according to claim 7, wherein the step of having the message user process the synchronization message according to the password includes: The user of the message determines the processing status of the synchronization message based on the password status, and obtains the processing result. The message user processes the synchronization message based on the processing result.

9. The treatment method according to claim 8, characterized in that, The step of the message user processing the synchronization message according to the processing result includes: In response to determining that the processing result is unprocessed, the message user performs business processing according to the synchronization message, sets the password status corresponding to the synchronization message to processed, and stores the password status in the password database. The data in the password database is synchronized to the database periodically according to a pre-set second time interval. In response to determining that the processing result has been processed, the message user abandons processing the synchronization message.

10. A message processing device, characterized by include: The acquisition module is configured to acquire the data operation set from the database; The building module is configured to build callback functions based on the data operation set; The callback function is obtained by registering the initial callback function; the initial callback function is obtained by selecting at least one callback container from the callback container set; the injection timing of the initial callback function is set according to the data operation set; the callback container set is obtained by obtaining the callback container corresponding to each data operation in the data operation set; the callback container is an independently running program process corresponding to the callback function. The messaging module is configured to use the response data of the callback function as a synchronization message generated by the synchronization plugin.

11. A message processing device, characterized by include: The synchronization module is configured to synchronously retrieve changed data from the database. The module for sending retrieval requests is configured to send retrieval requests to the synchronization component via the synchronization plugin; The synchronization plugin is a plugin built based on object-relational mapping and callback functions; the synchronization component is an open-source data change synchronization component. The acquisition module is configured to acquire the working status of the synchronization component according to the acquisition request using the synchronization component, and send the working status to the synchronization plugin, so that the synchronization plugin can determine whether to generate a synchronization message based on the changed data using the synchronization plugin or the synchronization component, wherein the working status includes a normal state and an abnormal state. The generation module is configured to generate a synchronization message based on the changed data using the synchronization plugin and / or the synchronization component. The sending module is configured to send the synchronization message to a synchronization message queue for processing by the message user.

12. An electronic device, comprising: The method includes a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 9.

13. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 9.

Images