A data processing method, device, equipment and readable storage medium

Abstract

The application discloses a data processing method, device and equipment and a readable storage medium. The method comprises the following steps: creating a local transaction, obtaining data information corresponding to a target business request in a to-be-processed state in the local transaction, and using the data information as business data; creating a dynamic proxy method, and performing asynchronous processing on a target business corresponding to the business data according to the dynamic proxy method; if the processing on the target business is successful, updating the state of the business data according to the processing result; and if the processing on the target business is unsuccessful, reprocessing the business data. In this way, the creation of the local transaction and the dynamic proxy ensures that the data will not be lost, avoids the invasiveness of the business code, reduces the system coupling degree through asynchronous processing, improves the system performance, and realizes the final consistency of the data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] When a server processes business requests, it typically needs to establish a data transmission connection with a database and retrieve and process the data within that database. This data can be stored in multiple databases of different types. Retrieving and processing data from a single database requires program code that matches the database type. Current technologies primarily use distributed transactions in two phases and compensating transactions for processing. This necessitates programmers handling compensating code for different scenarios, and can also lead to problems such as synchronization blocking and data inconsistency. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method, apparatus, device, and readable storage medium that ensures data is not lost and avoids intrusive business code by creating local transactions and dynamic proxies. At the same time, asynchronous processing reduces system coupling, improves system performance, and achieves eventual data consistency.

[0004] To achieve the above objectives:

[0005] In a first aspect, embodiments of this application provide a data processing method, including the following steps:

[0006] Create a local transaction and obtain the data information corresponding to the target business request that is in a pending state in the local transaction. The data information is used as business data.

[0007] Create a dynamic proxy method, and perform asynchronous processing on the target business corresponding to the business data according to the dynamic proxy method;

[0008] If the target service is successfully processed, the status of the service data is updated based on the processing result.

[0009] If the processing of the target service is unsuccessful, the service data will be processed again.

[0010] Optionally, the step of obtaining data information corresponding to the target business request that is in a pending state in the local transaction, wherein the data information is used as business data, includes:

[0011] The database supporting the target service is queried by polling, and the data information corresponding to the target service request that is in the pending state in the local transaction is obtained from the database; the target service request includes a request for the status information of the target service to be sent and a request for the status information of the processing exception that needs to be retried.

[0012] The acquired target business request is cached to generate a message queue, and after the business request is sent, the status of the target business information cached in the message queue is changed to the sent status.

[0013] Optionally, the method for creating a dynamic proxy, which asynchronously processes the target business corresponding to the business data according to the dynamic proxy method, includes:

[0014] Based on the business data, create a proxy object corresponding to the target business;

[0015] The business data is obtained through the proxy object and asynchronously written into the database that supports the target business.

[0016] Optionally, if the processing of the target service is successful, updating the status of the service data according to the processing result includes:

[0017] If the target business is successfully processed, the state of the target business is changed in a local transaction through asynchronous processing based on the result of the target business processing, and the target business is migrated to a preset historical table for asynchronous updating.

[0018] Optionally, if the processing of the target service fails, the service data is reprocessed, including:

[0019] If the target service processing fails, the target service is marked as processing failed, and the service request information is resent during the next poll.

[0020] Optionally, the method further includes:

[0021] If the number of retransmissions of the service data exceeds the preset retry threshold, the service data processing is deemed to have failed.

[0022] If the business data processing is determined to have failed, an alert message is issued, and the processing result is asynchronously migrated to a preset failure table in a local transaction for asynchronous updating.

[0023] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0024] The business monitoring unit is used to create a local transaction and obtain the data information corresponding to the target business request that is in the pending state in the local transaction. The data information is used as business data.

[0025] A data processing unit is used to create a dynamic proxy method and asynchronously process the target business corresponding to the business data according to the dynamic proxy method.

[0026] The message synchronization unit is used to update the status of the service data according to the processing result if the processing of the target service is successful; and to reprocess the service data if the processing of the target service is unsuccessful.

[0027] Optionally, the service monitoring unit is specifically used for:

[0028] The database supporting the target service is queried by polling, and the data information corresponding to the target service request in the pending state of the local transaction is obtained from the database; the target service request includes the request for the status information of the target service to be sent and the request for the status information of the processing exception that needs to be retried; the obtained target service request is cached to generate a message queue, and after the service request is sent, the status of the target service information cached in the message queue is changed to the sent status.

[0029] Thirdly, embodiments of this application disclose an electronic device, including: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the data processing method as described in the first aspect.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to implement the data processing method as described in the first aspect.

[0031] The data processing method, apparatus, device, and readable storage medium provided in this application embodiment include: creating a local transaction; acquiring data information corresponding to a target business request in a pending state within the local transaction, wherein the data information is used as business data; creating a dynamic proxy method; asynchronously processing the target business corresponding to the business data according to the dynamic proxy method; if the processing of the target business is successful, updating the state of the business data according to the processing result; if the processing of the target business is unsuccessful, reprocessing the business data. Thus, by creating a local transaction and a dynamic proxy, data loss is ensured, and intrusive business code is avoided. Simultaneously, asynchronous processing reduces system coupling, improves system performance, and achieves eventual data consistency. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the main flow of a data processing method provided in a preferred embodiment of the present invention;

[0033] Figure 2 A detailed flowchart of a data processing method provided in a preferred embodiment of the present invention is shown below;

[0034] Figure 3 This is a schematic diagram illustrating the specific process of business message state transformation in a data processing method according to a preferred embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the structure of a data processing device according to a preferred embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of a data processing device provided in another preferred embodiment of the present invention. Detailed Implementation

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0038] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0039] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0040] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0041] See Figure 1 This application provides a data processing method, which can be executed by a data processing device provided in this application. The data processing device can be implemented in software and / or hardware. In this embodiment, the data processing device is applied to a server as an example. The data processing method provided in this embodiment includes the following steps:

[0042] Step S101: Create a local transaction and obtain the data information corresponding to the target business request that is in the pending state in the local transaction. The data information is used as business data.

[0043] In one embodiment, the database supporting the target service is queried by polling, and the data information corresponding to the target service request that is in the pending state in the local transaction is obtained from the database; the target service request includes a request for the status information of the target service to be sent and a request for the status information of the processing exception that needs to be retried; here, the polling interval can be set according to different timeliness requirements, and different dimensions of isolation can be performed for different service messages.

[0044] Here, the acquired target service requests are cached to generate a message queue to reduce unnecessary message table scans. When target service information is generated or needs to be reprocessed, the target service is cached and its status is updated to "pending to send" in the message queue. After the service request is sent, the status of the target service cached in the message queue is updated to "sent".

[0045] Step S102: Create a dynamic proxy method and perform asynchronous processing on the target business corresponding to the business data according to the dynamic proxy method.

[0046] In one embodiment, a proxy object corresponding to the target business is created based on the acquired business data to realize service invocation and message status processing, avoiding intrusion by the business proxy. Here, the business data is obtained through the proxy object, and the business data is asynchronously written to the database supporting the target business. This includes the proxy object asynchronously invoking the business data on the server side.

[0047] Step S103: If the processing of the target service is successful, update the status of the service data according to the processing result.

[0048] In one embodiment, if the target service is successfully processed, the state of the target service is changed in a local transaction through asynchronous processing based on the result of the target service processing, and the target service is migrated to a preset historical table for asynchronous updating.

[0049] Step S104: If the processing of the target service is unsuccessful, the service data shall be processed again.

[0050] In one embodiment, if the target service processing fails, the target service is marked as processing failed, and the service request information is resent during the next poll. It is determined whether the number of resentments of the service data exceeds a preset return threshold. Here, if it is determined that the number of resentments of the service data does not exceed the preset return threshold, the service request information is resent according to the returned service data. If the number of resentments of the service data exceeds the preset retry threshold, the service data processing is determined to have failed, the service request is no longer resent, a warning message is issued, and the processing result is asynchronously migrated to a preset failure table in a local transaction for asynchronous updating.

[0051] In summary, the data processing method provided in the above embodiments processes business in a timely manner through polling, reduces system coupling by asynchronously processing target business, improves system performance, and achieves eventual data consistency.

[0052] See Figure 2 This is another data processing method provided in the embodiments of this application. Figure 2 The method described in the text can be for... Figure 1 The method described in more detail is described here. Figure 2 Steps S201 to S202 in the process can correspond to Figure 1 Step S101 in the process, Figure 2 Steps S203 to S204 in the above can correspond to Figure 1 Step S102 in the process, Figure 2 Step S205 in the text can correspond to Figure 1 Steps S103 to S104 in the above steps, the data processing method provided in this embodiment specifically includes the following steps:

[0053] Step S201: Create a local transaction.

[0054] Step S202: Task scheduling polls the message table.

[0055] In one embodiment, the database supporting the target service is queried by polling, and the data information corresponding to the target service request that is in the pending state in the local transaction is obtained from the database; the target service request includes a request for the status information of the target service to be sent and a request for the status information of the processing exception that needs to be retried; here, the polling interval can be set according to different timeliness requirements, and different dimensions of isolation can be performed for different service messages.

[0056] Step S203: Enter the MQ message queue.

[0057] In one embodiment, the acquired target service request is cached to generate a message queue to reduce unnecessary message table scanning. When target service information is generated or needs to be reprocessed, the target service is cached and its status is updated to "pending to send" in the message queue. After the service request is sent, the status of the target service cached in the message queue is updated to "sent".

[0058] Step S204: The dynamic proxy performs asynchronous processing.

[0059] In one embodiment, a proxy object corresponding to the target business is created based on the acquired business data to realize service invocation and message status processing, avoiding intrusion by the business proxy. Here, the business data is obtained through the proxy object, and the business data is asynchronously written to the database supporting the target business. This includes the proxy object asynchronously invoking the business data on the server side.

[0060] Step S205: Update the status accordingly based on the processing result.

[0061] In one embodiment, if the target service is successfully processed, the state of the target service is changed in a local transaction through asynchronous processing based on the result of the target service processing, and the target service is migrated to a preset historical table for asynchronous updating.

[0062] In one embodiment, if the target service processing fails, the target service is marked as processing failed, and the system waits for the next polling to resend the service request information. It is then determined whether the number of resends of the service data exceeds a preset retry threshold. Here, if the number of resends of the service data does not exceed the preset retry threshold, the service request information is resent based on the returned service data. If the number of resends of the service data exceeds the preset retry threshold, the service data processing is determined to have failed, the service request is no longer resent, a warning message is issued, and the processing result is asynchronously migrated to a preset failure table in a local transaction for asynchronous updating.

[0063] In summary, the data processing method provided in the above embodiments creates a local transaction to integrate writing and business processing within a single transaction, ensuring that data is not lost. It also uses a dynamic proxy to implement service calls and message status processing, avoiding intrusion of business code and eliminating the need to rely on third-party services during business processing.

[0064] See Figure 3 This document provides a detailed description of the business message state transition in a data processing method according to an embodiment of this application, including:

[0065] Step S301: Business messages to be processed.

[0066] In one embodiment, the database supporting the service is queried by polling, and the service information in the pending state of the local transaction is obtained from the database. The service information in the pending state includes the status information of the service to be sent and the status information of the service to be retried due to processing exception.

[0067] Step S302: Send service request information.

[0068] In one embodiment, the acquired pending business information is cached to generate a message queue to reduce unnecessary message table scans. When business information is generated or needs to be reprocessed, the business is cached and its status is updated to "pending to send" in the message queue. After the business request is sent, the status of the business cached in the message queue is updated to "sent".

[0069] Step S303: Determine whether the business was successfully processed.

[0070] Step S304: If the business processing is successful, then the business is migrated to the historical table.

[0071] In one embodiment, if the business processing is successful, the state of the business is changed in a local transaction through asynchronous processing based on the business processing result, and the business is migrated to a preset historical table for asynchronous updating.

[0072] Step S305: If the service processing is unsuccessful, determine whether the number of times the service is resent exceeds the preset retry threshold.

[0073] In one embodiment, if the service processing fails, the target service is marked as processing failed, and the service request information is resent in the next poll. It is also determined whether the number of times the service data is resent exceeds a preset retry threshold.

[0074] Step S306: If it is determined that the number of unsuccessful retransmissions exceeds the preset retry threshold, then the service is migrated to the failure table.

[0075] In one embodiment, if the number of retransmissions of the service data exceeds a preset retry threshold, the service data processing is determined to have failed, the service request is no longer retransmitted, a warning message is issued, and the processing result is asynchronously migrated to a preset failure table in a local transaction for asynchronous updating.

[0076] In summary, the business message state transformation method provided above reduces system coupling and improves system performance through asynchronous processing. Simultaneously, it ensures data consistency by rolling back business data from failed calls.

[0077] Based on the same inventive concept as the foregoing embodiments, the method provided by the foregoing embodiments will be described in detail below through a specific example.

[0078] See Figure 4 The data processing apparatus provided in this application embodiment includes a service monitoring unit, a data processing unit, and a message synchronization unit, wherein...

[0079] The business monitoring unit is used to create a local transaction and obtain the data information corresponding to the target business request that is in the pending state in the local transaction. The data information is used as business data.

[0080] A data processing unit is used to create a dynamic proxy method and asynchronously process the target business corresponding to the business data according to the dynamic proxy method.

[0081] The message synchronization unit is used to update the status of the service data according to the processing result if the processing of the target service is successful; and to reprocess the service data if the processing of the target service is unsuccessful.

[0082] In one embodiment, the service monitoring unit is specifically used to query the database supporting the target service through polling, and obtain the data information corresponding to the target service request that is in the pending state in the local transaction from the database; the target service request includes a request for the status information of the target service to be sent and a request for the status information of the processing exception that needs to be retried; here, the polling interval can be set according to different timeliness requirements, and different dimensions of isolation can be performed for different service messages.

[0083] Here, the acquired target service requests are cached to generate a message queue to reduce unnecessary message table scans. When target service information is generated or needs to be reprocessed, the target service is cached and its status is updated to "pending to send" in the message queue. After the service request is sent, the status of the target service cached in the message queue is updated to "sent".

[0084] In one embodiment, the data processing unit is specifically used to create a proxy object corresponding to the target business based on the acquired business data, so as to realize service invocation and message status processing and avoid intrusion of the business proxy. Here, the business data is obtained through the proxy object, and the business data is asynchronously written to the database supporting the target business, including asynchronous invocation processing on the server side when the proxy object processes the business data.

[0085] In one embodiment, the message synchronization unit is specifically used to, if the target service is successfully processed, modify the state of the target service in a local transaction through asynchronous processing based on the target service processing result, and migrate the target service to a preset historical table for asynchronous updating.

[0086] Here, if the target service processing fails, the target service is marked as processing failed, and the service request information is resent in the next poll. It is also determined whether the number of resentments of the service data exceeds a preset retry threshold. If the number of resentments of the service data does not exceed the preset retry threshold, the service request information is resent based on the returned service data. If the number of resentments of the service data exceeds the preset retry threshold, the service data processing is determined to have failed, the service request is no longer resent, a warning message is issued, and the processing result is asynchronously migrated to a preset failure table in a local transaction for asynchronous updating.

[0087] In summary, the data processing apparatus provided in the above embodiments helps to reduce system coupling, improve data processing efficiency, and ensure data consistency by creating local transactions through the business monitoring unit, creating dynamic proxies through the data processing unit, and asynchronously processing business data through the message synchronization unit to perform asynchronous state updates based on the processing results.

[0088] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a data processing apparatus, such as... Figure 5 As shown, the device includes: a processor 410 and a memory 411 storing a computer program; wherein, Figure 5The processor 410 shown in the diagram does not indicate that there is only one processor 410, but only indicates the positional relationship of the processor 410 relative to other devices. In practical applications, there can be one or more processors 410; similarly, Figure 5 The memory 411 illustrated herein has the same meaning, that is, it is only used to indicate the positional relationship of memory 411 relative to other devices. In practical applications, there can be one or more memories 411. When the processor 410 runs the computer program, the data processing method applied to the above-described device is implemented.

[0089] The device may also include at least one network interface 412. The various components of the device are coupled together via a bus system 413. It is understood that the bus system 413 is used to implement communication between these components. In addition to a data bus, the bus system 413 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general designated all buses as Bus System 413.

[0090] The memory 411 can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 411 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0091] The memory 411 in this embodiment of the invention is used to store various types of data to support the operation of the device. Examples of such data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment of the invention can be included in the application.

[0092] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is executed by a processor, it implements the data processing method applied to the above-mentioned device. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figure 1 The description of the illustrated embodiments will not be repeated here.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0095] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing method, characterized by, The method comprises the following steps: creating a local transaction, obtaining data information corresponding to a target service request in a to-be-processed state in the local transaction, the data information being used as service data; creating a dynamic proxy method, and performing asynchronous processing on a target service corresponding to the service data according to the dynamic proxy method; if the processing on the target service is successful, updating the state of the service data according to the processing result; if the processing on the target service is unsuccessful, reprocessing the service data; the creating of the dynamic proxy method, the asynchronous processing on the target service corresponding to the service data according to the dynamic proxy method comprises: creating a proxy object corresponding to the target service according to the service data; obtaining the service data through the proxy object, and writing the service data into a database supporting the target service asynchronously; if the processing on the target service is successful, changing the state of the target service in the local transaction according to the processing result of the target service through asynchronous processing, and migrating the target service to a preset history table for asynchronous updating. the obtaining of the data information corresponding to the target service request in the to-be-processed state in the local transaction, the data information being used as service data comprises:

2. The method of claim 1, wherein, querying a database supporting the target service through polling, and obtaining the data information corresponding to the target service request in the to-be-processed state in the local transaction from the database; the target service request comprises a request for to-be-sent state information of the target service and a request for retrying state information of an exception processing; caching the obtained target service request to generate a message queue, and changing the state of the target service information cached in the message queue to a sent state after the service request is sent. if the processing on the target service is unsuccessful, reprocessing the service data comprises:

3. The method of claim 1, wherein, if the processing on the target service is unsuccessful, marking the target service as processing failure, and waiting for the next polling to resend the service request information. The method further comprises:

4. The method of claim 3, wherein, if the number of resending times of the service data exceeds a preset retry number threshold, determining that the service data processing fails; if it is determined that the service data processing fails, issuing a warning information, and migrating the target service to a preset failure table in the local transaction for asynchronous updating according to the processing result. The device comprises:

5. A data processing apparatus, characterized by, a service monitoring unit configured to create a local transaction, and obtain data information corresponding to a target service request in a to-be-processed state in the local transaction, the data information being used as service data; a data processing unit configured to create a proxy object corresponding to the target service according to the service data, and obtain the service data through the proxy object, and write the service data into a database supporting the target service asynchronously. ​ The message synchronization unit is configured to, if the target service processing is successful, change the state of the target service in the local transaction according to the target service processing result through asynchronous processing, and migrate the target service to a preset history table for asynchronous updating; and if the processing of the target service is unsuccessful, re-process the service data.

6. The apparatus of claim 5, wherein, The service monitoring unit is specifically configured to: query a database supporting the target service through polling, and obtain data information of a target service request in a to-be-processed state in the local transaction from the database; the target service request includes a request for to-be-sent state information of the target service and a request for retrying state information of a processing exception; cache the obtained target service request to generate a message queue, and change the target service information state cached in the message queue to a sent state after the target service request is sent. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the data processing method in any one of claims 1 to 4.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the data processing method in any one of claims 1 to 4.

Images