Method, System, Device and Storage Medium for Data Processing

Abstract

The present disclosure relates to the field of big data technology, and in particular to a method and system for data processing, a device, and a storage medium. The method includes: in response to a received transaction request, executing at least one processing flow for transaction corresponding to the current transaction request according to the transaction request; collecting result data of the processing flow for transaction corresponding to the same transaction request; integrating the result data of the processing flow for transaction corresponding to the same transaction request, so as to write the transaction data obtained after integration into a message queue and send it to a data processing end, which can organize the result data of all processing flows for transaction corresponding to one user request into one message and write it into the message queue at one time, and can reduce the pressure on the message queue.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular, to a method and system for data processing, a device, and a storage medium. Background Art

[0002] In current Internet systems, it is necessary to analyze the data generated by complex services, such as analyzing logs, analyzing the reasons when an exception occurs, analyzing the indicator data of a certain service, etc., in order to understand the operation of the system by analyzing the data, clarify whether there are any abnormalities, whether there are points to be optimized, and how the specific operation of the service is, etc.

[0003] In existing technical solutions, the data generated by services is usually analyzed in the following three ways: First, analyze the system logs and view the system logs through keywords for analysis; Second, analyze through transaction data and view its operation by analyzing the transaction data; Third, collect and analyze data through probes. This solution belongs to non-embedded in the service, automatically collects system interfaces and displays them uniformly.

[0004] However, the above three methods have high operation difficulty, low data accuracy, high threshold, poor practicability, and poor flexibility. In addition, when a user request corresponds to multiple sub-process steps, if messages are sent for each process and step, then multiple messages need to be sent for one user request, which places a great pressure on the message queue that transfers the data generated by the service to the data processing end. Summary of the Invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide a method and system for data processing, a device, and a storage medium.

[0006] In a first aspect, embodiments of the present disclosure provide a method for data processing, including:

[0007] In response to a received transaction request, execute at least one processing process for the transaction corresponding to the current transaction request according to the transaction request;

[0008] Collect the result data of the processing process for the transaction corresponding to the same transaction request;

[0009] Integrate the result data of the processing process for the transaction corresponding to the same transaction request, so as to write the integrated transaction data into a message queue and send it to a data processing end.

[0010] In a possible implementation manner, the collecting the result data of the processing process for the transaction corresponding to the same transaction request includes:

[0011] Group and collect the result data of all processing flows for transactions corresponding to the same transaction request according to the relevance between the processing flows, so as to integrate all the grouped and collected result data corresponding to the same request.

[0012] In a possible implementation manner, before writing the transaction-related data obtained after integration into the message queue and sending it to the data processing end, the method further includes:

[0013] Compress the transaction-related data obtained after integration, so as to write the compressed transaction-related data into the message queue and send it to the data processing end.

[0014] In a possible implementation manner, the collection of the result data of the processing flows for transactions corresponding to the same transaction request is implemented by calling an interface. Second, an embodiment of the present disclosure provides a data processing method, which is applied to a data processing end, and the method includes:

[0015] Receive the message in the message queue and perform specialization processing on the transaction-related data in the message, where the specialization processing includes keyword extraction, secondary verification of data, and data filling;

[0016] Write the transaction-related data after specialization processing into the database;

[0017] In response to the received data query request, extract the corresponding transaction-related data from the database according to the keyword in the query request for the data query requester to view.

[0018] In a possible implementation manner, the method further includes:

[0019] Monitor the number of messages in the message queue and determine whether the number of messages in the message queue exceeds a preset threshold:

[0020] When the number of messages in the message queue exceeds the preset threshold, start an asynchronous thread to process the messages in the message queue.

[0021] In a possible implementation manner, the transaction-related data in the message is compressed data. Before performing specialization processing on the transaction-related data in the message, the method further includes:

[0022] Decompress the transaction-related data in the message, so as to perform specialization processing on the decompressed transaction-related data.

[0023] Third, an embodiment of the present disclosure provides a data processing system, including:

[0024] A data generation end, configured to, in response to a received transaction request, execute at least one processing flow for transaction corresponding to the current transaction request according to the transaction request; collect result data of the processing flow for transaction corresponding to the same transaction request; integrate the result data of the processing flow for transaction corresponding to the same transaction request, so as to write the integrated transaction data into a message queue and send it to a data processing end;

[0025] A data processing end, configured to receive messages from the message queue.

[0026] In a possible implementation manner, the data processing end is further configured to:

[0027] Perform specialization processing on the transaction data in the message, where the specialization processing includes keyword extraction, secondary verification of data, and data filling; write the specialized transaction data into a database; in response to a received data query request, extract corresponding transaction data from the database according to the keyword in the query request for the data query requester to view.

[0028] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0029] The memory is used to store a computer program;

[0030] The processor is configured to implement the above data processing method when executing the program stored on the memory.

[0031] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the above data processing method when executed by a processor.

[0032] The above technical solutions provided by the embodiments of the present disclosure have at least some or all of the following advantages compared with the prior art:

[0033] The data processing method and system according to the embodiments of the present disclosure, in response to a received transaction request, execute at least one processing flow for transaction corresponding to the current transaction request according to the transaction request; collect result data of the processing flow for transaction corresponding to the same transaction request; integrate the result data of the processing flow for transaction corresponding to the same transaction request, so as to write the integrated transaction data into a message queue and send it to a data processing end, can organize the result data of all processing flows for transaction corresponding to one user request into one message and write it into the message queue at one time, and can reduce the pressure on the message queue. Description of the Drawings

[0034] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments in accordance with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0035] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or related technologies. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0036] Figure 1 Schematically shows a flowchart of a method for data processing according to an embodiment of the present disclosure;

[0037] Figure 2 Schematically shows a detailed flowchart of step S2 according to an embodiment of the present disclosure;

[0038] Figure 3 Schematically shows a flowchart of a method for data processing according to another embodiment of the present disclosure;

[0039] Figure 4 Schematically shows a flowchart of a method for data processing according to still another embodiment of the present disclosure;

[0040] Figure 5 Schematically shows a flowchart of the processing of a message queue in the prior art;

[0041] Figure 6 Schematically shows a flowchart of the processing of a message queue according to an embodiment of the present disclosure;

[0042] Figure 7 Schematically shows a block diagram of the structure of a system for data processing according to an embodiment of the present disclosure;

[0043] FIG. 8(a) schematically shows a flowchart of the operation of the data generation end of a system for data processing according to an embodiment of the present disclosure;

[0044] FIG. 8(b) schematically shows a flowchart of the operation of the data processing end of a system for data processing according to an embodiment of the present disclosure; and

[0045] Figure 9 Schematically shows a block diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0047] See also Figure 1 , an embodiment of the present disclosure provides a data processing method, which is applied to a data generation end, and includes the following steps: S1, in response to a received transaction request, executing at least one processing flow for a transaction corresponding to the current transaction request according to the transaction request;

[0048] S2, collecting result data of the processing flow for the transaction corresponding to the same transaction request;

[0049] In actual applications, the collection of result data of all the processing flows for transactions corresponding to the same request is implemented by calling an interface. The method of calling an interface (API) for data collection can directly use a convenient encapsulation method. The data generation end can be a business system, and the result data of all the processing flows for transactions corresponding to the same request can be various business data, logs, and parameters of the business system. The specific data collected depends on the business itself. For example, when a user places an order at the checkout counter, the business system collects information such as the inventory of goods and the user's remaining balance.

[0050] S3, integrating the result data of the transaction processing flow corresponding to the same transaction request, compressing the integrated transaction data, and writing the compressed transaction data into a message queue and sending it to the data processing end.

[0051] In actual applications, integration can be aggregation and deduplication. Message queues are important components in distributed systems, mainly solving problems such as application coupling, asynchronous messages, and traffic peak shaving. Common message queues include ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, etc. You can choose one of them according to your business situation. If the data accuracy requirement is not 100%, it is recommended to use Kafka. Kafka's batch processing mechanism for data can provide greater throughput. Businesses that require data not to be lost can use Rocket message queues, which can ensure that messages are not lost, and the throughput can meet most business requirements. For startup companies with weaker technology, or systems that require rapid online use and small business volume, it is recommended to use Rabbit message queues, which are ready to use out of the box.

[0052] In practical applications, some business data is large in volume and needs to be compressed. Among them, the most commonly used compression algorithm is Gzip (the abbreviation of several file compression programs).

[0053] In practical applications, the writing to the message queue can be in a thread-synchronized or asynchronous manner. The asynchronous manner can ensure that the mainstream business of the system is not affected at all. In the synchronous manner, if the data sending to Kafka fails, the entire transaction will also fail, affecting the mainstream business. Which specific manner to adopt depends on each business system. For example, if only key logs are analyzed, then the asynchronous manner is the best; if the content of the sent message cannot be lost (such as after a user places an order, the commodity needs to be removed from the shopping cart, and in this case the message must not be lost, and the synchronous manner can be used for sending), then the synchronous manner is used for sending.

[0054] In this embodiment, referring to Figure 2 , in step S2, the collection of the result data of the processing flow for the transaction corresponding to the same transaction request includes:

[0055] S21, grouping and collecting the result data of all the processing flows for the transaction corresponding to the same request according to the relevance between the processing flows, so as to integrate all the grouped and collected result data corresponding to the same request.

[0056] In practical applications, when the user request is for the user to apply for a coupon, all the processing flows for the transaction corresponding to the same request include: 1. First, query the user's basic information; 2. Filter the risk control rules through the basic information; 3. Query the coupon rules to be issued; 4. Judge whether the user meets the rule conditions for issuing the coupon; 5. Send the coupon to the user. The result data of all the processing flows for the transaction includes: Data 1: User name: Zhang San; User basic information: Zhang San, age 19, from Beijing; Working as a programmer. Data 2: Zhang San, the risk control filtering result is a low-risk user; Data 3: Zhang San, the issued coupon number is 11111; Data 4: The rule for coupon number 1111 is to be issued to users in Beijing, and the usage validity period is 2 months; Data 5: Successfully sent coupon 11111 to user Zhang San. Since there is a logical relevance between process 1 and process 2, the corresponding data 1 and data 2 of process 1 and process 2 are collected as a group; there is a logical relevance between processes 3, 4, and 5, so the corresponding data 3, data 4, and data 5 of processes 3, 4, and 5 are collected as a group.

[0057] In this way, some common information can be found among the data from 1 to 5. Perform secondary data processing on this. The processing result is: Zhang San, 19 years old, from Beijing, working as a programmer. This user is a low-risk user for risk control. Coupon 11111 is issued. The coupon rule is to issue it to users in Beijing, and the usage validity period is 2 months; sending successful.

[0058] See Figure 3 , Another embodiment of the present disclosure provides a data processing method, which is applied to a data processing end. The method includes:

[0059] S31, Receive the message in the message queue, decompress the data for transaction in the message, so as to perform specialization processing on the decompressed data for transaction. Among them, the specialization processing includes keyword extraction, secondary verification of data, and data filling;

[0060] In practical applications, the secondary verification of data can be that the system log shows that the inventory has been exhausted, and the data generation end verifies whether the inventory is really exhausted. In the specialization processing, the processing logics of different systems are different. For example, keyword extraction is performed on the data (of course, keyword extraction can be done at the data production end, but the same message may have multiple data processing ends, and the processing methods of different processing ends are different, so performing keyword extraction at the data processing end can be more flexible). Or verify the data again. For example, if the received message is that the commodity inventory has been exhausted, resulting in the failure of the user's purchase, we can verify here whether the inventory has been exhausted.

[0061] S32, Write the data for transaction after specialization processing into the database;

[0062] In practical applications, write the data for transaction after specialization processing into the DB database, es (Elasticsearch is a search server based on Lucene. It provides a distributed multi-user full-text search engine, which can achieve real-time search, stable, reliable, fast, and easy to install and use) database or hbase database according to a specific format. It is preferably recommended to use the es database, which can conveniently search for data.

[0063] S33, In response to the received data query request, extract the corresponding data for transaction from the database according to the keyword in the query request for the data query requester to view.

[0064] In this embodiment, see Figure 4 , The method further includes:

[0065] S41, Monitor the number of messages in the message queue, and judge whether the number of messages in the message queue exceeds a preset threshold:

[0066] If so, execute step S42;

[0067] If not, execute the step of receiving messages from the message queue;

[0068] S42. Start an asynchronous thread to execute the step of receiving messages from the message queue for the messages of each thread.

[0069] In practical applications, when the number of messages in the message queue is small, as Figure 5 shown, the step of directly receiving messages from the message queue is executed by the application system. However, for some large-scale systems, in the case of millions of tps (Transactions Per Second, the number of messages processed per second), the number of messages in the message queue is very large, which puts great pressure on the services of the data processing end. When the pressure is too high, the messages in the message queue cannot be consumed in time, resulting in serious data backlog. For this situation, a sub-thread as Figure 6 shown can be used to process the messages in the message queue.

[0070] As Figure 7 shown, taking the actual situation of a business where a user requests a coupon for a user as an example, the data processing method of this embodiment is explained.

[0071] When the data generation end processes the business logic, it calls the API method to collect data for the data to be collected. The API method here is a method encapsulated for convenient use and can be directly used. The underlying layer includes the following steps:

[0072] 1) Compress the collected data.

[0073] 2) After the data is compressed, use the message queue for transmission.

[0074] 3) Here, writing to the message queue can be done in a thread-synchronized or asynchronous manner.

[0075] The data processing end executes the following steps:

[0076] 1) Decompress the data. After receiving the data, decompress the data. The decompressed data is the same as the business data of the data generation end.

[0077] 2) Data processing. The processing here depends on the specific business logic. Different systems have different processing logics. For example, keyword extraction can be performed on the data (of course, keyword extraction can be done at the data production end, but the same message may have multiple data processing ends, and different processing ends have different processing methods, so performing keyword extraction at the data processing end can be more flexible). Or verify the data again. For example, if the received message is that the commodity inventory has been exhausted, resulting in the failure of the user's purchase, we can verify here whether the inventory has been exhausted.

[0078] 3) Data storage. After the data processing is completed, the required key data is stored, which can be stored in ES, DB, or HBase. I recommend using ES, which can facilitate data search. Specifically, where to store depends on the business requirements, and we will not elaborate further here.

[0079] The data display end performs the following steps:

[0080] The data display end is the same as described above. Provide an interface to query data from ES, and search for data by entering keywords on the page. The keywords on the page display end are generally relatively simple, facilitating the use of personnel in different positions.

[0081] Using the data processing method of this embodiment, the required data can be collected, analyzed, and displayed in real time. The operation difficulty is relatively low, and the accuracy of the data depends on the user. If high-precision data needs to be collected, the data in the transaction link can be collected multiple times. Since the data collected can also be processed a second time at the data processing end, the practicability and flexibility of the entire architecture are very high.

[0082] The data processing method of this embodiment is implemented based on tools such as message queues and ES to collect, process, and store data in a simple and convenient manner, facilitating data analysis, solving customer complaints, etc.

[0083] The above process can be used for some simple processes. However, for several major mainstream shopping websites currently, for a single user request, multiple sub-processes and steps are involved. If messages are sent for each process and step, multiple messages need to be sent for a single user request, which places a great pressure on the message queue. For this, an architecture solution as shown in Figure 8(a) is provided.

[0084] As shown in Figure 8(a), for multiple messages in a single user operation, the system collects and summarizes multiple data, and performs secondary simple processing. For example, the user's basic information does not need to be retained in multiple copies, and only the content of each sub-process needs to be recorded. After summarizing and processing the data of multiple sub-processes and then compressing and sending, the pressure on the message queue can be greatly reduced.

[0085] For some large-scale systems, in the case of millions of transactions per second (tps), the data in the message queue is extremely large, exerting great pressure on the services of the data processing end. When the pressure is too high, the data in the message queue cannot be consumed in a timely manner, resulting in serious data backlog. For this situation, the architecture solution shown in Figure 8(b) can be adopted.

[0086] In the solution shown in Figure 8(b), after receiving the message from the message queue, an asynchronous thread is directly started for processing, which can quickly process the messages and improve the processing efficiency.

[0087] Embodiments of the present disclosure also provide a data processing system, including:

[0088] A data generation end, which is used to respond to a received transaction request, execute at least one processing process for transactions corresponding to the current transaction request according to the transaction request; collect the result data of the processing process for transactions corresponding to the same transaction request; integrate the result data of the processing process for transactions corresponding to the same transaction request, so as to write the integrated transaction data into the message queue and send it to the data processing end;

[0089] A data processing end, which is used to receive the messages in the message queue and perform specialization processing on the transaction data in the messages, where the specialization processing includes keyword extraction, secondary verification of the data, and data filling; write the specialized transaction data into the database; in response to a received data query request, extract the corresponding transaction data from the database according to the keywords in the query request for the data query requester to view.

[0090] A data display end, which is used to display the queried data, perform front-end display on the data processed by the data generation end, and display the content to be viewed according to various query conditions. The content displayed here can be conveniently viewed by personnel such as operation, product, and R & D.

[0091] The system of this embodiment can be a computer software platform, etc.

[0092] The data processing system of this embodiment extracts, processes, and saves business data in a simple way, supports viewing and data analysis, is very helpful for solving production problems, customer complaint problems, data analysis, etc., and has the advantages of low operation difficulty, high data accuracy, low threshold, strong practicability, and high flexibility.

[0093] Using the data processing system of this embodiment, different services can flexibly configure access points according to needs, with little impact on system performance, and collect, process, and store data without affecting the mainstream services.

[0094] For the implementation processes of the functions and roles of each part in the above system, please refer to the implementation processes of the corresponding steps in the above method for details, which will not be elaborated here.

[0095] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0096] Based on the same inventive concept, referring to Figure 9 As shown, the electronic device provided by the third exemplary embodiment of the present disclosure includes a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140. Among them, the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0097] The memory 1130 is used to store computer programs;

[0098] When the processor 1110 is used to execute the program stored in the memory 1130, it realizes the following data processing method:

[0099] In response to the received transaction request, execute at least one processing flow for the transaction corresponding to the current transaction request according to the transaction request;

[0100] Collect the result data of the processing flow for the transaction corresponding to the same transaction request;

[0101] Integrate the result data of the processing flow for the transaction corresponding to the same transaction request, so as to write the integrated transaction data into the message queue and send it to the data processing end.

[0102] The above communication bus 1140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0103] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.

[0104] The memory 1130 may include a random access memory (RAM), or may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory 1130 may also be at least one storage device located far from the aforementioned processor 1110.

[0105] The aforementioned processor 1110 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0106] Based on the same inventive concept, the fourth exemplary embodiment of the present disclosure also provides a computer-readable storage medium. A computer program is stored on the above-mentioned computer-readable storage medium, and when the computer program is executed by a processor, the data processing method described above is implemented.

[0107] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist separately without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the data processing method according to the embodiments of the present disclosure is implemented.

[0108] According to the embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or combined with an instruction execution system, device, or device.

[0109] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0110] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for data processing, characterized in that, Applied to the data generation end, the method includes: In response to the received transaction request, execute at least one processing flow for the transaction corresponding to the current transaction request according to the transaction request; Collect the result data of the processing flow for the transaction corresponding to the same transaction request; Integrate the result data of the processing flow for the transaction corresponding to the same transaction request, so as to write the transaction data obtained after integration into the message queue and send it to the data processing end, The collecting the result data of the processing flow for the transaction corresponding to the same transaction request includes: Group and collect the result data of all the processing flows for the transaction corresponding to the same transaction request according to the relevance between the processing flows, so as to integrate all the grouped and collected result data corresponding to the same request, The data generation end is a business system, and the result data of all the processing flows for the transaction corresponding to the same request are various business data, logs, and parameters of the business system, The integration is summarization and deduplication, For at least two processes with logical relevance, collect the data corresponding to the at least two processes as a group; wherein, the logical relevance represents the dependency relationship between the processing flows.

2. The method according to claim 1, wherein Before writing the transaction data obtained after integration into the message queue and sending it to the data processing end, the method further includes: Compress the transaction data obtained after integration, so as to write the compressed transaction data into the message queue and send it to the data processing end.

3. The method according to claim 1, wherein The collecting the result data of the processing flow for the transaction corresponding to the same transaction request is implemented by calling an interface.

4. A method for data processing, characterized in that Applied to the data processing end, the method includes: Receive the message in the message queue and perform specialization processing on the transaction data in the message, wherein the specialization processing includes keyword extraction, secondary verification of the data, and data filling; Write the specialized transaction data into the database; In response to the received data query request, extract the corresponding transaction data from the database according to the keyword in the query request for the data query requester to view, The message in the message queue is obtained at the data generation end through the following steps: Collect the result data of the processing flow for the transaction corresponding to the same transaction request; Integrate the result data of the processing flow for the transaction corresponding to the same transaction request, The collecting the result data of the processing flow for the transaction corresponding to the same transaction request includes: Group and collect the result data of all the processing flows for the transaction corresponding to the same transaction request according to the relevance between the processing flows, so as to integrate all the grouped and collected result data corresponding to the same request, The data generation end is a business system, and the result data of all the processing flows for the transaction corresponding to the same request are various business data, logs, and parameters of the business system, The integration is summarization and deduplication, For at least two processes with logical relevance, collect the data corresponding to the at least two processes as a group; wherein, the logical relevance represents the dependency relationship between the processing flows.

5. The method according to claim 4, wherein The method further includes: Monitor the number of messages in the message queue and determine whether the number of messages in the message queue exceeds a preset threshold: When the number of messages in the message queue exceeds the preset threshold, start an asynchronous thread to process the messages in the message queue.

6. The method according to claim 4, characterized in that, The data for transactions in the messages is compressed data. Before performing specialization processing on the data for transactions in the messages, the method further includes: Decompress the data for transactions in the messages to perform specialization processing on the decompressed data for transactions.

7. A data processing system, characterized in that, It includes: A data generation end, which is used to respond to a received transaction request and execute at least one processing process for transactions corresponding to the current transaction request according to the transaction request; Collect the result data of the processing processes for transactions corresponding to the same transaction request; Integrate the result data of the processing processes for transactions corresponding to the same transaction request, so as to write the integrated data for transactions into the message queue and send it to the data processing end; A data processing end, which is used to receive the messages in the message queue, The collection of the result data of the processing processes for transactions corresponding to the same transaction request includes: Group and collect the result data of all processing processes for transactions corresponding to the same transaction request according to the relevance between the processing processes, so as to integrate the result data of all grouped collections corresponding to the same request, The data generation end is a business system, and the result data of all processing processes for transactions corresponding to the same request are various business data, logs, and parameters of the business system, The integration is summarization and deduplication, For at least two processes with logical relevance, collect the data corresponding to the at least two processes as a group; wherein, the logical relevance represents the dependency relationship between the processing processes.

8. The system according to claim 7, wherein The data processing end is further used for: Perform specialization processing on the data for transactions in the messages, wherein the specialization processing includes keyword extraction, secondary verification of data, and data filling; write the specialized processed data for transactions into the database; in response to a received data query request, extract the corresponding data for transactions from the database according to the keywords in the query request for the data query requester to view.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store computer programs; When the processor is used to execute the programs stored on the memory, it realizes the data processing method described in any one of claims 1-3 or 4-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it realizes the data processing method described in any one of claims 1-3 or 4-6.

Images