Method and system for supporting batch editing processing capability

Abstract

The invention provides a method and a system for supporting batch editing processing capability, which are used for processing platform activity data and micro-service activity data in batches, and the method comprises the following steps: after a back-end service receives a batch modification request, generating a unique identifier of the request; writing the data partitions into a plurality of message queues, and returning a unique identifier of the request; the plurality of consumption scripts consume data in the information queue in parallel; when consuming data, the consumption script processes the data one by one according to each activity; the processing process comprises platform activity data processing and micro-service activity data processing; after the consumption script processes one activity every time, the completed activity id and the processing state are updated into the state table; and when each message is processed, the back-end service returns a processing result. The method has the advantages that the throughput of batch operation in a high-concurrency scene is improved; the final consistency of the data is ensured when the cross-micro service data is updated; and a real-time progress feedback mechanism of a user interaction level is provided.

Description

Technical Field

[0001] The present application belongs to the field of batch data processing technology, and specifically relates to a method and system for supporting batch editing processing capabilities. Background Art

[0002] Microservice architecture has become the mainstream architecture model of Internet platforms. Some online education platforms also use microservice architecture model in their system implementation. Microservice architecture mainly includes platform layer and microservice layer. Figure 1 As shown in the figure, in online education, the smallest service unit is an activity, which can include homework, classes, quizzes, discussions, and recordings. Each class can have many different activities. In terms of architectural design, each class has its own LMS (Learning Management System) activity list. The activity platform connects to different activity microservices to organize and manage activity service capabilities.

[0003] In actual application, in order to facilitate teachers to manage class activity data, it is often necessary to batch edit 500+ activity data (for example, batch edit activity start / end time, teaching teachers, etc.). These data to be edited are stored in the activity platform and the microservices of each activity. Therefore, the update will involve the consistency update of the LMS activity platform data and multiple microservices data.

[0004] like Figure 2 As shown, for batch data updates, the general request processing process includes:

[0005] 1. The front end submits batch processing requests;

[0006] 2. After receiving the data, the backend service will process it according to each activity;

[0007] 3. For each activity, update the LMS platform activity data, and then call each microservice API based on the activity type to update the microservice data;

[0008] 4. Return the results after all activities are updated.

[0009] The above technical solution for batch data update has the following defects:

[0010] 1. Obvious performance bottleneck: Processing 1,000 active data items at a time takes approximately 38 seconds, and DB transaction locks are heavily occupied.

[0011] 2. Poor system stability: Failures in LMS platform activity updates and any microservice call processing will result in overall transaction rollbacks and high retry costs;

[0012] 3. Poor user experience: Front-end requests are blocked for a long time, and the processing progress cannot be perceived in real time. Summary of the Invention

[0013] The purpose of this application is to overcome the defects of poor system stability and poor user experience during batch data processing.

[0014] To achieve the above objectives, this application proposes a method that supports batch editing processing capabilities for batch processing of platform activity data and microservice activity data, the method comprising:

[0015] Step S1: After receiving the batch modification request, the backend service generates a unique identifier TokenId for the request;

[0016] Step S2: The backend service writes the data partitions into multiple message queues and returns the requested TokenId;

[0017] Step S3: The consumption script monitors the message queue; multiple consumption scripts consume data in the message queue in parallel; when consuming data, the consumption script processes each activity one by one; the processing process includes processing platform activity data and microservice activity data; after processing each activity, the consumption script updates the completed activity ID and processing status to the status table; when each message processing is completed, the backend service returns the processing result.

[0018] As an improvement to the above method, the backend service writes data partitions into multiple message queues, including: the backend service divides the data into multiple shards according to a set number of pieces, and writes them into one or more message queues.

[0019] As an improvement to the above method, the multiple consumption scripts consume the data in the information queue in parallel, specifically, multiple consumer groups process the data in the information queue in shards.

[0020] As an improvement to the above method, when consuming data, the consumption script processes each activity one by one, including: first updating the platform activity data, and then calling the microservice API through the gRPC protocol to update the microservice activity data.

[0021] As an improvement to the above method, the consumption script ensures the consistency of platform activity data and microservice activity data through transactions when consuming data.

[0022] As an improvement to the above method, the state table is Redis and is recorded in Redis bitmap data format.

[0023] As an improvement to the above method, the backend service returns the processing result by pushing the result via WebSocket.

[0024] The present application also provides a system that supports batch editing processing capabilities, which is implemented based on the above method, and includes:

[0025] The backend service is used to generate a unique identifier TokenId for the request after receiving a batch modification request; write the data partition to one or more message queues and return the requested TokenId; and return the processing result when each message is processed.

[0026] Message queue, used to store batch modification request messages;

[0027] A state table, used to store the state of activity processing;

[0028] The consumption script is used to monitor the message queue and consume the data in the message queue in parallel. When consuming data, it processes each activity one by one. After each activity is processed, the completed activity ID and processing status are updated in the status table.

[0029] Compared with the prior art, the advantages of this application are:

[0030] 1. The technical solution of this application improves the throughput of batch operations in high-concurrency scenarios. By adopting an asynchronous + parallel consumption approach, the originally batch-processed data is split into controllable small data units, which are processed and consumed in groups, thereby improving the system processing throughput while also ensuring the timeliness of processing.

[0031] 2. The eventual consistency of data is guaranteed when data is updated across microservices. Before Kafka data sharding, transaction messages are written and the service retry mechanism is used to ensure the eventual consistency of transaction processing data.

[0032] 3. A real-time progress feedback mechanism is provided at the user interaction level. After parallel consumption processing is completed, Redis bitmap is used to store the processing status (success or failure) of each activity. The result data will be pushed to the application app in real time via WebSocket, allowing users to obtain and perceive the processing progress in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The following is a diagram of the microservice architecture;

[0034] Figure 2 The figure shows the processing process of the existing system when batch data is updated;

[0035] Figure 3 Shown is a timing diagram of a method for supporting batch editing processing capabilities;

[0036] Figure 4 Shown is the core flow chart of parallel consumption processing. DETAILED DESCRIPTION

[0037] The technical solution of this application is described in detail below with reference to the accompanying drawings.

[0038] In order to improve the concurrency and reliability of batch operation processing and ensure the eventual consistency of data, the method and system for supporting batch editing processing capabilities provided in this application adopt asynchronous + parallel consumption to avoid asynchronous execution timeout. At the same time, Redis bitmap is used to store the active processing status, and the result data is pushed to the application APP in real time via WebSocket, so that users can obtain and perceive the processing progress in real time.

[0039] Example 1

[0040] like Figure 3 As shown, the methods that support batch editing processing capabilities include:

[0041] Step 1: The application receives the batch management operation initiated by the user and initiates a batch modification request to the backend service. The request data content includes: user uid, activity ids (array List format), and modification content fields (such as activity teacher, activity start time, activity end time, etc.).

[0042] Step 2: After receiving the batch modification request, the backend service generates a unique TokenId based on the input parameters as the unique identifier for this request operation.

[0043] Step 3: The backend service preprocesses the data and writes the data partitions to one or more message queues after basic verification. At the same time, the request returns the TokenId, which makes it easy for the application to initiate a completion status monitoring query based on this representation.

[0044] Step 4: Multiple consumer scripts monitor and read the message queue data content respectively, and consume in parallel (multiple ConsumerGroups are processed by shards).

[0045] Step 5: When processing consumption, the consumption script processes each activity one by one. Each activity needs to process platform activity data and microservice data.

[0046] Step 6: Status synchronization: After processing each activity, the consumption script updates the completed activity ID and processing status to the status table. The status table marks which activities have been successfully processed and which activities have failed.

[0047] Step 7: Result Notification: When a message is processed, the backend service pushes the final processing result (using TokenId as the request identifier) ​​to the app in real time until the request is processed. The push method can be WebSocket or other methods.

[0048] When using the above process to batch process data, we found that when the batch operation activity volume exceeds 1k, it is easy to cause the processing script consumption timeout. Therefore, we optimized the above process. The specific optimization contents include:

[0049] 1. After the backend service (RestAPI) receives the batch processing request, it verifies the basic data and generates a unique identifier TokenId for the request based on the input parameters. It then fragments the data and writes it to the asynchronous message queue (the specific rule is 128 items per fragment).

[0050] 2. Parallel consumption by consumer scripts: Multiple consumer groups are processed on a sharded basis. The process is as follows: first, platform activity data is updated, then microservice APIs are called via the gRPC protocol to update business activity data to ensure consistency with platform activity. This process uses transactions to ensure consistency between platform activity data and microservice activity data. The client of gRPC streaming management is the consumer script, and the server is the individual activity microservices.

[0051] 3. Processing Status Maintenance: The status table can be maintained in Redis, using the Redis bitmap data format. After the consuming script processes each activity change, it updates the Redis bitmap to maintain the activity completion status. Activity completion status can be: 1 for success; 0 for failure (for unprocessed activity IDs, no such status record exists).

[0052] Parallel consumption processing core process such as Figure 4 shown.

[0053] For the implementation of parallel consumption queues, this embodiment can build a high-throughput processing architecture through Kafka's partitioning mechanism and consumer group mode. In other embodiments, distributed message processing MQs such as Apache RocketMQ, RabbitMQ (cluster mode), and Apache Pulsar can also be used to achieve the same effect.

[0054] This embodiment uses Redis, a high-performance storage medium, and its bitmap data format to implement the activity processing status table (storing data elements in an SDS data structure in String format). In other embodiments, Memcached, Cassandra, TiDB, and other KV storage databases can also be used to achieve the same effect.

[0055] Example 2

[0056] The present application also provides a system that supports batch editing processing capabilities, which is implemented based on the above method, and includes:

[0057] The backend service is used to generate a unique identifier TokenId for the request after receiving a batch modification request; write the data partition to one or more message queues and return the requested TokenId; and return the processing result when each message is processed.

[0058] Message queue, used to store batch modification request messages;

[0059] A state table, used to store the state of activity processing;

[0060] The consumption script is used to monitor the message queue and consume the data in the message queue in parallel. When consuming data, it processes each activity one by one. After each activity is processed, the completed activity ID and processing status are updated in the status table.

[0061] The present application may also provide a computer device comprising: at least one processor, memory, at least one network interface, and a user interface. The various components in the device are coupled together via a bus system. It will be understood that the bus system is used to enable communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0062] The user interface may include a display, a keyboard, or a pointing device, such as a mouse, a trackball, a touchpad, or a touch screen.

[0063] It is understood that the memory in the embodiments disclosed in the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0064] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset or an extension thereof: an operating system and applications.

[0065] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and handle hardware-based tasks. Application programs include various application programs, such as media players and browsers, which are used to implement various application services. The program that implements the method of the embodiment of the present disclosure can be included in the application program.

[0066] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in the application program, to:

[0067] Perform the steps of the above method.

[0068] The above method can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The above-disclosed methods, steps, and logic block diagrams can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the above-disclosed method can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0069] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, or other electronic units or combinations thereof for performing the functions described herein.

[0070] For software implementation, the technology of the present application can be implemented by executing the functional modules (e.g., procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0071] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.

[0072] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of this application and are not intended to limit the scope of the present invention. Although this application has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application and should be encompassed by the claims of this application.

Claims

1. A method for supporting batch editing processing capabilities for batch processing of platform activity data and microservice activity data, the method comprising: Step S1: After receiving the batch modification request, the backend service generates a unique identifier TokenId for the request; Step S2: The backend service writes the data partitions into multiple message queues and returns the requested TokenId; Step S3: The consumption script monitors the message queue; Multiple consumer scripts consume data in the information queue in parallel; When consuming data, the consumption script processes each activity one by one. The processing process includes the processing of platform activity data and microservice activity data. After processing each activity, the consumption script updates the completed activity ID and processing status to the status table; when each message processing is completed, the backend service returns the processing result.

2. The method for supporting batch editing processing capabilities according to claim 1, characterized in that: The backend service writes the data partitions into multiple message queues, including: the backend service divides the data into multiple fragments according to a set number of pieces, and writes them into one or more message queues.

3. The method for supporting batch editing processing capabilities according to claim 1, characterized in that: The multiple consumption scripts consume the data in the information queue in parallel, specifically, multiple consumer groups process the data in the information queue in slices.

4. The method for supporting batch editing processing capabilities according to claim 1, wherein: When consuming data, the consumption script processes each activity one by one, including: first updating the platform activity data, and then calling the microservice API through the gRPC protocol to update the microservice activity data.

5. The method for supporting batch editing processing capability according to claim 4, characterized in that: The consumption script ensures the consistency of platform activity data and microservice activity data through transactions when consuming data.

6. The method for supporting batch editing processing capabilities according to claim 1, characterized in that: The state table is Redis, and is recorded in Redisbitmap data format.

7. The method for supporting batch editing processing capability according to claim 1, characterized in that: The backend service returns the processing results by using WebSocket to push the results.

8. A system supporting batch editing processing capabilities, implemented based on the method of any one of claims 1 to 7, characterized in that: The system comprises: The backend service is used to generate a unique identifier TokenId for the request after receiving a batch modification request; write the data partition to one or more message queues and return the requested TokenId; and return the processing result when each message is processed. Message queue, used to store batch modification request messages; A state table, used to store the state of active processing; and The consumption script is used to monitor the message queue and consume the data in the message queue in parallel. When consuming data, it processes each activity one by one. After each activity is processed, the completed activity ID and processing status are updated in the status table.

Images