Event stream processing method and apparatus, computer device, and storage medium
By creating and controlling the number of concurrent coroutine scopes in event stream processing, the problem of high resource consumption and low efficiency caused by uncontrollable concurrency in RxJava is solved, and efficient and flexible event stream processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN INT FINANCIAL LEASING CO LTD
- Filing Date
- 2022-09-20
- Publication Date
- 2026-06-26
AI Technical Summary
In existing technologies, the number of concurrent threads in RxJava cannot be precisely controlled, resulting in excessive system resource consumption and reduced efficiency of stream event processing.
By creating a scope that can be executed by the coroutine code, storing the scope in an atomic operation class, the on/off state and concurrency of coroutines can be precisely controlled, and event stream processing can be executed within the scope corresponding to the coroutine code to generate asynchronous coroutine consumer ports for synchronous processing.
It solves the problem of excessive system resource consumption in traditional concurrent processing methods, while improving the efficiency and flexibility of event stream processing, simplifying code complexity, and making event stream processing more efficient.
Smart Images
Figure CN115390920B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to event stream processing methods, apparatus, computer devices, and storage media. Background Technology
[0002] Event stream processing is one of the most common scenarios in code development. It is suitable for end-to-end streaming processing and can extract, transform and analyze events generated at the source end in real time and load them to the target end.
[0003] In a typical event stream processing scenario, the process involves three steps: event enqueueing, event processing, and event dequeueing with the result. However, if the event processing time is unpredictable, such as file operations or network operations, it can slow down the consumption speed of the entire event queue, resulting in unnecessary performance waste.
[0004] When dealing with time-consuming events in the event queue, related technologies can use RxJava's event stream processing method to process all events in the event queue concurrently at once, or process them one by one. However, the number of concurrent threads in RxJava cannot be precisely controlled, resulting in excessive system resource consumption and reduced efficiency of stream event processing. Summary of the Invention
[0005] In view of this, the present invention provides an event stream processing method, apparatus, computer device and storage medium, the main purpose of which is to solve the problem that the number of concurrent threads in RxJava cannot be precisely controlled in the prior art, resulting in excessive system resource consumption and reduced efficiency of stream event processing.
[0006] According to one aspect of the present invention, an event stream processing method is provided, comprising:
[0007] Create a scope in which the running coroutine code can be executed, and store the scope in an atomic operation class so that the atomic operation class holds a shared scope;
[0008] When a processing instruction for an event stream in the message queue is initiated, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction.
[0009] Within the shared scope corresponding to the current coroutine code, perform coroutine consumption processing on events in the message queue, and determine whether the number of events processed by the coroutine exceeds the preset concurrency value when performing coroutine consumption processing;
[0010] If the limit is not exceeded, then an asynchronous coroutine consumer port is generated for the events processed by the coroutine through the shared scope, and the asynchronous coroutine consumer port is used to perform synchronous processing of the events.
[0011] Furthermore, the scope in which the creation and execution of the coroutine code can be implemented is stored in an atomic operation class so that the atomic operation class holds a shared scope, including:
[0012] Declare and initialize the coroutine scope object for the event stream processing scenario;
[0013] Define the global or custom scope that the coroutine code can execute in the coroutine scope object, and set the lifecycle maintained by the coroutine code in the event flow processing scenario so that the execution cycle of the coroutine code is consistent with the lifecycle maintained by the event flow processing scenario.
[0014] The global scope or custom scope is stored in an atomic operation class so that the atomic operation class holds a shared scope.
[0015] Furthermore, the method also includes:
[0016] Create a message queue for processing event streams, receive events to be processed, and encapsulate the data required for the events to be processed in the order of event processing from beginning to end and cache it at the end of the message queue.
[0017] Furthermore, the step of extracting the shared scope corresponding to the current coroutine code from the atomic operation class according to the processing instruction when initiating the event stream processing instruction in the message queue includes:
[0018] When the processing instruction for the event stream in the message queue is initiated, a basic loop pattern is generated within the shared scope, and the basic loop pattern is used to obtain the shared scope corresponding to the current coroutine code from the atomic operation class.
[0019] If the atomic operation class does not have a shared scope corresponding to the current coroutine code, a new scope variable is generated, and the new scope variable replaces the shared scope held in the atomic operation class through atomication operation to obtain the replacement result;
[0020] Determine whether to end the basic loop pattern based on the replacement result.
[0021] Furthermore, if the replacement result is successful, then determining whether to end the basic loop mode based on the replacement result includes:
[0022] Based on the replacement result, output the shared scope held in the atomic operation class after replacement, and end the basic loop pattern;
[0023] If the replacement result is a replacement failure, then determine whether to end the basic loop mode based on the replacement result, including:
[0024] The new scope variable is destroyed based on the replacement result, and the basic loop pattern is re-executed to obtain the shared scope corresponding to the current coroutine code from the atomic operation class.
[0025] Further, the step of performing coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and determining whether the number of events processed by the coroutine exceeds a preset concurrency value during the coroutine consumption processing, includes:
[0026] Within the shared scope corresponding to the current coroutine code, perform coroutine consumption processing on events in the message queue, and use a coroutine counter to record the current number of concurrent tasks during the coroutine consumption processing.
[0027] When the asynchronous coroutine consumption port detects that any event in the concurrent coroutine consumption process has been completed, it updates the current number of concurrent tasks recorded in the coroutine counter.
[0028] Based on the current number of concurrent tasks, determine whether the number of events processed by the coroutine exceeds the preset concurrency value;
[0029] After performing coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and determining whether the number of events processed by the coroutine exceeds a preset concurrency value during the coroutine consumption processing, the method includes:
[0030] If the number of events exceeds the limit, the processing instructions for the event stream in the message queue will be closed until the number of coroutines executing in the event stream in the message queue does not exceed the preset concurrency value.
[0031] Furthermore, the method also includes:
[0032] When the event stream in the message queue has finished processing, it is determined whether the shared scope held in the atomic operation class has reached the execution cycle of the coroutine code. If so, the shared scope held in the atomic operation class is destroyed.
[0033] According to another aspect of the present invention, an event stream processing apparatus is provided, comprising:
[0034] A module is created to create a scope in which the code of a running coroutine can be executed. This scope is stored in an atomic operation class so that the atomic operation class holds a shared scope.
[0035] The extraction module is used to extract the shared scope corresponding to the current coroutine code from the atomic operation class according to the processing instruction of the event stream in the start message queue.
[0036] The judgment module is used to perform coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and to determine whether the number of events processed by the coroutine consumption exceeds a preset concurrency value when performing coroutine consumption processing.
[0037] The processing module is used to generate asynchronous coroutine consumption ports for events that are processed by coroutines through the shared scope if no limit is reached, and to perform synchronous processing of events using the asynchronous coroutine consumption ports.
[0038] Furthermore, the creation module is specifically used to declare and initialize a coroutine scope object for the event flow processing scenario; define the global scope or custom scope that can be executed by running coroutine code in the coroutine scope object, and set the lifecycle maintained by the coroutine code in the event flow processing scenario so that the execution cycle of the coroutine code is consistent with the lifecycle maintained by the event flow processing scenario; and store the global scope or custom scope in an atomic operation class so that the atomic operation class holds a shared scope.
[0039] Furthermore, the device also includes:
[0040] The caching module is used to create a message queue for processing event streams, receive events to be processed, and encapsulate the data required for the events to be processed in the order of event processing from beginning to end and cache it at the end of the message queue.
[0041] Furthermore, the extraction module is specifically used to generate a basic loop pattern within the shared scope when the processing instruction of the event stream in the message queue is started; the basic loop pattern is used to obtain the shared scope corresponding to the current coroutine code from the atomic operation class; if the shared scope corresponding to the current coroutine code does not exist in the atomic operation class, a new scope variable is generated, and the new scope variable is used to replace the shared scope held in the atomic operation class through atomic operation to obtain a replacement result; and the basic loop pattern is terminated based on the replacement result.
[0042] Furthermore, the extraction module is specifically used to output the shared scope held in the replaced atomic operation class according to the replacement result, and end the basic loop mode; if the replacement result is a replacement failure, then determine whether to end the basic loop mode according to the replacement result, including: destroying the new scope variable according to the replacement result, and re-executing the basic loop mode to obtain the shared scope corresponding to the current coroutine code from the atomic operation class.
[0043] Furthermore, the judgment module is specifically used to perform coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and to use a coroutine counter to record the current number of concurrent tasks during the coroutine consumption processing; when the asynchronous coroutine consumption port detects that any event in the events being processed by coroutines simultaneously has been consumed, it updates the current number of concurrent tasks recorded in the coroutine counter; and determines whether the number of events processed by coroutines exceeds a preset concurrency value based on the current number of concurrent tasks.
[0044] The device further includes:
[0045] The shutdown module is used to perform coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and after determining whether the number of events processed by the coroutine exceeds a preset concurrency value during the coroutine consumption processing, if it exceeds the preset concurrency value, then shut down the processing instructions of the event stream in the message queue until the number of coroutines executed in the event stream in the message queue does not exceed the preset concurrency value.
[0046] Furthermore, the device also includes:
[0047] The destruction module is used to determine whether the shared scope held in the atomic operation class has reached the execution cycle of the coroutine code when the event stream in the message queue has been processed. If so, the shared scope held in the atomic operation class is destroyed.
[0048] According to another aspect of the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of an event stream processing method.
[0049] According to another aspect of the present invention, a computer storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of an event stream processing method.
[0050] By employing the above technical solution, the present invention provides an event stream processing method, apparatus, computer device, and storage medium. By creating a scope executable by running coroutine code and storing the scope in an atomic operation class, the atomic operation class holds a shared scope. When a processing instruction for an event stream in a message queue is initiated, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction. Coroutine consumption processing is performed on the events in the message queue within the shared scope corresponding to the current coroutine code. During coroutine consumption processing, it is determined whether the number of events processed by the coroutine exceeds a preset concurrency value. If it does not exceed the preset concurrency value, an asynchronous coroutine consumption port is generated for the events processed by the coroutine through the shared scope. The asynchronous coroutine consumption port is used for synchronous processing of the executed events. Compared with the existing RxJava event stream processing method, this application solves the problems of excessive system resource consumption in traditional concurrent processing methods and low efficiency of single-threaded event stream processing by precisely controlling the switching on and off of coroutines and the number of concurrent processes. This makes event stream processing more flexible and efficient. At the same time, due to the characteristics of coroutine code, the concurrent event stream is not thrown asynchronously, but in a synchronous manner, which simplifies the code complexity and makes the event stream processing process more flexible and efficient. Attached Figure Description
[0051] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0052] Figure 1 This is a schematic diagram of an application environment for an event stream processing method according to an embodiment of the present invention;
[0053] Figure 2 This is a flowchart illustrating an event stream processing method according to an embodiment of the present invention;
[0054] Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S10;
[0055] Figure 4 yes Figure 2 A schematic diagram of a specific implementation method for step S20;
[0056] Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S30;
[0057] Figure 6 This is a schematic diagram of an event stream processing device according to an embodiment of the present invention;
[0058] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;
[0059] Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0060] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0061] The event stream processing method provided in this embodiment of the invention can be applied to, for example, Figure 1 In this application environment, the client communicates with the server via a network. The server can create a scope that the coroutine code can execute, and store this scope in an atomic operation class so that the atomic operation class holds a shared scope. When a processing instruction for an event stream in the message queue is initiated, the server extracts the shared scope corresponding to the current coroutine code from the atomic operation class according to the processing instruction. Within the shared scope corresponding to the current coroutine code, coroutine consumption processing is performed on the events in the message queue. During the coroutine consumption processing, it is determined whether the number of events processed by the coroutine exceeds a preset concurrency value. If it does not exceed the limit, an asynchronous coroutine consumption port is generated for the events processed by the coroutine through the shared scope, and the asynchronous coroutine consumption port is used for the synchronous processing of the events. In this invention, by precisely controlling the switching on and off of coroutines and the number of concurrent processes, the problems of excessive system resource consumption in traditional concurrent processing methods and low efficiency in single-threaded event stream processing are solved, making event stream processing more flexible and efficient. At the same time, due to the characteristics of coroutine code, the concurrently processed event stream is not thrown asynchronously, but in a synchronous-like manner, simplifying code complexity and making the event stream processing process more flexible and efficient. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.
[0062] Please see Figure 2 As shown, Figure 2 A flowchart illustrating an event stream processing method provided in an embodiment of the present invention includes the following steps:
[0063] S10. Create a scope in which the running coroutine code can be executed, and store the scope in an atomic operation class so that the atomic operation class holds a shared scope.
[0064] Coroutines, often referred to as microthreads, are a user-mode context switching technique that allows code blocks to switch execution between each other. Typically, coroutines used in programming languages need to be used in conjunction with their scope; otherwise, the coroutine's lifecycle cannot be accurately controlled, leading to memory leaks or other problems. The scope that coroutine code can execute is an interface used to track internally started coroutines and provide organized and unified management of them (structured concurrency). Therefore, coroutines cannot be started outside this scope. The specific scope that coroutine code can execute involves the coroutine's execution range, lifecycle, and automatic lifecycle management. It can be created in two ways: using the `coroutineScope` function or the `supervisorScope` function. In scopes created using these methods, the context automatically inherits the parent coroutine's context.
[0065] In practical applications, multiple coroutines may simultaneously modify variables declared within a scope while performing event handling operations. For example, coroutine A checks if a scope exists; if it does, it can directly use the scope to perform the handling operation. However, if the scope is closed by coroutine B while it is being used, the variables declared within that scope will also change, leading to a conflict in the scope's operations. To prevent multiple coroutines from simultaneously modifying variables declared within a scope, the scope can be stored in an atomic operation class. This atomic operation class can share the scope with coroutines and ensure that the scope is not modified by other coroutines while a coroutine is using it.
[0066] It's important to understand that, considering the lifecycle of coroutine code in real-world application scenarios, we can also define the lifecycle of the scope maintained within those scenarios. Specifically, for example... Figure 3 In step S10, namely creating a scope that the running coroutine code can execute, and storing the scope in an atomic operation class so that the atomic operation class holds a shared scope, the following steps are included:
[0067] S11. Declare and initialize the coroutine scope object for the event stream processing scenario.
[0068] S12. Define the global or custom scope that the coroutine code can execute in the coroutine scope object, and set the lifecycle maintained by the coroutine code in the event flow processing scenario so that the execution cycle of the coroutine code is consistent with the lifecycle maintained by the event flow processing scenario.
[0069] S13. Store the global scope or custom scope in an atomic operation class so that the atomic operation class holds a shared scope.
[0070] Generally, the scope in which coroutine code can be executed can be divided into two main categories: global scope and user-defined scope. In the global scope, all created variables are stored as properties of the `window` object, and all created functions are stored as methods of the `window` object. User-defined scope operates within a function; the name of the code is only valid within the function. Calling a function creates a new function scope, which is destroyed after the function finishes execution. Each time a function is called, a new function scope is created.
[0071] Here, you can define the global or custom scope in the coroutine scope object to execute coroutine code, and set the lifecycle maintained by the coroutine code in the event flow processing scenario. This ensures that the execution cycle of the coroutine code is consistent with the lifecycle maintained by the event flow processing scenario. Variables declared in the global scope are global variables, which can be used anywhere. Global variables are only destroyed when the event flow processing ends, which consumes more memory. Variables declared in a custom scope are local variables, which can only be used inside a function. However, they are initialized when the code block they belong to is executed and destroyed when the code block finishes executing, which saves more memory.
[0072] It should be noted that all coroutine code has a scope. Scopes written inside functions are user-defined scopes. If a function contains other functions, another scope can be created within that scope, forming a scope chain. Specifically, when a function operates on a variable, it first searches for it in its own scope. If the variable is found, it is used directly. If not, it searches up the parent scope. If the variable is not found in the global scope, an error is thrown.
[0073] In practical use, the lifecycle can be understood as the period from the creation to the destruction of a variable. The lifecycle of a local scope begins when the variable enters the scope and ends when it leaves the scope. The lifecycle of the global scope is the entire program.
[0074] S20. When the processing instruction for the event stream in the message queue is started, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction.
[0075] In this embodiment of the invention, considering the processing order of events in the event stream, a message queue for processing the event stream can be created before initiating the processing instruction in the message queue. The message queue receives events to be processed and encapsulates the data required for each event in the order of event processing, caching it at the end of the message queue. Here, the message queue acts as a container for storing event messages; when an event needs to be processed, it can be retrieved from the message queue. Message queues are an important component in distributed systems; using message queues can improve system performance and reduce peak loads and coupling through asynchronous processing.
[0076] The event stream processing method provided by this invention can be applied to client-initiated event stream processing in various scenarios. This event stream can be formed by different data operation requests, such as data query, data upload, and data transformation. The specific operation content and information of the events in the event stream depend on the event consumer. Taking an image upload event as an example, the operation content to be passed to the message queue is image upload, and the operation information includes the local file address of the image. Subsequently, the server can obtain the corresponding image according to the image file address area corresponding to the event in the message queue and perform the upload operation. After the current event in the message queue is processed, the processing of the next event is executed.
[0077] It's important to understand that the processing method for event streams in a message queue can be configured through event stream tools. For example, you can set the concurrency of events, the priority order of events, etc. The start of an event stream can be triggered by the built-in processing commands of the event stream tool. Specifically, such as... Figure 4 As shown, in step S20, that is, when the processing instruction for the event stream in the message queue is started, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction, including the following steps:
[0078] S21. When the processing instruction of the event stream in the message queue is started, a basic loop pattern is generated within the shared scope, and the shared scope corresponding to the current coroutine code is obtained from the atomic operation class using the basic loop pattern.
[0079] S22. If the atomic operation class does not have a shared scope corresponding to the current coroutine code, a new scope variable is generated, and the new scope variable is used to replace the shared scope held in the atomic operation class through atomication operation to obtain the replacement result.
[0080] S23. Determine whether to end the basic loop mode based on the replacement result.
[0081] When the instruction to process the event stream in the message queue is initiated, it means that the coroutine code has executed the event processing operation. For the currently processed event, the coroutine code will first extract the latest shared scope from the atomic operation class. If the shared scope does not exist in the atomic operation class or the shared scope has become inactive, for example, it does not exist when the code runs for the first time, or the shared scope is manually closed after the coroutine code finishes running to avoid wasting resources, then a new scope can be generated and replaced in the atomic operation class.
[0082] This approach considers the possibility of conflicting replacement operations from other coroutines. The replacement result includes both successful and failed replacements. If the replacement is successful, the shared scope held by the atomic operation class after the replacement is output, and the basic loop ends. If the replacement fails, the new scope variable is destroyed, and the basic loop is re-executed to retrieve the shared scope corresponding to the current coroutine code from the atomic operation class. Specifically, after performing the replacement operation, the result can be returned. If the replacement is successful, it means no other coroutine is performing a new scope replacement operation, and the atomic operation class holds the newly generated scope, ending the basic loop. If the replacement fails, it means another coroutine is also performing a new scope replacement operation, causing a replacement conflict. In this case, the new scope is destroyed, and the basic loop is re-executed to retrieve the shared scope from the atomic operation class again. If the shared scope exists in the atomic operation class, it indicates the replacement operation performed by the other coroutine is successful, and the processing operation is performed using the shared scope in the current atomic operation class.
[0083] S30. Within the shared scope corresponding to the current coroutine code, perform coroutine consumption processing on the events in the message queue, and determine whether the number of events processed by the coroutine exceeds the preset concurrency value when performing coroutine consumption processing.
[0084] In this embodiment, to improve the processing efficiency of the event stream, a preset concurrency value can be set in advance. This preset concurrency value is equivalent to the maximum number of concurrent events allowed in the message queue, and is equivalent to the threshold number of synchronous events allowed in the message queue. Asynchronous coroutine consumption processing is used to execute the concurrent processing of events in the message queue. Specifically, during each processing of events in the event stream, it can be determined whether the number of events processed by the coroutine consumption has reached the maximum concurrency. If the maximum concurrency has been reached, it means that the number of events currently being processed concurrently has reached the set value, and the current event cannot be processed concurrently, so the coroutine startup is ignored. If the maximum concurrency has not been reached, it means that the number of events currently being processed concurrently has not reached the set value, and asynchronous coroutine consumption processing can be used to execute the concurrent processing of events, thereby ensuring that the number of concurrent events processed in the message queue is maintained within the maximum concurrency.
[0085] It's important to understand that since the start and end of events in a message queue need to be determined, and to accurately record concurrently processed events, a coroutine counter can be used to record the number of concurrent events in the message queue. Each time a coroutine executes the processing of an event, the coroutine counter increments by 1. Specifically, as... Figure 5 As shown, step S30, which involves performing coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and determining whether the number of events processed by the coroutine exceeds a preset concurrency value during the coroutine consumption process, includes the following steps:
[0086] S31. Perform coroutine consumption processing on the events in the message queue within the shared scope corresponding to the current coroutine code, and use a coroutine counter to record the current number of concurrent tasks when performing coroutine consumption processing.
[0087] S32. When the asynchronous coroutine consumption port detects that any event in the simultaneously executing coroutine consumption process has been consumed, the number of current concurrent tasks recorded in the coroutine counter is updated.
[0088] S33. Determine whether the number of events processed by the coroutine exceeds the preset concurrency value based on the current number of concurrent tasks.
[0089] The preset concurrency settings here can be configured according to the actual event flow processing scenario. For scenarios with high event processing difficulty, event processing is relatively time-consuming. If multiple events are started simultaneously and coroutines are executed to consume and process them, the efficiency of event processing will be low. In this case, a smaller preset concurrency value can be set. For example, for time-consuming file operations or network operations, the preset concurrency can be set to 2. For scenarios with low event processing difficulty, a larger preset concurrency value can be set. For example, for fast data operations or access operations, the preset concurrency can be set to 5. The specific preset concurrency value is not limited.
[0090] S40. If the time limit is not exceeded, then an asynchronous coroutine consumer port is generated for the events processed by the coroutine through the shared scope, and the asynchronous coroutine consumer port is used to perform synchronous processing of the events.
[0091] Understandably, if the preset concurrency value is not exceeded, it means that more events can be added to the current message queue for synchronous coroutine consumption and processing. Then, asynchronous coroutine consumption ports can be generated through the shared scope, and multiple events can be processed by coroutines simultaneously using the asynchronous coroutine consumption ports.
[0092] Accordingly, considering that the number of concurrent processing operations needs to be within the preset concurrency value to ensure the efficiency of event processing, if it exceeds the preset concurrency value, the processing instructions of the event stream in the message queue will be closed until the number of coroutines executing the event stream in the message queue does not exceed the preset concurrency value.
[0093] Furthermore, to avoid wasting resources in the message queue, when the event stream in the message queue has finished processing, it can be determined whether the shared scope held by the atomic operation class has reached the end of the coroutine's execution cycle. If so, the shared scope held by the atomic operation class can be destroyed. Here, when the number of event streams in the message queue is 0, the shared scope held by the atomic operation class can be made inactive.
[0094] This embodiment provides an event stream processing method. By creating a scope that can be executed by running coroutine code, the scope is stored in an atomic operation class so that the atomic operation class holds a shared scope. When the processing instruction for the event stream in the message queue is started, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction. Coroutine consumption processing is performed on the events in the message queue within the shared scope corresponding to the current coroutine code. When performing coroutine consumption processing, it is determined whether the number of events processed by the coroutine exceeds a preset concurrency value. If it does not exceed the preset concurrency value, an asynchronous coroutine consumption port is generated for the events processed by the coroutine through the shared scope. The asynchronous coroutine consumption port is used to synchronously process the events. Compared with the existing RxJava event stream processing method, this application solves the problems of excessive system resource consumption in traditional concurrent processing methods and low efficiency of single-threaded event stream processing by precisely controlling the switching on and off of coroutines and the number of concurrent processes. This makes event stream processing more flexible and efficient. At the same time, due to the characteristics of coroutine code, the concurrent event stream is not thrown asynchronously, but in a synchronous manner, which simplifies the code complexity and makes the event stream processing process more flexible and efficient.
[0095] In one embodiment, an event stream processing apparatus is provided, which corresponds one-to-one with the event stream processing methods described in the above embodiments. For example... Figure 6 As shown, the event stream processing device includes a creation module 51, an extraction module 52, a judgment module 53, and a judgment module 54. Detailed descriptions of each functional module are as follows:
[0096] Create module 51 to create a scope that can be executed by running coroutine code, and store the scope in an atomic operation class so that the atomic operation class holds a shared scope;
[0097] Extraction module 52 is used to extract the shared scope corresponding to the current coroutine code from the atomic operation class according to the processing instruction of the event stream in the start message queue.
[0098] The judgment module 53 is used to perform coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and to determine whether the number of events processed by the coroutine consumption exceeds the preset concurrency value when performing coroutine consumption processing.
[0099] The processing module 54 is used to generate an asynchronous coroutine consumption port for the event to be processed by the coroutine through the shared scope if the limit is not exceeded, and to use the asynchronous coroutine consumption port to perform synchronous processing of the event.
[0100] In one embodiment, the creation module 51 is specifically used to declare and initialize a coroutine scope object for an event flow processing scenario; define a global scope or custom scope that can be executed by running coroutine code in the coroutine scope object, and set the lifecycle maintained by the coroutine code in the event flow processing scenario so that the running cycle of the coroutine code is consistent with the lifecycle maintained by the event flow processing scenario; and store the global scope or custom scope in an atomic operation class so that the atomic operation class holds a shared scope.
[0101] In one embodiment, the device further includes:
[0102] The caching module is used to create a message queue for processing event streams, receive events to be processed, and encapsulate the data required for the events to be processed in the order of event processing from beginning to end and cache it at the end of the message queue.
[0103] In one embodiment, the extraction module 52 is specifically configured to, when initiating the processing instruction of the event stream in the message queue, generate a basic loop pattern within the shared scope, use the basic loop pattern to obtain the shared scope corresponding to the current coroutine code from the atomic operation class; if the shared scope corresponding to the current coroutine code does not exist in the atomic operation class, generate a new scope variable, and replace the shared scope held in the atomic operation with the new scope variable through atomic operations to obtain a replacement result; determine whether to end the basic loop pattern based on the replacement result.
[0104] In one embodiment, the extraction module 52 is further configured to output the shared scope held in the replaced atomic operation class according to the replacement result, and end the basic loop mode; if the replacement result is a replacement failure, then determine whether to end the basic loop mode according to the replacement result, including: destroying the new scope variable according to the replacement result, and re-executing the basic loop mode to obtain the shared scope corresponding to the current coroutine code from the atomic operation class.
[0105] In one embodiment, the judgment module 53 is specifically used to perform coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and to use a coroutine counter to record the current number of concurrent tasks during the coroutine consumption processing; when the asynchronous coroutine consumption port detects that any event in the events being processed by coroutines simultaneously has been consumed, it updates the current number of concurrent tasks recorded in the coroutine counter; and determines whether the number of events processed by coroutines exceeds a preset concurrency value based on the current number of concurrent tasks.
[0106] The device further includes:
[0107] The shutdown module is used to perform coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and after determining whether the number of events processed by the coroutine exceeds a preset concurrency value during the coroutine consumption processing, if it exceeds the preset concurrency value, then shut down the processing instructions of the event stream in the message queue until the number of coroutines executed in the event stream in the message queue does not exceed the preset concurrency value.
[0108] In one embodiment, the device further includes:
[0109] The destruction module is used to determine whether the shared scope held in the atomic operation class has reached the execution cycle of the coroutine code when the event stream in the message queue has been processed. If so, the shared scope held in the atomic operation class is destroyed.
[0110] This embodiment provides an event stream processing device. By creating a scope that can be executed by running coroutine code, the scope is stored in an atomic operation class so that the atomic operation class holds a shared scope. When a processing instruction for an event stream in a message queue is started, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction. Coroutine consumption processing is performed on the events in the message queue within the shared scope corresponding to the current coroutine code. When performing coroutine consumption processing, it is determined whether the number of events processed by the coroutine exceeds a preset concurrency value. If it does not exceed the preset concurrency value, an asynchronous coroutine consumption port is generated for the events processed by the coroutine through the shared scope. The asynchronous coroutine consumption port is used to synchronously process the events. Compared with the existing RxJava event stream processing method, this application solves the problems of excessive system resource consumption in traditional concurrent processing methods and low efficiency of single-threaded event stream processing by precisely controlling the switching on and off of coroutines and the number of concurrent processes. This makes event stream processing more flexible and efficient. At the same time, due to the characteristics of coroutine code, the concurrent event stream is not thrown asynchronously, but in a synchronous manner, which simplifies the code complexity and makes the event stream processing process more flexible and efficient.
[0111] For specific limitations regarding the event stream processing device, please refer to the limitations of the event stream processing method above, which will not be repeated here. Each module in the aforementioned event stream processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.
[0112] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of an event flow processing method on the server side.
[0113] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of an event flow processing method.
[0114] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:
[0115] Create a scope in which the running coroutine code can be executed, and store the scope in an atomic operation class so that the atomic operation class holds a shared scope;
[0116] When a processing instruction for an event stream in the message queue is initiated, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction.
[0117] Within the shared scope corresponding to the current coroutine code, perform coroutine consumption processing on events in the message queue, and determine whether the number of events processed by the coroutine exceeds the preset concurrency value when performing coroutine consumption processing;
[0118] If the limit is not exceeded, then an asynchronous coroutine consumer port is generated for the events processed by the coroutine through the shared scope, and the asynchronous coroutine consumer port is used to perform synchronous processing of the events.
[0119] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0120] Create a scope in which the running coroutine code can be executed, and store the scope in an atomic operation class so that the atomic operation class holds a shared scope;
[0121] When a processing instruction for an event stream in the message queue is initiated, the shared scope corresponding to the current coroutine code is extracted from the atomic operation class according to the processing instruction.
[0122] Within the shared scope corresponding to the current coroutine code, perform coroutine consumption processing on events in the message queue, and determine whether the number of events processed by the coroutine exceeds the preset concurrency value when performing coroutine consumption processing;
[0123] If the limit is not exceeded, then an asynchronous coroutine consumer port is generated for the events processed by the coroutine through the shared scope, and the asynchronous coroutine consumer port is used to perform synchronous processing of the events.
[0124] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0125] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0127] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. An event stream processing method, characterized in that, The method includes: Create a scope that the code of the running coroutine can execute, and store the scope in an atomic operation class so that the atomic operation class holds a shared scope. The atomic operation class is used to share the scope with the coroutine and maintain the scope from being modified by other coroutines while the coroutine uses the scope. When the processing instruction for the event stream in the message queue is initiated, a basic loop pattern is generated within the shared scope. The basic loop pattern is used to obtain the shared scope corresponding to the current coroutine code from the atomic operation class. If the shared scope corresponding to the current coroutine code does not exist in the atomic operation class, a new scope variable is generated, and the new scope variable replaces the shared scope held in the atomic operation class through atomic operations to obtain a replacement result. Based on the replacement result, it is determined whether to end the basic loop pattern. Within the shared scope corresponding to the current coroutine code, coroutine consumption processing is performed on the events in the message queue, and during each processing of the current event in the event stream, it is determined whether the number of events processed by the coroutine has reached the maximum concurrency. If the maximum concurrency is reached, it is determined that the current event cannot be processed concurrently; if the maximum concurrency is not reached, asynchronous coroutine consumption is used to process the event concurrently. An asynchronous coroutine consumption port is generated for the event to be processed by the shared scope, and multiple events are processed by coroutine consumption at the same time using the asynchronous coroutine consumption port.
2. The method according to claim 1, characterized in that, The scope that the code for creating and running coroutines can execute is stored in an atomic operation class so that the atomic operation class holds a shared scope, including: Declare and initialize the coroutine scope object for the event stream processing scenario; Define the global or custom scope that the coroutine code can execute in the coroutine scope object, and set the lifecycle maintained by the coroutine code in the event flow processing scenario so that the execution cycle of the coroutine code is consistent with the lifecycle maintained by the event flow processing scenario. The global scope or custom scope is stored in an atomic operation class so that the atomic operation class holds a shared scope.
3. The method according to claim 1, characterized in that, The method further includes: Create a message queue for processing event streams, receive events to be processed, and encapsulate the data required for the events to be processed in the order of event processing from beginning to end and cache it at the end of the message queue.
4. The method according to claim 1, characterized in that, If the replacement result is successful, then determining whether to end the basic loop mode based on the replacement result includes: Based on the replacement result, output the shared scope held in the atomic operation class after replacement, and end the basic loop pattern; If the replacement result is a replacement failure, then determine whether to end the basic loop mode based on the replacement result, including: The new scope variable is destroyed based on the replacement result, and the basic loop pattern is re-executed to obtain the shared scope corresponding to the current coroutine code from the atomic operation class.
5. The method according to any one of claims 1-4, characterized in that, The step of performing coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and determining whether the number of events processed by the coroutine exceeds a preset concurrency value during the coroutine consumption process, includes: Within the shared scope corresponding to the current coroutine code, perform coroutine consumption processing on events in the message queue, and use a coroutine counter to record the current number of concurrent tasks during the coroutine consumption processing. When the asynchronous coroutine consumption port detects that any event in the concurrent coroutine consumption process has been completed, it updates the current number of concurrent tasks recorded in the coroutine counter. Based on the current number of concurrent tasks, determine whether the number of events processed by the coroutine exceeds the preset concurrency value; After performing coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and determining whether the number of events processed by the coroutine exceeds a preset concurrency value during the coroutine consumption processing, the method includes: If the number of events exceeds the limit, the processing instructions for the event stream in the message queue will be closed until the number of coroutines executing in the event stream in the message queue does not exceed the preset concurrency value.
6. The method according to any one of claims 1-4, characterized in that, The method further includes: When the event stream in the message queue has finished processing, it is determined whether the shared scope held in the atomic operation class has reached the execution cycle of the coroutine code. If so, the shared scope held in the atomic operation class is destroyed.
7. An event stream processing apparatus, characterized in that, The device includes: A creation module is used to create a scope that can be executed by running coroutine code. The scope is stored in an atomic operation class so that the atomic operation class holds a shared scope. The atomic operation class is used to share the scope with coroutines and to keep the scope from being modified by other coroutines while the coroutines are using the scope. The extraction module is used to generate a basic loop pattern within the shared scope when a processing instruction for an event stream in the startup message queue is invoked. This basic loop pattern is then used to retrieve the shared scope corresponding to the current coroutine code from the atomic operation class. If the shared scope corresponding to the current coroutine code does not exist in the atomic operation class, a new scope variable is generated, and the new scope variable replaces the shared scope held in the atomic operation class through an atomic operation, resulting in a replacement. The module then determines whether to end the basic loop pattern based on the replacement result. The judgment module is used to perform coroutine consumption processing on events in the message queue within the shared scope corresponding to the current coroutine code, and to determine whether the number of events processed by the coroutine has reached the maximum concurrency during each processing of the current event in the event stream. The processing module is used to determine that the current event cannot be processed concurrently if the maximum concurrency is reached; if the maximum concurrency is not reached, it uses asynchronous coroutine consumption to process the event concurrently. It generates asynchronous coroutine consumption ports for the events to be processed by coroutines through the shared scope, and uses the asynchronous coroutine consumption ports to process multiple events simultaneously.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.