An event stream processing method and apparatus

Abstract

The application relates to the technical field of event stream processing, in particular to an event stream processing method and device, which can establish a connection with a server, acquire a real-time data stream sent by the server, trigger a corresponding event processing function, process and analyze the real-time data stream by using the event processing function, judge whether the response is successful, if yes, filter the real-time data stream according to pre-acquired business requirements to obtain an event stream, perform conversion and aggregation operations on the event stream, and generate data after event stream processing. It can be understood that the technical scheme provided by the application integrates event stream processing on an event processing function, and the function is used in the client to pass the acquired event stream to the function, so that data after event stream processing can be generated, a polling mode is not needed, the processing efficiency is high, and the developer can more conveniently process the event stream sent by the server.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of event stream processing, in particular to an event stream processing method and device. BACKGROUND

[0002] EventStream is an open source network protocol for streaming data to subscribers. The protocol is mainly used for server push technology, such as WebSockets and Comet. In web development, it can be used to implement server-side push so that the client can receive data updates on the server in a timely manner. EventStream Processing (ESP) refers to computing and analyzing stream events in near real-time, matching events with event query statements, and notifying listeners.

[0003] However, the traditional event stream processing method uses a polling method to process event streams, which has the problems of performance and traffic waste, resulting in low processing efficiency. SUMMARY

[0004] Therefore, the purpose of the present application is to provide an event stream processing method and device to solve the problems of performance and traffic waste and low processing efficiency in the prior art event stream processing method.

[0005] According to a first aspect of an embodiment of the present application, an event stream processing method is provided, applied to a client, comprising:

[0006] establishing a connection with a server, obtaining a real-time data stream sent by the server, and triggering a corresponding event processing function;

[0007] processing and analyzing the real-time data stream using the event processing function, and determining whether the response is successful;

[0008] If yes, filtering the real-time data stream according to the pre-acquired business requirements to obtain an event stream therein, performing conversion and aggregation operations on the event stream, and generating data after event stream processing.

[0009] Preferably, the connection with the server comprises:

[0010] creating an EventSource object on the client side through JavaScript;

[0011] connecting the EventSource object to the EventStream event stream on the server side and requesting to subscribe to the event stream.

[0012] Preferably, the obtaining of the real-time data stream sent by the server comprises:

[0013] The client uses the event listening mechanism of JavaScript to capture the real-time data stream pushed by the server.

[0014] Preferably, the processing and analyzing of the real-time data stream by the event processing function further comprises:

[0015] If the response fails, an error is thrown and error information, or a status code and a status message, is returned to the upper layer function.

[0016] Preferably, the returning of the error information, or the status code and the status message, to the upper layer function further comprises:

[0017] parsing the content of the real-time data stream that fails to respond into error information in JSON format;

[0018] if the parsing fails, an empty object is returned; if the parsing succeeds, the parsed error information is stored in an error variable;

[0019] if the error variable is not empty, the content in the error variable is converted into a JSON string and thrown;

[0020] if the error variable is empty, a status code and a status text of the response are obtained and thrown.

[0021] Preferably, the filtering of the real-time data stream to obtain an event stream, and the conversion and aggregation operation on the event stream, comprise:

[0022] creating a parser object, and using the parser object to parse the real-time data stream from the server into a JavaScript object, the real-time data stream being an EventSource message;

[0023] when the parser parses a valid event type data object representing the event stream, executing a defined callback function;

[0024] passing the parsed content as a parameter to the callback function, and obtaining data processed by the callback function.

[0025] Preferably, the parsing of the valid event type data object representing the event stream comprises:

[0026] iteratively reading data blocks from the body of a resp object, the resp object being an information object of the real-time data stream obtained from a remote server;

[0027] decoding each data block into a UTF-8 string, and using a parser to pass the string to the callback function.

[0028] Preferably, the method further comprises:

[0029] When receiving the disconnect instruction, the client disconnects the communication connection established with the server, and stops acquiring the event stream.

[0030] According to a second aspect of the embodiments of the present application, an event stream processing device is provided, comprising:

[0031] a host and a memory connected to the host;

[0032] the memory, in which program instructions are stored;

[0033] the host is configured to execute the program instructions stored in the memory, and execute the method according to any one of the preceding embodiments.

[0034] The technical solutions provided by the embodiments of the present application can have the following beneficial effects:

[0035] It can be understood that the technical solutions provided by the present application can establish a connection with a server, acquire a real-time data stream sent by the server, trigger a corresponding event processing function, process and analyze the real-time data stream by using the event processing function, determine whether the response is successful, if so, filter the event stream from the real-time data stream according to the pre-acquired business requirements, and generate data processed by the event stream. It can be understood that the technical solutions provided by the present application integrate the event stream processing on the event processing function, and pass the acquired event stream to the function in the client by using the function, so as to generate the data processed by the event stream. Without using the polling mode, the processing efficiency is high, and the developers can more conveniently process the event stream sent by the server.

[0036] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0038] Figure 1 is a step schematic diagram of an event stream processing method according to an exemplary embodiment;

[0039] Figure 2 is a processing flow schematic diagram according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not intended to represent all embodiments in accordance with the present application. Rather, they merely represent typical embodiments in accordance with a portion of the application, as detailed in the appended claims.

[0041] Embodiment One

[0042] Figure 1 is a schematic diagram of steps of an event stream processing method according to an exemplary embodiment, see Figure 1 An event stream processing method is provided, applied to a client, comprising:

[0043] Step S11, a connection is established with a server, and real-time data stream sent by the server is acquired in real time, triggering a corresponding event processing function;

[0044] Step S12, the real-time data stream is processed and analyzed using the event processing function, and it is determined whether the response is successful;

[0045] Step S13, if yes, the real-time data stream is filtered to obtain an event stream therein according to a pre-acquired business requirement, and the event stream is converted and aggregated to generate data after event stream processing.

[0046] It can be understood that the technical solution provided by the present embodiment can establish a connection with a server, acquire real-time data stream sent by the server, trigger a corresponding event processing function, process and analyze the real-time data stream using the event processing function, determine whether the response is successful, filter the real-time data stream to obtain an event stream therein if the response is successful according to a pre-acquired business requirement, convert and aggregate the event stream, and generate data after event stream processing. It can be understood that the technical solution provided by the present embodiment integrates event stream processing on an event processing function, and the function is used in a client to pass the acquired event stream to the function, so as to generate data after event stream processing, without using a polling mode, with high processing efficiency, and making it more convenient for developers to process event stream sent by a server.

[0047] It should be noted that in step S11, the connection with the server includes:

[0048] An EventSource object is created in the client through JavaScript;

[0049] The EventSource object is connected to an EventStream event stream on the server side, and an event stream subscription is requested.

[0050] In a specific implementation, JavaScript (JS for short) is a lightweight, interpreted or just-in-time compiled programming language with first-class functions. Although best known as the underlying engine that drives web browsers, it is also used in non-browser environments including desktop and server programs. JavaScript is a prototype-based, multi-paradigm dynamic scripting language that supports object-oriented, imperative, declarative, functional programming styles.

[0051] In an actual application, this step can establish a connection between the client and the server, and preferably, the client establishes a connection with the server through WebSocket or the like. WebSocket is a protocol for web browsers to create a full-duplex communication over a single TCP connection. It enables reliable, real-time, event-based, two-way communication in web applications, which makes the server actively push data to the client without the client initiating a request.

[0052] It should be noted that the real-time data stream sent by the server includes:

[0053] The client uses the event listening mechanism of JavaScript to capture the real-time data stream pushed by the server in real time.

[0054] In a specific implementation, after the connection between the client and the server is established, the server can use the EventStream protocol to send the real-time data stream to the client, and then the client uses the event listening mechanism of JavaScript to capture the real-time data pushed by the server. The real-time data stream contains a corresponding event stream.

[0055] It should be noted that after the real-time data stream is processed and analyzed by the event processing function, it further includes:

[0056] If the response fails, an error is thrown, and error information, or a status code and a status message, is returned to the upper layer function.

[0057] In specific practice, since the server side will continuously push data to the client side, some of the pushed data are event streams, and some are not. Taking the client side requesting the openai interface as an example, after requesting the openai interface, a result response (hereinafter referred to as resp) is obtained, which is an information object obtained from a remote server and contains an event stream. The resp and the onMessage callback function are passed as parameters to an event processing function (parseSSEResponse function). The event processing function is used to process and analyze the event stream, to determine whether the response is successful (the successful status code is 200-299), and to check whether the result ok attribute returned by the event stream is true. If yes, the response is successful, otherwise the response fails. If the response fails, an error is thrown and error information is returned to the upper layer function, or the status code and status message of the response are returned.

[0058] It should be noted that the error information, or the status code and status message, returned to the upper layer function also includes:

[0059] parsing the content of the real-time data stream that fails to respond into JSON format error information;

[0060] If the parsing fails, an empty object is returned; if the parsing succeeds, the parsed error information is stored in the error variable;

[0061] If the error variable is not empty, the content in the error variable is converted into a JSON string and thrown;

[0062] If the error variable is empty, the status code and status text of the response are obtained and thrown.

[0063] JSON: full name for JavaScript Object Notation (JavaScript Object Notation), which is a lightweight data exchange format. It is based on a subset of the JavaScript language and uses key-value pairs to represent data.

[0064] In the above example, when processing a failed response, first try to parse the content of the response into JSON format, if the parsing fails, return an empty object, this parsed error information will be stored in the error variable. When processing the error information, if the error is not empty, it is converted into a JSON string and thrown, otherwise the status code and status text of the response are thrown.

[0065] It should be noted that the filtering of the real-time data stream to obtain the event stream therein, and the conversion and aggregation operation of the event stream, include:

[0066] creating a parser object that parses a live data stream from a server into a JavaScript object, the live data stream being an EventSource message;

[0067] when the parser parses a valid event-type data object representing the event stream, executing a defined callback function;

[0068] passing the parsed content as an argument to the callback function, and obtaining the processed data of the event stream through the callback function.

[0069] In specific practice, a library called eventsource-parser is used, in which the createParser function aims to create a parser object that can parse an EventSource message sent from a server into a JavaScript object; here, only event-type data is considered. When the parser parses a valid event-type data object, a defined callback function onMessage is executed, and the parsed message content is passed as an argument to the callback function.

[0070] It should be noted that the valid event-type data object representing the event stream includes:

[0071] iterating to read data blocks from the body of a resp object, the resp object being an information object of a live data stream obtained from a remote server;

[0072] decoding each data block into a UTF-8 string, and using the parser to pass the string to the callback function.

[0073] In specific practice, the for-await-of syntax can be used to iterate to read data blocks from the body of the resp object, and the for await...of syntax is used here, indicating that we will wait for the arrival of each chunk during iteration. The streamAsyncIterable function can be used to convert a possibly chunked, asynchronously sent response body into an iterable byte stream, and the streamAsyncIterable function is a helper function for response iteration functions. In each data block, the TextDecoder is used to decode the data block into a UTF-8 string, and the feed method of the parser is used to pass the string to the parser.

[0074] It should be noted that the method further includes:

[0075] when receiving a disconnection instruction, disconnecting the communication connection established between the client and the server, and stopping obtaining the event stream.

[0076] In the specific practice, the client disconnects the connection with the server by closing the WebSocket connection and the like when the event stream is not needed to be subscribed.

[0077] Figure 2 is a processing flow diagram shown according to an exemplary embodiment, see Figure 2 The technical solution shown in the embodiment judges whether the response is successful by using the event processing function (parseSSEResponse function) when the request result resp containing the event stream is obtained from the server, executes the corresponding error step if the response fails, executes the success step if the response succeeds, reads the event stream data by using the asynchronous iterator, and throws the read data string.

[0078] It can be understood that the technical solution shown in the embodiment can integrate the steps into a function, import the function in the JavaScript file of the client, access the interface to obtain the SSE data at the place where the SSE data needs to be parsed, pass the result to the function, and define the onMessage callback function to process the response data. That is, the data stream can be parsed and processed, so that the developer can more conveniently process the server sent event.

[0079] SSE (Server-Sent Events) is a one-way communication protocol between a browser and a server defined by an HTML5 specification. It allows the server to push data to the client's web page without the client explicitly requesting by polling or other means.

[0080] Embodiment Two

[0081] An event stream processing device is provided, comprising:

[0082] A host controller and a memory connected to the host controller;

[0083] The memory, wherein the memory stores program instructions;

[0084] The host controller is configured to execute the program instructions stored in the memory to execute the method of any one of the above embodiments.

[0085] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0086] It should be noted that, in the description of the present application, the terms "first", "second" and the like are used only for descriptive purposes, and cannot be construed as indicating or implying relative importance. In addition, in the description of the present application, the meaning of "a plurality of" is at least two, unless otherwise specified.

[0087] Any process or method descriptions or descriptions of the flow diagrams described herein or otherwise described in the present application can be understood as representing the steps of the code modules, segments or portions of the respective codes, and the scope of the preferred embodiments of the present application includes additional implementation in which the functions performed by the code modules, segments or portions are carried out in a different order, in substantially simultaneous fashion, or in reverse order, and the like, as will be understood by those skilled in the art.

[0088] It should be understood that parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0089] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, which includes one or a combination of the steps of the method embodiments when executed.

[0090] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can exist physically, or two or more units can be integrated in one module. The above integrated module can be realized in the form of hardware or in the form of software functional module. The integrated module, if realized in the form of software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0091] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0092] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0093] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary, and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. An event stream processing method applied to a client, characterized in that, The method comprises the following steps: establishing a connection with a server, obtaining a real-time data stream sent by the server, and triggering a corresponding event processing function; the step of establishing a connection with the server comprises the following steps: creating an EventSource object on a client side by using JavaScript; and connecting the EventSource object to an EventStream event stream on a server side, and requesting to subscribe to the event stream; processing and analyzing the real-time data stream by using the event processing function, and determining whether the response is successful; if the response is successful, filtering the real-time data stream to obtain an event stream in the real-time data stream, and performing conversion and aggregation operations on the event stream to generate data processed by the event stream according to pre-acquired business requirements; the step of filtering the real-time data stream to obtain an event stream in the real-time data stream, and performing conversion and aggregation operations on the event stream comprises the following steps: creating a parser object, parsing the real-time data stream from the server side into a JavaScript object by using the parser object, and the real-time data stream is an EventSource message; when the parser parses a valid event type data object representing the event stream, executing a defined callback function; and passing the parsed content as a parameter to the callback function to obtain data processed by the event stream through the callback function.

2. The method of claim 1, wherein, the step of obtaining the real-time data stream sent by the server side comprises the following steps: capturing the real-time data stream pushed in real time by the server side on the client side by using an event listening mechanism of JavaScript.

3. The method of claim 1, wherein, after the step of processing and analyzing the real-time data stream by using the event processing function, the method further comprises the following steps: if the response fails, throwing an error, and returning error information, or a status code and a status message to an upper layer function.

4. The method of claim 3, wherein, the step of returning error information, or a status code and a status message to the upper layer function further comprises the following steps: parsing the content of the real-time data stream with a failed response into error information in a JSON format; if the parsing fails, returning an empty object; if the parsing succeeds, storing the parsed error information in an error variable; if the error variable is not empty, converting the content in the error variable into a JSON string and throwing the JSON string; if the error variable is empty, obtaining and throwing a status code and a status text of a response.

5. The method of claim 1, wherein, the step of parsing a valid event type data object representing the event stream comprises the following steps: iteratively reading data blocks from a body of a resp object, the resp object being an information object of the real-time data stream obtained from a remote server; decoding each data block into a UTF-8 string, and passing the string to the callback function by using a parser.

6. The method according to any one of claims 1 to 5, characterized in that, the method further comprises the following steps: when a disconnection instruction is received, disconnecting a communication connection established between the client and the server, and stopping obtaining the event stream.

7. An event stream processing device, characterized by The method comprises the following steps: a host and a memory connected to the host; the memory stores program instructions; the host executes the program instructions stored in the memory to execute the method according to any one of claims 1 to 6.

Images