Streaming data processing method and system
By establishing an internal message queue in the stream channel and converting the stream messages into a message format, the problems of high resource consumption and long parsing time in the existing technology are solved, efficient streaming data processing is achieved, and the system resource consumption and improvement is reduced. Processing efficiency.
Patent Information
- Application Number
- CN202411211712.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-03
AI Technical Summary
The existing streaming data processing system cannot know the data type, resulting in increased resource consumption and excessive parsing time, which affects processing efficiency.
Establish an internal message queue in the stream channel, convert the stream messages into message format and pass them to the queue, and process them according to the message format during consumption, reduce the parsing needs, and support the delivery of multiple types of data through one channel.
Reduces resource consumption, improves streaming data processing efficiency, reduces workload, and ensures system performance.
Smart Images
Figure CN119248533A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for streaming data processing. Background Art
[0002] At present, most of the streaming data functions exist in the system, which are mainly used for the decoupling of upstream and downstream modules. Usually, the upstream module will pass various messages to the downstream module through the stream channel. After receiving these messages, the downstream module will parse the streaming data and then perform subsequent message processing. Since the streaming data cannot know the data type, different streaming channels are usually established for different types of streaming data during data transmission. The streaming data is transmitted through the corresponding streaming channels to distinguish the type of streaming data. However, with the increase of streaming channels, resource consumption is increased and the performance of the system is reduced. Although some improvements have been made later, such as: transmitting 2-3 types of streaming data through one streaming channel to reduce the number of streaming channels and reduce resource consumption, in the subsequent analysis of the downstream module, since multiple types of streaming data use the same streaming channel for transmission, the downstream module does not know the specific type of streaming data and can only perform multiple analyses. When one analysis fails, another method is used for analysis, resulting in too long analysis time, increasing time consumption and also increasing resource consumption, affecting the processing efficiency of streaming data. Summary of the invention
[0003] The technical problem to be solved by the present invention is: the present invention provides a method and system for processing streaming data, which improves the processing efficiency of streaming data, reduces resource consumption, and ensures system performance.
[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0005] In a first aspect, the present invention provides a method for streaming data processing, comprising the steps of:
[0006] An internal message queue is established in the stream channel. When a stream message is generated, the stream message is converted into streaming data through the stream channel and a message format of the stream message is generated, and the message format is transmitted to the internal message queue.
[0007] When the streaming data is consumed, the message format is synchronously acquired from the internal message queue, and the streaming data is consumed according to the message format.
[0008] The beneficial effects of the present invention are as follows: when a stream message is generated, it is converted into streaming data through a stream channel, and at the same time, a message format of the stream message is generated, and the message format is passed to an internal message queue established in the stream channel, so that when the streaming data is consumed, the corresponding message format is synchronously obtained from the internal message queue to consume the streaming data, that is, the conversion of streaming data is directly realized in the stream channel, so that upstream modules, downstream modules and consumers only need to process objects without message conversion, and consume streaming data according to the message format, that is, upstream modules and downstream modules do not need to parse streaming data, thereby reducing workload and resource consumption, and supporting one stream channel to transmit multiple types of streaming data in the form of generating a message format and passing it to the internal message queue, without the need for multiple parsings, and only according to the message format can be consumed, thereby reducing resource consumption, improving the overall streaming data processing efficiency, and ensuring system performance.
[0009] Optionally, converting the stream message into streaming data through the stream channel and generating a message format of the stream message includes:
[0010] The stream message is converted into streaming data through the stream channel and the message type of the stream message is obtained at the same time, and a message format corresponding to the message type is obtained from a preset mapping relationship table according to the message type.
[0011] It can be known from the above description that the message format corresponding to the message type of the stream message is obtained by means of a preset mapping relationship table, thereby ensuring the accuracy of the message format.
[0012] Optionally, consuming the streaming data according to the message format includes:
[0013] It is determined whether the consumption is successful. If so, the consumption status of the streaming data is marked as normal consumption. If not, the consumption status of the streaming data is marked as abnormal consumption.
[0014] According to the above description, the consumption status of streaming data is marked as normal consumption or abnormal consumption according to whether the consumption is successful, which makes it easier to understand the consumption situation and facilitate subsequent management.
[0015] Optionally, it also includes:
[0016] The streaming data with abnormal consumption status is shifted to the abnormal message queue, and the streaming data in the abnormal message queue is re-consumed a preset number of times. If the preset number of re-consumptions all fail, an alarm is issued for the streaming data. If at least one of the preset number of re-consumptions is successful, the consumption status of the streaming data is updated to normal consumption.
[0017] According to the above description, the abnormally consumed streaming data is moved to the abnormal consumption queue for re-consumption, which improves the consumption success rate while avoiding its impact on the internal message queue, improves the overall streaming data processing efficiency, and ensures system performance.
[0018] Optionally, the preset number of times is [1,3].
[0019] In a second aspect, the present invention provides a system for stream data processing, comprising:
[0020] A conversion module, used to establish an internal message queue in the stream channel, and when a stream message is generated, convert the stream message into streaming data through the stream channel and generate a message format of the stream message, and pass the message format to the internal message queue;
[0021] The consumption module is used to synchronously obtain the message format from the internal message queue when consuming the streaming data, and consume the streaming data according to the message format.
[0022] The beneficial effects of the present invention are as follows: when a stream message is generated, it is converted into streaming data through a stream channel, and at the same time, a message format of the stream message is generated, and the message format is passed to an internal message queue established in the stream channel, so that when the streaming data is consumed, the corresponding message format is synchronously obtained from the internal message queue to consume the streaming data, that is, the conversion of streaming data is directly realized in the stream channel, so that upstream modules, downstream modules and consumers only need to process objects without message conversion, and consume streaming data according to the message format, that is, upstream modules and downstream modules do not need to parse streaming data, thereby reducing workload and resource consumption, and supporting one stream channel to transmit multiple types of streaming data in the form of generating a message format and passing it to the internal message queue, without the need for multiple parsings, and only according to the message format can be consumed, thereby reducing resource consumption, improving the overall streaming data processing efficiency, and ensuring system performance.
[0023] Optionally, the conversion module is specifically:
[0024] The stream message is converted into streaming data through the stream channel and the message type of the stream message is obtained at the same time, and a message format corresponding to the message type is obtained from a preset mapping relationship table according to the message type.
[0025] It can be known from the above description that the message format corresponding to the message type of the stream message is obtained by means of a preset mapping relationship table, thereby ensuring the accuracy of the message format.
[0026] Optionally, the consumption module includes:
[0027] The judgment module is used to judge whether the consumption is successful. If so, the consumption status of the streaming data is marked as normal consumption; if not, the consumption status of the streaming data is marked as abnormal consumption.
[0028] According to the above description, the consumption status of streaming data is marked as normal consumption or abnormal consumption according to whether the consumption is successful, which makes it easier to understand the consumption situation and facilitate subsequent management.
[0029] Also includes:
[0030] The retry module is used to shift the streaming data with abnormal consumption status to the abnormal message queue, and re-consume the streaming data in the abnormal message queue for a preset number of times. If the preset number of re-consumptions all fail, an alarm is issued for the streaming data. If at least one of the preset number of re-consumptions succeeds, the consumption status of the streaming data is updated to normal consumption.
[0031] According to the above description, the abnormally consumed streaming data is moved to the abnormal consumption queue for re-consumption, which improves the consumption success rate while avoiding its impact on the internal message queue, improves the overall streaming data processing efficiency, and ensures system performance.
[0032] Optionally, the preset number of times is [1,3]. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flow chart of a method for stream data processing provided by this embodiment;
[0034] Figure 2 A schematic diagram of the overall flow of a method for stream data processing provided in this embodiment;
[0035] Figure 3 A schematic diagram of the structure of a streaming data processing system provided in this embodiment.
[0036] [Description of Reference Numerals]
[0037] 1. A system for stream data processing;
[0038] 2. Conversion module;
[0039] 3. Consumption module; 31. Judgment module;
[0040] 4. Retry module. DETAILED DESCRIPTION
[0041] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.
[0042] Embodiment 1
[0043] Please refer to Figure 1 to Figure 2 The present invention provides a method for streaming data processing, comprising the steps of:
[0044] S1. Establish an internal message queue in a stream channel. When a stream message is generated, convert the stream message into streaming data through the stream channel and generate a message format of the stream message, and pass the message format to the internal message queue.
[0045] In this embodiment, if Figure 2 As shown, the production and consumption of streaming data are sequential. An internal message queue is established in the stream channel. When a stream message is generated, the stream message is converted into streaming data through the stream channel and the message format of the stream data is generated at the same time. The message format is passed to the established internal message queue.
[0046] At this time, the converting of the stream message into streaming data and generating the message format of the stream message through the stream channel in step S1 includes:
[0047] S11. Convert the stream message into streaming data through the stream channel and obtain the message type of the stream message at the same time, and obtain a message format corresponding to the message type from a preset mapping relationship table according to the message type.
[0048] In this embodiment, if Figure 2 As shown, when a stream message is generated, what is transmitted is a message object, which includes a message type and specific data content. Therefore, through the message type of the stream message, the corresponding message format is obtained from a preset mapping relationship table, that is, a mapping relationship table between message type and message format, according to the message type.
[0049] S2. When consuming the streaming data, synchronously obtain the message format from the internal message queue, and consume the streaming data according to the message format.
[0050] In this embodiment, if Figure 2 As shown, when streaming data is consumed, the message format is synchronously obtained from the internal message queue, and direct consumption of the streaming message is achieved according to the message format.
[0051] At this time, consuming the streaming data according to the message format in step S2 includes:
[0052] S21. Determine whether the consumption is successful. If so, mark the consumption status of the streaming data as normal consumption. If not, mark the consumption status of the streaming data as abnormal consumption.
[0053] In this embodiment, if Figure 2 As shown, if the consumption is successful, the consumption status of the streaming data will be marked as normal consumption. If the consumption fails, the consumption status of the streaming data will be marked as abnormal consumption.
[0054] In this embodiment, the streaming data with abnormal consumption status will be further processed, and the specific steps are as follows:
[0055] The streaming data with abnormal consumption status is shifted to the abnormal message queue, and the streaming data in the abnormal message queue is re-consumed a preset number of times. If the preset number of re-consumptions all fail, an alarm is issued for the streaming data. If at least one of the preset number of re-consumptions is successful, the consumption status of the streaming data is updated to normal consumption.
[0056] In this embodiment, the streaming data with abnormal consumption status will be shifted to the abnormal consumption queue. At this time, the abnormal consumption queue can be pre-established or currently established. The streaming data in the abnormal message queue is re-consumed a preset number of times. The preset number at this time is [1,3]. When the preset number is 1, 1 re-consumption is performed. When the preset number is 2, if both re-consumptions fail, an alarm is issued for the streaming data. If at least 1 success occurs, the consumption status of the streaming and data is updated to normal consumption. At this time, if the first re-consumption is successful, the consumption status can be directly updated without re-consumption.
[0057] Embodiment 2
[0058] Please refer to Figure 3 The present invention provides a streaming data processing system 1, comprising: a conversion module 2, a consumption module 3, a judgment module 31 and a retry module 4.
[0059] The conversion module 2 is used to establish an internal message queue in the stream channel. When a stream message is generated, the stream message is converted into streaming data through the stream channel and a message format of the stream message is generated, and the message format is transmitted to the internal message queue.
[0060] The consumption module 3 is used to synchronously obtain the message format from the internal message queue when consuming the streaming data, and consume the streaming data according to the message format.
[0061] Specifically, the conversion module 2 is specifically:
[0062] The stream message is converted into streaming data through the stream channel and the message type of the stream message is obtained at the same time, and a message format corresponding to the message type is obtained from a preset mapping relationship table according to the message type.
[0063] Specifically, the consumption module 3 includes:
[0064] The judging module 31 is used to judge whether the consumption is successful. If so, the consumption status of the streaming data is marked as normal consumption. If not, the consumption status of the streaming data is marked as abnormal consumption.
[0065] Specifically, it also includes:
[0066] The retry module 4 is used to move the streaming data whose consumption status is abnormal consumption to the abnormal message queue, and re-consume the streaming data in the abnormal message queue a preset number of times. If the preset number of re-consumptions all fail, an alarm is issued for the streaming data. If at least one of the preset number of re-consumptions is successful, the consumption status of the streaming data is updated to normal consumption.
[0067] Specifically, the preset number of times is [1, 3].
[0068] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.
[0069] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0070] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.
[0071] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.
[0072] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0073] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.
[0074] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.
Claims
1. A method for stream data processing, characterized in that: include: An internal message queue is established in the stream channel. When a stream message is generated, the stream message is converted into streaming data through the stream channel and a message format of the stream message is generated, and the message format is transmitted to the internal message queue. When the streaming data is consumed, the message format is synchronously acquired from the internal message queue, and the streaming data is consumed according to the message format.
2. A method for stream data processing according to claim 1, characterized in that: The converting the stream message into streaming data through the stream channel and generating the message format of the stream message at the same time comprises: The stream message is converted into streaming data through the stream channel and the message type of the stream message is obtained at the same time, and a message format corresponding to the message type is obtained from a preset mapping relationship table according to the message type.
3. A method for stream data processing according to claim 1, characterized in that: The consuming the streaming data according to the message format comprises: It is determined whether the consumption is successful. If so, the consumption status of the streaming data is marked as normal consumption. If not, the consumption status of the streaming data is marked as abnormal consumption.
4. A method for stream data processing according to claim 3, characterized in that: Also includes: The streaming data with abnormal consumption status is shifted to the abnormal message queue, and the streaming data in the abnormal message queue is re-consumed a preset number of times. If the preset number of re-consumptions all fail, an alarm is issued for the streaming data. If at least one of the preset number of re-consumptions is successful, the consumption status of the streaming data is updated to normal consumption.
5. A method for stream data processing according to claim 4, characterized in that: The preset number of times is [1,3].
6. A system for stream data processing, characterized in that: include: A conversion module, used to establish an internal message queue in the stream channel, and when a stream message is generated, convert the stream message into streaming data through the stream channel and generate a message format of the stream message, and pass the message format to the internal message queue; The consumption module is used to synchronously obtain the message format from the internal message queue when consuming the streaming data, and consume the streaming data according to the message format.
7. A streaming data processing system as claimed in claim 6, characterized in that: The conversion module is specifically: The stream message is converted into streaming data through the stream channel and the message type of the stream message is obtained at the same time, and a message format corresponding to the message type is obtained from a preset mapping relationship table according to the message type.
8. A streaming data processing system as claimed in claim 6, characterized in that: The consumption module includes: The judgment module is used to judge whether the consumption is successful. If so, the consumption status of the streaming data is marked as normal consumption; if not, the consumption status of the streaming data is marked as abnormal consumption.
9. A streaming data processing system as claimed in claim 8, characterized in that: Also includes: The retry module is used to shift the streaming data with abnormal consumption status to the abnormal message queue, and re-consume the streaming data in the abnormal message queue for a preset number of times. If the preset number of re-consumptions all fail, an alarm is issued for the streaming data. If at least one of the preset number of re-consumptions succeeds, the consumption status of the streaming data is updated to normal consumption.
10. A streaming data processing system according to claim 9, characterized in that: The preset number of times is [1,3].