A go-streams-based edge data real-time computing method and system
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 上海沄熹科技有限公司
- Filing Date
- 2022-12-02
- Publication Date
- 2026-07-07
AI Technical Summary
When there is a large amount of user historical behavior data, it is difficult to quickly capture valuable information, resulting in low efficiency in content or product recommendations.
We employ a real-time edge data computing method based on go-streams, using a streaming processing library to decompose, filter, partition, and merge data, enabling rapid processing of massive amounts of high-concurrency data and achieving data updates and scheduled releases.
It enables the rapid capture and distribution of valuable user information, and allows for real-time data updates and publishing.
Smart Images
Figure CN115774741B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of edge data technology, specifically to a method and system for real-time edge data computation based on go-streams. Background Technology
[0002] Currently, content or product recommendations are typically made to users by collecting and analyzing their historical behavior data. However, when faced with massive amounts of historical user behavior data, it becomes difficult to quickly extract valuable user data for effective content or product recommendations.
[0003] Real-time recommendation is a common application scenario of real-time stream computing technology. It analyzes behavioral data collected from users' mobile phones in real time to discover users' interests and preferences, and recommends content or products that users may be interested in.
[0004] Based on this, a method and system for real-time edge data computation based on go-streams were designed and developed. Summary of the Invention
[0005] This invention addresses the needs and shortcomings of current technological development by providing a real-time edge data computing method and system based on go-streams. This method analyzes streaming data in real time and processes massive amounts of high-concurrency data accurately in one go, quickly capturing and distributing valuable information to users, and enabling data updates and real-time publishing.
[0006] First, the present invention provides a real-time edge data computation method based on go-streams, and the technical solution adopted to solve the above-mentioned technical problems is as follows:
[0007] A real-time edge data computation method based on go-streams, implemented using the go-streams streaming library, includes the following implementation details:
[0008] (i) Update and release: Collect real-time edge data, decompose the collected data, perform filtering according to filtering conditions, select data targets and integrate them into the target format to realize data update and release;
[0009] (ii) Scheduled release: Collect real-time edge data and classify it according to the same time requirements. Then, perform different partitioning operations according to the parameter list through the partitioning function to realize the scheduled release of data;
[0010] (III) Merging and optimizing update release and scheduled release: Through the partition function, the update release data is passed to the specified partition for update release processing, the scheduled release logic remains unchanged, and invalid data is processed.
[0011] Optionally, the specific process for implementing step (one) based on the streaming library go-streams to update and publish data is as follows:
[0012] Data flows out from the Source and into a FlatMap, which breaks the data down into individual data points and then outputs them out.
[0013] The outgoing data enters the Filter for filtering.
[0014] Data that does not meet the filtering criteria is discarded.
[0015] Data that meets the filtering criteria enters the next FlatMap, which integrates the data into the target format and then sends it to the Sink.
[0016] Optionally, step (ii) is performed, and the specific process for implementing timed data publishing based on the streaming library go-streams is as follows:
[0017] Data flows out from the Source and into the FlatMap, which breaks the data down into individual data points and then outputs them out.
[0018] The Partition function is used to partition the data flowing out of FlatMap according to the parameter list, so that data with the same time frequency enters the same flow.
[0019] For any given flow, data enters the corresponding TumblingWindow according to set rules. After a set time interval, the data flowing into the TumblingWindow is passed to the FlatMap. The FlatMap processes the incoming data and then outputs it.
[0020] Merge the outflow data from different flows in FlatMap and send it to Sink through a new FlatMap.
[0021] Optionally, during step (iii), the update and scheduled data releases flow from the Source into FlatMap. FlatMap then breaks down the data into individual data points and releases them.
[0022] Use the Partition function to determine whether the data output from FlatMap meets the parameter requirements of the parameter list.
[0023] If the parameter requirement corresponding to the current data is not found in the parameter list, then this data is passed to the ignoreFilter.
[0024] If the parameter requirement corresponding to the current data is found in the parameter list, the data that needs to be transformed is passed to the Filter according to the parameter conditions, the data that needs to be sent periodically is passed to different flows, and finally all data is merged and sent to the Sink through a new FlatMap.
[0025] Secondly, the technical solution adopted by the real-time edge data computing system based on go-streams of the present invention to solve the above-mentioned technical problems is as follows:
[0026] A real-time edge data computing system based on go-streams includes the following modules based on the go-streams streaming library:
[0027] The real-time data processing module is used to collect real-time edge data, decompose the collected data, perform filtering according to filtering conditions, select data targets and integrate them into the target format, and realize data update and publication.
[0028] The data timed processing module is used to collect real-time edge data and classify it according to the same time requirements. Then, it performs different partitioning operations based on the parameter list through a partitioning function to realize the timed release of data.
[0029] The data merging and processing module is used to pass the updated and published data to the specified partition for update and publication processing through the partitioning function. The timed publication logic remains unchanged, and invalid data is processed.
[0030] Optionally, the specific process for the real-time data processing module to implement data update and publishing based on the streaming library go-streams is as follows:
[0031] Data flows out from the Source and into a FlatMap, which breaks the data down into individual data points and then outputs them out.
[0032] The outgoing data enters the Filter for filtering.
[0033] Data that does not meet the filtering criteria is discarded.
[0034] Data that meets the filtering criteria enters the next FlatMap, which integrates the data into the target format and then sends it to the Sink.
[0035] Optionally, the specific process for the timed data processing module, based on the go-streams streaming library, to implement timed data publishing is as follows:
[0036] Data flows out from the Source and into the FlatMap, which breaks the data down into individual data points and then outputs them out.
[0037] The Partition function is used to partition the data flowing out of FlatMap according to the parameter list, so that data with the same time frequency enters the same flow.
[0038] For any given flow, data enters the corresponding TumblingWindow according to set rules. After a set time interval, the data flowing into the TumblingWindow is passed to the FlatMap. The FlatMap processes the incoming data and then outputs it.
[0039] Merge the outflow data from different flows in FlatMap and send it to Sink through a new FlatMap.
[0040] Optionally, the specific process executed by the data merging and processing module based on the streaming library go-streams is as follows:
[0041] Updated and scheduled data releases flow from the Source into the FlatMap, where the FlatMap breaks down the data into individual data points and then releases them.
[0042] Use the Partition function to determine whether the data output from FlatMap meets the parameter requirements of the parameter list.
[0043] If the parameter requirement corresponding to the current data is not found in the parameter list, then this data is passed to the ignoreFilter.
[0044] If the parameter requirement corresponding to the current data is found in the parameter list, the data that needs to be transformed is passed to the Filter according to the parameter conditions, the data that needs to be sent periodically is passed to different flows, and finally all data is merged and sent to the Sink through a new FlatMap.
[0045] The real-time edge data computing method and system based on go-streams of the present invention have the following advantages compared with the prior art:
[0046] This invention can analyze streaming data in real time and accurately process massive amounts of high-concurrency data at once. It can quickly capture and distribute information that is valuable to users, and realize the updating and real-time release of data. Attached Figure Description
[0047] Appendix Figure 1 This is a module connection block diagram of Embodiment 2 of the present invention;
[0048] Appendix Figure 2These are flowcharts illustrating the data update and publishing implementation based on go-streams in Embodiments 1 and 2 of the present invention.
[0049] Appendix Figure 3 These are flowcharts illustrating the timed data publishing based on go-streams in Embodiments 1 and 2 of the present invention.
[0050] Appendix Figure 4 This is a flowchart illustrating the optimization of update publishing and scheduled publishing based on go-streams in Embodiments 1 and 2 of the present invention. Detailed Implementation
[0051] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.
[0052] It is important to note that:
[0053] Source indicates the input data source;
[0054] FlatMap represents an extended form of the map structure. In the following examples, we will pass a function to flatMap that returns a collection for each input, accepting one element and generating zero, one, or more elements;
[0055] ignoreFilter means to ignore the filter;
[0056] Filter represents a predicate filter that filters incoming elements. In the following embodiment, if an incoming element matches the predicate, it is passed downstream via the Filter; otherwise, it is discarded.
[0057] Sink represents the output data source;
[0058] flow represents a data flow;
[0059] Merge represents the operation of combining multiple data streams into a single data stream.
[0060] Example 1: ignoreFilter
[0061] Combined with appendix Figure 2 , 3 4. This embodiment proposes a real-time edge data computation method based on go-streams. This method is implemented based on the streaming processing library go-streams, and the specific implementation includes:
[0062] (i) Update and release: Collect real-time edge data, decompose the collected data, perform filtering according to the filtering conditions, select the data target and integrate it into the target format to realize data update and release.
[0063] Step (1) The specific process of implementing data update and publishing based on the streaming library go-streams is as follows:
[0064] Data flows out from the Source and into a FlatMap, which breaks the data down into individual data points and then outputs them out.
[0065] The outgoing data enters the Filter for filtering.
[0066] Data that does not meet the filtering criteria is discarded.
[0067] Data that meets the filtering criteria enters the next FlatMap, which integrates the data into the target format and then sends it to the Sink.
[0068] (ii) Scheduled release: Collect real-time edge data and classify it according to the same time requirements. Then, perform different partitioning operations according to the parameter list through the partitioning function to realize the scheduled release of data.
[0069] Step (II) The specific process for implementing timed data publishing based on the streaming library go-streams is as follows:
[0070] Data flows out from the Source and into the FlatMap, which breaks the data down into individual data points and then outputs them out.
[0071] The Partition function is used to partition the data flowing out of FlatMap according to the parameter list, so that data with the same time frequency enters the same flow.
[0072] For any given flow, data enters the corresponding TumblingWindow according to set rules. After a set time interval, the data flowing into the TumblingWindow is passed to the FlatMap. The FlatMap processes the incoming data and then outputs it.
[0073] Merge the outflow data from different flows in FlatMap and send it to Sink through a new FlatMap.
[0074] (III) Merging and optimizing update release and scheduled release: Through the partition function, the update release data is passed to the specified partition for update release processing, the scheduled release logic remains unchanged, and invalid data is processed.
[0075] When performing step (iii), the update and scheduled release data flow out from the Source and into the FlatMap. The FlatMap then breaks down the data into individual data points and releases them out.
[0076] Use the Partition function to determine whether the data output from FlatMap meets the parameter requirements of the parameter list.
[0077] If the parameter requirement corresponding to the current data is not found in the parameter list, then this data is passed to the ignoreFilter.
[0078] If the parameter requirement corresponding to the current data is found in the parameter list, the data that needs to be transformed is passed to the Filter according to the parameter conditions, the data that needs to be sent periodically is passed to different flows, and finally all data is merged and sent to the Sink through a new FlatMap.
[0079] Example 2:
[0080] Combined with appendix Figure 1-4 This embodiment proposes a real-time edge data computing system based on go-streams, which includes the following modules based on the go-streams streaming library:
[0081] The real-time data processing module is used to collect real-time edge data, decompose the collected data, perform filtering according to filtering conditions, select data targets and integrate them into the target format, and realize data update and publication.
[0082] The data timed processing module is used to collect real-time edge data and classify it according to the same time requirements. Then, it performs different partitioning operations based on the parameter list through a partitioning function to realize the timed release of data.
[0083] The data merging and processing module is used to pass the updated and published data to the specified partition for update and publication processing through the partitioning function. The timed publication logic remains unchanged, and invalid data is processed.
[0084] In this embodiment, the specific process of data update and publishing implemented by the real-time data processing module based on the streaming library go-streams is as follows:
[0085] Data flows out from the Source and into a FlatMap, which breaks the data down into individual data points and then outputs them out.
[0086] The outgoing data enters the Filter for filtering.
[0087] Data that does not meet the filtering criteria is discarded.
[0088] Data that meets the filtering criteria enters the next FlatMap, which integrates the data into the target format and then sends it to the Sink.
[0089] In this embodiment, the specific process of the data timed processing module implementing timed data publishing based on the streaming library go-streams is as follows:
[0090] Data flows out from the Source and into the FlatMap, which breaks the data down into individual data points and then outputs them out.
[0091] The Partition function is used to partition the data flowing out of FlatMap according to the parameter list, so that data with the same time frequency enters the same flow.
[0092] For any given flow, data enters the corresponding TumblingWindow according to set rules. After a set time interval, the data flowing into the TumblingWindow is passed to the FlatMap. The FlatMap processes the incoming data and then outputs it.
[0093] Merge the outflow data from different flows in FlatMap and send it to Sink through a new FlatMap.
[0094] In this embodiment, the specific process executed by the data merging and processing module based on the streaming library go-streams is as follows:
[0095] Updated and scheduled data releases flow from the Source into the FlatMap, where the FlatMap breaks down the data into individual data points and then releases them.
[0096] Use the Partition function to determine whether the data output from FlatMap meets the parameter requirements of the parameter list.
[0097] If the parameter requirement corresponding to the current data is not found in the parameter list, then this data is passed to the ignoreFilter.
[0098] If the parameter requirement corresponding to the current data is found in the parameter list, the data that needs to be transformed is passed to the Filter according to the parameter conditions, the data that needs to be sent periodically is passed to different flows, and finally all data is merged and sent to the Sink through a new FlatMap.
[0099] In summary, the edge data real-time computing method and system based on go-streams of the present invention can analyze streaming data in real time and accurately process massive amounts of high-concurrency data at once. It can quickly capture and distribute information that is valuable to users, and realize the updating and real-time publishing of data.
[0100] Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principle of the present invention shall fall within the patent protection scope of the present invention.
Claims
1. A method for real-time edge data computation based on go-streams, characterized in that... This method is implemented based on the streaming library go-streams, and its specific implementation includes: (I) Update and Release: Collect real-time edge data, decompose the collected data, perform filtering according to filtering conditions, select data targets and integrate them into the target format to realize data update and release; the specific process is as follows: data flows out from the Source and enters a FlatMap. The FlatMap decomposes the data into single-point data and flows out. The outflowing data enters the Filter for filtering. Data that does not meet the filtering conditions is discarded. Data that meets the filtering conditions enters the next FlatMap. The FlatMap integrates the data into the target format and then sends it to the Sink; (II) Scheduled Release: Real-time edge data is collected and categorized according to the same time requirements. Then, the data is released on a scheduled basis by performing different partitioning operations according to the parameter list through the partitioning function. The specific process is as follows: Data flows out from the Source and enters the FlatMap. The FlatMap decomposes the data into single-point data and flows out. The Partition function is used to perform different partitioning operations on the data flowing out of the FlatMap according to the parameter list, so that data with the same time frequency enters the same flow. For any flow, the data enters the corresponding TumblingWindow according to the set rules. After the data flowing into the TumblingWindow reaches the set time interval, it is passed into the FlatMap. The FlatMap processes the incoming data and flows out. Merge operation is performed on the data flowing out of the FlatMap in different flows, and the data is sent to the Sink through a new FlatMap. (III) Merging and Optimizing Update and Scheduled Releases: Through the partitioning function, the update release data is passed to the specified partition for update release processing. The scheduled release logic remains unchanged, and invalid data is processed. In this process, update and scheduled release data flow out from the Source and enter the FlatMap. The FlatMap decomposes the data into single-point data and flows out. The Partition function is used to determine whether the data flowing out of the FlatMap meets the parameter requirements of the parameter list. If the parameter requirements corresponding to the current data are not found in the parameter list, the data is passed to the ignoreFilter. If the parameter requirements corresponding to the current data are found in the parameter list, the data that needs to be transformed is passed to the Filter according to the parameter conditions. Data that needs to be sent on a scheduled basis is passed to different flows. Finally, all data is merged and sent to the Sink through a new FlatMap.
2. A real-time edge data computing system based on go-streams, characterized in that, It includes the following modules based on the streaming library go-streams: The real-time data processing module is used to collect real-time edge data, decompose the collected data, perform filtering according to filtering conditions, select data targets and integrate them into the target format, and realize data update and publication. The data timed processing module is used to collect real-time edge data and classify it according to the same time requirements. Then, it performs different partitioning operations based on the parameter list through a partitioning function to realize the timed release of data. The data merging and processing module is used to pass the updated and published data to the specified partition for update and publication through the partitioning function. The timed publication logic remains unchanged, and invalid data is processed. The specific process of data update and release implemented by the real-time data processing module based on the streaming processing library go-streams is as follows: Data flows out from the Source and enters a FlatMap. The FlatMap decomposes the data into single-point data and flows out. The outflowing data enters the Filter for filtering. Data that does not meet the filtering conditions is discarded. Data that meets the filtering conditions enters the next FlatMap. The FlatMap integrates the data into the target format and then sends it to the Sink. The specific process of the data timing processing module based on the streaming processing library go-streams to implement timed data publishing is as follows: Data flows out from the Source and enters the FlatMap. The FlatMap decomposes the data into single-point data and flows out. The Partition function is used to perform different partitioning processing on the data flowing out of the FlatMap according to the parameter list, so that data with the same time frequency enters the same flow. For any flow, the data enters the corresponding TumblingWindow according to the set rules. After the data flowing into the TumblingWindow reaches the set time interval, it is passed into the FlatMap. The FlatMap processes the incoming data and flows out. Merge operation is performed on the outflow data of the FlatMap in different flows, and then it is sent to the Sink through a new FlatMap. The specific process executed by the data merging and processing module based on the streaming processing library go-streams is as follows: Updated and periodically published data flow out from the Source and enter the FlatMap. The FlatMap decomposes the data into single-point data and flows them out. The Partition function is used to determine whether the data flowing out of the FlatMap meets the parameter requirements of the parameter list. If the parameter requirements corresponding to the current data are not found in the parameter list, the data is passed to the ignoreFilter. If the parameter requirements corresponding to the current data are found in the parameter list, the data that needs to be transformed is passed to the Filter according to the parameter conditions. Data that needs to be sent periodically is passed to different flows. Finally, all data is merged and sent to the Sink through a new FlatMap.