Method and apparatus for performing data operations on real-time data streams and data sets

Abstract

The present application discloses a method and device for performing data operations on real-time data streams and data sets, and the specific implementation scheme is: in response to receiving a calculation request, respectively obtaining the data set to be analyzed corresponding to the calculation request and the real-time data stream corresponding to the calculation request; partitioning the data set to be analyzed and the real-time data stream to generate each sub-data set and each sub-data stream; loading each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread; in response to each task node having been loaded, performing a preset calculation operation on the sub-data stream and the sub-data set in the task node to generate the calculation result of the task node; according to the number of task nodes and the calculation result of each task node, determining the final calculation result corresponding to the calculation request. The scheme implements a data calculation method for partitioning real-time data streams and large data sets and then performing associated calculations, which ensures the accuracy of calculations and improves the efficiency of real-time data calculations.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of data processing technology, and in particular to a method and device for performing data operations on real-time data streams and data sets. Background Art

[0002] In real-time computing tasks, there will be a scenario where a certain link needs to filter the entire data stream with unified data rules. The traditional technical solution is to use broadcast technology to copy and distribute a small data set to each task node. Each task node directly reads data from the local during the data calculation process. However, the existing real-time computing framework has restrictions on the size of broadcast variable data. When the size of the data set exceeds a certain scale, the broadcast technology cannot be used to broadcast public variables in real-time computing.

[0003] For the above scenarios, existing real-time computing technology provides a computing solution based on time windows. In real-time stream processing, each message is processed once, but sometimes we need to do some aggregate data processing for different situations, for example: how many SKU inventory statuses have changed multiple times in the past minute. In this case, in scenarios with high real-time requirements, the computing solution based on time windows will cause long data delays and cannot meet real-time requirements. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for performing data operations on real-time data streams and data sets.

[0005] According to a first aspect of the present application, a method for performing data operations on real-time data streams and data sets is provided, the method comprising: in response to receiving a calculation request, respectively obtaining a data set to be analyzed corresponding to the calculation request and a real-time data stream corresponding to the calculation request; partitioning the data set to be analyzed and the real-time data stream to generate respective sub-data sets corresponding to the data set to be analyzed and respective sub-data streams corresponding to the real-time data stream, wherein each sub-data set and each sub-data stream have a one-to-one correspondence based on the same partitioning method; loading each sub-data set and the sub-data stream corresponding to the sub-data set into each task node corresponding to the partition sub-thread; in response to each task node having been loaded, performing a preset calculation operation on the sub-data stream and the sub-data set in the task node to generate a calculation result of the task node; and determining a final calculation result corresponding to the calculation request according to the number of task nodes and the calculation result of each task node.

[0006] In some embodiments, the data set to be analyzed and the real-time data stream are partitioned to generate sub-data sets corresponding to the data set to be analyzed and sub-data streams corresponding to the real-time data stream, including: partitioning one of the data set to be analyzed and the real-time data stream to generate classification identifiers of each sub-data set and each sub-data set corresponding to the data set to be analyzed or each sub-data stream and each sub-data stream corresponding to the real-time data stream; determining each sub-data stream corresponding to each sub-data set or each sub-data stream corresponding to each sub-data set according to the classification identifier of each sub-data set or the classification identifier of each sub-data stream.

[0007] In some embodiments, a real-time data stream is partitioned to generate sub-data streams and classification identifiers of each sub-data stream corresponding to the real-time data stream, including: extracting fields from each message of the real-time data stream to generate partition variables corresponding to each message of the real-time data stream, wherein the extraction principle is to ensure that the partition variables are unique and the values ​​are evenly distributed; performing a hash modulo operation on each message of the real-time data stream according to the partition variables and the number of partitions corresponding to each message of the real-time data stream to obtain each sub-data stream and classification identifiers of each sub-data stream corresponding to each message of the real-time data stream.

[0008] In some embodiments, each sub-data set and the sub-data stream corresponding to the sub-data set are loaded into each task node corresponding to the partitioned sub-thread, including: loading each sub-data set and the sub-data stream corresponding to the sub-data set into each task node corresponding to the partitioned sub-thread of the sub-data set; or loading each sub-data stream and the sub-data set corresponding to the sub-data stream into each task node corresponding to the partitioned sub-thread of the sub-data stream.

[0009] In some embodiments, preset computing operations are performed on the sub-data streams and sub-data sets in the task node to generate computing results of the task node, including: according to the parallelism of the task node, preset computing operations are performed on the sub-data streams and sub-data sets in the task node to generate computing results of the task node, wherein the task node parallelism is used to characterize the number of partition sub-threads executed by each task node.

[0010] In some embodiments, before partitioning the data set to be analyzed and the real-time data stream and generating each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream, it also includes: determining the partitioning method and the number of task nodes according to the size of the data set to be analyzed and the computing resources.

[0011] In some embodiments, the partitioning method includes: the number of partitions, the partitioning algorithm, and the partitioning rules.

[0012] According to a second aspect of the present application, there is provided a device for performing data operations on real-time data streams and data sets, the device comprising: an acquisition unit, configured to, in response to receiving an operation request, respectively acquire a data set to be analyzed corresponding to the operation request and a real-time data stream corresponding to the operation request; a partitioning unit, configured to partition the data set to be analyzed and the real-time data stream, and generate each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream, wherein each sub-data set and each sub-data stream correspond one-to-one based on the same partitioning method; a loading unit, configured to load each sub-data set and the sub-data stream corresponding to the sub-data set into each task node corresponding to the partition sub-thread; a computing unit, configured to, in response to each task node having been loaded, perform a preset computing operation on the sub-data stream and the sub-data set in the task node, and generate a computing result of the task node; an analyzing unit, configured to determine the final computing result corresponding to the computing request according to the number of task nodes and the computing result of each task node.

[0013] In some embodiments, the partitioning unit includes: a partitioning module, configured to partition one of the data set to be analyzed and the real-time data stream, and generate classification identifiers of each sub-data set and each sub-data set corresponding to the data set to be analyzed or classification identifiers of each sub-data stream and each sub-data stream corresponding to the real-time data stream; a determination module, configured to determine each sub-data stream corresponding to each sub-data set or each sub-data stream corresponding to each sub-data set based on the classification identifiers of each sub-data set or the classification identifiers of each sub-data stream.

[0014] In some embodiments, the partitioning unit includes: an extraction module, configured to extract fields from each message of the real-time data stream, and generate a partition variable corresponding to each message of the real-time data stream, wherein the extraction principle is to ensure that the partition variable is unique and the numerical value is evenly distributed; a calculation module, configured to perform a hash modulus operation on each message of the real-time data stream according to the partition variable and the number of partitions corresponding to each message of the real-time data stream, to obtain each sub-data stream corresponding to each message of the real-time data stream and a classification identifier of each sub-data stream.

[0015] In some embodiments, the loading unit includes: a loading module configured to load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partitioned sub-thread of the sub-data set; or to load each sub-data stream and the sub-data set corresponding to the sub-data stream to each task node corresponding to the partitioned sub-thread of the sub-data stream.

[0016] In some embodiments, the computing unit includes: a computing module, which is configured to perform preset computing operations on the sub-data streams and sub-data sets in the task node according to the parallelism of the task node, and generate a computing result of the task node, wherein the parallelism of the task node is used to characterize the number of partition sub-threads executed by each task node.

[0017] In some embodiments, the apparatus further comprises: a determination unit configured to determine a partitioning method and the number of task nodes according to the size of the data set to be analyzed and computing resources.

[0018] According to the third aspect of the present application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner in the first aspect.

[0019] According to a fourth aspect of the present application, the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute a method as described in any implementation manner in the first aspect.

[0020] The technology adopted in the present application is to respond to receiving a computing request, respectively obtain the data set to be analyzed corresponding to the computing request and the real-time data stream corresponding to the computing request, partition the data set to be analyzed and the real-time data stream, generate each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream, wherein each sub-data set and each sub-data stream correspond one-to-one based on the same partitioning method, load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread, and in response to each task node having been loaded, perform preset computing operations on the sub-data stream and the sub-data set in the task node to generate the computing result of the task node, and determine the final computing result corresponding to the computing request according to the number of task nodes and the computing result of each task node. The results show that a method is implemented to partition real-time data streams and large data sets using a unified partitioning method in real-time computing and to perform associated computing on the partitioned data. The problem that the existing real-time computing framework has restrictions on the size of broadcast variable data and cannot support large data sets is solved. The problem that the existing technology will cause long delays and cannot meet the real-time requirements in scenarios with high real-time requirements is solved. The problem that data overflows from the memory when the data volume is large, resulting in reduced cluster applicability, is avoided. This solution can ensure that each data in the real-time data stream can hit the data in the large data set to ensure the accuracy of the calculation, and can reduce the network overhead of each task node through partitioned calculation, thereby improving the efficiency of real-time data calculation.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0023] Figure 1 is a schematic diagram of a first embodiment of a method for performing data operations on real-time data streams and data sets according to the present application;

[0024] Figure 2 It is a scene graph that can implement the method for performing data operations on real-time data streams and data sets according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a second embodiment of a method for performing data operations on real-time data streams and data sets according to the present application;

[0026] Figure 4 is a structural schematic diagram of an embodiment of a device for performing data operations on real-time data streams and data sets according to the present application;

[0027] Figure 5 It is a block diagram of an electronic device used to implement the method for performing data operations on real-time data streams and data sets in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] Figure 1 A schematic diagram 100 of a first embodiment of a method for performing data operations on real-time data streams and data sets according to the present application is shown. The method for performing data operations on real-time data streams and data sets comprises the following steps:

[0031] Step 101 , in response to receiving a computing request, respectively obtaining a to-be-analyzed data set corresponding to the computing request and a real-time data stream corresponding to the computing request.

[0032] In this embodiment, when the execution subject receives a calculation request, the data set to be analyzed corresponding to the calculation request and the real-time data stream corresponding to the calculation request can be obtained from other electronic devices or locally through a wired connection or a wireless connection. The data set to be analyzed can be a data set required for the calculation request in any application scenario, and the real-time data stream is a data stream received in real time during the calculation process. It should be pointed out that the above-mentioned wireless connection method may include but is not limited to 3G, 4G, 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0033] Step 102 : partition the data set to be analyzed and the real-time data stream to generate sub-data sets corresponding to the data set to be analyzed and sub-data streams corresponding to the real-time data stream.

[0034] In this embodiment, the execution subject can use the partitioning method to partition the data set to be analyzed and the real-time data stream obtained in step 101, respectively, to generate each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream. The purpose of the partitioning is to make the size of each generated sub-data set and the size of each generated sub-data stream meet the data size requirements of the data operation, and each sub-data set and each sub-data stream correspond one-to-one based on the same partitioning method, that is, the data in each sub-data set and the data in each sub-data stream are associated. The partitioning method is determined according to the association relationship between the data set to be analyzed and the real-time data stream to be analyzed.

[0035] Step 103, load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread, and in response to each task node having been loaded, perform preset calculation operations on the sub-data stream and sub-data set in the task node to generate a calculation result for the task node.

[0036] In this embodiment, the execution entity can load each associated sub-data set and each sub-data stream into each task node corresponding to the partitioned sub-thread. After each task node is loaded with the sub-data set and sub-data stream, the preset calculation operation is performed on the sub-data stream and sub-data set in the task node to generate the calculation result of the task node.

[0037] Step 104 , determining the final computing result corresponding to the computing request according to the number of task nodes and the computing result of each task node.

[0038] In this embodiment, the execution entity may summarize and analyze the operation results of each task node according to the number of task nodes and the operation result of each task node, and determine the final operation result corresponding to the operation request.

[0039] Continue to see Figure 2 , the method 200 for performing data operations on real-time data streams and data sets in this embodiment runs in an electronic device 201. When the electronic device 201 receives a computing request, the electronic device 201 first obtains the data set to be analyzed corresponding to the computing request and the real-time data stream 202 corresponding to the computing request, and then the electronic device 201 partitions the data set to be analyzed and the real-time data stream, generates each sub-data set corresponding to the data set to be analyzed and each sub-data stream 203 corresponding to the real-time data stream, and then the electronic device 201 loads each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread, and in response to each task node being loaded, performs a preset computing operation on the sub-data stream and the sub-data set in the task node, and generates the computing result 204 of the task node, and finally the electronic device 201 determines the final computing result 205 corresponding to the computing request according to the number of task nodes and the computing result of each task node.

[0040] For example, the scenario is real-time warning of out-of-stock and safety stock in supply chain inventory management. In inventory management, it is necessary to make real-time warnings for out-of-stock and safety stock of a certain range of SKUs. When the data volume of the SKU whitelist that needs to make real-time inventory warnings reaches a certain volume (tens of millions), the real-time computing broadcast technology cannot support it. Using this method, the real-time inventory message and warning whitelist data are partitioned in real time, and the partitioned data are placed in the same task for associated calculations, so that real-time out-of-stock and safety stock warnings can be made for the SKUs in the whitelist.

[0041] To further illustrate, the scenario is real-time oversold prediction for platform merchants. Since the inventory management of platform merchants is relatively rough, the available inventory set in the merchant backend is not necessarily the real physical inventory, which leads to frequent oversold situations of merchants on the platform. In order to manage merchants and strictly control the occurrence of oversold situations, it is necessary to make real-time predictions on whether merchants have oversold behavior. Since the number of merchants managed by the platform reaches hundreds of thousands and the number of SKUs on sale exceeds tens of millions, it is unrealistic to load all SKUs that need to be oversold into the memory of a single node at once when making real-time oversold predictions for each SKU. Through this method, real-time inventory messages and potential oversold items are partitioned for real-time prediction, which can solve the problem that a single perfect prediction node cannot load all oversold SKUs for real-time prediction.

[0042] The method for performing data operations on real-time data streams and data sets provided by the above-mentioned embodiments of the present application adopts the method of, in response to receiving a calculation request, respectively obtaining the data set to be analyzed corresponding to the calculation request and the real-time data stream corresponding to the calculation request, partitioning the data set to be analyzed and the real-time data stream, generating each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream, wherein each sub-data set and each sub-data stream correspond one-to-one based on the same partitioning method, loading each sub-data set and the sub-data stream corresponding to the sub-data set into each task node corresponding to the partition sub-thread, and in response to each task node having been loaded, performing a preset calculation operation on the sub-data stream and the sub-data set in the task node to generate a calculation result of the task node, and according to the number of task nodes and the calculation result of each task node , determine the final calculation result corresponding to the calculation request, and realize a method of partitioning real-time data streams and large data sets using a unified partitioning method in real-time calculations and performing associated calculations on the partitioned data. This solves the problem that the existing real-time calculation framework has restrictions on the size of broadcast variable data and cannot support large data sets. It solves the problem that the existing technology will cause long delays and cannot meet the real-time requirements in scenarios with high real-time requirements. It avoids the problem that when the data volume is large, data will overflow from the memory, resulting in reduced cluster applicability. This solution can ensure that every data in the real-time data stream can hit the data in the large data set to ensure the accuracy of the calculation, and can reduce the network overhead of each task node through partition calculation, thereby improving the efficiency of real-time data calculation.

[0043] Further references Figure 3 , which shows a schematic diagram 300 of a second embodiment of a method for performing data operations on real-time data streams and data sets. The process of the method includes the following steps:

[0044] Step 301 : in response to receiving a computing request, respectively obtaining a data set to be analyzed corresponding to the computing request and a real-time data stream corresponding to the computing request.

[0045] Step 302: Determine the partitioning method and the number of task nodes according to the size of the data set to be analyzed and the computing resources.

[0046] In this embodiment, the execution entity may determine the partitioning method and the number of task nodes according to the size of the data set to be analyzed and the number of computing resources. The partitioning method may include: the number of partitions, the partitioning algorithm and the partitioning rules.

[0047] Step 303 : partition the data set to be analyzed, and generate sub-data sets corresponding to the data set to be analyzed and classification identifiers of the sub-data sets.

[0048] In this embodiment, the execution subject may partition the data set to be analyzed according to the partitioning method determined in step 302, and generate sub-data sets corresponding to the data set to be analyzed and classification identifiers of the sub-data sets.

[0049] Step 304: Determine each sub-data stream corresponding to each sub-data set according to the classification identifier of each sub-data set.

[0050] In this embodiment, the execution entity can use the second partitioning method to partition the real-time data stream according to the classification identifier of each sub-data set to obtain each sub-data stream, and each sub-data stream is associated with each sub-data set based on the classification identifier. The second partitioning method may be different from the partitioning method of the data set to be analyzed.

[0051] In some optional implementations of this embodiment, partitioning the data set to be analyzed and the real-time data stream to generate each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream includes: partitioning the real-time data stream to generate each sub-data stream corresponding to the real-time data stream and the classification identifier of each sub-data stream; determining each sub-data set corresponding to each sub-data stream according to the classification identifier of each sub-data stream. The method of implementing data association partitioning from the perspective of real-time data stream makes data partitioning more flexible and diverse.

[0052] In some optional implementations of this embodiment, the real-time data stream is partitioned to generate each sub-data stream and classification identifier of each sub-data stream corresponding to the real-time data stream, including: extracting fields from each message of the real-time data stream to generate partition variables corresponding to each message of the real-time data stream, wherein the extraction principle is to ensure that the partition variables are unique and the values ​​are evenly distributed; performing a hash modulus operation on each message of the real-time data stream according to the partition variables and the number of partitions corresponding to each message of the real-time data stream to obtain each sub-data stream and classification identifier of each sub-data stream corresponding to each message of the real-time data stream. Processing is performed on each message received in real time to avoid the problem of not being able to meet the requirements of data real-time performance and overflow due to excessive data volume, thereby improving data processing efficiency and data security.

[0053] Step 305: Load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread.

[0054] In this embodiment, the execution entity may load each sub-data set and the sub-data stream corresponding to the sub-data set to the same task node corresponding to the partition sub-thread.

[0055] In some optional implementations of the present embodiment, each sub-data set and the sub-data stream corresponding to the sub-data set are loaded into each task node corresponding to the partition sub-thread, including: loading each sub-data set and the sub-data stream corresponding to the sub-data set into each task node corresponding to the partition sub-thread of the sub-data set; or loading each sub-data stream and the sub-data set corresponding to the sub-data stream into each task node corresponding to the partition sub-thread of the sub-data stream. Performing data operations based on the task nodes of the sub-data set or the sub-data stream itself saves the time for storing data in other task nodes, simplifies the operation process, and improves the operation efficiency.

[0056] Step 306 , performing preset computing operations on the sub-data streams and sub-data sets in the task node according to the parallelism of the task node, and generating a computing result of the task node.

[0057] In this embodiment, the execution subject performs preset computing operations on the sub-data streams and sub-data sets in the task node according to the task node parallelism, and generates the computing result of the task node, wherein the task node parallelism is used to characterize the number of partition sub-threads executed by each task node, and the task node parallelism is set according to the amount of data of each task node. For example, the task node parallelism is 1, which means that each task node executes 1 sub-thread to process data.

[0058] Step 307: Determine the final computing result corresponding to the computing request according to the number of task nodes and the computing result of each task node.

[0059] In this embodiment, the specific operations of steps 301 and 307 are the same as Figure 1 The operations of steps 101 and 104 in the illustrated embodiment are substantially the same and will not be described in detail herein.

[0060] from Figure 3 It can be seen that Figure 1Compared with the corresponding embodiments, the schematic diagram 300 of the method for performing data operations on real-time data streams and data sets in this embodiment adopts, according to the size of the data set to be analyzed and the computing resources, to determine the partitioning method and the number of task nodes, to partition the data set to be analyzed, to generate each sub-data set and the classification identifier of each sub-data set corresponding to the data set to be analyzed, to determine each sub-data stream corresponding to each sub-data set according to the classification identifier of each sub-data set, to load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread, to perform preset operation on the sub-data stream and sub-data set in the task node according to the parallelism of the task node, to generate the operation result of the task node, to realize a real-time data calculation method for partitioning real-time data streams and large data sets and associating the partitioned data, to set the parallelism of task execution according to the actual data amount, to improve the speed of task execution, and to improve the real-time data processing efficiency.

[0061] Further references Figure 4 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for performing data operations on real-time data streams and data sets. The device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0062] like Figure 4 As shown, the device 400 for performing data operations on real-time data streams and data sets in this embodiment includes: an acquisition unit 401, a partitioning unit 402, a loading unit 403, a computing unit 404 and an analyzing unit 405, wherein the acquisition unit is configured to, in response to receiving a computing request, respectively acquire the data set to be analyzed corresponding to the computing request and the real-time data stream corresponding to the computing request; the partitioning unit is configured to partition the data set to be analyzed and the real-time data stream, and generate each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream, wherein each sub-data set and each sub-data stream correspond to each other based on the same partitioning method; the loading unit is configured to load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread; the computing unit is configured to, in response to each task node having been loaded, perform a preset computing operation on the sub-data stream and the sub-data set in the task node to generate the computing result of the task node; the analyzing unit is configured to determine the final computing result corresponding to the computing request according to the number of task nodes and the computing result of each task node.

[0063] In this embodiment, the specific processing of the acquisition unit 401, the partition unit 402, the loading unit 403, the operation unit 404 and the analysis unit 405 of the device 400 for performing data operations on real-time data streams and data sets and the technical effects brought by them can be referred to respectively. Figure 1 The relevant descriptions of steps 101 to 104 in the corresponding embodiment are not repeated here.

[0064] In some optional implementations of the present embodiment, the partitioning unit includes: a partitioning module, configured to partition one of the data set to be analyzed and the real-time data stream, and generate each sub-data set and classification identifier of each sub-data set corresponding to the data set to be analyzed or each sub-data stream and classification identifier of each sub-data stream corresponding to the real-time data stream; a determination module, configured to determine each sub-data stream corresponding to each sub-data set or each sub-data stream corresponding to each sub-data set according to the classification identifier of each sub-data set or the classification identifier of each sub-data stream.

[0065] In some optional implementations of this embodiment, the partitioning unit includes: an extraction module, configured to extract fields from each message of the real-time data stream, and generate partition variables corresponding to each message of the real-time data stream, wherein the extraction principle is to ensure that the partition variables are unique and the numerical values ​​are evenly distributed; a calculation module, configured to perform a hash modulus operation on each message of the real-time data stream according to the partition variables and the number of partitions corresponding to each message of the real-time data stream, to obtain each sub-data stream corresponding to each message of the real-time data stream and a classification identifier of each sub-data stream.

[0066] In some optional implementations of the present embodiment, the loading unit includes: a loading module configured to load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partitioned sub-thread of the sub-data set; or to load each sub-data stream and the sub-data set corresponding to the sub-data stream to each task node corresponding to the partitioned sub-thread of the sub-data stream.

[0067] In some optional implementations of this embodiment, the computing unit includes: a computing module, configured to perform preset computing operations on the sub-data streams and sub-data sets in the task node according to the parallelism of the task node, and generate a computing result of the task node, wherein the task node parallelism is used to characterize the number of partition sub-threads executed by each task node.

[0068] In some optional implementations of this embodiment, the device further includes: a determination unit configured to determine a partitioning method and the number of task nodes according to the size of the data set to be analyzed and computing resources.

[0069] In some optional implementations of this embodiment, the partitioning method includes: the number of partitions, the partitioning algorithm and the partitioning rule.

[0070] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0071] like Figure 5 , is a block diagram of an electronic device for a method of performing data operations on real-time data streams and data sets according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0072] like Figure 5 As shown, the electronic device includes: one or more processors 501, a memory 502, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 501 is taken as an example.

[0073] The memory 502 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform a method for performing data operations on real-time data streams and data sets provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform a method for performing data operations on real-time data streams and data sets provided in the present application.

[0074] The memory 502 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the method for performing data operations on real-time data streams and data sets in the embodiment of the present application (for example, the attached Figure 4The processor 501 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 502, that is, the method for performing data operations on real-time data streams and data sets in the above method embodiment is implemented.

[0075] The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of an electronic device for performing data operations on real-time data streams and data sets, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 502 may optionally include a memory remotely arranged relative to the processor 501, and these remote memories may be connected to the electronic device for performing data operations on real-time data streams and data sets via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0076] The electronic device for performing data operations on real-time data streams and data sets may also include: an input device 503 and an output device 504. The processor 501, the memory 502, the input device 503 and the output device 504 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0077] The input device 503 can receive input digital or character information, and generate key signal input related to user settings and function control of electronic devices for data operations on real-time data streams and data sets, such as input devices such as touch screens, keypads, mice, track pads, touch pads, indicator rods, one or more mouse buttons, trackballs, joysticks, etc. The output device 504 may include a display device, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0078] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0080] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0081] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0082] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0083] According to the technical solution of the embodiment of the present application, in response to receiving a calculation request, respectively obtaining the data set to be analyzed corresponding to the calculation request and the real-time data stream corresponding to the calculation request, partitioning the data set to be analyzed and the real-time data stream, generating respective sub-data sets corresponding to the data set to be analyzed and respective sub-data streams corresponding to the real-time data stream, wherein each sub-data set and each sub-data stream correspond one-to-one based on the same partitioning method, loading each sub-data set and the sub-data stream corresponding to the sub-data set into each task node corresponding to the partition sub-thread, in response to each task node having been loaded, performing preset calculation operations on the sub-data stream and the sub-data set in the task node, generating the calculation result of the task node, and determining the task node corresponding to the calculation request according to the number of task nodes and the calculation result of each task node. The final calculation result realizes a method of partitioning real-time data streams and large data sets using a unified partitioning method in real-time computing and performing associated calculations on the partitioned data. It solves the problem that the existing real-time computing framework has restrictions on the size of broadcast variable data and cannot support large data sets. It also solves the problem that the existing technology will cause long delays and cannot meet real-time requirements in scenarios with high real-time requirements. It avoids the problem that when the data volume is large, data will overflow from the memory, resulting in reduced cluster applicability. This solution can ensure that every data in the real-time data stream can hit the data in the large data set to ensure the accuracy of the calculation, and can reduce the network overhead of each task node through partitioned calculation, thereby improving the efficiency of real-time data calculation.

[0084] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0085] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for performing data operations on real-time data streams and data sets, the method comprising: In response to receiving the operation request, respectively acquiring a data set to be analyzed corresponding to the operation request and a real-time data stream corresponding to the operation request; Partitioning the data set to be analyzed and the real-time data stream to generate sub-data sets corresponding to the data set to be analyzed and sub-data streams corresponding to the real-time data stream, wherein the sub-data sets and the sub-data streams correspond to each other in a one-to-one manner based on the same partitioning method; Load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread; in response to each task node having been loaded, perform a preset operation on the sub-data stream and the sub-data set in the task node to generate the operation result of the task node; According to the number of task nodes and the calculation result of each task node, a final calculation result corresponding to the calculation request is determined.

2. The method according to claim 1, wherein: Partitioning the data set to be analyzed and the real-time data stream to generate sub-data sets corresponding to the data set to be analyzed and sub-data streams corresponding to the real-time data stream includes: Partitioning one of the data set to be analyzed and the real-time data stream, generating each sub-data set and classification identifier of each sub-data set corresponding to the data set to be analyzed or each sub-data stream and classification identifier of each sub-data stream corresponding to the real-time data stream; According to the classification identifiers of the sub-data sets or the classification identifiers of the sub-data streams, the sub-data streams corresponding to the sub-data sets or the sub-data sets corresponding to the sub-data streams are determined.

3. The method according to claim 2, wherein: Partitioning the real-time data stream to generate each sub-data stream corresponding to the real-time data stream and a classification identifier of each sub-data stream includes: Extracting fields from each message of the real-time data stream to generate partition variables corresponding to each message of the real-time data stream, wherein the extraction principle is to ensure that the partition variables are unique and the values ​​are evenly distributed; According to the partition variables and the number of partitions corresponding to each message of the real-time data stream, a hash modulo operation is performed on each message of the real-time data stream to obtain each sub-data stream corresponding to each message of the real-time data stream and a classification identifier of each sub-data stream.

4. The method according to claim 1, wherein: The step of loading each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread includes: Load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partitioned sub-thread of the sub-data set; or load each sub-data stream and the sub-data set corresponding to the sub-data stream to each task node corresponding to the partitioned sub-thread of the sub-data stream.

5. The method according to claim 1, wherein: The performing of a preset operation on the sub-data stream and the sub-data set in the task node to generate the operation result of the task node includes: According to the parallelism of the task node, a preset computing operation is performed on the sub-data stream and the sub-data set in the task node to generate a computing result of the task node, wherein the task node parallelism is used to characterize the number of the partition sub-threads executed by each task node.

6. The method according to claim 1, wherein: Before partitioning the data set to be analyzed and the real-time data stream to generate each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream, the method further includes: The partitioning method and the number of task nodes are determined according to the size of the data set to be analyzed and the computing resources.

7. The method according to claim 1, wherein: The partitioning method includes: the number of partitions, a partitioning algorithm and a partitioning rule.

8. A device for performing data operations on real-time data streams and data sets, the device comprising: an acquisition unit, configured to, in response to receiving a computing request, respectively acquire a data set to be analyzed corresponding to the computing request and a real-time data stream corresponding to the computing request; a partitioning unit configured to partition the data set to be analyzed and the real-time data stream to generate each sub-data set corresponding to the data set to be analyzed and each sub-data stream corresponding to the real-time data stream, wherein each sub-data set and each sub-data stream have a one-to-one correspondence based on the same partitioning method; A loading unit configured to load each sub-data set and a sub-data stream corresponding to the sub-data set to each task node corresponding to the partition sub-thread; A computing unit is configured to, in response to each task node having been loaded, perform a preset computing operation on the sub-data stream and the sub-data set in the task node to generate a computing result of the task node; The analysis unit is configured to determine a final operation result corresponding to the operation request according to the number of task nodes and the operation result of each task node.

9. The device according to claim 8, wherein: The partition unit comprises: a partitioning module configured to partition one of the data set to be analyzed and the real-time data stream, and generate each sub-data set and classification identifier of each sub-data set corresponding to the data set to be analyzed or each sub-data stream and classification identifier of each sub-data stream corresponding to the real-time data stream; The determination module is configured to determine each sub-data set corresponding to each sub-data stream or each sub-data set corresponding to each sub-data stream according to the classification identifier of each sub-data set or the classification identifier of each sub-data stream.

10. The device according to claim 9, wherein: The partition unit comprises: An extraction module is configured to extract fields from each message of the real-time data stream and generate a partition variable corresponding to each message of the real-time data stream, wherein the extraction principle is to ensure that the partition variable is unique and the value distribution is uniform; The computing module is configured to perform a hash modulo operation on each message of the real-time data stream according to the partition variable and the number of partitions corresponding to each message of the real-time data stream, so as to obtain each sub-data stream corresponding to each message of the real-time data stream and a classification identifier of each sub-data stream.

11. The device according to claim 8, wherein: The loading unit comprises: A loading module is configured to load each sub-data set and the sub-data stream corresponding to the sub-data set to each task node corresponding to the partitioned sub-thread of the sub-data set; or to load each sub-data stream and the sub-data set corresponding to the sub-data stream to each task node corresponding to the partitioned sub-thread of the sub-data stream.

12. The device according to claim 8, wherein: The computing unit comprises: The operation module is configured to perform preset operation on the sub-data streams and sub-data sets in the task node according to the parallelism of the task node to generate the operation result of the task node, wherein the parallelism of the task node is used to characterize the number of sub-threads of the partition executed by each task node.

13. The apparatus according to claim 8, further comprising: The determination unit is configured to determine the partitioning method and the number of task nodes according to the size of the data set to be analyzed and the computing resources.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images