Stream data calculation method, calculation device, and electronic device

Abstract

The present application provides a method, a computing device, and an electronic device for computing stream data. The method includes: obtaining the primary key value corresponding to the target stream data and the primary key value corresponding to the dimension table, and obtaining the target stream data primary key value and the dimension table primary key value; determining the first target partition of the dimension table according to the partitioning strategy based on the dimension table primary key value, and obtaining the first mapping relationship between the first target partition and the node, determining the target node corresponding to the first target partition according to the first mapping relationship, and storing the dimension table to the target node; determining the second target partition of the target stream data according to the partitioning strategy based on the target stream data primary key value, and obtaining the second mapping relationship between the second target partition and the node, determining the target node corresponding to the second target partition according to the second mapping relationship, and storing the target stream data to the target node, so as to perform streaming computing on the target stream data and the dimension table on the target node. This solves the problem of decreased stream data computing performance caused by storing a large number of dimension tables on the same node.

Description

Technical Field

[0001] The present application relates to the field of stream data calculation, and in particular to a stream data calculation method, a calculation device, a computer-readable storage medium, and an electronic device. Background Art

[0002] In current stream computing business scenarios, it is common to combine streaming data with dimension tables for calculation. Taking FLINK (an open source stream processing framework developed by Apache) as an example, dimension tables with small data volumes can usually be loaded and stored in the assigned task JVM (Java Virtual Machine, JVM for short). However, static data tables with large data volumes (hereinafter referred to as dimension tables) cannot be directly loaded into the JVM (which will cause memory overflow). Dimension tables are generally placed in a third-party in-memory data database such as HBase (an open source database). During task execution, data is obtained from HBase for calculation. For example, Figure 1 As shown, the computing framework reads data from streaming data sources, partitions it, and sends it across the network to multiple compute nodes (compute nodes 1 to n) for computation or transformation (data processing 1 to n). When a data processing node needs dimension table data, it connects to an HBase instance to retrieve the data. The data is then transferred from the HBase instance nodes (hbase1, hbase2, ..., hbaseN) to the data processing nodes for computation. Finally, the data is merged and output on compute node m. When the dimension table data volume is large, network I / O latency is significant and significantly impacts performance. Furthermore, when the dimension table is large, it is difficult for compute nodes to buffer all the data on a single node due to memory constraints.

[0003] The above method uses traditional in-memory storage, which has poor scalability and can lead to memory bottlenecks when data volumes reach a certain level. Using a typical distributed in-memory database, streaming data cannot be combined with dimension table data over short distances at task nodes, resulting in a certain performance overhead on the network, which becomes more pronounced with larger data volumes. For businesses with high performance requirements, combining a typical in-memory database with streaming data cannot meet business requirements.

[0004] Therefore, a method is needed to solve the problem of performance degradation caused by storing a large amount of dimension table data on the same node during stream data computing. Summary of the Invention

[0005] The main purpose of the present application is to provide a method, a computing device, a computer-readable storage medium and an electronic device for computing stream data, so as to at least solve the problem in the prior art of performance degradation caused by storing a large amount of dimension table data on the same node during the computation of stream data.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for calculating stream data is provided, comprising: obtaining target stream data and a dimension table corresponding to the target stream data, and obtaining a primary key value corresponding to the target stream data and a primary key value corresponding to the dimension table, to obtain the target stream data primary key value and the dimension table primary key value, wherein the target stream data primary key value and the dimension table primary key value are the same; determining a first target partition of the dimension table according to a partitioning strategy based on the dimension table primary key value, and obtaining a first mapping relationship between the first target partition and a node, determining a target node corresponding to the first target partition according to the first mapping relationship, and storing the dimension table to the target node, wherein the first target partition is the dimension table Corresponding partition, the partitioning strategy is a strategy for performing operations on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value; determining the second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, and obtaining the second mapping relationship between the second target partition and the node, determining the target node corresponding to the second target partition according to the second mapping relationship, storing the target stream data in the target node, and performing streaming calculations on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data.

[0007] Optionally, the first target partition of the dimension table is determined according to the partition strategy based on the primary key value of the dimension table, including: performing a cyclic redundancy check on the primary key value of the dimension table to obtain a first check value, and obtaining the number of the first target partitions, performing a modulo operation on the first check value and the number of the first target partitions to obtain a first remainder, and determining that the partition corresponding to the same serial number as the first remainder is the first target partition, wherein each first target partition corresponds to a serial number; the second target partition of the target stream data is determined according to the primary key value of the target stream data according to the partition strategy, including: performing a cyclic redundancy check on the primary key value of the target stream data to obtain a second check value, and obtaining the number of the second target partitions, performing a modulo operation on the second check value and the number of the second target partitions to obtain a second remainder, and determining that the partition corresponding to the same serial number as the second remainder is the second target partition, wherein each second target partition corresponds to a serial number.

[0008] Optionally, the target node includes a target master node and multiple target slave nodes, and storing the target flow data in the target node includes: calculating the busyness of the target master node and the multiple target slave nodes, wherein the level of the busyness indicates the level of the CPU usage and memory usage of the node, and the higher the CPU usage and the memory usage, the higher the busyness; when the busyness of the target master node and the multiple target slave nodes are all less than a preset threshold, storing the target flow data in the target master node or the target slave node with the lowest busyness; when the busyness of any one of the target master node or the target slave node is greater than or equal to the preset threshold, storing the target flow data in other nodes except the target node.

[0009] Optionally, calculating the busyness of the target master node and the plurality of target slave nodes comprises: using the formula The busyness is calculated, where z represents the busyness, a represents the weight of the CPU usage, x represents the CPU usage, b represents the weight of the memory usage, y represents the memory usage, m represents the threshold of the CPU usage, and n represents the threshold of the memory usage.

[0010] Optionally, when the busyness of any one of the target master node or the target slave node is greater than or equal to the preset threshold, the target flow data is stored in other nodes other than the target node, including: calculating the busyness of the master node and multiple slave nodes of the other nodes, and calculating the average busyness of the master node and multiple slave nodes of the other nodes to obtain the average busyness value corresponding to each of the other nodes; comparing the sizes of multiple busyness averages, and storing the target flow data in the other node with the lowest busyness average.

[0011] Optionally, after storing the target flow data in the other node with the lowest average busyness, the method further includes: obtaining the dimension table corresponding to the dimension table primary key value that is the same as the primary key value of the target flow data; and storing the dimension table in the other node with the lowest average busyness.

[0012] Optionally, performing streaming calculations on the target stream data and the dimension table on the target node includes: reading the dimension table corresponding to the target stream data according to the dimension table primary key through a dimension table reading data client, and performing streaming calculations on the target stream data and the dimension table.

[0013] According to another aspect of the present application, a computing device for stream data is provided, comprising: a determining unit for acquiring target stream data and a dimension table corresponding to the target stream data, and determining a primary key value of the target stream data and a primary key value of the dimension table, to obtain the target stream data primary key value and the dimension table primary key value, wherein the target stream data primary key value and the dimension table primary key value are the same; a first storage unit for determining a first target partition of the dimension table according to a partitioning strategy based on the dimension table primary key value, and acquiring a first mapping relationship between the first target partition and a node, determining a target node corresponding to the first target partition according to the first mapping relationship, and storing the dimension table to the target node, wherein the first target partition is the primary key value corresponding to the dimension table. Partitioning, the partitioning strategy is a strategy for performing operations on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value; a second storage unit is used to determine the second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, and obtain a second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculations on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned computing methods.

[0015] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the computing methods described.

[0016] Applying the technical solution of the present application, the target stream data to be stored and its corresponding dimension table are obtained, and then the primary key value is obtained. The target stream data and its dimension table have the same primary key value. The primary key value is calculated through the same partitioning strategy to determine the first target partition of the dimension table and the second target partition of the target stream data. Then, the target node is determined according to the first mapping relationship and the second mapping relationship to store the target stream data in the node where the corresponding dimension table is located. In this way, the distribution of the target stream data and its dimension table data is completed through the partitioning strategy, avoiding the problem of reduced node computing performance caused by storing a large amount of stream data and dimension table data in the same node; at the same time, the target stream data and the dimension table are allocated to the same node for stream data calculation, thus avoiding unnecessary data transmission such as transmitting data from different nodes, so that the node has efficient computing performance. Compared with the method in the prior art where the dimension tables corresponding to a large amount of stream data are stored in the same node, and the method in the process of calculating the stream data requires additional data to be retrieved from the storage end of the dimension table data, resulting in reduced computing performance, the present application realizes the distributed storage and precise scheduling of data, avoids network congestion and the problem of reduced computing performance. Therefore, it can solve the problem of reduced stream data computing performance in the prior art and achieve the effect of precise scheduling and calculation of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0018] Figure 1 A flow chart of a method for calculating stream data in the prior art provided by an embodiment of the present application is shown;

[0019] Figure 2 A hardware structure block diagram of a mobile terminal showing a method for calculating stream data provided by an embodiment of the present application;

[0020] Figure 3 A schematic diagram illustrating a flow chart of a method for calculating stream data provided in an embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram of a storage method for a dimension table in a specific method for calculating stream data provided in an embodiment of the present application is shown;

[0022] Figure 5 A schematic diagram of a method for storing target stream data in a specific method for calculating stream data provided by an embodiment of the present application is shown;

[0023] Figure 6 A schematic diagram of a target flow data storage method considering busyness in a specific flow data calculation method provided by an embodiment of the present application is shown;

[0024] Figure 7 A structural block diagram of a stream data computing device provided in an embodiment of the present application is shown.

[0025] The above drawings include the following reference numerals:

[0026] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0031] FLINK: A framework and distributed processing engine for stateful computation on unbounded and bounded data streams.

[0032] JVM: Java Virtual Machine.

[0033] HBase: A distributed, column-oriented open source database.

[0034] Shuffle: A mechanism for redistributing data.

[0035] Stream data: Stream data is a set of sequential, large-scale, fast, and continuous data sequences.

[0036] Dimension table: a table corresponding to each dimension of data.

[0037] Distributed storage: Generally speaking, it can be divided into two categories: one is disk-based distributed storage, such as HDFS, ELASTICSEARCH, etc., and the other is memory-based distributed storage, such as REDIS, AEROSPIKE, etc., which has the advantages of distribution, high availability, massive storage, and generality.

[0038] Distributed memory database: Memory components such as REDIS and AEROSPIKE have the advantages of being distributed, highly available, high-performance, and supporting SQL-like languages.

[0039] Streaming computing: Components such as SPARK STREAMING and FLINK have the advantages of being distributed, scalable, and having high performance.

[0040] Distributed task scheduling: Taking YARN as an example, it has advantages such as multiple resource scheduling modes, high availability, and scalability.

[0041] As introduced in the background technology, in the prior art, a large amount of dimension table data is stored on the same node during stream data calculation, resulting in performance degradation. To solve the problem of performance degradation caused by stream data being stored on the same node, the embodiments of the present application provide a stream data calculation method, a calculation device, a computer-readable storage medium, and an electronic device.

[0042] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0043] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 2 FIG. 1 is a hardware structure diagram of a mobile terminal for a method for calculating stream data according to an embodiment of the present invention. Figure 2 As shown, the mobile terminal may include one or more ( Figure 2 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 2 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 2More or fewer components than shown, or with Figure 2 Different configurations shown.

[0044] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the method for calculating stream data in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0045] In this embodiment, a method for calculating streaming data running on a mobile terminal, a computer terminal, or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Figure 3 Flowchart of the method for calculating stream data according to an embodiment of the present application. Figure 3 As shown, the method includes the following steps:

[0047] Step S201: Obtain target stream data and a dimension table corresponding to the target stream data, and obtain a primary key value corresponding to the target stream data and a primary key value corresponding to the dimension table, to obtain the target stream data primary key value and the dimension table primary key value, wherein the target stream data primary key value and the dimension table primary key value are the same;

[0048] Specifically, during the stream data calculation process, the corresponding dimension table is generally stored in a specific dimension table data storage table, such as REDIS-based dimension table data. During the dimension table storage process, REDIS writes to the client to obtain the dimension table and primary key value, i.e., the dimension table primary key value. The dimension table data requires a pre-designed REDIS key to ensure that subsequent stream data can be appropriately matched. A routing strategy is then used to calculate the corresponding partition based on the dimension table primary key value. The target stream data also has a corresponding key value, which can be a customer ID, for example. It should be noted that the target stream data and the corresponding dimension table actually represent the same set of data and therefore have the same primary key value. In the following text, "REDIS" and "redis" are different spellings but convey the same meaning. "KEY" and "key" are also different spellings but convey the same meaning.

[0049] Step S202: Determine a first target partition of the dimension table according to the primary key value of the dimension table in accordance with a partitioning strategy, obtain a first mapping relationship between the first target partition and a node, determine a target node corresponding to the first target partition according to the first mapping relationship, and store the dimension table in the target node, wherein the first target partition is a partition corresponding to the dimension table, and the partitioning strategy is a strategy for performing operations on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value.

[0050] Specifically, CRC16(KEY)%16384 is used to calculate the first target partition according to the dimension table primary key value KEY. CRC is a cyclic redundancy check. 16384 represents the number of first target partitions, which can be adjusted according to actual conditions. There are multiple first target partitions, slot0, slot1, ..., slot16383. Each target partition has a corresponding node. The first target partition has a first mapping relationship with the node. For example: the dimension table partition stored in node 1, that is, the first target partition, is slot0...sloti. REDIS instance 1 is executed at node 1. The dimension table partition that can be stored at node 2 is sloti+1...slot2i. REDIS instance 2 is executed at node 2. The dimension table partition that can be stored at node n is slotx...slot16383. REDIS instance n is executed at node n.

[0051] Step S203: Determine the second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, obtain the second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculation on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data.

[0052] Specifically, after the dimension table is partitioned according to the first target partition, in order to more conveniently execute tasks in the node, the target stream data corresponding to the dimension table data in the target node is also stored in the target node. Since the target stream data and the dimension table have the same primary key value, the target stream data is also partitioned according to the primary key value. The above-mentioned CRC16(KEY)%16384 is also used to calculate the partition of the target stream data, namely the second target partition. The first mapping relationship between the second target partition and the node is the same as the second mapping relationship between the first target partition and the node, that is, one stream data and its corresponding dimension table are on the same node. In this way, when calculating the stream data, there is no need to retrieve its corresponding dimension table data from other nodes, eliminating unnecessary network transmission. The above method has the following two requirements: first, the task execution unit needs to be started in advance and resident, and calculate the received tasks at any time; second, the task execution unit and the dimension table storage unit (REDIS instance) must correspond one to one and be deployed on the same node to solve the problem of pulling data across the network.

[0053] Through this embodiment, the target stream data to be stored and its corresponding dimension table are obtained, and then the primary key value is obtained. The target stream data and its dimension table have the same primary key value. The primary key value is calculated through the same partitioning strategy to determine the first target partition of the dimension table and the second target partition of the target stream data. Then, the target node is determined according to the first mapping relationship and the second mapping relationship to store the target stream data in the node where the corresponding dimension table is located. In this way, the distribution of the target stream data and its dimension table data is completed through the partitioning strategy, avoiding the problem of node computing performance degradation caused by storing a large amount of stream data and dimension table data on the same node; at the same time, the target stream data and dimension table are allocated to the same node for stream data calculation, thus avoiding unnecessary data transmission such as transmitting data from different nodes, so that the node has efficient computing performance. Compared with the method in the prior art where the dimension tables corresponding to a large amount of stream data are stored on the same node, and the method in the process of stream data calculation requires additional data to be retrieved from the storage end of the dimension table data, resulting in a degradation of computing performance, the present application realizes the decentralized storage and precise scheduling of data, avoids network congestion and the problem of degradation of computing performance. Therefore, it can solve the problem of reduced stream data computing performance in the prior art and achieve the effect of precise scheduling and calculation of data.

[0054] During the specific implementation process, the above-mentioned step S202 can be implemented through the following steps: performing a cyclic redundancy check on the above-mentioned dimension table primary key value to obtain a first check value, and obtaining the number of the above-mentioned first target partitions, performing a modulo operation on the above-mentioned first check value and the number of the above-mentioned first target partitions to obtain a first remainder, and determining that the partition corresponding to the same serial number as the above-mentioned first remainder is the above-mentioned first target partition, wherein each of the above-mentioned first target partitions corresponds to a serial number; determining the second target partition of the above-mentioned target stream data according to the above-mentioned partitioning strategy based on the above-mentioned target stream data primary key value, including: performing a cyclic redundancy check on the above-mentioned target stream data primary key value to obtain a second check value, and obtaining the number of the above-mentioned second target partitions, performing a modulo operation on the above-mentioned second check value and the number of the above-mentioned second target partitions to obtain a second remainder, and determining that the partition corresponding to the same serial number as the above-mentioned second remainder is the above-mentioned second target partition, wherein each of the above-mentioned second target partitions corresponds to a serial number. This method determines the first target partition and the second target partition by using the partition strategy, i.e., cyclic redundancy check, to measure the primary key value of the dimension table and the primary key value of the target stream data and taking the modulus with the number of partitions. In this way, the dimension table and stream data can be partitioned according to the partition strategy to realize distributed task scheduling.

[0055] Specifically, REDIS stores dimension table data. The REDIS write side can reuse JEDIS functionality, using a CRC16(KEY)%16384 method to partition and assign slots. As mentioned above, dimension table data requires a well-defined REDISKEY to ensure that subsequent stream data can be properly matched. The first target partition can be 16384, or can be set based on the actual application scenario. CRC16(KEY) is the first checksum. CRC16(KEY)%16384 yields the first remainder, for example, 2, which stores the dimension table data in the slot2 partition. Similarly, after the target stream data enters the stream processing device, the slot corresponding to the dimension table data for the target stream data is calculated based on CRC16(KEY)%16384 (where the KEY is present in the target stream data and is the same as the KEY in the dimension table, such as the customer ID). This is the slot2 partition mentioned above. The corresponding node is then found based on the slot and node mapping table maintained by the REDIS client, indicating that the data should be sent to the found node.

[0056] In order to avoid problems such as excessive node load caused by multiple tasks being executed simultaneously in a computing node, in some optional implementations, the above-mentioned target node includes a target master node and multiple target slave nodes, and the above-mentioned step S203 can be implemented by the following steps: Step S2031: Calculate the busyness of the above-mentioned target master node and multiple target slave nodes, wherein the level of the above-mentioned busyness indicates the level of CPU usage and memory usage of the above-mentioned nodes, and the higher the above-mentioned CPU usage and the above-mentioned memory usage, the higher the above-mentioned busyness; Step S2032: When the above-mentioned busyness of the above-mentioned target master node and multiple above-mentioned target slave nodes are less than the preset threshold, the above-mentioned target flow data is stored in the above-mentioned target master node or the above-mentioned target slave node with the lowest busyness; Step S2033: When the above-mentioned busyness of any of the above-mentioned target master node or the above-mentioned target slave node is greater than or equal to the above-mentioned preset threshold, the above-mentioned target flow data is stored in other nodes except the above-mentioned target node. Before storing the target flow data in the target node, this method calculates the busyness of the target node and determines the storage node based on the busyness. This can avoid problems such as too many tasks on the same node, uneven resource utilization, and reduced computing performance.

[0057] During the specific implementation process, each target node includes a master node and multiple slave nodes. In order to reduce the problem of node resource pressure caused by the data belonging to a single node in a certain period of time, that is, the task is assigned to a single node, we also need to combine the node busyness (resources) to determine which computing node the final task (data) should be sent to, that is, calculate the busyness of multiple master nodes and slave nodes in each target node to understand the utilization rate of each node. The busyness is evaluated by the CPU and memory usage. When the busyness of the master node and multiple slave nodes in the target node does not exceed the preset threshold, the target stream data is stored in the node with the lowest busyness (which can be a master node or a slave node) for calculation. When the busyness of any node (which can be a master node or a slave node) is greater than or equal to the preset threshold, it indicates that the node is busy, and the target stream data is stored in other nodes.

[0058] In some optional implementations, the above step S2031 can be implemented by the following steps: The busyness is calculated, where z represents the busyness, a represents the weight of the CPU usage, x represents the CPU usage, b represents the weight of the memory usage, y represents the memory usage, m represents the CPU usage threshold, and n represents the memory usage threshold. This method calculates the busyness based on the CPU and memory usage according to the above formula, which can accurately determine the current state of the target node and determine whether to store the target flow data on the target node.

[0059] Specifically, the above m represents the threshold of CPU usage. For example, when the CPU usage is greater than 80%, m is 80%. n represents the threshold of memory usage. For example, when the memory usage is greater than 60%, m is 60%. e represents a natural number.

[0060] To balance the computational load of each node, step S2033 can be implemented by calculating the busyness of the master node and multiple slave nodes of the other nodes, and calculating the average busyness of the master node and multiple slave nodes of the other nodes to obtain the average busyness corresponding to each of the other nodes; comparing the multiple average busyness values, and storing the target flow data in the other node with the lowest average busyness value. This method calculates the busyness of other nodes and stores the target flow data in the node with the lowest busyness, thereby better utilizing node resources and avoiding situations where one node is overloaded while other nodes are idle.

[0061] During the specific implementation process, calculating the busyness of other nodes includes calculating the busyness of the master node and slave node of each of the other nodes. In order to better compare the busyness between nodes, the busyness of the master node and multiple slave nodes is averaged through the average busyness of each node to obtain the average busyness of the node, and the target flow data is stored in the node with the lowest average busyness to balance the computing load of each node.

[0062] In some optional embodiments, after step S2033, the method further includes the following steps: obtaining the dimension table corresponding to the dimension table primary key value that is the same as the primary key value of the target stream data; and storing the dimension table on the other node with the lowest average busyness. After storing the target stream data on the other node, the method also stores the dimension table data corresponding to the target stream data on the same node. This allows stream data calculations to be performed on the node with the lowest average busyness, avoiding the transmission of dimension table data and improving computational efficiency and performance.

[0063] Specifically, since the corresponding dimension table data needs to be retrieved during the stream data calculation process, as mentioned above, the stream data and the dimension table data are stored in the same target node for calculation. This can avoid the steps of transmitting the dimension table data and directly calculate in the same node, thereby improving the node's computing efficiency and performance. Therefore, after storing the target stream data in other nodes, the dimension table data is stored in this node together.

[0064] To facilitate computation of streaming data, the aforementioned step S203 of performing streaming computation on the target streaming data and the dimension table at the target node can be implemented by: utilizing a dimension table reading client to read the dimension table corresponding to the target streaming data based on the dimension table primary key, and then performing streaming computation on the target streaming data and the dimension table. This method facilitates computation of streaming data by utilizing the dimension table reading client at the target node to read the dimension table data corresponding to the target streaming data and then performing computation.

[0065] During the specific implementation process, when computing stream data in the target node, it is only necessary to read the corresponding dimension table through a dimension table reading client such as a REDIS reading client, so that the dimension table can participate in the computing of the corresponding stream data.

[0066] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for calculating stream data of the present application will be described in detail below with reference to specific embodiments.

[0067] This embodiment relates to a specific method for calculating stream data, including the following steps:

[0068] Step S1: Figure 4 This is a diagram of the storage method for the dimension table. The primary key (primary key value) is obtained from the Redis write client.

[0069] Step S2: Determine the partition of the partition dimension table by the routing strategy (partition strategy), that is, determine the slot (first target partition) by calculating the value of slotNum=CRC16(key)%16384;

[0070] Step S3: The target node is then determined by the first mapping relationship between the first target partition and the node, and the dimension table is stored in the target node. For example, Redis instance 1 is executed on node 1, and the partitions corresponding to node 1 are slot0, ..., sloti; Redis instance 2 is executed on node 2, and the partitions corresponding to node 2 are sloti+1, ..., slot2i; Redis instance n is executed on node n, and the partitions corresponding to node n are slotx, ..., slot16383; then the Redis read client is used to read the dimension table data from the above nodes, and the connection node selection strategy is to prioritize localhost (local data);

[0071] Step S4: Figure 5Schematic diagram of the target stream data storage method. The source data (stream data) is partitioned according to the primary key (target stream data primary key). The partitioning algorithm (partitioning strategy) is the same as the partitioning of the dimension table data, that is, partitionNum = CRC (key) % 16384, thereby determining the partition of the target stream data (the second target partition).

[0072] Step S5: search the slots mapping table (second mapping relationship) according to slotNum, find the node to which the partition data is to be sent, the redis slots mapping table (including the mapping relationship between slotNum and node), and the redis slots mapping table partitionNum is equal to slotNum;

[0073] Step S6: By calculating the node busyness (resources), it is determined which node the final task should be executed on (i.e., the direction of stream data transmission), and the target stream data is sent to the target node. For example, stream data shard 1 is stored in node 1, and the task execution unit in node 1 reads data from Redis instance 1 through Redis. The client obtains data (the dimension table corresponding to the target stream data) through the primary key key, and joins the dimension table with stream data shard 1 for stream computing. Similarly, stream data shard m is stored in node n, and the task execution unit in node n reads data from Redis instance n through Redis. The client obtains data (the dimension table corresponding to the target stream data) through the primary key key, and joins the dimension table with stream data shard m for stream computing. Finally, Redis1, ..., Redisn form a Redis cluster and merge the output.

[0074] Step S7: Figure 6 Schematic diagram of the target stream data storage method considering busyness. Calculate CRC(key)%16384 to obtain slotNum, look up the slot and node mapping table, obtain the node (master node) where the corresponding master slot is located and the slave node (slave node), calculate the busyness of the master and slave nodes, and determine whether the busyness exceeds the threshold. If so, randomly select other nodes as the final task node. If not, select the node with the lowest busyness as the task node.

[0075] The embodiments of the present application also provide a computing device for stream data. It should be noted that the computing device for stream data in the embodiments of the present application can be used to execute the computing method for stream data provided in the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0076] The following introduces the computing device for stream data provided in the embodiments of the present application.

[0077] Figure 7 Schematic diagram of a computing device for stream data according to an embodiment of the present application. Figure 7 As shown, the device includes:

[0078] a determining unit 10 configured to obtain target stream data and a dimension table corresponding to the target stream data, and determine a primary key value of the target stream data and a primary key value of the dimension table, to obtain the primary key value of the target stream data and the primary key value of the dimension table, wherein the primary key value of the target stream data and the primary key value of the dimension table are the same;

[0079] Specifically, during the calculation of stream data, the corresponding dimension table is generally stored in a specific dimension table data storage table, such as the dimension table data stored based on REDIS. During the storage of the dimension table, the dimension table and primary key KEY value, i.e., the dimension table primary key value, are obtained by writing to the client through REDIS. The dimension table data needs to have a pre-designed REDIS KEY to ensure that subsequent stream data can be matched in an appropriate manner. The routing strategy is then used to calculate the corresponding partition based on the dimension table primary key value. The target stream data also has a corresponding KEY value, which can be a customer ID, etc. It should be noted that the above-mentioned target stream data and the corresponding dimension table actually represent the same set of data and therefore have the same primary key value.

[0080] A first storage unit 20 is configured to determine a first target partition of the dimension table according to a partitioning strategy based on the primary key value of the dimension table, obtain a first mapping relationship between the first target partition and a node, determine a target node corresponding to the first target partition according to the first mapping relationship, and store the dimension table in the target node, wherein the first target partition is a partition corresponding to the dimension table, and the partitioning strategy is a strategy for performing operations on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value;

[0081] Specifically, CRC16(KEY)%16384 is used to calculate the first target partition according to the dimension table primary key value KEY. CRC is a cyclic redundancy check. 16384 represents the number of first target partitions, which can be adjusted according to actual conditions. There are multiple first target partitions, slot0, slot1, ..., slot16383. Each target partition has a corresponding node. The first target partition has a first mapping relationship with the node. For example: the dimension table partition stored in node 1, that is, the first target partition, is slot0...sloti. REDIS instance 1 is executed at node 1. The dimension table partition that can be stored at node 2 is sloti+1...slot2i. REDIS instance 2 is executed at node 2. The dimension table partition that can be stored at node n is slotx...slot16383. REDIS instance n is executed at node n.

[0082] The second storage unit 30 is used to determine the second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, and obtain the second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculation on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data.

[0083] Specifically, after the dimension table is partitioned according to the first target partition, in order to more conveniently execute tasks in the node, the target flow data corresponding to the dimension table data in the target node is also stored in the target node. Since the target flow data and the dimension table have the same primary key value, the target flow data is also partitioned according to the primary key value. The above-mentioned CRC16(KEY)%16384 is also used to calculate the partition of the target flow data, namely the second target partition. The first mapping relationship between the second target partition and the node is the same as the second mapping relationship between the first target partition and the node, that is, one flow data and its corresponding dimension table are on the same node. In this way, when the flow data is calculated, there is no need to retrieve its corresponding dimension table data from other nodes, eliminating unnecessary network transmission. The above-mentioned device has the following two requirements: first, the task execution unit needs to be started in advance and resident, and calculate the received tasks at any time; second, the task execution unit and the dimension table storage unit (REDIS instance) are required to correspond one to one and be deployed on the same node to solve the problem of pulling data across the network.

[0084] Through this embodiment, the target stream data to be stored and its corresponding dimension table are obtained, and then the primary key value is obtained. The target stream data and its dimension table have the same primary key value. The primary key value is calculated using the same partitioning strategy to determine the first target partition of the dimension table and the second target partition of the target stream data. Then, the target node is determined based on the first mapping relationship and the second mapping relationship to store the target stream data in the node where the corresponding dimension table is located. In this way, the target stream data and its dimension table data are distributed through the partitioning strategy, avoiding the problem of node computing performance degradation caused by storing a large amount of stream data and dimension table data on the same node; at the same time, the target stream data and dimension table are distributed to the same node for stream data calculation, thus avoiding unnecessary data transmission such as transmitting data from different nodes, so that the node has efficient computing performance. Compared with the device in the prior art, in which the dimension tables corresponding to a large amount of stream data are stored on the same node, and in the process of stream data calculation, data needs to be retrieved from the storage end of the dimension table data, resulting in a degradation of computing performance, the present application realizes decentralized storage and precise scheduling of data, avoids network congestion and the problem of degradation of computing performance. Therefore, it can solve the problem of reduced stream data computing performance in the prior art and achieve the effect of precise data scheduling and calculation.

[0085] In a specific implementation process, the first storage unit includes a first determination module for performing a cyclic redundancy check on the primary key value of the dimension table to obtain a first check value, and obtaining the number of the first target partitions, performing a modulo operation on the first check value and the number of the first target partitions to obtain a first remainder, and determining the partition corresponding to the same sequence number as the first remainder as the first target partition, wherein each of the first target partitions corresponds to a sequence number; the second storage unit includes a second determination module for performing a cyclic redundancy check on the primary key value of the target stream data to obtain a second check value, and obtaining the number of the second target partitions, performing a modulo operation on the second check value and the number of the second target partitions to obtain a second remainder, and determining the partition corresponding to the same sequence number as the second remainder as the second target partition, wherein each of the second target partitions corresponds to a sequence number. The device determines the first target partition and the second target partition by performing a cyclic redundancy check on the primary key value of the dimension table and the primary key value of the target stream data according to the partitioning strategy, i.e., performing a modulo operation on the primary key value of the dimension table and the primary key value of the target stream data. In this way, the dimension table and stream data can be partitioned according to the partitioning strategy to achieve distributed task scheduling.

[0086] Specifically, REDIS stores dimension table data. The REDIS write side can reuse JEDIS functionality, using a CRC16(KEY)%16384 method to partition and assign slots. As mentioned above, dimension table data requires a well-defined REDISKEY to ensure that subsequent stream data can be properly matched. The first target partition can be 16384, or can be set based on the actual application scenario. CRC16(KEY) is the first checksum. CRC16(KEY)%16384 yields the first remainder, for example, 2, which stores the dimension table data in the slot2 partition. Similarly, after the target stream data enters the stream processing device, the slot corresponding to the dimension table data for the target stream data is calculated based on CRC16(KEY)%16384 (where the KEY is present in the target stream data and is the same as the KEY in the dimension table, such as the customer ID). This is the slot2 partition mentioned above. The corresponding node is then found based on the slot and node mapping table maintained by the REDIS client, indicating that the data should be sent to the found node.

[0087] In order to avoid the problem that multiple tasks are executed simultaneously in a computing node, causing excessive node load, etc., in some optional embodiments, the above-mentioned target node includes a target master node and multiple target slave nodes, and the above-mentioned second storage unit includes a first computing module, a first storage module and a second storage module, wherein the first computing module is used to calculate the busyness of the above-mentioned target master node and the multiple target slave nodes, wherein the level of the above-mentioned busyness indicates the level of the CPU usage and the memory usage of the above-mentioned node, and the higher the above-mentioned CPU usage and the above-mentioned memory usage, the higher the above-mentioned busyness; the first storage module is used to store the above-mentioned target flow data to the above-mentioned target master node or the above-mentioned target slave node with the lowest busyness when the above-mentioned busyness of the above-mentioned target master node and the multiple above-mentioned target slave nodes are all less than a preset threshold; the second storage module is used to store the above-mentioned target flow data to other nodes except the above-mentioned target node when the above-mentioned busyness of any of the above-mentioned target master node or the above-mentioned target slave node is greater than or equal to the above-mentioned preset threshold. Before storing the target flow data in the target node, the device calculates the busyness of the target node and determines the storage node according to the busyness. This can avoid problems such as too many tasks on the same node, uneven resource utilization, and reduced computing performance.

[0088] During the specific implementation process, each target node includes a master node and multiple slave nodes. In order to reduce the problem of node resource pressure caused by the data belonging to a single node in a certain period of time, that is, the task is assigned to a single node, we also need to combine the node busyness (resources) to determine which computing node the final task (data) should be sent to, that is, calculate the busyness of multiple master nodes and slave nodes in each target node to understand the utilization rate of each node. The busyness is evaluated by the CPU and memory usage. When the busyness of the master node and multiple slave nodes in the target node does not exceed the preset threshold, the target stream data is stored in the node with the lowest busyness (which can be a master node or a slave node) for calculation. When the busyness of any node (which can be a master node or a slave node) is greater than or equal to the preset threshold, it indicates that the node is busy, and the target stream data is stored in other nodes.

[0089] In some optional embodiments, the first calculation module includes a first calculation submodule for calculating The busyness is calculated, where z represents the busyness, a represents the weight of the CPU usage, x represents the CPU usage, b represents the weight of the memory usage, y represents the memory usage, m represents the threshold of the CPU usage, and n represents the threshold of the memory usage. The device calculates the busyness according to the above formula based on the CPU and memory usage, thereby accurately determining the current state of the target node and determining whether to store the target flow data in the target node.

[0090] Specifically, the above m represents the threshold of CPU usage. For example, when the CPU usage is greater than 80%, m is 80%. n represents the threshold of memory usage. For example, when the memory usage is greater than 60%, m is 60%. e represents a natural number.

[0091] To balance the computational load of each node, the second storage module includes a second computing submodule and a third storage submodule. The second computing submodule is configured to calculate the busyness of the master node and multiple slave nodes of the aforementioned other nodes, and to calculate the average busyness of the master node and multiple slave nodes of the aforementioned other nodes, thereby obtaining the average busyness corresponding to each of the aforementioned other nodes. The third storage submodule is configured to compare the multiple average busyness values ​​and store the target flow data in the aforementioned other node with the lowest average busyness value. This device calculates the busyness of other nodes and stores the target flow data in the node with the lowest busyness value. This allows for better utilization of node resources and avoids situations where one node is overloaded while other nodes are idle.

[0092] During the specific implementation process, calculating the busyness of other nodes includes calculating the busyness of the master node and slave node of each of the other nodes. In order to better compare the busyness between nodes, the busyness of the master node and multiple slave nodes is averaged through the average busyness of each node to obtain the average busyness of the node, and the target flow data is stored in the node with the lowest average busyness to balance the computing load of each node.

[0093] In some optional embodiments, the apparatus further includes an acquisition unit and a third storage unit, wherein the acquisition unit is configured to acquire the dimension table corresponding to the dimension table primary key value identical to the target stream data primary key value; and the third storage unit is configured to store the dimension table in the other node having the lowest average busyness. After storing the target stream data in the other node, the apparatus also stores the dimension table data corresponding to the target stream data in the same node. This allows stream data calculations to be performed on the node with the lowest average busyness, avoiding the transmission of dimension table data and improving computational efficiency and performance.

[0094] Specifically, since the corresponding dimension table data needs to be retrieved during the stream data calculation process, as mentioned above, the stream data and the dimension table data are stored in the same target node for calculation. This can avoid the steps of transmitting the dimension table data and directly calculate in the same node, thereby improving the node's computing efficiency and performance. Therefore, after storing the target stream data in other nodes, the dimension table data is stored in this node together.

[0095] To facilitate computation on stream data, the second storage unit further includes a second computation module configured to read the dimension table corresponding to the target stream data based on the dimension table primary key via a dimension table reading client, and perform streaming computation on the target stream data and the dimension table. This device, in the target node, reads the dimension table data corresponding to the target stream data via the dimension table reading client and then performs computation, thus facilitating computation on stream data.

[0096] During the specific implementation process, when computing stream data in the target node, it is only necessary to read the corresponding dimension table through a dimension table reading client such as a REDIS reading client, so that the dimension table can participate in the computing of the corresponding stream data.

[0097] The stream data computing device includes a processor and a memory. The determination unit, first storage unit, and second storage unit are stored in the memory as program units. The processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0098] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be configured, and the kernel parameters can be adjusted to achieve precise scheduling of stream data and dimension table data.

[0099] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0100] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for calculating stream data.

[0101] Specifically, the calculation method of stream data includes:

[0102] Step S201: Obtain target stream data and a dimension table corresponding to the target stream data, and obtain a primary key value corresponding to the target stream data and a primary key value corresponding to the dimension table, to obtain the target stream data primary key value and the dimension table primary key value, wherein the target stream data primary key value and the dimension table primary key value are the same;

[0103] Specifically, during the calculation of stream data, the corresponding dimension table is generally stored in a specific dimension table data storage table, such as the dimension table data stored based on REDIS. During the storage of the dimension table, the dimension table and primary key KEY value, i.e., the dimension table primary key value, are obtained by writing to the client through REDIS. The dimension table data needs to have a pre-designed REDIS KEY to ensure that subsequent stream data can be matched in an appropriate manner. The routing strategy is then used to calculate the corresponding partition based on the dimension table primary key value. The target stream data also has a corresponding KEY value, which can be a customer ID, etc. It should be noted that the above-mentioned target stream data and the corresponding dimension table actually represent the same set of data and therefore have the same primary key value.

[0104] Step S202: Determine a first target partition of the dimension table according to the primary key value of the dimension table in accordance with a partitioning strategy, obtain a first mapping relationship between the first target partition and a node, determine a target node corresponding to the first target partition according to the first mapping relationship, and store the dimension table in the target node, wherein the first target partition is a partition corresponding to the dimension table, and the partitioning strategy is a strategy for performing operations on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value.

[0105] Specifically, CRC16(KEY)%16384 is used to calculate the first target partition according to the dimension table primary key value KEY. CRC is a cyclic redundancy check. 16384 represents the number of first target partitions, which can be adjusted according to actual conditions. There are multiple first target partitions, slot0, slot1, ..., slot16383. Each target partition has a corresponding node. The first target partition has a first mapping relationship with the node. For example: the dimension table partition stored in node 1, that is, the first target partition, is slot0...sloti. REDIS instance 1 is executed at node 1. The dimension table partition that can be stored at node 2 is sloti+1...slot2i. REDIS instance 2 is executed at node 2. The dimension table partition that can be stored at node n is slotx...slot16383. REDIS instance n is executed at node n.

[0106] Step S203: Determine the second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, obtain the second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculation on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data.

[0107] Specifically, after the dimension table is partitioned according to the first target partition, in order to more conveniently execute tasks in the node, the target stream data corresponding to the dimension table data in the target node is also stored in the target node. Since the target stream data and the dimension table have the same primary key value, the target stream data is also partitioned according to the primary key value. The above-mentioned CRC16(KEY)%16384 is also used to calculate the partition of the target stream data, namely the second target partition. The first mapping relationship between the second target partition and the node is the same as the second mapping relationship between the first target partition and the node, that is, one stream data and its corresponding dimension table are on the same node. In this way, when calculating the stream data, there is no need to retrieve its corresponding dimension table data from other nodes, eliminating unnecessary network transmission. The above method has the following two requirements: first, the task execution unit needs to be started in advance and resident, and calculate the received tasks at any time; second, the task execution unit and the dimension table storage unit (REDIS instance) must correspond one to one and be deployed on the same node to solve the problem of pulling data across the network.

[0108] Optionally, the first target partition of the dimension table is determined according to the partition strategy based on the primary key value of the dimension table, including: performing a cyclic redundancy check on the primary key value of the dimension table to obtain a first check value, and obtaining the number of the first target partitions, performing a modulo operation on the first check value and the number of the first target partitions to obtain a first remainder, and determining that the partition corresponding to the same serial number as the first remainder is the first target partition, wherein each of the first target partitions corresponds to a serial number; the second target partition of the target stream data is determined according to the primary key value of the target stream data according to the partition strategy, including: performing a cyclic redundancy check on the primary key value of the target stream data to obtain a second check value, and obtaining the number of the second target partitions, performing a modulo operation on the second check value and the number of the second target partitions to obtain a second remainder, and determining that the partition corresponding to the same serial number as the second remainder is the second target partition, wherein each of the second target partitions corresponds to a serial number.

[0109] Optionally, the target node includes a target master node and multiple target slave nodes, and storing the target flow data in the target node includes: calculating the busyness of the target master node and the multiple target slave nodes, wherein the level of the busyness indicates the level of CPU usage and memory usage of the node, and the higher the CPU usage and the memory usage, the higher the busyness; when the busyness of the target master node and the multiple target slave nodes are less than a preset threshold, storing the target flow data in the target master node or the target slave node with the lowest busyness; when the busyness of any target master node or the target slave node is greater than or equal to the preset threshold, storing the target flow data in other nodes except the target node.

[0110] Optionally, calculating the busyness of the target master node and the plurality of target slave nodes includes: using the formula The busyness is calculated, where z represents the busyness, a represents the weight of the CPU usage, x represents the CPU usage, b represents the weight of the memory usage, y represents the memory usage, m represents the CPU usage threshold, and n represents the memory usage threshold.

[0111] Optionally, when the busyness of any of the above-mentioned target master nodes or the above-mentioned target slave nodes is greater than or equal to the above-mentioned preset threshold, the above-mentioned target flow data is stored in other nodes other than the above-mentioned target node, including: calculating the busyness of the master node and multiple slave nodes of the above-mentioned other nodes, and calculating the average busyness of the master node and multiple slave nodes of the above-mentioned other nodes, to obtain the average busyness value corresponding to each of the above-mentioned other nodes; comparing the sizes of multiple above-mentioned busyness average values, and storing the above-mentioned target flow data in the above-mentioned other nodes with the lowest above-mentioned busyness average value.

[0112] Optionally, after storing the target flow data in the other node with the lowest average busyness, the method further includes: obtaining the dimension table corresponding to the dimension table primary key value that is the same as the target flow data primary key value; and storing the dimension table in the other node with the lowest average busyness.

[0113] Optionally, performing streaming calculations on the target stream data and the dimension table on the target node includes: reading the dimension table corresponding to the target stream data according to the dimension table primary key through the dimension table reading data client, and performing streaming calculations on the target stream data and the dimension table.

[0114] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:

[0115] Step S201: Obtain target stream data and a dimension table corresponding to the target stream data, and obtain a primary key value corresponding to the target stream data and a primary key value corresponding to the dimension table, to obtain the target stream data primary key value and the dimension table primary key value, wherein the target stream data primary key value and the dimension table primary key value are the same;

[0116] Step S202: Determine a first target partition of the dimension table according to the primary key value of the dimension table in accordance with a partitioning strategy, obtain a first mapping relationship between the first target partition and a node, determine a target node corresponding to the first target partition according to the first mapping relationship, and store the dimension table in the target node, wherein the first target partition is a partition corresponding to the dimension table, and the partitioning strategy is a strategy for performing operations on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value.

[0117] Step S203: Determine the second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, obtain the second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculation on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data.

[0118] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0119] Optionally, the first target partition of the dimension table is determined according to the partition strategy based on the primary key value of the dimension table, including: performing a cyclic redundancy check on the primary key value of the dimension table to obtain a first check value, and obtaining the number of the first target partitions, performing a modulo operation on the first check value and the number of the first target partitions to obtain a first remainder, and determining that the partition corresponding to the same serial number as the first remainder is the first target partition, wherein each of the first target partitions corresponds to a serial number; the second target partition of the target stream data is determined according to the primary key value of the target stream data according to the partition strategy, including: performing a cyclic redundancy check on the primary key value of the target stream data to obtain a second check value, and obtaining the number of the second target partitions, performing a modulo operation on the second check value and the number of the second target partitions to obtain a second remainder, and determining that the partition corresponding to the same serial number as the second remainder is the second target partition, wherein each of the second target partitions corresponds to a serial number.

[0120] Optionally, the target node includes a target master node and multiple target slave nodes, and storing the target flow data in the target node includes: calculating the busyness of the target master node and the multiple target slave nodes, wherein the level of the busyness indicates the level of CPU usage and memory usage of the node, and the higher the CPU usage and the memory usage, the higher the busyness; when the busyness of the target master node and the multiple target slave nodes are less than a preset threshold, storing the target flow data in the target master node or the target slave node with the lowest busyness; when the busyness of any target master node or the target slave node is greater than or equal to the preset threshold, storing the target flow data in other nodes except the target node.

[0121] Optionally, calculating the busyness of the target master node and the plurality of target slave nodes includes: using the formula The busyness is calculated, where z represents the busyness, a represents the weight of the CPU usage, x represents the CPU usage, b represents the weight of the memory usage, y represents the memory usage, m represents the CPU usage threshold, and n represents the memory usage threshold.

[0122] Optionally, when the busyness of any of the above-mentioned target master nodes or the above-mentioned target slave nodes is greater than or equal to the above-mentioned preset threshold, the above-mentioned target flow data is stored in other nodes other than the above-mentioned target node, including: calculating the busyness of the master node and multiple slave nodes of the above-mentioned other nodes, and calculating the average busyness of the master node and multiple slave nodes of the above-mentioned other nodes, to obtain the average busyness value corresponding to each of the above-mentioned other nodes; comparing the sizes of multiple above-mentioned busyness average values, and storing the above-mentioned target flow data in the above-mentioned other nodes with the lowest above-mentioned busyness average value.

[0123] Optionally, after storing the target flow data in the other node with the lowest average busyness, the method further includes: obtaining the dimension table corresponding to the dimension table primary key value that is the same as the target flow data primary key value; and storing the dimension table in the other node with the lowest average busyness.

[0124] Optionally, performing streaming calculations on the target stream data and the dimension table on the target node includes: reading the dimension table corresponding to the target stream data according to the dimension table primary key through the dimension table reading data client, and performing streaming calculations on the target stream data and the dimension table.

[0125] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0126] Step S201: Obtain target stream data and a dimension table corresponding to the target stream data, and obtain a primary key value corresponding to the target stream data and a primary key value corresponding to the dimension table, to obtain the target stream data primary key value and the dimension table primary key value, wherein the target stream data primary key value and the dimension table primary key value are the same;

[0127] Step S202: Determine a first target partition of the dimension table according to the primary key value of the dimension table in accordance with a partitioning strategy, obtain a first mapping relationship between the first target partition and a node, determine a target node corresponding to the first target partition according to the first mapping relationship, and store the dimension table in the target node, wherein the first target partition is a partition corresponding to the dimension table, and the partitioning strategy is a strategy for performing operations on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value.

[0128] Step S203: Determine the second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, obtain the second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculation on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data.

[0129] Optionally, the first target partition of the dimension table is determined according to the partition strategy based on the primary key value of the dimension table, including: performing a cyclic redundancy check on the primary key value of the dimension table to obtain a first check value, and obtaining the number of the first target partitions, performing a modulo operation on the first check value and the number of the first target partitions to obtain a first remainder, and determining that the partition corresponding to the same serial number as the first remainder is the first target partition, wherein each of the first target partitions corresponds to a serial number; the second target partition of the target stream data is determined according to the primary key value of the target stream data according to the partition strategy, including: performing a cyclic redundancy check on the primary key value of the target stream data to obtain a second check value, and obtaining the number of the second target partitions, performing a modulo operation on the second check value and the number of the second target partitions to obtain a second remainder, and determining that the partition corresponding to the same serial number as the second remainder is the second target partition, wherein each of the second target partitions corresponds to a serial number.

[0130] Optionally, the target node includes a target master node and multiple target slave nodes, and storing the target flow data in the target node includes: calculating the busyness of the target master node and the multiple target slave nodes, wherein the level of the busyness indicates the level of CPU usage and memory usage of the node, and the higher the CPU usage and the memory usage, the higher the busyness; when the busyness of the target master node and the multiple target slave nodes are less than a preset threshold, storing the target flow data in the target master node or the target slave node with the lowest busyness; when the busyness of any target master node or the target slave node is greater than or equal to the preset threshold, storing the target flow data in other nodes except the target node.

[0131] Optionally, calculating the busyness of the target master node and the plurality of target slave nodes includes: using the formula The busyness is calculated, where z represents the busyness, a represents the weight of the CPU usage, x represents the CPU usage, b represents the weight of the memory usage, y represents the memory usage, m represents the CPU usage threshold, and n represents the memory usage threshold.

[0132] Optionally, when the busyness of any of the above-mentioned target master nodes or the above-mentioned target slave nodes is greater than or equal to the above-mentioned preset threshold, the above-mentioned target flow data is stored in other nodes other than the above-mentioned target node, including: calculating the busyness of the master node and multiple slave nodes of the above-mentioned other nodes, and calculating the average busyness of the master node and multiple slave nodes of the above-mentioned other nodes, to obtain the average busyness value corresponding to each of the above-mentioned other nodes; comparing the sizes of multiple above-mentioned busyness average values, and storing the above-mentioned target flow data in the above-mentioned other nodes with the lowest above-mentioned busyness average value.

[0133] Optionally, after storing the target flow data in the other node with the lowest average busyness, the method further includes: obtaining the dimension table corresponding to the dimension table primary key value that is the same as the target flow data primary key value; and storing the dimension table in the other node with the lowest average busyness.

[0134] Optionally, performing streaming calculations on the target stream data and the dimension table on the target node includes: reading the dimension table corresponding to the target stream data according to the dimension table primary key through the dimension table reading data client, and performing streaming calculations on the target stream data and the dimension table.

[0135] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0136] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0141] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0142] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0144] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0145] 1) In the calculation method of stream data of the present application, the target stream data to be stored and its corresponding dimension table are obtained, and then the primary key value is obtained. The target stream data and its dimension table have the same primary key value. The primary key value is calculated through the same partitioning strategy to determine the first target partition of the dimension table and the second target partition of the target stream data. Then, the target node is determined according to the first mapping relationship and the second mapping relationship to store the target stream data in the node where the corresponding dimension table is located. In this way, the distribution of the target stream data and its dimension table data is completed through the partitioning strategy, avoiding the problem of node computing performance degradation caused by storing a large amount of stream data and dimension table data in the same node; at the same time, the target stream data and the dimension table are allocated to the same node for stream data calculation, thus avoiding unnecessary data transmission such as transmitting data from different nodes, so that the node has efficient computing performance. Compared with the method in the prior art where the dimension tables corresponding to a large amount of stream data are stored in the same node, and the method in the process of stream data calculation requires additional data to be retrieved from the storage end of the dimension table data, resulting in a degradation of computing performance, the present application realizes the decentralized storage and precise scheduling of data, avoids network congestion and the problem of degradation of computing performance. Therefore, it can solve the problem of reduced stream data computing performance in the prior art and achieve the effect of precise scheduling and calculation of data.

[0146] 2) In the stream data computing device of the present application, the target stream data to be stored and its corresponding dimension table are obtained, and then the primary key value is obtained. The target stream data and its dimension table have the same primary key value. The primary key value is calculated using the same partitioning strategy to determine the first target partition of the dimension table and the second target partition of the target stream data. Then, the target node is determined based on the first mapping relationship and the second mapping relationship to store the target stream data in the node where its corresponding dimension table is located. In this way, the distribution of the target stream data and its dimension table data is completed through the partitioning strategy, avoiding problems such as the degradation of node computing performance caused by the storage of a large amount of stream data and dimension table data in the same node; at the same time, the target stream data and the dimension table are allocated to the same node for stream data calculation, thus avoiding unnecessary data transmission such as transmitting data from different nodes, so that the node has efficient computing performance. Compared with the prior art, in which dimension tables corresponding to a large amount of stream data are stored in the same node, and in which data needs to be retrieved from the storage end of the dimension table data during the calculation of the stream data, resulting in a decrease in computing performance, the present application realizes decentralized storage and precise scheduling of data, avoiding network congestion and the problem of decreased computing performance. Therefore, it can solve the problem of decreased computing performance of stream data in the prior art and achieve the effect of precise scheduling and calculation of data.

[0147] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for calculating stream data, characterized in that: include: Obtain target stream data and a dimension table corresponding to the target stream data, and obtain a primary key value corresponding to the target stream data and a primary key value corresponding to the dimension table, to obtain the target stream data primary key value and the dimension table primary key value, wherein the target stream data primary key value and the dimension table primary key value are the same; Determine a first target partition of the dimension table according to a partitioning strategy based on the primary key value of the dimension table, obtain a first mapping relationship between the first target partition and a node, determine a target node corresponding to the first target partition according to the first mapping relationship, and store the dimension table in the target node, wherein the first target partition is a partition corresponding to the dimension table, and the partitioning strategy is a strategy of performing an operation on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value; Determine a second target partition of the target stream data according to the partition strategy based on the primary key value of the target stream data, obtain a second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculation on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data; Determining a first target partition of the dimension table according to a partitioning strategy based on the dimension table primary key value includes: Performing a cyclic redundancy check on the primary key value of the dimension table to obtain a first check value, obtaining the number of the first target partitions, performing a modulo operation on the first check value and the number of the first target partitions to obtain a first remainder, and determining that the partition corresponding to the same sequence number as the first remainder is the first target partition, wherein each first target partition corresponds to a sequence number; Determining a second target partition of the target stream data according to the partition strategy based on the primary key value of the target stream data includes: Perform a cyclic redundancy check on the primary key value of the target stream data to obtain a second check value, and obtain the number of the second target partitions. Perform a modulo operation on the second check value and the number of the second target partitions to obtain a second remainder, and determine that the partition corresponding to the same serial number as the second remainder is the second target partition, where each second target partition corresponds to a serial number.

2. The calculation method according to claim 1, characterized in that The target node includes a target master node and a plurality of target slave nodes, and storing the target stream data in the target node includes: Calculating the busyness of the target master node and the plurality of target slave nodes, wherein the busyness indicates the CPU usage and memory usage of the nodes, and the higher the CPU usage and the memory usage, the higher the busyness; When the busyness of the target master node and the plurality of target slave nodes are all less than a preset threshold, storing the target flow data in the target master node or the target slave node with the lowest busyness; When the busyness of any one of the target master node or the target slave node is greater than or equal to the preset threshold, the target flow data is stored in other nodes except the target node.

3. The calculation method according to claim 2, characterized in that Calculating the busyness of the target master node and the plurality of target slave nodes, including: By formula The busyness is calculated, where z represents the busyness, a represents the weight of the CPU usage, x represents the CPU usage, b represents the weight of the memory usage, y represents the memory usage, m represents the threshold of the CPU usage, and n represents the threshold of the memory usage.

4. The calculation method according to claim 2, characterized in that When the busyness of any one of the target master node or the target slave node is greater than or equal to the preset threshold, storing the target flow data to other nodes except the target node includes: Calculating the busyness of the master node and multiple slave nodes of the other nodes, and calculating the average busyness of the master node and multiple slave nodes of the other nodes, to obtain the average busyness corresponding to each of the other nodes; The sizes of the multiple busyness average values ​​are compared, and the target flow data is stored in the other node with the lowest busyness average value.

5. The calculation method according to claim 4, characterized in that After storing the target flow data in the other node with the lowest average busyness, the method further includes: Acquire the dimension table corresponding to the dimension table primary key value that is the same as the target stream data primary key value; The dimension table is stored in the other node with the lowest average busyness.

6. The calculation method according to claim 1, characterized in that Performing stream computing on the target stream data and the dimension table on the target node includes: The client reads the dimension table corresponding to the target stream data according to the dimension table primary key through the dimension table, and performs stream calculation on the target stream data and the dimension table.

7. A computing device for stream data, characterized in that: include: a determining unit, configured to obtain target stream data and a dimension table corresponding to the target stream data, and determine a primary key value of the target stream data and a primary key value of the dimension table, to obtain the primary key value of the target stream data and the primary key value of the dimension table, wherein the primary key value of the target stream data and the primary key value of the dimension table are the same; A first storage unit is configured to determine a first target partition of the dimension table according to a partitioning strategy based on a primary key value of the dimension table, obtain a first mapping relationship between the first target partition and a node, determine a target node corresponding to the first target partition according to the first mapping relationship, and store the dimension table in the target node, wherein the first target partition is a partition corresponding to the dimension table, and the partitioning strategy is a strategy for performing an operation on the primary key value to partition the target stream data and the dimension table corresponding to the primary key value; A second storage unit is configured to determine a second target partition of the target stream data according to the partitioning strategy based on the primary key value of the target stream data, obtain a second mapping relationship between the second target partition and the node, determine the target node corresponding to the second target partition according to the second mapping relationship, store the target stream data in the target node, and perform streaming calculation on the target stream data and the dimension table on the target node, wherein the second target partition is the partition corresponding to the target stream data, and the second target partition represents the partition corresponding to the target stream data; The first storage unit includes: a first determination module, configured to perform a cyclic redundancy check on the primary key value of the dimension table to obtain a first check value, obtain the number of the first target partitions, perform a modulo operation on the first check value and the number of the first target partitions to obtain a first remainder, and determine that a partition corresponding to a sequence number identical to the first remainder is the first target partition, wherein each first target partition corresponds to a sequence number; The second storage unit includes: The second determination module is used to perform a cyclic redundancy check on the primary key value of the target stream data to obtain a second check value, and obtain the number of the second target partitions, perform a modulo operation on the second check value and the number of the second target partitions to obtain a second remainder, and determine that the partition corresponding to the same serial number as the second remainder is the second target partition, wherein each second target partition corresponds to a serial number.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the computing method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the computing method according to any one of claims 1 to 6.

Images