Data stream processing method and device based on merge tree model and storage medium

Abstract

Embodiments of the present specification provide a data stream processing method and device based on a merge tree model and a storage medium. The data stream processing method comprises: obtaining source data; obtaining a vertex set and an edge data set according to the source data; obtaining an extreme point set and a rule point set according to the vertex set; obtaining a candidate saddle point set according to the extreme point set, the rule point set and the edge data set; obtaining an extreme point-candidate saddle point pair set according to the extreme point set and the candidate saddle point set; obtaining an extreme point-saddle point pair set according to the extreme point-candidate saddle point pair set and a preset vertex distribution rule; adding the extreme point-saddle point pair set to a preset merge tree; and the communication strategy comprises pushing, pulling and publishing / subscription. The technical solution provided by the present application is used to solve the problems in the prior art that the data volume is increasing, the data operation amount is large, the efficiency is low when the distributed nodes update the data, and it is difficult to achieve good performance when the data volume is large or the control flow is complex.

Description

Technical Field

[0001] This document relates to the field of large-scale data processing technology, and in particular to a data stream processing method, apparatus and storage medium based on a merged tree model. Background Technology

[0002] The data flow model, or data-centric execution model, guides parallelization by extracting the flow of data within an application, which is of great significance for optimizing data-centric topology analysis algorithms.

[0003] Existing methods use topology analysis algorithms to process computational fluid dynamics simulation data, explore the parallelism of applications from the perspective of control flow, and perform data simulation, feature extraction, and parallel data analysis and visualization.

[0004] However, with the continuous improvement of computing power and the increasing amount of data, traditional data analysis models are unable to solve the problem of low efficiency when updating data by distributed nodes in existing technologies. At the same time, since the problem addressed by the algorithm is unstructured, it is difficult to achieve good performance when the amount of data is large or the control flow is complex. Summary of the Invention

[0005] Based on the above analysis, this application aims to propose a data flow processing method, apparatus, and storage medium based on a merged tree model. Based on the merged tree data flow model, a data flow graph is generated through data nodes and communication strategies, thereby improving the input, output, and interaction speed of large-scale data in data analysis; and ensuring data consistency and accuracy during frequent data synchronization.

[0006] Firstly, one or more embodiments of this specification provide a data stream processing method based on a merging tree model, including:

[0007] Obtain source data;

[0008] Based on the source data, the vertex set and edge set are obtained;

[0009] Based on the vertex set, the extreme point set and the regular point set are obtained;

[0010] Based on the extreme point set, the regular point set, and the edge dataset, a candidate saddle point set is obtained;

[0011] Based on the set of extreme points and the set of candidate saddle points, a set of extreme point-saddle point pairs is obtained;

[0012] Add the set of extreme point saddle points to the preset merge tree;

[0013] The communication strategies include push, pull, and publish / subscribe.

[0014] Furthermore, the plurality of data nodes includes: a first data node, a second data node, and a third data node;

[0015] Based on the source data, a vertex set and an edge set are obtained, including:

[0016] The first data node generates the vertex set and the edge set based on the source data; it pushes the vertex set to the second data node and the edge set to the third data node.

[0017] Furthermore, the plurality of data nodes also includes: a fourth data node and a fifth data node;

[0018] Based on the vertex set, the extreme point set and the regular point set are obtained, including:

[0019] The second data node constructs an extreme point pointing chain starting from each element of the vertex set; generates the extreme point set and the rule point set according to the extreme point pointing chain; pushes the extreme point set to the fourth data node and the rule point set to the fifth data node.

[0020] Furthermore, the plurality of data nodes also includes: a sixth data node;

[0021] Based on the extreme point set, the regular point set, and the edge dataset, a candidate saddle point set is obtained, including:

[0022] The third data node publishes the edge dataset;

[0023] The fourth data node pushes the set of extreme points to the sixth data node;

[0024] The fifth data node pushes the rule point set to the sixth data node;

[0025] The sixth data node subscribes to the edge dataset in the third data node;

[0026] The sixth data node generates the candidate saddle point set based on the extreme point set, the rule point set, and the edge dataset.

[0027] Furthermore, the plurality of data nodes includes: a seventh data node;

[0028] Based on the set of extreme points and the set of candidate saddle points, a set of extreme point-saddle point pairs is obtained, including:

[0029] The sixth data node pushes the candidate saddle point set to the seventh data node;

[0030] The seventh data node pulls the extreme point set from the fourth data node; the seventh data node generates the extreme point-candidate saddle point pair set based on the extreme point set and the candidate saddle point set; the elements in the extreme point-candidate saddle point pair set are sorted to generate the extreme point-saddle point pair set.

[0031] Furthermore, the set of extreme point saddle points is added to a preset merge tree, including:

[0032] The seventh data node pushes the extreme point saddle point pair set into the merge tree.

[0033] Furthermore, the set of extreme point-saddle point pairs includes at least one extreme point-saddle point pair; each extreme point-saddle point pair includes an extreme point and a saddle point, comprising:

[0034] Based on the extreme point-saddle point pair set and the preset vertex allocation rule, the intermediate point set between the extreme point and the saddle point is obtained, and then the extreme point-saddle point pair set and the intermediate point set are deleted from the current topology graph.

[0035] Secondly, one or more embodiments of this specification provide a data stream processing device based on a merging tree model, including: an acquisition module, a first generation module, a second generation module, a first synthesis module, a second synthesis module, and an output module;

[0036] The acquisition module is used to acquire source data; the first generation module is used to obtain a vertex set and an edge set based on the source data; the second generation module is used to obtain an extreme point set and a regular point set based on the vertex set; the first synthesis module is used to obtain a candidate saddle point set based on the extreme point set, the regular point set, and the edge set; the second synthesis module is used to obtain an extreme point-saddle point pair set based on the extreme point set and the candidate saddle point set; the output module is used to add the extreme point-saddle point pair set to a preset merging tree; the communication strategy includes push, pull, and publish / subscribe.

[0037] Thirdly, one or more embodiments of this specification provide a computing device, including: a memory, a processor, and computer instructions stored in the memory and executable on the processor.

[0038] When the processor executes the computer instructions, it implements the steps of the data node processing method according to any one of the first aspects.

[0039] Fourthly, one or more embodiments of this specification provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data node processing method described in any of the first aspects.

[0040] Compared with the prior art, this application can achieve at least the following technical effects:

[0041] 1. This application transmits data based on a preset data node transmission and communication strategy. Therefore, when data in a data node is updated, the updated data can be transmitted to the corresponding node or temporarily stored in that data node. In contrast, with existing distributed nodes, when data on any node is updated, the server needs to update that data on all nodes in real time. Thus, compared to existing technologies, this application requires less computation and occupies less memory when transmitting data, thereby reducing storage costs and improving data synchronization efficiency.

[0042] 2. This application provides a standardized definition of the communication strategy and adopts three delivery modes: push, pull, and publish / subscribe. This eliminates the need for developers to set the execution order of the application from the perspective of control flow, reducing their working time, improving work efficiency, and fully leveraging the parallelism of data. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a data flow processing method based on a merging tree model, provided for one or more embodiments of this specification;

[0045] Figure 2 A logical scheduling graph for generating vertex sets and edge sets from source data in a parallel merging tree data stream provided in one or more embodiments of this specification;

[0046] Figure 3 In a parallel merging tree data stream provided by one or more embodiments of this specification, a logical scheduling graph is used to construct extreme point pointing chains from the vertex dataset to generate extreme point sets and regular point sets.

[0047] Figure 4 In a parallel merging tree data stream provided in one or more embodiments of this specification, a logical scheduling diagram for extracting data streams from candidate saddle points of a candidate saddle point set generated from a set of regular points is generated by determining whether vertices point to different extreme points.

[0048] Figure 5In a parallel merging tree data stream provided in one or more embodiments of this specification, the extreme point-candidate saddle point pair set is globally sorted, and the logical scheduling graph of the extreme point-saddle point pair set is extracted.

[0049] Figure 6 In a parallel merging tree data stream provided by one or more embodiments of this specification, a logical scheduling graph is extracted to combine extreme points and candidate saddle points and extract the set of maximum extreme point-candidate saddle point pairs among the candidate saddle points of extreme points.

[0050] Figure 7 The present invention relates to a logical scheduling diagram of a vertex allocation data stream in a parallel merging tree data stream provided in one or more embodiments of this specification, which determines the allocation of regular points based on extreme point-saddle point pairs and regular point data.

[0051] Figure 8 A logical scheduling graph for generating new extreme points, regular points, and edge sets in a parallel merging tree data stream provided in one or more embodiments of this specification;

[0052] Figure 9 A schematic diagram of the structure of a data stream processing device based on a merging tree model provided for one or more embodiments of this specification;

[0053] Figure 10 A schematic diagram of the structure of an electronic device based on a merging tree model provided for one or more embodiments of this specification;

[0054] Figure 11 This is a schematic diagram of the structure of a data stream storage medium based on a merged tree model, provided for one or more embodiments of this specification. Detailed Implementation

[0055] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0056] The increasing I / O bandwidth and interactivity requirements, as well as the frequent data synchronization or blocking operations, pose significant challenges to current data analysis and storage technologies. Based on these problems and scenarios, this application provides a data stream processing method based on a merging tree model.

[0057] The merging tree operation is implemented by establishing a data flow model. This data flow model separates the implementation of the topology analysis algorithm from the scheduling process, which requires introducing a logical scheduling graph of the data flow between the description model and the physical scheduling model. Through this logical scheduling graph, the communication method and the computational layer are completely separated, and can be transparently mapped to different runtime backends subsequently.

[0058] In this embodiment, data nodes and communication strategies are constructed using a data flow description language. The data flow description language is designed as a declarative interface, including descriptive elements such as data flow graphs (DFDs), data nodes, and data dependencies (Edges), and is implemented using C++ or text files. The system provides a pre-loading function for the data flow description model, parsing it into a global directed acyclic graph for further computational management and scheduling within the workspace.

[0059] Furthermore, a declarative approach is used to standardize the connections between nodes and communication methods in the merged tree data flow graph. Specifically, the target topology analysis algorithm is abstracted into a logical representation of the data flow graph through declarative specification. This declarative approach standardizes the definition of nodes and dependencies within the data flow graph, facilitating flexible design and extension by users as needed. Once a DFD is declared, the parsing layer can load the DFD into the system space, completing the instantiation of the DAG. Simultaneously, the entire description layer is supported by parallel operation primitives as the underlying API environment.

[0060] Regarding communication methods, this invention defines three types of message communication models, suitable for the needs of data request and transmission in topology analysis. These message communication methods are described in detail below.

[0061] Specifically, "Push" means to push data. After completing a calculation, the current data node actively pushes the calculation result to one or more associated nodes. This usually results in the current data being partitioned from one equivalence class into multiple new equivalence classes. For example, after completing a data calculation, node A actively pushes the calculation result to associated node B or node C.

[0062] Pull means to retrieve data. The data needed by the current data node may have already been calculated during the previous data flow. In this case, a pull operation needs to be performed from the target data node. If the data is ready, the pull operation is complete. For example, after node A completes the data calculation, it stores the result in memory. When node B needs the data from node A, node B performs a pull operation to obtain the data from node A.

[0063] Publish / Subscribe indicates a publish / subscribe pattern. If the data required by the currently subscribed data node is updated, the updated data needs to be published to the buffer, and then retrieved and consumed by the subscribing node. For example, if node B needs node A's data after node A's data is published, it can obtain the corresponding data from node A through the subscription pattern.

[0064] At the computational level, this invention constructs a data flow scheduling model by setting up data nodes.

[0065] Specifically, this invention employs a k-way reduction graph to construct the data flow scheduling model for the algorithm. The k-way reduction graph describes a given algorithm module, a set of idempotent computation nodes, and the data flow connecting them. The logical scheduling graph requires two parameters: the number of input grid blocks, *n*, and the user-specified reduction factor, *k*. The parameter *n* corresponds to the parallelism of the analysis method, i.e., the number of data blocks obtained from the partition; the parameter *k* specifies the reduction degree required at each level. For example, when *k* is 2, each pair of nodes at each level undergoes reduction. In this invention, the scheduling graph provides three different types of scheduling nodes: leaf nodes for initiating input, internal nodes for reduction, and root computation nodes. To link the logical scheduling graph to the implementation process, it is only necessary to allocate different computations to each computation node; this paper chooses callback functions to accomplish this binding process.

[0066] The algorithm model for generating a merged tree data flow graph is constructed by connecting data nodes through communication methods. First, all data objects used and evolved from the algorithm need to be identified. This step requires appropriately selecting the granularity of data extraction to ensure the simplicity and ease of use of subsequent data flows. Second, the dependencies between data are explored to determine the data flow direction and dependency clusters of each data node, i.e., input and output edges. Based on this, a data flow graph is constructed, combining all nodes through data flows to form a complete algorithm model.

[0067] In this embodiment, to achieve the aforementioned technical solution, multiple data nodes are required. These data nodes include: a first data node, a second data node, a third data node, a fourth data node, a fifth data node, a sixth data node, and a seventh data node. The specific implementation method is further explained using flowcharts and logical scheduling diagrams. The specific method is as follows: Figure 1-8 As shown in the figure, the following steps are included:

[0068] Step 1: Obtain source data. Based on the source data, obtain the vertex set and edge set.

[0069] In the embodiments of this application, such as Figure 2As shown, the first data node, InputGrid, obtains source data. Specifically, the first data node, InputGrid, interacts with external data to obtain the source data file, generates grid block data, and the grid block data is divided into blocks and the source data is generated according to the number of leaf nodes specified by the current system.

[0070] The first data node, InputGrid, splits the source data to obtain a vertex set and an edge set; the vertex set is pushed to the second data node, Vn, and the edge set is pushed to the third data node, En.

[0071] Step 2: Based on the vertex set, obtain the extreme point set and the regular point set.

[0072] In the embodiments of this application, such as Figure 3 As shown, the second data node Vn constructs an extreme point pointing chain starting from each element vertex of the vertex set. The extreme point set and the regular point set are generated from the original vertex set data according to the extreme point pointing chain. The extreme point set is pushed to the fourth data node Mn, and the regular point set is pushed to the fifth data node Rn.

[0073] Step 3: Obtain the candidate saddle point set based on the extreme point set, the regular point set, and the edge set.

[0074] In the embodiments of this application, such as Figure 4 As shown, the fourth data node Mn pushes the extreme point set to the sixth data node Sn, the fifth data node Rn pushes the regular point set to the sixth data node Sn, and the sixth data node Sn subscribes to the edge dataset in the third data node En; the sixth data node Sn judges whether the vertex set points to different extreme point sets based on the edge dataset and extracts the candidate saddle point set data.

[0075] Step 4: Based on the set of extreme points and the set of candidate saddle points, obtain the set of extreme point-saddle point pairs.

[0076] In the embodiments of this application, such as Figure 5 and Figure 6 As shown, the sixth data node Sn pushes the candidate saddle point set to the seventh data node SEn. The seventh data node SEn pulls the extreme point set from the fourth data node Mn. SEn first sorts the extreme points in the extreme point set in descending order, and then sorts the corresponding candidate saddle points in the candidate saddle point set in descending order based on the extreme points. Based on the sorted extreme points and candidate saddle points, an extreme point-candidate saddle point pair set is generated. In the extreme point-candidate saddle point pair set, the seventh data node SEn selects the largest candidate saddle point corresponding to each extreme point as the saddle point. Based on the saddle point and the extreme point, an extreme point-saddle point pair set is generated.

[0077] Step 5: Add the extreme point-saddle point pair set to the preset merge tree.

[0078] In this embodiment, the seventh data node SEn pushes the extreme point-saddle point pair set into the merge tree.

[0079] In the embodiments of this application, such as Figure 7 As shown, the set of extreme point-saddle point pairs includes at least one extreme point-saddle point pair; each extreme point-saddle point pair includes one extreme point and one saddle point.

[0080] Based on the extreme point-saddle point pair set and the preset vertex assignment rule Assignn, the intermediate point set between extreme points and saddle points is obtained, and then the extreme point-saddle point pair set and the intermediate point set are deleted from the current topology graph.

[0081] In addition, such as Figure 8 As shown, when the extreme point-saddle point pair set and the intermediate point set are deleted from the current topology graph, a new extreme point set NMn and a new regular point set NRn are generated. The new extreme point set and the new regular point set are generated according to the vertex allocation rules. At this time, the previous saddle point set will become the regular point set or the new extreme point set. When constructing the new logical schedule, the new data of each point is pulled and the original edge dataset data is subscribed to generate a new edge dataset NEn. The calculation continues, and the newly obtained extreme point-saddle point pair set is added to the merge tree.

[0082] This application provides a data stream processing device based on a merging tree model, such as... Figure 9 As shown, it includes: an acquisition module 101, a first generation module 102, a second generation module 103, a first synthesis module 104, a second synthesis module 105, and an output module 106;

[0083] The acquisition module 101 reads the source data and inputs it into the device; the first generation module 102 splits the source data into the vertex set and the edge set; the second generation module 103 splits the vertex set into the extreme point set and the regular point set; the first synthesis module 104 combines the extreme point set, the regular point set, and the edge set to obtain a candidate saddle point set; the second synthesis module 105 combines the extreme point set and the candidate saddle point set to obtain an extreme point-saddle point pair set; and the output module 106 adds the extreme point-saddle point pair set to a preset merge tree.

[0084] This application provides a computing device, such as... Figure 10 As shown, it includes a memory 202, a processor 201, an input device 203, an output device 204, a communication device 205, and computer instructions stored in the memory and executable on the processor;

[0085] When the processor 201 executes the computer instructions, it implements the steps of the data node processing method in the above embodiments.

[0086] This application provides a computer-readable storage medium, such as... Figure 11 As shown, it stores computer instructions, which, when executed by processor 301, implement the steps of the data node processing method in the above embodiment. The memory 302 stores computer program 303 and data calculated by the data node.

[0087] It should be noted that the embodiments concerning storage media in this specification and the embodiments concerning blockchain-based service provision methods in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding blockchain-based service provision method described above, and the repeated parts will not be described again.

[0088] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0089] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0090] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0091] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0092] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0093] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0098] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0099] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0100] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0102] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0103] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A method for processing data stream based on a merge tree model, characterized in that, Based on the merging tree, multiple data nodes are pre-configured, and these multiple data nodes perform the following steps through a preset communication strategy, including: Obtain source data; Based on the source data, the vertex set and edge set are obtained; Based on the vertex set, the extreme point set and the regular point set are obtained; Based on the extreme point set, the regular point set, and the edge dataset, a candidate saddle point set is obtained; Based on the set of extreme points and the set of candidate saddle points, a set of extreme point-saddle point pairs is obtained; Add the set of extreme point-saddle point pairs to the preset merge tree; The communication strategies include push, pull, and publish / subscribe; Wherein, the candidate saddle point set is the set of vertices in the vertex set for which all neighbor vertices corresponding to ascending edges belong to at least two extrema; the saddle point is the candidate saddle point with the maximum value among all candidate saddle points contained in the monotonic path to the extrema; the regular point set is the set of vertices in the vertex set excluding the extrema point set and the saddle point. The plurality of data nodes includes: a first data node, a second data node, and a third data node; Based on the source data, a vertex set and an edge set are obtained, including: The first data node generates the vertex set and the edge dataset based on the source data; it pushes the vertex set to the second data node and the edge dataset to the third data node. The plurality of data nodes also includes: a fourth data node and a fifth data node; Based on the vertex set, the extreme point set and the regular point set are obtained, including: The second data node constructs an extreme point pointing chain starting from each element of the vertex set; generates the extreme point set and the rule point set according to the extreme point pointing chain; pushes the extreme point set to the fourth data node and the rule point set to the fifth data node; The plurality of data nodes also includes: a sixth data node; Based on the extreme point set, the regular point set, and the edge dataset, a candidate saddle point set is obtained, including: The third data node publishes the edge dataset; The fourth data node pushes the set of extreme points to the sixth data node; The fifth data node pushes the rule point set to the sixth data node; The sixth data node subscribes to the edge dataset in the third data node; The sixth data node generates the candidate saddle point set based on the extreme point set, the rule point set, and the edge dataset; The plurality of data nodes includes: a seventh data node; Based on the set of extreme points and the set of candidate saddle points, a set of extreme point-saddle point pairs is obtained, including: The sixth data node pushes the candidate saddle point set to the seventh data node; The seventh data node pulls the extreme point set from the fourth data node; the seventh data node generates the extreme point-candidate saddle point pair set based on the extreme point set and the candidate saddle point set; the elements in the extreme point-candidate saddle point pair set are sorted to generate the extreme point-saddle point pair set.

2. The method according to claim 1, characterized in that, Adding the set of extreme point saddle points to a preset merge tree includes: The seventh data node pushes the extreme point-saddle point pair set into the merge tree.

3. The method according to claim 1 or 2, characterized in that, The set of extreme point-saddle point pairs includes at least one extreme point-saddle point pair; each extreme point-saddle point pair includes an extreme point and a saddle point, and the method further includes: Based on the extreme point-saddle point pair set and the preset vertex allocation rule, the intermediate point set between the extreme point and the saddle point is obtained, and then the extreme point-saddle point pair set and the intermediate point set are deleted from the current topology graph.

4. A dataflow processing apparatus based on a merge tree model, characterized in that, include: The module includes an acquisition module, a first generation module, a second generation module, a first synthesis module, a second synthesis module, and an output module. The acquisition module is used to acquire source data; The first generation module is used to obtain a vertex set and an edge set based on the source data; the second generation module is used to obtain an extreme point set and a regular point set based on the vertex set; the first synthesis module is used to obtain a candidate saddle point set based on the extreme point set, the regular point set, and the edge set; the second synthesis module is used to obtain an extreme point-saddle point pair set based on the extreme point set and the candidate saddle point set; the output module is used to add the extreme point-saddle point pair set to a preset merge tree; Communication strategies include push, pull, and publish / subscribe; wherein, the candidate saddle point set is the set of vertices in the vertex set whose neighbor vertices corresponding to all ascending edges belong to at least two extrema; the saddle point is the candidate saddle point with the maximum value among all candidate saddle points contained in the monotonic path to the extrema; the regular point set is the set of vertices in the vertex set other than the extrema point set and the saddle point. The multiple data nodes include: the first data node, the second data node, and the third data node; Based on the source data, a vertex set and an edge set are obtained, including: The first data node generates the vertex set and the edge dataset based on the source data; it pushes the vertex set to the second data node and the edge dataset to the third data node. The plurality of data nodes also includes: a fourth data node and a fifth data node; Based on the vertex set, the extreme point set and the regular point set are obtained, including: The second data node constructs an extreme point pointing chain starting from each element of the vertex set; generates the extreme point set and the rule point set according to the extreme point pointing chain; pushes the extreme point set to the fourth data node and the rule point set to the fifth data node; The plurality of data nodes also includes: a sixth data node; Based on the extreme point set, the regular point set, and the edge dataset, a candidate saddle point set is obtained, including: The third data node publishes the edge dataset; The fourth data node pushes the set of extreme points to the sixth data node; The fifth data node pushes the rule point set to the sixth data node; The sixth data node subscribes to the edge dataset in the third data node; The sixth data node generates the candidate saddle point set based on the extreme point set, the rule point set, and the edge dataset; The plurality of data nodes includes: a seventh data node; Based on the set of extreme points and the set of candidate saddle points, a set of extreme point-saddle point pairs is obtained, including: The sixth data node pushes the candidate saddle point set to the seventh data node; The seventh data node pulls the extreme point set from the fourth data node; the seventh data node generates the extreme point-candidate saddle point pair set based on the extreme point set and the candidate saddle point set; the elements in the extreme point-candidate saddle point pair set are sorted to generate the extreme point-saddle point pair set.

5. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer instructions, it implements the steps of the data node processing method according to any one of claims 1-3.

6. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data node processing method according to any one of claims 1-3.

Images