A method and device for processing streaming data
By initializing work nodes and containers in a streaming big data processing system, instantiating business plug-ins, and using Spout and Bolt components to process streaming data, the existing system configuration solidification and low efficiency are solved, and efficient and reliable flow is achieved Data processing framework that supports complex business needs.
Patent Information
- Application Number
- CN201910932329.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-09-29
AI Technical Summary
Existing streaming big data processing systems such as Storm and Flink cannot meet complex business needs due to their configuration solidification and low processing efficiency, especially in the Internet and Internet of Things scenarios for personalized services, intelligent analysis and in-process decision-making. .
By initializing work nodes, obtaining business plug-in packages, and instantiating business plug-ins in containers, using Spout and Bolt components to process streaming data, supporting distributed storage and dynamic scheduling, realizing a high availability and high reliability streaming data processing framework .
It realizes distributed, high availability and high reliability streaming data processing, supports business-party plug-ins development and expansion, and can dynamically schedule and monitor streaming data processing to meet complex business needs.
Smart Images

Figure CN112667304B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a streaming data processing method and device. Background Art
[0002] In application scenarios such as the Internet and the Internet of Things, complex business needs such as personalized services, improved user experience, intelligent analysis, and in-process decision-making have put forward higher requirements for streaming data processing technology. For example, the search business involves crawling data from web pages, cleaning data, and sorting search results based on algorithms. The data involved is different from data of fixed size or length, but streaming data, so it is generally processed through a streaming data processing framework. At present, the more widely used streaming big data processing systems include Storm, Flink and other technical methods, but due to the reasons of fixed configuration and low streaming processing efficiency, they cannot meet more complex business needs. Summary of the invention
[0003] In view of the above problems, the present invention is proposed to provide a streaming data processing method and device that overcomes the above problems or at least partially solves the above problems.
[0004] According to one aspect of the present invention, a streaming data processing method is provided, comprising:
[0005] Initialize several working nodes;
[0006] Obtain the business plug-in package for each business;
[0007] Starting a container corresponding to each service in the working node, and instantiating a corresponding service plug-in in the container according to the service plug-in package;
[0008] The streaming data of the corresponding business is processed through the instantiated business plug-in.
[0009] Optionally, the method further comprises:
[0010] Providing a front-end page, receiving the service plug-in package of each service through the front-end page, and saving it in the corresponding service directory of the distributed storage;
[0011] The acquiring of the service plug-in of each service includes: acquiring the service plug-in package of each service from each service directory.
[0012] Optionally, the structure of the service plug-in package includes: a service plug-in, configuration information, and library information on which the service plug-in depends.
[0013] Optionally, instantiating a corresponding service plug-in according to the service plug-in package in the container includes:
[0014] Parsing the service plug-in package to determine a service topology diagram;
[0015] Initialize a streaming data distribution component Spout and several streaming data processing components Bolt, and determine the component relationship between Spout and Bolt according to the business topology diagram;
[0016] The tasks determined according to the business plug-in are executed in the Spout and the Bolt respectively.
[0017] Optionally, the component relationship includes at least one of the following:
[0018] Randomly distribute streaming data, distribute streaming data by domain, broadcast streaming data.
[0019] Optionally, the initializing a plurality of working nodes includes: starting a working process corresponding to each working node;
[0020] The initialization streaming data distribution component Spout and several streaming data processing components Bolt include:
[0021] Run Spout and Bolt in a single-process multi-threaded model, and determine the number of Spout task threads and Bolt task threads based on the preset parallelism;
[0022] The executing the tasks determined according to the business plug-in in the Spout and the Bolt respectively includes: starting a plurality of execution threads in the working process, executing the task processing logic of the Spout and the Bolt through the execution threads, and realizing the corresponding Spout task threads and Bolt task threads.
[0023] Optionally, the execution thread includes: a task distribution thread;
[0024] The execution of the task processing logic of Spout and Bolt by the execution thread to implement the corresponding Spout task thread and Bolt task thread includes:
[0025] The task distribution thread is used to realize task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread.
[0026] Optionally, the task distribution is achieved by triggering an event mechanism.
[0027] Optionally, the bolt includes a synchronous bolt and an asynchronous bolt;
[0028] The task distribution of the asynchronous Bolt is specifically triggered by an event notification mechanism.
[0029] Optionally, the event is implemented in C++ language, and the Bolt includes a third-party Bolt implemented in a language other than C++ language; the read event and the write event of the task distribution of the third-party Bolt are separated.
[0030] Optionally, when exiting, the Spout and the Bolt first exit the event loop and then exit the corresponding task thread.
[0031] Optionally, the data transmitted by the task distribution is implemented based on an encapsulated serialized data structure.
[0032] Optionally, the data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
[0033] Optionally, the processing speed of the Spout is dynamically configured through a funnel algorithm, and / or, the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the corresponding Bolt at the upper level of the business topology diagram.
[0034] Optionally, the method further comprises:
[0035] Obtain node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism;
[0036] Get the performance information of each working node in normal status.
[0037] Optionally, the method further comprises:
[0038] If any one of the performance information and the node status information triggers an alarm condition, the corresponding alarm information is sent in a preset manner.
[0039] Optionally, the method further comprises:
[0040] Providing a front-end page, through which detection configuration information is obtained;
[0041] Start a detection Spout to load the detection configuration information, build a service detection topology, start several service bolts and a capacity expansion bolt, each service bolt corresponds to a service and includes a collection bolt and a decision bolt;
[0042] Allocate the IP list of the corresponding service to each collection bolt, so that each collection bolt collects the performance information of the corresponding service according to the corresponding IP list and passes it to the corresponding decision bolt;
[0043] The scaling bolt aggregates the decision information of each decision bolt and generates corresponding scaling information.
[0044] The number of containers is adjusted according to the expansion and contraction information.
[0045] Optionally, the method further comprises:
[0046] The expansion and contraction information is displayed through the front-end page.
[0047] According to another aspect of the present invention, there is provided a streaming data processing device, comprising:
[0048] An initialization unit, suitable for initializing several working nodes;
[0049] A business unit, adapted to obtain a business plug-in package of each business, start a container corresponding to each business in the working node, and instantiate a corresponding business plug-in in the container according to the business plug-in package;
[0050] The processing unit is adapted to process the streaming data of the corresponding service through the instantiated service plug-in.
[0051] Optionally, the device further comprises:
[0052] An interaction unit, adapted to provide a front-end page, receive a service plug-in package of each service through the front-end page, and save it in a corresponding service directory of the distributed storage;
[0053] The service unit is adapted to obtain the service plug-in package of each service from each service directory.
[0054] Optionally, the structure of the service plug-in package includes: a service plug-in, configuration information, and library information on which the service plug-in depends.
[0055] Optionally, the service unit is adapted to parse the service plug-in package to determine a service topology map;
[0056] Initialize a streaming data distribution component Spout and several streaming data processing components Bolt, and determine the component relationship between Spout and Bolt according to the business topology diagram;
[0057] The tasks determined according to the business plug-in are executed in the Spout and the Bolt respectively.
[0058] Optionally, the component relationship includes at least one of the following:
[0059] Randomly distribute streaming data, distribute streaming data by domain, broadcast streaming data.
[0060] Optionally, the initialization unit is adapted to start a work process corresponding to each work node; run Spout and Bolt in a single-process multi-threaded model, and determine the number of Spout task threads and Bolt task threads according to a preset parallelism;
[0061] The business unit is suitable for starting a plurality of execution threads in a work process, and executing the task processing logic of Spout and Bolt through the execution threads to realize the corresponding Spout task threads and Bolt task threads.
[0062] Optionally, the execution thread includes: a task distribution thread;
[0063] The business unit is suitable for implementing task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread through the task distribution thread.
[0064] Optionally, the task distribution is achieved by triggering an event mechanism.
[0065] Optionally, the bolt includes a synchronous bolt and an asynchronous bolt;
[0066] The task distribution of the asynchronous Bolt is specifically triggered by an event notification mechanism.
[0067] Optionally, the event is implemented in C++ language, and the Bolt includes a third-party Bolt implemented in a language other than C++ language; the read event and the write event of the task distribution of the third-party Bolt are separated.
[0068] Optionally, when exiting, the Spout and the Bolt first exit the event loop and then exit the corresponding task thread.
[0069] Optionally, the data transmitted by the task distribution is implemented based on an encapsulated serialized data structure.
[0070] Optionally, the data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
[0071] Optionally, the processing speed of the Spout is dynamically configured through a funnel algorithm, and / or, the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the corresponding Bolt at the upper level of the business topology diagram.
[0072] Optionally, the device further comprises:
[0073] The detection unit is adapted to obtain node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism; and obtain performance information of each working node in a normal state.
[0074] Optionally, the device further comprises:
[0075] The alarm unit is adapted to send corresponding alarm information in a preset manner if any one of the performance information and the node status information triggers an alarm condition.
[0076] Optionally, the detection unit is suitable for providing a front-end page, and obtaining detection configuration information through the front-end page; starting a detection Spout to load the detection configuration information, building a business detection topology map, starting several groups of business Bolts and a scaling Bolt, each group of business Bolts corresponds to a business, and includes a collection Bolt and a decision Bolt; allocating an IP list of the corresponding business to each collection Bolt, so that each collection Bolt collects performance information of the corresponding business according to the corresponding IP list and passes it to the corresponding decision Bolt; the scaling Bolt summarizes the decision information of each decision Bolt to generate corresponding scaling information; and adjusting the number of containers according to the scaling information.
[0077] Optionally, the detection unit is further adapted to display the expansion and contraction information through the front-end page.
[0078] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes any of the above methods.
[0079] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the one or more programs implement any of the methods described above.
[0080] As can be seen from the above, the technical solution of the present invention is to initialize several working nodes; obtain the business plug-in package of each business; start the container corresponding to each business in the working node, and instantiate the corresponding business plug-in in the container according to the business plug-in package; and process the streaming data of the corresponding business through the instantiated business plug-in. The beneficial effect is that a distributed, high-availability, and high-reliability streaming data processing framework can be realized, which supports the business party to develop plug-ins and expand nodes, and can realize the tracking and processing of streaming data, thereby performing dynamic scheduling and monitoring.
[0081] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0083] Figure 1 A schematic flow chart of a method for processing streaming data according to an embodiment of the present invention is shown;
[0084] Figure 2 A schematic structural diagram of a streaming data processing device according to an embodiment of the present invention is shown;
[0085] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown;
[0086] Figure 4 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0087] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0088] Figure 1 FIG. 2 shows a flow chart of a method for processing streaming data according to an embodiment of the present invention. Figure 1 As shown, the method includes:
[0089] Step 110, initializing several working nodes.
[0090] The embodiment of the present invention can be specifically implemented through a stream processing framework. When the stream processing framework is initialized, several working nodes are initialized first. At this time, no business is running. The nodes can be considered empty, waiting to complete the subsequent process according to the business needs. It can be seen that this is different from stream processing frameworks such as Storm. The configuration can be flexibly adjusted according to the business needs, and it is not a fixed configuration, which meets the business needs in complex scenarios.
[0091] Step 120: Obtain the service plug-in package of each service.
[0092] By implementing the functions of various services in the form of plug-ins, the implementation mode of the services can be made more flexible, and the services have the advantages of easy deployment and convenient operation and maintenance. The present invention can implement various functions of the services in the form of an integrated plug-in package. When processing streaming data, it is necessary to first obtain the service plug-in package, parse the service plug-in and the configuration information related to the service plug-in, etc., to provide a basis for subsequent processing work.
[0093] Step 130: Start a container corresponding to each service in the working node, and instantiate a corresponding service plug-in in the container according to the service plug-in package.
[0094] Containers are an implementation method that is becoming more and more popular among business personnel. They have been widely used due to their flexibility, high resource utilization, and strong robustness. The embodiments of the present invention can be implemented by docking with existing container architectures. For example, scheduling can be achieved at the container level by cooperating with existing container management servers. In the working nodes, containers are used as the technical basis. When implementing business functions, corresponding business plug-ins are instantiated according to the business plug-in packages deployed in the containers, thereby realizing business functions. In this way, each business is independent of each other, and can determine the data flow between each other based on business relevance. It can also achieve a flexible deployment of the number of containers based on the different priorities of tasks, resource usage, and other characteristics, making more fair and reasonable allocation and scheduling, and effectively improving the operating efficiency of streaming data processing.
[0095] Step 140: Process the streaming data of the corresponding service through the instantiated service plug-in.
[0096] As can be seen Figure 1 The method shown can realize a distributed, high-availability, and high-reliability streaming data processing framework, support business parties to develop plug-ins, expand nodes, and realize the tracking and processing of streaming data, thereby performing dynamic scheduling and monitoring.
[0097] In one embodiment of the present invention, the above method also includes: providing a front-end page, receiving the business plug-in package of each business through the front-end page, and saving it in the corresponding business directory of distributed storage; obtaining the business plug-in of each business includes: obtaining the business plug-in package of each business from each business directory.
[0098] The business plug-in package can come from the platform or be uploaded by the user. Users can upload through the front-end page, which receives the business plug-in packages of each business, and these business plug-in packages will be stored in the corresponding business directory of the distributed storage. When these plug-in packages need to be called, the business plug-in packages of each business are obtained from each business directory. In this way, the plug-in acquisition method can be more flexible and diverse, and it can show stronger adaptability when facing different business needs.
[0099] For the streaming data processing framework, it only needs to disclose the structure of the business plug-in package, so that developers of each business line can write their own business plug-in packages based on the structure. For example, given a class object, the developers of the business line only need to inherit the corresponding class object and perform corresponding custom design in a specific method function.
[0100] In one embodiment of the present invention, in the above method, the structure of the service plug-in package includes: a service plug-in, configuration information, and library information on which the service plug-in depends.
[0101] For example, the structure of the service plug-in package 1 may include service plug-ins, configuration information etc of the service plug-ins, information of the library lib that the service plug-ins depend on, etc. In this way, all necessary information required for the stable operation of the service plug-in package is included.
[0102] In one embodiment of the present invention, in the above method, instantiating the corresponding business plug-in in the container according to the business plug-in package includes: parsing the business plug-in package to determine the business topology map; initializing the streaming data distribution component Spout and several streaming data processing components Bolt, and determining the component relationship between Spout and Bolt according to the business topology map; and executing each task determined according to the business plug-in in Spout and Bolt respectively.
[0103] Here, Spout and Bolt borrow the concepts from the Storm framework, that is, they distribute and process data respectively, but it does not mean that the implementation needs to be consistent with the Storm framework. For example, Storm is implemented in Java language, which can easily cause the GC (garbage collection) jamming problem of the JVM (Java virtual machine). Therefore, C++ and other languages can be used instead to implement the embodiments of the present invention.
[0104] When implementing business functions, business plug-ins have their own implementation process. After parsing the business plug-in package, the business topology map can be obtained. When processing streaming data, the components in the business are initialized first to complete the corresponding configuration work. Among them, the Spout component is responsible for distributing streaming data, and the Bolt component is responsible for specific processing. After determining the business topology map, the relationship between the topology map and the related topology maps can be further determined, so that there is a global relationship map and the overall business topology map is constructed. Finally, the business function is implemented through this overall topology map and the Spout and Bolt components in the plug-in. In this way, in massive data and complex processing processes, the business topology map with related relationships can be quickly constructed into the overall business topology map, thereby providing a global perspective to review the operating status of each link of the business operation, which is conducive to searching for business data processing and decision-making. The implementation process of the business function can be displayed in the form of an intuitive topology map, which is conducive to monitoring from the overall perspective and thus controlling the process of business function implementation.
[0105] Determine whether the data source of the Spout matches the data output of the Bolt. If they match, there is an association relationship. For example, if the data source of the Spout of a search sub-business corresponds to the data output of another search sub-business Bolt, then the data source of the Spout is determined to match the data output of the Bolt and there is an association relationship. The data source and data output are determined based on the message queue. The message queue is a container for storing messages during the transmission process. It is used to improve system performance and peak shaving and reduce system coupling through asynchronous processing. The message queue can be used to determine between the data source and data output. In this way, the association relationship of the search sub-business and the message transmission method are clarified.
[0106] In one embodiment of the present invention, in the above method, the component relationship includes at least one of the following: randomly allocating streaming data, allocating streaming data by domain, and broadcasting streaming data.
[0107] Random distribution of streaming data is a common and frequently used data stream group. It contains a parameter (data source component) that sends tuples to randomly selected bolts to ensure that each consumer receives a similar number of tuples, so that each task is evenly distributed. Domain distribution of streaming data allows one or more domains of the tuple to control how tuples are sent to each bolt. It is possible to send value sets with the same domain combination to the same bolt. For example, if the word domain is used as the data stream group, the word-normalizer bolt (used to normalize words) will only send tuples with the same word to the same word-counter bolt (word counting) instance. Broadcast streaming data makes a copy of the tuple for each instance that receives the data. This grouping method is used to send signals to all bolts. For example, taking the refresh cache as an example, a refresh cache signal can be sent to all bolts. Of course, the above component relationships are only examples, and different component relationships can be used to form topologies in specific implementations.
[0108] In one embodiment of the present invention, in the above method, initializing several working nodes includes: starting a working process corresponding to each working node; initializing a streaming data distribution component Spout and several streaming data processing components Bolt includes: running Spout and Bolt in a single-process multi-threaded model, and determining the number of Spout task threads and Bolt task threads according to a preset parallelism; executing each task determined according to the business plug-in in Spout and Bolt respectively includes: starting several execution threads in the working process, executing the task processing logic of Spout and Bolt through the execution threads, and realizing the corresponding Spout task threads and Bolt task threads.
[0109] A thread is the smallest unit of program execution, while a process is the smallest unit of resource allocation. You can start several related execution threads in the working process first. Since Spout and Bolt in the execution thread have their own task processing logic, the corresponding Spout task threads and Bolt task threads are implemented according to these task processing logics. In this way, the Spout task threads and Bolt task threads corresponding to the tasks are implemented through Spout and Bolt and the task processing logic inside the two, and the execution of the program is realized. Both Spout and Bolt run in a single-process multi-threaded model. The number of threads of each component running is specified by the initial configuration parallelism, which realizes fine-grained resource allocation.
[0110] In one embodiment of the present invention, in the above method, the execution thread includes: a task distribution thread; executing the task processing logic of Spout and Bolt through the execution thread to implement the corresponding Spout task thread and Bolt task thread includes: implementing task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread through the task distribution thread.
[0111] Each thread of a Spout / Bolt in the task process worker can be called a task. Different Spout / Bolt tasks can share a physical thread, which is called an executor. Specifically, an OutputDispatcher independent thread is used to distribute tasks to different Bolts.
[0112] In one embodiment of the present invention, in the above method, task distribution is implemented by triggering an event mechanism. The worker process triggers corresponding asynchronous task processing through the event mechanism to improve operation efficiency.
[0113] In one embodiment of the present invention, in the above method, Bolt includes synchronous Bolt and asynchronous Bolt; the task distribution of asynchronous Bolt is specifically triggered by an event notification mechanism. The asynchronous bolt distributes tasks through the event notification mechanism and supports asynchronous task processing, and maintains independent execution units internally.
[0114] In one embodiment of the present invention, in the above method, the event is implemented in C++ language, and the Bolt includes a third-party Bolt implemented in a language other than C++ language; the read event and the write event of the task distribution of the third-party Bolt are separated.
[0115] Specifically, the entire streaming processing framework can also be implemented in C++. As mentioned above, most of the current streaming processing frameworks are implemented in Java, which can easily cause JVM GC jamming problems, and there are serious problems with improper resource use caused by the mixing of multiple businesses. The internal operating status of the business cannot be monitored, and the business performance and quality tracking cannot be met. The streaming framework implemented in C++ can effectively solve this problem. In addition, it can also support third-party Bolts developed in other programming languages, which are also triggered by an event mechanism. The difference is that when communicating with a third-party Bolt, reading and writing are separated, and each task is written to the corresponding buffer. At the same time, the read event only processes the corresponding task when the third-party Bolt returns, which will not cause task blocking.
[0116] In one embodiment of the present invention, in the above method, when exiting, Spout and Bolt first exit the event loop and then exit the corresponding task thread. Specifically, considering the problem of task accumulation, a buffer mechanism is used for safe exit, and each running thread will not be stopped until each Spout / Bolt task is processed. A time threshold can also be set, such as forcing a timeout exit if it has not exited within 60 seconds.
[0117] In one embodiment of the present invention, in the above method, the data transmitted by task distribution is implemented based on the encapsulated serialized data structure. For data serialization and deserialization of various data structures such as Bool, Int, Double, String, Vector, Array, etc., it only needs to be encapsulated into the corresponding ObjectPtr to realize message transmission.
[0118] In one embodiment of the present invention, in the above method, the data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
[0119] In terms of reliability, the bitset data structure is encapsulated into the task to track the status of each Bolt, and each Bolt node name corresponds to the bit in the tracing structure (that is, the flag bit in the data structure). The state transition mechanism is used to implement three different processing methods: emit, ack, and fail. When a failure occurs, three mechanisms are supported: 0: resend the message until the message is ACKed; 1: resend the message until the number of retries is reached; 2: no longer resend the message, and process it at most once. At the same time, under the above-mentioned 1st and 2nd failures, failure reentry or non-reentry mechanisms are also supported.
[0120] The Exactly-once mechanism can be used to ensure the atomicity of semantics, that is, through the task tracing mechanism, each task is executed once and only once. Under the failure reentrant mechanism, the failure information of the failed task can be recorded in different tracing bits. When the output dispatcher task is distributed, the Bolt that has been processed can be avoided according to its tracing bit.
[0121] In one embodiment of the present invention, in the above method, the processing speed of the Spout is dynamically configured through a funnel algorithm, and / or, the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the superior of the corresponding Bolt in the business topology diagram.
[0122] By using the speed limit method, the maximum processing speed of the Spout can be dynamically configured to limit the failure rate of task processing. In addition, it can also support step-by-step back pressure. When the downstream processing task slows down or an error occurs, the speed of the Bolt processing the task will decrease. Since the upstream back pressure eventually reduces the Spout pulling speed of the data source, the balance of the streaming processing speed is finally achieved. Specifically, it can be controlled by setting the task queue size.
[0123] In the specific implementation, the business side can inherit synchronous Bolt, asynchronous Bolt, and Bolt that supports third-party languages through class inheritance, and build the streaming processing topology required by the business. The business side can also define the topology structure by loading different configuration files.
[0124] In one embodiment of the present invention, the method further includes: obtaining node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism; and obtaining performance information of each working node in a normal state. The promethues indicator format is supported, and its metrics can record the counter value of each running state, such as task failure, task success, the number of tasks distributed to each Bolt, and other information. At the same time, a restful interface API is also provided to display the internal running state of each topology thread.
[0125] The heartbeat mechanism is a mechanism that sends a custom structure, such as a heartbeat packet, at regular intervals to let the other party know that it is still alive, so as to ensure the validity of the connection. In order to obtain node status information from the distributed database, a heartbeat mechanism can be set up to obtain node status information and performance information of each working node in normal state through the heartbeat mechanism. In this way, performance detection is achieved.
[0126] In one embodiment of the present invention, the method further includes: if any one of the performance information and the node status information triggers an alarm condition, sending corresponding alarm information in a preset manner.
[0127] By setting a preset alarm mechanism and presetting any one of the performance information and node status information to trigger the alarm condition, when insufficient performance or abnormal node status occurs during operation, the corresponding alarm information will be sent in a preset manner, thereby achieving effective monitoring and early warning of performance information and node status.
[0128] In one embodiment of the present invention, the method further includes: providing a front-end page, and obtaining detection configuration information through the front-end page; starting a detection Spout to load the detection configuration information, building a business detection topology map, starting several groups of business Bolts and a scaling Bolt, each group of business Bolts corresponds to a business, and includes a collection Bolt and a decision Bolt; allocating an IP list of the corresponding business to each collection Bolt, so that each collection Bolt collects performance information of the corresponding business according to the corresponding IP list and passes it to the corresponding decision Bolt; the scaling Bolt summarizes the decision information of each decision Bolt to generate corresponding scaling information; and adjusts the number of containers according to the scaling information.
[0129] In order to enable the system to have intelligent elastic scheduling capabilities, realize real-time monitoring of business redundancy, and automatically scale up and down as needed, it is necessary to collect performance information of the corresponding business, and then summarize and make decisions based on the collected business information to obtain the corresponding scaling information, so as to adjust the number of containers. Specifically, firstly, the plug-in and configuration information are detected according to the front-end page, and then a Spout is started to load the detection configuration information, and the business is divided according to different types of business and the topology map of business detection is constructed respectively. Then, several groups of business Bolts and a scaling Bolt corresponding to each business are started. Each group of business Bolts contains a collection Bolt and a decision Bolt. Among them, Bolt is responsible for receiving the information flow emitted by the Spout component and completing the specific processing logic. In complex business logic, multiple Bolt components can be connected in series, and different functions can be implemented in each Bolt component, so as to realize the overall processing logic. The execution of the corresponding business generally has a specific IP rule. Therefore, the IP list of the corresponding business can be assigned to each collection Bolt, so that each collection Bolt collects the performance information of the corresponding business according to the corresponding IP list. After obtaining the performance information of the corresponding business, it is passed to the corresponding decision bolt, and then the expansion and contraction bolt summarizes the decision information of each decision bolt and generates the corresponding expansion and contraction information, and finally adjusts the number of containers according to the expansion and contraction information. In this way, the intelligent elastic scheduling function is fully realized, which can monitor the redundancy of the business in real time and automatically expand and contract as needed.
[0130] It can be seen that this scheduling actually forms two levels of scheduling, namely, business-level scheduling and container-level scheduling. The streaming processing framework in the prior art can often only practice container-level scheduling, so it is impossible to control the status of each business, and the resource utilization is not good enough. The advantage of the embodiment of the present invention is that in the scenario of streaming processing, it cleverly uses the streaming data processing method to make decisions, so as to control the status of different businesses, not only can the number of containers be scheduled as a whole, but also the number of containers for each business can be scheduled separately, so as to utilize resources to a greater extent.
[0131] In one embodiment of the present invention, the method further includes: displaying the expansion and contraction information through a front-end page.
[0132] In order to observe the running status from a global perspective more intuitively, the corresponding expansion and contraction information can be displayed through the front-end page. In this way, the expansion and contraction information can be displayed through the front-end page in a timely manner, which is conducive to observation from a global perspective.
[0133] Figure 2 FIG. 2 shows a schematic diagram of the structure of a streaming data processing device according to an embodiment of the present invention. Figure 2 As shown, the device 200 includes:
[0134] An initialization unit 210, adapted to initialize a number of working nodes;
[0135] The embodiment of the present invention can be specifically implemented through a stream processing framework. When the stream processing framework is initialized, several working nodes are initialized first. At this time, no business is running. The nodes can be considered empty, waiting to complete the subsequent process according to the business needs. It can be seen that this is different from stream processing frameworks such as Storm. The configuration can be flexibly adjusted according to the business needs, and it is not a fixed configuration, which meets the business needs in complex scenarios.
[0136] The business unit 220 is adapted to obtain a business plug-in package of each business, start a container corresponding to each business in a working node, and instantiate a corresponding business plug-in in the container according to the business plug-in package;
[0137] By implementing the functions of various businesses in the form of plug-ins, the implementation mode of the business can be made more flexible, with the advantages of easy deployment and convenient operation and maintenance. The present invention can implement various business functions in the form of an integrated plug-in package. When processing streaming data, it is necessary to first obtain the business plug-in package, from which the business plug-in and the configuration information related to the business plug-in are parsed, so as to provide a basis for subsequent processing work. Containers are an implementation method that is increasingly popular with business personnel. Due to their flexibility, high resource utilization, strong robustness and other advantages, they have been widely used. The embodiments of the present invention can be implemented by docking with the existing container architecture, for example, scheduling is implemented by cooperating with the existing container management server at the container level. In the working node, the container is used as the technical basis. When implementing the business function, the corresponding business plug-in is instantiated according to the business plug-in package deployed in the container, so as to realize the business function. In this way, each business is independent of each other, and the data flow direction between each other can be determined according to the business relevance. It can also realize a flexible deployment of the number of containers in combination with the different priorities of tasks, resource occupancy and other characteristics, make a more fair and reasonable allocation and scheduling, and effectively improve the running efficiency of streaming data processing.
[0138] The processing unit 230 is adapted to process the streaming data of the corresponding service through the instantiated service plug-in.
[0139] As can be seen Figure 2 The device shown can realize a distributed, high-availability, and high-reliability streaming data processing framework, support business parties to develop plug-ins, expand nodes, and can track and process streaming data, thereby performing dynamic scheduling and monitoring.
[0140] In one embodiment of the present invention, the above-mentioned device also includes: an interaction unit, suitable for providing a front-end page, receiving a business plug-in package of each business through the front-end page, and saving it in a corresponding business directory of distributed storage; a business unit 220, suitable for obtaining a business plug-in package of each business from each business directory.
[0141] The business plug-in package can come from the platform or be uploaded by the user. Users can upload through the front-end page, which receives the business plug-in packages of each business, and these business plug-in packages will be stored in the corresponding business directory of the distributed storage. When these plug-in packages need to be called, the business plug-in packages of each business are obtained from each business directory. In this way, the plug-in acquisition method can be more flexible and diverse, and it can show stronger adaptability when facing different business needs.
[0142] For the streaming data processing framework, it only needs to disclose the structure of the business plug-in package, so that developers of each business line can write their own business plug-in packages based on the structure. For example, given a class object, the developers of the business line only need to inherit the corresponding class object and perform corresponding custom design in a specific method function.
[0143] In one embodiment of the present invention, in the above-mentioned device, the structure of the service plug-in package includes: a service plug-in, configuration information, and library information on which the service plug-in depends.
[0144] For example, the structure of the service plug-in package 1 may include service plug-ins, configuration information etc of the service plug-ins, information of the library lib that the service plug-ins depend on, etc. In this way, all necessary information required for the stable operation of the service plug-in package is included.
[0145] In one embodiment of the present invention, in the above-mentioned device, the business unit 220 is suitable for parsing the business plug-in package to determine the business topology map; initializing the streaming data distribution component Spout and several streaming data processing components Bolt, and determining the component relationship between Spout and Bolt according to the business topology map; and executing each task determined according to the business plug-in in Spout and Bolt respectively.
[0146] Here, Spout and Bolt borrow the concepts from the Storm framework, that is, they distribute and process data respectively, but it does not mean that the implementation needs to be consistent with the Storm framework. For example, Storm is implemented in Java language, which can easily cause the GC (garbage collection) jamming problem of the JVM (Java virtual machine). Therefore, C++ and other languages can be used instead to implement the embodiments of the present invention.
[0147] When implementing business functions, business plug-ins have their own implementation process. After parsing the business plug-in package, the business topology map can be obtained. When processing streaming data, the components in the business are initialized first, and the corresponding configuration work has been completed. Among them, the Spout component is responsible for distributing streaming data, and the Bolt component is responsible for specific processing. After determining the business topology map, the relationship between the topology map and the related topology maps can be further determined, so that there is a global relationship map and the overall business topology map is constructed. Finally, the business function is implemented through this overall topology map and the Spout and Bolt components in the plug-in. In this way, in massive data and complex processing processes, the business topology map with related relationships can be quickly constructed into the overall business topology map, thereby providing a global perspective to review the operating status of each link of the business operation, which is conducive to searching for business data processing and decision-making. The implementation process of the business function can be displayed in the form of an intuitive topology map, which is conducive to monitoring from the overall perspective and thus controlling the process of business function implementation.
[0148] In one embodiment of the present invention, in the above-mentioned apparatus, the component relationship includes at least one of the following: randomly allocating streaming data, allocating streaming data by domain, and broadcasting streaming data.
[0149] Random distribution of streaming data is a common and frequently used data stream group. It contains a parameter (data source component) that sends tuples to randomly selected bolts to ensure that each consumer receives a similar number of tuples, so that each task is evenly distributed. Domain distribution of streaming data allows one or more domains of the tuple to control how tuples are sent to each bolt. It is possible to send value sets with the same domain combination to the same bolt. For example, if the word domain is used as the data stream group, the word-normalizer bolt (used to normalize words) will only send tuples with the same word to the same word-counter bolt (word counting) instance. Broadcast streaming data makes a copy of the tuple for each instance that receives the data. This grouping method is used to send signals to all bolts. For example, taking the refresh cache as an example, a refresh cache signal can be sent to all bolts. Of course, the above component relationships are only examples, and different component relationships can be used to form topologies in specific implementations.
[0150] In one embodiment of the present invention, in the above-mentioned device, the initialization unit 210 is suitable for starting a working process corresponding to each working node; running Spout and Bolt in a single-process multi-threaded model, and determining the number of Spout task threads and Bolt task threads according to a preset parallelism; the business unit 220 is suitable for starting a plurality of execution threads in the working process, executing the task processing logic of Spout and Bolt through the execution threads, and realizing the corresponding Spout task threads and Bolt task threads.
[0151] A thread is the smallest unit of program execution, while a process is the smallest unit of resource allocation. You can start several related execution threads in the working process first. Since Spout and Bolt in the execution thread have their own task processing logic, the corresponding Spout task threads and Bolt task threads are implemented according to these task processing logics. In this way, the Spout task threads and Bolt task threads corresponding to the tasks are implemented through Spout and Bolt and the task processing logic inside the two, and the execution of the program is realized. Both Spout and Bolt run in a single-process multi-threaded model. The number of threads of each component running is specified by the initial configuration parallelism, which realizes fine-grained resource allocation.
[0152] In one embodiment of the present invention, in the above device, the execution thread includes: a task distribution thread; and a business unit 220, adapted to implement task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread through the task distribution thread.
[0153] Each thread of a Spout / Bolt in the task process worker can be called a task. Different Spout / Bolt tasks can share a physical thread, which is called an executor. Specifically, an OutputDispatcher independent thread is used to distribute tasks to different Bolts.
[0154] In one embodiment of the present invention, in the above device, task distribution is implemented by triggering an event mechanism. The worker process triggers corresponding asynchronous task processing through the event mechanism to improve operation efficiency.
[0155] In one embodiment of the present invention, in the above device, Bolt includes synchronous Bolt and asynchronous Bolt; the task distribution of asynchronous Bolt is specifically triggered by an event notification mechanism. The asynchronous bolt distributes tasks through the event notification mechanism and supports asynchronous task processing, and maintains independent execution units internally.
[0156] In one embodiment of the present invention, in the above device, the event is implemented by C++ language, and the Bolt includes a third-party Bolt implemented by a language other than C++ language; the read event and the write event of the task distribution of the third-party Bolt are separated.
[0157] Specifically, the entire streaming processing framework can also be implemented in C++. As mentioned above, most of the current streaming processing frameworks are implemented in Java, which can easily cause JVM GC jamming problems, and there are serious problems with improper resource use caused by the mixing of multiple businesses. The internal operating status of the business cannot be monitored, and the business performance and quality tracking cannot be met. The streaming framework implemented in C++ can effectively solve this problem. In addition, it can also support third-party Bolts developed in other programming languages, which are also triggered by an event mechanism. The difference is that when communicating with a third-party Bolt, reading and writing are separated, and each task is written to the corresponding buffer. At the same time, the read event only processes the corresponding task when the third-party Bolt returns, which will not cause task blocking.
[0158] In one embodiment of the present invention, in the above device, when exiting, Spout and Bolt first exit the event loop and then exit the corresponding task thread. Specifically, considering the problem of task accumulation, a buffer mechanism is used for safe exit, and each running thread will be stopped only when each Spout / Bolt task is processed. A time threshold can also be set, such as forcing a timeout exit if it has not exited within 60 seconds.
[0159] In one embodiment of the present invention, in the above device, the data transmitted by task distribution is realized based on the encapsulated serialized data structure. For data serialization and deserialization of various data structures such as Bool, Int, Double, String, Vector, Array, etc., it only needs to be encapsulated into the corresponding ObjectPtr to realize message transmission.
[0160] In one embodiment of the present invention, in the above-mentioned device, the data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
[0161] In terms of reliability, the bitset data structure is encapsulated into the task to track the status of each Bolt, and each Bolt node name corresponds to the bit in the tracing structure (that is, the flag bit in the data structure). The state transition mechanism is used to implement three different processing methods: emit, ack, and fail. When a failure occurs, three mechanisms are supported: 0: resend the message until the message is ACKed; 1: resend the message until the number of retries is reached; 2: no longer resend the message, and process it at most once. At the same time, under the above-mentioned 1st and 2nd failures, failure reentry or non-reentry mechanisms are also supported.
[0162] The Exactly-once mechanism can be used to ensure the atomicity of semantics, that is, through the task tracing mechanism, each task is executed once and only once. Under the failure reentrant mechanism, the failure information of the failed task can be recorded in different tracing bits. When the output dispatcher task is distributed, the Bolt that has been processed can be avoided according to its tracing bit.
[0163] In one embodiment of the present invention, in the above-mentioned device, the processing speed of the Spout is dynamically configured through a funnel algorithm, and / or, the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the superior of the corresponding Bolt in the business topology diagram.
[0164] By using the speed limit method, the maximum processing speed of the Spout can be dynamically configured to limit the failure rate of task processing. In addition, it can also support step-by-step back pressure. When the downstream processing task slows down or an error occurs, the speed of the Bolt processing the task will decrease. Since the upstream back pressure eventually reduces the Spout pulling speed of the data source, the balance of the streaming processing speed is finally achieved. Specifically, it can be controlled by setting the task queue size.
[0165] In the specific implementation, the business side can inherit synchronous Bolt, asynchronous Bolt, and Bolt that supports third-party languages through class inheritance, and build the streaming processing topology required by the business. The business side can also define the topology structure by loading different configuration files.
[0166] In one embodiment of the present invention, in the above-mentioned device, the device also includes: a detection unit, which is suitable for obtaining node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism; and the performance information of each working node in a normal state is obtained. The promethues indicator format is supported, and its metrics can record the counter value of each running state, such as task failure, task success, the number of tasks distributed to each Bolt, and other information. At the same time, a restful interface API is also provided to display the internal running status of each topology thread.
[0167] The heartbeat mechanism sends a custom structure, such as a heartbeat packet, at regular intervals to let the other party know that it is still alive, so as to ensure the validity of the connection. In order to obtain node status information from the distributed database, a heartbeat mechanism can be set up to obtain node status information and performance information of each working node in normal state through the heartbeat mechanism. In this way, performance detection is realized.
[0168] In one embodiment of the present invention, in the above-mentioned device, the device further includes: an alarm unit, adapted to send corresponding alarm information in a preset manner if any one of the performance information and the node status information triggers an alarm condition.
[0169] By setting a preset alarm mechanism and presetting any one of the performance information and node status information to trigger the alarm condition, when insufficient performance or abnormal node status occurs during operation, the corresponding alarm information will be sent in a preset manner, thereby achieving effective monitoring and early warning of performance information and node status.
[0170] In one embodiment of the present invention, in the above-mentioned device, the detection unit is suitable for providing a front-end page, and obtaining the detection configuration information through the front-end page; starting a detection Spout to load the detection configuration information, building a business detection topology map, starting several groups of business Bolts and a scaling Bolt, each group of business Bolts corresponds to a business, and includes a collection Bolt and a decision Bolt; allocating an IP list of the corresponding business to each collection Bolt, so that each collection Bolt collects the performance information of the corresponding business according to the corresponding IP list and passes it to the corresponding decision Bolt; the scaling Bolt summarizes the decision information of each decision Bolt to generate corresponding scaling information; and adjusting the number of containers according to the scaling information.
[0171] In order to enable the system to have intelligent elastic scheduling capabilities, realize real-time monitoring of business redundancy, and automatically scale up and down as needed, it is necessary to collect performance information of the corresponding business, and then summarize and make decisions based on the collected business information to obtain the corresponding scaling information to adjust the number of containers. Specifically, firstly, the plug-in and configuration information are detected according to the front-end page, and then a Spout is started to load the detection configuration information. According to the different types of business, they are divided and the topology diagram of business detection is constructed respectively. Then, several groups of business Bolts and a scaling Bolt corresponding to each business are started. Each group of business Bolts contains a collection Bolt and a decision Bolt. Among them, Bolt is responsible for receiving the information flow emitted by the Spout component and completing the specific processing logic. In complex business logic, multiple Bolt components can be connected in series, and different functions can be implemented in each Bolt component, so as to realize the overall processing logic. The execution of the corresponding business generally has a specific IP rule. Therefore, the IP list of the corresponding business can be assigned to each collection Bolt, so that each collection Bolt collects the performance information of the corresponding business according to the corresponding IP list. After obtaining the performance information of the corresponding business, it is passed to the corresponding decision bolt, and then the expansion and contraction bolt summarizes the decision information of each decision bolt and generates the corresponding expansion and contraction information, and finally adjusts the number of containers according to the expansion and contraction information. In this way, the intelligent elastic scheduling function is fully realized, which can monitor the redundancy of the business in real time and automatically expand and contract as needed.
[0172] It can be seen that this scheduling actually forms two levels of scheduling, namely, business-level scheduling and container-level scheduling. The streaming processing framework in the prior art can often only practice container-level scheduling, so it is impossible to control the status of each business, and the resource utilization is not good enough. The advantage of the embodiment of the present invention is that in the scenario of streaming processing, it cleverly uses the streaming data processing method to make decisions, so as to control the status of different businesses, not only can the number of containers be scheduled as a whole, but also the number of containers for each business can be scheduled separately, so as to utilize resources to a greater extent.
[0173] In one embodiment of the present invention, in the above-mentioned device, the detection unit is further adapted to display the expansion and contraction information through a front-end page.
[0174] In order to observe the running status from a global perspective more intuitively, the corresponding expansion and contraction information is displayed through the front-end page. In this way, the expansion and contraction information is displayed on the front-end page in a timely manner, which is conducive to observation from a global perspective.
[0175] It should be noted that the specific implementation of the above-mentioned device embodiments can be carried out with reference to the specific implementation of the above-mentioned corresponding method embodiments, which will not be repeated here.
[0176] In summary, the technical solution of the present invention initializes several working nodes; obtains the business plug-in package of each business; starts the container corresponding to each business in the working node, and instantiates the corresponding business plug-in in the container according to the business plug-in package; processes the streaming data of the corresponding business through the instantiated business plug-in. The beneficial effect is that a distributed, high-availability, and high-reliability data processing framework can be realized, which supports the business party to develop plug-ins and expand nodes, and can realize the tracking and processing of data, thereby performing dynamic scheduling and monitoring.
[0177] It should be noted that:
[0178] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general-purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for the purpose of disclosing the best mode of the present invention.
[0179] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0180] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.
[0181] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.
[0182] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.
[0183] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the streaming data processing device according to an embodiment of the present invention. The present invention can also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0184] For example, Figure 3A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. The electronic device 300 includes a processor 310 and a memory 320 arranged to store computer executable instructions (computer readable program code). The memory 320 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 320 has a storage space 330 for storing a computer readable program code 331 for executing any method step in the above method. For example, the storage space 330 for storing computer readable program code may include individual computer readable program codes 331 for implementing various steps in the above method respectively. The computer readable program code 331 can be read from or written into one or more computer program products. These computer program products include program code carriers such as a hard disk, a compact disk (CD), a memory card or a floppy disk. Such a computer program product is typically, for example Figure 4 The computer-readable storage medium. Figure 4 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown. The computer-readable storage medium 400 stores a computer-readable program code 331 for executing the method steps according to the present invention, which can be read by the processor 310 of the electronic device 300. When the computer-readable program code 331 is executed by the electronic device 300, the electronic device 300 executes the various steps in the method described above. Specifically, the computer-readable program code 331 stored in the computer-readable storage medium can execute the method shown in any of the above embodiments. The computer-readable program code 331 can be compressed in an appropriate form.
[0185] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.
[0186] The embodiment of the present invention discloses A1, a streaming data processing method, comprising:
[0187] Initialize several working nodes;
[0188] Obtain the business plug-in package for each business;
[0189] Starting a container corresponding to each service in the working node, and instantiating a corresponding service plug-in in the container according to the service plug-in package;
[0190] The streaming data of the corresponding business is processed through the instantiated business plug-in.
[0191] A2. The method as described in A1, wherein the method further comprises:
[0192] Providing a front-end page, receiving the service plug-in package of each service through the front-end page, and saving it in the corresponding service directory of the distributed storage;
[0193] The acquiring of the service plug-in of each service includes: acquiring the service plug-in package of each service from each service directory.
[0194] A3. The method as described in A1, wherein the structure of the service plug-in package includes: a service plug-in, configuration information, and library information on which the service plug-in depends.
[0195] A4. The method as described in A1, wherein instantiating a corresponding service plug-in in the container according to the service plug-in package comprises:
[0196] Parsing the service plug-in package to determine a service topology diagram;
[0197] Initialize a streaming data distribution component Spout and several streaming data processing components Bolt, and determine the component relationship between Spout and Bolt according to the business topology diagram;
[0198] The tasks determined according to the business plug-in are executed in the Spout and the Bolt respectively.
[0199] A5. The method as described in A4, wherein the component relationship includes at least one of the following:
[0200] Randomly distribute streaming data, distribute streaming data by domain, broadcast streaming data.
[0201] A6. The method as described in A4, wherein the initializing several working nodes comprises: starting a working process corresponding to each working node;
[0202] The initialization streaming data distribution component Spout and several streaming data processing components Bolt include:
[0203] Run Spout and Bolt in a single-process multi-threaded model, and determine the number of Spout task threads and Bolt task threads based on the preset parallelism;
[0204] The executing the tasks determined according to the business plug-in in the Spout and the Bolt respectively includes: starting a plurality of execution threads in the working process, executing the task processing logic of the Spout and the Bolt through the execution threads, and realizing the corresponding Spout task threads and Bolt task threads.
[0205] A7. The method as described in A6, wherein the execution thread includes: a task distribution thread;
[0206] The execution of the task processing logic of Spout and Bolt by the execution thread to implement the corresponding Spout task thread and Bolt task thread includes:
[0207] The task distribution thread is used to realize task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread.
[0208] A8. The method as described in A7, wherein the task distribution is achieved by triggering an event mechanism.
[0209] A9. The method as described in A8, wherein the bolts include synchronous bolts and asynchronous bolts;
[0210] The task distribution of the asynchronous Bolt is specifically triggered by an event notification mechanism.
[0211] A10. The method as described in A8, wherein the event is implemented in C++ language, the bolt includes a third-party bolt implemented in a language other than C++ language, and the read event and write event of the task distribution of the third-party bolt are separated.
[0212] A11. The method as described in A8, wherein:
[0213] When exiting, the Spout and the Bolt first exit the event loop and then exit the corresponding task thread.
[0214] A12. A method as described in A8, wherein the data transmitted by the task distribution is implemented based on an encapsulated serialized data structure.
[0215] A13. A method as described in A12, wherein the data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
[0216] A14. A method as described in A4, wherein the processing speed of the Spout is dynamically configured through a funnel algorithm, and / or the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the corresponding Bolt at the upper level of the business topology diagram.
[0217] A15. The method as described in A1, wherein the method further comprises:
[0218] Obtain node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism;
[0219] Get the performance information of each working node in normal status.
[0220] A16. The method as described in A15, wherein the method further comprises:
[0221] If any one of the performance information and the node status information triggers an alarm condition, the corresponding alarm information is sent in a preset manner.
[0222] A17. The method as described in A15, wherein the method further comprises:
[0223] Providing a front-end page, through which detection configuration information is obtained;
[0224] Start a detection Spout to load the detection configuration information, build a service detection topology, start several service bolts and a capacity expansion bolt, each service bolt corresponds to a service and includes a collection bolt and a decision bolt;
[0225] Allocate the IP list of the corresponding service to each collection bolt, so that each collection bolt collects the performance information of the corresponding service according to the corresponding IP list and passes it to the corresponding decision bolt;
[0226] The scaling bolt aggregates the decision information of each decision bolt and generates corresponding scaling information.
[0227] The number of containers is adjusted according to the expansion and contraction information.
[0228] A18. The method as described in A17, wherein the method further comprises:
[0229] The expansion and contraction information is displayed through the front-end page.
[0230] The embodiment of the present invention further discloses B19, a streaming data processing device, comprising:
[0231] An initialization unit, suitable for initializing several working nodes;
[0232] A business unit, adapted to obtain a business plug-in package of each business, start a container corresponding to each business in the working node, and instantiate a corresponding business plug-in in the container according to the business plug-in package;
[0233] The processing unit is adapted to process the streaming data of the corresponding service through the instantiated service plug-in.
[0234] B20. The device as described in B19, wherein the device further comprises:
[0235] An interaction unit, adapted to provide a front-end page, receive a service plug-in package of each service through the front-end page, and save it in a corresponding service directory of the distributed storage;
[0236] The service unit is adapted to obtain the service plug-in package of each service from each service directory.
[0237] B21. The device as described in B19, wherein the structure of the service plug-in package includes: a service plug-in, configuration information, and library information on which the service plug-in depends.
[0238] B22. The device as described in B19, wherein:
[0239] The business unit is adapted to parse the business plug-in package to determine a business topology diagram; initialize a streaming data distribution component Spout and a plurality of streaming data processing components Bolt, and determine a component relationship between Spout and Bolt according to the business topology diagram; and respectively execute tasks determined according to the business plug-in in Spout and Bolt.
[0240] B23. The device as described in B22, wherein the component relationship includes at least one of the following:
[0241] Randomly distribute streaming data, distribute streaming data by domain, broadcast streaming data.
[0242] B24. The device as described in B22, wherein
[0243] The initialization unit is adapted to start a work process corresponding to each work node; run Spout and Bolt in a single-process multi-threaded model, and determine the number of Spout task threads and Bolt task threads according to a preset parallelism;
[0244] The business unit is suitable for starting a plurality of execution threads in a work process, and executing the task processing logic of Spout and Bolt through the execution threads to realize the corresponding Spout task threads and Bolt task threads.
[0245] B25. The apparatus as described in B24, wherein the execution thread includes: a task distribution thread;
[0246] The business unit is suitable for implementing task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread through the task distribution thread.
[0247] B26. An apparatus as described in B25, wherein the task distribution is achieved by triggering an event mechanism.
[0248] B27. The device as described in B25, wherein the bolt includes a synchronous bolt and an asynchronous bolt;
[0249] The task distribution of the asynchronous Bolt is specifically triggered by an event notification mechanism.
[0250] B28. The device as described in B26, wherein the event is implemented in C++ language, and the bolt includes a third-party bolt implemented in a language other than C++ language; the read event and the write event of the task distribution of the third-party bolt are separated.
[0251] B29. The device as described in B26, wherein
[0252] When exiting, the Spout and the Bolt first exit the event loop and then exit the corresponding task thread.
[0253] B30. An apparatus as described in B26, wherein the data transmitted by the task distribution is implemented based on an encapsulated serialized data structure.
[0254] B31. An apparatus as described in B30, wherein the data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
[0255] B32. An apparatus as described in B22, wherein the processing speed of the Spout is dynamically configured through a funnel algorithm, and / or the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the corresponding Bolt at the upper level of the business topology diagram.
[0256] B33. The device as described in B19, wherein the device further comprises:
[0257] The detection unit is adapted to obtain node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism; and obtain performance information of each working node in a normal state.
[0258] B34. The device as described in B33, wherein the device further comprises:
[0259] The alarm unit is adapted to send corresponding alarm information in a preset manner if any one of the performance information and the node status information triggers an alarm condition.
[0260] B35. The device as described in B33, wherein:
[0261] The detection unit is suitable for providing a front-end page, and obtaining detection configuration information through the front-end page; starting a detection Spout to load the detection configuration information, building a business detection topology map, starting several groups of business Bolts and a scaling Bolt, each group of business Bolts corresponds to a business, and includes a collection Bolt and a decision Bolt; allocating an IP list of the corresponding business to each collection Bolt, so that each collection Bolt collects performance information of the corresponding business according to the corresponding IP list and passes it to the corresponding decision Bolt; the scaling Bolt summarizes the decision information of each decision Bolt to generate corresponding scaling information; and adjusting the number of containers according to the scaling information.
[0262] B36. The device as described in B35, wherein:
[0263] The detection unit is further adapted to display the expansion and contraction information through the front-end page.
[0264] An embodiment of the present invention also discloses C37, an electronic device, wherein the electronic device comprises: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to execute a method as described in any one of A1-A18.
[0265] An embodiment of the present invention also discloses D38, a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, implement the method as described in any one of A1-A18.
Claims
1. A streaming data processing method, comprising: Initialize several working nodes; Obtain the business plug-in package for each business; Starting a container corresponding to each service in the working node, and instantiating a corresponding service plug-in in the container according to the service plug-in package; Process the streaming data of the corresponding business through the instantiated business plug-in; Instantiating the corresponding service plug-in according to the service plug-in package in the container includes: Parsing the service plug-in package to determine a service topology diagram; Initialize a streaming data distribution component Spout and several streaming data processing components Bolt, and determine the component relationship between Spout and Bolt according to the business topology diagram; The tasks determined according to the business plug-in are executed in the Spout and the Bolt respectively.
2. The method of claim 1, wherein: The method further comprises: Providing a front-end page, receiving the service plug-in package of each service through the front-end page, and saving it in the corresponding service directory of the distributed storage; The acquiring of the service plug-in of each service includes: acquiring the service plug-in package of each service from each service directory.
3. The method of claim 1, wherein: The structure of the service plug-in package includes: a service plug-in, configuration information, and library information that the service plug-in depends on.
4. The method of claim 1, wherein: The component relationship includes at least one of the following: Randomly distribute streaming data, distribute streaming data by domain, broadcast streaming data.
5. The method of claim 1, wherein: Initializing a plurality of working nodes includes: starting a working process corresponding to each working node; The initialization streaming data distribution component Spout and several streaming data processing components Bolt include: Run Spout and Bolt in a single-process multi-threaded model, and determine the number of Spout task threads and Bolt task threads based on the preset parallelism; The executing the tasks determined according to the business plug-in in the Spout and the Bolt respectively includes: starting a plurality of execution threads in the working process, executing the task processing logic of the Spout and the Bolt through the execution threads, and realizing the corresponding Spout task threads and Bolt task threads.
6. The method of claim 5, wherein: The execution threads include: task distribution threads; The execution of the task processing logic of Spout and Bolt by the execution thread to implement the corresponding Spout task thread and Bolt task thread includes: The task distribution thread is used to realize task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread.
7. The method of claim 6, wherein: The task distribution is achieved by triggering an event mechanism.
8. The method of claim 7, wherein: The bolt includes a synchronous bolt and an asynchronous bolt; The task distribution of the asynchronous Bolt is specifically triggered by an event notification mechanism.
9. The method of claim 7, wherein: The event is implemented in C++ language, and the bolt includes a third-party bolt implemented in a language other than C++ language; the read event and the write event of the task distribution of the third-party bolt are separated.
10. The method of claim 7, wherein: When exiting, the Spout and the Bolt first exit the event loop and then exit the corresponding task thread.
11. The method of claim 7, wherein: The data transmitted by the task distribution is implemented based on an encapsulated serialized data structure.
12. The method of claim 11, wherein: The data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
13. The method of claim 1, wherein: The processing speed of the Spout is dynamically configured through a funnel algorithm, and / or the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the corresponding Bolt at the upper level of the business topology graph.
14. The method of claim 1, wherein: The method further comprises: Obtain node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism; Get the performance information of each working node in normal status.
15. The method of claim 14, wherein: The method further comprises: If any one of the performance information and the node status information triggers an alarm condition, the corresponding alarm information is sent in a preset manner.
16. The method of claim 14, wherein: The method further comprises: Providing a front-end page, through which detection configuration information is obtained; Start a detection Spout to load the detection configuration information, build a service detection topology, start several service bolts and a capacity expansion bolt, each service bolt corresponds to a service and includes a collection bolt and a decision bolt; Allocate the IP list of the corresponding service to each collection bolt, so that each collection bolt collects the performance information of the corresponding service according to the corresponding IP list and passes it to the corresponding decision bolt; The scaling bolt aggregates the decision information of each decision bolt and generates corresponding scaling information. The number of containers is adjusted according to the expansion and contraction information.
17. The method of claim 16, wherein: The method further comprises: The expansion and contraction information is displayed through the front-end page.
18. A streaming data processing device, comprising: An initialization unit, suitable for initializing several working nodes; A business unit, adapted to obtain a business plug-in package of each business, start a container corresponding to each business in the working node, and instantiate a corresponding business plug-in in the container according to the business plug-in package; A processing unit, adapted to process streaming data of a corresponding service through an instantiated service plug-in; The business unit is specifically adapted to parse the business plug-in package to determine a business topology diagram; initialize a streaming data distribution component Spout and a plurality of streaming data processing components Bolt, and determine a component relationship between Spout and Bolt according to the business topology diagram; and respectively execute tasks determined according to the business plug-in in Spout and Bolt.
19. The device of claim 18, wherein: The device also includes: An interaction unit, adapted to provide a front-end page, receive a service plug-in package of each service through the front-end page, and save it in a corresponding service directory of the distributed storage; The service unit is adapted to obtain the service plug-in package of each service from each service directory.
20. The apparatus of claim 18, wherein: The structure of the service plug-in package includes: a service plug-in, configuration information, and library information that the service plug-in depends on.
21. The apparatus of claim 18, wherein: The component relationship includes at least one of the following: Randomly distribute streaming data, distribute streaming data by domain, broadcast streaming data.
22. The apparatus of claim 18, wherein: The initialization unit is adapted to start a work process corresponding to each work node; run Spout and Bolt in a single-process multi-threaded model, and determine the number of Spout task threads and Bolt task threads according to a preset parallelism; The business unit is suitable for starting a plurality of execution threads in a work process, and executing the task processing logic of Spout and Bolt through the execution threads to realize the corresponding Spout task threads and Bolt task threads.
23. The device of claim 22, wherein: The execution threads include: task distribution threads; The business unit is suitable for implementing task distribution between the Spout task thread and the Bolt task thread, and between each Bolt task thread through the task distribution thread.
24. The device of claim 23, wherein: The task distribution is achieved by triggering an event mechanism.
25. The apparatus of claim 23, wherein: The bolt includes a synchronous bolt and an asynchronous bolt; The task distribution of the asynchronous Bolt is specifically triggered by an event notification mechanism.
26. The apparatus of claim 24, wherein: The event is implemented in C++ language, and the bolt includes a third-party bolt implemented in a language other than C++ language; the read event and the write event of the task distribution of the third-party bolt are separated.
27. The apparatus of claim 24, wherein: When exiting, the Spout and the Bolt first exit the event loop and then exit the corresponding task thread.
28. The apparatus of claim 24, wherein: The data transmitted by the task distribution is implemented based on an encapsulated serialized data structure.
29. The apparatus of claim 28, wherein: The data structure includes a flag bit for data tracking, so as to achieve reliability of task processing through a data tracking mechanism.
30. The apparatus of claim 18, wherein: The processing speed of the Spout is dynamically configured through a funnel algorithm, and / or the processing speed of the Spout is determined by step-by-step back pressure based on the processing speed of each Bolt; the step-by-step back pressure includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the corresponding Bolt at the upper level of the business topology graph.
31. The apparatus of claim 18, wherein: The device also includes: The detection unit is adapted to obtain node status information from a distributed database; the node status information is generated by each working node through a heartbeat mechanism; and obtain performance information of each working node in a normal state.
32. The apparatus of claim 31, wherein: The device also includes: The alarm unit is adapted to send corresponding alarm information in a preset manner if any one of the performance information and the node status information triggers an alarm condition.
33. The apparatus of claim 31, wherein: The detection unit is adapted to provide a front-end page, through which the detection configuration information is acquired; A detection Spout is started to load the detection configuration information, a service detection topology is constructed, and several groups of service Bolts and a scaling Bolt are started. Each group of service Bolts corresponds to a service and includes a collection Bolt and a decision Bolt. An IP list of a corresponding service is allocated to each collection Bolt, so that each collection Bolt collects performance information of the corresponding service according to the corresponding IP list and transmits it to the corresponding decision Bolt. The scaling Bolt summarizes the decision information of each decision Bolt to generate corresponding scaling information. The number of containers is adjusted according to the scaling information.
34. The apparatus of claim 33, wherein: The detection unit is further adapted to display the expansion and contraction information through the front-end page.
35. An electronic device, wherein: The electronic device comprises: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor performs the method according to any one of claims 1 to 17.
36. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the method according to any one of claims 1 to 17.
Citation Information
Patent Citations
Software architecture system
CN103309674A
Data processing method and system
CN105468626A
Data processing method, system, interface port and server
CN107133020A
Portable software platform and method for quickly finishing customized business, and robot
CN107273137A
Data-driven and plug-in defined event engine
US20070073739A1
