A method and apparatus for processing streaming data of a search service

Abstract

The application discloses a kind of search service's stream data processing method and device.The method includes: obtaining the service plug-in package of each search sub-service;Each service plug-in package is parsed, and the service topology graph of each search sub-service is determined;According to each service topology graph, the stream data of corresponding search sub-service is handled, and according to each service topology graph, the service total topology graph is reconstructed.Affirmative effect is in, according to the service logic of search sub-service, the service topology graph of search sub-service is determined, then according to the association of each service topology graph, service total topology graph is reconstructed, and according to service topology graph, the stream data of corresponding search sub-service is handled.In mass data and complex processing procedure, the service topology graph with association can be quickly structured into total service topology graph, to provide a kind of global perspective to review the running state of each link of business operation, conducive to search service perfect data processing and decision.

Description

Technical Field

[0001] This invention relates to the field of search technology, and more specifically to a streaming data processing method and apparatus for search operations. Background Technology

[0002] In scenarios with large data volumes, such as search operations, higher demands are placed on streaming data processing technologies. For example, search operations involve multiple different parts, such as crawling data from web pages, cleaning the data, and ranking search results according to algorithms. The data involved is not fixed-size or fixed-length data, but streaming data. Therefore, it is generally processed using methods capable of handling large-scale streaming data. Currently, widely used streaming big data processing systems include technologies such as Storm and Flink. However, due to fixed configurations, low streaming processing efficiency, and difficulty in identifying operational problems, they cannot meet more complex business needs. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide a streaming data processing method and apparatus for search services that overcomes or at least partially solves the above problems.

[0004] According to one aspect of the present invention, a streaming data processing method for search services is provided, comprising:

[0005] Obtain the business plugin packages for each search sub-service;

[0006] Analyze each business plugin package to determine the business topology of each search sub-business;

[0007] Process the streaming data of the corresponding search sub-businesses based on each business topology map, and reconstruct the overall business topology map based on each business topology map.

[0008] Optionally, processing the streaming data of the corresponding search sub-services according to each service topology map includes:

[0009] Initialize the streaming data distribution component Spout and several streaming data processing components Bolt, and determine the component relationship between Spout and Bolt according to the business topology diagram;

[0010] Each task, as determined by the business plugin, is executed in Spout and Bolt respectively.

[0011] Optionally, constructing the overall business topology map based on each business topology map includes:

[0012] Identify the relationships between the Spouts of each search sub-business and the Bolts of other search sub-businesses, and construct a general business topology map based on the identified relationships.

[0013] Optionally, identifying the association between the Spout of each search sub-service and the Bolt of other search sub-services includes:

[0014] Determine if the Spout's data source matches the Bolt's data output; if they match, a relationship exists.

[0015] The data source and the data output are determined based on the message queue.

[0016] Optionally, the component relationship includes at least one of the following:

[0017] Randomly allocated streaming data, domain-based allocated streaming data, and broadcast streaming data.

[0018] Optionally, the initialization of the streaming data distribution component Spout and several streaming data processing components Bolt includes: running Spout and Bolt in a single-process multi-threaded model, and determining the number of Spout task threads and Bolt task threads according to a preset degree of parallelism; the execution of each task determined by the business plugin in Spout and Bolt respectively includes:

[0019] Several execution threads are started in the startup process. The execution threads execute the task processing logic of Spout and Bolt, thereby realizing the corresponding Spout task thread and Bolt task thread.

[0020] Optionally, the method further includes: displaying the overall business topology diagram through a front-end page.

[0021] According to another aspect of the present invention, a streaming data processing apparatus for a search service is provided, comprising:

[0022] The acquisition unit is suitable for acquiring the business plugin packages of each search sub-business.

[0023] The parsing unit is suitable for parsing various business plugin packages and determining the business topology of each search sub-business.

[0024] The construction unit is suitable for processing streaming data of corresponding search sub-businesses based on each business topology map, and for reconstructing the overall business topology map based on each business topology map.

[0025] Optionally, the construction unit is adapted to initialize a streaming data distribution component Spout and several streaming data processing components Bolt, and to determine the component relationship between Spout and Bolt according to the business topology diagram;

[0026] Each task, as determined by the business plugin, is executed in Spout and Bolt respectively.

[0027] Optionally, the construction unit is adapted to identify the relationship between the Spout of each search sub-service and the Bolt of other search sub-services, and to construct a general business topology map based on the identified relationship.

[0028] Optionally, the construction unit is adapted to determine whether the data source of Spout matches the data output of Bolt, and if they match, there is an association relationship;

[0029] The data source and the data output are determined based on the message queue.

[0030] Optionally, the component relationship includes at least one of the following:

[0031] Randomly allocated streaming data, domain-based allocated streaming data, and broadcast streaming data.

[0032] Optionally, the construction unit is adapted to run Spout and Bolt in a single-process multi-threaded model, and determines the number of Spout task threads and Bolt task threads according to a preset parallelism; it starts several execution threads in the started working process, and executes the task processing logic of Spout and Bolt through the execution threads, thereby realizing the corresponding Spout task threads and Bolt task threads.

[0033] Optionally, the method further includes: displaying the overall business topology diagram through a front-end page.

[0034] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.

[0035] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method described above.

[0036] As described above, the technical solution of this invention involves acquiring the business plugin packages of each search sub-service; parsing each business plugin package to determine the business topology diagram of each search sub-service; processing the streaming data of the corresponding search sub-service based on each business topology diagram; and reconstructing the overall business topology diagram based on each business topology diagram. The beneficial effect is that the business topology diagram of the search sub-service is determined based on the business logic of the search sub-service, and then the overall business topology diagram is reconstructed based on the relationships between each business topology diagram, and the streaming data of the corresponding search sub-service is processed based on the business topology diagram. In massive amounts of data and complex processing flows, business topology diagrams with relationships can be quickly constructed into an overall business topology diagram, thereby providing a global perspective to examine the operational status of each link in the business operation, which is beneficial for improving data processing and decision-making in the search business.

[0037] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0038] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0039] Figure 1 A flowchart illustrating a streaming data processing method for a search service according to an embodiment of the present invention is shown.

[0040] Figure 2 A schematic diagram of the structure of a streaming data processing apparatus for a search service according to an embodiment of the present invention is shown;

[0041] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown;

[0042] Figure 4 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention is shown. Detailed Implementation

[0043] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0044] Figure 1 A flowchart illustrating a streaming data processing method for a search service according to an embodiment of the present invention is shown. Figure 1 As shown, the method includes:

[0045] Step 110: Obtain the business plugin packages for each search sub-business.

[0046] Business plugin packages can come from the platform or be uploaded by users. Users can upload them through the front-end page, which receives the plugin packages for each business and stores them in the corresponding business directories in distributed storage. When these plugin packages are needed, they are retrieved from the respective business directories. This makes plugin retrieval more flexible and diverse, demonstrating stronger adaptability to different business needs. For streaming data processing frameworks, only the structure of the business plugin packages needs to be exposed, allowing developers from each business line to write their own business plugin packages based on this structure. For example, given a class object, developers from each business line only need to inherit from the corresponding class object and perform corresponding custom designs in specific method functions.

[0047] Implementing various business functions through a plug-in approach allows for more flexible implementation methods and offers advantages such as ease of deployment and convenient operation and maintenance. When processing streaming data, it is necessary to first obtain the business plug-in package to provide a foundation for subsequent processing.

[0048] Step 120: parse each business plugin package to determine the business topology diagram of each search sub-business.

[0049] When implementing business functions, business plugins have their own unique workflows. By parsing each business plugin package, the business plugins and their related configuration information can be derived, thereby determining the business topology diagram of each search sub-business. In this way, the business topology diagram of the search sub-business is constructed.

[0050] Step 130: Process the streaming data of the corresponding search sub-businesses according to each business topology map, and reconstruct the overall business topology map according to each business topology map.

[0051] In streaming data processing, the components of the business are first initialized to complete the corresponding configuration. After determining the search sub-business topology, the relationships between the topology and related topologies can be further defined, resulting in a global relational graph and constructing the overall business topology. This allows the implementation process of business functions to be displayed in an intuitive topology diagram, facilitating global monitoring and control over the process.

[0052] It can be seen that, as Figure 1 The method described above can determine the business topology map of the search sub-business based on its business logic, then reconstruct the overall business topology map based on the relationships between the various business topology maps, and process the streaming data of the corresponding search sub-businesses according to the business topology map. In the face of massive amounts of data and complex processing flows, it can quickly construct the overall business topology map from the related business topology maps, thus providing a global perspective to examine the operational status of each stage of the business, which is beneficial for improving search business data processing and decision-making.

[0053] In one embodiment of the present invention, the above method, in which streaming data of corresponding search sub-services are processed according to each service topology map, includes: initializing a streaming data distribution component Spout and several streaming data processing components Bolt; determining the component relationship between Spout and Bolt according to the service topology map; and executing each task determined according to the service plugin in Spout and Bolt respectively.

[0054] Here, Spout and Bolt borrow concepts from the Storm framework, namely, to distribute and process data separately. However, this does not mean that the implementation needs to be consistent with the Storm framework. For example, Storm is implemented in Java, which can easily cause JVM (Java Virtual Machine) GC (garbage collection) lag issues. Therefore, C++ or other languages ​​can be used to implement the embodiments of this invention.

[0055] Business plugins have their own implementation flow when implementing business functions. Parsing the business plugin package yields a business topology diagram. The structure of a business plugin package can include: business plugins, configuration information, and information about the libraries the business plugins depend on. For example, the structure of business plugin package 1 can include business plugins (plugins), configuration information such as etc., and information about the libraries (lib) that the business plugins depend on. This includes all the necessary information for the stable operation of the business plugin package. During streaming data processing, each component in the business is initialized first to complete the corresponding configuration work. The Spout component is responsible for distributing streaming data, while the Bolt component is responsible for the specific processing. After determining the business topology diagram, the relationships within and between related topologies can be further determined, resulting in a global relational graph. Ultimately, the business functions are implemented through this overall topology diagram and the Spout and Bolt components within the plugins. This allows the implementation flow of business functions to be displayed in an intuitive topology diagram format, facilitating global monitoring and control of the business function implementation process.

[0056] In one embodiment of the present invention, the above method can employ a containerized approach, instantiating corresponding business plugins within a container based on a business plugin package to implement the functions corresponding to each business. Containers are increasingly popular among business users due to their flexibility, high resource utilization, and robustness, leading to their widespread application. Embodiments of the present invention can interface with existing container architectures, for example, by cooperating with existing container management servers at the container level to achieve scheduling. Using containers as the technological foundation in the worker nodes, when implementing business functions, corresponding business plugins are instantiated based on the business plugin package deployed within the container, thereby realizing the business functions. This ensures that each business is independent while allowing for data flow determination based on business relationships. Furthermore, it enables flexible deployment of the number of containers based on task priorities, resource consumption, and other characteristics, resulting in a fairer and more reasonable allocation and scheduling, effectively improving the efficiency of streaming data processing.

[0057] In one embodiment of the present invention, the above method, in which the overall business topology map is constructed based on each business topology map, includes: identifying the relationship between the Spout of each search sub-business and the Bolt of other search sub-businesses, and constructing the overall business topology map based on the identified relationship.

[0058] For the search business, the Spouts of each search sub-business are related to the Bolts of other search sub-businesses, and there are also relationships between the topologies of each search sub-business. Based on these relationships, the topologies are reconstructed and combined to construct the overall business topology. In this way, the functional implementation of the search business can be displayed in an intuitive topology diagram, which is beneficial for global monitoring and control of the business function implementation process.

[0059] In one embodiment of the present invention, the method described above, in which the relationship between the Spout of each search sub-service and the Bolt of other search sub-services is identified, includes: determining whether the data source of the Spout matches the data output of the Bolt; if they match, then a relationship exists; the data source and the data output are determined based on the message queue.

[0060] For example, if the data source of a Spout in one search sub-service corresponds exactly to the data output of a Bolt in another search sub-service, then the Spout's data source and the Bolt's data output are considered to match, indicating a relationship. A message queue is a container that stores messages during transmission. It's used to improve system performance, smooth out peak loads, and reduce system coupling through asynchronous processing. Message queues can be used to define the relationship between the data source and the data output. This clarifies the relationship between the search sub-services and the message transmission method.

[0061] In one embodiment of the present invention, the component relationship in the above method includes at least one of the following: random allocation of streaming data, allocation of streaming data by domain, and broadcast streaming data.

[0062] Randomly distributed streaming data is a common and frequently used data stream grouping method. It includes a parameter (data source component), where the data source sends tuples to randomly selected Bolts, ensuring each consumer receives approximately the same number of tuples, resulting in a more even distribution of tasks. Domain-based streaming data distribution allows control over how tuples are sent to different Bolts based on one or more domains of the tuple. This allows sets of values ​​with the same domain combination to be sent to the same Bolt. For example, with a word counter, if the data stream is grouped by the word domain, the word-normalizer Bolt (used for word normalization) will only send tuples containing the same word to the same word-counter Bolt instance. Broadcast streaming data copies the tuples to each receiving instance. This grouping method is used to send signals to all Bolts. For example, to refresh a cache, a refresh cache signal can be sent to all Bolts. Of course, the above component relationships are just examples; different component relationships can be used to form the topology in specific implementations.

[0063] In one embodiment of the present invention, the method described above, initializing the streaming data distribution component Spout and several streaming data processing components Bolt, includes: running Spout and Bolt in a single-process multi-threaded model, and determining the number of Spout task threads and Bolt task threads according to a preset degree of parallelism; executing each task determined according to the business plugin in Spout and Bolt respectively includes: starting several execution threads in the started working process, executing the task processing logic of Spout and Bolt through the execution threads, and implementing the corresponding Spout task threads and Bolt task threads.

[0064] A thread is the smallest unit of program execution, while a process is the smallest unit of resource allocation. Several related execution threads can be started within a worker process. Since the Spouts and Bolts within these threads have their own task processing logic, corresponding Spout and Bolt task threads are implemented based on this logic. In this way, the corresponding Spout and Bolt task threads are implemented through Spouts and Bolts and their internal task processing logic, thus achieving program execution. Both Spouts and Bolts run in a single-process, multi-threaded model, with the number of threads for each component specified by the initial parallelism configuration, achieving fine-grained resource allocation.

[0065] In one embodiment of the present invention, the execution thread in the above method includes a task dispatch thread; executing the task processing logic of Spout and Bolt through the execution thread to realize the corresponding Spout task thread and Bolt task thread includes: realizing task dispatch between Spout task thread and Bolt task thread, and between each Bolt task thread through the task dispatch thread.

[0066] In the task process worker, each Spout / Bolt thread can be called a task. Tasks from different Spouts / Bolts can share a physical thread, called the executor. A separate OutputDispatcher thread then distributes tasks to different Bolts.

[0067] In one embodiment of the present invention, the task distribution in the above method is triggered by an event mechanism. The worker process internally implements asynchronous task processing via the event mechanism, thereby improving operational efficiency.

[0068] In one embodiment of the present invention, the Bolt in the above method includes synchronous Bolt and asynchronous Bolt; the task distribution of asynchronous Bolt is specifically implemented through an event notification mechanism. Asynchronous Bolt task distribution is achieved through an event notification mechanism, and it also supports asynchronous task processing, internally maintaining independent execution units.

[0069] In one embodiment of the present invention, in the above method, the events are implemented using the C++ language, and Bolt includes third-party Bolts implemented in languages ​​other than C++; the read events and write events of the task distribution of the third-party Bolt are separated.

[0070] Specifically, the entire streaming framework can also be implemented in C++. As mentioned earlier, most current streaming frameworks are implemented in Java, which easily leads to JVM GC lag issues, serious resource misuse problems caused by multi-service deployments, and an inability to monitor the internal operation status of services, thus failing to meet business performance and quality tracking requirements. A streaming framework implemented in C++ can effectively solve this problem. Furthermore, it can support third-party Bolts developed in other programming languages, also triggered by an event mechanism. The difference lies in separating reading and writing when communicating with third-party Bolts. Each task is written to a corresponding buffer, and the read event is only processed when the third-party Bolt returns, thus avoiding task blocking.

[0071] In one embodiment of the present invention, in the above method, Spout and Bolt exit the event loop first, and then exit the corresponding task thread upon exiting. Specifically, considering the issue of task backlog, a buffering mechanism is used for safe exit. Each running thread is stopped only after each Spout / Bolt task has been processed. Alternatively, a time threshold can be set, such as forcing a timeout exit if the thread has not exited within 60 seconds.

[0072] In one embodiment of the present invention, the data transmitted by task distribution in the above method is based on a serialized data structure. For serialization and deserialization of various data structures such as Bool, Int, Double, String, Vector, and Array, it is only necessary to encapsulate them into the corresponding ObjectPtr to achieve message transmission.

[0073] In one embodiment of the present invention, the data structure in the above method includes a data tracking flag to achieve the reliability of task processing through a data tracking mechanism.

[0074] In terms of reliability, a bitset data structure is encapsulated within a task to track the status of each Bolt. Each Bolt node name corresponds to a bit in the tracing structure (i.e., a flag bit in the data structure). A state transition mechanism implements three different handling methods: emit, ack, and fail. When a failure occurs, three mechanisms are supported: 0: retransmit the message until it is ACKed; 1: retransmit the message until the maximum number of retries is reached; 2: do not retransmit the message, processing a maximum of one attempt. Furthermore, for the first and second types of failures, reentrancy or non-reentrancy mechanisms are also supported.

[0075] An exactly-once mechanism can be used to guarantee the atomicity of semantics, meaning that each task is executed exactly once through task tracing. With a failure-reentrant mechanism, failure information for failed tasks can be recorded in different tracing bits. When the output dispatcher distributes tasks, it avoids already processed Bolts based on their tracing bits.

[0076] In one embodiment of the present invention, in the above method, the processing speed of the Spout is dynamically configured by a funnel algorithm, and / or the processing speed of the Spout is determined by backpressure at each level based on the processing speed of each Bolt; the backpressure at each level includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the upper level of the corresponding Bolt in the business topology diagram.

[0077] By employing rate limiting to dynamically configure the maximum processing speed of the Spout, the task failure rate can be reduced. Additionally, it supports tiered backpressure; when a downstream task slows down or encounters an error, the Bolt processing that task will decrease in speed. This backpressure upstream ultimately reduces the data source Spout's fetch speed, achieving a balance in streaming processing speed. This can be controlled by setting the task queue size.

[0078] In practical implementation, business users can inherit synchronous Bolt, asynchronous Bolt, and Bolt supporting third-party languages ​​through class inheritance, and construct the streaming processing topology graph required by their business needs. Business users can also define the topology graph structure by loading different configuration files.

[0079] In one embodiment of the present invention, the method further includes: displaying a general business topology diagram through a front-end page.

[0080] To provide a more intuitive, global perspective on the operational status, the corresponding scaling information is displayed on the front-end page. This ensures that scaling information is displayed promptly on the front-end page, facilitating observation from a holistic perspective.

[0081] In one embodiment of the present invention, in order to more intuitively observe the operating status from a global perspective, the corresponding scaling information can be displayed through the front-end page. This achieves timely display of scaling information through the front-end page, facilitating observation from a global perspective.

[0082] In one embodiment of the present invention, the above method can obtain node status information generated through a heartbeat mechanism from a distributed database, thereby obtaining performance information of each worker node in a normal state. The heartbeat mechanism is a mechanism that periodically sends a custom structure, such as a heartbeat packet, to let the other party know that it is still alive, ensuring the validity of the connection. To obtain node status information from the distributed database, a heartbeat mechanism can be set up to obtain node status information and performance information of each worker node in a normal state. This achieves performance detection.

[0083] In one embodiment of the present invention, a preset alarm mechanism can be set and any one of the performance information and node status information can be preset to trigger an alarm. When insufficient performance or abnormal node status occurs during operation, the corresponding alarm information is sent in a preset manner, thereby realizing effective monitoring and early warning of performance information and node status.

[0084] In one embodiment of the present invention, in order to enable the system to have intelligent elastic scheduling capabilities, realize real-time monitoring of service redundancy, and automatically scale up and down as needed, it is necessary to collect the performance information of the corresponding services, and then summarize and make decisions based on the collected service information to obtain the corresponding scaling information, thereby adjusting the number of containers. Specifically, firstly, the plugin and configuration information are detected based on the front-end page, and then a Spout is started to load the detection configuration information. The services are divided according to different types, and a service detection topology is constructed for each. Next, several groups of service Bolts and a scaling Bolt corresponding to each service are started. Each group of service Bolts includes a collection Bolt and a decision Bolt. The Bolt is responsible for receiving the information stream emitted by the Spout component and completing the specific processing logic. In complex service logic, multiple Bolt components can be connected in series, each Bolt component implementing its own different functions, thereby realizing the overall processing logic. The execution of corresponding services generally has specific IP patterns; therefore, a corresponding service IP list can be assigned to each collection Bolt so that each collection Bolt can collect the performance information of the corresponding service according to the corresponding IP list. After obtaining the performance information of the relevant service, it is passed to the corresponding decision Bolt. Then, the scaling Bolt summarizes the decision information from each decision Bolt and generates corresponding scaling information. Finally, the number of containers is adjusted according to the scaling information. In this way, the intelligent elastic scheduling function is fully implemented, which can monitor the redundancy of the service in real time and automatically scale up or down as needed.

[0085] As can be seen, this scheduling actually forms two levels of scheduling: business-level scheduling and container-level scheduling. Existing streaming processing frameworks often only perform container-level scheduling, which fails to grasp the actual state of each business and results in poor resource utilization. The advantage of this invention lies in its clever use of streaming data processing methods for decision-making in a streaming processing scenario, thereby controlling the state of different businesses. It can not only schedule the overall number of containers but also schedule the number of containers for each business separately, maximizing resource utilization.

[0086] Figure 2 A schematic diagram of a streaming data processing apparatus for a search service according to an embodiment of the present invention is shown. Figure 2 As shown, the device 200 includes:

[0087] Acquisition unit 210 is adapted to acquire the business plugin packages of each search sub-business;

[0088] Business plugin packages can come from the platform or be uploaded by users. Users can upload them through the front-end page, which receives the plugin packages for each business and stores them in the corresponding business directories in distributed storage. When these plugin packages are needed, they are retrieved from the respective business directories. This makes plugin retrieval more flexible and diverse, demonstrating stronger adaptability to different business needs. For streaming data processing frameworks, only the structure of the business plugin packages needs to be exposed, allowing developers from each business line to write their own business plugin packages based on this structure. For example, given a class object, developers from each business line only need to inherit from the corresponding class object and perform corresponding custom designs in specific method functions.

[0089] Implementing various business functions through a plug-in approach allows for more flexible implementation methods and offers advantages such as ease of deployment and convenient operation and maintenance. When processing streaming data, it is necessary to first obtain the business plug-in package to provide a foundation for subsequent processing.

[0090] The parsing unit 220 is suitable for parsing each business plugin package and determining the business topology diagram of each search sub-business;

[0091] When implementing business functions, business plugins have their own unique workflows. By parsing each business plugin package, the business plugins and their related configuration information can be derived, thereby determining the business topology diagram of each search sub-business. In this way, the business topology diagram of the search sub-business is constructed.

[0092] The construction unit 230 is adapted to process the streaming data of the corresponding search sub-business according to each business topology map, and to reconstruct the overall business topology map according to each business topology map.

[0093] In streaming data processing, the components of the business are first initialized to complete the corresponding configuration. After determining the search sub-business topology, the relationships between the topology and related topologies can be further defined, resulting in a global relational graph and constructing an approximate overall topology. This allows the implementation process of business functions to be displayed in an intuitive topology diagram, facilitating global monitoring and control over the implementation process.

[0094] It can be seen that, as Figure 2The described device can determine the business topology map of the search sub-business based on its business logic, then reconstruct the overall business topology map based on the relationships between the various business topology maps, and process the streaming data of the corresponding search sub-businesses according to the business topology map. In the face of massive amounts of data and complex processing flows, it can quickly construct the overall business topology map from the related business topology maps, thereby providing a global perspective to examine the operational status of each stage of the business operation, which is beneficial for improving search business data processing and decision-making.

[0095] In one embodiment of the present invention, in the above-described apparatus, the construction unit 230 is adapted to initialize a streaming data distribution component Spout and a plurality of streaming data processing components Bolt, determine the component relationship between Spout and Bolt according to the business topology diagram, and execute each task determined according to the business plugin in Spout and Bolt respectively.

[0096] Here, Spout and Bolt borrow concepts from the Storm framework, namely, to distribute and process data separately. However, this does not mean that the implementation needs to be consistent with the Storm framework. For example, Storm is implemented in Java, which can easily cause JVM (Java Virtual Machine) GC (garbage collection) lag issues. Therefore, C++ or other languages ​​can be used to implement the embodiments of this invention.

[0097] Business plugins have their own implementation flow when implementing business functions. Parsing the business plugin package yields a business topology diagram. The structure of a business plugin package can include: business plugins, configuration information, and information about the libraries the business plugins depend on. For example, the structure of business plugin package 1 can include business plugins (plugins), configuration information such as etc., and information about the libraries (lib) that the business plugins depend on. This includes all the necessary information for the stable operation of the business plugin package. During streaming data processing, each component in the business is initialized first to complete the corresponding configuration work. The Spout component is responsible for distributing streaming data, while the Bolt component is responsible for the specific processing. After determining the business topology diagram, the relationships within and between related topologies can be further determined, resulting in a global relational graph. Ultimately, the business functions are implemented through this overall topology diagram and the Spout and Bolt components within the plugins. This allows the implementation flow of business functions to be displayed in an intuitive topology diagram format, facilitating global monitoring and control of the business function implementation process.

[0098] In one embodiment of the present invention, the above-described apparatus can employ a containerized approach, instantiating corresponding business plugins within containers based on business plugin packages to implement the functions corresponding to each business. Containers are increasingly popular among business users due to their flexibility, high resource utilization, and robustness, leading to their widespread application. Embodiments of the present invention can interface with existing container architectures, for example, by cooperating with existing container management servers at the container level to achieve scheduling. Using containers as the technological foundation in the worker nodes, when implementing business functions, corresponding business plugins are instantiated based on the business plugin packages deployed within the containers, thereby realizing the business functions. This ensures that each business is independent while allowing for the determination of data flow between them based on business relationships. Furthermore, it enables flexible deployment of the number of containers based on different task priorities and resource usage characteristics, resulting in a fairer and more reasonable allocation and scheduling, effectively improving the efficiency of streaming data processing.

[0099] In one embodiment of the present invention, the construction unit 230 in the above-described apparatus is adapted to identify the relationship between the Spout of each search sub-service and the Bolt of other search sub-services, and to construct a general service topology map based on the identified relationship.

[0100] For the search business, the Spouts of each search sub-business are related to the Bolts of other search sub-businesses, and there are also relationships between the topologies of each search sub-business. Based on these relationships, the topologies are reconstructed and combined to construct the overall business topology. In this way, the functional implementation of the search business can be displayed in an intuitive topology diagram, which is beneficial for global monitoring and control of the business function implementation process.

[0101] In one embodiment of the present invention, in the above-described apparatus, the construction unit 230 is adapted to determine whether the data source of the Spout matches the data output of the Bolt, and if they match, there is an association relationship; the data source and the data output are determined based on the message queue.

[0102] For example, if the data source of a Spout in one search sub-service corresponds exactly to the data output of a Bolt in another search sub-service, then the Spout's data source and the Bolt's data output are considered to match, indicating a relationship. A message queue is a container that stores messages during transmission. It's used to improve system performance, smooth out peak loads, and reduce system coupling through asynchronous processing. Message queues can be used to define the relationship between the data source and the data output. This clarifies the relationship between the search sub-services and the message transmission method.

[0103] In one embodiment of the present invention, the component relationships in the above-described apparatus include at least one of the following: random allocation of streaming data, domain allocation of streaming data, and broadcast streaming data.

[0104] Randomly distributed streaming data is a common and frequently used data stream grouping method. It includes a parameter (data source component), where the data source sends tuples to randomly selected Bolts, ensuring each consumer receives approximately the same number of tuples, resulting in a more even distribution of tasks. Domain-based streaming data distribution allows control over how tuples are sent to different Bolts based on one or more domains of the tuple. This allows sets of values ​​with the same domain combination to be sent to the same Bolt. For example, with a word counter, if the data stream is grouped by the word domain, the word-normalizer Bolt (used for word normalization) will only send tuples containing the same word to the same word-counter Bolt instance. Broadcast streaming data copies the tuples to each receiving instance. This grouping method is used to send signals to all Bolts. For example, to refresh a cache, a refresh cache signal can be sent to all Bolts. Of course, the above component relationships are just examples; different component relationships can be used to form the topology in specific implementations.

[0105] In one embodiment of the present invention, in the above-described apparatus, the construction unit 230 is adapted to run Spout and Bolt in a single-process multi-threaded model, and determines the number of Spout task threads and Bolt task threads according to a preset parallelism; it starts several execution threads in the started working process, and executes the task processing logic of Spout and Bolt through the execution threads, thereby realizing the corresponding Spout task threads and Bolt task threads.

[0106] A thread is the smallest unit of program execution, while a process is the smallest unit of resource allocation. Several related execution threads can be started within a worker process. Since the Spouts and Bolts within these threads have their own task processing logic, corresponding Spout and Bolt task threads are implemented based on this logic. In this way, the corresponding Spout and Bolt task threads are implemented through Spouts and Bolts and their internal task processing logic, thus achieving program execution. Both Spouts and Bolts run in a single-process, multi-threaded model, with the number of threads for each component specified by the initial parallelism configuration, achieving fine-grained resource allocation.

[0107] In one embodiment of the present invention, the execution thread in the above-described apparatus includes: a task distribution thread; and a business unit 220, adapted to implement task distribution between the Spout task thread and the Bolt task thread, and among the Bolt task threads, through the task distribution thread.

[0108] In the task process worker, each Spout / Bolt thread can be called a task. Tasks from different Spouts / Bolts can share a physical thread, called the executor. A separate OutputDispatcher thread then distributes tasks to different Bolts.

[0109] In one embodiment of the present invention, in the above-described apparatus, task distribution is triggered by an event mechanism. The worker process internally implements asynchronous task processing triggered by the event mechanism, thereby improving operational efficiency.

[0110] In one embodiment of the present invention, the Bolt in the above-described apparatus includes synchronous Bolts and asynchronous Bolts; the task distribution of asynchronous Bolts is specifically triggered by an event notification mechanism. Asynchronous Bolts distribute tasks through an event notification mechanism, while also supporting asynchronous task processing, and internally maintain independent execution units.

[0111] In one embodiment of the present invention, in the above-described apparatus, the events are implemented using the C++ language, and the Bolt includes third-party Bolts implemented in languages ​​other than C++; the read events and write events of the task distribution of the third-party Bolts are separated.

[0112] Specifically, the entire streaming framework can also be implemented in C++. As mentioned earlier, most current streaming frameworks are implemented in Java, which easily leads to JVM GC lag issues, serious resource misuse problems caused by multi-service deployments, and an inability to monitor the internal operation status of services, thus failing to meet business performance and quality tracking requirements. A streaming framework implemented in C++ can effectively solve this problem. Furthermore, it can support third-party Bolts developed in other programming languages, also triggered by an event mechanism. The difference lies in separating reading and writing when communicating with third-party Bolts. Each task is written to a corresponding buffer, and the read event is only processed when the third-party Bolt returns, thus avoiding task blocking.

[0113] In one embodiment of the present invention, in the above-described device, Spout and Bolt exit the event loop first, and then exit the corresponding task thread upon exiting. Specifically, considering the issue of task backlog, a buffering mechanism is used for safe exit. Each running thread is stopped only after each Spout / Bolt task has been completed. Alternatively, a time threshold can be set, such as forcing a timeout exit if the task has not exited within 60 seconds.

[0114] In one embodiment of the present invention, in the above-described apparatus, the data transmitted by task distribution is based on a serialized data structure. For serialization and deserialization of various data structures such as Bool, Int, Double, String, Vector, and Array, it is sufficient to encapsulate them into corresponding ObjectPtrs to achieve message transmission.

[0115] In one embodiment of the present invention, the data structure in the above-described apparatus includes a data tracking flag to achieve reliability of task processing through a data tracking mechanism.

[0116] In terms of reliability, a bitset data structure is encapsulated within a task to track the status of each Bolt. Each Bolt node name corresponds to a bit in the tracing structure (i.e., a flag bit in the data structure). A state transition mechanism implements three different handling methods: emit, ack, and fail. When a failure occurs, three mechanisms are supported: 0: retransmit the message until it is ACKed; 1: retransmit the message until the maximum number of retries is reached; 2: do not retransmit the message, processing a maximum of one attempt. Furthermore, for the first and second types of failures, reentrancy or non-reentrancy mechanisms are also supported.

[0117] An exactly-once mechanism can be used to guarantee the atomicity of semantics, meaning that each task is executed exactly once through task tracing. With a failure-reentrant mechanism, failure information for failed tasks can be recorded in different tracing bits. When the output dispatcher distributes tasks, it avoids already processed Bolts based on their tracing bits.

[0118] In one embodiment of the present invention, in the above-described apparatus, the processing speed of the Spout is dynamically configured by a funnel algorithm, and / or the processing speed of the Spout is determined by backpressure at each level based on the processing speed of each Bolt; the backpressure at each level includes: setting the length of the task queue of each Bolt, and when the task queue is full, providing feedback to the upper level of the corresponding Bolt in the business topology diagram.

[0119] By employing rate limiting to dynamically configure the maximum processing speed of the Spout, the task failure rate can be reduced. Additionally, it supports tiered backpressure; when a downstream task slows down or encounters an error, the Bolt processing that task will decrease in speed. This backpressure upstream ultimately reduces the data source Spout's fetch speed, achieving a balance in streaming processing speed. This can be controlled by setting the task queue size.

[0120] In practical implementation, business users can inherit synchronous Bolt, asynchronous Bolt, and Bolt supporting third-party languages ​​through class inheritance, and construct the streaming processing topology graph required by their business needs. Business users can also define the topology graph structure by loading different configuration files.

[0121] In one embodiment of the present invention, the device further includes: displaying a general business topology diagram through a front-end page.

[0122] To provide a more intuitive, global perspective on the operational status, the corresponding scaling information is displayed on the front-end page. This ensures that scaling information is displayed promptly on the front-end page, facilitating observation from a holistic perspective.

[0123] In one embodiment of the present invention, in order to more intuitively observe the operating status from a global perspective, the corresponding scaling information can be displayed through the front-end page. This achieves timely display of scaling information through the front-end page, facilitating observation from a global perspective.

[0124] In one embodiment of the present invention, the above-described apparatus can obtain node status information generated through a heartbeat mechanism from a distributed database, thereby acquiring performance information of each worker node in a normal state. The heartbeat mechanism is a mechanism that periodically sends a custom structure, such as a heartbeat packet, to let the other party know that it is still alive, ensuring the validity of the connection. To obtain node status information from the distributed database, a heartbeat mechanism can be set up to obtain node status information and performance information of each worker node in a normal state. This achieves performance detection.

[0125] In one embodiment of the present invention, a preset alarm mechanism can be set and any one of the performance information and node status information can be preset to trigger an alarm. When insufficient performance or abnormal node status occurs during operation, the corresponding alarm information is sent in a preset manner, thereby realizing effective monitoring and early warning of performance information and node status.

[0126] In one embodiment of the present invention, in order to enable the system to have intelligent elastic scheduling capabilities, realize real-time monitoring of service redundancy, and automatically scale up and down as needed, it is necessary to collect the performance information of the corresponding services, and then summarize and make decisions based on the collected service information to obtain the corresponding scaling information, thereby adjusting the number of containers. Specifically, firstly, the plugin and configuration information are detected based on the front-end page, and then a Spout is started to load and detect the configuration information. The services are divided according to different types, and a service detection topology is constructed for each. Next, several groups of service Bolts and a scaling Bolt corresponding to each service are started. Each group of service Bolts includes a collection Bolt and a decision Bolt. The Bolt is responsible for receiving the information stream emitted by the Spout component and completing the specific processing logic. In complex service logic, multiple Bolt components can be connected in series, with each Bolt component implementing its own different functions to achieve the overall processing logic. Executing the corresponding service generally has specific IP patterns; therefore, a corresponding service IP list can be assigned to each collection Bolt so that each collection Bolt can collect the performance information of the corresponding service according to the corresponding IP list. After obtaining the performance information of the relevant service, it is passed to the corresponding decision Bolt. Then, the scaling Bolt summarizes the decision information from each decision Bolt and generates corresponding scaling information. Finally, the number of containers is adjusted according to the scaling information. In this way, the intelligent elastic scheduling function is fully implemented, which can monitor the redundancy of the service in real time and automatically scale up or down as needed.

[0127] As can be seen, this scheduling actually forms two levels of scheduling: business-level scheduling and container-level scheduling. Existing streaming processing frameworks often only perform container-level scheduling, which fails to grasp the actual state of each business and results in poor resource utilization. The advantage of this invention lies in its clever use of streaming data processing methods for decision-making in a streaming processing scenario, thereby controlling the state of different businesses. It can not only schedule the overall number of containers but also schedule the number of containers for each business separately, maximizing resource utilization.

[0128] In summary, the technical solution of this invention involves acquiring the business plugin packages of each search sub-service; parsing each business plugin package to determine the business topology diagram of each search sub-service; processing the streaming data of the corresponding search sub-service based on each business topology diagram; and reconstructing the overall business topology diagram based on each business topology diagram. The beneficial effect is that the business topology diagram of each search sub-service is determined based on its business logic, and then the overall business topology diagram is reconstructed based on the relationships between the business topology diagrams. The streaming data of the corresponding search sub-services is then processed based on the business topology diagrams. In the face of massive amounts of data and complex processing flows, business topology diagrams with interrelationships can be quickly constructed into an overall business topology diagram, thereby providing a global perspective to examine the operational status of each stage of the business operation, which is beneficial for improving data processing and decision-making in the search business.

[0129] It should be noted that:

[0130] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0131] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0132] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0133] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0134] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0135] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the streaming data processing apparatus for search services according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0136] For example, Figure 3A schematic diagram of an electronic device according to an embodiment of the present invention is shown. The electronic device 300 includes a processor 310 and a memory 320 arranged to store computer-executable instructions (computer-readable program code). The memory 320 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 320 has a storage space 330 for storing computer-readable program code 331 for performing any of the method steps described above. For example, the storage space 330 for storing computer-readable program code may include various computer-readable program codes 331 respectively for implementing the various steps in the above methods. The computer-readable program code 331 can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. Such computer program products are typically, for example... Figure 4 The aforementioned computer-readable storage medium. Figure 4 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown. The computer-readable storage medium 400 stores computer-readable program code 331 for performing the method steps according to the present invention, which can be read by the processor 310 of an electronic device 300. When the computer-readable program code 331 is executed by the electronic device 300, it causes the electronic device 300 to perform the various steps of the method described above. Specifically, the computer-readable program code 331 stored in the computer-readable storage medium can perform the methods shown in any of the above embodiments. The computer-readable program code 331 can be compressed in a suitable form.

[0137] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0138] Embodiments of the present invention disclose A1, a streaming data processing method for search services, comprising:

[0139] Obtain the business plugin packages for each search sub-service;

[0140] Analyze each business plugin package to determine the business topology of each search sub-business;

[0141] Process the streaming data of the corresponding search sub-businesses based on each business topology map, and reconstruct the overall business topology map based on each business topology map.

[0142] A2. The method as described in A1, wherein processing the streaming data of the corresponding search sub-services according to each service topology map includes:

[0143] Initialize the streaming data distribution component Spout and several streaming data processing components Bolt, and determine the component relationship between Spout and Bolt according to the business topology diagram;

[0144] Each task, as determined by the business plugin, is executed in Spout and Bolt respectively.

[0145] A3. The method described in A2, wherein constructing the overall business topology map based on each business topology map includes:

[0146] Identify the relationships between the Spouts of each search sub-business and the Bolts of other search sub-businesses, and construct a general business topology map based on the identified relationships.

[0147] A4. The method as described in A3, wherein identifying the association between the Spout of each search sub-service and the Bolt of other search sub-services includes:

[0148] Determine if the Spout's data source matches the Bolt's data output; if they match, a relationship exists.

[0149] The data source and the data output are determined based on the message queue.

[0150] A5. The method as described in A2, wherein the component relationship includes at least one of the following:

[0151] Randomly allocated streaming data, domain-based allocated streaming data, and broadcast streaming data.

[0152] A6. The method as described in A2, wherein the initialization of the streaming data distribution component Spout and the plurality of streaming data processing components Bolt includes:

[0153] Run Spout and Bolt in a single-process multi-threaded model, and determine the number of Spout task threads and Bolt task threads according to the preset parallelism.

[0154] The execution of the tasks determined by the business plugin in Spout and Bolt respectively includes:

[0155] Several execution threads are started in the startup process. The execution threads execute the task processing logic of Spout and Bolt, thereby realizing the corresponding Spout task thread and Bolt task thread.

[0156] A7. The method described in A1, wherein the method further includes: displaying the overall business topology diagram through a front-end page.

[0157] Embodiments of the present invention also disclose B8, a streaming data processing apparatus for search services, comprising:

[0158] The acquisition unit is suitable for acquiring the business plugin packages of each search sub-business.

[0159] The parsing unit is suitable for parsing various business plugin packages and determining the business topology of each search sub-business.

[0160] The construction unit is suitable for processing streaming data of corresponding search sub-businesses based on each business topology map, and for reconstructing the overall business topology map based on each business topology map.

[0161] B9. The apparatus as described in B8, wherein the building unit is adapted to initialize a streaming data distribution component Spout and several streaming data processing components Bolt, and to determine the component relationship between Spout and Bolt according to the service topology diagram;

[0162] Each task, as determined by the business plugin, is executed in Spout and Bolt respectively.

[0163] B10. The apparatus as described in B8, wherein the building unit is adapted to identify the relationship between the Spout of each search sub-service and the Bolt of other search sub-services, and to build a general service topology map based on the identified relationship.

[0164] B11. The apparatus as described in B8, wherein the building unit is adapted to determine whether the data source of Spout matches the data output of Bolt, and if they match, there is an association relationship;

[0165] The data source and the data output are determined based on the message queue.

[0166] B12. The apparatus as described in B9, wherein the component relationships include at least one of the following:

[0167] Randomly allocated streaming data, domain-based allocated streaming data, and broadcast streaming data.

[0168] B13. The apparatus as described in B9, wherein the building unit is adapted to run Spout and Bolt in a single-process multi-threaded model, and determines the number of Spout task threads and Bolt task threads according to a preset degree of parallelism; starts several execution threads in the started working process, and executes the task processing logic of Spout and Bolt through the execution threads, thereby realizing the corresponding Spout task threads and Bolt task threads.

[0169] B14. The apparatus as described in B8, further comprising: displaying the overall business topology diagram via a front-end page.

[0170] Embodiments of the present invention also disclose C15, an electronic device comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform a method as described in any one of A1-A7.

[0171] Embodiments of the present invention also disclose D16, a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the method as described in any one of A1-A7.

Claims

1. A method for processing streaming data of a search service, comprising: obtaining service plug-in packages of each search sub-service; parsing each service plug-in package to determine a service topology of each search sub-service; processing streaming data of each search sub-service according to the service topology, and reconstructing a total service topology according to the service topology; wherein the processing of the streaming data of each search sub-service according to the service topology comprises: initializing a streaming data distribution component Spout and a plurality of streaming data processing components Bolts, and determining a component relationship between the Spout and the Bolts according to the service topology; executing each task determined according to the service plug-in in the Spout and the Bolts; wherein the functions corresponding to each service are implemented according to the service plug-in package by instantiating the corresponding service plug-in in a container according to a containerization method.

2. The method of claim 1, wherein, the reconstructing of the total service topology according to the service topology comprises: identifying an association relationship between the Spout of each search sub-service and the Bolts of other search sub-services, and constructing the total service topology according to the identified association relationship.

3. The method of claim 2, wherein, the identifying of the association relationship between the Spout of each search sub-service and the Bolts of other search sub-services comprises: judging whether a data source of the Spout matches a data output of the Bolts, and if so, there is an association relationship; the data source and the data output are determined according to a message queue.

4. The method of claim 1, wherein, the component relationship comprises at least one of the following: randomly distributing streaming data, distributing streaming data by domain, and broadcasting streaming data.

5. The method of claim 1, wherein, the initializing of the streaming data distribution component Spout and the plurality of streaming data processing components Bolts comprises: running the Spout and the Bolts in a single-process multi-thread model, and determining the number of Spout task threads and Bolt task threads according to a preset parallelism degree; the executing of each task in the Spout and the Bolts according to the service plug-in comprises: starting a plurality of execution threads in a started worker process, executing the task processing logic of the Spout and the Bolts through the execution threads, and implementing the corresponding Spout task threads and Bolt task threads.

6. The method of claim 1, wherein, The method further comprises: displaying the total service topology through a front-end page. 7.A device for processing streaming data of a search service, comprising: an obtaining unit adapted to obtain service plug-in packages of each search sub-service; a parsing unit adapted to parse each service plug-in package to determine a service topology of each search sub-service; a constructing unit adapted to process streaming data of each search sub-service according to the service topology, and reconstruct a total service topology according to the service topology. The constructing unit is adapted to initialize a stream data distribution component Spout and a plurality of stream data processing components Bolts, determine a component relationship between the Spout and the Bolts according to the business topology graph, and execute respective tasks determined according to the business plug-ins in the Spout and the Bolts respectively; wherein, according to a containerization method, respective business plug-ins are instantiated according to the business plug-in packages in the containers to implement functions corresponding to the business plug-ins.

8. The apparatus of claim 7, wherein the constructing unit is adapted to identify an association relationship between a Spout of each search sub-business and a Bolt of another search sub-business, and construct a total business topology graph according to the identified association relationship.

9. The apparatus of claim 8, wherein the constructing unit is adapted to judge whether a data source of the Spout matches a data output of the Bolt, and if the data source matches the data output, the association relationship exists. The data source and the data output are determined according to a message queue.

10. The apparatus of claim 7, wherein the component relationship comprises at least one of the following: randomly distributing stream data, distributing stream data by domain, and broadcasting stream data.

11. The apparatus of claim 7, wherein the constructing unit is adapted to run the Spout and the Bolts in a single-process multi-thread model, determine a number of Spout task threads and Bolt task threads according to a preset parallelism degree, start a plurality of execution threads in a started work process, execute task processing logic of the Spout and the Bolts through the execution threads, and implement respective Spout task threads and Bolt task threads through the execution threads.

12. The apparatus of claim 7, further comprising: The total business topology graph is displayed through a front-end page.

13. An electronic device, comprising: The electronic device includes a processor and a memory arranged to store computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-6.

14. A computer readable storage medium, wherein, The computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method of any of claims 1-6.

Images