Dynamic monitoring and scheduling method for multi-source heterogeneous computer data flow

Abstract

The application relates to the technical field of data processing, and discloses a dynamic monitoring and scheduling method for multi-source heterogeneous computer data flow, which comprises the following steps: analyzing multi-source heterogeneous data flow of a target computer to obtain data flow description information; performing resource quantitative evaluation on the data flow description information to obtain resource estimation quantity; dividing a to-be-scheduled batch of the multi-source heterogeneous data flow, and extracting data features of the data flow in the to-be-scheduled batch; selecting an adaptive processing engine type for the data flow in the to-be-scheduled batch in a preset processing engine template; constructing an execution topology graph according to the processing engine type and the data features, so as to determine a dynamic execution plan of the data flow in the to-be-scheduled batch, and distributing the data flow in the to-be-scheduled batch to a corresponding processing engine instance for execution, and monitoring the running state and resource occupation of the processing engine instance in real time; and the application can improve the dynamic monitoring and scheduling efficiency of computer data flow.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams. Background Technology

[0002] In the processing of multi-source heterogeneous computer data streams, existing technologies only parse data streams at the basic packet parsing level. They fail to comprehensively extract and integrate core characteristics such as source identifiers, data arrival rates, and data volumes, resulting in an inability to generate accurate data stream descriptions. This leads to fundamental data-level biases in subsequent resource assessment and scheduling decisions, making it difficult to adapt to the complex characteristics of multi-source heterogeneous data streams. Furthermore, existing technologies lack an effective mapping mechanism between historical data stream processing characteristics and actual resource consumption. Resource demand assessments are often based on empirical estimations without dynamic corrections incorporating real-time data characteristics. This results in significant discrepancies between estimated and actual resource consumption, easily leading to insufficient or redundant resource allocation and impacting the overall efficiency of data stream processing.

[0003] Existing data stream scheduling schemes mostly employ fixed scheduling batch division and processing engine selection methods, failing to dynamically adapt to the real-time data characteristics and resource estimates of the data streams. This makes it difficult to match the optimal processing engine to data streams with different protocol types and data states. Furthermore, during the execution plan formulation phase, the data dependencies and parallelism requirements between processing nodes are not fully considered, resulting in an unreasonable execution topology design. This leads to problems such as node load imbalance and data transmission link disruptions during data stream processing. In addition, existing monitoring mechanisms only provide post-processing feedback on data stream results, failing to sample and monitor the running status and resource usage of processing engine instances in real time. This makes it difficult to promptly detect and adjust problems such as unreasonable resource allocation and engine malfunctions during processing, further reducing the overall effectiveness of monitoring and scheduling multi-source heterogeneous data streams. Summary of the Invention

[0004] This invention provides a dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams, comprising: S1. Analyze the multi-source heterogeneous data stream of the target computer to obtain the data stream description information of the target computer; S2. Based on the mapping relationship between the processing characteristics of historical data streams and the actual resource consumption of the target computer, the resource quantification assessment of the data stream description information is performed to obtain the estimated resource amount of the multi-source heterogeneous data stream. S3. According to the preset scheduling period, the multi-source heterogeneous data stream is divided into batches to be scheduled, and the data features of the data streams in the batches to be scheduled are extracted. S4. Based on the data characteristics and the estimated resource quantity, select an appropriate processing engine type for the data stream in the batch to be scheduled from the preset processing engine template. S5. Based on the processing engine type and the data characteristics, construct the execution topology diagram of the data flow within the batch to be scheduled, so as to determine the dynamic execution plan of the data flow within the batch to be scheduled; S6. According to the dynamic execution plan, the data streams in the batch to be scheduled are distributed to the corresponding processing engine instances for processing, and the running status and resource usage of the processing engine instances are monitored in real time.

[0006] In a preferred embodiment, parsing the multi-source heterogeneous data streams of the target computer to obtain data stream description information of the target computer includes: Deep protocol stack parsing is performed on the raw data packets of the target computer, and the multi-source heterogeneous data streams in the raw data packets are extracted. The source identifier of the multi-source heterogeneous data stream is obtained by extracting the characteristic identifier from the transport layer header of the multi-source heterogeneous data stream. Based on the source identifier, deep protocol parsing is performed on the application layer protocol header of the multi-source heterogeneous data stream to obtain the application layer metadata of the multi-source heterogeneous data stream. The data packet arrival frequency is statistically analyzed on the application layer metadata to obtain the data arrival rate of the multi-source heterogeneous data stream, and the load length is aggregated and accumulated on the application layer metadata to obtain the data volume of the multi-source heterogeneous data stream. By integrating the source identifier, the data arrival rate, and the data volume, the data stream description information of the target computer is obtained.

[0007] In a preferred embodiment, the resource quantification assessment of the data stream description information based on the mapping relationship between the processing characteristics of historical data streams and the actual resource consumption of the target computer, to obtain the estimated resource quantity of the multi-source heterogeneous data stream, includes: Acquire the historical data stream of the target computer and extract the processing features of the historical data stream; Based on the processing features and the actual resource consumption records of the target computer, a consumption value association mapping is performed on the historical data stream to obtain a historical feature consumption association entry library of the target computer. The historical feature consumption associated entry library is clustered and representative samples are selected to construct the feature consumption benchmark reference set of the target computer; The data stream description information is mapped to the feature consumption benchmark reference set to obtain a target benchmark reference entry that matches the data stream description information. The resource estimates of the multi-source heterogeneous data stream are obtained by quantifying and evaluating the consumption value of the target benchmark reference item.

[0008] In a preferred embodiment, the step of quantifying and evaluating the consumption value of the target benchmark reference item to obtain the resource estimate of the multi-source heterogeneous data stream includes: By performing historical feature backtracking on the target benchmark reference entry, the benchmark resource consumption, reference data volume, and reference data arrival rate of the target benchmark reference entry are obtained; Real-time feature extraction is performed on the data stream description information to obtain the current data volume and current data arrival rate of the data stream description information; The resource estimate of the multi-source heterogeneous data stream is obtained by weighting and fusing the baseline resource consumption, the reference data volume, the reference data arrival rate, the current data volume, and the current data arrival rate with a dynamic correction factor.

[0009] In a preferred embodiment, the formula for calculating the resource estimate is as follows: ; In the formula, This indicates the estimated amount of resources. This represents the baseline resource consumption. Indicates the current data volume level. Indicates the magnitude of the reference data. This indicates the current data arrival rate. This indicates the arrival rate of the reference data. This indicates the preset data volume impact factor. This represents the preset rate influence factor.

[0010] In a preferred embodiment, the step of dividing the multi-source heterogeneous data streams into batches to be scheduled according to a preset scheduling period, and extracting data features of the data streams within each batch to be scheduled, includes: Based on a preset scheduling period, the multi-source heterogeneous data stream is divided into continuous time window slices to obtain a batch of data to be scheduled corresponding to the scheduling period. Data stream identifiers are collected from the batch of data to be scheduled to obtain a real-time data stream identifier list for the batch of data to be scheduled. Based on the real-time data stream identifier list, deep parsing of the application layer protocol type is performed on the data streams within the data batch to be scheduled to obtain the protocol type label and data stream status identifier of the data streams within the data batch to be scheduled. Multidimensional aggregation statistics are performed on the protocol type label and the data stream status identifier to obtain the data characteristics of the data stream within the batch to be scheduled.

[0011] In a preferred embodiment, the step of selecting a suitable processing engine type for the data stream within the batch to be scheduled, based on the data characteristics and the estimated resource quantity, from a preset processing engine template, includes: Based on the data characteristics, the preset processing engine templates are screened for functional adaptability to obtain candidate engine templates for the data streams in the batch to be scheduled. The resource capacity parameter of the candidate engine template is parsed to obtain the resource capacity level identifier corresponding to the candidate engine template; Based on the estimated resource quantity, the satisfaction level of the resource capacity level identifier is compared to obtain the secondary candidate engine template of the data stream in the batch to be scheduled. A comprehensive adaptation evaluation is performed on the secondary candidate engine templates to obtain the processing engine type that the data stream in the batch to be scheduled is adapted to.

[0012] In a preferred embodiment, constructing an execution topology graph of the data streams within the scheduled batch based on the processing engine type and the data characteristics to determine the dynamic execution plan for the data streams within the scheduled batch includes: Based on the processing engine type and the data characteristics, the data flow direction of the data stream in the batch to be scheduled is analyzed to obtain the processing nodes of the data stream in the batch to be scheduled and the data dependency chain between the nodes. The data dependency chain is hierarchically divided to obtain the topological hierarchy identifier of the processing node; Based on the parallelism requirement identifier in the data characteristics, the resource parallelism of the processing node is configured to obtain the set of parallel execution units of the processing node. The set of parallel execution units and the topology level identifier are combined to construct the execution topology graph of the data flow within the batch to be scheduled. By binding engine instances to the execution topology graph, a dynamic execution plan for the data flow within the batch to be scheduled is obtained.

[0013] In a preferred embodiment, the step of binding engine instances to the execution topology graph to obtain a dynamic execution plan for the data flow within the batch to be scheduled includes: Based on the processing nodes in the execution topology graph and the set of parallel execution units, a node load balancing prediction is performed on the preset distributed engine cluster to obtain a list of candidate engine instances for the data stream in the batch to be scheduled. The candidate engine instance list is evaluated by determining the matching degree of node data features to obtain the preferred engine instance list for the processing node; Based on the list of preferred engine instances and the data flow dependency relationship of the processing nodes, a data routing mapping is performed between the processing nodes and the preferred engine instances to obtain a binding relationship mapping table between the processing nodes and the target engine instances. The binding relationship mapping table is executed in a timing arrangement to generate a dynamic execution plan that includes node startup order, engine instance allocation information, and data stream push path.

[0014] In a preferred embodiment, the step of distributing the data streams within the scheduled batch to corresponding processing engine instances for processing according to the dynamic execution plan, and monitoring the running status and resource usage of the processing engine instances in real time, includes: Based on the engine instance allocation information in the dynamic execution plan, network address resolution is performed on the processing engine instances in the distributed engine cluster to obtain the target engine instance communication endpoint set corresponding to the data stream in the batch to be scheduled. Verify the current schedulable status of the target engine instance communication endpoint set to obtain a list of ready engine instances for the data stream in the batch to be scheduled. Based on the data stream push path in the dynamic execution plan and the data characteristics of the data stream in the batch to be scheduled, the data stream in the batch to be scheduled is protocol adapted and encapsulated to obtain a set of transmissible data units that match the ready engine instance. The transmissible data unit set and the ready engine instance list are routed and pushed step by step, and the ready engine instances are started to perform concurrent execution processing on the transmissible data unit set. The CPU utilization, memory usage, and input / output throughput of the ready engine instance are sampled in real time during the processing to generate a running status monitoring record of the processing engine instance.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention obtains complete descriptive information by deeply analyzing the data stream, constructs a benchmark reference set by combining the mapping relationship between historical features and actual resource consumption, and then completes accurate resource prediction by weighted fusion through dynamic correction factors. At the same time, it selects and adapts processing engines based on data features and resource predictions to avoid resource allocation redundancy or insufficiency, and greatly improves the rationality of scheduling and resource utilization efficiency.

[0016] 2. This invention constructs an execution topology graph by parsing data dependencies, and completes engine instance binding and execution timing orchestration by combining load balancing prediction, forming a dynamic execution plan adapted to the characteristics of the data flow. Furthermore, during the data flow processing, the CPU, memory, IO throughput and other indicators of the engine instance are sampled and monitored in real time to keep track of the running status, ensuring the efficient and stable operation of the data flow processing, and improving the dynamics and reliability of the overall monitoring and scheduling. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams provided in an embodiment of the present invention. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams according to an embodiment of the present invention. In this embodiment, the dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams includes: S1. Analyze the multi-source heterogeneous data stream of the target computer to obtain the data stream description information of the target computer; In this embodiment of the invention, the step of parsing the multi-source heterogeneous data stream of the target computer to obtain the data stream description information of the target computer includes: Deep protocol stack parsing is performed on the raw data packets of the target computer, and the multi-source heterogeneous data streams in the raw data packets are extracted. The source identifier of the multi-source heterogeneous data stream is obtained by extracting the characteristic identifier from the transport layer header of the multi-source heterogeneous data stream. Based on the source identifier, deep protocol parsing is performed on the application layer protocol header of the multi-source heterogeneous data stream to obtain the application layer metadata of the multi-source heterogeneous data stream. The data packet arrival frequency is statistically analyzed on the application layer metadata to obtain the data arrival rate of the multi-source heterogeneous data stream, and the load length is aggregated and accumulated on the application layer metadata to obtain the data volume of the multi-source heterogeneous data stream. By integrating the source identifier, the data arrival rate, and the data volume, the data stream description information of the target computer is obtained.

[0021] For the raw data packets received by the target computer, the parsing method proceeds sequentially from the physical layer upwards according to the network protocol stack. This involves parsing the data link layer encapsulation structure, the network layer addressing information, the transport layer control information, and the application layer service information. In the process of disassembling the data packet encapsulation structure layer by layer, the different data sources, different data organization formats, different transmission channels, and different service types carried within the data packet are identified one by one. Various data streams that are independent of each other and have different data structures are completely separated and extracted from the overall data carrier of the raw data packet, forming multi-source heterogeneous data streams that can be independently processed.

[0022] For the multi-source heterogeneous data streams extracted from the original data packets, the data area of ​​the transport layer header corresponding to each data stream is determined one by one. The data source address information, destination address information, source port information, destination port information, session connection identifier, and transmission control related identifier information contained in the transport layer header are completely read. The above information that can uniquely characterize the source and transmission attributes of the data stream is integrated and formed into a fixed identifier content. This identifier content can uniquely distinguish different multi-source heterogeneous data streams, and finally the source identifier of the multi-source heterogeneous data stream is obtained.

[0023] Based on the extracted source identifiers, each multi-source heterogeneous data stream is precisely matched with its corresponding application layer protocol header. According to the standard message format, field definition, and interaction rules of the corresponding application layer protocol, each functional field in the application layer protocol header is completely disassembled, identified, and extracted to obtain information such as protocol type, interaction instructions, data identifier, session state, business attributes, and data interaction method. The above information together constitutes the content that can completely represent the application layer data attributes, and finally obtains the application layer metadata of the multi-source heterogeneous data stream.

[0024] For the obtained application layer metadata, the actual arrival time of each data packet is continuously recorded according to the order in which the data packets arrive at the target computer. The number of data packets arriving within a continuous time interval is statistically analyzed according to a fixed time span. By summarizing the number of arrivals in multiple continuous time intervals, a data arrival rate that can intuitively reflect the speed and density of data transmission is formed. At the same time, for the effective payload of the data packets corresponding to the application layer metadata, the actual data length of each payload segment is determined one by one. The payload lengths of all data packets corresponding to the same multi-source heterogeneous data stream are continuously accumulated in sequence to finally obtain a data volume that can completely reflect the overall scale and total amount of data.

[0025] Each multi-source heterogeneous data stream is associated with its source identifier, data arrival rate, and data volume. These three types of attribute information are integrated and encapsulated according to a unified logical structure to form a comprehensive information set that can fully and completely reflect the source characteristics, transmission rate characteristics, and data scale characteristics of the data stream. This comprehensive information set can completely describe the overall attributes of the data stream inside the target computer, and finally obtain the data stream description information of the target computer.

[0026] S2. Based on the mapping relationship between the processing characteristics of historical data streams and the actual resource consumption of the target computer, the resource quantification assessment of the data stream description information is performed to obtain the estimated resource amount of the multi-source heterogeneous data stream. In this embodiment of the invention, the resource quantification assessment of the data stream description information based on the mapping relationship between the processing characteristics of historical data streams and the actual resource consumption of the target computer, to obtain the estimated resource quantity of the multi-source heterogeneous data stream, includes: Acquire the historical data stream of the target computer and extract the processing features of the historical data stream; Based on the processing features and the actual resource consumption records of the target computer, a consumption value association mapping is performed on the historical data stream to obtain a historical feature consumption association entry library of the target computer. The historical feature consumption associated entry library is clustered and representative samples are selected to construct the feature consumption benchmark reference set of the target computer; The data stream description information is mapped to the feature consumption benchmark reference set to obtain a target benchmark reference entry that matches the data stream description information. The resource estimates of the multi-source heterogeneous data stream are obtained by quantifying and evaluating the consumption value of the target benchmark reference item.

[0027] The step of quantifying and evaluating the consumption value of the target benchmark reference item to obtain the resource estimate of the multi-source heterogeneous data stream includes: By performing historical feature backtracking on the target benchmark reference entry, the benchmark resource consumption, reference data volume, and reference data arrival rate of the target benchmark reference entry are obtained; Real-time feature extraction is performed on the data stream description information to obtain the current data volume and current data arrival rate of the data stream description information; The resource estimate of the multi-source heterogeneous data stream is obtained by weighting and fusing the baseline resource consumption, the reference data volume, the reference data arrival rate, the current data volume, and the current data arrival rate with a dynamic correction factor.

[0028] The formula for calculating the estimated resource quantity is as follows: ; In the formula, This indicates the estimated amount of resources. This represents the baseline resource consumption. Indicates the current data volume level. Indicates the magnitude of the reference data. This indicates the current data arrival rate. This indicates the arrival rate of the reference data. This indicates the preset data volume impact factor. This represents the preset rate influence factor.

[0029] In the calculation of resource estimates, the baseline resource consumption is derived from the system resource consumption under historical operating conditions obtained by backtracking historical features of the target baseline reference items. The current data volume is derived from the overall scale information of the current data stream obtained by real-time feature extraction of the data stream description information. The reference data volume is derived from the overall scale information of the corresponding data under historical operating conditions obtained by backtracking historical features of the target baseline reference items. The current data arrival rate is derived from the transmission speed information of the current data stream obtained by real-time feature extraction of the data stream description information. The reference data arrival rate is derived from the data transmission speed information under historical operating conditions obtained by backtracking historical features of the target baseline reference items. The data volume impact factor and rate impact factor are fixed impact coefficients pre-set according to the actual business scenario.

[0030] This calculation process is used to dynamically adjust resource consumption based on historical baseline resource consumption, taking into account the differences in data volume and data arrival rate between the current data stream and the historical reference data stream. This results in a resource estimation result that is more in line with the actual operating scenario of the current multi-source heterogeneous data stream, and achieves accurate mapping of resource consumption from historical experience to the current scenario.

[0031] During the calculation, when the current data volume is greater than the reference data volume, the overall calculation result will be improved based on the baseline resource consumption, and the improvement will increase as the ratio of the current data volume to the reference data volume increases. When the current data arrival rate is greater than the reference data arrival rate, the overall calculation result will be improved based on the baseline resource consumption, and the improvement will increase as the ratio of the current data arrival rate to the reference data arrival rate increases. When the current data volume is less than the reference data volume or the current data arrival rate is less than the reference data arrival rate, the overall calculation result will be reduced based on the baseline resource consumption, and the reduction will increase as the corresponding ratio decreases.

[0032] The system log area, data cache area, and historical operation record storage area of ​​the target computer are used to comprehensively retrieve all data stream information generated and saved in the past continuous operation periods. This data stream information includes interactive data generated by different business processes, transmission data received by different network interfaces, communication data initiated by different applications, and background data generated by different system services. All collected past data streams are uniformly organized and collected to form a complete and comprehensive historical data stream of the target computer. Feature extraction is carried out on each of the collected historical data streams. The source attributes, transmission rate performance, total data volume, data interaction form, and data processing logic of each historical data stream are broken down in turn. All the above information that can fully reflect the data stream processing process is integrated and solidified to form the processing characteristics of the historical data stream that can uniquely characterize the processing attributes of each historical data stream.

[0033] The processing features of each extracted historical data stream are precisely matched with the actual system resource consumption records generated by the target computer during the same runtime period when the historical data stream was generated. The actual resource consumption records include the real-time usage of the central processing unit, the allocation and usage of physical memory, the usage of network bandwidth, the read and write usage of disk storage, and the resource usage of input and output interfaces. A fixed one-to-one mapping relationship is established between each set of processing features and the corresponding actual resource consumption. The associated entries are generated one by one according to the form that a single processing feature corresponds to a complete set of resource consumption records. All the generated associated entries are collected and stored in a unified format to form a historical feature consumption associated entry library covering various historical operating scenarios of the target computer.

[0034] All related entries within the historical feature consumption related entry library are centrally categorized according to the similarity of processing features. Related entries with the same or similar source attributes, the same or similar transmission rate, the same or similar data volume, the same or similar data interaction form, and the same or similar processing logic are grouped into the same category. Within each category, all related entries under that category are traversed, and related entries whose features are all at the middle level of the category and can fully represent the overall attributes of the category are selected as representative samples of that category. The representative samples selected from all categories are uniformly summarized, and after removing duplicate and abnormal samples, they are standardized and organized to construct a feature consumption benchmark reference set for the target computer that can provide a reference standard for subsequent evaluation.

[0035] The source identifier, data arrival rate, and data volume contained in the previously generated data flow description information are compared item by item with each benchmark reference entry in the feature consumption benchmark reference set according to the same dimension. During the comparison process, the data flow source characteristics, data transmission rate characteristics, and total data volume characteristics are checked in turn to ensure that each characteristic is consistent with the benchmark reference entry until a benchmark reference entry that completely matches all the characteristics of the data flow description information is found. This entry is the target benchmark reference entry that matches the data flow description information.

[0036] A complete historical feature backtracking operation is performed on the matched target benchmark reference item. Along the historical data processing path corresponding to the item, the system resource consumption, overall data scale, and data transmission speed of the item during the historical operation are traced and extracted in sequence. The traced system resource consumption is solidified as the benchmark resource consumption, the overall data scale is solidified as the reference data volume, and the data transmission speed is solidified as the reference data arrival rate. Finally, the benchmark resource consumption, reference data volume, and reference data arrival rate of the target benchmark reference item are obtained in complete form.

[0037] Real-time feature extraction is performed on the data stream description information generated in the early stage and used for matching and comparison. From all the contents contained in the data stream description information, information reflecting the overall scale of the current data stream and information reflecting the transmission speed of the current data stream are accurately located and separated. The information related to the overall scale of the current data stream is determined as the current data volume, and the information related to the transmission speed of the current data stream is determined as the current data arrival rate. Finally, the current data volume and current data arrival rate of the data stream description information are obtained.

[0038] The obtained baseline resource consumption, reference data volume, reference data arrival rate, current data volume, and current data arrival rate are combined in an orderly manner according to the logical relationship of actual business operation. The influence of the dynamic correction factor is determined based on the degree of difference between historical reference characteristics and current real-time characteristics. The dynamic correction factor is applied to the baseline resource consumption, reference data volume, reference data arrival rate, current data volume, and current data arrival rate in sequence, so that various feature information is weighted and fused according to the actual operation scenario. During the weighted fusion process, the complete combination of historical baseline attributes and current real-time attributes is maintained. Through the weighted adjustment of segment information and overall fusion, the resource estimation of multi-source heterogeneous data stream is finally obtained.

[0039] S3. According to the preset scheduling period, the multi-source heterogeneous data stream is divided into batches to be scheduled, and the data features of the data streams in the batches to be scheduled are extracted. In this embodiment of the invention, the step of dividing the multi-source heterogeneous data streams into batches to be scheduled according to a preset scheduling period and extracting data features of the data streams within the batches to be scheduled includes: Based on a preset scheduling period, the multi-source heterogeneous data stream is divided into continuous time window slices to obtain a batch of data to be scheduled corresponding to the scheduling period. Data stream identifiers are collected from the batch of data to be scheduled to obtain a real-time data stream identifier list for the batch of data to be scheduled. Based on the real-time data stream identifier list, deep parsing of the application layer protocol type is performed on the data streams within the data batch to be scheduled to obtain the protocol type label and data stream status identifier of the data streams within the data batch to be scheduled. Multidimensional aggregation statistics are performed on the protocol type label and the data stream status identifier to obtain the data characteristics of the data stream within the batch to be scheduled.

[0040] According to a pre-set and fixed scheduling cycle in the system, the continuously transmitted and updated multi-source heterogeneous data stream is divided into continuous and non-overlapping time windows in the time dimension. During the division process, the start and end times of each time window are strictly determined according to the time span of the scheduling cycle. The complete and continuous multi-source heterogeneous data stream is sequentially divided into multiple data segments of equal length and continuous time sequence. Each data segment maintains a strict time correspondence with the scheduling cycle. All the divided data segments are combined to form a batch of data to be scheduled corresponding to the scheduling cycle.

[0041] For each batch of data to be scheduled, the unique and non-repeatable identification characters of each data stream are located and extracted one by one. All the extracted identification characters are sorted in order according to the transmission order of the data stream in the batch of data to be scheduled. The same identification characters that appear repeatedly are deduplicated and retained. All valid identification characters are collected and formed into a complete set of entries. This set of entries can fully display the unique identifier of all data streams in the batch of data to be scheduled, and finally obtain the real-time data stream identifier list of the batch of data to be scheduled.

[0042] Based on each data stream identifier recorded in the real-time data stream identifier list, the corresponding target data stream is located and matched one by one within the batch of data to be scheduled. For each successfully matched data stream, it is parsed layer by layer from the data link layer up to the application layer, completely disassembling the header information and interaction content of the application layer protocol. According to the standard format of the application layer protocol, the specific protocol type of each data stream is determined and the corresponding protocol type label is generated. At the same time, the current connection establishment status, data transmission progress, interaction execution stage and business operation status of each data stream are parsed and the corresponding data stream status identifier is generated.

[0043] The protocol type labels and data flow status identifiers corresponding to all data flows within the batch to be scheduled are simultaneously aggregated and statistically analyzed from multiple dimensions. The data flows are classified and statistically analyzed according to different types of protocol type labels, and the number of data flows corresponding to each type of protocol is recorded. The data flows are also classified and statistically analyzed according to different status types of data flow status identifiers, and the number of data flows corresponding to each status is recorded. The statistical results of the protocol type dimension and the statistical results of the data flow status dimension are organically integrated to form data features that can comprehensively reflect the distribution of data flow types, status distribution and overall composition within the batch to be scheduled.

[0044] S4. Based on the data characteristics and the estimated resource quantity, select an appropriate processing engine type for the data stream in the batch to be scheduled from the preset processing engine template. In this embodiment of the invention, the step of selecting a suitable processing engine type for the data stream within the batch to be scheduled, based on the data characteristics and the estimated resource quantity, from a preset processing engine template, includes: Based on the data characteristics, the preset processing engine templates are screened for functional adaptability to obtain candidate engine templates for the data streams in the batch to be scheduled. The resource capacity parameter of the candidate engine template is parsed to obtain the resource capacity level identifier corresponding to the candidate engine template; Based on the estimated resource quantity, the satisfaction level of the resource capacity level identifier is compared to obtain the secondary candidate engine template of the data stream in the batch to be scheduled. A comprehensive adaptation evaluation is performed on the secondary candidate engine templates to obtain the processing engine type that the data stream in the batch to be scheduled is adapted to.

[0045] Based on the data characteristics of the data streams within the batch to be scheduled, the system iterates through and compares each of the pre-configured and stored processing engine templates. It matches the protocol processing type, data processing mode, data stream state adaptation scenario, protocol type label, data stream state identifier, data stream type distribution, and data stream state distribution contained in the data characteristics of each processing engine template. All processing engine templates that can fully adapt to all attribute requirements in the data characteristics are selected. All the selected processing engine templates are combined to form the candidate engine templates for the data streams within the batch to be scheduled.

[0046] A comprehensive analysis of the internal resource carrying capacity of each selected candidate engine template is conducted. The maximum data processing scale, the highest data transmission rate, the maximum memory resources, the maximum network bandwidth, and the maximum number of concurrent data streams that each candidate engine template can handle during runtime are determined. The above-mentioned contents that reflect the resource carrying capacity of the candidate engine template are uniformly converted into corresponding level labels. These level labels can intuitively distinguish the resource carrying capacity of different candidate engine templates, and finally, the resource capacity level label corresponding to each candidate engine template is obtained.

[0047] The resource estimates of the multi-source heterogeneous data streams obtained in the previous calculation are used as the basis for resource demand. The resource capacity level identifiers corresponding to each candidate engine template are compared item by item to determine whether the resource carrying capacity of the candidate engine template can fully cover all the resource demand corresponding to the resource estimates. Only candidate engine templates that fully meet the resource carrying capacity and have no resource gaps are retained, while candidate engine templates that cannot meet the resource demand are eliminated. All the remaining candidate engine templates together constitute the secondary candidate engine templates for the data streams in the batch to be scheduled.

[0048] The obtained secondary candidate engine templates are evaluated in a multi-dimensional comprehensive fit assessment. The fit between each secondary candidate engine template and the data stream in the batch to be scheduled is assessed in terms of protocol processing accuracy, data processing efficiency, resource matching fit, long-term operation stability, and data stream state adaptability. The results of all assessment dimensions are integrated and judged, and the secondary candidate engine template with the highest comprehensive fit and which fully matches the data stream processing requirements is selected. The type corresponding to this engine template is the processing engine type adapted to the data stream in the batch to be scheduled.

[0049] S5. Based on the processing engine type and the data characteristics, construct the execution topology diagram of the data flow within the batch to be scheduled, so as to determine the dynamic execution plan of the data flow within the batch to be scheduled; In this embodiment of the invention, the step of constructing an execution topology graph of the data stream within the batch to be scheduled based on the processing engine type and the data characteristics, in order to determine the dynamic execution plan of the data stream within the batch to be scheduled, includes: Based on the processing engine type and the data characteristics, the data flow direction of the data stream in the batch to be scheduled is analyzed to obtain the processing nodes of the data stream in the batch to be scheduled and the data dependency chain between the nodes. The data dependency chain is hierarchically divided to obtain the topological hierarchy identifier of the processing node; Based on the parallelism requirement identifier in the data characteristics, the resource parallelism of the processing node is configured to obtain the set of parallel execution units of the processing node. The set of parallel execution units and the topology level identifier are combined to construct the execution topology graph of the data flow within the batch to be scheduled. By binding engine instances to the execution topology graph, a dynamic execution plan for the data flow within the batch to be scheduled is obtained.

[0050] The step of binding engine instances to the execution topology to obtain a dynamic execution plan for the data stream within the batch to be scheduled includes: Based on the processing nodes in the execution topology graph and the set of parallel execution units, a node load balancing prediction is performed on the preset distributed engine cluster to obtain a list of candidate engine instances for the data stream in the batch to be scheduled. The candidate engine instance list is evaluated by determining the matching degree of node data features to obtain the preferred engine instance list for the processing node; Based on the list of preferred engine instances and the data flow dependency relationship of the processing nodes, a data routing mapping is performed between the processing nodes and the preferred engine instances to obtain a binding relationship mapping table between the processing nodes and the target engine instances. The binding relationship mapping table is executed in a timing arrangement to generate a dynamic execution plan that includes node startup order, engine instance allocation information, and data stream push path.

[0051] Based on the complete processing logic of the determined processing engine type and the data characteristics of the data stream within the batch to be scheduled, including protocol type, data stream status, and data transmission direction, the entire transmission direction of the data stream within the batch to be scheduled is broken down and analyzed segment by segment from access reception to parsing, processing, aggregation, conversion, and output forwarding. Combining the functional modules corresponding to the processing engine type, each functional processing node that the data stream must pass through in the entire processing flow is located. At the same time, the data input source and data output destination between two adjacent processing nodes are sorted out one by one, clarifying the association logic that the output data of the previous processing node is the only input data of the next processing node. The upstream and downstream relationships of all processing nodes are connected and sorted out, and finally the processing nodes of the data stream within the batch to be scheduled and the data dependency chain between the nodes are obtained.

[0052] Based on the upstream and downstream data transmission logic recorded in the data dependency chain between nodes, all processing nodes are hierarchically distinguished according to the actual execution order of data processing. Processing nodes that do not have any preceding data dependencies and can directly receive the raw data stream are uniformly classified as the top-level starting level. Processing nodes that must rely on the output data of the starting level processing nodes to perform processing operations are uniformly classified as the intermediate level. Processing nodes that must rely on the output data of the intermediate level processing nodes to complete the final processing are uniformly classified as the bottom level. The hierarchical division of all processing nodes is completed in sequence according to the transmission direction of data dependencies. Each divided level is assigned a unique identifier, which can intuitively reflect the vertical position of the processing node in the overall data processing flow, and finally obtain the topological hierarchical identifier of the processing node.

[0053] From the data characteristics of the data stream within the batch to be scheduled, the parallelism requirement identifier that can represent the number of data streams that can be processed synchronously is accurately extracted. Based on the actual processing requirements corresponding to the parallelism requirement identifier, the corresponding parallel processing resources are matched for each processing node. According to the actual needs of parallel processing, multiple independent and simultaneously running execution units are set for a single processing node. Each execution unit can independently complete the processing operation of the corresponding processing node. All execution units corresponding to all processing nodes are uniformly collected and orderly organized to form a unit combination that can synchronously carry out data stream processing operations, and finally the set of parallel execution units of the processing node is obtained.

[0054] The set of parallel execution units corresponding to the processing nodes and the topology hierarchy identifiers are organically combined according to the actual logical relationship of data processing. All processing nodes are arranged in the order of the upper and lower levels determined by the topology hierarchy identifiers. At the same time, all parallel execution units corresponding to each processing node are associated and bound under each node. The connection relationship between processing nodes at different levels and the synchronous execution relationship between parallel execution units at the same level are clarified. Through the orderly combination of the hierarchical structure and parallel execution units, a complete and clear data flow processing structure is formed. This structure can fully demonstrate the data flow processing flow and parallel execution method, and finally, the execution topology diagram of the data flow in the batch to be scheduled is constructed.

[0055] Based on the processing requirements of all processing nodes recorded in the execution topology diagram and the resource consumption requirements corresponding to the set of parallel execution units, the current running status of all available engine instances in the preset distributed engine cluster is predicted one by one. The current remaining resources, processing load and actual processing capacity of each engine instance are checked to determine whether each engine instance can stably bear all the processing requirements of the corresponding processing nodes and parallel execution units. All engine instances that can meet the bearing requirements are centrally summarized and arranged in an orderly manner to finally obtain a list of candidate engine instances for the data stream in the batch to be scheduled.

[0056] Each engine instance in the candidate engine instance list is meticulously compared with the data characteristics matched by the corresponding processing node. The degree of fit between the engine instance and the actual needs of the processing node is determined in terms of application layer protocol processing capability, data arrival rate carrying capacity, and data volume processing capability. Only engine instances that meet the highest standard of fit and fully match the processing requirements are retained. These selected high-quality engine instances are then systematically organized and uniformly collected to finally obtain the preferred engine instance list for the processing node.

[0057] Based on the engine instance information in the preferred engine instance list and the established data flow dependencies between processing nodes, each processing node is matched with the corresponding engine instance with the highest compatibility in the preferred engine instance list. The unique corresponding connection between the processing node and the target engine instance is determined strictly according to the data flow transmission path. All matched correspondences are uniformly recorded and standardized to obtain the binding relationship mapping table between processing nodes and target engine instances.

[0058] The execution sequence of the completed binding relationship mapping table is arranged in an orderly manner according to the execution order of the data dependency relationship chain and the overall structural requirements of the execution topology. During the arrangement process, the startup order of all processing nodes is accurately determined, the engine instance allocation information corresponding to each processing node is clearly marked, and the data flow transmission and push path from the original access point to the corresponding processing node and then to the target engine instance is scientifically planned. All the above-mentioned arranged information is integrated, summarized and standardized, and finally a dynamic execution plan containing node startup order, engine instance allocation information and data flow push path is generated.

[0059] S6. According to the dynamic execution plan, the data streams in the batch to be scheduled are distributed to the corresponding processing engine instances for processing, and the running status and resource usage of the processing engine instances are monitored in real time.

[0060] In this embodiment of the invention, the step of distributing the data streams within the scheduled batch to the corresponding processing engine instances for processing according to the dynamic execution plan, and monitoring the running status and resource usage of the processing engine instances in real time, includes: Based on the engine instance allocation information in the dynamic execution plan, network address resolution is performed on the processing engine instances in the distributed engine cluster to obtain the target engine instance communication endpoint set corresponding to the data stream in the batch to be scheduled. Verify the current schedulable status of the target engine instance communication endpoint set to obtain a list of ready engine instances for the data stream in the batch to be scheduled. Based on the data stream push path in the dynamic execution plan and the data characteristics of the data stream in the batch to be scheduled, the data stream in the batch to be scheduled is protocol adapted and encapsulated to obtain a set of transmissible data units that match the ready engine instance. The transmissible data unit set and the ready engine instance list are routed and pushed step by step, and the ready engine instances are started to perform concurrent execution processing on the transmissible data unit set. The CPU utilization, memory usage, and input / output throughput of the ready engine instance are sampled in real time during the processing to generate a running status monitoring record of the processing engine instance.

[0061] Based on the allocation information of all engine instances recorded in the dynamic execution plan, each allocated processing engine instance is precisely located within the distributed engine cluster by instance identifier. For each located processing engine instance, a complete network address resolution is performed to determine the network access address, dedicated data communication port, data transmission physical interface, and external data access node corresponding to the processing engine instance. All the communication access related information obtained by resolution is uniformly collected and standardized to form a set of communication endpoints that can achieve accurate data flow transmission, effective docking, and normal interaction. Finally, the set of communication endpoints of the target engine instance corresponding to the data flow in the batch to be scheduled is obtained.

[0062] For each processing engine instance corresponding to a communication endpoint in the target engine instance communication endpoint set, a comprehensive and detailed verification of its current schedulable status is carried out. This involves checking the network connectivity stability, system operation normality, hardware resource idleness, service response timeliness, and data processing capacity of each processing engine instance. This confirms that each processing engine instance fully meets all the conditions for receiving, parsing, and processing data streams. All processing engine instances that pass all status verifications and meet the real-time scheduling execution requirements are centrally collected, and processing engine instances with network interruptions, operational abnormalities, insufficient resources, or service failures are removed. Finally, a list of ready engine instances for the data streams in the batch to be scheduled is obtained.

[0063] Based on the pre-planned data stream push path in the dynamic execution plan, and the data characteristics related to the application layer protocol type, data format attributes, and transmission rate requirements of the data stream in the batch to be scheduled, the original data stream in the batch to be scheduled is subjected to targeted adaptation and complete encapsulation according to the protocol specifications, transmission format standards, and data structure requirements supported by the ready engine instance. The encapsulated independent transmittable units are then arranged in an orderly manner according to the order of the data stream push path. All the adapted and encapsulated independent units that meet the transmission and processing requirements together form a set of data units that can be normally recognized, stably received, and correctly processed by the ready engine instance, ultimately resulting in a set of transmittable data units that match the ready engine instance.

[0064] According to the hierarchical order of the data flow push path and the upstream and downstream transmission logic in the dynamic execution plan, each data unit in the set of transmittable data units is precisely matched one-to-one with the corresponding target engine instance in the list of ready engine instances. The matched data units are pushed to the dedicated receiving port of the corresponding ready engine instance through network routing. After all transmittable data units have been pushed in place, all ready engine instances are started in sequence, so that multiple ready engine instances can simultaneously perform parallel parsing, transformation and execution processing on their respective transmittable data units.

[0065] Throughout the entire data processing process of all ready engine instances, the real-time CPU utilization, real-time physical memory utilization, and real-time input / output data throughput of each ready engine instance are continuously collected and recorded at fixed time intervals. Each piece of running data is categorized, summarized, and updated in real time according to the unique identifier of the corresponding processing engine instance, forming a continuous record that can fully reflect the real-time running status, resource utilization, and data processing status of each processing engine instance, and finally generating a running status monitoring record of the processing engine instances.

[0066] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0067] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams, characterized in that, The method includes: S1. Analyze the multi-source heterogeneous data stream of the target computer to obtain the data stream description information of the target computer; S2. Based on the mapping relationship between the processing characteristics of historical data streams and the actual resource consumption of the target computer, the resource quantification assessment of the data stream description information is performed to obtain the estimated resource amount of the multi-source heterogeneous data stream. S3. According to the preset scheduling period, the multi-source heterogeneous data stream is divided into batches to be scheduled, and the data features of the data streams in the batches to be scheduled are extracted. S4. Based on the data characteristics and the estimated resource quantity, select an appropriate processing engine type for the data stream in the batch to be scheduled from the preset processing engine template. S5. Based on the processing engine type and the data characteristics, construct the execution topology diagram of the data flow within the batch to be scheduled, so as to determine the dynamic execution plan of the data flow within the batch to be scheduled; S6. According to the dynamic execution plan, the data streams in the batch to be scheduled are distributed to the corresponding processing engine instances for processing, and the running status and resource usage of the processing engine instances are monitored in real time.

2. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 1, characterized in that, The process of parsing the multi-source heterogeneous data streams of the target computer to obtain data stream description information of the target computer includes: Deep protocol stack parsing is performed on the raw data packets of the target computer, and the multi-source heterogeneous data streams in the raw data packets are extracted. The source identifier of the multi-source heterogeneous data stream is obtained by extracting the characteristic identifier from the transport layer header of the multi-source heterogeneous data stream. Based on the source identifier, deep protocol parsing is performed on the application layer protocol header of the multi-source heterogeneous data stream to obtain the application layer metadata of the multi-source heterogeneous data stream. The data packet arrival frequency is statistically analyzed on the application layer metadata to obtain the data arrival rate of the multi-source heterogeneous data stream, and the load length is aggregated and accumulated on the application layer metadata to obtain the data volume of the multi-source heterogeneous data stream. By integrating the source identifier, the data arrival rate, and the data volume, the data stream description information of the target computer is obtained.

3. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 1, characterized in that, The mapping relationship between the processing characteristics of historical data streams and the actual resource consumption of the target computer is used to perform resource quantification assessment on the data stream description information to obtain the resource estimate of the multi-source heterogeneous data stream, including: Acquire the historical data stream of the target computer and extract the processing features of the historical data stream; Based on the processing features and the actual resource consumption records of the target computer, a consumption value association mapping is performed on the historical data stream to obtain a historical feature consumption association entry library of the target computer. The historical feature consumption associated entry library is clustered and representative samples are selected to construct the feature consumption benchmark reference set of the target computer; The data stream description information is mapped to the feature consumption benchmark reference set to obtain a target benchmark reference entry that matches the data stream description information. The resource estimates of the multi-source heterogeneous data stream are obtained by quantifying and evaluating the consumption value of the target benchmark reference item.

4. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 3, characterized in that, The step of quantifying and evaluating the consumption value of the target benchmark reference item to obtain the resource estimate of the multi-source heterogeneous data stream includes: By performing historical feature backtracking on the target benchmark reference entry, the benchmark resource consumption, reference data volume, and reference data arrival rate of the target benchmark reference entry are obtained; Real-time feature extraction is performed on the data stream description information to obtain the current data volume and current data arrival rate of the data stream description information; The resource estimate of the multi-source heterogeneous data stream is obtained by weighting and fusing the baseline resource consumption, the reference data volume, the reference data arrival rate, the current data volume, and the current data arrival rate with a dynamic correction factor.

5. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 4, characterized in that, The formula for calculating the estimated resource quantity is as follows: ； In the formula, This indicates the estimated amount of resources. This represents the baseline resource consumption. Indicates the current data volume level. Indicates the magnitude of the reference data. This indicates the current data arrival rate. This indicates the arrival rate of the reference data. This indicates the preset data volume impact factor. This represents the preset rate influence factor.

6. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 1, characterized in that, The step of dividing the multi-source heterogeneous data streams into batches to be scheduled according to a preset scheduling period and extracting data features of the data streams within each batch to be scheduled includes: Based on a preset scheduling period, the multi-source heterogeneous data stream is divided into continuous time window slices to obtain a batch of data to be scheduled corresponding to the scheduling period. Data stream identifiers are collected from the batch of data to be scheduled to obtain a real-time data stream identifier list for the batch of data to be scheduled. Based on the real-time data stream identifier list, deep parsing of the application layer protocol type is performed on the data streams within the data batch to be scheduled to obtain the protocol type label and data stream status identifier of the data streams within the data batch to be scheduled. Multidimensional aggregation statistics are performed on the protocol type label and the data stream status identifier to obtain the data characteristics of the data stream within the batch to be scheduled.

7. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 1, characterized in that, Based on the data characteristics and the estimated resource quantity, the step of selecting a suitable processing engine type for the data stream within the batch to be scheduled from a preset processing engine template includes: Based on the data characteristics, the preset processing engine templates are screened for functional adaptability to obtain candidate engine templates for the data streams in the batch to be scheduled. The resource capacity parameter of the candidate engine template is parsed to obtain the resource capacity level identifier corresponding to the candidate engine template; Based on the estimated resource quantity, the satisfaction level of the resource capacity level identifier is compared to obtain the secondary candidate engine template of the data stream in the batch to be scheduled. A comprehensive adaptation evaluation is performed on the secondary candidate engine templates to obtain the processing engine type that the data stream in the batch to be scheduled is adapted to.

8. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 1, characterized in that, The step of constructing an execution topology graph of the data stream within the batch to be scheduled based on the processing engine type and the data characteristics, in order to determine the dynamic execution plan of the data stream within the batch to be scheduled, includes: Based on the processing engine type and the data characteristics, the data flow direction of the data stream in the batch to be scheduled is analyzed to obtain the processing nodes of the data stream in the batch to be scheduled and the data dependency chain between the nodes. The data dependency chain is hierarchically divided to obtain the topological hierarchy identifier of the processing node; Based on the parallelism requirement identifier in the data characteristics, the resource parallelism of the processing node is configured to obtain the set of parallel execution units of the processing node. The set of parallel execution units and the topology level identifier are combined to construct the execution topology graph of the data flow within the batch to be scheduled. By binding engine instances to the execution topology graph, a dynamic execution plan for the data flow within the batch to be scheduled is obtained.

9. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 8, characterized in that, The step of binding engine instances to the execution topology to obtain a dynamic execution plan for the data stream within the batch to be scheduled includes: Based on the processing nodes in the execution topology graph and the set of parallel execution units, a node load balancing prediction is performed on the preset distributed engine cluster to obtain a list of candidate engine instances for the data stream in the batch to be scheduled. The candidate engine instance list is evaluated by determining the matching degree of node data features to obtain the preferred engine instance list for the processing node; Based on the list of preferred engine instances and the data flow dependency relationship of the processing nodes, a data routing mapping is performed between the processing nodes and the preferred engine instances to obtain a binding relationship mapping table between the processing nodes and the target engine instances. The binding relationship mapping table is executed in a timing arrangement to generate a dynamic execution plan that includes node startup order, engine instance allocation information, and data stream push path.

10. The dynamic monitoring and scheduling method for multi-source heterogeneous computer data streams as described in claim 9, characterized in that, The step of distributing the data streams within the scheduled batch to the corresponding processing engine instances for processing according to the dynamic execution plan, and monitoring the running status and resource usage of the processing engine instances in real time, includes: Based on the engine instance allocation information in the dynamic execution plan, network address resolution is performed on the processing engine instances in the distributed engine cluster to obtain the target engine instance communication endpoint set corresponding to the data stream in the batch to be scheduled. Verify the current schedulable status of the target engine instance communication endpoint set to obtain a list of ready engine instances for the data stream in the batch to be scheduled. Based on the data stream push path in the dynamic execution plan and the data characteristics of the data stream in the batch to be scheduled, the data stream in the batch to be scheduled is protocol adapted and encapsulated to obtain a set of transmissible data units that match the ready engine instance. The transmissible data unit set and the ready engine instance list are routed and pushed step by step, and the ready engine instances are started to perform concurrent execution processing on the transmissible data unit set. The CPU utilization, memory usage, and input / output throughput of the ready engine instance are sampled in real time during the processing to generate a running status monitoring record of the processing engine instance.

Images