Data processing and resource configuration method and system based on multi-objective optimization

Abstract

The application discloses a data processing and resource configuration method and system based on multi-target optimization, and the method comprises the following steps: listening to the real-time running state of a running end to determine the running data collection mode of the running end; generating a plurality of labeled running data subsets according to the running data collection mode, and preprocessing each running data subset; constructing a running end business model according to the business processing attributes of the running end; wherein the business processing attributes comprise business flow path sub-attributes and business architecture sub-attributes; setting a multi-target engine for the running end business model, determining a plurality of data processing paths according to the optimization algorithm of the multi-target engine; inputting a part of the running data subsets into each data processing path, predicting the resource bottleneck trend of the running end according to the output result of the data processing path; and predictively adjusting the resource configuration strategy according to the resource bottleneck trend.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a data processing and resource allocation method and system based on multi-objective optimization. Background Technology

[0002] Enterprises and other entities with complex operational systems typically employ numerous servers. These servers can execute business processes independently or form a collaborative business architecture. Different customers have varying processing requirements, leading to significant differences in the business flow transmission paths and the business architecture layers invoked by each business within the runtime environment. Currently, to adapt to these differentiated processing needs, the runtime environment often requires temporary setup or adjustment of corresponding business processing architectures for different business types. This approach of adapting to each business individually not only increases the configuration complexity of the runtime environment but, more seriously, the lack of a unified modeling and reuse mechanism for existing business flow transmission paths and business architectures prevents the runtime environment from quickly matching the optimal processing path for each business in multi-business concurrent scenarios.

[0003] Meanwhile, in multi-service concurrent scenarios, the runtime needs to simultaneously consider multiple optimization objectives, including but not limited to minimizing service processing latency, maximizing system throughput, and balancing resource usage among various services. However, these optimization objectives often have mutually restrictive relationships. For example, reducing processing latency may require more memory resources, while increasing throughput may require increased computing power. Existing technologies lack a path selection mechanism that can effectively balance these conflicting objectives, leading to resource allocation decisions that often prioritize one aspect over another, making it difficult to achieve global optimization.

[0004] Furthermore, different business processes require varying types and quantities of resources, and these resource demands fluctuate dynamically with changes in the real-time load on the runtime. Existing resource allocation strategies are typically based on fixed rules or reactive alarm triggers, meaning adjustments are only made passively after resource bottlenecks have occurred and business performance has degraded. This reactive approach cannot meet the needs of scenarios with high timeliness requirements for business processing. Accurately predicting resource bottleneck trends and implementing resource allocation in advance is crucial for improving the reliability and efficiency of resource allocation.

[0005] In summary, in response to the difficulties in adapting business paths, the complexity of balancing multiple objectives, and the lag in post-event response to resource bottlenecks faced by the runtime in multiple business scenarios, there is an urgent need for a technical solution that can uniformly construct the runtime business processing model, realize multi-objective optimized path selection, and predictive resource allocation. Summary of the Invention

[0006] In view of the above problems, the present invention provides a data processing and resource allocation method based on multi-objective optimization, including: Monitor the real-time running status of the running terminal to determine the running data collection mode of the running terminal; generate several tagged running data subsets according to the running data collection mode, and preprocess each running data subset; Based on the business processing attributes of the runtime, a runtime business model is constructed; wherein, the business processing attributes include business flow path sub-attributes and business architecture sub-attributes; a multi-objective engine is set for the runtime business model, and several data processing paths are determined according to the optimization algorithm of the multi-objective engine; A subset of runtime data is selected and input into each data processing path. Based on the output results of the data processing paths, the resource bottleneck trend of the runtime is predicted. Based on the resource bottleneck trend, the resource allocation strategy is predictively adjusted.

[0007] Preferably, the real-time operating status of the runtime is monitored to determine the runtime data collection mode; based on the runtime data collection mode, several tagged runtime data subsets are generated, and each runtime data subset is preprocessed, including: Monitor the real-time execution status of each business thread under the runtime; wherein, the real-time execution status includes the real-time data processing rate and the real-time result output rate of each business thread; determine the distribution of the effective running time interval of each business thread based on the real-time execution status; Based on the distribution of effective running time intervals and the location of each virtual interval for each business thread, the running data collection mode for the running terminal is determined; wherein, the running data collection mode includes the collection time and the process address where the collection is performed; According to the aforementioned operational data acquisition mode, raw operational data queues from each business thread are collected; each raw operational data queue is processed for source and modality tagging to generate several operational data subsets; Based on the modal labels and data volume of each subset of running data, noise reduction and deduplication preprocessing are performed on each subset of running data.

[0008] Preferably, a runtime business model is constructed based on the runtime's business processing attributes; a multi-objective engine is set for the runtime business model, and several data processing paths are determined based on the optimization algorithm of the multi-objective engine, including: Obtain the service flow transmission path information and historical service architecture configuration information of the running terminal from its historical logs. The path usage of the service flow transmission path information is identified to obtain service flow path sub-attributes; wherein, the service flow path sub-attributes include the distribution location of service flow transmission paths in normal working state; The historical business architecture configuration information is subjected to architecture-level integrity identification to obtain business architecture sub-attributes; wherein, the business architecture sub-attributes include business architecture types that are in a valid and complete state; Based on the business flow path sub-attributes and the business architecture sub-attributes, a business flow transmission path set and a business architecture set are formed to construct a runtime business model; wherein, the runtime business model includes a business flow transmission path for processing business and a business architecture with embedded computing plugins; Analyze customer needs to obtain the types of business processing results the customer expects to obtain; generate a multi-objective engine based on the business processing result types; The multi-objective engine filters all service flow transmission paths under the runtime service model to determine a set of candidate data processing paths. The multi-objective engine is used to solve for the best path or path combination that satisfies the Pareto optimality condition in the candidate data processing paths based on the trade-off relationship between each optimization objective.

[0009] Preferably, a subset of runtime data is selected and input into each data processing path. Based on the output results of the data processing paths, the resource bottleneck trend of the runtime is predicted. Based on the resource bottleneck trend, the resource allocation strategy is predictively adjusted, including: Analyze customer needs to determine target business scenarios; filter the subset of operational data that matches the characteristics of the target business scenarios, and input the subset of operational data one by one into each data processing path; The output results of several intermediate ports that connect the data processing path with the business architecture under the running end business model are obtained, and the credibility of the output results is identified in order to predict the resource bottleneck trend of the running end at the corresponding position of each intermediate port. The resource bottleneck trend is traced to determine the predicted location of the resource bottleneck trend and the resource gap attribute, and the resource allocation strategy for the business architecture is predictively adjusted; wherein, the resource allocation strategy includes the scheduling location and / or scheduling quantity of computing resources and / or memory resources.

[0010] Preferably, predicting the resource bottleneck trend of the runtime based on the output results of the data processing path includes: Obtain the current queue length to be processed, current resource utilization rate, current average response latency, historical peak queue length, resource utilization warning threshold and baseline response latency of each intermediate port on the data processing path; Based on the current pending queue length, current resource utilization rate, current average response latency, historical peak queue length, resource utilization warning threshold, and baseline response latency of each intermediate port, determine the resource bottleneck urgency index of each intermediate port. Among them, the resource bottleneck urgency index of each intermediate port The calculation formula is: ; The resource bottleneck urgency index is used to characterize the urgency of the intermediate port experiencing a resource bottleneck. The larger the value of the resource bottleneck urgency index, the higher the urgency of the intermediate port experiencing a resource bottleneck. This represents the current length of the queue to be processed at this intermediate port. This indicates the current resource utilization rate of the intermediate port; This indicates the resource usage warning threshold for this intermediate port; This represents the current average response latency of the intermediate port. This is the baseline response delay for the intermediate port; This represents the queue pressure weighting coefficient. This represents the resource pressure weighting coefficient. This represents the delay pressure weighting coefficient, and satisfies... ; This represents the delay sensitivity coefficient; This indicates the elimination of the zero fault tolerance factor; Based on the ranking of the resource bottleneck urgency index of each intermediate port from high to low, the intermediate ports with resource bottleneck risk and their risk levels are determined. Preferably, after determining the intermediate port with resource bottleneck risk and its risk level, the method further includes: Intermediate ports whose resource bottleneck urgency index exceeds a preset risk threshold are selected as target intermediate ports for resource gap assessment. The step of determining the predicted location of the resource bottleneck trend and the resource gap attributes, and predictively adjusting the resource allocation strategy for the business architecture, includes: Obtain the predicted net resource gap, bottleneck remaining time interval, and system scheduling response window for each target intermediate port; The resource gap warning value for each target intermediate port is calculated based on its predicted net resource gap, remaining bottleneck time interval, and system scheduling response window; where the resource gap warning value for each target intermediate port is... The calculation formula is: ; The larger the resource gap warning value, the more resources the target intermediate port needs to replenish and the more urgent the scheduling; where, This indicates the predicted net resource gap at the intermediate port of the target; This indicates the remaining time interval of the bottleneck at the intermediate port of the target. ; This represents the system scheduling response window of the running terminal to which the intermediate port of the target belongs; This represents the resource type adjustment coefficient for the intermediate port of the target; This indicates the time urgency reference range of the target's intermediate port; This represents the attenuation rate coefficient at the intermediate port of the target. Based on the ranking of the resource gap warning values ​​of each target intermediate port from high to low, predictive adjustments to the resource allocation strategy are performed sequentially on each target intermediate port.

[0011] A data processing and resource allocation system based on multi-objective optimization includes: The data acquisition mode determination module is used to monitor the real-time operating status of the running terminal to determine the data acquisition mode for the running terminal. The data subset processing module is used to generate several labeled subsets of running data according to the running data acquisition mode, and to preprocess each subset of running data. The business model construction module is used to construct a runtime business model based on the runtime's business processing attributes; wherein, the business processing attributes include business flow path sub-attributes and business architecture sub-attributes; The path determination module is used to set a multi-objective engine for the runtime business model and determine several data processing paths according to the optimization algorithm of the multi-objective engine. The resource configuration module is used to filter a subset of running data and input it to each data processing path. Based on the output results of the data processing path, it predicts the resource bottleneck trend of the running end. Based on the resource bottleneck trend, it predictively adjusts the resource configuration strategy.

[0012] Preferably, the acquisition mode determination module is used to monitor the real-time operating status of the running terminal to determine the operating data acquisition mode for the running terminal, including: Monitor the real-time execution status of each business thread under the runtime; wherein, the real-time execution status includes the real-time data processing rate and the real-time result output rate of each business thread; determine the distribution of the effective running time interval of each business thread based on the real-time execution status; Based on the distribution of effective running time intervals and the location of each virtual interval for each business thread, the running data collection mode for the running terminal is determined; wherein, the running data collection mode includes the collection time and the process address where the collection is performed; The data subset processing module is used to generate several labeled subsets of running data according to the running data acquisition mode, and to preprocess each subset of running data, including: According to the aforementioned operational data acquisition mode, raw operational data queues from each business thread are collected; each raw operational data queue is processed for source and modality tagging to generate several operational data subsets; Based on the modal labels and data volume of each subset of running data, noise reduction and deduplication preprocessing are performed on each subset of running data.

[0013] Preferably, the business model construction module is used to construct a runtime business model based on the business processing attributes of the runtime terminal, including: Obtain the service flow transmission path information and historical service architecture configuration information of the running terminal from its historical logs. The path usage of the service flow transmission path information is identified to obtain service flow path sub-attributes; wherein, the service flow path sub-attributes include the distribution location of service flow transmission paths in normal working state; The historical business architecture configuration information is subjected to architecture-level integrity identification to obtain business architecture sub-attributes; wherein, the business architecture sub-attributes include business architecture types that are in a valid and complete state; Based on the business flow path sub-attributes and the business architecture sub-attributes, a business flow transmission path set and a business architecture set are formed to construct a runtime business model; wherein, the runtime business model includes a business flow transmission path for processing business and a business architecture with embedded computing plugins; The path determination module is used to set a multi-objective engine for the runtime business model, and determine several data processing paths according to the optimization algorithm of the multi-objective engine, including: Analyze customer needs to obtain the types of business processing results the customer expects to obtain; generate a multi-objective engine based on the business processing result types; The multi-objective engine filters all service flow transmission paths under the runtime service model to determine a set of candidate data processing paths. The multi-objective engine is used to solve for the best path or path combination that satisfies the Pareto optimality condition in the candidate data processing paths based on the trade-off relationship between each optimization objective.

[0014] Preferably, the resource configuration module is used to filter a subset of runtime data and input it to each data processing path; based on the output results of the data processing paths, it predicts the resource bottleneck trend of the runtime; and based on the resource bottleneck trend, it predictively adjusts the resource configuration strategy, including: Analyze customer needs to determine target business scenarios; filter the subset of operational data that matches the characteristics of the target business scenarios, and input the subset of operational data one by one into each data processing path; The output results of several intermediate ports that connect the data processing path with the business architecture under the running end business model are obtained, and the credibility of the output results is identified in order to predict the resource bottleneck trend of the running end at the corresponding position of each intermediate port. The resource bottleneck trend is traced to determine the predicted location of the resource bottleneck trend and the resource gap attribute, and the resource allocation strategy for the business architecture is predictively adjusted; wherein, the resource allocation strategy includes the scheduling location and / or scheduling quantity of computing resources and / or memory resources.

[0015] The beneficial effects of the above-mentioned technical solutions provided in the embodiments of the present invention include at least the following: The data processing and resource allocation method and system based on multi-objective optimization provided in the above embodiments of the present invention adaptively determine the data collection mode by monitoring the real-time operating status of the runtime and tagging the collected operating data, thereby achieving effective perception and data governance of the runtime's operating status. Based on this, a runtime business model is constructed using historical information of the runtime's business flow transmission paths and business architecture configuration information. This digitally abstracts and reuses the runtime's existing business processing capabilities, fully utilizing the runtime's existing business flow transmission path resources and business architecture resources, thus avoiding repetitive architecture configuration work. Furthermore, by setting... This multi-objective engine, incorporating multiple optimization objective functions and Pareto optimality algorithms, selects candidate data processing paths from a set of business flow transmission paths that can balance multiple mutually constraining objectives such as processing latency, throughput, and resource occupancy, effectively resolving the trade-offs between multiple objectives. Finally, by inputting a subset of runtime data matching the target business scenario into each data processing path for simulation, and identifying credibility based on performance index deviations and logical constraints in the intermediate port output results, it predictively adjusts the scheduling position and quantity of computing and / or memory resources before resource bottlenecks actually occur, achieving a shift from passive response to proactive prevention in resource allocation. The various steps of this invention are interconnected, forming a complete closed-loop technical solution from runtime status awareness, business capability modeling, multi-objective path optimization to predictive resource allocation. This systematically improves the business processing efficiency, resource utilization, and comprehensive multi-objective optimization level of the runtime in multi-service concurrent scenarios.

[0016] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the data processing and resource allocation method based on multi-objective optimization provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a data processing and resource allocation system based on multi-objective optimization provided in an embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0020] In the description of this invention, it should be noted that the terms center, up, down, left, right, vertical, horizontal, inner, outer, far, near, front, and back, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms first, second, and third are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] In this application, the "running end" refers to a distributed computing system comprising several servers interconnected by a communication network, including but not limited to cloud server clusters, distributed computing node groups, or enterprise-level data center systems. A portion of these servers, through logical combination and functional collaboration, form a multi-level business architecture for processing specific business operations. For clarity, in this application, "collection mode" refers to the scheduling methods at the collection level, such as time strategy, frequency strategy, and address strategy for data collection, while "modality" in the modality label refers to the type and attributes of the data itself (such as text, image, time series, etc.). The two terms have a clear boundary within the terminology system.

[0023] Please see Figure 1 As shown in the embodiment of the present invention, a data processing and resource allocation method based on multi-objective optimization is provided. This data processing and resource allocation method based on multi-objective optimization includes: Monitor the real-time running status of the runtime to determine the runtime data collection mode; based on the runtime data collection mode, generate several tagged runtime data subsets and preprocess each runtime data subset; Based on the business processing attributes of the runtime, a runtime business model is constructed; the business processing attributes include business flow path sub-attributes and business architecture sub-attributes; a multi-objective engine is set for the runtime business model, and several data processing paths are determined based on the optimization algorithm of the multi-objective engine; A subset of runtime data is selected and input into each data processing path. Based on the output of the data processing path, the resource bottleneck trend at the runtime is predicted. Based on the resource bottleneck trend, the resource allocation strategy is predictively adjusted.

[0024] The above embodiments construct a runtime business model adapted to business processing and combine it with a multi-objective engine to form multiple data processing paths, providing adapted paths for each business. Furthermore, predictive resource allocation is implemented based on the output of the data processing paths, meeting the needs for effective multi-objective optimization and resource allocation of businesses.

[0025] The beneficial effects of the above embodiments are as follows: By monitoring the real-time operating status of the runtime and determining the operating data collection mode accordingly, adaptive perception and targeted data collection of the operating status are achieved; based on this, a runtime business model is constructed, and the historically accumulated business flow transmission path resources and business architecture resources of the runtime are uniformly digitally modeled, providing a reusable model foundation for subsequent path optimization; furthermore, by setting a multi-objective engine, several data processing paths suitable for multi-objective trade-offs are selected from the business model using optimization algorithms; finally, by simulating the operation of the data processing paths, the resource bottleneck trend of the runtime is predicted in advance, and the resource allocation strategy is predictively adjusted accordingly. This forms a complete technical closed loop of perception, modeling, optimization, prediction, and allocation, transforming the passive emergency response after a potential resource bottleneck occurs into proactive preventative adjustment before it occurs, reducing the risk of business performance degradation due to insufficient resources.

[0026] In another embodiment, the real-time operating status of the runtime is monitored to determine the runtime data collection mode; based on the runtime data collection mode, several tagged runtime data subsets are generated, and each runtime data subset is preprocessed, including: Monitor the real-time execution status of each business thread under the runtime; the real-time execution status includes the real-time data processing rate and the real-time result output rate of each business thread; determine the distribution of the effective running time interval of each business thread based on the real-time execution status; Based on the distribution of the effective running time intervals of each business thread and its location within the virtual interval, the running data collection mode for the running end is determined; wherein, the running data collection mode includes the collection time and the process address where the collection takes place; Based on the data collection mode, collect raw data queues from each business thread; perform source and modality tagging on each raw data queue to generate several subsets of running data; Based on the modal labels and data volume of each subset of running data, noise reduction and deduplication preprocessing are performed on each subset of running data.

[0027] In the above embodiments, the runtime includes several interconnected servers, some of which can be combined to form a business architecture. External input services are loaded into the business architecture, flowing within it and undergoing processing at various levels. It can be understood that each level of processing outputs a stage result, which is then input to another level to continue the business processing flow, ultimately outputting the final result at the bottom layer of the business architecture. As can be seen from the above, the input services flow along corresponding paths within the business architecture, and the distribution of these paths directly determines the computational mode the input services will receive. In actual operation, the operational status of each server under the runtime directly affects the actual processing performance of the services. In order for the constructed runtime business model to obtain effective resource scheduling and configuration in all data processing paths within itself, it is necessary to test the processing performance of each data processing path. The above-mentioned processing performance test requires the assistance of the running data of each server under the runtime. That is, it is necessary to first obtain the running data of each server, and then process the running data according to its source and modality to form a subset of running data that can be used to test the processing performance of each data processing path under the runtime business model, thus providing a data foundation for subsequent testing operations.

[0028] Specifically, the system monitors the real-time data processing rate (i.e., the amount of data processed per unit time) and the real-time result output rate (i.e., the amount of result data output per unit time during the execution of the business thread) of each business thread under the server. A threshold comparison is performed on the real-time data processing rate and the real-time result output rate. The first and second rate thresholds are preset fixed values ​​or adaptive thresholds dynamically calculated based on the historical running data of the server. The time intervals where the real-time data processing rate is greater than the first rate threshold and / or the real-time result output rate is greater than the second rate threshold are determined as effective running time intervals. All effective running time intervals are then arranged chronologically to form an effective running time interval distribution. Based on this, the collection time and frequency for collecting running data from the server are determined by the distribution of effective running time intervals. The process address where the data collection is performed is determined by the virtual interval location corresponding to the virtual network of the server for each business thread. The collection time may, but is not limited to, coincide with the effective running time interval, and the process address where the data collection is performed may, but is not limited to, be the process location within the business thread corresponding to the data collection. The virtual interval location refers to the logical address range or virtual local area network segment corresponding to each business thread in the virtual network topology of the runtime, used to identify the logical affiliation of the business thread in the distributed architecture of the runtime. The system pre-stores a mapping table of virtual interval locations to physical computing resources and process addresses. By querying this mapping table, the process address where the data collection is located is determined according to the virtual interval location. Considering that the runtime data collected from the business threads may include different modalities such as text and images, in order to adapt to the differences in format and data volume of different modal data, raw runtime data queues from each business thread are collected according to the runtime data collection mode. Then, the source and modality tags of each raw runtime data queue are processed to generate several runtime data subsets. Each runtime data subset has tags for the runtime data source address and modality type, providing a basis for subsequent selection of runtime data subsets. Finally, based on the modality type tag and data volume of each runtime data subset, preprocessing such as noise interference removal and duplicate data merging is performed on each runtime data subset to ensure the accuracy of the runtime data subsets.

[0029] The beneficial effects of the above embodiments are as follows: By monitoring the real-time data processing rate and real-time result output rate of each business thread under the running terminal, the effective running time interval of each business thread is determined based on the rate threshold, and the collection time and collection process address are determined by combining the logical belonging position of each thread in the virtual network. This reduces data collection during unnecessary periods and lowers the redundancy of data collection. At the same time, by performing source and modal tagging processing on the collected raw running data, a subset of running data with source address tags and modal type tags is generated, and noise reduction and deduplication preprocessing is performed, providing a high-quality and easily searchable data foundation for subsequent data filtering and path simulation.

[0030] In another embodiment, a runtime business model is constructed based on the runtime's business processing attributes; a multi-objective engine is set for the runtime business model, and several data processing paths are determined based on the multi-objective engine's optimization algorithm, including: Obtain the business flow transmission path information and historical business architecture configuration information from the historical logs of the runtime; The path usage of the service flow transmission path information is identified to obtain the service flow path sub-attributes; among them, the service flow path sub-attributes include the distribution location of the service flow transmission paths in normal working state; The historical business architecture configuration information is used to identify the architecture hierarchy integrity to obtain business architecture sub-attributes; among them, the business architecture sub-attributes include the business architecture types that are in a valid and complete state; Based on the business flow path sub-attributes and business architecture sub-attributes, a business flow transmission path set and a business architecture set are formed to construct the runtime business model; the runtime business model includes the business flow transmission path used to process business and the business architecture with embedded computing plugins. Analyze customer needs to obtain the types of business processing results the customer expects to obtain; generate a multi-objective engine based on the types of business processing results. The multi-objective engine filters all business flow transmission paths under the runtime business model to determine a set of candidate data processing paths. The multi-objective engine is used to find the best path or path combination that satisfies the Pareto optimality condition among the candidate data processing paths based on the trade-off relationship between each optimization objective.

[0031] In practice, the above embodiments involve the runtime establishing paths for external business flow transmission and multi-layered architectures for handling business during historical operations. These paths and architectures are retained internally for a period to facilitate future use. Therefore, the runtime's historical logs generate corresponding business flow transmission path information and business architecture configuration records. To filter currently usable business flow transmission paths and architectures, the runtime identifies the path usage of these paths, determining the distribution of normally functioning business flow transmission paths. Furthermore, it identifies the architectural hierarchy integrity of historical business architecture configuration information, identifying the types of business architectures in a valid and complete state. This results in the formation of business flow path sub-attributes and business architecture sub-attributes. It can be understood that these business flow path sub-attributes and business architecture sub-attributes represent the attributes of the business flow transmission paths and business architectures that the runtime can use. Based on these business flow path sub-attributes and business architecture sub-attributes, a set of business flow transmission paths and a set of business architectures are formed, each corresponding to a set of business flow transmission paths and business architectures that the runtime can use. Based on the aforementioned business flow transmission path set and business architecture set, a runtime business model is constructed to provide a foundation for subsequent business processing. This runtime business model is a digital mapping model built upon historical business flow transmission path information and business architecture configuration information to simulate the runtime's business processing capabilities. This model includes a business flow transmission path set and a business architecture set, providing a simulation environment for business processing logic during resource configuration optimization. Furthermore, customer requirements are analyzed to obtain the types of business processing results the customer expects. Based on these results, a multi-objective engine is generated. This engine contains multiple optimization objective functions corresponding to the business processing result type, and optimization algorithms for Pareto optimal solutions among these objective functions. These objective functions have mutually reinforcing or mutually restrictive relationships. This process filters all business flow transmission paths under the runtime business model, determining a set of candidate data processing paths. The multi-objective engine then uses these candidate paths to find the optimal path or path combination that satisfies the Pareto optimal condition, based on the trade-offs between the optimization objectives.

[0032] The beneficial effects of the above embodiments are as follows: By extracting business flow transmission path information and historical business architecture configuration information from the historical logs of the runtime, and identifying and filtering the usage status of the path (whether it is in normal working condition) and the hierarchical integrity of the architecture (whether it is in a valid and complete state), it is ensured that the constructed runtime business model only contains currently available business flow transmission paths and business architecture resources, thereby improving the consistency between the model and the actual operating environment. On this basis, by analyzing customer requirements to determine the business processing result type and generating a multi-objective engine accordingly, and then filtering candidate data processing paths from the set of business flow transmission paths, the multi-objective engine performs Pareto optimal solutions among various optimization objectives, thereby achieving a scientific trade-off between multiple mutually constraining objectives and overcoming the technical difficulty of simultaneously considering multiple objectives.

[0033] In another embodiment, a subset of runtime data is selected and input into each data processing path. Based on the output results of the data processing paths, the resource bottleneck trend at the runtime is predicted. Based on the resource bottleneck trend, the resource allocation strategy is predictively adjusted, including: Analyze customer needs to determine target business scenarios; filter the subset of operational data that matches the characteristics of the target business scenarios, and input a subset of operational data into each data processing path; Obtain the output results of several intermediate ports that connect the data processing path with the business architecture under the running business model, identify the credibility of the output results, and predict the resource bottleneck trend of the running end at the corresponding position of each intermediate port. The resource bottleneck trend is traced to determine the predicted location of the resource bottleneck trend and the attribute of the resource gap, and the resource allocation strategy for the business architecture is adjusted predictively; the resource allocation strategy includes the scheduling location and / or scheduling quantity of computing resources and / or memory resources.

[0034] In practical operation, the above embodiments analyze customer needs to determine the target business scenario. The target business scenario is obtained through natural language processing (NLP) technology, which performs intent recognition and key element extraction on the customer need text, including feature dimensions such as business type, data scale, and processing timeliness requirements. Based on the characteristics of the target business scenario, a matching subset of runtime data is selected. The matching method is to calculate the cosine similarity between the label vector of the runtime data subset and the feature vector of the target business scenario, and select the runtime data subset with a similarity greater than a preset threshold as the simulation input. The data processing path includes several intermediate ports, each of which interfaces with each level of the business architecture under the runtime business model. When a runtime data subset is input to the data processing path, corresponding results are output at each intermediate port. The credibility of the output results of each intermediate port is assessed using the following methods: (a) For the output of each intermediate port, extract its performance indicators (including processing time, throughput, and resource utilization rate), calculate the relative deviation rate between the performance indicators and the historical benchmark performance indicators of the corresponding position, and if the relative deviation rate exceeds the first deviation threshold, it is determined that the intermediate port has a resource bottleneck risk. (b) and / or, verify the logical constraint relationship of the output performance indicators between each intermediate port. The logical constraint relationship includes that the cumulative processing time between upstream and downstream ports should be less than the total time threshold, and the sum of the resource occupancy rates of each port should be less than the total resource limit. If the verification result does not meet the preset set of logical constraint conditions C, it is determined that the corresponding intermediate port chain has a resource bottleneck risk. (c) Based on the above deviation judgment and / or logical self-consistency judgment results, a weighted fusion method (weight coefficients are preset according to the business type or obtained through machine learning training) is used to calculate the comprehensive credibility score of each intermediate port. If the comprehensive credibility score is lower than the credibility threshold, it is determined that there is a resource bottleneck risk at the corresponding position of the intermediate port. The credibility threshold is preset according to the business type or dynamically adjusted.

[0035] If the confidence level is less than the confidence level threshold, a resource bottleneck risk is identified at the corresponding intermediate port, and the business architecture level connected to the intermediate port is determined as the predicted location of the resource bottleneck. The resource bottleneck trend is traced back to obtain the predicted location and the type and quantity of missing resources. The resource gap attribute refers to the type and quantity of computing and / or memory resources missing at the predicted location of the resource bottleneck. This attribute guides the formulation of resource allocation strategies, ensuring that appropriate types and quantities of computing and / or memory resources are scheduled to the corresponding business architecture level. This guarantees that the business architecture level has sufficient resource reserves before the resource bottleneck occurs, meeting the needs of effective multi-objective optimization and resource allocation for the business.

[0036] It needs to be further clarified that the relationship between the service flow transmission path and the data processing path is as follows: The service flow transmission path is the set of all physical or logical transmission channels historically retained and currently available on the runtime for carrying service data flow. This set reflects all available path resources of the runtime in terms of service processing capabilities. The data processing path, on the other hand, is a subset of effective paths selected by the multi-objective engine from the set of service flow transmission paths based on various optimization objective functions. These paths satisfy specific optimization constraints (including but not limited to latency minimization, throughput maximization, and resource occupancy balancing) and participate in multi-objective Pareto optimality solutions. In other words, the data processing path is a specific subset of the service flow transmission paths after multi-objective optimization selection; the two are related as a whole set and a subset.

[0037] The computing plugins refer to executable computing units that are pre-compiled and deployed in various layers of the business architecture, including but not limited to dynamic link libraries, containerized microservices, script engines, or hardware-accelerated operators. The computing plugins are used to perform corresponding data processing operations when business data flows through various layers of the business architecture. The computing plugins transfer data to each other through standardized interfaces, together forming a complete business processing chain.

[0038] The beneficial effects of the above embodiments are as follows: By analyzing customer needs to determine the target business scenario (including features such as business type, data scale, and processing timeliness requirements), and by selecting a subset of running data that matches the features of the target business scenario based on vector cosine similarity as simulation input, the high correlation between the simulation input data and the current business to be processed is ensured, improving the pertinence and effectiveness of path verification. At the same time, by performing multi-dimensional credibility identification on the output results of each intermediate port of the data processing path (judging the relative deviation rate of performance indicators, verifying the logical constraint relationship between ports, and weighted fusion comprehensive scoring), the specific intermediate port with resource bottleneck risk and its corresponding business architecture level can be accurately located, and the type and quantity of missing resources can be determined. Thus, before the resource bottleneck occurs, the appropriate type and quantity of resources can be accurately scheduled to the target location, realizing the precise positioning and on-demand replenishment of resource allocation.

[0039] In one embodiment, predicting the resource bottleneck trend of the runtime based on the output of the data processing path can be implemented as follows: Obtain the current queue length to be processed, current resource utilization rate, current average response latency, historical peak queue length, resource utilization warning threshold and baseline response latency of each intermediate port on the data processing path; Based on the current pending queue length, current resource utilization, current average response latency, historical peak queue length, resource utilization warning threshold, and baseline response latency of each intermediate port, the resource bottleneck urgency index for each intermediate port is determined. The formula for calculating the resource bottleneck urgency index for each intermediate port is as follows: ; The resource bottleneck urgency index is a dimensionless value used to characterize the urgency of the resource bottleneck occurring at the intermediate port. The larger the value of the resource bottleneck urgency index, the higher the urgency of the resource bottleneck occurring at the intermediate port. Wherein, the current queue length to be processed The number of pending tasks currently backlogged at the intermediate port is a dimensionless integer, read from the real-time cache queue of the task scheduler at the runtime; the historical peak queue length... This represents the maximum task backlog that the intermediate port has experienced during its historical operating cycle. It is a dimensionless integer obtained from the historical performance logs of the operating terminal; the current resource utilization rate... This indicates the actual percentage of current resource occupancy at the intermediate port. The resources include computing power and / or memory resources. The current resource occupancy rate is a dimensionless value ranging from 0 to 1, read in real-time from the runtime resource monitoring module. The resource occupancy warning threshold... This represents the system's preset maximum percentage of resource security usage, a dimensionless fixed value preset according to the operational security specifications; the current average response latency... The average time taken by the intermediate port to process a single task is expressed in units of time and is calculated in real time from the runtime performance monitoring logs using a sliding window; the baseline response latency is... This represents the standard processing time of the intermediate port under historical normal load conditions, expressed in units of time, and extracted from the historical performance baseline database of the running end. This represents the queue pressure weighting coefficient. This represents the resource pressure weighting coefficient. This represents the weighting coefficients for delayed pressure; all three are dimensionless positive numbers and satisfy the following conditions: The determination is made using either the analytic hierarchy process or offline machine learning training, depending on the business type. This represents the latency sensitivity coefficient, a dimensionless positive number, preset according to the business's sensitivity to real-time performance. This indicates the zero-tolerance factor, which is a pre-defined, extremely small positive constant, for example... When the length of the historical peak queue When it is zero, with replace The division calculation of the historical peak queue length item is involved; when the baseline response delay... When it is zero, with replace It participates in the division calculation of the baseline response delay term.

[0040] Based on the ranking of the resource bottleneck urgency index of each intermediate port from high to low, the intermediate ports with resource bottleneck risk and their risk levels are determined. Specifically, after calculating the resource bottleneck urgency index of each intermediate port, the system sorts the intermediate ports in descending order of the index value. The higher the index value, the greater the urgency of the resource bottleneck at the business architecture level corresponding to the intermediate port. Subsequently, intermediate ports whose resource bottleneck urgency index exceeds a preset risk threshold are selected as target intermediate ports for resource gap assessment, while intermediate ports that do not exceed the preset risk threshold are not included in the subsequent assessment. Based on this, for the identified target intermediate ports, the system calculates the resource gap warning value for each target intermediate port in descending order of its resource bottleneck urgency index, and executes predictive adjustments to the resource allocation strategy in descending order of the resource gap warning values. That is, the higher the warning value of the target intermediate port, the greater its resource gap and the more urgent the time. The system prioritizes scheduling computing power resources and / or memory resources for it, thereby realizing a complete resource bottleneck response process from risk positioning and target selection to gap quantification and orderly allocation.

[0041] In the above embodiments, the resource bottleneck is not caused by a single factor, but is the result of the combined pressure of queue backlog, resource consumption, and response latency. This formula integrates these three dimensions into a unified calculation framework: the first term is the queue pressure term, which is the ratio of the current queue length to the historical peak queue length, reflecting the relative level of task backlog at this intermediate port; the second term is the resource pressure term, which is the ratio of the current resource utilization rate to the preset warning threshold, reflecting the current resource consumption safety margin at this intermediate port; the third term is the latency deterioration term, which adopts an exponential function form with the natural constant e as the base, i.e. Its design principle is to characterize the avalanche effect of response latency: when the actual response latency begins to exceed the historical baseline, the task queuing and resource contention within the system will enter a nonlinear positive feedback loop, and the risk of resource bottlenecks will deteriorate exponentially rather than simply grow linearly.

[0042] The beneficial effects of the above embodiments are as follows: This formula transforms the qualitative judgment that originally relied on human experience or simple threshold comparison into a standardized quantitative score. The system only needs to calculate the RBUI value of each intermediate port and sort them in descending order to quickly and objectively filter out the highest risk positions and their urgency ranking from a large number of ports, avoiding the subjectivity and randomness of human judgment. At the same time, the introduction of the exponential term makes the formula have extremely high predictive sensitivity in latency-sensitive business scenarios. It can identify potential bottlenecks in advance when the response latency just begins to deteriorate and has not yet caused significant system degradation, thus gaining valuable lead time for subsequent resource allocation and realizing the transformation from passive response to proactive prevention.

[0043] In one embodiment, after determining the intermediate port with resource bottleneck risk and its risk level, the process may further include: Intermediate ports whose resource bottleneck urgency index exceeds a preset risk threshold are selected as target intermediate ports for resource gap assessment. At this point, correspondingly, determining the predicted location of the resource bottleneck trend and the resource gap attributes, and predictively adjusting the resource allocation strategy for the business architecture, may include: Obtain the predicted net resource gap, bottleneck remaining time interval, and system scheduling response window for each target intermediate port; The resource gap warning value for each target intermediate port is calculated based on its predicted net resource gap, remaining bottleneck time interval, and system scheduling response window. The formula for calculating the resource gap warning value for each target intermediate port is as follows: ; Wherein, the time urgency coefficient is The dimension of the resource gap warning value is consistent with the dimension of the predicted net resource gap, and is used to characterize the amount of resources that the target intermediate port needs to supplement and the urgency of its scheduling. The larger the value of the resource gap warning value, the more resources the target intermediate port needs to supplement and the more urgent the scheduling. Among them, the predicted net resource gap The remaining time interval of the bottleneck represents the difference between the amount of resources required by the target intermediate port at a future predicted time and the amount of resources currently available, measured in resource units. It is calculated by simulating a subset of the running data into the data processing path and then calculating based on the simulation output. This represents the time span from the current moment to the predicted moment when the bottleneck actually occurs. The dimension is time, and it is a non-negative value. It is calculated by extrapolating from the current resource consumption rate and combining it with the inflection point timestamp from the simulation output. When the predicted bottleneck has already occurred, it is taken as... The system scheduling response window It represents the average response time required by the runtime resource scheduling system from issuing a resource allocation instruction to completing the actual resource expansion. The unit of measurement is time, and it is obtained based on the historical average time consumption statistics of the runtime resource scheduler. This represents the resource type adjustment coefficient, which is a dimensionless positive number. It is preset according to the resource type and is used to distinguish the weight differences of different resource types in terms of supply urgency. This represents the baseline range for time urgency. It is a dimensionless positive number used to control the maximum additional warning range caused by time urgency. It is preset according to business priority. This represents the decay rate coefficient, a dimensionless positive number used to control how quickly the urgency decreases over time, and is preset according to the rate of elastic scaling of runtime resources.

[0044] Based on the ranking of the resource gap warning values ​​of each target intermediate port from high to low, predictive adjustments to the resource allocation strategy are performed on each target intermediate port in sequence. Specifically, after calculating the resource gap warning value for each target intermediate port, the system sorts all target intermediate ports in descending order of the value. The higher the resource gap warning value, the greater the resource supply required by the target intermediate port and the higher the urgency of the supply. Based on the sorting results, the system performs predictive adjustments to the resource allocation strategy for each target intermediate port in sequence: that is, it prioritizes determining the resource gap attributes (including the type identifier and quantity of the required computing power resources and / or memory resources) and the corresponding business architecture level position for the top-ranked target intermediate ports, and then pre-schedules the corresponding resources to the business architecture level according to the scheduling position and quantity. After completion, the system processes the next target intermediate port in sequence until the resource allocation adjustment of all target intermediate ports is completed. Thus, the order of the resource gap warning values ​​directly determines the order in which each target intermediate port receives resource supply, ensuring that limited resources are prioritized to serve the port with the largest gap and the most urgent time, thereby realizing a complete resource bottleneck response process from risk positioning, target screening, gap quantification to orderly allocation.

[0045] In the above embodiments, after determining that a certain intermediate port has a resource bottleneck risk, the system still needs to quantify two issues: how much resource needs to be added and how urgent the situation is. This formula integrates these two dimensions into a unified calculation framework. The purpose of the formula is to determine the net shortage of basic resources. This is the difference between the simulated peak resource demand and the current available resource quantity, providing a quantitative basis for resource allocation. The key to the formula is multiplying this by a pre-existing time urgency coefficient. .when When it approaches zero (a bottleneck is about to occur), the exponential term The time urgency coefficient approaches 1 and reaches its maximum value. The warning value was significantly amplified, indicating that immediate deployment was necessary; when Much larger When the bottleneck is still far away, the index term approaches 0, the urgency coefficient approaches 1, and the warning value falls back to the basic level, indicating that it only needs to be included in the regular scheduling plan.

[0046] The beneficial effects of the above embodiments: This formula enables the system to simultaneously output the quantity basis for resource scheduling ( The formula combines the contribution of the resource gap (the size of the resource gap) and the priority (the contribution of the time urgency coefficient) to achieve a joint quantification of the quantity and urgency of the resource shortage. This overcomes the core defect of conventional solutions that only focus on the size of the resource gap while ignoring the proximity of the bottleneck. It avoids resource mismatch problems such as the actual occurrence of the bottleneck due to supply delays or the premature occupation of a large amount of resources by non-urgent ports, thus crowding out the supply capacity of urgent ports. At the same time, the output of this formula is directly used to predictively adjust the resource allocation strategy for each target intermediate port, ensuring that limited computing power and memory resources are prioritized to serve the target intermediate port with the largest resource gap and the most urgent time, thereby improving the timeliness and utilization of resource scheduling.

[0047] Please see Figure 2 As shown, an embodiment of this application provides a data processing and resource allocation system based on multi-objective optimization. This data processing and resource allocation system based on multi-objective optimization includes: The data acquisition mode determination module is used to monitor the real-time operating status of the runtime and thereby determine the data acquisition mode for the runtime. The data subset processing module is used to generate several labeled subsets of running data according to the running data acquisition mode, and to preprocess each subset of running data. The business model building module is used to build the runtime business model based on the runtime's business processing attributes; among which, the business processing attributes include business flow path sub-attributes and business architecture sub-attributes. The path determination module is used to set a multi-objective engine for the runtime business model and determine several data processing paths based on the optimization algorithm of the multi-objective engine. The resource configuration module is used to filter a subset of runtime data and input it into each data processing path. Based on the output results of the data processing path, it predicts the resource bottleneck trend at the runtime end and adjusts the resource configuration strategy predictively based on the resource bottleneck trend.

[0048] The beneficial effects of the above embodiments are that the data processing and resource allocation system based on multi-objective optimization constructs a runtime business model adapted to business processing, and combines a multi-objective engine to form multiple data processing paths, providing adapted paths for each business, and also implements predictive resource allocation based on the output of the data processing paths, thereby meeting the needs for effective multi-objective optimization and resource allocation of businesses.

[0049] In another embodiment, the acquisition mode determination module is used to monitor the real-time operating status of the runtime terminal to determine the data acquisition mode for the runtime terminal, including: Monitor the real-time execution status of each business thread under the runtime; the real-time execution status includes the real-time data processing rate and the real-time result output rate of each business thread; determine the distribution of the effective running time interval of each business thread based on the real-time execution status; Based on the distribution of the effective running time intervals of each business thread and its location within the virtual interval, the running data collection mode for the running end is determined; wherein, the running data collection mode includes the collection time and the process address where the collection takes place; The data subset processing module is used to generate several labeled subsets of runtime data based on the runtime data acquisition mode, and to preprocess each subset of runtime data, including: Based on the data collection mode, collect raw data queues from each business thread; perform source and modality tagging on each raw data queue to generate several subsets of running data; Based on the modal labels and data volume of each subset of running data, noise reduction and deduplication preprocessing are performed on each subset of running data.

[0050] In another embodiment, the business model building module is used to build a runtime business model based on the runtime's business processing attributes, including: Obtain the business flow transmission path information and historical business architecture configuration information from the historical logs of the runtime; The path usage of the service flow transmission path information is identified to obtain the service flow path sub-attributes; among them, the service flow path sub-attributes include the distribution location of the service flow transmission paths in normal working state; The historical business architecture configuration information is used to identify the architecture hierarchy integrity to obtain business architecture sub-attributes; among them, the business architecture sub-attributes include the business architecture types that are in a valid and complete state; Based on the business flow path sub-attributes and business architecture sub-attributes, a business flow transmission path set and a business architecture set are formed to construct the runtime business model; the runtime business model includes the business flow transmission path used to process business and the business architecture with embedded computing plugins. The path determination module is used to set a multi-objective engine for the runtime business model, and determine several data processing paths according to the optimization algorithm of the multi-objective engine (modification: consistent with the description of the path determination module in the amended claim 7, modifying the original determination of several data processing paths corresponding to multiple objectives to determine several data processing paths according to the optimization algorithm of the multi-objective engine), including: Analyze customer needs to obtain the types of business processing results the customer expects to obtain; generate a multi-objective engine based on the types of business processing results. The multi-objective engine filters all business flow transmission paths under the runtime business model to determine a set of candidate data processing paths. The multi-objective engine is used to find the best path or path combination that satisfies the Pareto optimality condition among the candidate data processing paths based on the trade-off relationship between each optimization objective.

[0051] In another embodiment, the resource configuration module is used to filter a subset of runtime data input to each data processing path, predict the resource bottleneck trend at the runtime based on the output results of the data processing paths, and predictively adjust the resource configuration strategy based on the resource bottleneck trend, including: Analyze customer needs to determine target business scenarios; filter the subset of operational data that matches the characteristics of the target business scenarios, and input a subset of operational data into each data processing path; Obtain the output results of several intermediate ports that connect the data processing path with the business architecture under the running business model, identify the credibility of the output results, and predict the resource bottleneck trend of the running end at the corresponding position of each intermediate port. The resource bottleneck trend is traced to determine the predicted location of the resource bottleneck trend and the attribute of the resource gap, and the resource allocation strategy for the business architecture is adjusted predictively; the resource allocation strategy includes the scheduling location and / or scheduling quantity of computing resources and / or memory resources.

[0052] The data processing and resource allocation system based on multi-objective optimization of the present invention has the same operation and effect as the aforementioned data processing and resource allocation method based on multi-objective optimization, and will not be described again here.

[0053] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. This disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A data processing and resource allocation method based on multi-objective optimization, characterized in that, include: Monitor the real-time operating status of the running terminal to determine the data collection mode for the running terminal; Based on the aforementioned operational data acquisition mode, several tagged operational data subsets are generated, and each operational data subset is preprocessed. Based on the business processing attributes of the runtime, a runtime business model is constructed; wherein, the business processing attributes include business flow path sub-attributes and business architecture sub-attributes; a multi-objective engine is set for the runtime business model, and several data processing paths are determined according to the optimization algorithm of the multi-objective engine; A subset of runtime data is selected and input into each data processing path. Based on the output results of the data processing paths, the resource bottleneck trend of the runtime is predicted. Based on the resource bottleneck trend, the resource allocation strategy is predictively adjusted.

2. The data processing and resource allocation method based on multi-objective optimization as described in claim 1, characterized in that: Monitor the real-time operating status of the running terminal to determine the data collection mode for the running terminal; Based on the aforementioned operational data acquisition mode, several tagged operational data subsets are generated, and each operational data subset is preprocessed, including: Monitor the real-time execution status of each business thread under the runtime; wherein, the real-time execution status includes the real-time data processing rate and the real-time result output rate of each business thread; determine the distribution of the effective running time interval of each business thread based on the real-time execution status; Based on the distribution of effective running time intervals and the location of each virtual interval for each business thread, the running data collection mode for the running terminal is determined; wherein, the running data collection mode includes the collection time and the process address where the collection is performed; According to the aforementioned operational data acquisition mode, raw operational data queues from each business thread are collected; each raw operational data queue is processed for source and modality tagging to generate several operational data subsets; Based on the modal labels and data volume of each subset of running data, noise reduction and deduplication preprocessing are performed on each subset of running data.

3. The data processing and resource allocation method based on multi-objective optimization as described in claim 1, characterized in that: Based on the business processing attributes of the runtime terminal, construct the runtime terminal business model; A multi-objective engine is set for the runtime business model, and several data processing paths are determined based on the optimization algorithm of the multi-objective engine, including: Obtain the service flow transmission path information and historical service architecture configuration information of the running terminal from its historical logs. The path usage of the service flow transmission path information is identified to obtain service flow path sub-attributes; wherein, the service flow path sub-attributes include the distribution location of service flow transmission paths in normal working state; The historical business architecture configuration information is subjected to architecture-level integrity identification to obtain business architecture sub-attributes; wherein, the business architecture sub-attributes include business architecture types that are in a valid and complete state; Based on the business flow path sub-attributes and the business architecture sub-attributes, a business flow transmission path set and a business architecture set are formed to construct a runtime business model; wherein, the runtime business model includes a business flow transmission path for processing business and a business architecture with embedded computing plugins; Analyze customer needs to obtain the types of business processing results the customer expects to obtain; generate a multi-objective engine based on the business processing result types; The multi-objective engine filters all service flow transmission paths under the runtime service model to determine a set of candidate data processing paths. The multi-objective engine is used to solve for the best path or path combination that satisfies the Pareto optimality condition in the candidate data processing paths based on the trade-off relationship between each optimization objective.

4. The data processing and resource allocation method based on multi-objective optimization as described in claim 1, characterized in that: A subset of runtime data is selected and input into each data processing path. Based on the output results of the data processing paths, the resource bottleneck trend of the runtime is predicted. Based on the aforementioned resource bottleneck trends, predictively adjust resource allocation strategies, including: Analyze customer needs to determine target business scenarios; filter the subset of operational data that matches the characteristics of the target business scenarios, and input the subset of operational data one by one into each data processing path; The output results of several intermediate ports that connect the data processing path with the business architecture under the running end business model are obtained, and the credibility of the output results is identified in order to predict the resource bottleneck trend of the running end at the corresponding position of each intermediate port. The resource bottleneck trend is traced to determine the predicted location of the resource bottleneck trend and the resource gap attribute, and the resource allocation strategy for the business architecture is predictively adjusted; wherein, the resource allocation strategy includes the scheduling location and / or scheduling quantity of computing resources and / or memory resources.

5. The data processing and resource allocation method based on multi-objective optimization according to claim 1, characterized in that, The step of predicting the resource bottleneck trend of the running terminal based on the output results of the data processing path includes: Obtain the current queue length to be processed, current resource utilization rate, current average response latency, historical peak queue length, resource utilization warning threshold and baseline response latency of each intermediate port on the data processing path; Based on the current pending queue length, current resource utilization rate, current average response latency, historical peak queue length, resource utilization warning threshold, and baseline response latency of each intermediate port, determine the resource bottleneck urgency index of each intermediate port. Among them, the resource bottleneck urgency index of each intermediate port The calculation formula is: ； The resource bottleneck urgency index is used to characterize the urgency of the intermediate port experiencing a resource bottleneck. The larger the value of the resource bottleneck urgency index, the higher the urgency of the intermediate port experiencing a resource bottleneck. This represents the current length of the queue to be processed at this intermediate port. This indicates the current resource utilization rate of the intermediate port; This indicates the resource usage warning threshold for this intermediate port; This represents the current average response latency of the intermediate port. This is the baseline response delay for the intermediate port; This represents the queue pressure weighting coefficient. This represents the resource pressure weighting coefficient. This represents the delay pressure weighting coefficient, and satisfies... ; This represents the delay sensitivity coefficient; This indicates the elimination of the zero fault tolerance factor; Based on the ranking of the resource bottleneck urgency index of each intermediate port from high to low, the intermediate ports with resource bottleneck risk and their risk levels are determined.

6. The data processing and resource allocation method based on multi-objective optimization according to claim 5, characterized in that, After identifying the intermediate ports with resource bottleneck risks and their risk levels, the process also includes: Intermediate ports whose resource bottleneck urgency index exceeds a preset risk threshold are selected as target intermediate ports for resource gap assessment. The step of determining the predicted location of the resource bottleneck trend and the resource gap attributes, and predictively adjusting the resource allocation strategy for the business architecture, includes: Obtain the predicted net resource gap, bottleneck remaining time interval, and system scheduling response window for each target intermediate port; The resource gap warning value for each target intermediate port is calculated based on its predicted net resource gap, remaining bottleneck time interval, and system scheduling response window; where the resource gap warning value for each target intermediate port is... The calculation formula is: ； The larger the resource gap warning value, the more resources the target intermediate port needs to replenish and the more urgent the scheduling; where, This indicates the predicted net resource gap at the intermediate port of the target; This indicates the remaining time interval of the bottleneck at the intermediate port of the target. ; This represents the system scheduling response window of the running terminal to which the intermediate port of the target belongs; This represents the resource type adjustment coefficient for the intermediate port of the target; This indicates the time urgency reference range of the target's intermediate port; This represents the attenuation rate coefficient at the intermediate port of the target. Based on the ranking of the resource gap warning values ​​of each target intermediate port from high to low, predictive adjustments to the resource allocation strategy are performed sequentially on each target intermediate port.

7. A data processing and resource allocation system based on multi-objective optimization, characterized in that, include: The data acquisition mode determination module is used to monitor the real-time operating status of the running terminal to determine the data acquisition mode for the running terminal. The data subset processing module is used to generate several labeled subsets of running data according to the running data acquisition mode, and to preprocess each subset of running data. The business model construction module is used to construct a runtime business model based on the runtime's business processing attributes; wherein, the business processing attributes include business flow path sub-attributes and business architecture sub-attributes; The path determination module is used to set a multi-objective engine for the runtime business model and determine several data processing paths according to the optimization algorithm of the multi-objective engine. The resource configuration module is used to filter a subset of running data and input it to each data processing path. Based on the output results of the data processing path, it predicts the resource bottleneck trend of the running end. Based on the resource bottleneck trend, it predictively adjusts the resource configuration strategy.

8. The data processing and resource allocation system based on multi-objective optimization as described in claim 7, characterized in that: The data acquisition mode determination module is used to monitor the real-time operating status of the running terminal to determine the data acquisition mode for the running terminal, including: Monitor the real-time execution status of each business thread under the runtime; wherein, the real-time execution status includes the real-time data processing rate and the real-time result output rate of each business thread; determine the distribution of the effective running time interval of each business thread based on the real-time execution status; Based on the distribution of effective running time intervals and the location of each virtual interval for each business thread, the running data collection mode for the running terminal is determined; wherein, the running data collection mode includes the collection time and the process address where the collection is performed; The data subset processing module is used to generate several labeled subsets of running data according to the running data acquisition mode, and to preprocess each subset of running data, including: According to the aforementioned operational data acquisition mode, raw operational data queues from each business thread are collected; each raw operational data queue is processed for source and modality tagging to generate several operational data subsets; Based on the modal labels and data volume of each subset of running data, noise reduction and deduplication preprocessing are performed on each subset of running data.

9. The data processing and resource allocation system based on multi-objective optimization as described in claim 8, characterized in that: The business model construction module is used to construct a runtime business model based on the runtime's business processing attributes, including: Obtain the service flow transmission path information and historical service architecture configuration information of the running terminal from its historical logs. The path usage of the service flow transmission path information is identified to obtain service flow path sub-attributes; wherein, the service flow path sub-attributes include the distribution location of service flow transmission paths in normal working state; The historical business architecture configuration information is subjected to architecture-level integrity identification to obtain business architecture sub-attributes; wherein, the business architecture sub-attributes include business architecture types that are in a valid and complete state; Based on the business flow path sub-attributes and the business architecture sub-attributes, a business flow transmission path set and a business architecture set are formed to construct a runtime business model; wherein, the runtime business model includes a business flow transmission path for processing business and a business architecture with embedded computing plugins; The path determination module is used to set a multi-objective engine for the runtime business model, and determine several data processing paths based on the optimization algorithm of the multi-objective engine, including: Analyze customer needs to obtain the types of business processing results the customer expects to obtain; generate a multi-objective engine based on the business processing result types; The multi-objective engine filters all service flow transmission paths under the runtime service model to determine a set of candidate data processing paths. The multi-objective engine is used to solve for the best path or path combination that satisfies the Pareto optimality condition in the candidate data processing paths based on the trade-off relationship between each optimization objective.

10. The data processing and resource allocation system based on multi-objective optimization as described in claim 7, characterized in that: The resource configuration module is used to filter a subset of running data and input it into each data processing path, and predict the resource bottleneck trend of the running end based on the output results of the data processing path. Based on the aforementioned resource bottleneck trends, predictively adjust resource allocation strategies, including: Analyze customer needs to determine target business scenarios; filter the subset of operational data that matches the characteristics of the target business scenarios, and input the subset of operational data one by one into each data processing path; The output results of several intermediate ports that connect the data processing path with the business architecture under the running end business model are obtained, and the credibility of the output results is identified in order to predict the resource bottleneck trend of the running end at the corresponding position of each intermediate port. The resource bottleneck trend is traced to determine the predicted location of the resource bottleneck trend and the resource gap attribute, and the resource allocation strategy for the business architecture is predictively adjusted; wherein, the resource allocation strategy includes the scheduling location and / or scheduling quantity of computing resources and / or memory resources.

Images