Big data task scheduling optimization method and system based on heterogeneous computing

Abstract

The invention provides a big data task scheduling optimization method and system based on heterogeneous computing, and relates to the technical field of big data process.The big data task scheduling optimization method comprises the steps that multi-source big data and transmission parameters of corresponding transmission links are obtained, and associated information is extracted from the multi-source big data; obtaining a target transmission link based on the transmission parameter; based on the associated information, obtaining a plurality of processing sub-items with identifiers and a corresponding execution sequence; obtaining an initial allocation scheme based on the associated information; obtaining running state information of the heterogeneous computing resources to obtain a verification result, and adjusting the initial allocation scheme in combination with the execution sequence to obtain a final allocation scheme; and based on the target transmission link and the final allocation scheme, the processing subitems with the identifiers are processed through the heterogeneous computing resources, and efficient, safe and ordered scheduling optimization of the multi-source big data in the heterogeneous computing environment is realized.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a method and system for optimizing big data task scheduling based on heterogeneous computing. Background Technology

[0002] In general big data intelligent classification and processing scenarios, the system needs to process massive amounts of data simultaneously and support high concurrency and parallel execution of multiple tasks. Such applications place high demands on the task scheduling mechanism: it must not only be able to perform intelligent classification and clustering based on the inherent characteristics of the data, but also dynamically adapt to the underlying heterogeneous computing resources and ensure efficient and low-loss delivery of data in the transmission link to achieve end-to-end low latency and high throughput processing.

[0003] Current mainstream solutions employ AI-driven task clustering combined with FPGA hardware acceleration architecture. They extract multi-source data features through convolutional neural networks and generate task clusters using the K-means algorithm, then map the tasks to FPGA logic units for parallel execution. At the same time, they introduce task dependency graphs and dynamic priority strategies to optimize the execution order, and supplement this with load monitoring to achieve resource pressure balancing, thereby improving overall processing efficiency.

[0004] However, existing solutions have certain shortcomings. For example, they do not fully consider the impact of transmission link status on task scheduling, lack the ability to perceive and adaptively adjust physical layer transmission parameters in real time, which may lead to delays or packet loss during data transmission due to link degradation. In addition, their resource allocation process lacks a verification mechanism for the reliability of the runtime system status, making it difficult to ensure the reliability and consistency of scheduling decisions in complex and ever-changing deployment environments. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for optimizing big data task scheduling based on heterogeneous computing, so as to solve the problems in the prior art that it is difficult to ensure the reliability and consistency of scheduling decisions in complex and ever-changing deployment environments.

[0006] Firstly, this application provides a big data task scheduling optimization method based on heterogeneous computing, including:

[0007] Obtain multi-source big data and corresponding transmission parameters of transmission links, and extract related information from multi-source big data;

[0008] Based on transmission parameters, edge computing technology is used to calibrate the transmission link to obtain the target transmission link;

[0009] Based on the correlation information, the multi-source big data is divided into multiple processing sub-items, and the identification information of the corresponding target transmission link is assigned to each processing sub-item, resulting in multiple identified processing sub-items and their corresponding execution order.

[0010] Based on the association information, the computational requirement parameters of each identified processing sub-item are matched with the configurable parameters of heterogeneous computing resources to obtain an initial allocation scheme.

[0011] Obtain the running status information of heterogeneous computing resources, use blockchain verification technology to verify the credibility of the running status information and the initial allocation scheme, obtain the verification results, and adjust the initial allocation scheme in combination with the execution order to obtain the final allocation scheme;

[0012] Based on the target transmission link and the final allocation scheme, the tagged processing sub-items are processed through heterogeneous computing resources to achieve scheduling optimization of multi-source big data.

[0013] Optionally, the runtime status information of heterogeneous computing resources is obtained, and the credibility of the runtime status information and the initial allocation scheme is verified using blockchain verification technology. The verification result is then used to adjust the initial allocation scheme based on the execution order, resulting in the final allocation scheme, including:

[0014] Obtain the running status information of heterogeneous computing resources, and based on preset verification rules, use the blockchain verification technology pre-configured by each verification node in the blockchain network to verify the credibility of the running status information and the initial allocation scheme, and obtain the verification result of each verification node.

[0015] If the number of verification nodes that pass the verification is greater than the preset number, the final verification result is determined to be a successful verification, and the resource allocation priority of each marked processing sub-item is determined according to the execution order.

[0016] If the number of verification nodes that fail the verification is greater than the preset number, then the unreasonable resource allocation items in the initial allocation scheme are corrected to obtain an intermediate allocation scheme, and the resource allocation priority of each marked processing sub-item is determined in combination with the execution order.

[0017] Based on the resource allocation priority, the resource allocation ratios corresponding to each marked processing sub-item in the initial or intermediate allocation scheme are adjusted to obtain the final allocation scheme.

[0018] Optionally, if the number of verification nodes that fail verification is greater than the preset number, then unreasonable resource allocation items in the initial allocation scheme are corrected to obtain an intermediate allocation scheme, and the resource allocation priority of each identifiable processing sub-item is determined in conjunction with the execution order, including:

[0019] If the number of verification nodes that fail verification is greater than the preset number, the reasons for failure associated with the verification nodes that fail verification will be compared with the resource allocation items in the initial allocation scheme to determine unreasonable resource allocation items.

[0020] For each unreasonable resource allocation item, the resource information that matches the reason for failure is filtered from the configurable parameters, and the resource information is bound to the corresponding identified processing sub-item in the unreasonable resource allocation item in the initial allocation scheme to obtain the intermediate allocation scheme.

[0021] Based on the intermediate allocation scheme and the data association attributes of the calculation requirement parameters, the resource allocation priority of each identified processing sub-item is determined.

[0022] Optionally, based on transmission parameters, edge computing technology is used to calibrate the transmission link to obtain the target transmission link, including:

[0023] Edge computing technology is used to calculate the first deviation value between the transmission rate and the preset standard transmission rate, the second deviation value between the transmission delay and the preset standard transmission delay, and the third deviation value between the signal attenuation degree and the preset standard signal attenuation degree in the transmission parameters.

[0024] Based on the first deviation value, the second deviation value, and the third deviation value, the transmission power, transmission frequency, and transmission protocol of the transmission link are calibrated respectively to obtain the target transmission link.

[0025] Optionally, based on the correlation information, the multi-source big data is divided into multiple processing sub-items, and each processing sub-item is assigned the identification information of the corresponding target transmission link, resulting in multiple identified processing sub-items and their corresponding execution order, including:

[0026] Based on the associated information, the feature dimensions of multi-source big data are determined, and the similarity, correlation and attribute overlap of each feature dimension are calculated. Based on the similarity, correlation and attribute overlap of each feature dimension, the multi-source big data is divided into multiple data combinations.

[0027] Calculate the processing complexity of each data combination based on its data size and dependencies.

[0028] Based on the processing complexity of each data combination, each data combination is divided into multiple processing sub-items, and the identification information of the corresponding target transmission link is assigned to each processing sub-item, resulting in multiple identifiable processing sub-items.

[0029] The dependencies between the identified processing sub-items are hierarchically resolved and topologically sorted to determine the execution order of each identified processing sub-item.

[0030] Optionally, based on the association information, the computational requirement parameters of each identified processing sub-item are matched with the configurable parameters of heterogeneous computing resources to obtain an initial allocation scheme, including:

[0031] Based on the computational requirement parameters of each identified processing sub-item and the configurable parameters of heterogeneous computing resources, an initial matching rule is established between the computational requirement parameters and the configurable parameters. The computational requirement parameters include computation type, computational quantity, and computational response requirements, while the configurable parameters include computational capacity, computational load, and adaptation type.

[0032] Based on the data association attributes in the association information, the parameter weights of each parameter in the initial matching rule are adjusted to obtain the adjusted matching rule;

[0033] Based on the adjusted matching rules, the allocation result of each labeled processing sub-item is determined, and the allocation results of all labeled processing sub-items are integrated to obtain the initial allocation scheme.

[0034] Optionally, based on the target transmission link and the final allocation scheme, the identifiable processing sub-items are processed using heterogeneous computing resources to achieve scheduling optimization for multi-source big data, including:

[0035] According to the final allocation scheme, the identified processing sub-items are transmitted to the computing units corresponding to the heterogeneous computing resources through the target transmission link. Each computing unit processes the identified processing sub-items that have no dependencies in parallel and monitors the processing progress of each computing unit.

[0036] If the processing progress of a computing unit lags behind the preset progress, then based on the final allocation scheme, idle resources in heterogeneous computing resources are allocated to computing units whose processing progress lags behind the preset progress, and the marked processing sub-items are processed in parallel. If the processing progress of a computing unit exceeds the preset progress, then according to the final allocation scheme, the marked processing sub-items are processed in parallel to achieve scheduling optimization of multi-source big data.

[0037] Secondly, this application provides a big data task scheduling optimization system based on heterogeneous computing, comprising:

[0038] The acquisition module is used to acquire multi-source big data and the transmission parameters of the corresponding transmission links, and extract related information from the multi-source big data.

[0039] The calibration module is used to calibrate the transmission link based on transmission parameters and using edge computing technology to obtain the target transmission link;

[0040] The allocation module is used to divide multi-source big data into multiple processing sub-items based on the association information, and assign the corresponding target transmission link identification information to each processing sub-item to obtain multiple identifiable processing sub-items and their corresponding execution order.

[0041] The allocation module is also used to match the computational requirement parameters of each identified processing sub-item with the configurable parameters of heterogeneous computing resources based on the association information to obtain an initial allocation scheme.

[0042] The calibration module is also used to obtain the running status information of heterogeneous computing resources, use blockchain verification technology to verify the credibility of the running status information and the initial allocation scheme, obtain the verification results, and adjust the initial allocation scheme in combination with the execution order to obtain the final allocation scheme.

[0043] The processing module is used to process identifiable sub-items based on the target transmission link and the final allocation scheme through heterogeneous computing resources, so as to achieve scheduling optimization of multi-source big data.

[0044] Thirdly, this application provides an electronic device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor, used to execute computer programs, implements the steps of a big data task scheduling optimization method based on heterogeneous computing as described in the first aspect above.

[0047] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the big data task scheduling optimization method based on heterogeneous computing described in the first aspect above.

[0048] This application provides a big data task scheduling optimization method based on heterogeneous computing. By acquiring multi-source big data and the transmission parameters of corresponding transmission links, and extracting related information from the multi-source big data, it enhances the scheduling strategy's ability to collaboratively understand data content and transmission conditions; reduces transmission delays or data distortion caused by physical layer fluctuations, and ensures high-reliability delivery of task data during the scheduling process; achieves fine-grained task decomposition and link binding, improving the orderliness and traceability of the end-to-end processing flow; ensures the authenticity of resource status data and the rationality of allocation schemes, avoids scheduling deviations caused by false or outdated status information, and improves the system's decision robustness in complex deployment environments.

[0049] Furthermore, the system acquires the operational status information of heterogeneous computing resources, and based on preset verification rules, verifies the credibility of the operational status information and the initial allocation scheme using blockchain verification technology pre-configured on each verification node in the blockchain network, obtaining the verification result of each verification node. If the verification result shows that the number of verification nodes that have passed verification is greater than a preset number, the final verification result is determined to be a successful verification, and the resource allocation priority of each identifiable processing sub-item is determined according to the execution order. If the verification result shows that the number of verification nodes that have failed verification is greater than the preset number, unreasonable resource allocation items in the initial allocation scheme are corrected to obtain an intermediate allocation scheme, and the resource allocation priority of each identifiable processing sub-item is determined in conjunction with the execution order. Based on the resource allocation priority, the resource allocation ratio corresponding to each identifiable processing sub-item in the initial allocation scheme or intermediate allocation scheme is adjusted to obtain the final allocation scheme.

[0050] This solution addresses the unreliable scheduling issue caused by the lack of reliable runtime state verification in existing solutions, improves the consistency and anti-interference capability of scheduling decisions in high-concurrency, multi-task parallel scenarios, and enhances the system's adaptive response level to abnormal states of heterogeneous resources. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a big data task scheduling optimization method based on heterogeneous computing, provided for an embodiment of this application;

[0053] Figure 2 A schematic diagram illustrating a process for obtaining a final allocation scheme, provided as an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of the structure of a big data task scheduling and optimization system based on heterogeneous computing, provided in an embodiment of this application. Detailed Implementation

[0055] To address the scheduling challenges faced by high concurrency and parallel execution of multiple tasks in general big data intelligent classification and processing scenarios, while existing mainstream solutions can achieve a certain degree of parallel optimization through AI feature clustering and FPGA acceleration, they are difficult to coordinately perceive the status of transmission links and ensure the credibility of scheduling decisions. This can easily lead to delays or packet loss when the link deteriorates, and there is a lack of verification mechanisms for the authenticity of the operating status of heterogeneous resources, which affects scheduling consistency and reliability.

[0056] To address this, this application extracts the correlation information of multi-source big data and the transmission parameters of the corresponding transmission links, uses edge computing to dynamically calibrate the links to obtain a stable target transmission path, and divides the processing sub-items and execution order accordingly. Then, it matches the computing requirements of the sub-items with the heterogeneous computing resource capabilities to generate an initial allocation scheme. Furthermore, it introduces blockchain verification technology to verify the credibility of the resource operation status and the initial scheme, and dynamically adjusts the resource allocation in combination with the execution order. Finally, under the synergistic drive of the target transmission link and the trusted scheduling scheme, it completes efficient and reliable big data processing, thereby solving the dual bottlenecks of transmission adaptability and scheduling credibility in existing technologies.

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The core of this application is to provide a big data task scheduling optimization method based on heterogeneous computing, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0059] Step 101: Obtain multi-source big data and the transmission parameters of the corresponding transmission links, and extract related information from the multi-source big data.

[0060] In this step, multi-source big data refers to a massive collection of data originating from different data-generating entities, different data types, or different data collection channels. A transmission link refers to the physical or logical connection channel used to transmit data. Transmission parameters refer to specific indicators describing the transmission performance of the transmission link. Related information refers to the feature-level relationships between different data within multi-source big data.

[0061] In this embodiment, firstly, massive amounts of data from different channels and the performance indicators of the corresponding transmission channels are collected. Secondly, the correlation data between different data at the feature level is mined and extracted from the collected massive amounts of data, providing a basis for subsequent transmission link calibration and data partitioning.

[0062] For example, Platform A, as a general-purpose service platform focused on multi-source data integration and processing, needs to optimize the scheduling of massive amounts of data in heterogeneous scenarios. First, Platform A collects multi-source big data from three channels: 1) equipment operation data collected by sensors from 200 devices distributed across industrial production scenarios; 2) business transaction data generated by Platform A's own business systems; and 3) user interaction data generated by users through the Web and APP terminals. This multi-source big data is transmitted to Platform A's data receiving module via four independent transmission links, each corresponding to a unique link number.

[0063] Platform A obtains transmission parameters for each transmission link through a link monitoring module, specifically including transmission rate, transmission delay, and signal attenuation. Subsequently, Platform A employs data feature correlation analysis to extract correlation information from multi-source big data, specifically including correlations between device operation data and business transaction frequency, correlations between user click behavior and business transaction types, and correlations between device energy consumption data and user feedback content.

[0064] Step 102: Based on the transmission parameters, the transmission link is calibrated using edge computing technology to obtain the target transmission link.

[0065] In this step, edge computing technology refers to the technical approach of performing data processing, analysis, and computation at edge nodes close to the data source or data consumer.

[0066] Step 103: Based on the association information, the multi-source big data is divided into multiple processing sub-items, and the identification information of the corresponding target transmission link is assigned to each processing sub-item to obtain multiple identified processing sub-items and their corresponding execution order.

[0067] In this step, the identification information refers to the unique identification data used to uniquely distinguish different processing sub-items.

[0068] Step 104: Based on the association information, match the computational requirement parameters of each identified processing sub-item with the configurable parameters of heterogeneous computing resources to obtain an initial allocation scheme.

[0069] In this step, computational requirement parameters refer to the various performance requirements needed for the processing sub-item to complete its computational task. Heterogeneous computing resources refer to a collection of computing resources composed of various computing units with different architectures or types. Configurable parameters refer to the adjustable performance-related parameters of each computing unit in the heterogeneous computing resources.

[0070] Step 105: Obtain the running status information of heterogeneous computing resources, use blockchain verification technology to verify the credibility of the running status information and the initial allocation scheme, obtain the verification results, and adjust the initial allocation scheme in combination with the execution order to obtain the final allocation scheme.

[0071] In this step, the operational status information refers to the current working status data of each computing unit in the heterogeneous computing resources. Blockchain verification technology refers to the technical method of verifying the authenticity and rationality of data or solutions using the distributed node consensus mechanism of the blockchain network.

[0072] Step 106: Based on the target transmission link and the final allocation scheme, process the identified processing sub-items through heterogeneous computing resources to achieve scheduling optimization of multi-source big data.

[0073] This application's embodiments enhance the scheduling strategy's ability to collaboratively understand data content and transmission conditions by acquiring multi-source big data and corresponding transmission link parameters, and extracting related information from the multi-source big data; reduce transmission delays or data distortion caused by physical layer fluctuations, ensuring high-reliability delivery of task data during the scheduling process; achieve fine-grained task decomposition and link binding, improving the orderliness and traceability of the end-to-end processing flow; and ensure the authenticity of resource status data and the rationality of allocation schemes, avoiding scheduling deviations caused by false or outdated status information.

[0074] This application provides a specific embodiment. Step 102 involves calibrating the transmission link based on transmission parameters using edge computing technology to obtain the target transmission link. This specifically includes the following steps:

[0075] Step 201: Calculate the first deviation value between the transmission rate and the preset standard transmission rate, the second deviation value between the transmission delay and the preset standard transmission delay, and the third deviation value between the signal attenuation degree and the preset standard signal attenuation degree using edge computing technology.

[0076] In this embodiment, firstly, preset transmission rate reference values, transmission delay reference values, and signal attenuation reference values ​​are defined. Secondly, edge computing-related processing nodes are invoked to extract the actual transmission rate, transmission delay, and signal attenuation level from the transmission parameters. Finally, the deviations between the actual values ​​and the corresponding preset reference values ​​are calculated. Specifically, the first deviation value is equal to the absolute value of the difference between the transmission rate and the preset standard transmission rate; the second deviation value is equal to the absolute value of the difference between the transmission delay and the preset standard transmission delay; and the third deviation value is equal to the absolute value of the difference between the actual signal attenuation level and the preset standard signal attenuation level.

[0077] Step 202: Based on the first deviation value, the second deviation value, and the third deviation value, calibrate the transmission power, transmission frequency, and transmission protocol of the transmission link to obtain the target transmission link.

[0078] In this embodiment, the magnitudes of the first, second, and third deviation values ​​are first analyzed to identify the performance bottleneck of the transmission link corresponding to each deviation value. Secondly, for the transmission rate deviation corresponding to the first deviation value, the transmission power of the transmission link is adjusted to compensate for the insufficient rate or to reduce power consumption to avoid waste. For the transmission delay deviation corresponding to the second deviation value, the transmission frequency is adjusted to optimize signal transmission timeliness. For the signal attenuation deviation corresponding to the third deviation value, the transmission protocol is adjusted to reduce signal loss. Finally, through the coordinated adjustment of transmission power, transmission frequency, and transmission protocol, the actual transmission parameters of the transmission link are brought closer to the preset standard, resulting in a target transmission link with satisfactory performance.

[0079] The embodiments of this application ensure the stability and timeliness of data transmission, provide reliable link support for the transmission and processing of subsequent processing sub-items, and avoid data loss or processing delays caused by substandard transmission link performance.

[0080] For example, suppose platform A has four transmission links with transmission rates of 200Mbps, 300Mbps, 250Mbps, and 350Mbps, transmission delays of 40ms, 35ms, 45ms, and 30ms, and signal attenuation of 15%, 12%, 18%, and 10%, respectively. Simultaneously, a preset standard transmission rate of 300Mbps, a preset standard transmission delay of 30ms, and a preset standard signal attenuation of 10% are set. Based on this, edge computing technology is used to calculate the first, second, and third deviation values ​​for each of the four links. The calculations show that the first deviation value for the first link is 100Mbps, the second deviation value is 10ms, and the third deviation value is 5%. Subsequently, calibration operations are performed based on the three deviation values ​​calculated for each of the four links. For the first link, its transmission power is adjusted from 12dBm to 22dBm, its transmission frequency is adjusted from 2.4GHz to 5.8GHz, and its transmission protocol is changed from TCP to UDP. For the other links, the transmission power, transmission frequency, and transmission protocol are similarly adjusted according to their respective deviation values. After the above calibration, the actual transmission parameters of the four links are close to the preset standard, thus obtaining four target transmission links.

[0081] This application provides a specific embodiment. Step 103 involves dividing multi-source big data into multiple processing sub-items based on association information, and assigning corresponding target transmission link identification information to each processing sub-item to obtain multiple identified processing sub-items and their corresponding execution order. The specific steps include the following:

[0082] Step 301: Based on the correlation information, determine the feature dimensions of the multi-source big data, calculate the similarity, correlation and attribute overlap of each feature dimension, and divide the multi-source big data into multiple data combinations based on the similarity, correlation and attribute overlap of each feature dimension.

[0083] In this step, feature dimension refers to the specific feature categories possessed by each type of data in multi-source big data.

[0084] In this embodiment, firstly, based on the association information, the specific feature categories of different types of data in multi-source big data are sorted out. Secondly, by calculating the similarity, correlation and attribute overlap between each feature category, the degree of feature association between different data is clarified. Finally, according to the classification criteria of the degree of feature association, the multi-source big data with close association is grouped into one category to form multiple data combinations.

[0085] Step 302: Calculate the processing complexity of each data combination based on its data size and dependencies.

[0086] In this step, data size refers to the total number of bytes of data included in each data combination. Dependency refers to the tightness of the logical relationship between the data items within a data combination.

[0087] In this embodiment, the total number of data bytes for each data combination is first obtained, and the quantitative value of the logical correlation between the data within each data combination is determined. Secondly, corresponding weight values ​​are assigned to the data size and the degree of correlation. The processing complexity is equal to the sum of the data size multiplied by the data size weight and the correlation dependence multiplied by the correlation dependence weight.

[0088] Step 303: Based on the processing complexity of each data combination, split each data combination into multiple processing sub-items, and assign the corresponding target transmission link identification information to each processing sub-item to obtain multiple identifiable processing sub-items.

[0089] In this embodiment, a splitting threshold is first set based on the processing complexity of each data combination to determine the maximum processing complexity that each processing sub-item can handle. Then, each data combination is split into multiple processing sub-items according to the splitting threshold, ensuring that the processing complexity of each sub-item does not exceed the set threshold. Finally, the unique identification information of each link is obtained from the target transmission link, and corresponding identification information is assigned to each split processing sub-item, establishing a unique correspondence between each processing sub-item and the target transmission link, resulting in multiple identified processing sub-items.

[0090] Step 304: Perform hierarchical resolution and topological sorting of the dependencies between each identified processing sub-item to determine the execution order of each identified processing sub-item.

[0091] In this step, dependency refers to the association between each identified processing sub-item, where the output data of the preceding processing sub-item serves as the input data of the subsequent processing sub-item.

[0092] In this embodiment, the relationships between each identified processing sub-item are first analyzed to identify which processing sub-items depend on the output data of other processing sub-items to be executed. Then, these relationships are hierarchically analyzed to divide them into different execution levels. Finally, the parallel execution order of processing sub-items at the same level and the sequential execution order of processing sub-items at different levels are determined by topological sorting, thus forming the complete execution order of each identified processing sub-item.

[0093] The embodiments of this application realize the orderly organization and fine-grained division of multi-source big data, so that the processing sub-items are precisely bound to the target transmission link, providing a clear execution basis for subsequent computing resource matching and parallel processing, and avoiding low processing efficiency caused by data disorder and chaotic execution order.

[0094] This application provides a specific embodiment. Step 104 involves matching the computational requirement parameters of each identified processing sub-item with the configurable parameters of heterogeneous computing resources based on the association information to obtain an initial allocation scheme. This specifically includes the following steps:

[0095] Step 401: Based on the computational requirement parameters of each identified processing sub-item and the configurable parameters of heterogeneous computing resources, establish initial matching rules between computational requirement parameters and configurable parameters. The computational requirement parameters include computation type, computational quantity, and computational response requirements, while the configurable parameters include computational capacity, computational load, and adaptation type.

[0096] In this step, computation type refers to the specific category of computational operations required to process the sub-item. Computational load refers to the amount of data that the sub-item needs to process to complete its computational task. Computational response requirement refers to the time limit required for the sub-item to complete its computational task. Computational capacity refers to the processing power of each computing unit in the heterogeneous computing resources. Computational load refers to the amount of computational tasks currently being carried by each computing unit in the heterogeneous computing resources. Adaptation type refers to the types of computational tasks that each computing unit in the heterogeneous computing resources is adapted to.

[0097] In this embodiment, the specific computational operation category, the amount of data to be processed, and the time limit for each identified processing sub-item are first clarified. Then, the computational processing capabilities of each computing unit in the heterogeneous computing resources, the amount of computational tasks currently being carried, and the types of computational tasks they are adapted to are analyzed. Finally, the correspondence between computational requirement parameters and configurable parameters is established, clarifying the preliminary correspondence logic between computation type and adaptation type, computational quantity and computational capability, and computational response requirements and computational load, thus forming initial matching rules.

[0098] Step 402: Based on the data association attributes in the association information, adjust the parameter weights of each parameter in the initial matching rule to obtain the adjusted matching rule.

[0099] In this step, data association attributes refer to the association characteristics and processing priority requirements between multi-source big data reflected in the association information.

[0100] In this embodiment of the application, the big data association characteristics and processing priority requirements reflected in the associated information are first analyzed in depth to clarify the importance of different computing requirement parameters in the overall task processing. Then, for each corresponding parameter in the initial matching rule, different importance weights are assigned according to the data association attributes. Finally, the weights of each parameter are adjusted to obtain the adjusted matching rule.

[0101] Step 403: Based on the adjusted matching rules, determine the allocation result of each labeled processing sub-item, and integrate the allocation results of all labeled processing sub-items to obtain the initial allocation scheme.

[0102] In this embodiment, the computational requirements of each identifiable processing sub-item are first compared with the configurable parameters of heterogeneous computing resources according to the adjusted matching rules. Then, based on the comparison results, the specific computing unit within the heterogeneous computing resources corresponding to each identifiable processing sub-item is determined, forming the allocation result for a single processing sub-item. Finally, the allocation results of all identifiable processing sub-items are collected, categorized, organized, and integrated to form an initial allocation scheme covering all processing sub-items.

[0103] The embodiments of this application achieve precise matching between computing needs and heterogeneous computing resources, avoiding resource waste or substandard task processing caused by rigid matching rules.

[0104] This application provides a specific embodiment, such as Figure 2 As shown, step 105 involves obtaining the running status information of heterogeneous computing resources, using blockchain verification technology to verify the credibility of the running status information and the initial allocation scheme, obtaining the verification result, and adjusting the initial allocation scheme based on the execution order to obtain the final allocation scheme. Specifically, this includes the following steps:

[0105] Step 501: Obtain the running status information of heterogeneous computing resources, and based on the preset verification rules, verify the credibility of the running status information and the initial allocation scheme through the blockchain verification technology pre-configured by each verification node in the blockchain network, and obtain the verification result of each verification node.

[0106] In this step, the preset verification rules refer to the judgment criteria pre-set for verifying the credibility of the running status information and the initial allocation scheme. A blockchain network is a network system composed of multiple distributed nodes used for data verification and notarization. A verification node is an independent node in the blockchain network that participates in credibility verification.

[0107] In this embodiment, the current working status data of each computing unit in the heterogeneous computing resources is first collected in real time, and a pre-defined verification standard for determining credibility is defined. Then, the running status information and the initial allocation scheme are synchronized to a blockchain network composed of multiple distributed nodes. Finally, the authenticity of the running status information and the rationality of the initial allocation scheme are verified by the blockchain verification method pre-configured by each independent node participating in the verification in the network, and the judgment result of each verification node is recorded.

[0108] Step 502: If the number of verification nodes that pass the verification is greater than the preset number, then the final verification result is determined to be that the verification is passed, and the resource allocation priority of each marked processing sub-item is determined according to the execution order.

[0109] In this step, the preset quantity refers to the pre-set threshold number of verification nodes used to determine the final verification result.

[0110] In this embodiment, the verification results of all verification nodes are first counted, and the number of verification nodes that pass the verification is compared with a pre-set threshold. If the number of nodes that pass the verification exceeds the threshold, the final judgment result is that the verification has passed, based on all the results. Finally, the priority level of each identified processing sub-item in resource allocation is determined by referring to the execution order of each identified processing sub-item and combining the task execution logic.

[0111] Step 503: If the verification result shows that the number of verification nodes that fail the verification is greater than the preset number, then the unreasonable resource allocation items in the initial allocation scheme are corrected to obtain an intermediate allocation scheme, and the resource allocation priority of each marked processing sub-item is determined in combination with the execution order.

[0112] In this step, unreasonable resource allocation items refer to resource allocation content in the initial allocation scheme that does not match the heterogeneous computing resource operation status information or preset verification rules.

[0113] Step 504: Adjust the resource allocation ratios corresponding to each marked processing sub-item in the initial or intermediate allocation scheme according to the resource allocation priority to obtain the final allocation scheme.

[0114] In this embodiment, the resource allocation priority of each marked processing sub-item is first determined. Then, based on the priority, if the final verification result is a pass, the resource allocation ratio corresponding to each processing sub-item in the initial allocation scheme is adjusted. If the final verification result is a fail, the resource allocation ratio corresponding to each processing sub-item in the intermediate allocation scheme is adjusted. The processing sub-item with higher priority is allocated a higher proportion of resources. Finally, all the adjusted resource allocation content is integrated to obtain the final allocation scheme.

[0115] The embodiments of this application implement reliable verification and dynamic optimization of resource allocation schemes, avoiding resource waste or task execution failure caused by mismatch between resource allocation and actual operating status, and ensuring the rationality and reliability of resource allocation.

[0116] For example, suppose the obtained runtime information of heterogeneous computing resources shows that: the current computing load of the CPU is 35%, and it is running smoothly without change; the current computing load of the GPU is 85%, which is a significant increase compared to the initial statistics and has far exceeded the preset 80% load threshold; the current computing load of the FPGA is 25%, with no significant change. Meanwhile, suppose there are 10 verification nodes configured in the blockchain network, and the preset verification rule is set to consider a verification passed if the computing load is less than or equal to 80% and the adaptation type matches, with a preset threshold of 6 successful verifications.

[0117] In this scenario, the aforementioned operational status information and initial allocation scheme are transmitted to the blockchain network, and trustworthiness is verified using the pre-configured blockchain verification technology of each verification node. Assuming that during the verification process, 7 verification nodes report GPU computing load exceeding limits (i.e., although the matching type is correct, available resources are insufficient), and the verification fails; while 3 verification nodes report that the verification passed. Statistical analysis shows that the number of verification nodes that passed is 3, which does not exceed the preset threshold of 6. Therefore, the final verification result is determined to be verification failure.

[0118] Subsequently, the failure reason of GPU overload associated with the 7 verification nodes was compared with the resource allocation items in the initial allocation scheme. It was determined that the decision to allocate 3 parallel computing sub-items to the GPU in the initial allocation scheme was an unreasonable resource allocation item. Next, based on the characteristics of this unreasonable item, the configurable parameters of heterogeneous computing resources were screened. It was found that the FPGA's computing load was only 25% and its adaptation type supported parallel computing, making it an idle resource. Therefore, the GPU in the original scheme was replaced by the FPGA, and the above 3 parallel computing sub-items were bound to the FPGA, thus generating an intermediate allocation scheme.

[0119] Furthermore, based on the execution order of each identifiable processing sub-item, the resource allocation priority of each sub-item is determined as follows: device data processing sub-item has the highest priority, followed by business data processing sub-item, then user data processing sub-item, and the comprehensive analysis processing sub-item has the lowest priority. According to this priority sequence, the resource allocation ratio corresponding to each identifiable processing sub-item in the intermediate allocation scheme is dynamically adjusted. Specifically, the resource allocation ratio of the device data processing sub-item is increased to 40%, the business data processing sub-item is adjusted to 30%, the user data processing sub-item is adjusted to 20%, and the comprehensive analysis processing sub-item is adjusted to 10%. After the above corrections and adjustments, a final allocation scheme that conforms to the current heterogeneous resource status and has undergone reliable verification is finally obtained.

[0120] This application provides a specific embodiment. In step 503, if the number of verification nodes that failed the verification is greater than the preset number, the unreasonable resource allocation items in the initial allocation scheme are corrected to obtain an intermediate allocation scheme. The resource allocation priority of each identifiable processing sub-item is then determined based on the execution order. Specifically, this includes the following steps:

[0121] Step 511: If the number of verification nodes that failed the verification is greater than the preset number, then the reasons for failure associated with the verification nodes that failed the verification are compared with the resource allocation items in the initial allocation scheme to determine the unreasonable resource allocation items.

[0122] In this embodiment of the application, firstly, it is confirmed that more than a preset number of verification nodes have reported that the verification has failed. Secondly, the specific reasons for failure associated with all verification nodes that failed the verification are collected. Then, these reasons for failure are compared and checked one by one with each resource allocation item in the initial allocation scheme to determine whether each resource allocation item conflicts with the reasons for failure. Finally, the resource allocation items with conflicts are determined to be unreasonable resource allocation items, thus locking in the target for subsequent correction work.

[0123] Step 512: Based on each unreasonable resource allocation item, filter out the resource information that matches the reason for failure from the configurable parameters, and bind the resource information with the corresponding identified processing sub-item in the unreasonable resource allocation item in the initial allocation scheme to obtain the intermediate allocation scheme.

[0124] In this step, resource information refers to specific resource data among the configurable parameters of heterogeneous computing resources that meet the requirements for failure reasons.

[0125] In this embodiment of the application, for each identified unreasonable resource allocation item, the resource conditions required by the corresponding failure reason are clarified. Then, specific resource data that meets the resource conditions are selected from the configurable parameters of heterogeneous computing resources. The selected resource information is then uniquely associated with the identified processing sub-items corresponding to the unreasonable resource allocation items in the initial allocation scheme. Finally, all unreasonable resource allocation items are replaced to form an intermediate allocation scheme.

[0126] Step 513: Determine the resource allocation priority of each identifiable processing sub-item based on the data association attributes of the intermediate allocation scheme and the calculation requirement parameters.

[0127] In this embodiment, the resource adaptation of each identifiable processing sub-item in the intermediate allocation scheme is first analyzed. Then, the data association attributes in the computational requirement parameters are extracted to clarify the computational importance and association characteristics of different processing sub-items. Next, combining the resource configuration logic of the intermediate allocation scheme and the priority of the data association attributes, the priority level of each identifiable processing sub-item in resource allocation is comprehensively determined, ultimately determining the resource allocation priority of each identifiable processing sub-item.

[0128] The embodiments of this application realize the accurate correction of unreasonable resource allocation items and the scientific determination of resource allocation priority, providing a reasonable basis for subsequent adjustment of resource allocation ratio, avoiding resource mismatch caused by blind correction, and ensuring the adaptability of resource allocation to task requirements.

[0129] This application provides a specific embodiment. Step 106 involves processing the identified processing sub-items using heterogeneous computing resources based on the target transmission link and the final allocation scheme to achieve scheduling optimization of multi-source big data. Specifically, this includes the following steps:

[0130] Step 601: According to the final allocation scheme, the identified processing sub-items are transmitted to the computing units corresponding to the heterogeneous computing resources through the target transmission link. Each computing unit performs parallel processing on the identified processing sub-items that have no dependency relationship, and monitors the processing progress of each computing unit.

[0131] In this embodiment, firstly, the heterogeneous computing resources corresponding to each identifiable processing sub-item are determined according to the final allocation scheme. Then, these identifiable processing sub-items are accurately transmitted to the corresponding computing units through the target transmission link. Next, identifiable processing sub-items that do not have dependencies in each computing unit are selected, and each computing unit processes these sub-items simultaneously. Finally, the task completion status of each computing unit is tracked and recorded in real time, and the processing progress is continuously monitored.

[0132] Step 602: If the processing progress of a computing unit lags behind the preset progress, then based on the final allocation scheme, idle resources in heterogeneous computing resources are allocated to computing units whose processing progress lags behind the preset progress, and the marked processing sub-items are processed in parallel. If the processing progress of a computing unit exceeds the preset progress, then according to the final allocation scheme, the marked processing sub-items are processed in parallel to achieve scheduling optimization of multi-source big data.

[0133] In this step, idle resources refer to heterogeneous computing resources that have not been assigned processing tasks or whose utilization is low and which are available for reallocation.

[0134] In this embodiment, firstly, the actual processing progress of each computing unit is compared with the preset progress. If the actual processing progress lags behind the preset progress, then based on the final allocation scheme, unallocated or underutilized reallocable resources are selected from the heterogeneous computing resources. These idle resources are then allocated to the computing unit with the lagging progress and work with that computing unit to process the marked processing sub-items in parallel. If the actual processing progress exceeds the preset progress, then the final allocation scheme remains unchanged, and the corresponding computing unit continues to process the marked processing sub-items in parallel. Through the differentiated processing of the two cases, the scheduling optimization of multi-source big data is finally achieved.

[0135] The embodiments of this application realize dynamic adaptation and efficient resource utilization for multi-source big data processing, avoid schedule imbalance and resource idleness, and ensure the stability and optimization effect of the scheduling process.

[0136] For example, suppose that after dividing multi-source big data based on correlation information, a total of 12 labeled processing sub-items are obtained. Among them, the computational requirements of processing sub-items 1 to 8 are general-purpose computing types with high computational response requirements; processing sub-items 9 to 11 are parallel computing types with large computational load and high data correlation attributes; processing sub-item 12 is a parallel computing type, but with relatively small computational load. At the same time, the configurable parameters of the obtained heterogeneous computing resources show that: CPUs have efficient logic control capabilities, GPUs excel in floating-point operations and high-concurrency processing, and FPGAs have advantages in specific pipeline parallel processing.

[0137] Based on this, initial matching rules for computational requirement parameters and configurable parameters are first established. General-purpose computational sub-items are initially matched to CPUs, and parallel computational sub-items are initially matched to GPUs or FPGAs. Subsequently, the parameter weights in the matching rules are adjusted according to the data association attributes in the association information. Since sub-items 9 to 11 have strong data association attributes, the weight of the computational capability parameter is increased, prioritizing their matching to GPUs with the highest theoretical computational capability. Sub-items 1 to 8, due to their high response requirements, continue to be matched to CPUs; and sub-item 12 is matched to FPGAs. Based on these adjusted matching rules, the allocation results for each identifiable sub-item are determined and integrated, ultimately resulting in the initial allocation scheme: sub-items 1 to 8 are allocated to CPUs, sub-items 9 to 11 to GPUs, and sub-item 12 to FPGAs.

[0138] Figure 3 This is a schematic diagram illustrating a specific implementation of a big data task scheduling and optimization system based on heterogeneous computing, as provided in this application. (Refer to...) Figure 3 The system may include:

[0139] The acquisition module 21 is used to acquire multi-source big data and the transmission parameters of the corresponding transmission links, and extract related information from the multi-source big data;

[0140] The calibration module 22 is used to calibrate the transmission link based on transmission parameters and using edge computing technology to obtain the target transmission link;

[0141] The allocation module 23 is used to divide multi-source big data into multiple processing sub-items based on the association information, and to assign the identification information of the corresponding target transmission link to each processing sub-item, so as to obtain multiple identifiable processing sub-items and their corresponding execution order.

[0142] The allocation module 23 is also used to match the computational requirement parameters of each identified processing sub-item with the configurable parameters of heterogeneous computing resources based on the association information to obtain an initial allocation scheme.

[0143] The calibration module 22 is also used to obtain the running status information of heterogeneous computing resources, use blockchain verification technology to verify the credibility of the running status information and the initial allocation scheme, obtain the verification result, and adjust the initial allocation scheme in combination with the execution order to obtain the final allocation scheme.

[0144] Processing module 24 is used to process the identified processing sub-items based on the target transmission link and the final allocation scheme through heterogeneous computing resources, so as to achieve scheduling optimization of multi-source big data.

[0145] This application provides an embodiment of a big data task scheduling optimization system based on heterogeneous computing to implement the aforementioned big data task scheduling optimization method based on heterogeneous computing. Therefore, the specific implementation of the big data task scheduling optimization system based on heterogeneous computing can be found in the embodiment section of the big data task scheduling optimization method based on heterogeneous computing described above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0146] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of a big data task scheduling optimization method based on heterogeneous computing as described above.

[0147] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for optimizing big data task scheduling based on heterogeneous computing.

[0148] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0149] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the embodiments of the big data task scheduling optimization method based on heterogeneous computing.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0151] The foregoing has provided a detailed description of a big data task scheduling optimization method and system based on heterogeneous computing provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A heterogeneous computing-based big data task scheduling optimization method, characterized in that, The method comprises the following steps: Obtain multi-source big data and transmission parameters of corresponding transmission links, and extract associated information from the multi-source big data; Based on the transmission parameters, calibrate the transmission links using edge computing technology to obtain target transmission links; Based on the associated information, divide the multi-source big data into multiple processing sub-items, and assign each processing sub-item with identification information of the corresponding target transmission link to obtain multiple labeled processing sub-items and corresponding execution sequences; Based on the associated information, match the computing requirement parameters of each labeled processing sub-item with the configurable parameters of the heterogeneous computing resources to obtain an initial allocation scheme; Obtain the running state information of the heterogeneous computing resources, use blockchain verification technology to verify the credibility of the running state information and the initial allocation scheme, obtain a verification result, and adjust the initial allocation scheme in combination with the execution sequence to obtain a final allocation scheme; Based on the target transmission link and the final allocation scheme, process the labeled processing sub-items through the heterogeneous computing resources to realize the scheduling optimization of the multi-source big data.

2. The method of claim 1, wherein, Obtain the running state information of the heterogeneous computing resources, use blockchain verification technology to verify the credibility of the running state information and the initial allocation scheme, obtain a verification result, and adjust the initial allocation scheme in combination with the execution sequence to obtain a final allocation scheme, comprising: Obtain the running state information of the heterogeneous computing resources, and based on a preset verification rule, use the blockchain verification technology pre-configured by each verification node in the blockchain network to verify the credibility of the running state information and the initial allocation scheme respectively to obtain a verification result of each verification node; If the number of verification nodes with a passed verification result is greater than a preset number, it is determined that the final verification result is passed, and the resource allocation priority of each labeled processing sub-item is determined according to the execution sequence; If the number of verification nodes with a failed verification result is greater than the preset number, unreasonable resource allocation items in the initial allocation scheme are corrected to obtain an intermediate allocation scheme, and the resource allocation priority of each labeled processing sub-item is determined in combination with the execution sequence; According to the resource allocation priority, the resource allocation proportion of each labeled processing sub-item in the initial allocation scheme or the intermediate allocation scheme is adjusted to obtain a final allocation scheme.

3. The method of claim 2, wherein, If the number of verification nodes with a failed verification result is greater than the preset number, unreasonable resource allocation items in the initial allocation scheme are corrected to obtain an intermediate allocation scheme, and the resource allocation priority of each labeled processing sub-item is determined in combination with the execution sequence, comprising: If the number of verification nodes with a failed verification result is greater than the preset number, compare the failed reasons associated with the verification nodes with the resource allocation items in the initial allocation scheme to determine the unreasonable resource allocation items; According to each unreasonable resource allocation item, filter out resource information matching the unqualified reason from the configurable parameters, and bind the resource information with the corresponding identified processing sub-item in the unreasonable resource allocation item in the initial allocation scheme to obtain an intermediate allocation scheme; According to the intermediate allocation scheme and the data correlation attribute of the calculation requirement parameter, determine the resource allocation priority of each identified processing sub-item.

4. The method of claim 1, wherein, Based on the transmission parameter, the transmission link is calibrated by using edge computing technology to obtain a target transmission link, including: Using edge computing technology to calculate the first deviation value of the transmission rate and the preset standard transmission rate, the second deviation value of the transmission delay and the preset standard transmission rate, and the third deviation value of the signal attenuation degree and the preset standard signal attenuation degree in the transmission parameter; According to the first deviation value, the second deviation value and the third deviation value, the transmission power, the transmission frequency and the transmission protocol of the transmission link are calibrated respectively to obtain the target transmission link.

5. The method of claim 1, wherein, Based on the correlation information, the multi-source big data is divided into multiple processing sub-items, and the identification information of the corresponding target transmission link is allocated to each processing sub-item to obtain multiple identified processing sub-items and the corresponding execution order, including: According to the correlation information, the feature dimension of the multi-source big data is determined, and the similarity, correlation and attribute overlap of each feature dimension are calculated, and according to the similarity, correlation and attribute overlap of each feature dimension, the multi-source big data is divided into multiple data combinations; According to the data size and correlation dependence of each data combination, the processing complexity of each data combination is calculated; According to the processing complexity of each data combination, each data combination is split into multiple processing sub-items, and the identification information of the corresponding target transmission link is allocated to each processing sub-item to obtain multiple identified processing sub-items; The dependency relationship between each identified processing sub-item is hierarchically analyzed and topologically sorted to determine the execution order of each identified processing sub-item.

6. The method of claim 1, wherein, Based on the correlation information, the calculation requirement parameter of each identified processing sub-item and the configurable parameter of the heterogeneous computing resource are matched to obtain an initial allocation scheme, including: According to the calculation requirement parameter of each identified processing sub-item and the configurable parameter of the heterogeneous computing resource, an initial matching rule of the calculation requirement parameter and the configurable parameter is established, the calculation requirement parameter includes calculation type, calculation amount and calculation response requirement, and the configurable parameter includes calculation ability, calculation load and adaptation type; According to the data correlation attribute in the correlation information, the parameter weight of each parameter in the initial matching rule is adjusted to obtain an adjusted matching rule; According to the adjusted matching rule, the allocation result of each identified processing sub-item is determined, and the allocation results of all identified processing sub-items are integrated to obtain an initial allocation scheme.

7. The method of claim 1, wherein, Based on the target transmission link and the final allocation scheme, the identified processing sub-item is processed by the heterogeneous computing resource to realize the scheduling optimization of the multi-source big data, including: According to the final allocation scheme, the identified processing sub-items are transmitted to the corresponding computing units of the heterogeneous computing resources through the target transmission link, the identified processing sub-items without dependency are processed in parallel by each computing unit, and the processing progress of each computing unit is monitored; If the processing progress of a computing unit lags behind the preset progress, the idle resources in the heterogeneous computing resources are allocated to the computing unit whose processing progress lags behind the preset progress based on the final allocation scheme, and the identified processing sub-items are processed in parallel, and if the processing progress of a computing unit exceeds the preset progress, the identified processing sub-items are processed in parallel according to the final allocation scheme, so as to realize the scheduling optimization of the multi-source big data.

8. A heterogeneous computing-based big data task scheduling optimization system, characterized in that, Comprise: An acquisition module is configured to acquire multi-source big data and transmission parameters of corresponding transmission links, and extract association information from the multi-source big data; A calibration module is configured to calibrate the transmission links based on the transmission parameters by using edge computing technology to obtain target transmission links; An allocation module is configured to divide the multi-source big data into a plurality of processing sub-items based on the association information, and assign identification information of the corresponding target transmission link to each processing sub-item to obtain a plurality of identified processing sub-items and a corresponding execution order; The allocation module is further configured to match the computing requirement parameters of each identified processing sub-item and the configurable parameters of the heterogeneous computing resources based on the association information to obtain an initial allocation scheme; The calibration module is further configured to acquire running state information of the heterogeneous computing resources, verify the running state information and the initial allocation scheme by using a blockchain verification technology to obtain a verification result, and adjust the initial allocation scheme in combination with the execution order to obtain a final allocation scheme; A processing module is configured to process the identified processing sub-items by using the heterogeneous computing resources based on the target transmission link and the final allocation scheme, so as to realize the scheduling optimization of the multi-source big data.

9. An electronic device, comprising: Comprise: A memory is configured to store a computer program; A processor is configured to execute the computer program to implement the steps of the big data task scheduling optimization method based on heterogeneous computing according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the big data task scheduling optimization method based on heterogeneous computing according to any one of claims 1 to 7.

Images