Data flow optimization method and system based on artificial intelligence

Abstract

The invention discloses a data flow optimization method and system based on artificial intelligence. The method comprises the steps of multi-modal data flow collection, digital twin basic modeling, decision enhancement, dynamic topology optimization and resource arrangement optimization. The invention relates to the technical field of network data flow management, adopts a data flow integration optimization scheme combining digital twin basic modeling, dynamic topological optimization and decision comprehensive optimization, not only improves the flexibility and real-time performance of a system, but also optimizes the efficiency and intelligent level of resource allocation; the quantum mapping and quantum optimization technology is adopted, adaptive noise and disturbance simulation are combined, the mapping precision and the dynamic adaptive capacity between a virtual layer and a physical layer are improved, and the state error between a virtual node and a physical node is optimized through quantum mapping parallel processing; hierarchical federated learning is combined with an adaptive learning strategy to optimize a data flow decision, and a joint optimization method of a quantum heuristic algorithm and a graph neural network is introduced into dynamic topological optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of network data stream management, and specifically refers to a data stream optimization method and system based on artificial intelligence. Background Art

[0002] The data stream optimization method based on artificial intelligence utilizes machine learning algorithms and intelligent analysis techniques to automatically identify and optimize bottlenecks, redundancies, and inefficiencies in the data stream. By monitoring and predicting data flow patterns in real time, this method can dynamically adjust data processing paths, resource allocation, and task scheduling, thereby improving data processing efficiency, reducing latency, and reducing resource consumption. Its role is to enhance the overall performance of the system, ensure the stability and efficiency of the data stream in complex environments, and is applicable to scenarios such as big data processing, the Internet of Things, and real-time analysis.

[0003] However, in the existing data stream optimization methods, there are technical problems that the existing network data streams have different data formats and data types, resulting in the digital twin modeling of the data stream being unable to accurately reflect the dynamic relationship between the physical system and the virtual model, nor can it implement the dynamic changes of the corresponding system. This further leads to deficiencies in the security, accuracy, and real-time performance of the existing data stream decisions; in the existing digital twin basic modeling process, there are problems that the modeling accuracy of the virtual topology mapping layer often lacks during the modeling processes of the physical layer and the virtual layer. At the same time, this also leads to poor dynamic adaptability of the virtual model and insufficient noise simulation; in the existing data stream decision optimization and dynamic topology optimization processes, there are problems that the computational complexity of the topology optimization of the decision model is high, and at the same time, the difficulty of real-time adjustment is also relatively high. In the dynamic topology optimization process, traditional methods often face technical problems of tight computing resources and great difficulty in real-time adjustment. Summary of the Invention

[0004] In view of the above situation, to overcome the defects of the existing technology, the present invention provides an artificial intelligence-based data stream optimization method and system. In the existing data stream optimization methods, due to the existence of different data formats and data types in the existing network data stream, the digital twin modeling for the data stream not only fails to accurately reflect the dynamic relationship between the physical system and the virtual model, but also cannot implement the dynamic changes of the corresponding system. This further leads to the technical problems that the security, accuracy, and real-time performance of the existing data stream decision-making are all insufficient. This solution creatively adopts a data stream integration and optimization solution that combines digital twin basic modeling, dynamic topology optimization, and decision comprehensive optimization, which not only improves the flexibility and real-time performance of the system, but also optimizes the efficiency and intelligent level of resource allocation. In the existing digital twin basic modeling process, there are problems such as insufficient modeling accuracy in the virtual topology mapping layer during the modeling process of the physical layer and the virtual layer. At the same time, this also leads to poor dynamic adaptability of the virtual model and insufficient noise simulation. This solution creatively adopts quantum mapping and quantum optimization technologies, combined with adaptive noise and perturbation simulation, to improve the mapping accuracy and dynamic adaptation ability between the virtual layer and the physical layer. Through parallel processing of quantum mapping, the state error between virtual nodes and physical nodes is optimized, making the virtual topology mapping more accurate. In the existing data stream decision optimization and dynamic topology optimization processes, there are problems such as high computational complexity in the topology optimization of the decision model and high difficulty in real-time adjustment. In the dynamic topology optimization process, traditional methods often face problems such as tight computational resources and high difficulty in real-time adjustment. This solution creatively adopts hierarchical federated learning combined with an adaptive learning strategy to optimize data stream decisions, and introduces a joint optimization method of quantum heuristic algorithm and graph neural network in dynamic topology optimization. By innovatively designing a hybrid improved reward function, considering various factors such as throughput, latency, security, and energy efficiency, to adapt to different business requirements and environmental conditions, the data stream decision-making process is optimized. At the same time, by combining the quantum heuristic algorithm and the graph neural network, an optimization method of introducing quantum superposition correction parameters is innovatively introduced, reducing the computational complexity of topology structure optimization, and improving the efficiency and accuracy of topology structure adjustment through the quantum heuristic optimization algorithm.

[0005] The technical solution adopted by the present invention is as follows: The artificial intelligence-based data stream optimization method provided by the present invention includes the following steps:

[0006] Step S1: Multimodal data stream acquisition;

[0007] Step S2: Digital twin basic modeling;

[0008] Step S3: Decision reinforcement;

[0009] Step S4: Dynamic topology optimization;

[0010] Step S5: Resource orchestration optimization.

[0011] Further, in step S1, the multi-modal data stream collection is used to collect multi-dimensional data and perform optimization and encoding. Specifically, through multi-source data collection, the original data of the data stream is obtained, and feature extraction and compression processing are performed on the original data of the data stream to obtain a multi-modal feature data set of the data stream.

[0012] The multi-modal feature data set of the data stream includes network performance features, computing resource features, storage device features, security level features, abnormal fault features, compression features, and time series features.

[0013] Further, in step S2, the digital twin basic modeling is used to construct a basic digital twin model required for data stream optimization. Specifically, based on the multi-modal feature data set of the data stream, through constructing a virtual topology mapping layer, digital twin basic modeling is performed to obtain a data stream digital twin model, including the following steps:

[0014] Step S21: Construct a virtual mapping. Specifically, by defining the virtualization representation of the data stream, a virtual topology mapping layer is constructed, including virtual node construction, virtual connection construction, and virtual policy construction. And during the construction of the virtual node, virtual connection, and virtual policy, quantum mapping and quantum optimization are introduced to improve the virtual mapping, and the physical layer of the data stream is topologically mapped to the virtual layer to obtain a virtual topology mapping layer.

[0015] The quantum mapping performs parallel mapping processing through quantum gate operations; the quantum optimization performs error optimization by minimizing the state error between virtual nodes and physical nodes.

[0016] The calculation formula of the quantum mapping is:

[0017] ;

[0018] In the formula, is the high-dimensional quantum state feature, which is used to represent the data stream feature of the virtual topology mapping layer after parallel mapping processing. N is the total number of features, which is used to represent the total number of original features in the multi-modal feature data set of the data stream. n is the feature index. is the quantum state phase factor. is the original feature data, which is used to represent the original features in the multi-modal feature data set of the data stream.

[0019] Step S22: Virtual model noise simulation. Specifically, by simulating events such as morning and perturbations, network instability events and abnormal events are simulated, and through adaptive noise and perturbation addition, virtual model noise simulation is performed to obtain random load virtual noise data.

[0020] Step S23: Virtual layer state prediction, specifically, constructing a standard time series prediction model to predict the virtual layer state, and combining the random load virtual noise data to perform simulation prediction on the changes of network load and traffic dynamic parameters, so as to obtain load fluctuation prediction data;

[0021] Step S24: Generation of dynamic transition matrix, specifically, generating a state transition probability matrix in a dynamic environment based on the load fluctuation prediction data and the virtual topology mapping layer, which is used to represent the transition probability between system states, and through the state transition probability matrix, perform the conversion representation of the data flow system state;

[0022] Step S25: Basic model calibration, specifically, by calculating the calibration parameter error, comparing the state differences between the virtual layer and the physical layer, and dynamically adjusting the parameters of the virtual model to obtain virtual node error reference data, and based on the virtual node error reference data, iteratively optimize and adjust virtual nodes, virtual connections and virtual policies to optimize the virtual topology mapping layer and obtain a data flow digital twin model.

[0023] Further, in step S3, the decision reinforcement is used to optimize the data flow decision through hierarchical federated learning. Specifically, based on the data flow multi-modal feature data set and the data flow digital twin model, a hierarchical federated learning method for hybrid decision optimization is adopted to perform decision reinforcement to obtain data flow decision reference data, including the following steps:

[0024] Step S31: Constructing a hierarchical federated learning architecture, specifically, constructing an edge layer, a fog computing layer and a cloud coordination layer to construct a hierarchical federated learning network structure, and introducing an adaptive learning strategy to perform dynamic adjustment of federated learning to obtain a federated learning basic model;

[0025] Step S32: Global policy aggregation and optimization, specifically, performing gradient aggregation on the fog computing layer through the cloud coordination layer to generate a global optimization policy model, and based on the global optimization policy model, performing global policy optimization;

[0026] The calculation formula for the global policy aggregation and optimization is:

[0027] ;

[0028] In the formula, is the global optimization decision model output by the cloud coordination layer, t is the time index, G is the total number of fog computing nodes, g is the fog computing node index, is the dynamic adjustment coefficient, which is specifically calculated by weighting the delay and load parameters, w m is the weight of the edge node, is the local gradient of the model parameter group of the locally trained model, is the aggregated gradient information output by the fog computing layer;

[0029] Step S33: Construct a hybrid improved reward function, specifically by designing a hybrid improved reward function to optimize the data flow decision-making and obtain a hybrid improved global optimization strategy;

[0030] The hybrid improved reward function is specifically constructed by throughput, latency, security, and energy efficiency metrics, and the optimal hybrid improved reward function is obtained by adjusting the weights according to the environment and business requirements;

[0031] The calculation formula of the optimal hybrid improved reward function is:

[0032] ;

[0033] In the formula, R is the optimal hybrid improved reward function, a is the throughput weight, Throu is the throughput parameter, which is used to represent the amount of data successfully transmitted per unit time, b is the latency weight, Latency is the latency parameter, which is used to represent the latency time of data transmission, c is the security weight, Security is the security parameter, which is used to represent the security evaluation value during data transmission, d is the energy efficiency weight, Energy is the energy efficiency parameter, which is used to represent the energy consumption evaluation value of the data flow system. The hybrid improved reward function is specifically constructed by throughput, latency, security, and energy efficiency metrics, and the optimal hybrid improved reward function is obtained by adjusting the weights according to the environment and business requirements;

[0034] Step S34: Decision reinforcement, specifically by sending the global optimization policy model generated by the cloud to the edge nodes, and making local decisions based on the global optimization policy model to obtain local decision data. By adjusting the optimal routing path, scheduling task, and network parameters according to the local decision data, data flow decision reference data is obtained;

[0035] The data flow decision reference data specifically includes a global optimization data flow decision strategy and distributed execution encrypted data.

[0036] Further, in step S4, the dynamic topology optimization is used to adjust the data flow topology structure in real time. Specifically, based on the data flow digital twin model and the data flow decision reference data, a graph neural network combined with a quantum heuristic algorithm is used for dynamic topology optimization to obtain dynamic topology structure data, including the following steps:

[0037] Step S41: Topology analysis enhancement, specifically by using a standard graph neural network to identify the bottleneck nodes and paths of the network topology, and through node feature update and topology feature extraction, data flow network topology analysis feature data is obtained;

[0038] Step S42: Network topology dynamic reconstruction, specifically, based on the data flow network topology analysis characteristic data, by designing a topology reconstruction function, obtaining the optimal topology configuration scheme data, and using a quantum-inspired optimization algorithm to perform dynamic topology adjustment to obtain the dynamic optimal topology structure data;

[0039] The calculation formula of the topology reconstruction function is:

[0040] TopoE = t 1 ×Throughput + t 2 ×LateReduction - t 3 ×ReconfigCost;

[0041] In the formula, TopoE is the topology reconstruction function, t 1 is the weight of throughput increase and decrease, Throughput is the throughput increase and decrease strategy, t 2 is the weight of path optimization, LateReduction is the path optimization strategy, t 3 is the weight of topology reconstruction cost, ReconfigCos is the topology reconstruction cost loss;

[0042] Step S43: Quantum-inspired optimization, specifically, on the basis of the standard quantum-inspired algorithm, by combining quantum superposition correction parameters, improving the quantum-inspired algorithm to obtain the quantum superposition inspired optimization algorithm, and optimizing the dynamic optimal topology structure data according to the quantum superposition inspired optimization algorithm to obtain the improved topology structure data;

[0043] Step S44: Dynamic topology verification, specifically, through performance evaluation and stability analysis, verifying the improved topology structure data through the data flow digital twin model, and comparing by simulating the running state of the new topology with the actual system to perform dynamic topology verification to obtain the dynamic topology structure data.

[0044] Furthermore, in step S5, the resource orchestration optimization is used to adjust the resource allocation structure, specifically, based on the dynamic topology structure data and the data flow decision reference data, performing multi-dimensional resource allocation optimization to obtain the data flow optimization comprehensive reference data;

[0045] The data flow optimization comprehensive reference data specifically includes bandwidth allocation data, computing resource allocation data, latency optimization data, energy consumption optimization reference data, and dynamic topology adaptability reference data.

[0046] The data flow optimization system based on artificial intelligence provided by the present invention includes a data flow perception layer, a digital twin layer, an intelligent decision-making layer, and an execution control layer;

[0047] The data stream perception layer is used for multi-modal data stream acquisition. Through multi-modal data stream acquisition, a data stream multi-modal feature dataset is obtained, and the data stream multi-modal feature dataset is sent to the digital twin layer;

[0048] The digital twin layer is used for digital twin basic modeling. Through digital twin basic modeling, a data stream digital twin model is obtained, and the data stream digital twin model is sent to the intelligent decision-making layer;

[0049] The intelligent decision-making layer is used for decision reinforcement and dynamic topology optimization. Through decision reinforcement and dynamic topology optimization, data stream decision reference data and dynamic topology structure data are obtained, and the data stream decision reference data and dynamic topology structure data are sent to the execution control layer;

[0050] The execution control layer is used for resource orchestration optimization. Through resource orchestration optimization, data stream optimization comprehensive reference data is obtained.

[0051] The beneficial effects achieved by the present invention using the above solution are as follows:

[0052] (1) Aiming at the technical problems existing in the existing data stream optimization methods, that is, due to the existence of different data formats and data types in the existing network data stream, the digital twin modeling of the data stream not only cannot accurately reflect the dynamic relationship between the physical system and the virtual model, but also cannot implement the dynamic changes of the corresponding system. This further leads to the deficiencies in the security, accuracy, and real-time performance of the existing data stream decision-making. This solution creatively adopts a data stream integration optimization solution that combines digital twin basic modeling, dynamic topology optimization, and decision comprehensive optimization, which not only improves the flexibility and real-time performance of the system, but also optimizes the efficiency and intelligent level of resource allocation;

[0053] (2) Aiming at the problems existing in the existing digital twin basic modeling process, that is, during the modeling process of the physical layer and the virtual layer, the modeling accuracy of the virtual topology mapping layer often lacks, and at the same time, this also leads to poor dynamic adaptability of the virtual model and insufficient noise simulation. This solution creatively adopts quantum mapping and quantum optimization technologies, combined with adaptive noise and perturbation simulation, to improve the mapping accuracy and dynamic adaptation ability between the virtual layer and the physical layer. Through parallel processing of quantum mapping, the state error between virtual nodes and physical nodes is optimized, making the virtual topology mapping more accurate;

[0054] (3) In view of the technical problems that in the existing data flow decision optimization and dynamic topology optimization processes, the computational complexity of the topology optimization of the decision-making model is high, and at the same time, the difficulty of real-time adjustment is also high. In the dynamic topology optimization process, traditional methods often face the problems of tight computational resources and great difficulty in real-time adjustment. This solution creatively adopts hierarchical federated learning combined with an adaptive learning strategy to optimize data flow decisions, and introduces a joint optimization method of a quantum heuristic algorithm and a graph neural network in dynamic topology optimization. By innovatively designing a hybrid improved reward function (step S33), considering multiple factors such as throughput, latency, security, and energy efficiency, to adapt to different service requirements and environmental conditions, the data flow decision-making process is optimized. At the same time, by combining a quantum heuristic algorithm and a graph neural network (steps S41 to S44), an optimization method of introducing quantum superposition correction parameters is innovatively introduced, reducing the computational complexity of topology structure optimization, and improving the efficiency and accuracy of topology structure adjustment through the quantum heuristic optimization algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a schematic flow chart of the data flow optimization method based on artificial intelligence provided by the present invention;

[0056] Figure 2 is a schematic diagram of the data flow optimization system based on artificial intelligence provided by the present invention;

[0057] Figure 3 is a schematic flow chart of the digital twin basic modeling in step S2;

[0058] Figure 4 is a schematic flow chart of the decision reinforcement in step S3;

[0059] Figure 5 is a schematic flow chart of the dynamic topology optimization in step S4.

[0060] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, but do not constitute a limitation to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0062] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc. indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0063] Example 1. Refer to Figure 1 , the data stream optimization method based on artificial intelligence provided by the present invention includes the following steps:

[0064] Step S1: Multimodal data stream acquisition;

[0065] Step S2: Digital twin basic modeling;

[0066] Step S3: Decision reinforcement;

[0067] Step S4: Dynamic topology optimization;

[0068] Step S5: Resource orchestration optimization.

[0069] By performing the above operations, in the existing data stream optimization methods, there are technical problems that in the existing network data streams, due to the existence of different data formats and data types, the digital twin modeling for the data stream not only cannot accurately reflect the dynamic relationship between the physical system and the virtual model, but also cannot implement the dynamic changes of the corresponding system, which further leads to deficiencies in the security, accuracy, and real-time performance of the existing data stream decision-making. The present solution creatively adopts a data stream integration optimization solution combining digital twin basic modeling, dynamic topology optimization, and decision comprehensive optimization, which not only improves the flexibility and real-time performance of the system, but also optimizes the efficiency and intelligent level of resource allocation.

[0070] Example 2. Refer to Figure 1 and Figure 2 , based on the above example, in step S1, the multimodal data stream acquisition is used to collect multi-dimensional data and perform optimization and encoding. Specifically, through multi-source data acquisition, the original data of the data stream is obtained, and feature extraction and compression processing are performed on the original data of the data stream to obtain a multi-modal feature data set of the data stream;

[0071] The multi-source data acquisition specifically collects multi-source heterogeneous data from networks, computing nodes, and storage devices; the original data of the data stream includes network performance data, computing node resource data, storage device performance data, security level data, environmental data, abnormal fault data, and data stream upstream and downstream information data;

[0072] The network performance data includes latency data, packet loss data, and jitter data;

[0073] The computing node resource data includes CPU utilization data, memory occupancy data, load data, and hardware temperature data;

[0074] The storage device performance data includes storage utilization data, read / write speed data, and input / output operation count data;

[0075] The security level data includes data flow security level data and data flow priority identification data;

[0076] The environmental data includes power status data and network topology change data;

[0077] The abnormal fault data includes network anomaly data, node fault data, and data flow anomaly data; The network anomalies specifically include congestion, network segmentation, packet loss, and error detection;

[0078] The upstream and downstream information data of the data flow includes data flow source data and data flow destination data;

[0079] The feature extraction and compression process specifically refers to performing basic preprocessing on the original data of the data flow through data cleaning and denoising, performing feature mapping through heuristic coding technology, and compressing the feature data through an adaptive compression algorithm to obtain a multi-modal feature dataset of the data flow;

[0080] The multi-modal feature dataset of the data flow includes network performance features, computing resource features, storage device features, security level features, abnormal fault features, compression features, and time series features.

[0081] Embodiment 3, refer to Figure 1 、 Figure 2 and Figure 3 Based on the above embodiment, in step S2, the digital twin basic modeling is used to construct a basic digital twin model required for data flow optimization. Specifically, according to the multi-modal feature dataset of the data flow, a digital twin basic model is constructed by building a virtual topology mapping layer, including the following steps:

[0082] Step S21: Construct a virtual mapping. Specifically, by defining a virtual representation of the data flow, a virtual topology mapping layer is constructed, including virtual node construction, virtual connection construction, and virtual policy construction. And during the construction of the virtual node, virtual connection, and virtual policy, quantum mapping and quantum optimization are introduced to improve the virtual mapping, and the physical layer of the data flow is mapped to the virtual layer through topology to obtain the virtual topology mapping layer;

[0083] The quantum mapping performs parallel mapping processing through quantum gate operations; the quantum optimization performs error optimization by minimizing the state error between virtual nodes and physical nodes.

[0084] The calculation formula of the virtual topology mapping layer is:

[0085] ;

[0086] In the formula, V Topo is the virtual topology mapping layer, V Node is the virtual node, V link is the virtual connection, V policy is the virtual policy.

[0087] The calculation formula of the quantum mapping is:

[0088] ;

[0089] In the formula, is the high-dimensional quantum state feature, which is used to represent the data stream feature of the virtual topology mapping layer after parallel mapping processing. N is the total number of features, which is used to represent the total number of original features in the data stream multimodal feature dataset. n is the feature index. is the quantum state phase factor. is the original feature data, which is used to represent the original features in the data stream multimodal feature dataset.

[0090] Step S22: Virtual model noise simulation, specifically, network instability events and abnormal events are simulated by simulating morning and perturbations, and virtual model noise simulation is performed by adding adaptive noise and perturbations to obtain random load virtual noise data.

[0091] Step S23: Virtual layer state prediction, specifically, a standard time series prediction model is constructed to predict the state of the virtual layer, and the change simulation prediction of network load and traffic dynamic parameters is combined with the random load virtual noise data to obtain load fluctuation prediction data.

[0092] The calculation formula of the load fluctuation prediction data is:

[0093] ;

[0094] In the formula, V Node,i is the load fluctuation prediction data, LSTM(·) is the standard time series prediction model, specifically used to represent the long short-term memory neural network model. V Node,i (t) is the state of the virtual node at time t, where i is the virtual node index, V link,i,j(t) is the virtual connection state corresponding to the i-th virtual node at time t, where j is the virtual connection index, V policy,i,,j,k (t) is the virtual policy state corresponding to the i-th virtual node and the j-th virtual connection at time t;

[0095] Step S24: Generate a dynamic transition matrix. Specifically, based on the load fluctuation prediction data and the virtual topology mapping layer, generate a state transition probability matrix in a dynamic environment, which is used to represent the transition probability between system states, and through the state transition probability matrix, perform the conversion representation of the data flow system state;

[0096] Step S25: Calibrate the basic model. Specifically, by calculating the calibration parameter error, compare the state differences between the virtual layer and the physical layer, and perform dynamic adjustment of the parameters of the virtual model to obtain the virtual node error reference data. And based on the virtual node error reference data, iteratively optimize and adjust the virtual nodes, virtual connections, and virtual policies to optimize the virtual topology mapping layer and obtain the data flow digital twin model;

[0097] The calculation formula of the calibration parameter error is:

[0098] ;

[0099] In the formula, is the calibration parameter error, is the virtual node state of the virtual layer, is the virtual node state corresponding to the physical layer.

[0100] By performing the above operations, in the existing digital twin basic modeling process, there are problems that the modeling accuracy of the virtual topology mapping layer often lacks in the modeling processes of the physical layer and the virtual layer. At the same time, this also leads to poor dynamic adaptability of the virtual model and insufficient noise simulation. This solution creatively adopts quantum mapping and quantum optimization technologies, combines adaptive noise and perturbation simulation, improves the mapping accuracy and dynamic adaptability between the virtual layer and the physical layer, optimizes the state error between the virtual node and the physical node through quantum mapping parallel processing, and makes the virtual topology mapping more accurate.

[0101] Example 4, refer to Figure 1 、 Figure 2 and Figure 4 Based on the above example, in step S3, the decision reinforcement is used to optimize the data flow decision through hierarchical federated learning. Specifically, based on the data flow multi-modal feature data set and the data flow digital twin model, a hierarchical federated learning method of hybrid decision optimization is adopted to perform decision reinforcement to obtain the data flow decision reference data, including the following steps:

[0102] Step S31: Construct a hierarchical federated learning architecture. Specifically, construct an edge layer, a fog computing layer, and a cloud coordination layer to build a hierarchical federated learning network structure, and introduce an adaptive learning strategy to perform dynamic adjustment of federated learning to obtain a basic federated learning model;

[0103] The edge layer is used to collect and locally train the model; the fog computing layer is used to process the gradients of edge nodes; the cloud coordination layer is used to receive the aggregated gradient information of the fog computing layer and generate a global optimization decision model;

[0104] The calculation formula of the edge layer is:

[0105] ;

[0106] In the formula, is the local gradient of the model parameter group of the local training model, m is the local training model index, is the local gradient of the loss function of the local training model;

[0107] The calculation formula of the fog computing layer is:

[0108] ;

[0109] In the formula, is the aggregated gradient information output by the fog computing layer, F is the total number of edge nodes, f is the edge node index, is the local gradient of the model parameter group of the local training model;

[0110] Step S32: Global policy aggregation and optimization. Specifically, the cloud coordination layer performs gradient aggregation of the fog computing layer and generates a global optimization policy model, and global policy optimization is performed based on the global optimization policy model;

[0111] The calculation formula of the global policy aggregation and optimization is:

[0112] ;

[0113] In the formula, is the global optimization decision model output by the cloud coordination layer, t is the time index, G is the total number of fog computing nodes, g is the fog computing node index, is the dynamic adjustment coefficient, which is specifically calculated by weighting the delay and load parameters, w m is the weight of the edge node, is the local gradient of the model parameter group of the local training model, is the aggregated gradient information output by the fog computing layer;

[0114] Step S33: Construct a hybrid improved reward function, specifically, by designing a hybrid improved reward function, optimize the data flow decision-making to obtain a hybrid improved global optimization strategy;

[0115] The hybrid improved reward function is specifically constructed by throughput, latency, security, and energy efficiency metrics, and by adjusting the weights according to environmental and business requirements, an optimal hybrid improved reward function is obtained;

[0116] The calculation formula of the optimal hybrid improved reward function is:

[0117] ;

[0118] In the formula, R is the optimal hybrid improved reward function, a is the throughput weight, Throu is the throughput parameter, which is used to represent the amount of data successfully transmitted per unit time, b is the latency weight, Latency is the latency parameter, which is used to represent the latency time of data transmission, c is the security weight, Security is the security parameter, which is used to represent the security evaluation value during data transmission, and d is the energy efficiency weight, Energy is the energy efficiency parameter, which is used to represent the energy consumption evaluation value of the data flow system;

[0119] Preferably, the value of the throughput weight a is 0.45, the value of the latency weight b is 0.25, the value of the security weight c is 0.3, and the value of the energy efficiency weight d is 0.2;

[0120] Step S34: Decision reinforcement, specifically, send the global optimization policy model generated by the cloud to the edge node, and perform local decision-making based on the global optimization policy model to obtain local decision-making data. By adjusting the optimal routing path, scheduling task, and network parameters according to the local decision-making data, data flow decision reference data is obtained;

[0121] The data flow decision reference data specifically includes a global optimization data flow decision-making strategy and distributed execution encrypted data.

[0122] Example Five, refer to Figure 1 、 Figure 2 and Figure 5 Based on the above example, in step S4, the dynamic topology optimization is used to adjust the data flow topology structure in real time. Specifically, based on the data flow digital twin model and the data flow decision reference data, a graph neural network combined with a quantum heuristic algorithm is used to perform dynamic topology optimization to obtain dynamic topology structure data, including the following steps:

[0123] Step S41: Topology analysis enhancement, specifically, using a standard graph neural network to identify the bottleneck nodes and paths of the network topology, and through node feature update and topology feature extraction, obtaining data on the topology analysis features of the data flow network;

[0124] Step S42: Network topology dynamic reconstruction, specifically, based on the data on the topology analysis features of the data flow network, by designing a topology reconstruction function, obtaining data on the optimal topology configuration plan, and using a quantum-inspired optimization algorithm to perform dynamic topology adjustment to obtain data on the dynamic optimal topology structure;

[0125] The calculation formula of the topology reconstruction function is:

[0126] TopoE = t 1 ×Throughput + t 2 ×LateReduction - t 3 ×ReconfigCost;

[0127] In the formula, TopoE is the topology reconstruction function, t 1 is the weight of throughput increase and decrease, Throughput is the throughput increase and decrease strategy, t 2 is the weight of path optimization, LateReduction is the path optimization strategy, t 3 is the weight of topology reconstruction cost, ReconfigCos is the topology reconstruction cost loss;

[0128] Preferably, the value of the weight t 1 of throughput increase and decrease is 0.4, the value of the weight t 2 of path optimization is 0.3, and the value of the weight t 3 of topology reconstruction cost is 0.3;

[0129] Step S43: Quantum-inspired optimization, specifically, on the basis of the standard quantum-inspired algorithm, by combining quantum superposition correction parameters, improving the quantum-inspired algorithm to obtain a quantum superposition inspired optimization algorithm, and based on the quantum superposition inspired optimization algorithm, optimizing the data on the dynamic optimal topology structure to obtain improved topology structure data;

[0130] The calculation formula for improving the quantum-inspired algorithm is:

[0131] ;

[0132] In the formula, is the output of the quantum superposition inspired optimization algorithm, where, is the iteration number index, is the output of the quantum superposition inspired optimization algorithm for the th iteration, is the quantum superposition correction weight, as a whole is the quantum superposition correction term, which is used to optimize the effect of parallel search in multiple solution spaces, is the quantum interference correction weight, as a whole is the quantum interference correction term, which is used to strengthen the search space through the coherence of quantum states, is the weight of the standard search strategy, as a whole is the classical search strategy term of the standard quantum heuristic algorithm, which is used to represent the search strategy of the standard quantum heuristic algorithm;

[0133] Step S44: Dynamic topology verification, specifically, through performance evaluation and stability analysis, verify the improved topology structure data through the data flow digital twin model, and compare the running state of the simulated new topology with the actual system to perform dynamic topology verification and obtain dynamic topology structure data.

[0134] By performing the above operations, in the existing data flow decision optimization and dynamic topology optimization processes, there are problems such as high computational complexity of topology optimization of the decision model and high difficulty in real-time adjustment. In the dynamic topology optimization process, traditional methods often face technical problems such as tight computational resources and large difficulty in real-time adjustment. This solution creatively uses hierarchical federated learning combined with an adaptive learning strategy to optimize data flow decisions, and introduces a joint optimization method of quantum heuristic algorithm and graph neural network in dynamic topology optimization. By innovatively designing a hybrid improvement reward function (step S33), combining multiple factors such as throughput, latency, security, and energy efficiency to adapt to different business requirements and environmental conditions, the data flow decision-making process is optimized. At the same time, by combining the quantum heuristic algorithm and the graph neural network (steps S41 to S44), an optimization method for quantum superposition correction parameters is innovatively introduced, reducing the computational complexity of topology structure optimization, and improving the efficiency and accuracy of topology structure adjustment through the quantum heuristic optimization algorithm.

[0135] Example six, refer to Figure 1 and Figure 2 , based on the above example, in step S5, the resource orchestration optimization is used to adjust the resource allocation structure, specifically, according to the dynamic topology structure data and the data flow decision reference data, perform multi-dimensional resource allocation optimization to obtain the comprehensive reference data for data flow optimization;

[0136] The comprehensive reference data for data flow optimization specifically includes bandwidth allocation data, computing resource allocation data, latency optimization data, energy consumption optimization reference data, and dynamic topology adaptability reference data.

[0137] Example seven, refer to Figure 1 and Figure 2, based on the above embodiment, the artificial intelligence-based data stream optimization system provided by the present invention includes a data stream perception layer, a digital twin layer, an intelligent decision-making layer, and an execution control layer;

[0138] The data stream perception layer is used for multi-modal data stream collection. Through multi-modal data stream collection, a data stream multi-modal feature data set is obtained, and the data stream multi-modal feature data set is sent to the digital twin layer;

[0139] The digital twin layer is used for digital twin basic modeling. Through digital twin basic modeling, a data stream digital twin model is obtained, and the data stream digital twin model is sent to the intelligent decision-making layer;

[0140] The intelligent decision-making layer is used for decision reinforcement and dynamic topology optimization. Through decision reinforcement and dynamic topology optimization, data stream decision reference data and dynamic topology structure data are obtained, and the data stream decision reference data and dynamic topology structure data are sent to the execution control layer;

[0141] The execution control layer is used for resource orchestration optimization. Through resource orchestration optimization, data stream optimization comprehensive reference data is obtained.

[0142] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0143] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention.

[0144] The above describes the present invention and its embodiments, and this description is not restrictive. What is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. Generally speaking, if those of ordinary skill in the art are inspired by it and design similar structural ways and embodiments without creative efforts without departing from the purpose of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A data flow optimization method based on artificial intelligence, characterized by: The method comprises the following steps: Step S1: multimodal data stream collection to obtain a multimodal feature data set of the data stream; Step S2: Digital twin basic modeling, by constructing a virtual topology mapping layer, digital twin basic modeling is performed to obtain a data flow digital twin model, including the following steps: Step S21: construct virtual mapping; Step S22: virtual model noise simulation; Step S23: virtual layer state prediction; Step S24: dynamic transfer matrix generation; Step S25: basic model calibration; In step S21, the virtual mapping is constructed by using quantum mapping and performing parallel mapping processing through quantum gate operation. The calculation formula is: ； In the formula, is a high-dimensional quantum state feature, used to represent the data stream feature of the virtual topology mapping layer after parallel mapping processing, N is the total number of features, used to represent the total number of original features in the multimodal feature data set of the data stream, n is the feature index, is the quantum state phase factor, is original feature data, used to represent the original features in the multimodal feature data set of the data stream; Step S3: Decision reinforcement, using a hierarchical federated learning method with hybrid decision optimization to perform decision reinforcement and obtain data flow decision reference data; Step S4: Dynamic topology optimization, using a graph neural network combined with a quantum heuristic algorithm to perform dynamic topology optimization and obtain dynamic topology structure data; Step S5: Optimize resource scheduling and obtain comprehensive reference data for data flow optimization.

2. The data flow optimization method based on artificial intelligence according to claim 1 is characterized in that: In step S1, the multimodal data stream acquisition is used to acquire multidimensional data and optimize and encode it, specifically, acquiring original data of the data stream through multi-source data acquisition, and performing feature extraction and compression processing on the original data stream to obtain a multimodal feature data set of the data stream; The data stream multimodal feature data set includes network performance features, computing resource features, storage device features, security level features, abnormal fault features, compression features and timing features.

3. The data flow optimization method based on artificial intelligence according to claim 2 is characterized in that: In step S2, the digital twin basic modeling is used to construct the basic digital twin model required for data flow optimization. Specifically, based on the multimodal feature data set of the data flow, a virtual topology mapping layer is constructed to perform digital twin basic modeling to obtain a data flow digital twin model, including the following steps: Step S21: constructing a virtual mapping, specifically, by defining a virtualized representation of a data flow, constructing a virtual topology mapping layer, including virtual node construction, virtual connection construction, and virtual policy construction, and by introducing quantum mapping and quantum optimization in the process of the virtual node construction, virtual connection construction, and virtual policy construction, improving the virtual mapping, mapping the data flow physical layer to the virtual layer through topology, and obtaining a virtual topology mapping layer; The quantum mapping performs parallel mapping processing through quantum gate operation; the quantum optimization performs error optimization by minimizing the state error between virtual nodes and physical nodes; Step S22: virtual model noise simulation, specifically, simulating network instability events and abnormal events by simulating morning and disturbance, and performing virtual model noise simulation by adding adaptive noise and disturbance to obtain random load virtual noise data; Step S23: virtual layer state prediction, specifically, constructing a standard time series prediction model to predict the virtual layer state, and combining the random load virtual noise data to simulate and predict the changes in network load and traffic dynamic parameters to obtain load fluctuation prediction data; Step S24: generating a dynamic transfer matrix, specifically generating a state transition probability matrix in a dynamic environment according to the load fluctuation prediction data and the pseudo-topology mapping layer, for representing the transition probability between system states, and performing a conversion representation of the data flow system state through the state transition probability matrix; Step S25: Basic model calibration, specifically, calculating the calibration parameter error, comparing the state difference between the virtual layer and the physical layer, and dynamically adjusting the parameters of the virtual model to obtain virtual node error reference data, and iteratively optimizing and adjusting virtual nodes, virtual connections and virtual strategies based on the virtual node error reference data to optimize the virtual topology mapping layer and obtain a data flow digital twin model.

4. The data flow optimization method based on artificial intelligence according to claim 3 is characterized in that: In step S3, the decision reinforcement is used to optimize the data flow decision through hierarchical federated learning, specifically, based on the data flow multimodal feature dataset and the data flow digital twin model, a hierarchical federated learning method of hybrid decision optimization is adopted to perform decision reinforcement to obtain data flow decision reference data, including the following steps: Step S31: construct a hierarchical federated learning architecture, specifically by constructing an edge layer, a fog computing layer, and a cloud coordination layer to construct a hierarchical federated learning network structure, and by introducing an adaptive learning strategy to dynamically adjust the federated learning to obtain a federated learning basic model; Step S32: global strategy aggregation optimization, specifically, performing gradient aggregation of the fog computing layer through the cloud coordination layer, and generating a global optimization strategy model, and performing global strategy optimization based on the global optimization strategy model; The calculation formula for the global strategy aggregation optimization is: ； In the formula, is the global optimization decision model output by the cloud coordination layer, t is the time index, G is the total number of fog computing nodes, g is the fog computing node index, is a dynamic adjustment coefficient, which is calculated by weighting the delay and load parameters. m is the weight of the edge node, is the local gradient of the model parameter set of the local training model, It is the aggregated gradient information output by the fog computing layer; Step S33: constructing a hybrid improved reward function, specifically, optimizing the data flow decision by designing a hybrid improved reward function, and obtaining a hybrid improved global optimization strategy; The hybrid improved reward function is specifically constructed by throughput, latency, security and energy efficiency indicators, and the optimal hybrid improved reward function is obtained by adjusting the weights according to the environment and business requirements; The calculation formula of the optimal hybrid improved reward function is: ； Wherein, R is the optimal hybrid improved reward function, a is the throughput weight, Throu is the throughput parameter, which is used to indicate the amount of data successfully transmitted per unit time, b is the delay weight, Latency is the delay parameter, which is used to indicate the delay time of data transmission, c is the security weight, Security is the security parameter, which is used to indicate the security evaluation value during data transmission, d is the energy efficiency weight, Energy is the energy efficiency parameter, which is used to indicate the energy consumption evaluation value of the data flow system. The hybrid improved reward function is specifically constructed by throughput, delay, security and energy efficiency indicators, and the optimal hybrid improved reward function is obtained by adjusting the weights according to the environment and business needs; Step S34: Decision reinforcement, specifically, sending the global optimization strategy model generated in the cloud to the edge node, and making local decisions based on the global optimization strategy model to obtain local decision data, and adjusting the optimal routing path, scheduling tasks and network parameters based on the local decision data to obtain data flow decision reference data.

5. The data flow optimization method based on artificial intelligence according to claim 4 is characterized in that: In step S3, the data flow decision reference data specifically includes a global optimization data flow decision strategy and distributed execution encrypted data.

6. The data flow optimization method based on artificial intelligence according to claim 5 is characterized in that: In step S4, the dynamic topology optimization is used to adjust the data flow topology structure in real time, specifically, based on the data flow digital twin model and the data flow decision reference data, a graph neural network combined with a quantum heuristic algorithm is used to perform dynamic topology optimization to obtain dynamic topology structure data, including the following steps: Step S41: topology analysis enhancement, specifically using a standard graph neural network to identify bottleneck nodes and paths of the network topology, and obtaining data flow network topology analysis feature data through node feature updating and topology feature extraction; Step S42: Dynamically reconstructing the network topology, specifically, analyzing the characteristic data of the data flow network topology, designing a topology reconstruction function, obtaining the optimal solution data of the topology configuration, and using a quantum heuristic optimization algorithm to perform dynamic topology adjustment to obtain dynamic optimal topology structure data; The calculation formula of the topology reconstruction function is: TopoE=t1×Throughput+t2×LateReduction-t3×ReconfigCost; Where TopoE is the topology reconstruction function, t1 is the throughput increase / decrease weight, Throughput is the throughput increase / decrease strategy, t2 is the path optimization weight, LateReduction is the path optimization strategy, t3 is the topology reconstruction cost weight, and ReconfigCos is the topology reconstruction cost loss. Step S43: quantum heuristic optimization, specifically, based on the standard quantum heuristic algorithm, by combining the quantum superposition correction parameter, the quantum heuristic algorithm is improved to obtain the quantum superposition heuristic optimization algorithm, and according to the quantum superposition heuristic optimization algorithm, the dynamic optimal topological structure data is optimized to obtain the improved topological structure data; Step S44: Dynamic topology verification, specifically, verifying the improved topology structure data through the data flow digital twin model through performance evaluation and stability analysis, and performing dynamic topology verification by simulating the operating status of the new topology and comparing it with the actual system to obtain dynamic topology structure data.

7. The data flow optimization method based on artificial intelligence according to claim 6 is characterized in that: In step S5, the resource scheduling optimization is used to adjust the resource allocation structure, specifically, to perform multi-dimensional resource allocation optimization based on the dynamic topology structure data and the data flow decision reference data to obtain data flow optimization comprehensive reference data; The data flow optimization comprehensive reference data specifically includes bandwidth allocation data, computing resource allocation data, delay optimization data, energy consumption optimization reference data and dynamic topology adaptability reference data.

8. An artificial intelligence-based data flow optimization system, used to implement the artificial intelligence-based data flow optimization method according to any one of claims 1 to 7, characterized in that: It includes data stream perception layer, digital twin layer, intelligent decision-making layer and execution control layer.

9. The artificial intelligence-based data flow optimization system according to claim 8, characterized in that: The data stream perception layer is used for multimodal data stream acquisition, obtains a data stream multimodal feature data set through multimodal data stream acquisition, and sends the data stream multimodal feature data set to the digital twin layer; The digital twin layer is used for digital twin basic modeling, obtains the data stream digital twin model through the digital twin basic modeling, and sends the data stream digital twin model to the intelligent decision-making layer; The intelligent decision layer is used for decision reinforcement and dynamic topology optimization, and obtains data flow decision reference data and dynamic topology structure data through decision reinforcement and dynamic topology optimization, and sends the data flow decision reference data and dynamic topology structure data to the execution control layer; The execution control layer is used for resource scheduling optimization, and through resource scheduling optimization, comprehensive reference data for data flow optimization is obtained.

Images