A method for improving the data flow effect based on artificial intelligence

Abstract

The present invention discloses a method for improving data flow effects based on artificial intelligence, which solves the problem that existing data flow methods cannot dynamically adjust the data flow processing process according to the real-time load of the blockchain and the priority level of the data flow, and are insufficiently adaptable to meet different business needs. The method includes filtering and processing redundant information of data flow files based on a data filter, hierarchically encrypting the data flow files, constructing and training a flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of the encrypted data flow, and the flow scheduling model distributes the encrypted data flow. In the present invention, the data filter and the flow scheduling model cooperate with each other to achieve filtering redundancy of the data flow files, optimize the scheduling flow processing, reduce the storage burden and network transmission pressure of the system, thereby improving the efficiency of data processing, and ensuring the priority execution of key tasks, further improving the response speed and throughput of the overall system.

Description

Technical Field

[0001] The present invention belongs to the field of data flow technology, and specifically relates to a method for improving data flow effects based on artificial intelligence. Background Art

[0002] The rapid development and widespread adoption of information technologies such as big data and the internet have led to a significant increase in the number of users of various internet applications, including collaborative office work, online education, and social networking. To meet these demands, data often needs to flow frequently between domains. In traditional single-domain data storage environments, data is accessed and used only within the domain, with no connection or interaction between domains. However, with the increasing demand for cross-domain use of massive amounts of data, independent domains need to communicate and share data. Establishing fast communication and transfer channels between domains, breaking down data barriers, and achieving efficient cross-domain data transfer and sharing has become an urgent challenge.

[0003] Chinese patent CN117493465A discloses a data flow method, device, electronic device, and storage medium, including: obtaining a pre-set metadata configuration table and metadata field configuration table, as well as data flow requirements; determining the master end and slave end of the data to be transferred based on the data flow requirements; and controlling the master end to transfer data with the slave end based on the metadata configuration table and metadata field configuration table. However, the existing data flow method controls the master end to transfer data with the slave end based on the metadata configuration table and metadata field configuration table. It fails to filter out redundant data, increases the amount of data to be transmitted, and fails to dynamically adjust the data flow processing flow according to the real-time load of the blockchain path and the priority of the data flow, resulting in insufficient adaptability to meet different business needs. To address the above problems, we propose an artificial intelligence-based data flow effect improvement method. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the existing technology and provide an artificial intelligence-based data flow effect improvement method, which solves the problem that the existing data flow method controls the master end to transfer data with the slave end according to the metadata configuration table and the metadata field configuration table and cannot filter out redundant data, thereby increasing the amount of data to be transmitted, and cannot dynamically adjust the data flow processing flow according to the real-time load of the blockchain and the priority level of the data flow, thereby insufficient adaptability to meet different business needs.

[0005] The present invention is implemented as follows: a method for improving data flow effects based on artificial intelligence, the method for improving data flow effects based on artificial intelligence specifically comprising:

[0006] In response to an instruction to receive at least one set of data flow transfer files pushed by a regional traffic node, receiving at least one set of data flow transfer files, wherein the data flow transfer files include a data flow ID, a data flow topic, a data flow type, and a service ID, and filtering redundant information of the data flow transfer files based on a data filter;

[0007] Loading the data flow file after redundant information filtering, encrypting the data flow file in a hierarchical manner based on the importance of the data, and outputting at least one set of encrypted data streams;

[0008] Analyze the encrypted data flow characteristics, build and train the flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of the encrypted data flow, and output a converged flow scheduling model;

[0009] The encrypted data stream is written into the convergent flow scheduling model. The flow scheduling model combines priority features, importance features, data features, and blockchain path load to allocate the encrypted data stream flow, and outputs a flow allocation map that represents regional traffic nodes, encryption levels, blockchain paths, and target traffic nodes.

[0010] Verify the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, push the encrypted data stream to the target traffic node;

[0011] If the identity and flow authority verification of the target traffic node fails, the flow warning instruction is triggered. The flow scheduling model responds to the flow warning instruction and calls the backup traffic node in real time based on the blockchain road load to execute the encrypted data flow secondary flow allocation task.

[0012] Preferably, the method for filtering redundant information of data flow files based on a data filter specifically includes:

[0013] Pre-built data filters, which are distributed and deployed on regional traffic nodes on each end side;

[0014] Load at least one set of data flow files, process abnormal values and missing values in the data flow files, and obtain standard data flow files;

[0015] Define a data filter to improve the preprocessing function of the Flink block. Preprocess the data flow file based on the preprocessing function. The preprocessing function is a variance probability distribution function of the time window. The preprocessing function receives the data flow file with a key, and each time window corresponds to a key.

[0016] The preprocessing function is expressed as:

[0017] (1)

[0018] in, It represents the variance probability output after the preprocessing function preprocesses the data flow file. Indicates the time window size, is the number of time windows, Indicates the time for data flow files to be preprocessed by the time window. Indicates data flow file input representation, Respectively represent the maximum and minimum values of the data flow file within the time window period;

[0019] According to the preprocessing function, the variance probability output representation of the data flow file after preprocessing introduces a redundancy consistency constraint, a filtering function is defined based on the redundancy consistency constraint, and the data flow file after the redundant information is filtered is output;

[0020] The redundant consistency constraint is expressed as:

[0021] (2)

[0022] (3)

[0023] The filter function is expressed as:

[0024] (4)

[0025] in, Indicates the time window size, are the height and width of the time window, is the average value of data flow files within the time window period, It is the output representation of the data flow file after being filtered by the filter function;

[0026] Filter out redundant information in data flow files, annotate redundant information with labels, merge the labels with redundant information, upload backups through the data cache layer, and regularly verify the integrity and recoverability of the backups.

[0027] Preferably, the data filter filters the data flow files uploaded by the regional traffic nodes according to a preset filtering frequency, and packages and compresses the data flow files before uploading. The data filter is composed of multiple microfilters. The microfilters provide upstream filtering services and downstream filtering services. The microfilters include an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit. The Flink block communicates with the data aggregation unit, the data cache layer, and the backup upload unit through an abstract interface. The improved Flink block optimizes the control of task parallelism by configuring the parallelism parameter for the Flink block, and writes the data flow file task at one time after aggregating multiple data flow file records.

[0028] Preferably, the method for hierarchically encrypting data flow files based on data importance specifically includes:

[0029] Obtain data flow files, use SALib library to perform quantitative analysis on the sensitivity of data flow files, and obtain data sensitivity coefficients;

[0030] Determine the value coefficient of data flow files based on hierarchical analysis method combined with random forest;

[0031] Obtain data sensitivity coefficient and value coefficient, define data importance function, and output data importance;

[0032] Define security parameters for data transfer files based on data importance ,property and dependency group information , generate the unique encryption code corresponding to the data flow file , output system public key based on hash encryption algorithm and hierarchical master keys ;

[0033] (5)

[0034] (6)

[0035] in, They are hash operations and hash operations for verification, Two random numbers generated for the hash algorithm, They are bilinear groups and bilinear maps of prime order respectively;

[0036] Get the system public key and hierarchical master keys , based on the system public key and hierarchical master keys Provide hierarchical encryption for data flow files.

[0037] Preferably, the value coefficient is determined by constructing a hierarchical model based on random forest local encoding and global hierarchical analysis. The hierarchical model consists of a target layer, a criterion layer, and a solution layer. The hierarchical model uses a random forest algorithm to train the data value features to obtain a convergent hierarchical model.

[0038] The data importance is calculated using the following formula:

[0039] (7)

[0040] in, Indicates the importance of data, Represent the data sensitivity coefficient and value coefficient respectively. Parameters representing the importance function.

[0041] Preferably, the method for constructing and training a flow scheduling model based on the priority features, importance features, and data features of encrypted data flows specifically includes:

[0042] The Pipeline model is used as the initial model. The Pipeline model consists of the Pipeline layer, the Valve layer, and the Context layer. The architecture of the predefined initial model is the ResNet architecture, and the artificial intelligence sliding window mechanism is introduced into the initial model.

[0043] The instability of the sliding window is calculated using the Courbet-Léberle divergence and is determined by the entropy of the flow distribution graph.

[0044] Obtain encrypted data streams, generate encrypted data stream labels based on priority features, importance features, and data features, and divide the encrypted data streams with generated labels into training sets and test sets;

[0045] The encrypted data stream is annotated by combining four sets of attribute features: priority feature, importance feature, data feature, and blockchain path load;

[0046] Select an appropriate loss function to measure the difference between the Pipeline model's predicted flow distribution map and the actual flow distribution map, and select an optimizer to update the Pipeline model parameters;

[0047] The initial model is iteratively trained using the training set until the initial model converges;

[0048] Obtain a test set, use the test set as input, execute the flow scheduling model, and determine whether the test result meets the preset accuracy threshold. If it meets the preset accuracy threshold, output a converged flow scheduling model.

[0049] Preferably, the flow scheduling model combines priority characteristics, importance characteristics, data characteristics, and blockchain circuit load to allocate the encrypted data flow, specifically including:

[0050] Load the encrypted data stream, use the encrypted data stream as input, execute the flow scheduling model, and determine the priority characteristics, importance characteristics, and data characteristics of the encrypted data stream;

[0051] The flow scheduling model is based on the Courbet-Léberle divergence to calculate the fusion of priority features, importance features, and data features to obtain the feature fusion value;

[0052] Determine the blockchain load based on the fitness function combined with the blockchain processing capacity, bandwidth utilization, and transaction confirmation time;

[0053] Combine the feature fusion value and the blockchain path load to calculate the allocation priority value, generate the allocation priority sequence, and allocate the encrypted data flow based on the allocation priority sequence;

[0054] The allocation priority value is calculated using the following formula:

[0055] (8)

[0056] in, Indicates the allocation priority value, is the number of encrypted data stream files, Indicates the number of blockchain paths, They are the current blockchain road load and the average blockchain road load, represents the feature fusion value, is the crossover probability coefficient of the fitness function.

[0057] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0058] In an embodiment of the present invention, a data filter is used to filter redundant information in data flow files, and a flow scheduling model is used to allocate encrypted data flow based on priority characteristics, importance characteristics, data characteristics, and blockchain load. The data filter and flow scheduling model work together to filter redundant data flow files, optimize scheduling flow processing, reduce the system's storage burden and network transmission pressure, thereby improving data processing efficiency, ensuring that critical tasks are executed first, and further improving the overall system's response speed and throughput. This overcomes the problems of existing data flow methods, such as the inability of the master end to filter out redundant data when controlling data flow with the slave end based on the metadata configuration table and metadata field configuration table, thereby increasing the amount of data required to be transmitted, and the inability to dynamically adjust the data flow processing process based on the real-time blockchain load and the priority level of the data flow, thereby lacking adaptability to different business needs.

[0059] In an embodiment of the present invention, redundant information in data flow files is filtered and processed through a data filter, and the data filter is composed of multiple microfilters. The microfilter includes an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit, thereby realizing filtering, conversion, and aggregation operations on real-time or batch data flow files to remove redundant, erroneous, and unnecessary data, thereby enabling quality control of data flow files to ensure data accuracy, integrity, and reliability.

[0060] In the embodiment of the present invention, the data importance function is defined by the data sensitivity coefficient and the value coefficient, the data importance is output, and the security parameters of the data flow file are defined based on the data importance. , thus overcoming the problem of limited system computing power and computing resources. Hierarchical encryption of data flow files can help users improve data flow efficiency and ensure the privacy of data flow files.

[0061] In the embodiment of the present invention, a flow scheduling model is trained by combining priority features, importance features, data features, and blockchain load. At the same time, the flow scheduling model uses the Pipeline model as the initial model, so that data flow can be prioritized according to the importance and urgency of the data, ensuring that critical tasks can obtain resources first, thereby improving overall efficiency. It can also reasonably allocate network bandwidth and storage resources through analysis of data features, avoid resource waste, and improve the load capacity of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the implementation flow of the method for improving data flow effects based on artificial intelligence provided by the present invention.

[0063] Figure 2 The present invention shows a schematic diagram of the implementation process of a method for filtering redundant information of data flow files based on a data filter.

[0064] Figure 3 The diagram shows the implementation process of the hierarchical encryption method for data flow files based on data importance.

[0065] Figure 4 A schematic diagram of the implementation process of a method for constructing and training a flow scheduling model based on the priority features, importance features, and data features of encrypted data streams is shown.

[0066] Figure 5 The figure shows a flow chart of the implementation process of the encrypted data flow allocation method by combining the flow scheduling model with priority characteristics, importance characteristics, data characteristics, and blockchain path load.

[0067] Figure 6 A structural diagram of a data flow effect improvement system based on artificial intelligence is shown. DETAILED DESCRIPTION

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0069] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0070] Existing data flow methods control the master end to transfer data with the slave end based on the metadata configuration table and the metadata field configuration table. However, they cannot filter out redundant data, which increases the amount of data that needs to be transmitted. In addition, they cannot dynamically adjust the data flow processing process according to the real-time load of the blockchain path and the priority level of the data flow, resulting in insufficient adaptability to meet different business needs. To address the above problems, we propose an artificial intelligence-based data flow effect improvement method. In short, when the method is implemented, it first responds to at least one set of data flow file reception instructions pushed by the regional traffic node, receives at least one set of data flow files, filters and processes redundant information in the data flow files based on the data filter, and after filtering, hierarchically encrypts the data flow files based on the importance of the data, outputs at least one set of encrypted data streams, and writes the encrypted data streams into the flow scheduling model. The flow scheduling model combines priority features, importance features, data features, and blockchain path load to allocate the encrypted data streams, and finally verifies the identity and flow permission of the target traffic node. If the identity and flow permission of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node. In an embodiment of the present invention, a data filter is used to filter redundant information in data flow files, and a flow scheduling model is used to allocate encrypted data flow based on priority characteristics, importance characteristics, data characteristics, and blockchain load. The data filter and flow scheduling model work together to filter redundant data flow files, optimize scheduling flow processing, reduce the system's storage burden and network transmission pressure, thereby improving data processing efficiency, ensuring that critical tasks are executed first, and further improving the overall system's response speed and throughput. This overcomes the problems of existing data flow methods, such as the inability of the master end to filter out redundant data when controlling data flow with the slave end based on the metadata configuration table and metadata field configuration table, thereby increasing the amount of data required to be transmitted, and the inability to dynamically adjust the data flow processing process based on the real-time blockchain load and the priority level of the data flow, thereby lacking adaptability to different business needs.

[0071] The embodiment of the present invention provides a method for improving data flow effect based on artificial intelligence. Figure 1 The following is a schematic diagram of an implementation process of a method for improving data flow effects based on artificial intelligence, which specifically includes:

[0072] Step S10, in response to the at least one set of data flow file receiving instructions pushed by the regional traffic node, receiving at least one set of data flow files, and filtering redundant information of the data flow files based on a data filter;

[0073] It should be noted that the data flow file includes but is not limited to data flow ID, data flow topic, data flow type, and business ID. The regional traffic node refers to the key node responsible for processing, forwarding and storing data in a specific area during the data flow process. Within the data center, the regional traffic node can be used to connect different server groups or racks to achieve efficient data flow within the data center. In a wide area network environment, the regional traffic node can serve as a data exchange point between different geographical locations to achieve cross-regional data flow.

[0074] Step S20, loading the data flow file after redundant information filtering, hierarchically encrypting the data flow file based on data importance, and outputting at least one set of encrypted data streams;

[0075] Step S30: Analyze the encrypted data flow characteristics, build and train a flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of the encrypted data flow, and output a converged flow scheduling model;

[0076] Step S40: Write the encrypted data stream into the converged flow scheduling model. The flow scheduling model allocates the encrypted data stream flow based on priority features, importance features, data features, and blockchain load, and outputs a flow distribution map that represents regional traffic nodes, encryption levels, blockchain paths, and target traffic nodes.

[0077] Step S50, verifying the identity and transfer authority of the target traffic node;

[0078] It's important to note that the identity of the target traffic node can be verified using JWT (JSON Web Token). JWT is an open JSON-based standard for communicating claims between parties. It consists of three parts: a header, a payload, and a signature. The advantage of JWT is that it's self-contained, can carry user identity and permission information, and is easily transferred between different services. Verification of transfer permissions can be achieved through an API gateway, which integrates with existing authentication systems such as LDAP, OAuth, and SAML to achieve unified authentication and single sign-on (SSO).

[0079] Step S60: If the identity and transfer authority of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node;

[0080] In step S70, if the identity and flow authority verification of the target flow node fails, a flow warning instruction is triggered. The flow scheduling model responds to the flow warning instruction and calls the backup flow node in real time based on the blockchain road load to perform the encrypted data flow secondary flow distribution task.

[0081] In this embodiment, after step S70 calls the backup traffic node in real time based on the blockchain road load to execute the secondary flow distribution task of the encrypted data flow, it is still necessary to verify the identity and flow authority of the backup traffic node. If it fails, in order to reduce the system data flow load, the tertiary flow distribution task will no longer be executed, and the traffic node abnormal instruction will be triggered.

[0082] When the method is implemented, it first responds to at least one group of data flow file receiving instructions pushed by the regional traffic node, receives at least one group of data flow files, filters and processes redundant information of the data flow files based on the data filter, encrypts the data flow files in a hierarchical manner based on the importance of the data after filtering, outputs at least one group of encrypted data streams, writes the encrypted data streams into the flow scheduling model, and the flow scheduling model allocates the encrypted data stream flow in combination with priority features, importance features, data features, and blockchain path load, and finally verifies the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node.

[0083] In an embodiment of the present invention, a data filter is used to filter redundant information in data flow files, and a flow scheduling model is used to allocate encrypted data flow based on priority characteristics, importance characteristics, data characteristics, and blockchain load. The data filter and flow scheduling model work together to filter redundant data flow files, optimize scheduling flow processing, reduce the system's storage burden and network transmission pressure, thereby improving data processing efficiency, ensuring that critical tasks are executed first, and further improving the overall system's response speed and throughput. This overcomes the problems of existing data flow methods, such as the inability of the master end to filter out redundant data when controlling data flow with the slave end based on the metadata configuration table and metadata field configuration table, thereby increasing the amount of data required to be transmitted, and the inability to dynamically adjust the data flow processing process based on the real-time blockchain load and the priority level of the data flow, thereby lacking adaptability to different business needs.

[0084] The embodiment of the present invention provides a method for filtering redundant information of data flow files based on a data filter. Figure 2 The following is a schematic diagram of a method for filtering redundant information in data flow files based on a data filter. The method for filtering redundant information in data flow files based on a data filter specifically includes:

[0085] Step S101: pre-build a data filter, where the data filter is distributed and deployed on the regional traffic nodes on each end side;

[0086] In this embodiment, consistent hashing can be used to distribute data filters across regional traffic nodes on each end side, ensuring load balancing and fault tolerance. When nodes join or leave, the consistent hashing algorithm can redistribute data filters, ensuring high system availability.

[0087] Step S102: loading at least one set of data flow files, processing abnormal values and missing values in the data flow files, and obtaining standard data flow files;

[0088] It should be noted that the abnormal values and missing values in the data flow files can be processed by using the labeling method combined with the square root transformation.

[0089] Step S103: Define a data filter to improve the preprocessing function of the Flink block, and preprocess the data flow file based on the preprocessing function. The preprocessing function is a variance probability distribution function of the time window. The preprocessing function receives the data flow file with keys, and each time window corresponds to a key.

[0090] The preprocessing function is expressed as:

[0091] (1)

[0092] in, It represents the variance probability output after the preprocessing function preprocesses the data flow file. Indicates the time window size, is the number of time windows, Indicates the time for data flow files to be preprocessed by the time window. Indicates data flow file input representation, Respectively represent the maximum and minimum values of the data flow file within the time window period;

[0093] Step S104, introducing a redundancy consistency constraint into the variance probability output representation after preprocessing the data flow file according to the preprocessing function, defining a filtering function based on the redundancy consistency constraint, and outputting the data flow file after the redundancy information is filtered out;

[0094] The redundant consistency constraint is expressed as:

[0095] (2)

[0096] (3)

[0097] The filter function is expressed as:

[0098] (4)

[0099] in, Indicates the time window size, are the height and width of the time window, is the average value of data flow files within the time window period, It is the output representation of the data flow file after being filtered by the filter function;

[0100] Step S105 , filter out redundant information of the data flow file, annotate the redundant information with a tag, merge the tag with the redundant information, upload the backup through the data cache layer, and regularly verify the integrity and recoverability of the backup.

[0101] In this embodiment, the data filter filters data transfer files uploaded by regional traffic nodes according to a pre-set filtering frequency, packages and compresses the data transfer files, and then uploads them. The data filter is composed of multiple microfilters, which provide upstream and downstream filtering services. The microfilters include an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit. The Flink block communicates with the data aggregation unit, data cache layer, and backup upload unit via abstract interfaces, including but not limited to List, Map, Comparable, Runnable, and AutoCloseable. The improved Flink block optimizes task parallelism by configuring the parallelism parameter in the Flink block, aggregating multiple data transfer file records and writing the data transfer file task in one go. Furthermore, container technologies such as Docker can be used to improve the Flink block, packaging Flink jobs and their dependencies into standardized units to facilitate consistent deployment and operation in different environments. This allows the optimized Flink block to support multiple deployment modes (such as standalone mode and cluster mode), allowing the most appropriate deployment mode to be selected based on actual needs. For example, during the development phase, you can choose local mode for rapid iteration; in a production environment, you should choose cluster mode to ensure high availability and scalability. In addition, you can use Flink's REST API to dynamically submit and manage jobs, further improving operational efficiency.

[0102] In an embodiment of the present invention, redundant information in data flow files is filtered and processed through a data filter, and the data filter is composed of multiple microfilters. The microfilter includes an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit, thereby realizing filtering, conversion, and aggregation operations on real-time or batch data flow files to remove redundant, erroneous, and unnecessary data, thereby enabling quality control of data flow files to ensure data accuracy, integrity, and reliability.

[0103] The embodiment of the present invention provides a method for hierarchical encryption of data flow files based on data importance. Figure 3 The following is a schematic diagram of the implementation process of a method for hierarchical encryption of data flow files based on data importance. The method for hierarchical encryption of data flow files based on data importance specifically includes:

[0104] Step S201: Obtain a data transfer file, and use the SALib library to perform a quantitative analysis on the sensitivity of the data transfer file to obtain a data sensitivity coefficient;

[0105] In the embodiment of the present invention, the analysis functions provided by the SALib library (such as Sobol analysis) are used to calculate sensitivity indices based on the model output. These indices can help understand the degree of influence of each input parameter on the model output.

[0106] Step S202, determining the value coefficient of the data flow file based on the hierarchical analysis method combined with random forest;

[0107] It should be noted that the value coefficient is determined by constructing a hierarchical model based on random forest local encoding and global hierarchical analysis. The hierarchical model consists of a target layer, a criterion layer, and a solution layer. The hierarchical model uses the random forest algorithm to train data value features to obtain a converged hierarchical model. When collecting data for hierarchical model training, characteristic data related to the value of the data flow, such as data quality indicators, usage frequency, and user feedback, is collected. This collected characteristic data is then trained using the random forest algorithm to obtain a model capable of predicting the value of the data flow.

[0108] In this embodiment, the criterion layer is used to construct a judgment matrix, combine the expert consultation method to determine the importance of data flow files, and compare and score the criteria pairwise based on importance. Then, the geometric mean method in the hierarchical analysis method is used to calculate the weight of the criteria. The criteria include but are not limited to data quality, data volume, data timeliness, and data security.

[0109] Step S203, obtaining the data sensitivity coefficient and value coefficient, defining the data importance function, and outputting the data importance;

[0110] In this embodiment, the data importance is calculated using the following formula:

[0111] (7)

[0112] in, Indicates the importance of data, Represent the data sensitivity coefficient and value coefficient respectively. The parameter representing the importance function. In this embodiment, the importance function may be a Gumbel-Copula function, and the parameter of the Gumbel-Copula function is 0.1-0.5.

[0113] Step S204: define security parameters of data transfer files based on data importance ,property and dependency group information , generate the unique encryption code corresponding to the data flow file , output system public key based on hash encryption algorithm and hierarchical master keys ;

[0114] (5)

[0115] (6)

[0116] in, They are hash operations and hash operations for verification, Two random numbers generated for the hash algorithm, They are bilinear groups and bilinear maps of prime order respectively;

[0117] Step S205: Obtain the system public key and hierarchical master keys , based on the system public key and hierarchical master keys The data flow files are encrypted in a hierarchical manner, and the data flow files are decrypted in a hierarchical manner by the EDSP according to the Lagrange interpolation theorem.

[0118] In the embodiment of the present invention, the data importance function is defined by the data sensitivity coefficient and the value coefficient, the data importance is output, and the security parameters of the data flow file are defined based on the data importance. , thus overcoming the problem of limited system computing power and computing resources. Hierarchical encryption of data flow files can help users improve data flow efficiency and ensure the privacy of data flow files.

[0119] The embodiment of the present invention provides a method for constructing and training a flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of encrypted data flows. Figure 4 A schematic diagram of the implementation process of a method for constructing and training a flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of encrypted data streams is shown. The method for constructing and training a flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of encrypted data streams specifically includes:

[0120] Step S301: Use the Pipeline model as the initial model. The Pipeline model consists of the Pipeline layer, the Valve layer, and the Context layer. The architecture of the predefined initial model is the ResNet architecture. An artificial intelligence sliding window mechanism is introduced into the initial model.

[0121] Step S302 , using Courbet-Léberle divergence to calculate the instability of the sliding window, and determining the instability of the sliding window by the entropy of the flow distribution graph;

[0122] Step S303: Obtain an encrypted data stream, generate an encrypted data stream label based on the priority feature, importance feature, and data feature, and divide the encrypted data stream with the generated label into a training set and a test set;

[0123] Step S304: annotate the encrypted data stream based on the four attribute features of priority, importance, data, and blockchain load.

[0124] Step S305: Select an appropriate loss function to measure the difference between the pipeline model's predicted flow distribution graph and the actual flow distribution graph, and select an optimizer to update the pipeline model parameters;

[0125] Step S306, iteratively training the initial model using the training set until the initial model converges;

[0126] Step S307: Obtain a test set, and execute the flow scheduling model using the test set as input;

[0127] Step S308, determining whether the test result meets a preset accuracy threshold;

[0128] Step S309: If the preset accuracy threshold is met, a converged flow scheduling model is output.

[0129] If it does not meet the preset accuracy threshold, return to step S30 to continue model iterative training.

[0130] In this embodiment, the preset accuracy threshold is set to 0.9-0.95, the ratio of the training set to the test set is 4:1, the training rounds of the model are set to 200-400, the flow scheduling model learning rate is set to 0.0002, the loss function of the flow scheduling model is binary cross entropy loss (BCELoss), and the optimizer can be an RMSprop optimizer.

[0131] In the embodiment of the present invention, a flow scheduling model is trained by combining priority features, importance features, data features, and blockchain load. At the same time, the flow scheduling model uses the Pipeline model as the initial model, so that data flow can be prioritized according to the importance and urgency of the data, ensuring that critical tasks can obtain resources first, thereby improving overall efficiency. It can also reasonably allocate network bandwidth and storage resources through analysis of data features, avoid resource waste, and improve the load capacity of the system.

[0132] The embodiment of the present invention provides a method for distributing encrypted data flow by combining priority characteristics, importance characteristics, data characteristics, and blockchain path load with a flow scheduling model. Figure 5 The flow diagram of the flow scheduling model combining priority characteristics, importance characteristics, data characteristics, and blockchain road load to allocate the encrypted data flow is shown. The flow scheduling model combining priority characteristics, importance characteristics, data characteristics, and blockchain road load to allocate the encrypted data flow specifically includes:

[0133] Step S401: Load the encrypted data stream, use the encrypted data stream as input, execute the flow scheduling model, and determine the priority characteristics, importance characteristics, and data characteristics of the encrypted data stream;

[0134] It should be noted that data features include but are not limited to data type, data format, data source, data quality, and data correlation. The flow scheduling model determines the priority features and importance features of the encrypted data flow. The data features can use correlation analysis, hypothesis testing and other methods to evaluate the importance and effectiveness of each feature.

[0135] Step S402: The flow scheduling model calculates the fusion of priority features, importance features, and data features based on the Courbet-Léberle divergence to obtain a feature fusion value;

[0136] In this embodiment, the Kullback-Leibler Divergence (KL Divergence) is a method for measuring the difference between two probability distributions. In feature fusion, we can use KL Divergence to measure the relative importance of different features, thereby obtaining a comprehensive feature fusion value. When calculating the feature fusion value, we first approximate the feature probability distribution based on kernel density estimation. Then, we calculate the KL divergence between the priority feature, important features, and data features. Then, we assign a weight to each feature based on the KL divergence. Finally, we calculate the weighted average of these values, which is used as the feature fusion value.

[0137] Step S403: Determine the blockchain load based on the fitness function combined with the blockchain processing capacity, bandwidth usage, and transfer determination time.

[0138] Step S404: Calculate the allocation priority value based on the feature fusion value and the blockchain load, generate an allocation priority sequence, and allocate the encrypted data flow based on the allocation priority sequence;

[0139] The allocation priority value is calculated using the following formula:

[0140] (8)

[0141] in, Indicates the allocation priority value, is the number of encrypted data stream files, Indicates the number of blockchain paths, They are the current blockchain road load and the average blockchain road load, represents the feature fusion value, is the crossover probability coefficient of the fitness function. In this embodiment, the crossover probability coefficient can be 0.1-0.3.

[0142] The embodiment of the present invention provides a data flow effect improvement system based on artificial intelligence, Figure 6 The following is a schematic diagram showing the structure of a system for improving data flow effects based on artificial intelligence, which specifically includes:

[0143] The information filtering module 100 receives at least one set of data flow transfer files in response to an instruction to receive at least one set of data flow transfer files pushed by a regional traffic node, wherein the data flow transfer files include a data flow ID, a data flow topic, a data flow type, and a service ID, and filters redundant information in the data flow transfer files based on a data filter;

[0144] The data stream encryption module 200 is used to load the data flow transfer file after redundant information filtering, encrypt the data flow transfer file in a hierarchical manner based on the importance of the data, and output at least one set of encrypted data streams;

[0145] The model building module 300 is used to analyze the encrypted data flow characteristics, build and train the flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of the encrypted data flow, and output a converged flow scheduling model;

[0146] The flow allocation module 400 is used to write the encrypted data flow into the flow scheduling model. The flow scheduling model allocates the encrypted data flow based on priority characteristics, importance characteristics, data characteristics, and blockchain load, and outputs a flow allocation map that represents regional traffic nodes, encryption levels, blockchains, and target traffic nodes.

[0147] The authority verification module 500 is used to verify the identity and flow authority of the target flow node. If the identity and flow authority of the target flow node are verified, the encrypted data flow is pushed to the target flow node;

[0148] The secondary distribution module 600 triggers a flow warning instruction if the identity and flow authority verification of the target traffic node fails. The flow scheduling model responds to the flow warning instruction and calls the backup link in real time based on the blockchain road load to perform the encrypted data flow secondary flow distribution task.

[0149] Another aspect of the present invention provides a computer-readable storage medium storing computer program instructions that can be executed by a processor. When the computer program instructions are executed, the method of any one of the above embodiments is implemented.

[0150] The computer-readable storage medium (e.g., memory) herein may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. By way of example and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which may act as external cache memory. By way of example and not limitation, RAM may be available in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

[0151] In summary, the present invention provides a method for improving data flow effects based on artificial intelligence. When the method is implemented, it first responds to at least one group of data flow file receiving instructions pushed by the regional traffic node, receives at least one group of data flow files, filters and processes redundant information of the data flow files based on the data filter, and after filtering, encrypts the data flow files in a hierarchical manner based on the importance of the data, outputs at least one group of encrypted data streams, writes the encrypted data streams into the flow scheduling model, and the flow scheduling model combines priority features, importance features, data features, and blockchain path load to distribute the encrypted data streams, and finally verifies the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node.

[0152] In an embodiment of the present invention, a data filter is used to filter redundant information in data flow files, and a flow scheduling model is used to allocate encrypted data flow based on priority characteristics, importance characteristics, data characteristics, and blockchain load. The data filter and flow scheduling model work together to filter redundant data flow files, optimize scheduling flow processing, reduce the system's storage burden and network transmission pressure, thereby improving data processing efficiency, ensuring that critical tasks are executed first, and further improving the overall system's response speed and throughput. This overcomes the problems of existing data flow methods, such as the inability of the master end to filter out redundant data when controlling data flow with the slave end based on the metadata configuration table and metadata field configuration table, thereby increasing the amount of data required to be transmitted, and the inability to dynamically adjust the data flow processing process based on the real-time blockchain load and the priority level of the data flow, thereby lacking adaptability to different business needs.

[0153] It should be noted that for the aforementioned embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0154] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the invention. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field can still combine, add, delete or make other adjustments to the features in the various embodiments of the present invention according to the circumstances without conflict, without making creative work, so as to obtain different other technical solutions that do not deviate from the concept of the present invention in essence, and these technical solutions also fall within the scope of protection of the present invention.

Claims

1. A method for improving data flow effect based on artificial intelligence, characterized in that: The method for improving data flow effects based on artificial intelligence specifically includes: In response to at least one set of data flow transfer file receiving instructions pushed by a regional traffic node, receiving at least one set of data flow transfer files, wherein the data flow transfer files include a data flow ID, a data flow topic, a data flow type, and a service ID, and filtering redundant information of the data flow transfer files based on a data filter; Loading the data flow file after redundant information filtering, encrypting the data flow file in a hierarchical manner based on the importance of the data, and outputting at least one set of encrypted data streams; Analyze the encrypted data flow characteristics, build and train the flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of the encrypted data flow, and output a converged flow scheduling model; The encrypted data stream is written into the convergent flow scheduling model. The flow scheduling model combines priority characteristics, importance characteristics, data characteristics, and blockchain path load to allocate the encrypted data stream flow, and outputs a flow allocation map that represents regional traffic nodes, encryption levels, blockchain paths, and target traffic nodes. The flow scheduling model combines priority characteristics, importance characteristics, data characteristics, and blockchain circuit load to allocate the flow of encrypted data flows, specifically including: Load the encrypted data stream, use the encrypted data stream as input, execute the flow scheduling model, and determine the priority characteristics, importance characteristics, and data characteristics of the encrypted data stream; The flow scheduling model is based on the Courbet-Léberle divergence to calculate the fusion of priority features, importance features, and data features to obtain the feature fusion value; Determine the blockchain load based on the fitness function combined with the blockchain processing capacity, bandwidth utilization, and transaction confirmation time; Combine the feature fusion value and the blockchain path load to calculate the allocation priority value, generate the allocation priority sequence, and allocate the encrypted data flow based on the allocation priority sequence; The allocation priority value is calculated using the following formula: （8） in, Indicates the allocation priority value, is the number of encrypted data stream files, Indicates the number of blockchain paths, They are the current blockchain road load and the average blockchain road load, represents the feature fusion value, is the crossover probability coefficient of the fitness function.

2. The method for improving data flow performance based on artificial intelligence according to claim 1, wherein: The method for improving data flow effects based on artificial intelligence specifically further includes: Verify the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, push the encrypted data stream to the target traffic node; If the identity and flow authority verification of the target traffic node fails, the flow warning instruction is triggered. The flow scheduling model responds to the flow warning instruction and calls the backup traffic node in real time based on the blockchain road load to perform the encrypted data flow secondary flow allocation task.

3. The method for improving data flow performance based on artificial intelligence according to claim 1, wherein: The method for filtering redundant information of data flow files based on a data filter specifically includes: Pre-built data filters, which are distributed and deployed on regional traffic nodes on each end side; Load at least one set of data flow files, process abnormal values and missing values in the data flow files, and obtain standard data flow files; Define a data filter to improve the preprocessing function of the Flink block. Preprocess the data flow file based on the preprocessing function. The preprocessing function is a variance probability distribution function of the time window. The preprocessing function receives the data flow file with a key, and each time window corresponds to a key. The preprocessing function is expressed as: （1） in, It represents the variance probability output after the preprocessing function preprocesses the data flow file. Indicates the time window size, is the number of time windows, Indicates the time for data flow files to be preprocessed by the time window. Indicates data flow file input representation, Respectively represent the maximum and minimum values of the data flow file within the time window period; According to the preprocessing function, the variance probability output representation of the data flow file after preprocessing introduces a redundancy consistency constraint, a filtering function is defined based on the redundancy consistency constraint, and the data flow file after the redundant information is filtered is output; The redundant consistency constraint is expressed as: （2） （3） The filter function is expressed as: （4） in, Indicates the time window size, are the height and width of the time window, is the average value of data flow files within the time window period, It is the output representation of the data flow file after being filtered by the filter function; Filter out redundant information in data flow files, annotate redundant information with labels, merge the labels with redundant information, upload backups through the data cache layer, and regularly verify the integrity and recoverability of the backups.

4. The method for improving data flow performance based on artificial intelligence according to claim 3, wherein: The data filter filters the data flow files uploaded by the regional traffic nodes according to a preset filtering frequency, and packages and compresses the data flow files before uploading. The data filter is composed of multiple microfilters, which provide upstream filtering services and downstream filtering services. The microfilter includes an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit. The Flink block communicates with the data aggregation unit, the data cache layer, and the backup upload unit through an abstract interface. The improved Flink block optimizes the control of task parallelism by configuring the parallelism parameter in the Flink block, and aggregates multiple data flow file records and writes the data flow file task at one time.

5. The method for improving data flow performance based on artificial intelligence according to claim 1, wherein: The method for hierarchically encrypting data flow files based on data importance specifically includes: Obtain data flow files, use SALib library to perform quantitative analysis on the sensitivity of data flow files, and obtain data sensitivity coefficients; Determine the value coefficient of data flow files based on hierarchical analysis method combined with random forest; Obtain data sensitivity coefficient and value coefficient, define data importance function, and output data importance; Define security parameters for data transfer files based on data importance ,property and dependency group information , generate the unique encryption code corresponding to the data flow file , output system public key based on hash encryption algorithm and hierarchical master keys ; （5） （6） in, They are hash operations and hash operations for verification, Two random numbers generated for the hash algorithm, They are bilinear groups and bilinear maps of prime order respectively; Get the system public key and hierarchical master keys , based on the system public key and hierarchical master keys Provide hierarchical encryption for data flow files.

6. The method for improving data flow performance based on artificial intelligence according to claim 5, characterized in that: The value coefficient is determined based on the hierarchical model constructed by random forest local encoding and global hierarchical analysis. The hierarchical model consists of the target layer, the criterion layer and the scheme layer. The hierarchical model uses the random forest algorithm to train the data value features to obtain a convergent hierarchical model.

7. The method for improving data flow performance based on artificial intelligence according to claim 6, characterized in that: The data importance is calculated using the following formula: （7） in, Indicates the importance of data, Represent the data sensitivity coefficient and value coefficient respectively. Parameters representing the importance function.

8. The method for improving data flow performance based on artificial intelligence according to any one of claims 2 to 7, characterized in that: The method for constructing and training a flow scheduling model based on the priority features, importance features, and data features of encrypted data flows specifically includes: The Pipeline model is used as the initial model. The Pipeline model consists of the Pipeline layer, the Valve layer, and the Context layer. The architecture of the predefined initial model is the ResNet architecture, and the artificial intelligence sliding window mechanism is introduced into the initial model. The instability of the sliding window is calculated using the Courbet-Léberle divergence and is determined by the entropy of the flow distribution graph. Obtain encrypted data streams, generate encrypted data stream labels based on priority features, importance features, and data features, and divide the encrypted data streams with generated labels into training sets and test sets; The encrypted data stream is annotated by combining four sets of attribute features: priority feature, importance feature, data feature, and blockchain path load; Select an appropriate loss function to measure the difference between the Pipeline model's predicted flow distribution map and the actual flow distribution map, and select an optimizer to update the Pipeline model parameters; The initial model is iteratively trained using the training set until the initial model converges; Obtain a test set, use the test set as input, execute the flow scheduling model, and determine whether the test result meets the preset accuracy threshold. If it meets the preset accuracy threshold, output a converged flow scheduling model.

Images