Data circulation effect improving method based on artificial intelligence

Abstract

The invention discloses a data flow effect improving method based on artificial intelligence, and solves the problem that an existing data flow method cannot dynamically adjust a data flow processing flow according to a real-time load condition of a block link and a priority level of a data flow, and is insufficient in adaptability to different service requirements. The method comprises the steps of filtering redundant information of a data transfer file based on a data filter, performing hierarchical encryption on the data transfer file, constructing and training a transfer scheduling model based on priority features, importance features and data features of an encrypted data stream, and performing transfer distribution on the encrypted data stream by the transfer scheduling model. According to the invention, the data filter and the circulation scheduling model cooperate to filter redundancy of the data circulation file, optimize scheduling circulation processing and reduce the storage burden and network transmission pressure of the system, so that the data processing efficiency is improved, key tasks are ensured to be executed preferentially, and the data processing efficiency is improved. And the response speed and throughput of the whole system are further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data flow, and specifically relates to a method for improving data flow effects based on artificial intelligence. Background Art

[0002] The rapid development and widespread popularization of information technologies such as big data and the Internet have led to a significant increase in the number of users of various Internet applications such as collaborative office, online education, and social networks. In order to meet the above needs, data often needs to be frequently transferred between different domains. In the traditional single-domain data storage environment, data transfer is accessed and used within the domain, and there is no connection or interaction between domains. However, with the increasing demand for cross-domain use of massive data, independent domains need to communicate and share. How to establish a fast connection and transfer channel between different domains, break down data barriers, and achieve efficient cross-domain data transfer and sharing has become an urgent problem to be solved.

[0003] Chinese patent CN117493465A discloses a data flow method, device, electronic device and storage medium, including: obtaining a pre-set metadata configuration table and a metadata field configuration table, as well as data flow requirements; determining the master end and the slave end of the data to be circulated according to the data flow requirements; controlling the master end to perform data flow with the slave end according to the metadata configuration table and the metadata field configuration table. However, the existing data flow method controls the master end to perform data flow with the slave end according to the metadata configuration table and the metadata field configuration table. It cannot filter out redundant data, increases the amount of data to be transmitted, and cannot dynamically adjust the data flow processing flow according to the real-time load of the blockchain and the priority level of the data flow, and is insufficient in adaptability to different business needs. In response to the above problems, we propose a method for improving data flow effects based on artificial intelligence. Summary of the invention

[0004] The purpose of the present invention is to address the shortcomings of the prior art and provide a method for improving data flow effects based on artificial intelligence, which solves the problem that the existing data flow method controls the master end to transfer data with the slave end according to the metadata configuration table and the metadata field configuration table and cannot filter out redundant data, increases the amount of data to be transmitted, and cannot dynamically adjust the data flow processing flow according to the real-time load of the blockchain and the priority level of the data flow, and is insufficient to adapt to different business needs.

[0005] The present invention is implemented as follows: a method for improving data flow effect based on artificial intelligence, the method for improving data flow effect based on artificial intelligence specifically comprising:

[0006] In response to at least one set of data flow transfer file receiving instructions pushed by the regional traffic node, at least one set of data flow transfer files is received, wherein the data flow transfer files include a data flow ID, a data flow topic, a data flow type, and a service ID, and redundant information of the data flow transfer files is filtered based on a data filter;

[0007] Loading the data flow file after redundant information filtering, encrypting the data flow file in a hierarchical manner based on the importance of the data, and outputting at least one set of encrypted data streams;

[0008] Analyze the encrypted data flow characteristics, build and train the flow scheduling model based on the encrypted data flow priority characteristics, importance characteristics, and data characteristics, and output a converged flow scheduling model;

[0009] The encrypted data stream is written into the flow scheduling model. The flow scheduling model allocates the encrypted data stream flow based on priority features, importance features, data features, and blockchain path load, and outputs a flow allocation map that represents regional traffic nodes, encryption levels, blockchain paths, and target traffic nodes.

[0010] Verify the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, push the encrypted data stream to the target traffic node;

[0011] If the identity and flow authority verification of the target traffic node fails, the flow warning instruction is triggered. The flow scheduling model responds to the flow warning instruction and calls the backup traffic node in real time based on the blockchain road load to execute the encrypted data flow secondary flow allocation task.

[0012] Preferably, the method for filtering redundant information of data flow files based on a data filter specifically includes:

[0013] Pre-build data filters, where the data filters are distributed and deployed on regional traffic nodes on each end side;

[0014] Load at least one set of data flow files, process abnormal values ​​and missing values ​​in the data flow files, and obtain standard data flow files;

[0015] A preprocessing function of a Flink block is defined to improve a data filter, and a data flow file is preprocessed based on the preprocessing function, wherein the preprocessing function is a variance probability distribution function of a time window, and the preprocessing function receives a data flow file with a key, and each time window corresponds to a key;

[0016] The preprocessing function is expressed as:

[0017]

[0018] Among them, E k,tIt represents the variance probability output after the preprocessing function preprocesses the data flow file, w k represents the time window size, i is the number of data types in the data flow file, t represents the time of the data flow file preprocessing after the time window, x(t) represents the input representation of the data flow file, maxx(t) and minx(t) represent the maximum and minimum values ​​of the data flow file within the time window period respectively;

[0019] According to the preprocessing function, the variance probability output representation of the data flow file after preprocessing introduces a redundant consistency constraint, defines a filtering function based on the redundant consistency constraint, and outputs the data flow file after redundant information filtering;

[0020] The redundant consistency constraint is expressed as:

[0021]

[0022] w k =m×n(3)

[0023] The filter function is expressed as:

[0024]

[0025] Among them, w k Indicates the time window size, m and n are the height and width of the time window respectively. is the mean value of the data flow file within the time window period, and f(x) is the output representation of the data flow file after filtering by the filter function;

[0026] Filter out redundant information in data flow files, annotate labels for redundant information, merge labels with redundant information, upload backups through the data cache layer, and regularly verify the integrity and recoverability of backups.

[0027] Preferably, the data filter filters the data flow files uploaded by the regional traffic nodes according to a preset filtering frequency, and uploads the data flow files after packaging and compression. The data filter is composed of multiple microfilters, which provide upstream filtering services and downstream filtering services. The microfilter includes an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit. The Flink block communicates with the data aggregation unit, the data cache layer, and the backup upload unit through an abstract interface. The improved Flink block optimizes the control of task parallelism by configuring parallelism parameters to the Flink block, and writes the data flow file task at one time after aggregating multiple data flow file records.

[0028] Preferably, the method for hierarchical encryption of data flow files based on data importance specifically includes:

[0029] Obtain data flow files, use SALib library to quantitatively analyze the sensitivity of data flow files, and obtain data sensitivity coefficients;

[0030] Determine the value coefficient of data flow files based on hierarchical analysis method combined with random forest;

[0031] Obtain data sensitivity coefficient and value coefficient, define data importance function, and output data importance;

[0032] Define the security parameter λ of the data transfer file based on the data importance, attribute a i And the correlation group information A s , generate the unique encryption code H corresponding to the data transfer file uid , output the system public key PK and hierarchical master key FMK based on the hash encryption algorithm;

[0033]

[0034] Among them, H1 and H2 are hash operations and hash operations for verification, respectively. α and β are two random numbers generated by the hash algorithm. g and g α ,e(g,g) β They are bilinear groups and bilinear maps of prime order respectively;

[0035] Obtain the system public key PK and the hierarchical master key FMK, and encrypt the data flow files hierarchically based on the system public key PK and the hierarchical master key FMK.

[0036] Preferably, the value coefficient is determined by constructing a hierarchical model based on random forest local encoding and global hierarchical analysis. The hierarchical model consists of a target layer, a criterion layer, and a solution layer. The hierarchical model uses a random forest algorithm to train data value features to obtain a convergent hierarchical model.

[0037] The data importance is calculated by the following formula:

[0038]

[0039] Among them, λ represents the importance of data, λ1 and λ2 represent the data sensitivity coefficient and value coefficient respectively, and κ represents the parameter of the importance function.

[0040] Preferably, the method for constructing and training a flow scheduling model based on the priority feature, importance feature, and data feature of the encrypted data flow specifically includes:

[0041] The Pipel ine model is used as the initial model. The Pipel ine model consists of the Pipel ine layer, the Valve layer, and the Context layer. The architecture of the predefined initial model is the ResNet architecture. The sliding window mechanism of artificial intelligence is introduced into the initial model.

[0042] The instability of the sliding window is calculated using the Courbet-Léberle divergence and is determined by the entropy of the flow distribution graph.

[0043] Obtain an encrypted data stream, generate an encrypted data stream label based on a priority feature, an importance feature, and a data feature, and divide the encrypted data stream with the generated label into a training set and a test set;

[0044] The encrypted data stream is annotated by combining four sets of attribute features: priority feature, importance feature, data feature, and blockchain path load;

[0045] Select an appropriate loss function to measure the difference between the model-predicted flow distribution map and the actual flow distribution map, and select an optimizer to update the model parameters;

[0046] The initial model is iteratively trained using the training set until the initial model converges;

[0047] Get the test set, use the test set as input, execute the flow scheduling model, and determine whether the test result meets the preset accuracy threshold. If it meets the preset accuracy threshold, output the converged flow scheduling model.

[0048] Preferably, the flow scheduling model combines priority characteristics, importance characteristics, data characteristics, and blockchain path load to allocate the encrypted data flow, specifically including:

[0049] Load the encrypted data stream, take the encrypted data stream as input, execute the flow scheduling model, and the flow scheduling model determines the priority characteristics, importance characteristics, and data characteristics of the encrypted data stream;

[0050] The flow scheduling model is based on the Courbet-Leberle divergence to fuse the priority features, importance features, and data features to obtain the feature fusion value;

[0051] Determine the blockchain load based on the fitness function combined with the blockchain processing capacity, bandwidth utilization, and flow determination time;

[0052] Combine the feature fusion value and the blockchain path load to calculate the allocation priority value, generate the allocation priority sequence, and allocate the encrypted data flow based on the allocation priority sequence;

[0053] The allocation priority value is calculated by the following formula:

[0054]

[0055] Among them, Y i represents the allocation priority value, s is the number of encrypted data stream files, l represents the number of blockchain paths, and l i , are the current blockchain load and the average blockchain load, respectively, T represents the feature fusion value, and τ is the crossover probability coefficient of the fitness function.

[0056] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0057] In the embodiment of the present invention, a data filter is used to filter and process redundant information of data flow files, and the encrypted data flow is distributed through the flow scheduling model in combination with priority features, importance features, data features, and blockchain path load. The data filter and the flow scheduling model cooperate to achieve filtering redundancy of data flow files, optimize scheduling flow processing, reduce the storage burden of the system and network transmission pressure, thereby improving the efficiency of data processing, and ensuring that key tasks are executed first, further improving the response speed and throughput of the overall system. It overcomes the problem that the existing data flow method controls the master end to transfer data with the slave end according to the metadata configuration table and the metadata field configuration table and cannot filter out redundant data, increases the amount of data to be transmitted, and cannot dynamically adjust the data flow processing process according to the real-time load of the blockchain path and the priority level of the data flow, and is not adaptable enough to meet different business needs.

[0058] In the embodiment of the present invention, redundant information of data flow files is filtered and processed by a data filter, and the data filter is composed of a plurality of microfilters, and the microfilter includes an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit, thereby realizing operations such as filtering, converting, and aggregating real-time or batch data flow files to remove redundant, erroneous, and unnecessary data, thereby being able to perform quality control on the data flow files to ensure the accuracy, integrity, and reliability of the data.

[0059] In the embodiment of the present invention, the data importance function is defined through the data sensitivity coefficient and the value coefficient, the data importance is output, and the security parameter λ of the data flow file is defined based on the data importance, thereby overcoming the problem of limited system computing power and computing resources. The hierarchical encryption of data flow files can help users improve data flow efficiency and ensure the privacy of data flow files.

[0060] In the embodiment of the present invention, the flow scheduling model is trained by combining priority features, importance features, data features, and blockchain road load. At the same time, the flow scheduling model uses the Pipel ine model as the initial model, so that data flow can be prioritized according to the importance and urgency of the data to ensure that key tasks can obtain resources first, thereby improving the overall efficiency. It can also reasonably allocate network bandwidth and storage resources through analysis of data features, avoid resource waste, and improve the load capacity of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the implementation flow of the method for improving data flow effect based on artificial intelligence provided by the present invention.

[0062] Figure 2 The present invention shows a schematic diagram of the implementation process of a method for filtering redundant information of data flow files based on a data filter.

[0063] Figure 3 The present invention shows a schematic diagram of the implementation process of the method for hierarchical encryption of data flow files based on data importance.

[0064] Figure 4 A schematic diagram of the implementation process of a method for constructing and training a flow scheduling model based on encrypted data flow priority features, importance features, and data features is shown.

[0065] Figure 5 The figure shows a flow chart of the implementation process of the encrypted data flow allocation method by combining the flow scheduling model with priority characteristics, importance characteristics, data characteristics, and blockchain path load.

[0066] Figure 6 A structural diagram of a data flow effect improvement system based on artificial intelligence is shown. DETAILED DESCRIPTION

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0068] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] The existing data flow method controls the master end to transfer data with the slave end according to the metadata configuration table and the metadata field configuration table, but cannot filter out redundant data, which increases the amount of data that needs to be transmitted, and cannot dynamically adjust the data flow processing process according to the real-time load of the blockchain and the priority level of the data flow, and is not adaptable enough to meet different business needs. In response to the above problems, we propose a method for improving the data flow effect based on artificial intelligence. In short, when the method is implemented, it first responds to at least one set of data flow file receiving instructions pushed by the regional traffic node, receives at least one set of data flow files, filters and processes the redundant information of the data flow files based on the data filter, and after filtering, the data flow files are hierarchically encrypted based on the importance of the data, outputs at least one set of encrypted data streams, and writes the encrypted data streams into the flow scheduling model. The flow scheduling model combines priority features, importance features, data features, and blockchain load to allocate encrypted data streams, and finally verifies the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node. In the embodiment of the present invention, a data filter is used to filter and process redundant information of data flow files, and the encrypted data flow is distributed through the flow scheduling model in combination with priority features, importance features, data features, and blockchain path load. The data filter and the flow scheduling model cooperate to achieve filtering redundancy of data flow files, optimize scheduling flow processing, reduce the storage burden of the system and network transmission pressure, thereby improving the efficiency of data processing, and ensuring that key tasks are executed first, further improving the response speed and throughput of the overall system. It overcomes the problem that the existing data flow method controls the master end to transfer data with the slave end according to the metadata configuration table and the metadata field configuration table and cannot filter out redundant data, increases the amount of data to be transmitted, and cannot dynamically adjust the data flow processing process according to the real-time load of the blockchain path and the priority level of the data flow, and is not adaptable enough to meet different business needs.

[0070] The embodiment of the present invention provides a method for improving data flow effect based on artificial intelligence. Figure 1 The following is a schematic diagram of the implementation process of a method for improving data flow effects based on artificial intelligence, wherein the method for improving data flow effects based on artificial intelligence specifically includes:

[0071] Step S10, in response to at least one set of data flow file receiving instructions pushed by the regional traffic node, receiving at least one set of data flow files, and filtering and processing redundant information of the data flow files based on a data filter;

[0072] It should be noted that the data flow file includes but is not limited to data flow ID, data flow topic, data flow type, and business ID. The regional traffic node refers to the key node responsible for processing, forwarding and storing data in a specific area during the data flow process. Within the data center, the regional traffic node can be used to connect different server groups or racks to achieve efficient data flow within the data center. In a wide area network environment, the regional traffic node can be used as a data exchange point between different geographical locations to achieve cross-regional data flow.

[0073] Step S20, loading the data flow file after the redundant information filtering process, hierarchically encrypting the data flow file based on the importance of the data, and outputting at least one set of encrypted data streams;

[0074] Step S30, analyzing the encrypted data flow features, building and training a flow scheduling model based on the encrypted data flow priority features, importance features, and data features, and outputting a converged flow scheduling model;

[0075] Step S40, write the encrypted data stream into the flow scheduling model, the flow scheduling model combines the priority characteristics, importance characteristics, data characteristics, and blockchain path load to allocate the encrypted data stream flow, and outputs a flow allocation map that represents regional traffic nodes, encryption levels, blockchain paths, and target traffic nodes.

[0076] Step S50, verifying the identity and flow authority of the target flow node;

[0077] It should be noted that the identity of the target traffic node can be verified by JWT (JSON Web Token). JWT is an open standard based on JSON for transmitting statements between parties. It consists of three parts: Header, Payload, and Signature. The advantage of JWT is that it is self-contained, can carry user identity information and permission information, and is easy to pass between different services. The verification of flow permissions can be verified by API gateway. API gateway integrates existing authentication systems such as LDAP, OAuth, SAML, etc. to achieve unified authentication and single sign-on (SSO).

[0078] Step S60: If the identity and transfer authority of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node;

[0079] In step S70, if the identity and flow authority verification of the target flow node fails, a flow warning instruction is triggered. The flow scheduling model responds to the flow warning instruction and calls the backup flow node in real time based on the blockchain road load to execute the encrypted data flow secondary flow allocation task.

[0080] In this embodiment, after step S70 calls the backup traffic node in real time based on the blockchain road load to execute the secondary flow allocation task of the encrypted data flow, it is still necessary to verify the identity and flow authority of the backup traffic node. If it fails, in order to reduce the system data flow load, the tertiary flow allocation task will no longer be executed, triggering the traffic node abnormal instruction.

[0081] When the method is implemented, it first responds to at least one group of data flow file receiving instructions pushed by the regional traffic node, receives at least one group of data flow files, filters and processes redundant information of the data flow files based on the data filter, encrypts the data flow files in a hierarchical manner based on the importance of the data after filtering, outputs at least one group of encrypted data streams, writes the encrypted data streams into the flow scheduling model, and the flow scheduling model allocates the encrypted data stream flow in combination with priority features, importance features, data features, and blockchain path loads, and finally verifies the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node.

[0082] In the embodiment of the present invention, a data filter is used to filter and process redundant information of data flow files, and the encrypted data flow is distributed through the flow scheduling model in combination with priority features, importance features, data features, and blockchain path load. The data filter and the flow scheduling model cooperate to achieve filtering redundancy of data flow files, optimize scheduling flow processing, reduce the storage burden of the system and network transmission pressure, thereby improving the efficiency of data processing, and ensuring that key tasks are executed first, further improving the response speed and throughput of the overall system. It overcomes the problem that the existing data flow method controls the master end to transfer data with the slave end according to the metadata configuration table and the metadata field configuration table and cannot filter out redundant data, increases the amount of data to be transmitted, and cannot dynamically adjust the data flow processing process according to the real-time load of the blockchain path and the priority level of the data flow, and is not adaptable enough to meet different business needs.

[0083] The embodiment of the present invention provides a method for filtering redundant information of data flow files based on a data filter. Figure 2 The schematic diagram of the implementation process of the method for filtering redundant information of data flow files based on the data filter is shown. The method for filtering redundant information of data flow files based on the data filter specifically includes:

[0084] Step S101, pre-building a data filter, wherein the data filter is distributed and deployed in a regional traffic node on each end side;

[0085] In this embodiment, the consistent hashing algorithm can be used to distribute the data filters on the regional traffic nodes on each end side to ensure load balancing and fault tolerance. When nodes join or leave, the consistent hashing algorithm can reallocate the data filters to ensure high availability of the system.

[0086] Step S102, loading at least one set of data flow files, processing abnormal values ​​and missing values ​​of the data flow files, and obtaining standard data flow files;

[0087] It should be noted that the abnormal values ​​and missing values ​​in the data flow files can be processed by combining the marking method with the square root transformation.

[0088] Step S103, defining a preprocessing function of a data filter to improve the Flink block, preprocessing the data flow file based on the preprocessing function, wherein the preprocessing function is a variance probability distribution function of a time window, and the preprocessing function receives a data flow file with a key, and each time window corresponds to a key;

[0089] The preprocessing function is expressed as:

[0090]

[0091] Among them, E k,t It represents the variance probability output after the preprocessing function preprocesses the data flow file, w k represents the time window size, i is the number of data types in the data flow file, t represents the time of the data flow file preprocessing after the time window, x(t) represents the input representation of the data flow file, maxx(t) and minx(t) represent the maximum and minimum values ​​of the data flow file within the time window period respectively;

[0092] Step S104, introducing a redundant consistency constraint into the variance probability output representation after preprocessing the data flow file according to the preprocessing function, defining a filtering function based on the redundant consistency constraint, and outputting the data flow file after redundant information filtering processing;

[0093] The redundant consistency constraint is expressed as:

[0094]

[0095] w k =m×n(3)

[0096] The filter function is expressed as:

[0097]

[0098] Among them, w kIndicates the time window size, m and n are the height and width of the time window respectively. is the mean value of the data flow file within the time window period, and f(x) is the output representation of the data flow file after filtering by the filter function;

[0099] Step S105, filter out redundant information of the data flow file, annotate the redundant information with a label, merge the label with the redundant information, upload the backup through the data cache layer, and regularly verify the integrity and recoverability of the backup.

[0100] In this embodiment, the data filter filters the data flow files uploaded by the regional traffic node according to the preset filtering frequency, and uploads the data flow files after packaging and compression. The data filter is composed of multiple micro filters, which provide upstream filtering services and downstream filtering services. The micro filter includes an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit. The Flink block communicates with the data aggregation unit, the data cache layer, and the backup upload unit through an abstract interface. The abstract interface includes but is not limited to a List interface, a Map interface, a Comparable interface, a Runnable interface, and an AutoCloseable interface. The improved Flink block realizes the optimization control of task parallelism by configuring parallelism parameters to the Flink block, and writes the data flow file task at one time after aggregating multiple data flow file records. At the same time, the Flink block can also be improved by using container technologies such as Docker, and the Flink job and its dependencies can be packaged into standardized units, which is convenient for consistent deployment and operation in different environments, so that the optimized Flink block supports multiple deployment modes (such as independent mode, cluster mode, etc.), and the most appropriate deployment mode can be selected according to actual needs. For example, in the development phase, you can choose local mode for rapid iteration; in the production environment, you should choose cluster mode to ensure high availability and scalability. In addition, you can also use Flink's REST API to dynamically submit and manage jobs, further improving operation and maintenance efficiency.

[0101] In the embodiment of the present invention, redundant information of data flow files is filtered and processed by a data filter, and the data filter is composed of a plurality of microfilters, and the microfilter includes an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit, thereby realizing operations such as filtering, converting, and aggregating real-time or batch data flow files to remove redundant, erroneous, and unnecessary data, thereby being able to perform quality control on the data flow files to ensure the accuracy, integrity, and reliability of the data.

[0102] The embodiment of the present invention provides a method for hierarchical encryption of data flow files based on data importance. Figure 3 The following is a schematic diagram of the implementation process of a method for hierarchical encryption of data flow files based on data importance. The method for hierarchical encryption of data flow files based on data importance specifically includes:

[0103] Step S201, obtaining a data flow file, using the SALib library to perform quantitative analysis on the sensitivity of the data flow file, and obtaining a data sensitivity coefficient;

[0104] In the embodiment of the present invention, the sensitivity index is calculated based on the model output using the analysis function provided by the SALib library (such as Sobol analysis). These indices can help understand the influence of each input parameter on the model output.

[0105] Step S202, determining the value coefficient of the data flow file based on the hierarchical analysis method combined with random forest;

[0106] It should be noted that the value coefficient is determined by constructing a hierarchical model based on random forest local encoding and global hierarchical analysis. The hierarchical model consists of a target layer, a criterion layer, and a solution layer. The hierarchical model uses a random forest algorithm to train the data value features to obtain a converged hierarchical model. When collecting data for hierarchical model training, feature data related to the value of the data flow, such as data quality indicators, usage frequency, user feedback, etc., are collected, and then the collected feature data is trained using a random forest algorithm to obtain a model that can predict the value of the data flow.

[0107] In this embodiment, the criterion layer is used to construct a judgment matrix, and the importance of the data flow file is determined in combination with the expert consultation method. The criteria are compared and scored pairwise based on the importance, and then the geometric mean method in the hierarchical analysis method is used to calculate the weights of the criteria. The criteria include but are not limited to data quality, data volume, data timeliness, and data security.

[0108] Step S203, obtaining data sensitivity coefficient and value coefficient, defining data importance function, and outputting data importance;

[0109] In this embodiment, the data importance is calculated by the following formula:

[0110]

[0111] Among them, λ represents the importance of data, λ1 and λ2 represent the data sensitivity coefficient and value coefficient respectively, and κ represents the parameter of the importance function. In this embodiment, the importance function can be a Gumbel-Copula function, and the parameter of the Gumbel-Copula function is 0.1-0.5.

[0112] Step S204: define the security parameter λ of the data transfer file based on the data importance, attribute a i And the correlation group information A s , generate the unique encryption code H corresponding to the data transfer file uid , output the system public key PK and hierarchical master key FMK based on the hash encryption algorithm;

[0113]

[0114] Among them, H1 and H2 are hash operations and hash operations for verification, respectively. α and β are two random numbers generated by the hash algorithm. g and g α ,e(g,g) β They are bilinear groups and bilinear maps of prime order respectively;

[0115] Step S205, obtain the system public key PK and the hierarchical master key FMK, hierarchically encrypt the data flow file based on the system public key PK and the hierarchical master key FMK, and the hierarchical decryption of the data flow file can be performed by EDSP according to the Lagrange interpolation theorem.

[0116] In the embodiment of the present invention, the data importance function is defined through the data sensitivity coefficient and the value coefficient, the data importance is output, and the security parameter λ of the data flow file is defined based on the data importance, thereby overcoming the problem of limited system computing power and computing resources. The hierarchical encryption of data flow files can help users improve data flow efficiency and ensure the privacy of data flow files.

[0117] The embodiment of the present invention provides a method for constructing and training a flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of encrypted data flows. Figure 4 The schematic diagram of the implementation process of the method for constructing and training the flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of the encrypted data flow is shown. The method for constructing and training the flow scheduling model based on the priority characteristics, importance characteristics, and data characteristics of the encrypted data flow specifically includes:

[0118] Step S301, taking the Pipel ine model as the initial model, the Pipel ine model consists of a Pipel ine layer, a Valve layer, and a Context layer, the architecture of the predefined initial model is a ResNet architecture, and an artificial intelligence sliding window mechanism is introduced into the initial model;

[0119] Step S302, using Courbet-Labler divergence to calculate the instability of the sliding window, and determining the instability of the sliding window through the entropy of the flow distribution graph;

[0120] Step S303, obtaining an encrypted data stream, generating an encrypted data stream label based on the priority feature, the importance feature, and the data feature, and dividing the encrypted data stream with the generated label into a training set and a test set;

[0121] Step S304, annotating the encrypted data stream by combining four sets of attribute features, namely, priority feature, importance feature, data feature, and blockchain path load;

[0122] Step S305, selecting an appropriate loss function to measure the difference between the model-predicted flow distribution graph and the actual flow distribution graph, and selecting an optimizer to update the model parameters;

[0123] Step S306, iteratively training the initial model using the training set until the initial model converges;

[0124] Step S307, obtaining a test set, taking the test set as input, and executing the flow scheduling model;

[0125] Step S308, determining whether the test result meets a preset accuracy threshold;

[0126] Step S309: if the preset accuracy threshold is met, a converged flow scheduling model is output.

[0127] If it does not meet the preset accuracy threshold, return to step S30 to continue model iterative training.

[0128] In this embodiment, the preset accuracy threshold is set to 0.9-0.95, the ratio of the training set to the test set is 4:1, the training rounds of the model are set to 200-400, the flow scheduling model learning rate is set to 0.0002, the loss function of the flow scheduling model is binary cross entropy loss (BCELoss), and the optimizer can be an RMSprop optimizer.

[0129] In the embodiment of the present invention, the flow scheduling model is trained by combining priority features, importance features, data features, and blockchain road load. At the same time, the flow scheduling model uses the Pipel ine model as the initial model, so that data flow can be prioritized according to the importance and urgency of the data to ensure that key tasks can obtain resources first, thereby improving the overall efficiency. It can also reasonably allocate network bandwidth and storage resources through analysis of data features, avoid resource waste, and improve the load capacity of the system.

[0130] The embodiment of the present invention provides a method for allocating encrypted data flow by combining a flow scheduling model with priority characteristics, importance characteristics, data characteristics, and blockchain path load. Figure 5The flow scheduling model combines the priority feature, importance feature, data feature, and blockchain road load to implement the method for allocating the encrypted data flow. The method for allocating the encrypted data flow by combining the priority feature, importance feature, data feature, and blockchain road load is shown, and specifically includes:

[0131] Step S401, loading the encrypted data stream, taking the encrypted data stream as input, executing the flow scheduling model, and the flow scheduling model determines the priority characteristics, importance characteristics, and data characteristics of the encrypted data stream;

[0132] It should be noted that data characteristics include but are not limited to data type, data format, data source, data quality, and data correlation. The flow scheduling model determines the priority characteristics and importance characteristics of the encrypted data flow. The data characteristics can use correlation analysis, hypothesis testing and other methods to evaluate the importance and effectiveness of each feature.

[0133] Step S402, the flow scheduling model calculates the fusion of priority features, importance features, and data features based on Courbet-Leberle divergence to obtain a feature fusion value;

[0134] In this embodiment, the Kullback-Leibler Divergence (KL divergence) is a method for measuring the difference between two probability distributions. In feature fusion, we can use KL divergence to measure the relative importance of different features, so as to obtain a comprehensive feature fusion value. When calculating the feature fusion value, firstly approximate the probability distribution of the feature based on the kernel density estimation, then calculate the KL divergence between the priority feature, the importance feature, and the data feature, then assign a weight to each feature according to the size of the KL divergence, and then calculate the weighted average, and use the weighted average as the feature fusion value.

[0135] Step S403, determining the blockchain load based on the fitness function combined with the blockchain processing capacity, bandwidth usage, and flow determination time.

[0136] Step S404, combining the feature fusion value and the blockchain path load to calculate the allocation priority value, generate an allocation priority sequence, and allocate the encrypted data flow based on the allocation priority sequence;

[0137] The allocation priority value is calculated by the following formula:

[0138]

[0139] Among them, Y i represents the allocation priority value, s is the number of encrypted data stream files, l represents the number of blockchain paths, and l i , are the current blockchain load and the average blockchain load, respectively, T represents the feature fusion value, and τ is the crossover probability coefficient of the fitness function. In this embodiment, the crossover probability coefficient may be 0.1-0.3.

[0140] The embodiment of the present invention provides a data flow effect improvement system based on artificial intelligence, Figure 6 The structure diagram of the data flow effect improvement system based on artificial intelligence is shown, and the data flow effect improvement system based on artificial intelligence specifically includes:

[0141] The information filtering module 100 receives at least one set of data flow transfer files in response to at least one set of data flow transfer file receiving instructions pushed by the regional traffic node, wherein the data flow transfer files include data flow ID, data flow topic, data flow type, and business ID, and filters and processes redundant information of the data flow transfer files based on the data filter;

[0142] The data stream encryption module 200 is used to load the data flow transfer file after the redundant information filtering process, hierarchically encrypt the data flow transfer file based on the importance of the data, and output at least one set of encrypted data streams;

[0143] The model building module 300 is used to analyze the encrypted data flow characteristics, build and train the flow scheduling model based on the encrypted data flow priority characteristics, importance characteristics, and data characteristics, and output a converged flow scheduling model;

[0144] The flow allocation module 400 is used to write the encrypted data flow into the flow scheduling model. The flow scheduling model allocates the encrypted data flow according to the priority characteristics, importance characteristics, data characteristics, and blockchain path load, and outputs a flow allocation diagram representing the regional traffic nodes, encryption levels, blockchain paths, and target traffic nodes.

[0145] The authority verification module 500 is used to verify the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, the encrypted data flow is pushed to the target traffic node;

[0146] The secondary distribution module 600 triggers a flow warning instruction if the identity and flow authority verification of the target traffic node fails. The flow scheduling model responds to the flow warning instruction and calls the backup link in real time based on the blockchain road load to execute the encrypted data flow secondary flow distribution task.

[0147] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, which stores computer program instructions, and the computer program instructions can be executed by a processor. When the computer program instructions are executed, the method of any one of the above embodiments is implemented.

[0148] The computer-readable storage medium (e.g., memory) herein can be a volatile memory or a nonvolatile memory, or can include both a volatile memory and a nonvolatile memory. As an example and not by way of limitation, a nonvolatile memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. A volatile memory can include a random access memory (RAM), which can act as an external cache memory. As an example and not by way of limitation, RAM can be obtained in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage device of the disclosed aspects is intended to include, but is not limited to, these and other suitable types of memories.

[0149] In summary, the present invention provides a method for improving data flow effect based on artificial intelligence. When the method is implemented, it first responds to at least one group of data flow file receiving instructions pushed by the regional traffic node, receives at least one group of data flow files, filters and processes the redundant information of the data flow files based on the data filter, and encrypts the data flow files in a hierarchical manner based on the importance of the data after filtering, outputs at least one group of encrypted data streams, writes the encrypted data streams into the flow scheduling model, and the flow scheduling model allocates the encrypted data stream flow in combination with priority features, importance features, data features, and blockchain path loads, and finally verifies the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, the encrypted data stream is pushed to the target traffic node.

[0150] In the embodiment of the present invention, a data filter is used to filter and process redundant information of data flow files, and the encrypted data flow is distributed through the flow scheduling model in combination with priority features, importance features, data features, and blockchain path load. The data filter and the flow scheduling model cooperate to achieve filtering redundancy of data flow files, optimize scheduling flow processing, reduce the storage burden of the system and network transmission pressure, thereby improving the efficiency of data processing, and ensuring that key tasks are executed first, further improving the response speed and throughput of the overall system. It overcomes the problem that the existing data flow method controls the master end to transfer data with the slave end according to the metadata configuration table and the metadata field configuration table and cannot filter out redundant data, increases the amount of data to be transmitted, and cannot dynamically adjust the data flow processing process according to the real-time load of the blockchain path and the priority level of the data flow, and is not adaptable enough to meet different business needs.

[0151] It should be noted that, for the above-mentioned embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0152] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field can still combine, add, delete or make other adjustments to the features in the various embodiments of the present invention according to the circumstances without conflict, without making creative work, so as to obtain different other technical solutions that do not deviate from the concept of the present invention in essence, and these technical solutions also belong to the scope of protection of the present invention.

Claims

1. A method for improving data flow effect based on artificial intelligence, characterized in that: The method for improving data flow effect based on artificial intelligence specifically includes: In response to at least one set of data flow transfer file receiving instructions pushed by the regional traffic node, at least one set of data flow transfer files is received, wherein the data flow transfer files include a data flow ID, a data flow topic, a data flow type, and a service ID, and redundant information of the data flow transfer files is filtered based on a data filter; Loading the data flow file after redundant information filtering, encrypting the data flow file in a hierarchical manner based on the importance of the data, and outputting at least one set of encrypted data streams; Analyze the encrypted data flow characteristics, build and train the flow scheduling model based on the encrypted data flow priority characteristics, importance characteristics, and data characteristics, and output a converged flow scheduling model; The encrypted data stream is written into the flow scheduling model. The flow scheduling model allocates the encrypted data stream flow based on priority features, importance features, data features, and blockchain path load, and outputs a flow allocation map that represents regional traffic nodes, encryption levels, blockchain paths, and target traffic nodes.

2. The method for improving data flow effect based on artificial intelligence according to claim 1, characterized in that: The method for improving data flow effect based on artificial intelligence specifically includes: Verify the identity and flow authority of the target traffic node. If the identity and flow authority of the target traffic node are verified, push the encrypted data stream to the target traffic node; If the identity and flow authority verification of the target traffic node fails, the flow warning instruction is triggered. The flow scheduling model responds to the flow warning instruction and calls the backup traffic node in real time based on the blockchain road load to execute the encrypted data flow secondary flow allocation task.

3. The method for improving data flow effect based on artificial intelligence as claimed in claim 1, characterized in that: The method for filtering redundant information of data flow files based on a data filter specifically includes: Pre-build data filters, where the data filters are distributed and deployed on regional traffic nodes on each end side; Load at least one set of data flow files, process abnormal values ​​and missing values ​​in the data flow files, and obtain standard data flow files; Define a data filter to improve the preprocessing function of the Flink block, preprocess the data flow file based on the preprocessing function, the preprocessing function is a variance probability distribution function of the time window, the preprocessing function receives the data flow file with a key, and each time window corresponds to a key; The preprocessing function is expressed as: Among them, E k,t It represents the variance probability output after the preprocessing function preprocesses the data flow file, w k represents the time window size, i is the number of data types in the data flow file, t represents the time of the data flow file preprocessing after the time window, x(t) represents the input representation of the data flow file, maxx(t) and minx(t) represent the maximum and minimum values ​​of the data flow file within the time window period respectively; According to the preprocessing function, the variance probability output representation of the data flow file after preprocessing introduces a redundant consistency constraint, defines a filtering function based on the redundant consistency constraint, and outputs the data flow file after redundant information filtering; The redundant consistency constraint is expressed as: In k =m×n(3) The filter function is expressed as: Among them, w k Indicates the time window size, m and n are the height and width of the time window respectively. is the mean value of the data flow file within the time window period, and f(x) is the output representation of the data flow file after filtering by the filter function; Filter out redundant information in data flow files, annotate labels for redundant information, merge labels with redundant information, upload backups through the data cache layer, and regularly verify the integrity and recoverability of backups.

4. The method for improving data flow effect based on artificial intelligence as claimed in claim 3, characterized in that: The data filter filters the data flow files uploaded by the regional traffic nodes according to a preset filtering frequency, and uploads the data flow files after packaging and compression. The data filter is composed of multiple microfilters. The microfilters provide upstream filtering services and downstream filtering services. The microfilters include an improved Flink block, a data aggregation unit, a data cache layer, and a backup upload unit. The Flink block communicates with the data aggregation unit, the data cache layer, and the backup upload unit through an abstract interface. The improved Flink block optimizes the control of task parallelism by configuring the parallelism parameter to the Flink block, and writes the data flow file task at one time after aggregating multiple data flow file records.

5. The method for improving data flow effect based on artificial intelligence according to claim 1, characterized in that: The method for hierarchical encryption of data flow files based on data importance specifically includes: Obtain data flow files, use SALib library to quantitatively analyze the sensitivity of data flow files, and obtain data sensitivity coefficients; Determine the value coefficient of data flow files based on hierarchical analysis method combined with random forest; Obtain data sensitivity coefficient and value coefficient, define data importance function, and output data importance; Define the security parameter λ of the data transfer file based on the data importance, attribute a i And the correlation group information A s , generate the unique encryption code H corresponding to the data transfer file uid , output the system public key PK and hierarchical master key FMK based on the hash encryption algorithm; Among them, H1 and H2 are hash operations and hash operations for verification, respectively. α and β are two random numbers generated by the hash algorithm. g and g α ,e(g,g) β They are bilinear groups and bilinear maps of prime order respectively; Obtain the system public key PK and the hierarchical master key FMK, and encrypt the data flow files hierarchically based on the system public key PK and the hierarchical master key FMK.

6. The method for improving data flow effect based on artificial intelligence as claimed in claim 5, characterized in that: The value coefficient is determined based on the hierarchical model constructed by random forest local encoding and global hierarchical analysis. The hierarchical model consists of a target layer, a criterion layer, and a solution layer. The hierarchical model uses a random forest algorithm to train the data value features to obtain a converged hierarchical model.

7. The method for improving data flow effect based on artificial intelligence according to claim 6, characterized in that: The data importance is calculated by the following formula: Among them, λ represents the importance of data, λ1 and λ2 represent the data sensitivity coefficient and value coefficient respectively, and κ represents the parameter of the importance function.

8. The method for improving data flow effect based on artificial intelligence according to any one of claims 2 to 7, characterized in that: The method for constructing and training a flow scheduling model based on the priority features, importance features, and data features of encrypted data flows specifically includes: The Pipeline model is used as the initial model. The Pipeline model consists of the Pipeline layer, the Valve layer, and the Context layer. The architecture of the predefined initial model is the ResNet architecture. The sliding window mechanism of artificial intelligence is introduced into the initial model. The instability of the sliding window is calculated using the Courbet-Léberle divergence and is determined by the entropy of the flow distribution graph. Obtain an encrypted data stream, generate an encrypted data stream label based on a priority feature, an importance feature, and a data feature, and divide the encrypted data stream with the generated label into a training set and a test set; The encrypted data stream is annotated by combining four sets of attribute features: priority feature, importance feature, data feature, and blockchain path load; Select an appropriate loss function to measure the difference between the model-predicted flow distribution map and the actual flow distribution map, and select an optimizer to update the model parameters; The initial model is iteratively trained using the training set until the initial model converges; Get the test set, use the test set as input, execute the flow scheduling model, and determine whether the test result meets the preset accuracy threshold. If it meets the preset accuracy threshold, output the converged flow scheduling model.

9. The method for improving data flow effect based on artificial intelligence as claimed in claim 8, characterized in that: The flow scheduling model combines priority characteristics, importance characteristics, data characteristics, and blockchain path load to allocate the encrypted data flow, specifically including: Load the encrypted data stream, take the encrypted data stream as input, execute the flow scheduling model, and the flow scheduling model determines the priority characteristics, importance characteristics, and data characteristics of the encrypted data stream; The flow scheduling model is based on the Courbet-Leberle divergence to fuse the priority features, importance features, and data features to obtain the feature fusion value; The blockchain load is determined based on the fitness function combined with the blockchain processing capacity, bandwidth utilization, and flow determination time.

10. The method for improving data flow effect based on artificial intelligence according to claim 9, characterized in that: The flow scheduling model combines priority characteristics, importance characteristics, data characteristics, and blockchain path load to allocate the encrypted data flow, and specifically includes: Combine the feature fusion value and the blockchain path load to calculate the allocation priority value, generate the allocation priority sequence, and allocate the encrypted data flow based on the allocation priority sequence; The allocation priority value is calculated by the following formula: Among them, Y i represents the allocation priority value, s is the number of encrypted data stream files, l represents the number of blockchain paths, and l i , are the current blockchain load and the average blockchain load, respectively, T represents the feature fusion value, and τ is the crossover probability coefficient of the fitness function.

Images