A multi-source data acquisition method and system based on adaptive interface

Abstract

The application provides a multi-source data acquisition method and system based on an adaptive interface, relates to the field of water conservancy industry data resource acquisition, and solves the technical problem of low water conservancy data acquisition efficiency caused by the dependence of the prior art on manual configuration of an interface during water conservancy data acquisition. The method comprises the following steps: taking a water conservancy data source of a known protocol type as a first data source; constructing a protocol identification model based on a convolutional neural network; training the protocol identification model through a preset loss function; obtaining the protocol type of the first data source through the protocol identification model; extracting key feature information of the protocol type through principal component analysis technology; constructing a decision tree based on the key feature information to generate corresponding adaptation rules; accessing a second data source; presetting a similarity threshold, calculating the similarity of the second data source and the adaptation rules; and if the similarity is not less than the preset similarity threshold, marking the second data source and the first data source as the same type of data source. The application is used in the process of multi-source data integration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of water conservancy industry data resource collection, and in particular relates to a multi-source data collection method and system based on adaptive interface. BACKGROUND

[0002] In the water conservancy industry, data sources are extensive and complex, different data sources have different interface types and communication protocols; for traditional data collection methods, data interface and protocol conversion rules are configured manually; for each new sensor, it takes 2-3 days to complete parameter research, program writing and testing, for hundreds of sensor types in the water conservancy industry, the cumulative working hours are extremely large, and when the sensor protocol or parameter changes, the program needs to be modified again, which may cause the system to shut down;

[0003] The prior art relies on manual configuration of interfaces and protocol conversion rules in the water conservancy data collection link, for extensive and complex water conservancy data sources, manual operation not only consumes time and effort, but also requires rewriting of adaptation code every time a new data source is connected, resulting in the technical problems of low water conservancy data collection efficiency and difficult adaptation. SUMMARY

[0004] The present application provides a multi-source data collection method and system based on adaptive interface, which solves the technical problems of low water conservancy data collection efficiency and difficult adaptation caused by the prior art relying on manual configuration of interfaces and protocol conversion rules in the water conservancy data collection link, for extensive and complex water conservancy data sources, manual operation not only consumes time and effort, but also requires rewriting of adaptation code every time a new data source is connected.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, a multi-source data collection method based on adaptive interface is provided, comprising: obtaining a plurality of water conservancy data sources of known protocol types, and taking the plurality of water conservancy data sources as first data sources;

[0007] A convolutional neural network is called, a protocol recognition model is constructed based on the convolutional neural network, a preset loss function is trained, the protocol recognition model is trained through the preset loss function, the protocol type of the first data source is obtained through the trained protocol recognition model, the key feature information of the protocol type is extracted through principal component analysis technology, and the corresponding adaptation rules are generated based on the key feature information, and the plurality of adaptation rules are stored in the form of rule feature vectors to a rule library;

[0008] Accessing a second data source; presetting a similarity threshold, calculating the similarity between the second data source and the adaptation rule; if the similarity is not less than the preset similarity threshold, marking the second data source and the first data source as the same type of data source, and setting the same type of data source to use the same adaptation rule; wherein the second data source is a water conservancy data source to be matched with the adaptation rule.

[0009] In a possible implementation of the above first aspect, the protocol recognition model comprises: an input layer, a convolution layer, a pooling layer, and a full connection layer.

[0010] The input layer is configured to call the first data source and integrate the first data source into a data sequence X= ; wherein, Xi represents the i-th element in the data sequence, and L is the length of the data sequence.

[0011] The convolution layer is configured to preset the j-th convolution kernel as , and the size of each convolution kernel is m; wherein, is a positive integer.

[0012] The convolution result of the j-th convolution kernel at position k is obtained through the formula ; wherein, is the bias term corresponding to the j-th convolution kernel. Xi represents the weight value of the i-th convolution result in the j-th convolution kernel. Through the convolution operation, a plurality of feature mapping sets

[0013] are obtained.

[0014] The pooling layer is configured to set the length of the pooling window as s, and the result of the j-th feature mapping after the pooling at position k is obtained through the formula ; wherein, is the global position of the i-th element in the k-th window.

[0015] Through the convolution operation, a plurality of feature vector sets are obtained.

[0016] The full connection layer is configured to connect the plurality of feature vector sets output by the pooling layer, and map the feature vector sets to a protocol category space through the full connection layer to obtain a protocol category probability distribution vector.

[0017] It should be noted that each convolution result corresponds to an element on the feature mapping set .

[0018] ​​In conjunction with the first aspect above, in one possible implementation, the step of mapping the feature vector set to the protocol category space through a fully connected layer to obtain the protocol category probability distribution vector includes:

[0019] Let W be the weight matrix of the fully connected layer and b be the bias vector;

[0020] The output formula for a fully connected layer is: ;in, The feature vector output by the pooling layer. is the preset activation function, o is the protocol category probability distribution vector; and each element in the protocol category probability distribution vector corresponds to the probability of a protocol type.

[0021] In conjunction with the first aspect above, in one possible implementation, the method for obtaining the preset loss function includes:

[0022] Obtain the first historical data source as training set samples ,in, For the nth historical first data source sample, This represents the corresponding protocol category label; n is a positive integer;

[0023] Through formula The loss function is calculated. Where N is the total number of data samples, and C is the total number of protocol categories. This represents the true label of the nth historical training set sample on the c-th protocol. The protocol identification model predicts the probability that the nth sample index belongs to the cth protocol class; where the true labels include 0 and 1.

[0024] It should be noted that the loss function is used to calculate the error between the protocol identification model's prediction and the true label; protocol category label. It's a general term, referring to the actual labels. It is the encoding form of the protocol category label; the true label is 1 when the samples in the training set belong to the same protocol category, and 0 when they do not belong to the same protocol category.

[0025] In conjunction with the first aspect above, in one possible implementation, the step of training the protocol recognition model through a preset loss function includes:

[0026] Formula based on backpropagation algorithm The weight matrix of the fully connected layer after the protocol recognition model is updated is calculated. ;in, The learning rate preset for the protocol recognition model. This represents the weight matrix of the fully connected layer before the update. This is the feature vector output after the nth sample is processed by the pooling layer. express Transpose of;

[0027] Invoke the preset loss function and obtain the updated loss function based on the weight matrix of the fully connected layer;

[0028] When the updated loss function is minimized, the protocol recognition model is marked as having completed training.

[0029] In conjunction with the first aspect above, in one possible implementation, obtaining the protocol type of the first data source through the trained protocol identification model includes:

[0030] The first data source is input into the trained protocol recognition model. The model outputs a protocol category probability distribution vector, and the category with the highest probability value in the vector is selected as the identified protocol type. ,in, , This represents the probability of the c-th type of protocol during the inference process of the protocol identification model.

[0031] It should be noted that, because the model focuses on the single input sample during inference, the sample index n is omitted in single-sample inference scenarios. The essence is .

[0032] In conjunction with the first aspect above, in one possible implementation, the extraction of key feature information of the protocol type using principal component analysis includes:

[0033] Principal component analysis was used to analyze the data format, interface parameters, and data transmission frequency of the first data source to obtain several features;

[0034] The matrix form of the first data source is set as follows: Where m is the number of the first data sources and n is the number of features; standardization of the first data sources yields a mean of 0 and a variance of 1 for each feature; ] represents a matrix containing the b-th feature of the a-th sample;

[0035] Through formula The covariance matrix of the standardized data was calculated. ;in, It is a standardized data matrix;

[0036] Perform eigenvalue decomposition on the covariance matrix Σ to obtain the eigenvalues. eigenvectors corresponding to eigenvalues ;in, ;

[0037] According to a preset principal component contribution rate threshold, selecting a feature vector corresponding to a first k characteristic value As a principal component; projecting the first data source onto the principal component to perform a dimension reduction operation to obtain a feature vector after dimension reduction , and marking the feature vector as key feature information.

[0038] In combination with the above first aspect, in a possible implementation manner, the constructing a decision tree based on the key feature information to generate a corresponding adaptation rule comprises:

[0039] S110: For any feature A, marking the value of the feature A as , and dividing the training data set D into n subsets according to the feature A;

[0040] S120: Assuming that the training data set of the first data source is D, containing n types of adaptation rule types, and the number of samples of the i-th type is , the information entropy is calculated by the formula ;

[0041] The information gain of the feature A to D is calculated by the formula ;

[0042] S130: The information gain ratio is calculated by the formula

[0043] The decision tree algorithm is called, and the feature with the largest information gain ratio is selected as the split attribute of the current node;

[0044] S140: Starting from the root node of the decision tree, splitting with the feature with the largest information gain ratio to form a child node, and repeating steps S110-S130 for each child node until all samples belong to the same type of adaptation rule, marking the end of the repetition process, and marking the leaf node of the decision tree at this time as the corresponding adaptation rule.

[0045] In combination with the above first aspect, in a possible implementation manner, the calculating the similarity between the second data source and the adaptation rule comprises:

[0046] Assuming that the rule feature vector of the first data source is , and the rule feature vector of the second data source is ;

[0047] The similarity is calculated by the formula .

[0048] ​​In a second aspect, the application provides a multi-source data acquisition system based on an adaptive interface, comprising a communication unit and a processing unit.

[0049] The communication unit is configured to acquire a plurality of water conservancy data sources of known protocol types, take the plurality of water conservancy data sources as first data sources, and access second data sources.

[0050] The processing unit is configured to call a convolutional neural network, construct a protocol identification model based on the convolutional neural network, preset a loss function, train the protocol identification model through the preset loss function, acquire the protocol types of the first data sources through the trained protocol identification model, extract key feature information of the protocol types through principal component analysis technology, construct a decision tree based on the key feature information to generate corresponding adaptation rules, store the adaptation rules in the form of rule feature vectors in a rule library, preset a similarity threshold, calculate the similarity between the second data sources and the adaptation rules, if the similarity is not less than the preset similarity threshold, mark the second data sources and the first data sources as the same type of data sources, and set the same type of data sources to use the same adaptation rule, and wherein the second data sources are water conservancy data sources to be matched with the adaptation rules.

[0051] The application provides a multi-source data acquisition method and system based on an adaptive interface, which can quickly and accurately identify various communication protocols and interface types through deep learning and machine learning algorithms. When encountering new data sources, whether using a new protocol or having special interface characteristics, the system can generate corresponding adaptation rules through protocol identification and feature analysis to effectively acquire new data sources, solve the problem of difficulty in adapting to new data sources in traditional technologies, and have strong scalability. The application reduces the technical threshold and operation difficulty: traditional data acquisition methods require high professional knowledge of technical personnel and need to be familiar with various protocols and interface configurations. The automatic process of the application makes the data acquisition process unnecessary to have complex manual operation, reduces the technical requirements for operators, reduces operation errors caused by human factors, and makes data acquisition work easier to carry out.

[0052] It should be understood that the descriptions of technical features, technical solutions, beneficial effects or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Instead, it can be understood that the description of a feature or beneficial effect means that the specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and beneficial effects described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A system architecture diagram of a multi-source data acquisition system based on an adaptive interface provided by an embodiment of the present application;

[0054] Figure 2 A flowchart of a multi-source data acquisition method based on an adaptive interface provided by an embodiment of the present application;

[0055] Figure 3 A flowchart of another multi-source data acquisition method based on an adaptive interface provided by an embodiment of the present application; DETAILED DESCRIPTION

[0056] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this document is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean: A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0057] It should be noted that in this application, the words "exemplary" or "for example" are used to mean serving as an example, instance or illustration. Any embodiment or design solution described as "exemplary" or "for example" in this application should not be interpreted as more preferred or more advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0058] To solve the technical problem that the prior art relies on manual configuration of interfaces when collecting water conservancy data, thereby causing low efficiency of water conservancy data collection, embodiments of the present application provide a multi-source data collection method based on adaptive interfaces, as shown in Figure 1 The method comprises the following steps.

[0059] Obtain a plurality of water conservancy data sources of known protocol types, and use the plurality of water conservancy data sources as first data sources.

[0060] Call a convolutional neural network, construct a protocol identification model based on the convolutional neural network, preset a loss function, train the protocol identification model through the preset loss function, obtain the protocol types of the first data sources through the trained protocol identification model, extract key feature information of the protocol types through principal component analysis technology, construct a decision tree based on the key feature information to generate corresponding adaptation rules, and store the adaptation rules in the form of rule feature vectors in a rule library.

[0061] Access a second data source, preset a similarity threshold, calculate the similarity between the second data source and the adaptation rules, if the similarity is not less than the preset similarity threshold, mark the second data source and the first data source as the same type of data source, and set the same type of data source to use the same adaptation rule, wherein the second data source is a water conservancy data source to be matched with the adaptation rules. Based on this, the present application realizes the automation and intelligentization of water conservancy multi-source data collection, and eliminates the steps of manual configuration of interfaces and protocol conversion rules in the traditional way. When a new data source is accessed, there is no need to re-write adaptation code, and the system automatically completes protocol identification, rule generation and other operations, greatly shortening the preparation time of data collection. For the same type of data source, the adaptation rules in the rule library can be directly called to quickly complete the collection, significantly improving the data collection efficiency and reducing the labor cost and time cost.

[0062] As shown in Figure 2 The multi-source data collection method based on adaptive interfaces provided by the embodiments of the present application comprises the following steps.

[0063] Extract the key feature information of the protocol types through principal component analysis technology.

[0064] In some implementations, the principal component analysis technology is used to analyze the data format, interface parameters and data transmission frequency of the first data sources to obtain a plurality of features.

[0065] The matrix form of the first data sources is set as wherein m is the number of the first data sources, and n is the number of features; the first data sources are standardized to obtain a mean value of 0 and a variance of 1 for each feature; represents a matrix containing the bth feature of the ath sample.

[0066] The formula is Computing the covariance matrix of the normalized data ; wherein, is the normalized data matrix;

[0067] Eigenvalue decomposition is performed on the covariance matrix Σ to obtain eigenvalues and eigenvectors corresponding to the eigenvalues ; wherein, ;

[0068] According to the preset principal component contribution rate threshold, the eigenvectors corresponding to the first k eigenvalues are selected as principal components; the first data source is projected onto the principal components for dimension reduction operation to obtain the dimension-reduced eigenvectors , and the eigenvectors are marked as key feature information.

[0069] Exemplarily, after identifying the communication protocol, the data format, interface parameter, data transmission frequency and other interface type related features of the data source are further analyzed; the key feature information is extracted by using the feature extraction algorithm in machine learning.

[0070] The principal component analysis method is used to analyze the data format, interface parameter, data transmission frequency and other interface type related features of the data source, and the key feature information is extracted.

[0071] (1) Data standardization: assuming that the original data matrix is , wherein m is the number of samples and n is the number of features. The data is standardized to make the mean of each feature 0 and the variance 1.

[0072] The standardization formula is: , wherein is the mean of the ith feature, is the variance of the bth feature.

[0073] (2) Computing the covariance matrix: the covariance matrix Σ of the normalized data is calculated, and the formula is: , wherein is the normalized data matrix;

[0074] (3) Eigenvalue decomposition: eigenvalue decomposition is performed on the covariance matrix Σ to obtain eigenvalues and corresponding eigenvectors ;

[0075] (4) Selecting principal components: according to the set principal component contribution rate threshold of 85% (i.e., retaining the principal components with a cumulative contribution rate of more than 85%), the eigenvectors corresponding to the first k eigenvalues As principal components. The original data is projected onto these principal components to obtain the feature vectors after dimension reduction , Y is the extracted key feature information, which is used for subsequent adaptive rule generation.

[0076] As Figure 3 shown, another multi-source data acquisition method based on adaptive interface provided by the embodiment of the application comprises:

[0077] Based on the key feature information, a decision tree is constructed to generate corresponding adaptive rules.

[0078] In some implementations, S110: for any feature A, the value of the feature A is marked as , and the training data set D is divided into n subsets according to the feature A.

[0079] S120: assuming that the training data set of the first data source is D, containing n types of adaptive rule types, the sample number of the i-th type is , the information entropy is calculated by the formula ;

[0080] The information gain of the feature A to D is calculated by the formula ;

[0081] S130: the information gain ratio is calculated by the formula ;

[0082] The decision tree algorithm is called, and the feature with the largest information gain ratio is selected as the split attribute of the current node.

[0083] S140: starting from the root node of the decision tree, the feature with the largest information gain ratio is split to form a child node, and steps S110-S130 are repeated for each child node until all samples belong to the same adaptive rule type, the repetition process is marked, and the leaf node of the decision tree at this time is marked as the corresponding adaptive rule.

[0084] For example, the decision tree algorithm is used to generate adaptive rules for the data source; the construction of the decision tree is based on existing water conservancy data acquisition adaptation experience and rule templates, and through the judgment and branch selection of the data source feature information, specific adaptive rules are generated, including data parsing rules, data conversion rules, and communication connection rules.

[0085] (1) Decision tree algorithm selection and construction basis

[0086] ​C4.5 decision tree algorithm is selected, which can process continuous and discrete features, and can select the optimal split attribute according to the information gain ratio, and is suitable for the scene with complex water conservancy data source characteristics. The construction of the decision tree is based on the existing water conservancy data collection adaptation experience and rule templates, and the protocol type, data format, interface parameter and other data source characteristic information are taken as the decision attributes, and the adaptation rules are taken as the leaf nodes.

[0087] (2) Decision tree construction and rule generation

[0088] 1) Calculate information entropy

[0089] Let the training data set be D, containing k types of adaptation rules, and the number of samples of the i-th type is The information entropy of D is: .

[0090] 2) Calculate information gain ratio

[0091] For feature A, its value is , according to A, D is divided into n subsets The information gain of feature A to D is: The information gain ratio is: , wherein , the feature with the maximum information gain ratio is selected as the split attribute of the current node.

[0092] 3) Recursively construct a decision tree

[0093] Starting from the root node, split with the feature with the maximum information gain ratio to form a child node, and repeat the above process for each child node until all samples belong to the same class or no feature can be split, at which time the leaf node is the generated adaptation rule.

[0094] Another multi-source data collection method based on an adaptive interface provided by the embodiment of the application comprises:

[0095] Accessing a second data source; presetting a similarity threshold, calculating the similarity between the second data source and the adaptation rule; if the similarity is not less than the preset similarity threshold, marking the second data source and the first data source as the same type of data source, and setting the same type of data source to use the same adaptation rule; wherein the second data source is a water conservancy data source to be matched with the adaptation rule.

[0096] Illustratively, the generated adaptation rules are stored in a rule library, and database technology is used for storage management. The rule library supports rule query, update and deletion operations, and can dynamically maintain the adaptation rules according to the changes of the data source.

[0097] When a new same kind of data source is accessed, the corresponding adaptive rule in the rule library is quickly called, the data source is collected, parsed and converted according to the rule, and the collected data is converted into a unified format and transmitted to the data aggregation platform;

[0098] The same kind of data source is identified by comparing the similarity of the protocol type and characteristic information of the newly accessed data source with the characteristic information stored in the rule library. Cosine similarity calculation is adopted, the data source characteristic vector is , the characteristic vector of a rule in the rule library is When S is greater than or equal to 0.8, it is determined that the data source is the same kind.

[0099] Based on the above technical scheme, the protocol identification model is adopted, a large number of known protocol type water conservancy data samples are trained, so that the model can accurately identify multiple water conservancy data communication protocols such as Modbus and MQTT. The model will preprocess the data, including cleaning and normalization, then extract the protocol characteristic vector and match it with the stored characteristic mode, realize the automatic identification of the protocol, and get rid of the dependence on manual identification of the protocol; the interface characteristic information obtained by protocol identification and characteristic analysis is used as input, and a decision tree algorithm is used to generate adaptive rules. The decision tree is based on the existing water conservancy data collection adaptation experience and rule template, and generates specific rules such as data parsing, conversion and communication connection through the judgment of the characteristic information, realizes the automatic generation of the adaptive rule, and realizes the automatic generation of the adaptive rule. The rule library stores the adaptive rule by using the database technology, supports the query, update, deletion and other operations of the rule, and can be dynamically maintained according to the change of the data source. When the same kind of data source is accessed again, the adaptive rule in the rule library can be quickly called, the rule reuse is realized, and the process of repeatedly generating the rule is avoided.

[0100] The above mainly introduces the scheme of the embodiments of the application from the perspective of device implementation. It can be understood that each device, for example, a multi-source data collection system based on an adaptive interface, contains at least one of the corresponding hardware structure and software module for executing each function. Those skilled in the art should easily realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software driven hardware depends on the specific application and design constraints of the technical scheme. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0101] The embodiments of the present application can divide the functional units of a multi-source data acquisition system based on an adaptive interface according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. It should be noted that the division of the units in the embodiments of the present application is illustrative, and is only a logical division. In actual implementation, another division manner can be used.

[0102] The units in the embodiments of the present application can also be referred to as modules. For example, the processing unit can be referred to as a processing module.

[0103] In the above embodiments, the software, hardware, firmware, or any combination thereof can be used to realize the above-mentioned embodiments wholly or partially. When the software program is used to realize the above-mentioned embodiments, the computer program product in the form of whole or part can be used to realize the above-mentioned embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the flow or function described in the embodiments of the present application is wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in the computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by the computer or include one or more data storage devices such as servers, data centers, etc. that can be integrated with the medium. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (solid state disk, SSD)), etc.

[0104] Although the present application is described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and implemented by those skilled in the art through viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. Some measures described in mutually different dependent claims can be combined and produce a good result.

[0105] Although the application has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the scope of the application. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation, as it should be understood that various modifications and equivalents can be used without departing from the spirit and scope of the application. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for multi-source data acquisition based on adaptive interface, characterized in that, The application relates to a water conservancy data source protocol identification method and device. Obtaining a plurality of water conservancy data sources of known protocol types, and taking the plurality of water conservancy data sources as first data sources; Calling a convolutional neural network, constructing a protocol identification model based on the convolutional neural network, presetting a loss function, training the protocol identification model through the preset loss function, obtaining the protocol types of the first data sources through the trained protocol identification model, extracting key feature information of the protocol types through principal component analysis technology, constructing a decision tree based on the key feature information to generate corresponding adaptation rules, and storing the adaptation rules in the form of rule feature vectors in a rule library; Accessing a second data source; Presetting a similarity threshold, calculating the similarity between the second data source and the adaptation rules, marking the second data source and the first data source as the same type of data source if the similarity is not less than the preset similarity threshold, and setting the same type of data source to use the same adaptation rule; wherein the second data source is a water conservancy data source to be matched with the adaptation rule; The method comprises the following steps: A1: calculating information entropy A2: calculating information gain ratio Let the training data set be D, containing k types of fitting rules, and the number of samples of the jth type is The information entropy of D is: ; A3: recursively constructing a decision tree For feature A, its value is , D is divided into n subsets according to A , the information gain of feature A on D is: The information gain ratio is: , wherein The feature with the largest information gain ratio is selected as the split attribute of the current node; is the information entropy of the training data set ; Starting from a root node, splitting the features with the maximum information gain ratio to form child nodes, and repeating the above process for each child node until all samples belong to the same class or no features can be split, at which time the leaf node is the generated adaptation rule; The similarity between the second data source and the adaptation rules is calculated by comparing the protocol types and feature information of the second data source with the feature information stored in the rule library to identify the same type of data source. The protocol identification model comprises an input layer, a convolutional layer, a pooling layer and a fully connected layer; The fully connected layer is used to connect a plurality of feature vector sets output by the pooling layer, and map the feature vector sets to a protocol category space through the fully connected layer to obtain a protocol category probability distribution vector.

2. The method of claim 1, wherein, The feature vector set is mapped to the protocol category space through the fully connected layer to obtain the protocol category probability distribution vector, which comprises: The input layer is used to call the first data source, and the first data source is integrated into a data sequence X=[ ]; wherein, represents the i-th element in the data sequence, and L is the length of the data sequence. The convolution layer is used for presetting the jth convolution kernel as , and the size of each convolution kernel is m; wherein, is a positive integer; The convolution result of the jth convolution kernel at position k is calculated by the formula ; wherein, is the bias term corresponding to the jth convolution kernel; is the bias term corresponding to the jth convolution kernel; represents the weight value of the ith convolution result in the jth convolution kernel; By convolution operation, a plurality of feature map sets are obtained ; The pooling layer is configured to set a length of a pooling window as s, and calculate a result of the jth feature map at a position k after the pooling by a formula ; wherein, is a global position of the ith element in the kth window.​ By convolution operation, a feature vector set is obtained ; Setting the weight matrix of the fully connected layer as W and the bias vector as b; 3. The method of claim 2, wherein, The preset loss function is obtained in the following manner: The protocol identification model is trained through the preset loss function, which comprises: The output formula of the full connection layer is: ; wherein, is a feature vector set output by the pooling layer, is a preset activation function, and o is a protocol category probability distribution vector; and each element in the protocol category probability distribution vector corresponds to the probability of a protocol type.

4. The method of claim 1, wherein, Calling the preset loss function, obtaining an updated loss function based on the fully connected layer weight matrix; obtaining a historical first data source as a training set sample wherein, is the nth historical first data source sample, is a corresponding protocol category label; n is a positive integer; Through formula The loss function is calculated. Where N is the total number of data samples, and C is the total number of protocol categories. This represents the true label of the nth historical training set sample on the c-th protocol. The protocol identification model predicts the probability that the nth sample index belongs to the cth protocol class; where the true labels include 0 and 1.

5. The method of claim 1, wherein, When the updated loss function is the minimum, the protocol identification model is marked as being trained. by backpropagation algorithm formula The weight matrix of the updated full connection layer of the protocol identification model is calculated ; wherein, the preset learning rate of the protocol identification model, denotes the weight matrix of the full connection layer before updating, is the feature vector output after the nth historical training set sample is processed by the pooling layer, denotes the transpose of The protocol types of the first data sources are obtained through the trained protocol identification model, which comprises: The key feature information of the protocol types is extracted through the principal component analysis technology, which comprises:

6. The method of claim 1, wherein, The principal component analysis technology is used to analyze the data format, interface parameters and data transmission frequency of the first data sources to obtain a plurality of features; The first data source is input into the trained protocol identification model, a protocol category probability distribution vector is output through the protocol identification model, and the category with the maximum probability value in the protocol category probability distribution vector is selected as the identified protocol type wherein, , represents the probability of the cth protocol in the inference process of the protocol identification model.

7. The method of claim 1, wherein, The similarity between the second data source and the adaptation rules is calculated, which comprises: The application relates to a water conservancy data source protocol identification method and device. The matrix form of the first data source is set as Wherein m is the number of the first data source, and n is the number of features; the first data source is normalized to obtain a mean value of 0 and a variance of 1 for each feature; The matrix of the bth feature containing the ath sample is represented as The covariance matrix of the normalized data is calculated by the formula ; wherein, is the normalized data matrix;​ Eigenvalue decomposition of the covariance matrix Σ yields eigenvalues and eigenvectors corresponding to the eigenvalues ; wherein ; According to a preset principal component contribution rate threshold, selecting the eigenvectors corresponding to the first k eigenvalues As the principal component; project the first data source onto the principal component for dimension reduction operation to obtain the dimension-reduced eigenvectors , and mark the eigenvectors as key feature information.

8. The method of claim 1, wherein, The communication unit is used to obtain a plurality of water conservancy data sources of known protocol types, take the plurality of water conservancy data sources as first data sources, and access a second data source. The rule feature vector of the first data source is set as The rule feature vector of the second data source is ; The similarity is calculated by the formula .​ 9. A multi-source data acquisition system based on adaptive interface, characterized in that, ​ ​ ​ The processing unit is configured to invoke a convolutional neural network, construct a protocol identification model based on the convolutional neural network, preset a loss function, train the protocol identification model through the preset loss function, obtain a protocol type of the first data source through the trained protocol identification model, extract key feature information of the protocol type through principal component analysis technology, construct a decision tree based on the key feature information to generate corresponding adaptation rules, and store the adaptation rules in the form of rule feature vectors in a rule library. A preset similarity threshold is configured to calculate a similarity between the second data source and the adaptation rules, and if the similarity is not less than the preset similarity threshold, the second data source and the first data source are marked as the same type of data source, and the same adaptation rule is set for the same type of data source, wherein the second data source is a water conservancy data source to be matched with the adaptation rules. The construction of the decision tree based on the key feature information to generate corresponding adaptation rules comprises: A1: calculating information entropy A2: calculating information gain ratio Let the training data set be D, containing k types of fitting rules, and the number of samples of the jth type is The information entropy of D is: ; A3: recursively constructing a decision tree For feature A, its value is , D is divided into n subsets according to A , the information gain of feature A on D is: The information gain ratio is: , wherein The feature with the largest information gain ratio is selected as the split attribute of the current node; is the information entropy of the training data set ; Starting from a root node, the feature with the maximum information gain ratio is split to form a child node, and the above process is repeated for each child node until all samples belong to the same class or no feature can be split, at which time the leaf node is the generated adaptation rule. The calculation of the similarity between the second data source and the adaptation rules comprises: The same type of data source is identified by comparing the protocol type and the feature information of the second data source with the feature information stored in the rule library. ​

Images