Iot gateway data processing method and related apparatus

Abstract

The application provides an Internet of Things gateway data processing method and related device, a traffic identification framework is acquired and a device set to be processed is determined; each decision unit in the traffic identification framework is walked through from a starting identification unit of the traffic identification framework, a traffic detection network corresponding to the walked decision unit is acquired, and a device set corresponding to the walked decision unit in the device set to be processed is determined; based on the traffic detection network corresponding to the walked decision unit and a traffic identification feature vector in the traffic detection network corresponding to the walked decision unit, the device set corresponding to the walked decision unit is identified, a device set corresponding to each sub-decision unit connected to the walked decision unit is determined, and each terminal identification unit in the traffic identification framework is obtained until each terminal identification unit in the traffic identification framework corresponds to a device set respectively. The application can improve the accuracy of device identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a method for processing data of an Internet of Things gateway and related apparatus. BACKGROUND

[0002] With the rapid development of Internet of Things technology, the number and types of Internet of Things devices are showing explosive growth. Among numerous Internet of Things application scenarios, Internet of Things gateways, as key devices connecting the perception layer and the network layer, undertake the important task of aggregating, processing and forwarding data from different devices. However, as the devices accessing the Internet of Things gateway become increasingly diverse and complex, how to efficiently and accurately identify and manage these devices has become a problem to be solved in the field of Internet of Things.

[0003] In traditional methods for identifying Internet of Things devices, a single feature or simple rules are often used to classify and identify devices. For example, only the basic information of a device (such as device model, manufacturer, etc.) or some simple operating parameters (such as power consumption, transmission rate, etc.) are used to determine the type of the device. This method may meet the basic identification needs when facing a small number of devices with a single type. However, when a large number of devices with various types are connected to an Internet of Things system, the identification method using a single feature or simple rules is not sufficient, and cannot accurately distinguish and identify the devices, resulting in low accuracy of device identification, which is not conducive to device management. On the other hand, some existing device identification methods often face problems such as high consumption of computing resources and low identification efficiency when processing large-scale data. Due to the lack of an effective hierarchical identification mechanism, all device data need to be uniformly processed and analyzed, resulting in high computational complexity and long identification time, which cannot meet the requirements of real-time application scenarios of the Internet of Things. SUMMARY

[0004] The purpose of the present application is to provide a method for processing data of an Internet of Things gateway and related apparatus.

[0005] The present application is implemented as follows:

[0006] In a first aspect, the present application provides a method for processing data of an Internet of Things gateway, the method comprising: obtaining a traffic identification framework and determining a set of devices to be processed; each non-terminal identification unit of the traffic identification framework corresponds to a first type of traffic detection network or a second type of traffic detection network, and in each identification track of the traffic identification framework, a decision unit corresponding to the second type of traffic detection network is located after a decision unit corresponding to the first type of traffic detection network; wherein the first type of traffic detection network is a traffic detection network including a traffic identification feature vector, and the second type of traffic detection network is a traffic detection network including a traffic deep feature vector; starting from a starting identification unit of the traffic identification framework, each decision unit in the traffic identification framework is walked through one by one, a traffic detection network corresponding to the walked decision unit is obtained, and a set of devices corresponding to the walked decision unit in the set of devices to be processed is determined; based on the traffic detection network corresponding to the walked decision unit and the traffic identification feature vector in the traffic detection network corresponding to the walked decision unit, the set of devices corresponding to the walked decision unit is identified, and a set of devices corresponding to each sub-decision unit connected to the walked decision unit is determined until a set of devices corresponding to each terminal identification unit in the traffic identification framework is obtained.

[0007] In a second aspect, the present application provides a data processing device, comprising: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method described above is implemented.

[0008] The method and system for processing data of an Internet of Things gateway provided by the embodiments of the present application can, when a set of devices to be processed needs to be identified, based on the obtained traffic identification framework, walk through each decision unit in the traffic identification framework one by one, identify the set of devices corresponding to the walked decision unit according to the walked decision unit, and obtain the set of devices corresponding to the sub-decision unit connected to the walked decision unit, so that the hierarchical identification of the set of devices to be processed according to the obtained traffic identification framework can be completed by walking through one by one. Because the set of devices to be processed can be hierarchically identified by the present application, the accuracy of device identification is effectively improved. In addition, because in the traffic identification framework, the decision unit corresponding to the second type of traffic detection network is located after the decision unit corresponding to the first type of traffic detection network, the present application first identifies the set of devices to be processed according to the basic index of the device, obtains a plurality of sets of devices separated from the set of devices to be processed, and then performs deep identification on the separated sets of devices according to the deep feature, thereby further increasing the accuracy. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0010] Figure 1 This is a flowchart of an IoT gateway data processing method provided in an embodiment of this application.

[0011] Figure 2 This is a schematic diagram of the composition of a data processing device provided in an embodiment of this application. Detailed Implementation

[0012] In this embodiment, the execution entity of the IoT gateway data processing method is a data processing device, including but not limited to the IoT gateway itself, or a server, personal computer, laptop computer, etc. connected to the IoT gateway. The data processing device can operate independently to implement this application, or it can access a network and implement this application through interactive operation with other data processing devices in the network. The network where the data processing device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, VPN network, etc.

[0013] This application provides an embodiment of an IoT gateway data processing method, such as... Figure 1 As shown, the method includes:

[0014] Step S100: Obtain the traffic identification framework and determine the set of devices to be processed; each non-terminal identification unit of the traffic identification framework corresponds to a first type of traffic detection network or a second type of traffic detection network. In each identification trajectory of the traffic identification framework, the decision unit corresponding to the second type of traffic detection network is located after the decision unit corresponding to the first type of traffic detection network; wherein, the first type of traffic detection network is a traffic detection network that includes traffic identification representation vectors of basic traffic features, and the second type of traffic detection network is a traffic detection network that includes traffic identification representation vectors of deep traffic features.

[0015] The traffic recognition framework can be understood as a hierarchical structure, for example, a tree structure. In this tree structure, each node corresponds to a category. Each non-terminal recognition unit in the tree structure (the terminal recognition unit is the end node of the tree structure) corresponds to a first type of traffic detection network or a second type of traffic detection network. The first type of traffic detection network is a traffic detection network that includes a traffic recognition feature vector of traffic basic features, and the second type of traffic detection network is a traffic detection network that includes a traffic recognition feature vector of traffic deep features. For example, for traffic recognition of smart cameras, traffic basic features can include packet size, transmission rate, etc., which are relatively easy to obtain and analyze; while traffic deep features can involve more complex information such as image data content, frame rate, etc.

[0016] In each recognition track of the traffic recognition framework (i.e. a path from the trunk node to the end node), the decision unit corresponding to the second type of traffic detection network is after the decision unit corresponding to the first type of traffic detection network. This means that in the recognition process, the data processing apparatus will first use traffic basic features for preliminary recognition, and then use traffic deep features for more detailed recognition. For example, in identifying smart devices, first determine the general type of the device according to basic features such as packet size, transmission rate, etc., such as determining whether it is a video device or a sensor device; and then further determine which smart camera or sensor it is according to deep features such as image content and frame rate.

[0017] In order to construct such a traffic recognition framework, the data processing apparatus can use machine learning algorithms and data analysis techniques. For example, by collecting a large amount of device traffic data, using clustering algorithms to classify the data, an initial tree structure is constructed. Then, according to the classification results, select appropriate traffic detection networks for each node. Specifically, feature selection algorithms can be used to select the most representative traffic basic features and traffic deep features from a large number of traffic features to construct a traffic recognition feature vector. For example, information gain, chi-square test, etc. can be used to evaluate the importance of features.

[0018] The role of the Internet of Things gateway is to aggregate data from different devices, so the set of devices to be processed is the set of all devices that need to be processed through the Internet of Things gateway. In practical applications, the data processing apparatus can determine the set of devices to be processed in various ways.

[0019] One possible way is through device registration information. When a device accesses the Internet of Things network, it will register with the Internet of Things gateway or related management system, providing basic information about the device, such as device ID, device type, device location, etc. The data processing device can obtain the information of all registered devices from the registration information, and thus determine the set of devices to be processed. Another way is through network scanning and detection. The data processing device can periodically scan the Internet of Things network, discover the devices in the network, and obtain some basic information about the devices, such as the MAC address and IP address of the devices. Through analysis and processing of this information, the type and status of the devices are determined, and thus the set of devices to be processed is determined. In addition, the data processing device can also determine the set of devices to be processed in combination with the historical data of device usage and user behavior data. For example, according to the user's operation records and usage frequency of the device, the devices frequently used by the user are analyzed, and these devices are taken as part of the set of devices to be processed. At the same time, according to the fault records and maintenance records of the devices, the devices that have faults or need maintenance are excluded, and it is ensured that the devices in the set of devices to be processed can work normally.

[0020] Step S200: Starting from the starting recognition unit of the traffic recognition framework, each decision unit in the traffic recognition framework is walked through one by one, the traffic detection network corresponding to the walked decision unit is obtained, and the set of devices corresponding to the walked decision unit in the set of devices to be processed is determined.

[0021] The traffic recognition framework has a specific structure, and the starting recognition unit is the starting point of the entire recognition process. The data processing device first locates to this starting recognition unit, and then walks through the decision units in the framework one by one according to certain rules. For example, in a traffic recognition framework based on tree structure, the starting recognition unit can be regarded as the main trunk node of the tree. The data processing device starts from this starting recognition unit and traverses the entire tree structure according to algorithms such as depth-first search or breadth-first search. Taking depth-first search as an example, the data processing device will access nodes as deeply as possible along a path until it cannot continue or reaches the target node, then backtracks to the previous step and continues to explore other paths.

[0022] When the data processing apparatus wanders to a certain decision unit, the traffic detection network corresponding to the decision unit needs to be acquired. As described above, each non-terminal recognition unit of the traffic recognition framework corresponds to a traffic detection network of the first kind or a traffic detection network of the second kind. For example, when the data processing apparatus wanders to a decision unit corresponding to the preliminary classification of an intelligent lighting device, the decision unit may correspond to a traffic detection network of the first kind, i.e., a traffic detection network containing traffic recognition feature vectors of traffic basic features. These traffic basic features can include the communication frequency of the device, the average length of the data packet, etc. The data processing apparatus can acquire the traffic detection network corresponding to the decision unit by querying a pre-established mapping table. This mapping table stores the correspondence between each decision unit and the corresponding traffic detection network, which has been established when the traffic recognition framework is constructed. When the traffic detection network corresponding to a certain decision unit needs to be acquired, the data processing apparatus only needs to look up the mapping table according to the identification of the decision unit.

[0023] While acquiring the traffic detection network corresponding to the decision unit that the data processing apparatus wanders to, the data processing apparatus also needs to determine the set of devices in the set of devices to be processed that correspond to the decision unit. This process is matched according to the classification features of the decision unit and the related information of each device in the set of devices to be processed. For example, assuming that the decision unit that the data processing apparatus currently wanders to is classified according to the communication protocol type of the device, and the communication protocol type is ZigBee. The data processing apparatus will traverse each device in the set of devices to be processed, and check the communication protocol information of the device. If the communication protocol of a device is ZigBee, then the device belongs to the set of devices corresponding to this decision unit.

[0024] Step S300: based on the traffic detection network corresponding to the decision unit that the data processing apparatus wanders to, and the traffic recognition feature vectors in the traffic detection network corresponding to the decision unit that the data processing apparatus wanders to, the set of devices corresponding to the decision unit that the data processing apparatus wanders to is recognized, the set of devices corresponding to each sub-decision unit connected to the decision unit that the data processing apparatus wanders to is determined, until the set of devices corresponding to each terminal recognition unit in the traffic recognition framework is obtained.

[0025] The traffic detection network is a model tool for analyzing and recognizing device traffic, which contains traffic recognition feature vectors. The traffic recognition feature vectors can reflect the characteristics of device traffic, and through the analysis of these characteristics, the device can be classified and recognized. For example, in an Internet of Things environment containing intelligent cameras and intelligent temperature and humidity sensors, the traffic detection network corresponding to the decision unit for recognizing intelligent cameras can include the frame rate, resolution, etc. of the video stream as the traffic recognition feature vectors; while the traffic detection network corresponding to the decision unit for recognizing intelligent temperature and humidity sensors can include the data update frequency, data value range, etc. as the traffic recognition feature vectors.

[0026] The data processing apparatus first acquires a device data stream set corresponding to the decision unit to which the walk is made. This device data stream set contains device data streams corresponding to each device in the device set corresponding to the decision unit. Then, the data processing apparatus selects device data items from each device data stream in the device data stream set based on the traffic identification feature vector in the traffic detection network corresponding to the decision unit to which the walk is made. For example, if the traffic identification feature vector contains the feature of the communication frequency of the device, the data processing apparatus extracts the data item related to the communication frequency from the data stream of each device. This process can be implemented by data parsing and feature extraction algorithms, such as using regular expression matching, data slicing, etc. to extract specific data items. Then, the data processing apparatus identifies the device set corresponding to the decision unit based on the selected device data items and the traffic detection network corresponding to the decision unit to which the walk is made, and determines the device set corresponding to each sub-decision unit connected to the decision unit. For example, assuming that the current decision unit has two sub-decision units corresponding to different types of smart home devices, the data processing apparatus determines which sub-decision unit the device belongs to according to the extracted device data items such as the power and operating mode of the device, and combines the classification model in the traffic detection network. This process can be implemented using classification algorithms in machine learning algorithms, such as decision tree classification algorithm, support vector machine classification algorithm, etc. Taking the decision tree classification algorithm as an example, the data processing apparatus traverses the decision tree according to the value of the device data item, and finally reaches a terminal node. The class corresponding to the terminal node is the device set corresponding to the sub-decision unit to which the device belongs.

[0027] The data processing apparatus will repeatedly perform the above identification process to continue walking from the current decision unit to its sub-decision units until it obtains the device set corresponding to each terminal identification unit in the traffic identification framework.

[0028] In order to more accurately identify the device, the data processing apparatus can use a multi-feature fusion implementation. For example, when extracting device data items, not only the features corresponding to a single traffic identification feature vector are considered, but also multiple features are analyzed. Assuming that in addition to the communication frequency, the data packet size of the device is also considered as a traffic identification feature vector, the two features can be fused into a comprehensive feature value by weighted summation, as follows:

[0029] ;

[0030] wherein, represents the comprehensive feature value, are the weights of the communication frequency feature and the data packet size feature, respectively, respectively are the communication frequency feature value and the data packet size feature value. By adjusting the weights, the importance of different features can be highlighted according to the actual situation.

[0031] In addition, in order to improve the efficiency and accuracy of identification, the data processing device can also use data preprocessing techniques such as data cleaning, data normalization, etc. Data cleaning can remove noise data and abnormal data in the device data stream, and data normalization can convert device data items of different dimensions to the same value range for comparison and analysis. For example, for communication frequency data with a value range of 0-100 and data packet size data with a value range of 0-1000, they can be converted to the range of 0-1 by linear normalization method, as follows:

[0032] ;

[0033] wherein, represents the normalized data, X represents the original data, respectively represent the minimum value and the maximum value of the original data.

[0034] As an implementation, in step S300, based on the flow detection network corresponding to the decision unit walked to and the flow identification feature vector in the flow detection network corresponding to the decision unit walked to, the device set corresponding to the decision unit walked to is identified, and the device set corresponding to each sub-decision unit connected to the decision unit walked to is determined, including:

[0035] Step S310: obtaining a device data stream set corresponding to the decision unit walked to; the obtained device data stream set includes device data streams respectively corresponding to each device in the device set corresponding to the decision unit walked to;

[0036] Step S320: based on the flow identification feature vector in the flow detection network corresponding to the decision unit walked to, selecting device data items from each device data stream in the device data stream set corresponding to the decision unit walked to respectively;

[0037] Step S330: based on the selected device data items, and based on the flow detection network corresponding to the decision unit walked to, the device set corresponding to the decision unit walked to is identified, and the device set corresponding to each sub-decision unit connected to the decision unit walked to is determined.

[0038] When the data processing device wanders to a certain decision unit in the traffic recognition framework, it needs to obtain the corresponding device data stream set. This device data stream set contains the device data stream corresponding to each device in the device set corresponding to the decision unit. In order to obtain these device data streams, the data processing device can adopt various implementation manners. One feasible manner is to collect data through the Internet of Things gateway. As the hub connecting various devices and networks, the Internet of Things gateway can receive and forward the data sent by the devices. The data processing device can send a query request to the Internet of Things gateway, specify the device information and data type to be obtained, and the Internet of Things gateway obtains the data from the corresponding device according to the request and returns it to the data processing device. For example, using the message queue mechanism based on the MQTT (Message Queuing Telemetry Transport) protocol, the device publishes data to a specific topic, and the Internet of Things gateway subscribes to these topics. When the data processing device needs to obtain data, it can obtain the data stream of the corresponding device by sending a subscription request to the gateway.

[0039] After obtaining the device data stream set, the data processing device selects the corresponding device data item from each device data stream according to the traffic recognition feature vector in the traffic detection network corresponding to the decision unit to which it wanders. The traffic recognition feature vector is a set of feature vectors that can reflect the characteristics of device traffic. The traffic recognition feature vectors corresponding to different decision units may be different. For example, for the decision unit of the intelligent lighting device, its traffic recognition feature vector may include the brightness value, color temperature value, energy consumption and other characteristics of the device.

[0040] The data processing device parses and analyzes the data stream of each device, extracts the corresponding device data item from it according to the definition of the traffic recognition feature vector. For example, for a device data stream of a smart light bulb, it may contain a series of data packets, each of which contains different field information such as timestamp, device ID, brightness value, color temperature value, etc. The data processing device extracts the corresponding brightness value and color temperature value as device data items from the data packets by parsing the data packets according to the brightness value and color temperature value features defined in the traffic recognition feature vector. In order to realize this process, the data processing device can use data parsing technology. For example, for device data streams based on TCP / IP protocol transmission, network packet analysis tools such as Wireshark can be used to parse the structure and content of the data packets. By writing a parsing program, according to the format and characteristics of the data packets, the required device data items are extracted. The specific parsing process may involve bit operations, byte conversion and other operations on binary data to ensure that the device data items are correctly extracted.

[0041] After obtaining the device data items, the data processing apparatus will use these data items and the flow detection network corresponding to the decision unit that the device has walked to, to identify the device set, so as to determine the device set corresponding to each sub-decision unit connected to the decision unit. The flow detection network is a model for classifying and identifying device data, which can divide devices into different categories according to the characteristics and patterns of device data items.

[0042] For example, in the example of intelligent lighting devices, it is assumed that the sub-decision units connected to the current decision unit correspond to different lighting modes, such as normal lighting mode, energy-saving lighting mode, and night lighting mode. The data processing apparatus will determine to which sub-decision unit each device belongs according to the selected device data items, such as brightness value, color temperature value, and energy consumption, combined with the classification rules and algorithms in the flow detection network.

[0043] A feasible implementation is to use a machine learning algorithm to construct the flow detection network. For example, a decision tree algorithm can be used to construct the flow detection network. The decision tree is a classification model based on tree structure, which constructs a series of decision rules through learning of training data, for classifying unknown data. When constructing the decision tree, a training data set is first needed, which contains device data items with known category labels. For example, the training data set of intelligent lighting devices may contain a large number of device brightness values, color temperature values, energy consumptions, and corresponding lighting mode category labels. Then, the decision tree algorithm is used to learn and train the training data set to generate a decision tree model. Each internal node of the decision tree represents a test on an attribute, each branch represents the output of the test, and each terminal node represents a category label. For example, in the decision tree of intelligent lighting devices, an internal node may be a test on the brightness value, if the brightness value is greater than a certain threshold, then follow one branch to continue judging; if the brightness value is less than or equal to the threshold, then follow another branch to continue judging. Finally, according to a series of tests and judgments, the terminal node is reached, and the lighting mode category to which the device belongs is obtained.

[0044] When identifying the device set, the data processing apparatus will input the selected device data items into the trained decision tree model, traverse and judge according to the rules of the decision tree, and finally determine the device set to which the device belongs. For example, for a smart bulb with a brightness value of 80, a color temperature value of 4000K, and an energy consumption of 5W, the data processing apparatus will input these data items into the decision tree model, and after a series of judgments, it is finally determined that the smart bulb belongs to the sub-decision unit corresponding to the normal lighting mode.

[0045] In addition to the decision tree algorithm, other machine learning algorithms such as support vector machines (SVM), neural networks, etc. can also be used to build a traffic detection network. Support vector machines separate different classes of data by finding the optimal hyperplane; neural networks perform nonlinear mapping and classification of data by constructing a multi-layer neural network. These algorithms can be selected and applied according to the specific application scenario and data characteristics, and the present application does not limit them.

[0046] Through the operations of steps S310 to S330, the data processing apparatus can accurately identify and classify the device set based on the device data stream set and the traffic detection network, determine the device set corresponding to each sub-decision unit connected to the decision unit reached, and provide a basis for subsequent device management and data analysis. For example, in a smart home system, according to the device set under different lighting modes, the system can implement an automatic lighting control strategy, improving energy utilization efficiency and user experience.

[0047] As an implementation, the traffic detection network includes a fusion hidden Markov model; the fusion hidden Markov model includes a plurality of hidden Markov models; step S330, based on the selected device data items and based on the traffic detection network corresponding to the decision unit reached, identifying the device set corresponding to the decision unit reached, determining the device set corresponding to each sub-decision unit connected to the decision unit reached, includes:

[0048] Step S331: For each device data stream in the device data stream set corresponding to the decision unit reached, load the device data items selected from the device data stream into each hidden Markov model in the traffic detection network corresponding to the decision unit reached, and obtain the identification confidence output by each hidden Markov model in the traffic detection network corresponding to the decision unit reached.

[0049] Step S332: Determine the target hidden Markov model in the traffic detection network corresponding to the decision unit reached that outputs the maximum identification confidence, and determine the target sub-decision unit in each sub-decision unit connected to the decision unit reached that is associated with the target hidden Markov model.

[0050] Step S333: Attribute the device corresponding to the device data stream to the target sub-decision unit.

[0051] A Hidden Markov Model (HMM) is a statistical model that describes a stochastic process with hidden states. In embodiments of the present application, each HMM can be regarded as a recognizer for a specific device feature pattern. For example, in a smart security system, there can be multiple types of sensor devices, such as cameras, infrared detectors, door / window sensors, etc. For a traffic detection network corresponding to a decision unit that recognizes camera devices, the HMMs therein can model different resolution, frame rate, etc. feature patterns, respectively.

[0052] When the data processing apparatus processes each device data stream in the set of device data streams corresponding to the decision unit that the agent walks to, it loads the device data items selected from the device data stream into each HMM in the traffic detection network. For example, for a camera device data stream, the selected device data items can include frame rate, resolution, bitrate, etc. information of the video stream. The data processing apparatus inputs these data items as inputs to each HMM in the traffic detection network in turn.

[0053] An HMM outputs a recognition confidence by calculating the probability of an observation sequence (i.e. the input device data items) under the model. Specifically, an HMM is composed of a hidden state sequence, an observation sequence, and parameters such as a state transition probability matrix and an observation probability matrix. Suppose a simple HMM is used to recognize the frame rate feature of a camera, its hidden states can represent different frame rate intervals, and the observation sequence is the actual frame rate data item input. The model describes the transition relationship between hidden states according to the state transition probability matrix, and the probability of observing a specific observation value under a certain hidden state according to the observation probability matrix.

[0054] The process of calculating the recognition confidence usually involves forward algorithm or backward algorithm, etc. Take the forward algorithm as an example. Its basic idea is to calculate the probability of the observation sequence up to time t by dynamic programming, given the model parameters. The specific formula is as follows:

[0055] ;

[0056] wherein, represents the probability of the hidden state being and the observation sequence being observed at time t, denotes the parameter set of the HMM, including the state transition probability matrix and the observation probability matrix, etc. The probability of the entire observation sequence, i.e. the recognition confidence, can be obtained by recursive calculation.

[0057] For example, for a camera device data item with a frame rate of 30 fps, when input into the hidden Markov models corresponding to different frame rate intervals, the models will calculate the probability of the frame rate data appearing in different hidden states according to the parameters inside the models, so as to obtain the recognition confidence of the model for the device data item. Assuming that the recognition confidence calculated by the model is 0.8, it means that the model considers that the probability of the device data item conforming to the corresponding characteristic mode is 80%.

[0058] After obtaining the recognition confidences output by the hidden Markov models in the flow detection network, the data processing apparatus compares the sizes of the recognition confidences to find the target hidden Markov model outputting the maximum recognition confidence. For example, in the above-mentioned intelligent security system, for a device data item of a camera device, there can be three hidden Markov models corresponding to different frame rate intervals (such as 15-25 fps, 25-35 fps, and 35-45 fps), and the recognition confidences output by the three hidden Markov models are 0.6, 0.8, and 0.4 respectively. The data processing apparatus compares the three recognition confidences to determine that the hidden Markov model outputting the maximum recognition confidence 0.8 is the target hidden Markov model.

[0059] After determining the target hidden Markov model, the data processing apparatus also needs to determine the target sub-decision unit associated with the target hidden Markov model in each sub-decision unit connected to the decision unit to which the data processing apparatus has wandered. This is because each hidden Markov model is usually associated with a specific sub-decision unit, representing a specific device characteristic mode or category. For example, in the intelligent security system, the sub-decision unit associated with the hidden Markov model corresponding to the frame rate of 25-35 fps can represent the high-definition camera category. The data processing apparatus finds the target sub-decision unit corresponding to the target hidden Markov model through the pre-established association mapping table of hidden Markov models and sub-decision units.

[0060] After determining the target hidden Markov model and the target sub-decision unit, the data processing apparatus attributes the device corresponding to the device data stream currently processed to the target sub-decision unit. For example, in the intelligent security system, for the camera device with a frame rate of 30 fps mentioned above, since the target hidden Markov model corresponding to the device is the model corresponding to the frame rate of 25-35 fps, and the model is associated with the target sub-decision unit of the high-definition camera category, the data processing apparatus will attribute the camera device to the sub-decision unit corresponding to the high-definition camera category.

[0061] To achieve this attribution operation, the data processing apparatus can record the correspondence between the device and the sub-decision unit in the device management database. When the device is attributed to the target sub-decision unit, the data processing apparatus updates the record in the database, and associates the identification information of the device with the identification information of the target sub-decision unit. In this way, in subsequent device management and data analysis, the device can be classified and processed according to the sub-decision unit to which the device belongs.

[0062] Through the operations of steps S331 to S333, the data processing apparatus can analyze and identify the device data items using the hidden Markov model, determine the target sub-decision unit to which the device belongs, and thus achieve accurate classification and management of the device. This identification method based on the hidden Markov model can effectively handle the uncertainty and randomness in the device data, improve the accuracy and reliability of device identification, and provide strong support for efficient operation and fine management of the Internet of Things system.

[0063] As an implementation manner, the method provided by the embodiments of the present application can further include:

[0064] Step S400: After obtaining the device set corresponding to each decision unit in the traffic identification framework, determining the generalization network corresponding to each decision unit in the traffic identification framework based on the device data stream set of the device set corresponding to each decision unit in the traffic identification framework.

[0065] Step S500: Identifying the generalization device set based on the generalization network corresponding to each decision unit in the traffic identification framework, wherein the generalization device set is different from the device set to be processed.

[0066] When the data processing apparatus obtains the device set corresponding to each decision unit in the traffic identification framework, it will further determine the generalization network of each decision unit using the device data stream set corresponding to these device sets. The generalization network, also known as the migration network, aims to construct a model that can adapt to different but related scenarios or data distributions through learning and analysis of existing data, thereby improving the generalization ability and adaptability of the model.

[0067] To determine the generalization network corresponding to each decision unit of the intelligent lighting device, the data processing apparatus can implement transfer learning in deep learning. Transfer learning is a method of using existing knowledge or models to solve new problems, which can apply the features and model structures learned in one field or task to another related field or task. Specifically, the data processing apparatus can first pre-train a base model on large-scale general device data. The base model can learn some general device data features, such as time series patterns of data, correlations between data, etc. Then, the data processing apparatus uses the device data stream set corresponding to the intelligent lighting device set as fine-tuning data to fine-tune the pre-trained base model. The fine-tuning process is to further optimize and adjust the parameters of the model based on the base model according to the characteristics of the intelligent lighting device data, so that the model can better adapt to the distribution and features of the intelligent lighting device data. For example, an optimization algorithm such as stochastic gradient descent (SGD) can be used to update the parameters of the model to minimize the loss function of the model on the fine-tuning data. The loss function can be selected according to the specific task and data characteristics, such as mean square error (MSE) loss function for regression tasks, cross-entropy loss function for classification tasks, etc.

[0068] In this way, the data processing apparatus can determine the corresponding generalization network for each decision unit in the traffic identification framework using its corresponding device data stream set. These generalization networks can learn the common features and specific patterns of different types of device data, providing strong support for subsequent identification of the generalization device set.

[0069] After determining the generalization network corresponding to each decision unit in the traffic identification framework, the data processing apparatus will use these generalization networks to identify the generalization device set. The generalization device set is a set of devices different from the processing device set, which may come from different environments, have different data distributions, but have some relevance to the processing device set. When identifying the generalization device set, the data processing apparatus inputs the data stream of each device in the generalization device set into the generalization network corresponding to the corresponding decision unit. The generalization network will extract and analyze the features of the input device data stream, and then output the identification result of the device.

[0070] To improve the accuracy and reliability of identification, the data processing apparatus can use multi-model fusion or ensemble learning implementation. For example, multiple generalization networks with different structures or parameter settings can be fused to combine their output results to determine the final identification result. One feasible fusion method is weighted averaging, that is, according to the performance and reliability of each generalization network, a weight is assigned to it, and then the output results are weighted and averaged to obtain the final result.

[0071] In this way, the data processing apparatus can effectively identify the generalization device set by using the generalization network, accurately classify and identify the devices even if there is some difference between the devices and the set of devices to be processed, and further expand the application range and adaptability of the Internet of Things gateway data processing method. For example, in the field of smart home, the device identification model learned in one family can be applied to other families to realize unified management and control of smart devices in different environments.

[0072] As an implementation manner, before obtaining the traffic identification framework, the method further includes a determination process of the traffic identification framework, specifically including:

[0073] Step S10: obtaining a set of example device data streams of an example device set; the set of example device data streams includes example device basic data corresponding to a plurality of device basic indicators and example device deep data corresponding to a plurality of device deep indicators;

[0074] Step S20: performing multi-level identification on the example device set according to the example device basic data corresponding to the plurality of device basic indicators in the set of example device data streams, for each identification result in each level, determining a traffic identification feature vector from the plurality of device basic indicators according to the identification result corresponding to the corresponding decision unit in the corresponding level, generating a traffic detection network corresponding to the corresponding decision unit and including the determined traffic identification feature vector, and simultaneously establishing a sub-decision unit connected with the corresponding decision unit in the identification framework;

[0075] Step S30: from the terminal identification unit of the identification framework obtained after the multi-level identification stops, performing multi-level identification again on a corresponding example decision unit device set identified from the example device set according to the example device deep data corresponding to the plurality of device deep indicators in the set of example device data streams, and for each identification result in each level, determining a traffic identification feature vector from the plurality of device deep indicators according to the identification result corresponding to the corresponding decision unit in the corresponding level, generating a traffic detection network corresponding to the corresponding decision unit and including the determined traffic identification feature vector, and simultaneously establishing a sub-decision unit connected with the corresponding decision unit in the identification framework;

[0076] Step S40: when the multi-level identification performed again stops, taking the corresponding identification framework as the traffic identification framework.

[0077] In step S10, the data processing apparatus first needs to obtain a set of example device data streams of an example device set. The example device set is a group of representative Internet of Things device samples, covering various types and functions of devices, and the set of example device data streams contains various data generated by these devices during operation.

[0078] These data can be divided into device basic data and device deep data, corresponding to multiple device basic indicators and multiple device deep indicators respectively. Device basic indicators are some basic features that are relatively easy to obtain and analyze, such as the communication frequency of the device, the packet size, the startup time of the device, etc. Device deep indicators involve more complex and professional features of the device, such as for a smart camera, its deep indicators may include the resolution, frame rate, color mode of the image, etc.; for a smart temperature and humidity sensor, the deep indicators may include the accuracy, measurement range of the sensor, etc.

[0079] In order to obtain these data, the data processing apparatus can be implemented in various ways. One possible way is to install data collection software or hardware modules on the example devices, which can monitor the running state of the devices in real time and send the relevant data to the data processing apparatus.

[0080] In step S20, after obtaining the set of example device data streams, the data processing apparatus performs multi-level identification on the set of example devices according to the device basic data therein. Multi-level identification is a process of gradual refinement, and each level of identification will further divide the device set into different subsets according to specific characteristics. For example, in the first level of identification, the data processing apparatus can divide the set of example devices into a subset of devices using TCP protocol and a subset of devices using UDP protocol according to the communication protocol type of the devices.

[0081] For each identification result in each level, the data processing apparatus determines a traffic identification feature vector according to the identification result of the corresponding decision unit. The traffic identification feature vector is a set of feature vectors that can effectively distinguish different device categories. For example, in a certain level of identification, it is found that different types of devices can be better distinguished according to the packet size of the device, so the packet size can be used as a component of the traffic identification feature vector corresponding to the decision unit of this level.

[0082] After determining the traffic identification feature vector, the data processing apparatus generates a traffic detection network corresponding to the corresponding decision unit. For example, as mentioned earlier, a machine learning algorithm such as decision tree algorithm can be used to construct the traffic detection network. The decision tree algorithm constructs a decision tree by learning from the training data (i.e. the example device basic data), each internal node represents a test on an attribute (i.e. a feature in the traffic identification feature vector), each branch represents the output of the test, and each terminal node represents a device category.

[0083] Simultaneously, the data processing device constructs sub-decision units connected to the corresponding decision units within the identification framework. These sub-decision units further subdivide equipment categories based on the current decision unit. To achieve multi-level identification and determine the traffic identification representation vector, the data processing device can employ a feature selection algorithm. The purpose of the feature selection algorithm is to select the most representative and discriminative features from numerous basic equipment indicators as the traffic identification representation vector. One feasible feature selection algorithm is the information gain algorithm, which evaluates the contribution of each feature to equipment classification by calculating its information gain. The formula for calculating information gain is as follows:

[0084] ;

[0085] Where S represents the current set of devices, A represents a basic metric for a certain device, Values(A) represents all values ​​of metric A, Sv represents the subset of devices when metric A takes the value v, and Entropy(S) represents the entropy of the device set S, calculated using the following formula:

[0086] ;

[0087] Where n represents the number of equipment categories, p i This represents the proportion of devices belonging to the i-th class in the device set S.

[0088] In step S30, after the multi-level identification based on the basic device indicators stops, the data processing device will start from the terminal identification unit of the obtained identification framework and perform another multi-level identification on the corresponding example decision unit device set based on the example device depth data. This step is to further utilize the device's depth features to perform a more refined classification of the devices based on the basic identification.

[0089] For example, in the previous identification process, devices have already been initially classified into different categories based on basic indicators such as communication protocols and data packet size. Now, for the set of devices corresponding to each terminal identification unit, the data processing device will perform further multi-level identification based on device depth indicators. For example, for a set of smart camera devices, further classification will be performed based on depth indicators such as image resolution and frame rate.

[0090] During each stage of the identification process, the data processing device also determines the traffic identification representation vector from various device depth indicators based on the identification results of the corresponding decision unit. For example, when re-identifying a smart camera, if it is found that the combination of image frame rate and resolution can better distinguish different models and uses of cameras, then frame rate and resolution can be used as the traffic identification representation vector corresponding to that decision unit.

[0091] Then, the data processing apparatus generates a traffic detection network corresponding to the corresponding decision unit, which contains the determined traffic identification feature vector. Machine learning algorithms such as support vector machine (SVM) algorithm can also be used to build the traffic detection network. The SVM algorithm separates different categories of device data by finding an optimal hyperplane, thereby achieving classification of devices.

[0092] Finally, the data processing apparatus assembles the sub-decision units connected to the corresponding decision unit in the identification framework. For example, for smart cameras, different sub-decision units can be built according to different value ranges of frame rate and resolution, such as high frame rate and high resolution camera subset, low frame rate and low resolution camera subset, etc.

[0093] In step S40, when the re-multistage identification based on the device depth index stops, the data processing apparatus takes the identification framework obtained at this time as the final traffic identification framework. This traffic identification framework has undergone preliminary identification based on device basic indicators and fine identification based on device depth indicators, and can more accurately classify and identify Internet of Things devices.

[0094] Through steps S10 to S40, the traffic identification framework is built, which combines the basic indicators and depth indicators of the device, and can comprehensively and carefully identify and classify Internet of Things devices, laying a foundation for efficient operation and management of the Internet of Things system.

[0095] As an implementation, in step S20, the example device set is identified in multiple stages according to the example device basic data corresponding to the multiple device basic indicators in the example device data stream set. For each identification result in each stage, the traffic identification feature vector is determined from the multiple device basic indicators according to the identification result corresponding to the corresponding decision unit in the corresponding stage, a traffic detection network corresponding to the corresponding decision unit is generated, which includes the determined traffic identification feature vector, and the sub-decision units connected to the corresponding decision unit in the identification framework are assembled, including:

[0096] Step S21: When generating the identification framework based on the example device basic data in the example device data stream set, the decision units in the newly generated level are walked through one by one, and the walked decision units are taken as the walking basic decision units.

[0097] In generating the recognition framework based on the example device base data, the data processing apparatus needs to analyze the example device base data in the example device data stream set, which contains various basic feature information of the device, such as the communication protocol type, data packet size, communication frequency, etc. of the device. The newly generated level refers to the hierarchical structure newly generated in the process of building the recognition framework, with the gradual analysis and division of the device features. Each level contains a number of decision units, which are nodes in the recognition framework, representing the judgment point of classifying the device according to a certain specific feature.

[0098] For example, in building the recognition framework of smart home devices, the first layer of decision units may be generated according to the communication protocol type of the device, such as TCP protocol devices and UDP protocol devices. When further subdivision of device categories is needed, a new level will be generated, and the decision units in the new level may be divided according to the application layer protocol type, data packet size, and other features of the device.

[0099] The data processing apparatus traverses the decision units in the newly generated level one by one, completing the traversal operation of the tree structure. The feasible traversal algorithms include depth-first search (DFS) and breadth-first search (BFS).

[0100] Depth-first search is an algorithm that traverses along the depth direction of the tree. It starts from the starting node, visits the nodes as deeply as possible, until it cannot continue or reaches the target node, then backtracks to the previous step and continues to explore other paths. Taking the construction of the smart home device recognition framework as an example, suppose the current newly generated level has three decision units corresponding to different communication protocol types (TCP, UDP, HTTP). When the data processing apparatus uses the depth-first search algorithm to traverse, it may first select the decision unit corresponding to the TCP protocol for access, taking it as the traversal base decision unit. Then, after completing the related operations of this decision unit, it further accesses the child decision units related to this decision unit in the next level, until it cannot continue to deepen, then backtracks to the previous level and selects the decision unit corresponding to the UDP protocol for access, and so on, until all decision units are accessed.

[0101] The breadth-first search is to access all neighbor nodes of the starting node first, and then access the neighbor nodes of these neighbor nodes in turn, and so on. In the example of the above smart home device identification framework, when the data processing apparatus uses the breadth-first search algorithm for walking, it will first access the decision units corresponding to the TCP, UDP, and HTTP protocols in turn, and then access all decision units in the next level until all decision units are accessed. Subsequent related concepts will not be described again.

[0102] Through such a walking process one by one, the data processing apparatus can comprehensively analyze each decision unit in the newly generated level, laying a foundation for subsequent identification framework construction and device identification work.

[0103] When the data processing apparatus determines the walking base decision unit in step S21, it needs to find the example decision unit device set corresponding to it. The example decision unit device set refers to the set of devices in the example device set that meet the characteristics or conditions represented by the walking base decision unit.

[0104] For example, in a smart home Internet of Things system, assume that the walking base decision unit is a decision unit determined based on the device communication protocol TCP. The example device set contains various smart devices, such as smart cameras, smart temperature and humidity sensors, smart door locks, etc. The data processing apparatus will traverse each device in the example device set and check the device's communication protocol information. If the communication protocol of a device is TCP protocol, then the device belongs to the example decision unit device set corresponding to this walking base decision unit.

[0105] To achieve this process, the data processing apparatus can use a database query implementation. Assume that the relevant information of the example device set is stored in a relational database, which has a table for storing the basic information of the device, including device ID, device type, communication protocol, etc. The data processing apparatus can use Structured Query Language (SQL) to perform query operations.

[0106] After determining the example decision unit device set, the data processing apparatus obtains the example decision unit data set corresponding to the walking base decision unit from the example device data stream set. The example decision unit data set contains the relevant data of each device in the example decision unit device set, which is the basis for further analysis and processing.

[0107] Continuing with the example of the smart home IoT system, for a given set of TCP protocol devices (i.e., the example decision unit device set), the data processing apparatus needs to obtain various data of these devices from the example device data stream set, such as the device's packet size, communication frequency, energy consumption, etc. The example device data stream set can be stored in a distributed file system or a database, and the data processing apparatus can match and obtain the corresponding data according to the device ID or other unique identifier.

[0108] Step S23: Based on the example device basic data in the example decision unit data set corresponding to the wandering basic decision unit, identify the example decision unit device set corresponding to the wandering basic decision unit, and select the traffic identification feature vector corresponding to the wandering basic decision unit from the multiple device basic indicators based on the identification result.

[0109] When the data processing apparatus obtains the example decision unit data set corresponding to the wandering basic decision unit, the example device basic data contained therein is an important basis for device identification. The purpose of device identification is to find the feature differences and regularities between these devices in order to classify them more meticulously.

[0110] For example, in a smart traffic IoT system, assume that the example decision unit device set corresponding to the wandering basic decision unit contains various types of traffic sensor devices, such as speed sensors, traffic volume sensors, and road humidity sensors, etc. The example device basic data may include indicators such as sensor sampling frequency, data transmission interval, data accuracy, etc. The data processing apparatus will analyze these basic data and try to find features that can distinguish different types of sensors. One feasible implementation is the data analysis and statistical method. The data processing apparatus can calculate the statistics of various device basic indicators, such as mean, variance, standard deviation, etc., to understand the distribution and characteristics of the data. For example, by calculating the mean and variance of the sampling frequency of different sensors, it can be found that the mean of the sampling frequency of the speed sensor may be higher and the variance may be smaller, because the change of the speed is relatively stable and requires a higher sampling frequency to accurately measure; while the mean of the sampling frequency of the traffic volume sensor may be relatively lower and the variance may be larger, because the change of the traffic volume has greater randomness. The data processing apparatus can also use data visualization techniques to assist the identification process. For example, drawing histograms, box plots, and other statistical charts of different device basic indicators to visually observe the distribution of the data. Through these charts, the differences between different types of devices in certain indicators can be more clearly found. For example, when drawing the histograms of the data accuracy indicators of the speed sensor and the traffic volume sensor, it may be found that the data accuracy distribution of the speed sensor is more concentrated, while the data accuracy distribution of the traffic volume sensor is relatively more dispersed.

[0111] After identifying the set of example decision unit devices, the data processing apparatus needs to select appropriate indicators from the plurality of device base indicators as the traffic recognition feature vector corresponding to the wandering base decision unit according to the identification result. The traffic recognition feature vector is a feature vector that can effectively distinguish different device categories, which is of great significance for subsequent construction of accurate traffic detection network and further refinement of the recognition framework.

[0112] There are many methods to select the traffic recognition feature vector. One is to select based on the distinguishing ability of the feature. The data processing apparatus can calculate the distinguishing degree indicators of each device base indicator, such as information gain, Gini index, etc., to evaluate the contribution of the indicator to device classification. Information gain refers to the degree of reduction in uncertainty of class information after knowing the value of a certain feature. The greater the information gain, the greater the contribution of the feature to classification.

[0113] Taking sensor devices in intelligent transportation systems as an example, suppose the device base indicators include sampling frequency, data transmission interval and data precision. The data processing apparatus can calculate the information gain of these three indicators respectively. Suppose the information gain of the sampling frequency is 0.8, the information gain of the data transmission interval is 0.6, and the information gain of the data precision is 0.4. According to the size of the information gain, the data processing apparatus can select the sampling frequency as an important part of the traffic recognition feature vector, because it has a greater contribution to distinguishing different types of sensor devices. The formula for calculating the information gain is referred to the foregoing description, which is not repeated here.

[0114] Another selection method is based on domain knowledge and experience. In some cases, according to the understanding of Internet of Things devices and past experience, some base indicators that are important for device classification can be directly determined as the traffic recognition feature vector.

[0115] Through this process, the data processing apparatus can accurately identify the set of example decision unit devices based on the example device base data, and select appropriate traffic recognition feature vectors, providing strong support for subsequent recognition framework construction and device classification.

[0116] Step S24: generating a plurality of sub-decision units connected with the wandering base decision unit, and generating a traffic detection network including a traffic recognition feature vector corresponding to the wandering base decision unit; the generated traffic detection network includes hidden Markov models associated with the plurality of sub-decision units connected with the wandering base decision unit.

[0117] In the construction process of the identification framework, in order to classify the devices more specifically, the data processing apparatus needs to further divide the example decision unit device set corresponding to the wandering basic decision unit according to the determined traffic identification feature vector, thereby generating multiple sub decision units. The sub decision unit is a decision node that classifies devices according to more specific features or conditions on the basis of the wandering basic decision unit.

[0118] For example, in an Internet of Things system of a smart factory, the example decision unit device set corresponding to the wandering basic decision unit is a group of production devices, and the traffic identification feature vector is determined as the energy consumption index of the devices. The data processing apparatus can generate multiple sub decision units according to different value ranges of the energy consumption index. Assuming that the value range of the energy consumption index is between 0-100kW, the data processing apparatus can divide it into three subintervals: 0-30kW, 30-70kW and 70-100kW, respectively corresponding to low energy consumption device sub decision unit, medium energy consumption device sub decision unit and high energy consumption device sub decision unit.

[0119] To achieve this division, the data processing apparatus can use the implementation of cluster analysis. Cluster analysis is a method of grouping data objects into multiple classes or clusters, so that data objects in the same cluster have high similarity, while data objects in different clusters have large differences. Feasible clustering algorithms include K-Means algorithm, DBSCAN algorithm, etc. Taking the K-Means algorithm as an example, it divides data points into K clusters, where K is the number of clusters specified in advance. The basic idea of the algorithm is to randomly select K initial centroids, then assign each data point to the cluster with the nearest centroid, then update the centroid of each cluster, and repeat this process until the centroid no longer changes or the predetermined number of iterations is reached.

[0120] In the above example of a smart factory, the data processing apparatus can use the K-Means algorithm to perform cluster analysis on the energy consumption data of the production devices, dividing them into different clusters, each corresponding to a sub decision unit. The specific steps are as follows:

[0121] 1. Determine the number of clusters K, here K=3.

[0122] 2. Randomly select 3 energy consumption data of production devices as initial centroids.

[0123] 3. Calculate the distance of each production device's energy consumption data to the 3 centroids, and assign it to the cluster with the nearest centroid.

[0124] 4. Update the centroid of each cluster, that is, calculate the mean of all data points in each cluster as the new centroid.

[0125] 5. Repeat steps 3 and 4 until the centroid no longer changes or a predetermined number of iterations is reached.

[0126] The implementation of constructing a hidden Markov model usually includes model training and parameter estimation. One feasible method is to use the Baum-Welch algorithm, which is an iterative algorithm based on the expectation maximization (EM) algorithm for estimating the parameters of a hidden Markov model. The basic idea of the Baum-Welch algorithm is to adjust the parameters of the model iteratively through iterative calculation, so that the probability of the observed sequence under the model is maximized. Specifically, the steps of the Baum-Welch algorithm are as follows:

[0127] 1. Initialize the parameters of the model, including the state transition probability matrix and the observation probability matrix, etc.

[0128] 2. Calculate the forward probability and the backward probability. The forward probability represents the probability of observing part of the observation sequence and the hidden state being s_i at time t given the model parameters; the backward probability represents the probability of observing part of the observation sequence from time t+1 to the end time given the model parameters.

[0129] 3. Calculate the expectation based on the forward probability and the backward probability, i.e., calculate the expected value of the model parameters given the observation sequence.

[0130] 4. Update the parameters of the model, using the calculated expectation to update the state transition probability matrix and the observation probability matrix, etc.

[0131] 5. Repeat steps 2-4 until the parameters of the model converge or a predetermined number of iterations is reached.

[0132] By generating multiple sub-decision units connected to the wandering basic decision unit and constructing a traffic detection network containing traffic identification feature vectors and associated hidden Markov models, the data processing device can more accurately classify and identify devices, improving the performance and reliability of the identification framework.

[0133] Step S25: Based on the hidden Markov models associated with the multiple sub-decision units connected to the wandering basic decision unit, identify the set of example decision unit devices corresponding to the wandering basic decision unit, and obtain the set of example decision unit devices corresponding to each sub-decision unit connected to the wandering basic decision unit.

[0134] Specifically, a Hidden Markov Model (HMM) includes key elements such as a hidden state sequence, an observation sequence, a state transition probability matrix, and an observation probability matrix. The hidden state sequence represents the actual state of the device at different times, such as the normal operating state or the fault state; the observation sequence is the device data that can be directly obtained; the state transition probability matrix describes the probability of the device transitioning from one hidden state to another; and the observation probability matrix represents the probability of observing a specific observation in a certain hidden state.

[0135] Once the data processing device acquires the set of example decision-making unit devices corresponding to the walking basic decision-making unit, it inputs the relevant data of each device as an observation sequence into the Hidden Markov Model associated with each sub-decision-making unit. Taking an intelligent warehousing system as an example, assume that the set of example decision-making unit devices corresponding to the walking basic decision-making unit includes various types of warehousing equipment, such as automated guided vehicles (AGVs), rack storage systems, and intelligent sorting robots. In step S24, three sub-decision-making units have been generated based on the equipment's operating efficiency indicators: a high-operating-efficiency equipment sub-decision-making unit, a medium-operating-efficiency equipment sub-decision-making unit, and a low-operating-efficiency equipment sub-decision-making unit, and each sub-decision-making unit is associated with a Hidden Markov Model.

[0136] For each device in the example decision unit device set, the data processing device inputs its operational efficiency-related data, such as the amount of goods handled per unit time and task completion time, as an observation sequence into the three Hidden Markov Models. Each Hidden Markov Model calculates the probability of the device being in each hidden state given the observation sequence based on its own parameters.

[0137] The specific computation process can use the forward algorithm. Assume the hidden state set of the Hidden Markov Model is... The observation sequence is The state transition probability matrix is ​​A, where Indicates from state Transition to state The probability of ; the observation probability matrix is ​​B, where Indicates the state The observed values The probability of the initial state is: ,in Indicates the initial state. The probability of.

[0138] The computation steps of the forward algorithm are as follows:

[0139] 1. Initialization: Calculate the forward probability at the initial time step. ,in denotes the hidden state at time t, and and the observation sequence .

[0140] 2. Recursion: for , compute .

[0141] 3. Termination: the probability of the observation sequence is , where denotes the parameters of the hidden Markov model.

[0142] Through the forward algorithm, each hidden Markov model can calculate the probability of the given device observation sequence. The data processing apparatus compares the probability values of the device under the hidden Markov model associated with each sub-decision unit, and assigns the device to the example decision unit device set corresponding to the sub-decision unit with the largest probability value.

[0143] Through the above identification process, the data processing apparatus can ultimately obtain the example decision unit device set corresponding to each sub-decision unit connected to the walk basic decision unit. Such division makes the identification framework more refined and accurate, and can better reflect the differences and characteristics between devices.

[0144] Step S26: After walking through all the decision units in the newly generated level, determine the next level of the newly generated level as the new newly generated level, and jump to the process of iteratively performing the one-by-one walking of the decision units in the new generated level. Stop the iteration when the example device basic data in the example device data stream set cannot generate a new decision unit.

[0145] When the data processing apparatus walks through the decision units in the newly generated level one by one, it records the walking state of each decision unit. When all decision units are accessed and the corresponding operations are completed, the data processing apparatus determines that the walking of the decision units in the newly generated level is completed. After completing the walking of the current level, the data processing apparatus determines the next level of the newly generated level as the new newly generated level.

[0146] The data processing apparatus jumps to the process of iteratively performing the one-by-one walking of the decision units in the new generated level, which continuously refines the identification framework to make it more accurate in classifying and identifying devices. As the iteration proceeds, the identification framework gradually delves into more specific feature levels of the devices, thereby improving the accuracy of identification.

[0147] In each iteration, the data processing apparatus repeats the operations of steps S21-S25. For example, in the second layer decision unit walk process of the smart medical Internet of Things system, for the vital sign monitoring device decision unit, the data processing apparatus, as the walk base decision unit, determines the example decision unit device set and data set corresponding thereto according to step S21, identifies the device set according to step S23, selects the traffic identification feature vector according to step S24, and finally identifies the device set by using the hidden Markov model according to step S25 to obtain the example decision unit device set corresponding to each sub decision unit.

[0148] When a new decision unit cannot be generated according to the example device base data in the example device data stream set, the iteration stops. This means that the example device base data cannot provide sufficient information to further subdivide the device categories under the current identification framework. Whether a new decision unit can be generated can be implemented in various ways.

[0149] A feasible method is to evaluate whether the newly generated decision unit has sufficient discrimination. For example, the feature similarity of the devices in the example decision unit device set corresponding to the new decision unit is calculated. If the feature similarity between the devices is high, that is, the new decision unit cannot effectively further distinguish the devices, it is considered that a new decision unit cannot be generated. Similarity metrics can be used to evaluate the feature similarity, such as cosine similarity. When the cosine similarity is close to 1, it means that the features of the two devices are very similar.

[0150] Another method is to check the size of the example decision unit device set corresponding to the new decision unit. If the number of devices in the set is too small to constitute a meaningful category, it is also considered that a new decision unit cannot be generated. For example, a minimum device number threshold N min is set. When the number of devices in the device set is less than N min , the iteration stops.

[0151] Through such an iteration and stopping mechanism, the data processing apparatus can construct an identification framework with reasonable hierarchical structure and accurate classification, providing a solid foundation for subsequent Internet of Things device identification and management.

[0152] As an implementation, step S23 includes identifying the example decision unit device set corresponding to the walk base decision unit based on the example device base data in the example decision unit data set corresponding to the walk base decision unit, and selecting the traffic identification feature vector corresponding to the walk base decision unit from the plurality of device base indicators based on the identification result.

[0153] Step S231: obtaining a target device basic index set, wherein the target device basic index set comprises target device basic indexes selected from a plurality of device basic indexes;

[0154] Step S232: identifying an example device set corresponding to the example decision unit based on the target device basic index set and example device basic data in the example decision unit data set corresponding to the example decision unit, and selecting a traffic identification feature vector corresponding to the example decision unit from the target device basic index set based on the identification result.

[0155] In step S231, the data processing apparatus selects representative and distinguishing target device basic indexes from a plurality of device basic indexes to form a target device basic index set. Device basic indexes are various parameters that describe the basic characteristics of a device. However, not all device basic indexes are equally important for the classification and identification of devices, so they need to be selected.

[0156] The data processing apparatus can select target device basic indexes in various ways. One possible method is to select based on the statistical properties of the data. For example, the variance of each device basic index can be calculated. A larger variance usually means that the index has a larger difference between different devices, and is more likely to be an effective feature for distinguishing devices. The formula for calculating variance is:

[0157] ;

[0158] wherein, denotes the variance, denotes the i-th data point, denotes the mean of the data, and n denotes the number of data points.

[0159] In addition to the variance-based selection method, the correlation between indexes can also be considered. If the correlation between two indexes is high, they may provide similar information, in which case one of the indexes can be selected as a target device basic index to avoid information redundancy. Correlation can be measured by calculating the correlation coefficient, such as the Pearson correlation coefficient. For two variables X and Y, the formula for calculating the Pearson correlation coefficient is:

[0160] ;

[0161] wherein, denotes the Pearson correlation coefficient, denotes the i-th data point of variables X and Y, respectively, denotes the mean of variables X and Y, respectively, and n denotes the number of data points.

[0162] By calculating the correlation coefficients between different device base indicators, the data processing apparatus can remove indicators with high correlation, retain independent and representative indicators, and form a target device base indicator set.

[0163] In step S232, after obtaining the target device base indicator set, the data processing apparatus uses these indicators and the example device base data in the example decision unit data set to identify the example decision unit device set. The purpose of identification is to find the characteristics that can most effectively distinguish different devices, i.e., the traffic identification representation vector.

[0164] One implementation is a classification algorithm, such as a decision tree algorithm, where each internal node represents a test on an attribute (i.e., a target device base indicator), each branch represents the output of the test, and each terminal node represents a device class. The data processing apparatus first constructs a training data set according to the target device base indicator set and the example device base data, where each data sample contains the values of the target device base indicators and the corresponding device class label.

[0165] The data processing apparatus uses the decision tree algorithm to learn the training data set and constructs a decision tree. In the construction process, the algorithm selects the best split attribute according to the classification ability of each target device base indicator. For example, first calculate the information gain when the sampling frequency is the split attribute, and the formula for calculating the information gain is:

[0166] ;

[0167] where S represents the current data set, A represents the attribute (i.e., the target device base indicator), Values(A) represents all values of attribute A, Sv represents the data set subset when attribute A takes value v, and Entropy(S) represents the entropy of data set S, and the calculation formula is:

[0168] ;

[0169] where n represents the number of device classes, and pi represents the proportion of the i-th device in the data set S.

[0170] Calculate the information gain when the sampling frequency is the split attribute:

[0171] First, calculate the entropy of the data set:

[0172] ;

[0173] When the sampling frequency is the split attribute, there are three values: 1Hz, 2Hz, and 3Hz. Calculate the corresponding subset entropy:

[0174] For the subset S1 with a sampling frequency of 1 Hz, containing 1 sample, belonging to category A, the entropy is ;

[0175] For the subset S2 with a sampling frequency of 2 Hz, containing 1 sample, belonging to category B, the entropy is ;

[0176] For the subset S3 with a sampling frequency of 3 Hz, containing 1 sample, belonging to category C, the entropy is ;

[0177] The information gain when the sampling frequency is used as the division attribute is:

[0178] ;

[0179] Similarly, the information gain when the data precision is used as the division attribute can be calculated. Suppose that the information gain when the data precision is used as the division attribute is 1.2.

[0180] Since the information gain when the sampling frequency is used as the division attribute is larger, it indicates that the sampling frequency has a stronger classification ability on the data, and thus the first internal node of the decision tree selects the sampling frequency as the division attribute.

[0181] By continuously selecting the best division attribute to construct the decision tree, the data processing apparatus finally obtains a decision tree that can effectively classify the set of example decision unit devices. After the decision tree is constructed, the data processing apparatus selects the index with the largest contribution to classification from the target device basic index set as the traffic recognition representation vector corresponding to the wandering basic decision unit according to the structure and node information of the decision tree. For example, in the above example, if the main nodes and key branch nodes of the decision tree are mainly divided based on the sampling frequency, the sampling frequency can be used as the traffic recognition representation vector.

[0182] As an implementation manner, the target device basic index set is determined by the following steps:

[0183] Step S201: determining the numerical fluctuation coefficients corresponding to a plurality of device basic indexes based on the set of example device data streams;

[0184] Step S202: removing the device basic indexes corresponding to the target numerical fluctuation coefficients from the plurality of device basic indexes to obtain a first transition device basic index set; the target numerical fluctuation coefficient is a numerical fluctuation coefficient not less than a preset coefficient threshold value;

[0185] Step S203: Based on the set of example device data streams, determine the cumulative frequency distribution corresponding to each device base index in the first transition device base index set respectively, and remove the device base index corresponding to the cumulative frequency distribution with a single change direction from the first transition device base index set, to obtain the second transition device base index set.

[0186] Step S204: Perform bimodal cross-point density detection on the cumulative frequency distribution corresponding to each device base index in the second transition device base index set respectively, to remove the device base index corresponding to the first target cumulative frequency distribution from the second transition device base index set, to obtain the target device base index set; the first target cumulative frequency distribution is the cumulative frequency distribution with a bimodal cross-point density not less than a first preset density threshold.

[0187] In step S201, the numerical fluctuation coefficient is used to measure the dispersion degree of the device base index data, which reflects the fluctuation of the data around its mean value. After obtaining the set of example device data streams, the data processing device calculates the numerical fluctuation coefficient of each device base index contained therein.

[0188] A feasible method for calculating the numerical fluctuation coefficient is to calculate the standard deviation. The standard deviation is the square root of the variance, and the calculation formula of the variance is:

[0189] ;

[0190] Wherein, denotes the variance, denotes the i-th data point, denotes the mean value of the data, and n denotes the number of data points. The standard deviation is the square root of the variance, i.e.:

[0191] ;

[0192] By calculating the numerical fluctuation coefficients of the plurality of device base indexes, the data processing device can understand the dispersion of each index data, providing a basis for subsequent selection of indexes.

[0193] The preset coefficient threshold is a pre-set threshold value for judging whether the numerical fluctuation of the device base index is too large. The data processing device compares the size relationship between the numerical fluctuation coefficient of each device base index and the preset coefficient threshold, removes the device base index with a numerical fluctuation coefficient not less than the preset coefficient threshold from the plurality of device base indexes, and the remaining device base indexes constitute the first transition device base index set.

[0194] The purpose of this is to remove those device base indicators with too large data fluctuations, as these indicators may be affected by more noise or abnormal factors, which is not conducive to subsequent accurate identification of devices.

[0195] The cumulative frequency distribution refers to arranging data in ascending order and calculating the cumulative frequency corresponding to each data point. The data processing apparatus will calculate the cumulative frequency distribution of each device base indicator in the first transition device base indicator set based on the data in the example device data stream set.

[0196] For example, for the temperature setting value of the intelligent air conditioner in the intelligent office environment, the data processing apparatus collects a series of temperature setting value data, such as [22℃, 23℃, 22℃, 24℃, 23℃, 25℃]. After sorting these data in ascending order, we get [22℃, 22℃, 23℃, 23℃, 24℃, 25℃]. Calculate the cumulative frequency of each data point, the cumulative frequency of the first data point 22℃ is 2 / 6≈0.33, the cumulative frequency of the second data point 23℃ is 4 / 6≈0.67, and so on. The cumulative frequency distribution with a single change direction means that the data of the device base indicator shows a monotonic increasing or monotonic decreasing trend, which may not provide enough distinguishing information. The data processing apparatus will check the cumulative frequency distribution of each device base indicator, remove the device base indicators corresponding to the cumulative frequency distribution with a single change direction from the first transition device base indicator set, and the remaining device base indicators constitute the second transition device base indicator set.

[0197] For example, if the cumulative frequency distribution of a certain device base indicator is always monotonically increasing, it means that the data of this indicator lacks diversity of change and may not be useful for distinguishing devices, so it will be removed.

[0198] The double-peak cross-point density detection is used to analyze the density of double-peak cross-points in the cumulative frequency distribution curve. The data processing apparatus will analyze the cumulative frequency distribution curve of each device base indicator in the second transition device base indicator set and calculate the density of double-peak cross-points.

[0199] For example, for the brightness adjustment range of the intelligent lighting device, its cumulative frequency distribution curve may exhibit a double-peak shape. The data processing apparatus detects the double-peak cross-points in the curve through a specific algorithm and calculates the density of these cross-points.

[0200] The first preset density threshold is a preset threshold for determining whether the bimodal intersection density is too high. If the bimodal intersection density of the cumulative frequency distribution of a certain device basis index is not less than the first preset density threshold, it indicates that the data distribution of the index is too complex or unstable, which may interfere with device identification. The data processing apparatus will remove the device basis index corresponding to such a first target cumulative frequency distribution from the second transition device basis index set, and the remaining device basis indexes form the target device basis index set.

[0201] For example, assuming that the first preset density threshold is 0.5, and the bimodal intersection density of the cumulative frequency distribution of a certain device basis index is 0.6, which is greater than the first preset density threshold, then the index will be removed.

[0202] After obtaining the target device basis index set, the data processing apparatus can further verify its effectiveness. One verification method is to use cross-validation techniques. Divide the example device data stream set into a training set and a validation set, use the target device basis index set to train a classification model on the training set, and then verify it on the validation set to evaluate the performance indicators of the model, such as accuracy, recall rate, F1 value, etc.

[0203] For example, a decision tree classification model is constructed using the target device basis index set. After training on the training set, prediction is performed on the validation set. Calculate the accuracy of the prediction result. If the accuracy reaches the expected threshold, it indicates that the target device basis index set is effective; otherwise, the determination process of the index set needs to be rechecked and adjusted.

[0204] As an implementation, the identification result includes multiple identification sub-results; in step S232, based on the target device basis index set and the example device basis data in the example decision unit data set corresponding to the wandering basis decision unit, the example decision unit device set corresponding to the wandering basis decision unit is identified, and the traffic identification feature vector corresponding to the wandering basis decision unit is selected from the target device basis index set based on the identification result, including:

[0205] Step S2321: starting from a single feature, each target device basis index in the target device basis index set is fused to obtain multiple fused device basis indexes at the current analysis level; each fused device basis index at the current analysis level includes a preset number of device basis indexes corresponding to the current analysis level.

[0206] Here, a single feature refers to a single device base indicator in the target device base indicator set. For example, assuming that the target device base indicator set contains multiple indicators such as "average power consumption", "data transmission rate", "response time", "working temperature", etc. of a device, "average power consumption" is one of the single features. The data processing apparatus starts the fusion operation from such a single feature. Fusion refers to combining multiple target device base indicators to form a new comprehensive indicator, i.e., a fused device base indicator. Each fused device base indicator at the current analysis level contains a preset number of device base indicators corresponding to the current analysis level. For example, if the preset number is 3, the data processing apparatus will select 3 different indicators from the target device base indicator set for fusion.

[0207] For example, the following gives a specific example of the fusion process. Assuming that the target device base indicator set is {Indicator A (average power consumption), Indicator B (data transmission rate), Indicator C (response time), Indicator D (working temperature), Indicator E (storage capacity)}, and the preset number is 3. The data processing apparatus will first select Indicator A, Indicator B, and Indicator C for fusion. This fusion can be simply understood as merging or associating the relevant data of the three indicators according to certain rules. For example, fusion can be achieved by simple addition or weighted summation. Assuming that the fusion method is weighted summation, the weight of Indicator A is set to 0.3, the weight of Indicator B is set to 0.4, and the weight of Indicator C is set to 0.3. For the relevant data of a device, the value of Indicator A is a, the value of Indicator B is b, and the value of Indicator C is c. The value of the fused device base indicator after fusion can be calculated by the formula: fused value = 0.3 * a + 0.4 * b + 0.3 * c.

[0208] After completing this set of fusion, the data processing apparatus will continue to select other combinations, such as Indicator A, Indicator B, and Indicator D for fusion, and calculate the fused value according to the weighted summation method described above. Then, there are combinations such as Indicator A, Indicator B, and Indicator E, Indicator A, Indicator C, and Indicator D, etc. until all possible combinations of selecting a preset number of indicators from the target device base indicator set are traversed.

[0209] In actual implementation, the data processing apparatus can use a loop-nested algorithm to implement this fusion operation. For example, a multiple loop structure in a programming language is used.

[0210] Step S2322: For each fused device base indicator in the plurality of fused device base indicators at the current analysis level, select the example device base data of the fused device base indicator from the example decision unit device base data corresponding to the wandering base decision unit.

[0211] In step S2321, the data processing apparatus has selected a preset number of indexes from the target device basic index set for fusion, and obtained a plurality of fused device basic indexes. These fused device basic indexes constitute the index set at the current analysis level. For example, assume that the fused device basic indexes at the current analysis level include fused index 1 (fused from indexes A, B, and C) and fused index 2 (fused from indexes A, B, and D).

[0212] The example decision unit device basic data set corresponding to the wandering basic decision unit refers to the basic data set of the example device associated with a specific wandering basic decision unit. This data set contains data records of various device basic indexes of a plurality of example devices. It can be imagined as a two-dimensional table, with rows representing different example devices and columns representing different device basic indexes. For example, assume that there is a data set containing 100 example devices, and each device has data records of indexes A, B, C, D, and E. This data set is an example decision unit device basic data set.

[0213] When performing this step, the data processing apparatus processes each fused device basic index at the current analysis level one by one. For each fused device basic index, it searches for the corresponding example device basic data in the example decision unit device basic data set. The specific searching and selecting process can be realized through data matching and indexing operations.

[0214] The following is illustrated by a specific example. Assume that there is a fused device basic index at the current analysis level, which is fused index 1 (fused from indexes A, B, and C), and the example decision unit device basic data set corresponding to the wandering basic decision unit is shown in Table 1:

[0215] Table 1-Example decision unit device basic data set

[0216] The data processing apparatus searches for records containing index A, B, and C data in this data set. For example device No. 1, the value of index A is 10, the value of index B is 20, and the value of index C is 30, which together constitute part of the example device basic data corresponding to fused index 1. Similarly, for example devices No. 2 and No. 3, the corresponding index A, B, and C data can also be obtained, which together constitute the example device basic data of fused index 1.

[0217] Step S2323: identifying the selected example device basic data corresponding to the fused device basic index to obtain an identification sub-result corresponding to the fused device basic index.

[0218] In the specific operation process, the data processing apparatus uses various data analysis and pattern recognition techniques to process the example device basic data. Recognition refers to judging whether the example device basic data meets a certain preset rule or pattern according to its characteristics and patterns, so as to determine whether it can be associated with a specific device category or state.

[0219] For example, assuming that the selected fusion device basic index is fused by the three indexes of "average power consumption", "data transmission rate" and "response time" of the device, and the example device basic data obtained by the data processing apparatus is a set of specific values of the three indexes of a plurality of example devices. The data processing apparatus can recognize these data by statistical analysis method. A feasible implementation manner is to calculate the statistical characteristics of the data, such as mean, variance and standard deviation, to understand the distribution and dispersion of the data.

[0220] By calculating these statistical characteristics, the data processing apparatus can understand the overall situation of the example devices in the "average power consumption" index. For example, if the mean is small and the variance is also small, it means that the average power consumption of these example devices is relatively stable and low; if the mean is large and the variance is large, it means that the average power consumption of the example devices is large.

[0221] In addition to statistical analysis, the data processing apparatus can also use pattern matching method for recognition. Pattern matching refers to comparing the characteristics of the example device basic data with the preset pattern or template to judge whether it matches. For example, for the two indexes of "data transmission rate" and "response time", there may be a preset pattern, that is, when the data transmission rate is within a certain range and the response time is within another specific range, it represents that the device is in normal working state. The data processing apparatus can compare the "data transmission rate" and "response time" values of each example device with this preset pattern, and if it is consistent, it is considered that the example device meets the preset normal working state pattern.

[0222] In addition, the data processing apparatus can also use cluster analysis. For example device basic data, the data processing apparatus can use clustering algorithms such as K-Means clustering algorithm to divide the example devices into different clusters according to their fusion device basic index values. By observing the distribution and characteristics of the clusters, it is judged whether a clear classification pattern can be formed. For example, if the clustering result shows that most of the example devices are clearly divided into several different clusters, and the devices in each cluster have similar characteristics in the fusion device basic index, it can be considered that these data have identifiable patterns.

[0223] After the analysis and processing of the example device base data are completed, the data processing apparatus gives a recognition sub-result according to the analysis result. The recognition sub-result is a judgment on whether the fusion device base index can effectively identify the device. For example, if it is found through statistical analysis that the distribution of data is relatively chaotic and has no obvious rules, the result of pattern matching shows that most of the example devices do not conform to the preset mode, and the clustering analysis cannot form a clear cluster structure, then the data processing apparatus will give a recognition sub-result representing unsuccessful recognition. On the contrary, if the statistical characteristics show that the data has obvious rules, the compliance of pattern matching is high, and the clustering analysis forms a clear classification, then the data processing apparatus will give a recognition sub-result representing successful recognition.

[0224] Step S2324: When the recognition sub-results corresponding to the multiple fusion device base indexes at the current analysis level are all results representing unsuccessful recognition, the current analysis level is expanded to obtain a next level.

[0225] In step S2323, after the data processing apparatus identifies the example device base data of each fusion device base index, a corresponding recognition sub-result is obtained. This recognition sub-result reflects whether the fusion device base index can effectively identify the device. For example, the recognition sub-result can be represented by a Boolean value, "True" representing successful recognition and "False" representing unsuccessful recognition.

[0226] When the data processing apparatus checks and finds that the recognition sub-results corresponding to all the fusion device base indexes at the current analysis level are all results representing unsuccessful recognition, it means that the fusion device base indexes currently constructed cannot meet the demand for accurate identification of the device under the existing combination of characteristics. For example, assume that there are three fusion device base indexes at the current analysis level, namely fusion index 1 (fused by indexes A, B, and C), fusion index 2 (fused by indexes A, B, and D), and fusion index 3 (fused by indexes A, C, and D). After the data processing apparatus identifies the example device base data corresponding to these three fusion indexes, the obtained recognition sub-results are all "False", i.e., unsuccessful recognition. This may be because the current selected index combination cannot fully reflect the feature differences of the device, resulting in the inability to accurately distinguish different types of devices.

[0227] At this time, the data processing apparatus needs to expand the current analysis level to obtain a next level. Level expansion refers to increasing the number of device base indexes included in the fusion device base index, thereby increasing the dimension of analysis. For example, in the above example, the fusion device base index at the current analysis level is fused by 3 device base indexes, so after level expansion, the fusion device base index at the next level may be fused by 4 device base indexes.

[0228] To realize the hierarchical expansion, the data processing apparatus can employ a loop-nested algorithm to traverse all possible new combinations of indicators. Specifically, assuming that the original target device base indicator set is {Indicator A, Indicator B, Indicator C, Indicator D, Indicator E}, when the fusion device base indicator under the current analysis level is fused from 3 indicators, now it is expanded to 4 indicators fusion. The data processing apparatus can traverse all possible combinations of 4 indicators through a four-fold loop.

[0229] Through such a hierarchical expansion operation, the data processing apparatus constructs a new set of fusion device base indicators, i.e., the fusion device base indicators of the next level. These new fusion device base indicators contain more device base indicator information, which can provide richer feature representation, thereby improving the recognition ability of the device. For example, under the new level, there can be a fusion indicator (fused from indicators A, B, C, E), which can more accurately capture some key features of the device, so that the data processing apparatus can successfully recognize the device in the subsequent recognition process.

[0230] In summary, when the fusion device base indicator under the current analysis level cannot effectively recognize the device, the data processing apparatus expands the analysis level, increases the number of device base indicators in the fusion device base indicator, constructs a new set of fusion device base indicators, provides more possibilities for subsequent recognition operations, and helps to improve the accuracy and success rate of device recognition.

[0231] Step S2325: enter the next time recognition sub-result recognition process, determine the next level as the new current analysis level, jump to the process of iteratively performing fusion on each target device base indicator in the plurality of device base indicators, end the iteration when a recognition sub-result representing successful recognition is obtained, and select the traffic recognition feature vector corresponding to the wandering base decision unit from the target device base indicator set based on the recognition sub-result representing successful recognition.

[0232] When the data processing apparatus performs step S2325, it enters the next time recognition sub-result recognition process. First, the data processing apparatus determines the next level as the new current analysis level, which means it will continue the recognition operation based on the new set of fusion device base indicators containing more combinations of device base indicators. Then, the data processing apparatus jumps to the process of fusing each target device base indicator in the plurality of device base indicators, i.e., returns to the operation similar to step S2321, and iteratively performs it.

[0233] In the iteration process, the data processing apparatus repeatedly performs the operations of fusing, selecting data, identifying, and judging until an identification sub-result representing successful identification is obtained, and then the iteration ends. For example, assuming that the initial target device basis index set contains indexes A, B, C, D, and E, and the preset number is 3 at the initial current analysis level, after the operations of steps S2321 to S2324, it is found that all the identification sub-results corresponding to the fused device basis indexes are not successful, and then the level is expanded, and the preset number becomes 4 at the new current analysis level.

[0234] The data processing apparatus selects 4 indexes from the target device basis index set for fusion again, such as fused index 1 (fused by indexes A, B, C, and D) and fused index 2 (fused by indexes A, B, C, and E). For each new fused device basis index, the data processing apparatus selects corresponding example device basis data from the example decision unit device basis data set corresponding to the wandering basis decision unit according to the manner of step S2322. For example, for fused index 1, the data processing apparatus searches for a record containing data corresponding to indexes A, B, C, and D in the data set.

[0235] Then, the data processing apparatus identifies the selected example device basis data according to the method of step S2323 to obtain new identification sub-results. If the identification sub-result corresponding to a certain fused device basis index (such as fused index 2) represents successful identification at this time, the iteration ends.

[0236] When the identification sub-result representing successful identification is obtained, the data processing apparatus selects the traffic identification feature vector corresponding to the wandering basis decision unit from the target device basis index set based on this result. For example, assuming that the identification sub-result corresponding to fused index 2 is successful identification, since fused index 2 is fused by indexes A, B, C, and E, the data processing apparatus selects indexes A, B, C, and E as the traffic identification feature vector corresponding to the wandering basis decision unit.

[0237] Through such an iteration process, the data processing apparatus can continuously adjust the combination of fused device basis indexes until a feature combination that can successfully identify a device is found, thereby determining a suitable traffic identification feature vector, providing an accurate feature basis for subsequent traffic detection network generation and the like, and improving the identification accuracy and efficiency of the device.

[0238] As an implementation manner, step S2323, identifying the selected example device basis data corresponding to the targeted fused device basis index to obtain an identification sub-result corresponding to the targeted fused device basis index, includes:

[0239] Step S23231: fitting a fusion hidden Markov model according to the selected example device basic data of the targeted fusion device basic index, to obtain a fusion hidden Markov model corresponding to the targeted fusion device basic index;

[0240] Step S23232: determining the recognition quality of the fusion hidden Markov model corresponding to the targeted fusion device basic index.

[0241] Step S23233: when the recognition quality of the fusion hidden Markov model corresponding to the targeted fusion device basic index meets a preset recognition quality condition, determining that the recognition result corresponding to the targeted fusion device basic index is a result representing successful recognition.

[0242] In step S23231, the data processing apparatus constructs a fusion hidden Markov model suitable for the fusion device basic index according to the selected example device basic data.

[0243] The process of model fitting by the data processing apparatus is to estimate the parameters of the model through the example device basic data. A commonly used fitting method is the Baum-Welch algorithm. This algorithm is a specific implementation of the Expectation-Maximization (EM) algorithm, which is used to estimate the parameters of a hidden Markov model given an observation sequence.

[0244] Specifically, the main steps of the Baum-Welch algorithm are as follows:

[0245] 1. Initialization: randomly initialize the parameters of the model, including the state transition probability matrix and the observation probability matrix.

[0246] 2. Expectation step (E step): according to the current model parameters, calculate the posterior probability of the hidden state given the observation sequence.

[0247] 3. Maximization step (M step): according to the posterior probability calculated in the E step, update the parameters of the model to maximize the probability of the observation sequence.

[0248] 4. Repeat the E step and the M step until the model parameters converge or reach a preset number of iterations.

[0249] Through such a fitting process, the data processing apparatus can obtain a fusion hidden Markov model for the fusion device basic index, which can describe the relationship between the hidden state and the observation sequence of the example device basic data.

[0250] In step S23232, after obtaining the fusion hidden Markov model, the data processing apparatus needs to evaluate the recognition quality of the model to judge its recognition ability for the device. The recognition quality is an indicator to measure the accuracy and reliability of the model.

[0251] A feasible evaluation method is to use the Cross-Validation technique. Cross-Validation divides the dataset into a training set and a validation set, trains the model on the training set, and then evaluates the performance of the model on the validation set. For example, the example device base data is divided according to a certain proportion (such as 80% training set and 20% validation set).

[0252] The data processing apparatus uses the training set data to train the fusion hidden Markov model, and then uses the validation set data to make predictions on the trained model. The prediction results are compared with the true labels (known device states or categories) of the validation set data, and evaluation indicators such as accuracy, recall, and F1 value are calculated.

[0253] The formula for calculating the accuracy is: Accuracy = (number of correctly predicted samples / total number of samples) x 100%. For example, in the validation set, there are 100 samples, and the model correctly predicts 80 samples, then the accuracy is (80 / 100) x 100% = 80%.

[0254] The formula for calculating the recall is: Recall = (true positives / actual positives) x 100%. Among them, true positives refer to the number of samples that are predicted as positive examples and are actually positive examples.

[0255] The F1 value is the harmonic mean of accuracy and recall, and the formula is: F1 value = 2 x (accuracy x recall) / (accuracy + recall).

[0256] By calculating these evaluation indicators, the data processing apparatus can comprehensively understand the recognition quality of the fusion hidden Markov model. For example, if the accuracy is high, it means that the model has good recognition accuracy for the device; if the recall is high, it means that the model can better identify positive example devices.

[0257] In step S23233, the preset recognition quality condition is a standard set in advance to judge whether the model is effective. For example, the preset recognition quality condition can be that the accuracy is greater than 85% and the recall is greater than 80%.

[0258] When the recognition quality of the fusion hidden Markov model calculated by the data processing apparatus meets the preset recognition quality condition, it is considered that the model can effectively identify the device. At this time, the data processing apparatus determines that the recognition result corresponding to the fusion device base index is a result representing successful recognition.

[0259] For example, it is calculated that the accuracy of the fusion hidden Markov model corresponding to a certain fusion device basic index is 90%, and the recall rate is 88%, both of which meet the preset recognition quality condition. This indicates that the model has high accuracy and reliability in identifying devices, and can well distinguish different types of devices according to the example device basic data of the fusion device basic index. Therefore, the data processing apparatus determines that the recognition result corresponding to the fusion device basic index is successful.

[0260] As an implementation form, in step S23232, the recognition quality of the fusion hidden Markov model corresponding to the fusion device basic index is determined, including:

[0261] In step S232321, a first Bayes factor corresponding to the wandering basic decision unit is obtained, and a first Bayes factor score is determined based on the first Bayes factor.

[0262] In step S232322, a second Bayes factor corresponding to the fusion hidden Markov model corresponding to the fusion device basic index is obtained, and a second Bayes factor score corresponding to the fusion hidden Markov model corresponding to the fusion device basic index is determined based on the second Bayes factor.

[0263] In step S232323, a Bayes factor change amount of the fusion hidden Markov model corresponding to the fusion device basic index is determined based on the first Bayes factor score and the second Bayes factor score, and the recognition quality of the fusion hidden Markov model corresponding to the fusion device basic index is determined based on the determined Bayes factor change amount.

[0264] The Bayes factor is a statistical quantity used to compare different models or hypotheses. It is based on Bayes' theorem and measures the strength of evidence for one hypothesis over another by comparing the likelihood of data under two hypotheses. In the embodiments of the present application, the data processing apparatus first obtains a first Bayes factor corresponding to the wandering basic decision unit.

[0265] For example, the wandering basic decision unit represents a classification node of a certain type of device, and the data processing apparatus has collected a batch of example data related to the type of device, which constitutes the basis for analysis. In order to obtain the first Bayes factor, the data processing apparatus needs to define two competing hypotheses, for example, hypothesis H1 represents that the device is in a normal working state, and hypothesis H0 represents that the device is in an abnormal working state.

[0266] The calculation formula of the Bayes factor is: ;

[0267] Among them, represents the Bayes factor, P(D|H1) is the probability of observing the data D under the assumption H1, and P(D|H0) is the probability of observing the data D under the assumption H0.

[0268] The data processing apparatus estimates the probabilities of observing the current example data under the two assumptions by analyzing historical data or prior knowledge, and thus calculates the first Bayes factor. For example, it is calculated that the first Bayes factor BF = 3, which means that the evidence strength of the assumption H1 is 3 times that of the assumption H0 under the current data.

[0269] Based on the first Bayes factor, the data processing apparatus determines the first Bayes factor score. The Bayes factor score can be a score mapped according to the numerical range of the Bayes factor. For example, a scoring rule can be defined as follows: when BF < 1, the score is set to 1, indicating that the assumption H0 is more likely; when 1≤ BF < 3, the score is set to 2, indicating that the evidence strengths of the two assumptions are close; and when BF ≥ 3, the score is set to 3, indicating that the assumption H1 is more likely. According to this rule, the first Bayes factor score in the above example is 3.

[0270] In step S232322, the data processing apparatus focuses on the fusion hidden Markov model corresponding to the fusion device basis index of interest. The fusion hidden Markov model is a statistical model that can describe a random process with hidden states, which models data through the relationship between the hidden state sequence and the observation sequence.

[0271] Similarly, to obtain the second Bayes factor, the data processing apparatus needs to define two competing assumptions. The assumption H2 indicates that the fusion hidden Markov model can accurately identify the state of the device, and the assumption H3 indicates that the fusion hidden Markov model cannot accurately identify the state of the device.

[0272] The data processing apparatus calculates the probabilities of observing the data under the two assumptions according to the fitting of the fusion hidden Markov model to the example device basis data, and thus obtains the second Bayes factor. For example, it is calculated that the second Bayes factor BF 23 = 5, which indicates that the evidence strength of the assumption H2 is 5 times that of the assumption H3 under the current data.

[0273] Then, the data processing apparatus determines the second Bayes factor score according to the second Bayes factor. A similar scoring rule as in step S232321 can be used, for example, when BF 23 < 1, the score is set to 1; when 1≤ BF 23≥ 3, the score is set to 3. According to this rule, the second Bayesian factor score in the above example is 3.

[0274] In step S2323, the Bayesian factor change reflects the change in evidence strength of the fused hidden Markov model relative to the hypothesis of the wandering base decision unit. The data processing device determines the Bayesian factor change by calculating the difference between the second Bayesian factor score and the first Bayesian factor score. For example, in the previous example, the first Bayesian factor score is 3 and the second Bayesian factor score is 3, so the Bayesian factor change is 3-3=0.

[0275] Based on the Bayesian factor change, the data processing device can determine the recognition quality of the fused hidden Markov model. Generally speaking, if the Bayesian factor change is positive and large, it means that the fused hidden Markov model provides stronger evidence support relative to the hypothesis of the wandering base decision unit, i.e. the recognition quality of the model is high; if the Bayesian factor change is negative and the absolute value is large, it means that the evidence support of the fused hidden Markov model is weak, and the recognition quality is low; if the Bayesian factor change is close to 0, it means that the evidence strengths of the two are similar, and the recognition quality of the model needs to be further evaluated.

[0276] For example, an identification quality evaluation rule can be set: when the Bayesian factor change is greater than 2, it is considered that the recognition quality of the fused hidden Markov model is high; when the Bayesian factor change is between -2 and 2, it is considered that the recognition quality is medium; when the Bayesian factor change is less than -2, it is considered that the recognition quality is low. In the above example, the Bayesian factor change is 0, so the data processing device will judge that the recognition quality of the fused hidden Markov model is medium.

[0277] The above respectively obtains the first Bayesian factor and its score corresponding to the wandering base decision unit, and the second Bayesian factor and its score corresponding to the fused hidden Markov model, and then calculates the Bayesian factor change, so as to determine the recognition quality of the fused hidden Markov model. This evaluation method based on Bayesian factor can effectively measure the change of evidence strength of the model under different hypotheses, and provides a quantitative basis for judging the recognition ability of the fused hidden Markov model to the device.

[0278] As an implementation manner, in step S2325, the traffic recognition feature vector corresponding to the wandering base decision unit is selected from the target device base index set based on the recognition sub-result representing recognition success, including:

[0279] Step S23251: selecting a candidate fusion device basic index corresponding to a recognition result representing recognition success from a plurality of fusion device basic indexes under the current analysis level, and obtaining a Bayesian factor change amount of a fusion hidden Markov model corresponding to each candidate fusion device basic index respectively;

[0280] Step S23252: selecting a target fusion hidden Markov model corresponding to the maximum Bayesian factor change amount from the fusion hidden Markov models corresponding to each candidate fusion device basic index respectively;

[0281] Step S23253: taking a target device basic index in the candidate fusion device basic index corresponding to the target fusion hidden Markov model as a traffic recognition feature vector corresponding to the walk-based decision unit.

[0282] Based on this, in step S24, the traffic detection network including the traffic recognition feature vector corresponding to the walk-based decision unit is generated, including:

[0283] Step S241: integrating the target fusion hidden Markov model and the traffic recognition feature vector corresponding to the targeted decision unit to obtain the traffic detection network corresponding to the walk-based decision unit.

[0284] In step S23251, under the current analysis level, the data processing apparatus has performed recognition operations on a plurality of fusion device basic indexes and obtained corresponding recognition results. When the recognition result corresponding to a certain fusion device basic index represents recognition success, the data processing apparatus selects the fusion device basic index as a candidate fusion device basic index.

[0285] For example, assuming that there are fusion device basic indexes F1 (fused by indexes A, B, and C), F2 (fused by indexes A, B, and D), and F3 (fused by indexes B, C, and D) under the current analysis level. After the recognition operation, the recognition result corresponding to F1 is recognition success, and the recognition results corresponding to F2 and F3 are recognition failure. Then, the data processing apparatus selects F1 as a candidate fusion device basic index. Next, the data processing apparatus obtains the Bayesian factor change amount of the fusion hidden Markov model corresponding to the candidate fusion device basic index. The Bayesian factor change amount is an index for measuring the change in evidence strength of the fusion hidden Markov model in the recognition process, which is obtained by calculating the difference between the first Bayesian factor score and the second Bayesian factor score in the previous step.

[0286] Assuming that for the candidate fusion device basic index F1, the first Bayesian factor score of the fusion hidden Markov model corresponding to it is 2 points, and the second Bayesian factor score is 4 points, then the Bayesian factor change amount is 4-2=2.

[0287] In step S23252, when there are multiple candidate fusion device base indicators, the data processing apparatus needs to select the model with the largest Bayesian factor change from the corresponding fusion hidden Markov models as the target fusion hidden Markov model. This is because the larger the Bayesian factor change, the stronger the evidence support provided by the model in the identification process, and the better the identification effect.

[0288] For example, assume that in addition to the above-mentioned F1, there is another candidate fusion device base indicator F4 (fused by indicators A, C, and D), and the Bayesian factor change of the corresponding fusion hidden Markov model is 3. Since 3 > 2, the data processing apparatus will select the fusion hidden Markov model corresponding to F4 as the target fusion hidden Markov model.

[0289] In step S23253, after determining the target fusion hidden Markov model, the data processing apparatus will take the target device base indicator in the candidate fusion device base indicator corresponding to the model as the flow identification feature vector corresponding to the wandering base decision unit. These flow identification feature vectors will be used for subsequent flow detection network generation.

[0290] For example, for the candidate fusion device base indicator F4 (fused by indicators A, C, and D) corresponding to the target fusion hidden Markov model, the data processing apparatus will take indicators A, C, and D as the flow identification feature vector corresponding to the wandering base decision unit.

[0291] In step S241, after obtaining the target fusion hidden Markov model and the flow identification feature vector, the data processing apparatus needs to integrate them to construct the flow detection network corresponding to the wandering base decision unit.

[0292] The integration process can be achieved by taking the flow identification feature vector as the input feature of the model, combining the structure and parameters of the target fusion hidden Markov model. For example, the target fusion hidden Markov model has a specific state transition probability matrix and an observation probability matrix, and the data processing apparatus can adjust or optimize these matrices according to the characteristics of the flow identification feature vector, so that the model can better adapt and process these features.

[0293] Assume that the state transition probability matrix of the target fusion hidden Markov model is P, the observation probability matrix is O, and the flow identification feature vector is V = [v1, v2, v3] (where v1, v2, and v3 correspond to the values of indicators A, C, and D, respectively). The data processing apparatus can update P and O according to the values of V, for example, by adjusting certain elements in the matrix to reflect the influence of the flow identification feature vector on the model state transition and observation.

[0294] The specific integration implementation can adopt a model parameter optimization algorithm, such as gradient descent method. Taking the gradient descent method as an example, the data processing apparatus can define a loss function for measuring the difference between the output of the flow detection network and the actual observation data. Then, by calculating the gradient of the loss function with respect to the model parameters, the model parameters are updated in the opposite direction of the gradient, gradually reducing the value of the loss function, until the convergence condition is reached.

[0295] The data processing apparatus updates the model parameters through continuous iteration to minimize the value of the loss function, thereby completing the integration of the target fusion hidden Markov model and the flow identification feature vector, and obtaining the flow detection network corresponding to the walk basic decision unit.

[0296] By selecting appropriate candidate fusion device basic indicators and target fusion hidden Markov models through steps S23251 to S23253, determining the flow identification feature vector, and then integrating the target fusion hidden Markov model and the flow identification feature vector in step S241, the flow detection network corresponding to the walk basic decision unit is constructed. This method can select the most representative features and models according to the identification results and the Bayesian factor change amount, thereby improving the identification ability and accuracy of the flow detection network.

[0297] As an implementation, step S30, from the terminal recognition unit of the recognition framework obtained after the multi-level identification is stopped, according to the example device depth data corresponding to the multiple device depth indicators in the example device data stream set, the corresponding example decision unit device set identified from the example device set is subjected to multi-level identification again, and for each level and each time of identification result, the flow identification feature vector is determined from the multiple device depth indicators according to the identification result corresponding to the corresponding decision unit of the corresponding level, and the flow detection network corresponding to the corresponding decision unit including the determined flow identification feature vector is generated, and the sub-decision unit connected with the corresponding decision unit in the recognition framework is assembled, including:

[0298] Step S31: starting from the terminal recognition unit of the recognition framework obtained after the multi-level identification is stopped, the decision units in the newly generated level are walked through one by one, and the walked decision units are determined as walk depth feature decision units;

[0299] Step S32: determining the example decision unit device set corresponding to the walk depth feature decision unit, and obtaining the example decision unit data set corresponding to the walk depth feature decision unit identified from the example device data stream set;

[0300] Step S33: based on the example device depth data in the example decision unit data set corresponding to the walk depth feature decision unit, identifying the example decision unit device set corresponding to the walk depth feature decision unit, obtaining an identification result, and based on the identification result, selecting the traffic identification feature vector corresponding to the walk depth feature decision unit from the plurality of device depth indicators;

[0301] Step S34: generating a plurality of sub-decision units connected to the walk depth feature decision unit, and generating a traffic detection network including the traffic identification feature vector corresponding to the walk depth feature decision unit; the generated traffic detection network includes the hidden Markov model associated with each of the plurality of sub-decision units connected to the walk depth feature decision unit;

[0302] Step S35: based on the hidden Markov model associated with each of the plurality of sub-decision units connected to the walk depth feature decision unit, identifying the example decision unit device set corresponding to the walk depth feature decision unit, obtaining the example decision unit device set corresponding to each of the sub-decision units connected to the walk depth feature decision unit;

[0303] Step S36: when the walk of the decision unit in the newly generated level ends, determining the next level of the newly generated level as a new newly generated level, jumping to the iterative execution of the process of walking through the decision units in the newly generated level, and stopping the iteration when the example device depth data in the example device data stream set cannot generate a new decision unit.

[0304] In step S31, after completing the multi-level identification based on the device basic indicators, the data processing apparatus starts to access the decision units in the newly generated level one by one from the terminal recognition unit of the recognition framework. For example, assume that the terminal recognition unit of the recognition framework has three, corresponding to device types A, B and C. In the newly generated level, some decision units are newly added above each terminal recognition unit, which are used for further identification based on device depth indicators. The data processing apparatus first starts from the terminal recognition unit corresponding to device type A, accesses the first decision unit in the newly generated level above it, and determines it as the walk depth feature decision unit.

[0305] In step S32, after determining the wandering depth feature decision unit, the data processing apparatus needs to find out the example decision unit device set associated with the decision unit. This means finding those example devices that are classified into the category represented by the decision unit in the previous identification process. For example, for the wandering depth feature decision unit determined above (assuming it corresponds to a new classification category based on device depth features), the data processing apparatus will search for all example devices classified into this category in the example device set, which constitutes the example decision unit device set corresponding to the wandering depth feature decision unit.

[0306] At the same time, the data processing apparatus also needs to obtain the example decision unit data set corresponding to the wandering depth feature decision unit from the example device data stream set. The example device data stream set contains example device depth data corresponding to various device depth indicators, and the data processing apparatus will filter out the depth data records corresponding to the devices in the example decision unit device set from the example device data stream set to form the example decision unit data set.

[0307] In step S33, the data processing apparatus uses the example device depth data in the example decision unit data set to further analyze the example decision unit device set. This may involve the use of various data analysis and machine learning algorithms, such as clustering analysis, classification algorithms, etc. For example, the data processing apparatus can use clustering algorithms to cluster the example device depth data and identify different device patterns or categories based on the clustering results.

[0308] Assuming that the example device depth data contains certain complex operating parameters of the device (such as power consumption fluctuation curve in a certain time period, burst frequency of data transmission, etc.), the data processing apparatus may find different clustering clusters corresponding to different operating states or characteristic patterns of the device by clustering analysis of these data. Based on the clustering results, the data processing apparatus obtains the identification results.

[0309] Based on this identification result, the data processing apparatus selects the traffic identification feature vector corresponding to the wandering depth feature decision unit from the various device depth indicators. The principle of selection is to determine the indicators with high discrimination degree for devices in the identification results. For example, if the clustering results show that a certain device depth indicator has obvious differences between different clustering clusters and can well distinguish different categories of devices, the data processing apparatus will select this indicator as the traffic identification feature vector corresponding to the wandering depth feature decision unit.

[0310] In step S34, after obtaining the traffic identification feature vector corresponding to the walk depth feature decision unit, the data processing apparatus generates a plurality of sub-decision units connected to the decision unit. These sub-decision units are used to further subdivide the device categories or features, so as to more accurately identify the devices. For example, different sub-decision units are generated according to different operating states or performance features of the devices, and each sub-decision unit represents a specific device state or feature category.

[0311] At the same time, the data processing apparatus generates a traffic detection network including the traffic identification feature vector. The traffic detection network is a model structure for deep identification of devices, which includes the hidden Markov models respectively associated with the plurality of sub-decision units connected to the walk depth feature decision unit. The hidden Markov model can effectively process sequence data with hidden states, and is suitable for analyzing the timing features and potential patterns in the device depth data.

[0312] For example, assuming that three sub-decision units are generated, corresponding to high-performance state, medium-performance state and low-performance state of the devices respectively. For each sub-decision unit, the data processing apparatus constructs a hidden Markov model associated therewith. In constructing the hidden Markov model, the traffic identification feature vector is taken as the input feature of the model, and through learning and training on the example device depth data, the parameters of the model, such as the state transition probability matrix and the observation probability matrix, are determined.

[0313] In step S35, the data processing apparatus uses the hidden Markov models respectively associated with the plurality of sub-decision units connected to the walk depth feature decision unit to re-identify the devices in the example decision unit device set. Specifically, for each device in the example decision unit device set, the data processing apparatus inputs the corresponding example device depth data of the device into the hidden Markov models associated with each sub-decision unit, and determines to which sub-decision unit the device belongs according to the output result of the model.

[0314] For example, for a specific example device, the data processing apparatus inputs its depth data into the hidden Markov model associated with the high-performance state sub-decision unit, the hidden Markov model associated with the medium-performance state sub-decision unit, and the hidden Markov model associated with the low-performance state sub-decision unit respectively. Assuming that the hidden Markov model associated with the high-performance state sub-decision unit outputs the highest probability, the data processing apparatus will classify the example device into the example decision unit device set corresponding to the high-performance state sub-decision unit.

[0315] Through the identification of all devices in the example decision unit device set, the data processing apparatus finally obtains the example decision unit device set respectively corresponding to each sub-decision unit connected to the walk depth feature decision unit.

[0316] In step S36, when the data processing apparatus completes the walk and related operations on all decision units in the newly generated level, the next level of the newly generated level is determined as a new newly generated level. Then, the data processing apparatus jumps back to step S31 to walk through the decision units in the new newly generated level one by one, and continues the above-mentioned operations of identification, generation of sub-decision units, construction of traffic detection network, etc.

[0317] This iteration process is repeated until no new decision unit can be generated according to the example device depth data in the example device data stream set. For example, when the data processing apparatus finds that no new device feature or category can be found to further subdivide the decision unit according to the existing example device depth data in a certain iteration, the iteration is stopped. At this time, the entire multi-level identification based on device depth indicators is completed, and the data processing apparatus completes the further improvement and optimization of the identification framework, so that the identification framework can more accurately identify and classify devices in depth.

[0318] Through the operations of steps S31 to S36, the decision units in the newly generated level are walked through and processed one by one from the terminal identification unit of the identification framework, new sub-decision units and traffic detection networks are continuously generated, and the example devices are identified and classified in depth until no new decision unit can be generated. This multi-level identification and iteration method can fully utilize the information in the example device depth data to improve the accuracy and precision of device identification.

[0319] As an implementation, in step S33, before the traffic identification feature vector corresponding to the walked depth feature decision unit is selected from the plurality of device depth indicators based on the example device depth data in the example decision unit data set corresponding to the walked depth feature decision unit, the method further comprises:

[0320] Step S3301: selecting a key device depth indicator from the plurality of device depth indicators based on the example decision unit data set corresponding to the walked depth feature decision unit, to obtain a key device depth indicator set;

[0321] Step S3302: determining the cumulative frequency distribution corresponding to each key device depth indicator in the key device depth indicator set, and removing the key device depth indicators corresponding to the cumulative frequency distribution with a single change direction from the key device depth indicator set to obtain a transition device depth indicator set;

[0322] Step S3303: Perform bimodal cross-point density detection on the cumulative frequency distribution corresponding to each key device depth indicator in the transition device depth indicator set, remove the key device depth indicator corresponding to the second target cumulative frequency distribution from the transition device depth indicator set to obtain the target device depth indicator set, wherein the second target cumulative frequency distribution is the cumulative frequency distribution with a bimodal cross-point density not less than a second preset density threshold.

[0323] Based on this, step S33, based on the example device depth data in the example decision unit data set corresponding to the wandering depth feature decision unit, identify the example decision unit device set corresponding to the wandering depth feature decision unit, obtain the identification result, and based on the identification result, select the traffic identification feature vector corresponding to the wandering depth feature decision unit from the plurality of device depth indicators, including:

[0324] Step S331: Based on the example device depth data in the example decision unit data set corresponding to the wandering depth feature decision unit and the target device depth indicator set, identify the example decision unit device set corresponding to the newly generated depth feature decision unit, obtain the identification result, and based on the identification result, select the traffic identification feature vector corresponding to the newly generated depth feature decision unit from the target device depth indicator set.

[0325] In step S3301, the data processing apparatus analyzes the plurality of device depth indicators based on the example decision unit data set corresponding to the wandering depth feature decision unit. The example decision unit data set contains various depth data information of the example devices related to the decision unit. The data processing apparatus will use a specific method or algorithm to evaluate the importance of each device depth indicator for distinguishing different device states or features. For example, suppose the plurality of device depth indicators include the "data encryption strength", "communication protocol complexity", "data processing delay", "memory occupancy rate" and the like of the device. The data processing apparatus can measure the importance of each indicator by calculating the information gain of each indicator. Information gain refers to the degree of reduction in uncertainty of the data set after knowing a certain feature (here, device depth indicator). The greater the information gain, the stronger the ability of the indicator to distinguish different device states or features.

[0326] In step S3302, for each key device depth indicator in the key device depth indicator set, the data processing apparatus calculates the corresponding cumulative frequency distribution. The cumulative frequency distribution refers to the cumulative frequency of each data point after arranging the data in ascending order, which can reflect the distribution characteristics of the data.

[0327] For example, for the key device depth indicator "data encryption strength", the data processing apparatus will collect the "data encryption strength" data of all example devices in the example decision unit data set, and sort these data from small to large. Assuming the data is [10, 20, 30, 30, 40, 50], and the corresponding frequencies are [1, 1, 2, 1, 1] (i.e. 10 appears once, 20 appears once, 30 appears twice, etc.). Then the cumulative frequency distribution can be calculated as: the cumulative frequency of the first data point 10 is 1 / 6, the cumulative frequency of the second data point 20 is (1+1) / 6 = 2 / 6, the cumulative frequency of the third data point 30 is (1+1+2) / 6 = 4 / 6, and so on.

[0328] Next, the data processing apparatus analyzes the change direction of the cumulative frequency distribution of each key device depth indicator. If the cumulative frequency distribution of a certain key device depth indicator shows a single change direction, such as always monotonically increasing or monotonically decreasing, it means that the data distribution of this indicator is relatively uniform, and it may not be able to well reflect the different characteristics or states of the device, so it needs to be removed from the key device depth indicator set.

[0329] In step S3303, the data processing apparatus performs bimodal cross-point density detection on the cumulative frequency distribution of each key device depth indicator in the transition device depth indicator set. The bimodal cross-point density refers to the density of the cross-points between the two peaks when the cumulative frequency distribution curve appears in a bimodal shape.

[0330] The data processing apparatus will set a second preset density threshold to determine whether the bimodal cross-point density is too high. If the bimodal cross-point density of the cumulative frequency distribution of a certain key device depth indicator is not less than the second preset density threshold, it means that the data distribution of this indicator is too complex or unstable, which may interfere with the subsequent device recognition, so it needs to be removed from the transition device depth indicator set.

[0331] In step S331, after obtaining the target device depth indicator set, the data processing apparatus will identify the example decision unit device set corresponding to the newly generated depth feature decision unit based on the example device depth data in the example decision unit data set corresponding to the target device depth indicator set and corresponding to the wandering depth feature decision unit.

[0332] For example, assuming that the target device depth indicator set contains "data encryption strength", the data processing apparatus will extract the "data encryption strength" data of all example devices from the example decision unit data set. Then, the data processing apparatus can use clustering algorithms, classification algorithms, etc. to analyze and process these data to identify different device patterns or categories.

[0333] Suppose that the data processing apparatus divides the devices in the example decision unit device set into two categories through the clustering algorithm, and the "data encryption strength" data of the devices in one category is more concentrated at a higher level, and the "data encryption strength" data of the devices in the other category is more concentrated at a lower level. This is the result obtained through identification. Based on this identification result, the data processing apparatus selects the traffic identification feature vector corresponding to the newly generated deep feature decision unit from the target device deep index set. For example, since "data encryption strength" can well distinguish the two categories of devices, the data processing apparatus selects "data encryption strength" as the traffic identification feature vector corresponding to the newly generated deep feature decision unit.

[0334] In summary, the data processing apparatus screens and processes the device deep indexes through steps S3301 to S3303 to obtain the target device deep index set, and then performs device identification based on the example device deep data corresponding to the target device deep index set in step S331, and selects a suitable traffic identification feature vector, thereby providing a basis for subsequent construction of a more accurate traffic detection network and further device identification.

[0335] The embodiments of the present application also provide a data processing apparatus, as shown in Figure 2 The data processing apparatus 100 includes a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, such as through a bus 102. Optionally, the data processing apparatus 100 can also include a transceiver 104. The computer program for implementing the above-mentioned Internet of Things gateway data processing method of the embodiments of the present application is stored in the memory, and is configured to be executed by one or more processors 101 to implement the technical solutions provided by the present application.

[0336] The above only describes some embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. An Internet of Things gateway data processing method, characterized in that, The method comprises: acquiring a traffic identification framework and determining a set of devices to be processed; each non-terminal identification unit of the traffic identification framework corresponds to a traffic detection network of a first type or a traffic detection network of a second type, and in each identification track of the traffic identification framework, a decision unit corresponding to the traffic detection network of the second type is behind a decision unit corresponding to the traffic detection network of the first type; wherein the traffic detection network of the first type is a traffic detection network of a traffic identification feature vector including traffic basic features, and the traffic detection network of the second type is a traffic detection network of a traffic identification feature vector including traffic deep features; starting from a starting identification unit of the traffic identification framework, walking through each decision unit in the traffic identification framework one by one, acquiring a traffic detection network corresponding to the walked decision unit, and determining a set of devices corresponding to the walked decision unit in the set of devices to be processed; based on the traffic detection network corresponding to the walked decision unit and the traffic identification feature vector in the traffic detection network corresponding to the walked decision unit, identifying the set of devices corresponding to the walked decision unit, determining a set of devices corresponding to each sub-decision unit connected to the walked decision unit, until a set of devices corresponding to each terminal identification unit in the traffic identification framework is obtained.

2. The method of claim 1, wherein, The method comprises: acquiring a set of device data streams corresponding to the walked decision unit; the acquired set of device data streams includes device data streams corresponding to each device in the set of devices corresponding to the walked decision unit; based on the traffic identification feature vector in the traffic detection network corresponding to the walked decision unit, selecting a device data item from each device data stream in the set of device data streams corresponding to the walked decision unit; based on the selected device data item and the traffic detection network corresponding to the walked decision unit, identifying the set of devices corresponding to the walked decision unit, and determining a set of devices corresponding to each sub-decision unit connected to the walked decision unit.

3. The method of claim 2, wherein, The traffic detection network comprises a fusion hidden Markov model; the fusion hidden Markov model comprises a plurality of hidden Markov models; and the method comprises: based on the selected device data item and the traffic detection network corresponding to the walked decision unit, identifying the set of devices corresponding to the walked decision unit, and determining a set of devices corresponding to each sub-decision unit connected to the walked decision unit. For each device data stream in the device data stream set corresponding to the walked-to decision unit, load the device data item selected from the device data stream into each hidden Markov model in the flow detection network corresponding to the walked-to decision unit, to obtain the identification confidence output by each hidden Markov model in the flow detection network corresponding to the walked-to decision unit; determine a target hidden Markov model outputting the maximum identification confidence in the flow detection network corresponding to the walked-to decision unit, and determine a target sub-decision unit associated with the target hidden Markov model in each sub-decision unit connected to the walked-to decision unit; attribute the device corresponding to the device data stream to the target sub-decision unit.

4. The method of claim 1, wherein, The method further comprises: when obtaining the device set corresponding to each decision unit in the traffic identification framework, determine the generalization network corresponding to each decision unit in the traffic identification framework based on the device data stream set of the device set corresponding to each decision unit in the traffic identification framework; identify the generalization device set based on the generalization network corresponding to each decision unit in the traffic identification framework, wherein the generalization device set is different from the processing device set.

5. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the traffic identification framework, the method further comprises: obtain an example device data stream set of an example device set; the example device data stream set includes example device basic data corresponding to a plurality of device basic indicators and example device deep data corresponding to a plurality of device deep indicators; perform multi-level identification on the example device set according to the example device basic data corresponding to the plurality of device basic indicators in the example device data stream set, and for each identification result in each level, determine a traffic identification feature vector from the plurality of device basic indicators according to the identification result corresponding to the corresponding decision unit in the corresponding level, generate a flow detection network corresponding to the corresponding decision unit including the determined traffic identification feature vector, and simultaneously establish a sub-decision unit connected to the corresponding decision unit in the identification framework; from the terminal identification unit of the identification framework obtained after the multi-level identification stops, perform multi-level identification again on the corresponding example decision unit device set identified from the example device set according to the example device deep data corresponding to the plurality of device deep indicators in the example device data stream set, and for each identification result in each level, determine a traffic identification feature vector from the plurality of device deep indicators according to the identification result corresponding to the corresponding decision unit in the corresponding level, generate a flow detection network corresponding to the corresponding decision unit including the determined traffic identification feature vector, and simultaneously establish a sub-decision unit connected to the corresponding decision unit in the identification framework; when the multi-level identification is performed again, the corresponding identification framework is taken as the traffic identification framework.

6. The method of claim 5, wherein, The plurality of device basic indicators in the example device data stream set respectively correspond to example device basic data, and the plurality of device basic indicators are used to identify the example device set in multiple levels. For each identification result in each level, a traffic identification feature vector corresponding to the walk base decision unit is determined from the plurality of device basic indicators according to the identification result corresponding to the corresponding decision unit in the corresponding level, and a traffic detection network including the determined traffic identification feature vector corresponding to the corresponding decision unit is generated. Meanwhile, a sub-decision unit connected to the corresponding decision unit in the identification framework is established, including: When generating an identification framework based on the example device basic data in the example device data stream set, the decision units in the newly generated level are walked through one by one, and the walked decision unit is taken as a walk base decision unit; An example decision unit device set corresponding to the walk base decision unit is determined, and an example decision unit data set corresponding to the walk base decision unit identified from the example device data stream set is obtained; Based on the example device basic data in the example decision unit data set corresponding to the walk base decision unit, the example decision unit device set corresponding to the walk base decision unit is identified, and a traffic identification feature vector corresponding to the walk base decision unit is selected from the plurality of device basic indicators based on the identification result; A plurality of sub-decision units connected to the walk base decision unit are generated, and a traffic detection network including the traffic identification feature vector corresponding to the walk base decision unit is generated. The generated traffic detection network includes a plurality of sub-decision units connected to the walk base decision unit, and each sub-decision unit is associated with a hidden Markov model; Based on the hidden Markov model associated with each sub-decision unit connected to the walk base decision unit, the example decision unit device set corresponding to the walk base decision unit is identified to obtain an example decision unit device set corresponding to each sub-decision unit connected to the walk base decision unit; When the walk of the decision units in the newly generated level is completed, the next level of the newly generated level is determined as a new newly generated level, and the process of walking through one by one the decision units in the newly generated level is iteratively executed. When no new decision unit can be generated based on the example device basic data in the example device data stream set, the iteration is stopped.

7. The method of claim 6, wherein, The plurality of device basic indicators in the example device data stream set respectively correspond to example device basic data, and the plurality of device basic indicators are used to identify the example device set in multiple levels. For each identification result in each level, a traffic identification feature vector corresponding to the walk base decision unit is determined from the plurality of device basic indicators according to the identification result corresponding to the corresponding decision unit in the corresponding level, and a traffic detection network including the determined traffic identification feature vector corresponding to the corresponding decision unit is generated. Meanwhile, a sub-decision unit connected to the corresponding decision unit in the identification framework is established, including: A target device basic indicator set is obtained, wherein the target device basic indicator set includes a target device basic indicator selected from the plurality of device basic indicators; identify, based on the target device basic index set and example device basic data in the example decision unit data set corresponding to the wandering basic decision unit, an example decision unit device set corresponding to the wandering basic decision unit, and select, based on a recognition result, a traffic recognition representation vector corresponding to the wandering basic decision unit from the target device basic index set.

8. The method of claim 7, wherein, The target device basic index set is determined by the following steps: Determine, based on the example device data stream set, a numerical fluctuation coefficient corresponding to each of the plurality of device basic indexes; Remove, from the plurality of device basic indexes, a device basic index corresponding to a target numerical fluctuation coefficient to obtain a first transition device basic index set; The target numerical fluctuation coefficient is a numerical fluctuation coefficient not less than a preset coefficient threshold value; Determine, based on the example device data stream set, a cumulative frequency distribution corresponding to each device basic index in the first transition device basic index set, and remove, from the first transition device basic index set, a device basic index corresponding to a cumulative frequency distribution with a single change direction to obtain a second transition device basic index set; Perform bimodal cross-point density detection on the cumulative frequency distribution corresponding to each device basic index in the second transition device basic index set to remove a device basic index corresponding to a first target cumulative frequency distribution from the second transition device basic index set to obtain the target device basic index set; the first target cumulative frequency distribution is a cumulative frequency distribution with a bimodal cross-point density not less than a first preset density threshold value; The recognition result includes a plurality of recognition sub-results; the identification of the example decision unit device set corresponding to the wandering basic decision unit based on the target device basic index set and the example device basic data in the example decision unit data set corresponding to the wandering basic decision unit, and the selection of the traffic recognition representation vector corresponding to the wandering basic decision unit from the target device basic index set based on the recognition result, includes: Starting from a single feature, fuse each target device basic index in the target device basic index set to obtain a plurality of fused device basic indexes at a current analysis level; each fused device basic index at the current analysis level includes a preset number of device basic indexes corresponding to the current analysis level; For each fused device basic index in the plurality of fused device basic indexes at the current analysis level, select example device basic data of the corresponding fused device basic index from the example decision unit device basic data set corresponding to the wandering basic decision unit; Identify the selected example device basic data of the corresponding fused device basic index to obtain a recognition sub-result corresponding to the corresponding fused device basic index; When the recognition sub-results corresponding to the plurality of fused device basic indexes at the current analysis level are all results representing unsuccessful recognition, expand the current analysis level to obtain a next level. In a next recognition sub-result recognition process, the next level is determined as a new current analysis level, iteration is performed on the process of fusing each target device basic index in the plurality of device basic indexes, the iteration is ended when a recognition sub-result representing recognition success is obtained, and a traffic recognition feature vector corresponding to the walk basic decision unit is selected from the target device basic index set based on the recognition sub-result representing recognition success.

9. The method of claim 8, wherein, The identifying the example device basic data of the selected targeted fusion device basic index to obtain a recognition sub-result corresponding to the targeted fusion device basic index comprises: fitting a fusion hidden Markov model according to the example device basic data of the selected targeted fusion device basic index to obtain a fusion hidden Markov model corresponding to the targeted fusion device basic index; determining the recognition quality of the fusion hidden Markov model corresponding to the targeted fusion device basic index; when the recognition quality of the fusion hidden Markov model corresponding to the targeted fusion device basic index meets a preset recognition quality condition, determining that a recognition result corresponding to the targeted fusion device basic index is a result representing recognition success.

10. A data processing apparatus, characterized by, comprise: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method of any one of claims 1-9 is implemented.

Images