Method and apparatus for determining valid data of device data, and computer program product

Abstract

The invention provides an effective data determination method and device for equipment data and a computer program product, and the method comprises the steps: obtaining the equipment data which comprises offline data, state data, control data and response data; the decision tree models are adopted to classify the corresponding equipment data to obtain valid data in the equipment data, the decision tree models are in one-to-one correspondence with the types of the equipment data, and the decision tree models are obtained by training the valid data and invalid data of one type of equipment data; the problem of data conflicts caused by effective judgment errors of equipment data in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and computer program product for determining valid data of device data. Background Art

[0002] With the rapid development of the Internet of Things, a large number of devices need to send or receive data to the IOT cloud service. Due to many factors such as network stability, network speed, service problems, etc. during the data transmission process, there is a conflict in the existing data when the device data reaches the user. Summary of the invention

[0003] The main purpose of the present application is to provide a method, apparatus and computer program product for determining valid data of device data, so as to at least solve the problem of data conflict caused by errors in valid judgment of device data in the prior art.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for determining valid data of device data is provided, comprising: acquiring device data, the device data including offline data, status data, control data and response data; using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the decision tree model corresponding to the type of the device data one by one, and the decision tree model is trained for valid data and invalid data of one type of the device data.

[0005] Optionally, before using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the method also includes: screening a first key feature from the features corresponding to the offline data according to feature importance, the features corresponding to the offline data include server information, device online time, and device offline time, and the feature importance of the first key feature is greater than any remaining features; constructing a first decision tree according to the first key feature; using historical offline data to train the first decision tree to obtain the decision tree model corresponding to the offline data, the historical offline data includes valid offline data and invalid offline data.

[0006] Optionally, before using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the method also includes: screening a second key feature from the features corresponding to the status data according to feature importance, the features corresponding to the status data include device control data and device response data, and the feature importance of the second key feature is greater than any remaining features; constructing a second decision tree based on the second key feature; and training the second decision tree using historical status data to obtain the decision tree model corresponding to the status data, the historical status data including valid status data and invalid status data.

[0007] Optionally, before using historical offline data to train the first decision tree, the method also includes: when the quantity difference ratio is greater than or equal to a predetermined ratio, using a resampling technique to increase the number of valid offline data or the number of invalid offline data, so that the quantity difference ratio is less than the predetermined ratio, and the quantity difference ratio is the ratio of the absolute value of the difference between the number of valid offline data and the number of invalid offline data to the sum of the number of valid offline data and the number of invalid offline data; or, increasing the cost weight of the target offline data, the target offline data is the one with less data between the valid offline data and the invalid offline data, and the cost weight is the weight for calculating the classification accuracy of the historical offline data.

[0008] Optionally, constructing a first decision tree according to the first key feature includes: constructing an initial decision tree according to the first key feature; and when the depth of the initial decision tree is greater than a predetermined depth, pruning the initial decision tree using a pruning technique to obtain the first decision tree.

[0009] Optionally, the first decision tree is trained using historical offline data to obtain the decision tree model corresponding to the offline data, including: using grid search or random search to tune the parameters of the first decision tree to obtain the optimal parameters of the first decision tree; adjusting the parameters of the first decision tree according to the optimal parameters, and using the historical offline data to train the first decision tree to obtain the decision tree model corresponding to the offline data.

[0010] Optionally, the method further includes: in a case where there is response data having the same random number as the control data and a timestamp later than the control data, determining that the control data is valid data; in a case where there is control data having the same random number as the response data and a timestamp earlier than the response data, determining that the response data is valid data.

[0011] Optionally, after classifying the corresponding device data using a decision tree model to obtain valid data in the device data, the method further includes: uploading the valid data in the device data to a server, and prohibiting invalid data in the device data from being uploaded to the server.

[0012] According to another aspect of the present application, a device for determining valid data of device data is provided, comprising: an acquisition unit, used to acquire device data, the device data including offline data, status data, control data and response data; a classification unit, used to classify the corresponding device data using a decision tree model to obtain valid data in the device data, the decision tree model having a one-to-one correspondence with the type of the device data, and the decision tree model being trained for valid data and invalid data of one type of the device data.

[0013] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, any one of the methods described above is implemented.

[0014] By applying the technical solution of the present application, in the method for determining valid data of the above-mentioned device data, a decision tree model is trained for each type of device data, and the training data is valid data and invalid data with classification numbers, so that the corresponding device data is classified through the decision tree model to obtain valid data in the device data. There is no problem of inaccurate valid data judgment through time sequence due to network delays, etc., which solves the problem of data conflict caused by errors in valid judgment of device data in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for determining valid data of device data provided in an embodiment of the present application is shown;

[0016] Figure 2 A schematic flow chart of a method for determining valid data of device data provided according to an embodiment of the present application is shown;

[0017] Figure 3 A schematic diagram of a process for solving a data conflict according to an embodiment of the present application is shown;

[0018] Figure 4 A structural block diagram of a device for determining valid data of equipment data provided according to an embodiment of the present application is shown.

[0019] The above drawings include the following reference numerals:

[0020] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] As introduced in the background technology, errors in valid judgment of device data in the prior art lead to data conflicts. To solve this technical problem, the embodiments of the present application provide a method, apparatus and computer program product for determining valid data of device data.

[0025] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for determining valid data of device data according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the valid data determination method of the device data in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0028] In this embodiment, a method for determining valid data of device data running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 2 FIG. 1 is a flowchart of a method for determining valid data of device data according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0030] Step S201, obtaining device data, the device data including offline data, status data, control data and response data;

[0031] Step S202, using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the decision tree model corresponds to the type of the device data one by one, and the decision tree model is trained with valid data and invalid data of one type of device data.

[0032] In the above-mentioned method for determining valid data of device data, a decision tree model is trained for each type of device data, and the training data is valid data and invalid data with classification numbers, so that the corresponding device data is classified through the decision tree model to obtain valid data in the device data. There is no problem of inaccurate valid data judgment through time sequence due to network delays, etc., which solves the problem of data conflict caused by errors in valid judgment of device data in the prior art.

[0033] It should be noted that the device data can be the data of the Internet of Things device, and when a data conflict occurs, the device data where the data conflict occurs can be directly obtained. First, the data type is determined, that is, one of the offline data, status data, control data and response data. The corresponding decision tree model is used to determine whether the conflicting device data is valid data. The invalid data is discarded to resolve the data conflict.

[0034] In order to construct a decision tree model corresponding to the offline data, in an optional implementation manner, before using the decision tree model to classify the corresponding device data to obtain valid data in the device data, the method further includes:

[0035] Step S301, selecting a first key feature from the features corresponding to the offline data according to feature importance, the features corresponding to the offline data include server information, device online time, and device offline time, and the feature importance of the first key feature is greater than any remaining features;

[0036] Step S302, constructing a first decision tree according to the first key feature;

[0037] Step S303: Use historical offline data to train the first decision tree to obtain the decision tree model corresponding to the offline data, where the historical offline data includes valid offline data and invalid offline data.

[0038] In the above implementation, when constructing a decision tree, it is usually necessary to select the most important features from a large number of features for splitting. Through feature importance evaluation (such as Gini impurity, information gain, etc.), the most critical features for distinguishing valid data can be screened out. To further optimize the feature selection process, an integrated learning method (such as random forest) can be used to evaluate the importance of each feature, and based on this, features such as server information, device online time, and device offline time can be screened to find the first key feature corresponding to the above offline data, so as to construct a first decision tree based on the above first key feature, and use historical offline data to train the above first decision tree to obtain the above decision tree model corresponding to the above offline data.

[0039] In order to construct a decision tree model corresponding to the state data, in an optional implementation manner, before using the decision tree model to classify the corresponding device data to obtain valid data in the device data, the method further includes:

[0040] Step S401, selecting a second key feature from the features corresponding to the state data according to feature importance, the features corresponding to the state data include device control data and device response data, and the feature importance of the second key feature is greater than any remaining feature;

[0041] Step S402, constructing a second decision tree according to the second key feature;

[0042] Step S403: Use historical state data to train the second decision tree to obtain the decision tree model corresponding to the state data, wherein the historical state data includes valid state data and invalid state data.

[0043] In the above implementation, the same method is used to screen features such as device control data and device response data to find the second key feature corresponding to the above status data, thereby constructing a second decision tree based on the above second key feature, and training the above second decision tree using historical status data to obtain the above decision tree model corresponding to the above status data.

[0044] In order to achieve data balance, in an optional implementation manner, before using historical offline data to train the first decision tree, the method further includes:

[0045] Step S501: when the quantity difference ratio is greater than or equal to the predetermined ratio, a resampling technique is used to increase the quantity of the valid offline data or the invalid offline data, so that the quantity difference ratio is less than the predetermined ratio, and the quantity difference ratio is the ratio of the absolute value of the difference between the quantity of the valid offline data and the quantity of the invalid offline data to the sum of the quantity of the valid offline data and the quantity of the invalid offline data;

[0046] Step S502, or, increasing the cost weight of the target offline data, the target offline data being the one with less data between the valid offline data and the invalid offline data, and the cost weight being the weight for calculating the classification accuracy of the historical offline data.

[0047] In the above implementation, there may be an imbalance between valid data and invalid data in the device data. The decision tree model may be biased towards the majority class when processing unbalanced data, resulting in inaccurate predictions for the minority class. To improve this, resampling techniques (such as SMOTE, ADASYN, etc.) can be used to balance the data set so that the number of valid data and invalid data is roughly the same, or a cost-sensitive learning method can be used to adjust the model's prediction weight for the minority class. For example, the number of valid data is 100 and the number of invalid data is 50. The cost weight of the valid data is set to 1, and the cost weight of the invalid data is set to 2. In this way, when calculating the classification accuracy, an invalid data classification error is counted twice, thereby making up for the quantity gap.

[0048] In order to improve the classification performance of the decision tree model, in an optional implementation, the above step S302 includes:

[0049] Step S3021, constructing an initial decision tree according to the first key feature;

[0050] Step S3022: When the depth of the initial decision tree is greater than a predetermined depth, the initial decision tree is pruned using a pruning technique to obtain the first decision tree.

[0051] In the above implementation, the decision tree model is prone to overfitting, especially when the depth of the tree is too large. By limiting the complexity of the tree through pruning techniques (such as pre-pruning and post-pruning), the generalization ability of the model can be improved. When the depth of the above initial decision tree is greater than the predetermined depth, the above initial decision tree is pruned using pruning techniques to obtain the above first decision tree, so as to avoid overfitting of the decision tree model corresponding to the first decision tree.

[0052] In order to further improve the classification performance of the decision tree model, in an optional implementation, the above step S303 includes:

[0053] Step S3031, using grid search or random search to tune the parameters of the first decision tree to obtain the optimal parameters of the first decision tree;

[0054] Step S3032: Adjust the parameters of the first decision tree according to the optimal parameters, and use the historical offline data to train the first decision tree to obtain the decision tree model corresponding to the offline data.

[0055] In the above implementation, parameter tuning methods such as grid search and random search are used to find the optimal decision tree parameters (such as tree depth, minimum number of samples, etc.), and the first decision tree after parameter tuning is used for training to obtain a decision tree model to further optimize model performance.

[0056] In order to determine the validity of the control data and the response data, in an optional implementation manner, the method further includes:

[0057] Step S601, in the case where the response data has the same random number as the control data and a timestamp later than the control data, determining that the control data is valid data;

[0058] Step S602: If there is the control data having the same random number as the response data and an earlier timestamp than the response data, determine that the response data is valid data.

[0059] In the above implementation, for the control data and response data, since this part of the data is highly correlated, and when the device is frequently controlled, there will be a situation where the control data of the device and the response data of the device are not one-to-one corresponding, an association relationship can be added to the two types of data, such as a random number, etc., and a comprehensive judgment is made by combining the random number and time to determine whether the device control and response data are valid. That is, if there is the above-mentioned response data with the same random number as the above-mentioned control data and a timestamp later than the above-mentioned control data, then the above-mentioned control data is valid data; if there is the above-mentioned control data with the same random number as the above-mentioned response data and a timestamp earlier than the above-mentioned response data, then the above-mentioned response data is valid data.

[0060] In order to avoid data conflicts, in an optional implementation manner, after classifying the corresponding device data using a decision tree model to obtain valid data in the device data, the method further includes:

[0061] Step S701, uploading valid data in the above device data to the server, and prohibiting invalid data in the above device data from being uploaded to the above server.

[0062] In the above implementation, only valid data in the device data is uploaded to the server for ease of use, and invalid data in the device data is prohibited from being uploaded to the server to prevent data conflicts between the invalid data and other data.

[0063] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of applying the valid data determination method of the device data of the present application to resolve data conflicts will be described in detail below in conjunction with specific embodiments.

[0064] This embodiment relates to a specific solution to data conflicts, such as Figure 3 As shown, the following steps are included:

[0065] Step S1: First, classify the device data, such as offline data, status data, control data and response data;

[0066] Step S2: Formulate different processing methods according to different types of data to solve the problem of device data conflict. For example, use a decision tree model to classify the corresponding device data to obtain valid data in the device data. The decision tree model corresponds to the type of device data one by one.

[0067] Step S3: Determine whether the conflicting data is valid according to the corresponding processing method. If it is invalid, it means that the data is invalid and the data conflict can be resolved. If both are valid, other operations are used to resolve the data conflict problem.

[0068] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] The embodiment of the present application also provides a valid data determination device for device data. It should be noted that the valid data determination device for device data of the embodiment of the present application can be used to execute the valid data determination method for device data provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0070] The following introduces the valid data determination device for device data provided in an embodiment of the present application.

[0071] Figure 4 is a structural block diagram of a device for determining valid data of device data according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0072] An acquisition unit 10 is used to acquire device data, wherein the device data includes offline data, status data, control data and response data;

[0073] The classification unit 20 is used to classify the corresponding device data using a decision tree model to obtain valid data in the device data. The decision tree model corresponds to the type of the device data one by one. The decision tree model is trained with valid data and invalid data of one type of device data.

[0074] In the valid data determination device for the above-mentioned equipment data, a decision tree model is trained for each type of equipment data, and the training data is valid data and invalid data with classification numbers, so that the corresponding equipment data is classified through the decision tree model to obtain the valid data in the equipment data. There is no problem of inaccurate valid data judgment through time sequence due to network delays, etc., which solves the problem of data conflict caused by incorrect valid judgment of equipment data in the prior art.

[0075] It should be noted that the device data can be the data of the Internet of Things device, and when a data conflict occurs, the device data where the data conflict occurs can be directly obtained. First, the data type is determined, that is, one of the offline data, status data, control data and response data. The corresponding decision tree model is used to determine whether the conflicting device data is valid data. The invalid data is discarded to resolve the data conflict.

[0076] In order to construct a decision tree model corresponding to the offline data, in an optional implementation manner, the above-mentioned device further includes:

[0077] A first screening unit is used to screen a first key feature from the features corresponding to the offline data according to feature importance before classifying the corresponding device data using a decision tree model to obtain valid data in the device data, wherein the features corresponding to the offline data include server information, device online time, and device offline time, and the feature importance of the first key feature is greater than any remaining feature;

[0078] A first construction unit, used to construct a first decision tree according to the first key feature;

[0079] The first training unit is used to train the first decision tree using historical offline data to obtain the decision tree model corresponding to the offline data. The historical offline data includes valid offline data and invalid offline data.

[0080] In the above implementation, when constructing a decision tree, it is usually necessary to select the most important features from a large number of features for splitting. Through feature importance evaluation (such as Gini impurity, information gain, etc.), the most critical features for distinguishing valid data can be screened out. To further optimize the feature selection process, an integrated learning method (such as random forest) can be used to evaluate the importance of each feature, and based on this, features such as server information, device online time, and device offline time can be screened to find the first key feature corresponding to the above offline data, so as to construct a first decision tree based on the above first key feature, and use historical offline data to train the above first decision tree to obtain the above decision tree model corresponding to the above offline data.

[0081] In order to construct a decision tree model corresponding to the state data, in an optional implementation manner, the above-mentioned device further includes:

[0082] a second screening unit, configured to screen a second key feature from features corresponding to the state data according to feature importance before classifying the corresponding device data using a decision tree model to obtain valid data in the device data, wherein the features corresponding to the state data include device control data and device response data, and the feature importance of the second key feature is greater than any remaining feature;

[0083] A second construction unit, used to construct a second decision tree according to the second key feature;

[0084] The second training unit is used to train the second decision tree using historical state data to obtain the decision tree model corresponding to the state data, wherein the historical state data includes valid state data and invalid state data.

[0085] In the above implementation, the same method is used to screen features such as device control data and device response data to find the second key feature corresponding to the above status data, thereby constructing a second decision tree based on the above second key feature, and training the above second decision tree using historical status data to obtain the above decision tree model corresponding to the above status data.

[0086] In order to achieve data balance, in an optional implementation manner, the above device further includes:

[0087] a first adjustment unit, configured to, before using historical offline data to train the first decision tree, use a resampling technique to increase the number of valid offline data or the number of invalid offline data when the number difference ratio is greater than or equal to a predetermined ratio, so that the number difference ratio is less than the predetermined ratio, and the number difference ratio is a ratio of an absolute value of a difference between the number of valid offline data and the number of invalid offline data to a sum of the number of valid offline data and the number of invalid offline data;

[0088] The second adjustment unit is used to either increase the cost weight of the target offline data, where the target offline data is the one with less data between the valid offline data and the invalid offline data, and the cost weight is the weight for calculating the classification accuracy of the historical offline data.

[0089] In the above implementation, there may be an imbalance between valid data and invalid data in the device data. The decision tree model may be biased towards the majority class when processing unbalanced data, resulting in inaccurate predictions for the minority class. To improve this, resampling techniques (such as SMOTE, ADASYN, etc.) can be used to balance the data set so that the number of valid data and invalid data is roughly the same, or a cost-sensitive learning method can be used to adjust the model's prediction weight for the minority class. For example, the number of valid data is 100 and the number of invalid data is 50. The cost weight of the valid data is set to 1, and the cost weight of the invalid data is set to 2. In this way, when calculating the classification accuracy, an invalid data classification error is counted twice, thereby making up for the quantity gap.

[0090] In order to improve the classification performance of the decision tree model, in an optional implementation, the first construction unit includes:

[0091] A first building module, used to build an initial decision tree according to the first key feature;

[0092] The first pruning module is used to prune the initial decision tree using a pruning technique to obtain the first decision tree when the depth of the initial decision tree is greater than a predetermined depth.

[0093] In the above implementation, the decision tree model is prone to overfitting, especially when the depth of the tree is too large. By limiting the complexity of the tree through pruning techniques (such as pre-pruning and post-pruning), the generalization ability of the model can be improved. When the depth of the above initial decision tree is greater than the predetermined depth, the above initial decision tree is pruned using pruning techniques to obtain the above first decision tree, so as to avoid overfitting of the decision tree model corresponding to the first decision tree.

[0094] In order to further improve the classification performance of the decision tree model, in an optional implementation, the first training unit includes:

[0095] An optimization module, used to optimize the parameters of the first decision tree by using a grid search or a random search to obtain the optimal parameters of the first decision tree;

[0096] The adjustment module is used to adjust the parameters of the first decision tree according to the optimal parameters, and train the first decision tree using the historical offline data to obtain the decision tree model corresponding to the offline data.

[0097] In the above implementation, parameter tuning methods such as grid search and random search are used to find the optimal decision tree parameters (such as tree depth, minimum number of samples, etc.), and the first decision tree after parameter tuning is used for training to obtain a decision tree model to further optimize model performance.

[0098] In order to determine the validity of the control data and the response data, in an optional implementation manner, the above-mentioned device further includes:

[0099] A first determining unit, configured to determine that the control data is valid data if the response data has the same random number as the control data and a timestamp later than that of the control data;

[0100] The second determining unit is configured to determine that the response data is valid data when there is the control data having the same random number as the response data and an earlier timestamp than the response data.

[0101] In the above implementation, for the control data and response data, since this part of the data is highly correlated, and when the device is frequently controlled, there will be a situation where the control data of the device and the response data of the device are not one-to-one corresponding, an association relationship can be added to the two types of data, such as a random number, etc., and a comprehensive judgment is made by combining the random number and time to determine whether the device control and response data are valid. That is, if there is the above-mentioned response data with the same random number as the above-mentioned control data and a timestamp later than the above-mentioned control data, then the above-mentioned control data is valid data; if there is the above-mentioned control data with the same random number as the above-mentioned response data and a timestamp earlier than the above-mentioned response data, then the above-mentioned response data is valid data.

[0102] In order to avoid data conflicts, in an optional implementation manner, the above device further includes:

[0103] The uploading unit is used to classify the corresponding device data using a decision tree model to obtain valid data in the device data, upload the valid data in the device data to the server, and prohibit invalid data in the device data from being uploaded to the server.

[0104] In the above implementation, only valid data in the device data is uploaded to the server for ease of use, and invalid data in the device data is prohibited from being uploaded to the server to prevent data conflicts between the invalid data and other data.

[0105] The effective data determination device of the device data includes a processor and a memory. The acquisition unit and the classification unit are stored in the memory as program units, and the processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0106] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of data conflict caused by error in valid judgment of device data in the prior art can be solved by adjusting kernel parameters.

[0107] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0108] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the valid data determination method of the device data.

[0109] Specifically, the method for determining valid data of device data includes:

[0110] Step S201, obtaining device data, the device data including offline data, status data, control data and response data;

[0111] Step S202, using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the decision tree model corresponds to the type of the device data one by one, and the decision tree model is trained with valid data and invalid data of one type of device data.

[0112] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the valid data determination method of the device data is executed when the program is run.

[0113] Specifically, the method for determining valid data of device data includes:

[0114] Step S201, obtaining device data, the device data including offline data, status data, control data and response data;

[0115] Step S202, using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the decision tree model corresponds to the type of the device data one by one, and the decision tree model is trained with valid data and invalid data of one type of device data.

[0116] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0117] Step S201, obtaining device data, the device data including offline data, status data, control data and response data;

[0118] Step S202, using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the decision tree model corresponds to the type of the device data one by one, and the decision tree model is trained with valid data and invalid data of one type of device data.

[0119] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0120] Step S201, obtaining device data, the device data including offline data, status data, control data and response data;

[0121] Step S202, using a decision tree model to classify the corresponding device data to obtain valid data in the device data, the decision tree model corresponds to the type of the device data one by one, and the decision tree model is trained with valid data and invalid data of one type of device data.

[0122] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0128] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0129] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0131] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0132] 1) In the valid data determination method of the device data of the present application, a decision tree model is trained for each type of device data, and the training data is valid data and invalid data with classification numbers, so that the corresponding device data is classified through the decision tree model to obtain the valid data in the device data. There is no problem of inaccurate valid data judgment through time sequence due to network delays, etc., which solves the problem of data conflict caused by errors in valid judgment of device data in the prior art.

[0133] 2) In the valid data determination device for device data of the present application, a decision tree model is trained for each type of device data, and the training data is valid data and invalid data with classification numbers, so that the corresponding device data is classified through the decision tree model to obtain valid data in the device data. There is no problem of inaccurate valid data judgment through time sequence due to network delays, etc., which solves the problem of data conflict caused by errors in valid judgment of device data in the prior art.

[0134] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining valid data of device data, characterized in that: include: Acquire device data, wherein the device data includes offline data, status data, control data and response data; A decision tree model is used to classify the corresponding device data to obtain valid data in the device data. The decision tree model corresponds to the type of the device data one by one. The decision tree model is trained with valid data and invalid data of one type of device data.

2. The method according to claim 1, characterized in that Before classifying the corresponding device data using a decision tree model to obtain valid data in the device data, the method further includes: Filtering a first key feature from the features corresponding to the offline data according to feature importance, the features corresponding to the offline data including server information, device online time, and device offline time, wherein the feature importance of the first key feature is greater than any remaining features; Constructing a first decision tree according to the first key feature; The first decision tree is trained using historical offline data to obtain the decision tree model corresponding to the offline data, wherein the historical offline data includes valid offline data and invalid offline data.

3. The method according to claim 1, characterized in that Before classifying the corresponding device data using a decision tree model to obtain valid data in the device data, the method further includes: Selecting a second key feature from the features corresponding to the status data according to feature importance, the features corresponding to the status data include device control data and device response data, and the feature importance of the second key feature is greater than any remaining feature; Constructing a second decision tree according to the second key feature; The second decision tree is trained using historical state data to obtain the decision tree model corresponding to the state data, wherein the historical state data includes valid state data and invalid state data.

4. The method according to claim 2, characterized in that: Before using historical offline data to train the first decision tree, the method further includes: When the quantity difference ratio is greater than or equal to the predetermined ratio, a resampling technique is used to increase the quantity of the valid offline data or the invalid offline data, so that the quantity difference ratio is less than the predetermined ratio, and the quantity difference ratio is the ratio of the absolute value of the difference between the quantity of the valid offline data and the quantity of the invalid offline data to the sum of the quantity of the valid offline data and the quantity of the invalid offline data; Alternatively, the cost weight of the target offline data is increased, the target offline data is the one with less data between the valid offline data and the invalid offline data, and the cost weight is the weight for calculating the classification accuracy of the historical offline data.

5. The method according to claim 2, characterized in that: Constructing a first decision tree according to the first key feature includes: Constructing an initial decision tree according to the first key feature; When the depth of the initial decision tree is greater than a predetermined depth, the initial decision tree is pruned using a pruning technique to obtain the first decision tree.

6. The method according to claim 2, characterized in that Using historical offline data to train the first decision tree to obtain the decision tree model corresponding to the offline data includes: Using grid search or random search to tune parameters of the first decision tree to obtain optimal parameters of the first decision tree; The first decision tree is adjusted according to the optimal parameters, and the first decision tree is trained using the historical offline data to obtain the decision tree model corresponding to the offline data.

7. The method according to claim 1, characterized in that The method further comprises: In the case where the response data has the same random number as the control data and a timestamp later than the control data, determining that the control data is valid data; In the case where the control data has the same random number as the response data and an earlier time stamp than the response data, the response data is determined to be valid data.

8. The method according to any one of claims 1 to 7, characterized in that After classifying the corresponding device data using a decision tree model to obtain valid data in the device data, the method further includes: Upload valid data in the device data to the server, and prohibit invalid data in the device data from being uploaded to the server.

9. A device for determining valid data of equipment data, characterized in that: include: An acquisition unit, used to acquire device data, wherein the device data includes offline data, status data, control data and response data; A classification unit is used to classify the corresponding device data using a decision tree model to obtain valid data in the device data. The decision tree model corresponds to the type of the device data one by one, and the decision tree model is trained for valid data and invalid data of one type of device data.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images