Data processing method and device, equipment and computer readable storage medium

Abstract

The application discloses a data processing method and device, equipment and a computer readable storage medium. The method comprises the following steps: obtaining a plurality of data sources of gateway data in a gateway management platform, the gateway management platform being used for managing a plurality of gateway devices, each gateway device corresponding to a plurality of gateway data elements; for each gateway data element, scoring the plurality of data sources based on the data quality of the plurality of data sources by using a random forest model to obtain a first score of each data source; for each gateway data element, determining the data source with the highest first score in the plurality of first scores as a target data source corresponding to the gateway data element; and in the gateway management platform, for each gateway data element of each gateway device, determining the gateway data in the target data source as target data corresponding to the gateway data element. According to the data processing method, the data quality of the gateway data in the gateway management platform can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a data processing method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] With the development of science and technology, gateway control functions can be realized on the basis of intelligent home gateway supporting hardware devices through specific application programs. Users can manage multiple gateways through the gateway control function in the application program. In order to manage multiple gateways, the application program often needs to be bound to multiple gateways first. Among them, the gateway data corresponding to multiple gateways can come from a gateway management platform. However, if the gateway data in the gateway management platform is inconsistent with the actual gateway data, the gateway binding will fail. Therefore, the data quality of the gateway data in the gateway management platform is crucial.

[0003] At present, the gateway data when the local gateway is shipped is usually obtained by multiple provincial home platforms respectively, and then the gateway data is transmitted to a unified gateway management platform.

[0004] However, since the data volume of the gateway data is large, the gateway data in the provincial home platform is often wrong or missing, and data may be further lost in the data transmission process, resulting in low data quality of the gateway data in the gateway management platform. SUMMARY

[0005] The embodiments of the present application provide a data processing method, device, equipment, computer readable storage medium and computer program product, which can improve the data quality of the gateway data in the gateway management platform.

[0006] In a first aspect, the embodiments of the present application provide a data processing method, which comprises:

[0007] obtaining multiple data sources of gateway data in a gateway management platform, the gateway management platform being used for managing multiple gateway devices, each of the gateway devices corresponding to multiple gateway data elements, and the gateway data elements representing device information of the gateway devices;

[0008] For each of the gateway data elements, a random forest model is used to score the multiple data sources based on the data quality of the multiple data sources, to obtain a first score of each of the data sources;

[0009] For each of the gateway data elements, the data source with the highest first score is determined as a target data source corresponding to the gateway data element in the multiple first scores;

[0010] In the gateway management platform, for each gateway data element of each gateway device, gateway data in the target data source is determined as target data corresponding to the gateway data element.

[0011] In a possible implementation, the random forest includes random forests respectively corresponding to a plurality of data types, and the scoring of the plurality of data sources based on data quality of the plurality of data sources by using the random forest model to obtain a first score of each data source includes:

[0012] A target data type corresponding to the gateway data element is determined from the plurality of data types;

[0013] The scoring of the plurality of data sources based on data quality of the plurality of data sources by using the random forest model respectively corresponding to the plurality of data types to obtain a second score of each data source corresponding to each random forest model;

[0014] A plurality of weights respectively corresponding to the second scores are determined based on the target data type;

[0015] The second scores are weighted and summed according to the plurality of weights to obtain a first score of each data source.

[0016] In a possible implementation, each random forest model includes a plurality of decision trees, and the scoring of the plurality of data sources based on data quality of the plurality of data sources by using the random forest model respectively corresponding to the plurality of data types to obtain a second score of each data source corresponding to each random forest model includes:

[0017] For each data type, each decision tree in the random forest model is used to score the plurality of data sources based on data quality of the plurality of data sources to obtain a third score of each data source;

[0018] For each decision tree of each data type, the data source with the highest third score is determined as a first data source from the plurality of third scores to obtain the first data source corresponding to the plurality of decision trees respectively;

[0019] For each data type, a second score of each data source corresponding to each random forest model is determined according to a number of the first data sources by using a voting method.

[0020] In a possible implementation, before the scoring of the multiple data sources based on the data quality of the multiple data sources by using the random forest model corresponding to the multiple data types, the method further includes:

[0021] obtaining multiple training subsets corresponding to each of the data types;

[0022] training an initial decision tree by using the multiple training subsets respectively for each of the data types to obtain the multiple decision trees;

[0023] determining a random forest model according to the multiple decision trees corresponding to each of the data types to obtain the random forest model corresponding to the multiple data types respectively.

[0024] In a possible implementation, the obtaining of the multiple training subsets corresponding to each of the data types includes:

[0025] obtaining multiple sample data corresponding to each of the data types and a sample label corresponding to each of the sample data;

[0026] determining a feature vector corresponding to each of the sample data based on a type feature of each of the data types;

[0027] constructing a training subset according to the feature vector corresponding to each of the sample data and the sample label to obtain one of the training subsets corresponding to each of the data types, and returning to the obtaining of the multiple sample data corresponding to each of the data types and the sample label corresponding to each of the sample data until the multiple training subsets corresponding to each of the data types are obtained.

[0028] In a possible implementation, before the determining of the feature vector corresponding to each of the sample data based on the type feature of each of the data types, the method further includes:

[0029] constructing a feature set based on the type feature of each of the data types;

[0030] the determining of the feature vector corresponding to each of the sample data based on the type feature of each of the data types includes:

[0031] determining the feature vector corresponding to each of the sample data based on the feature set of each of the data types.

[0032] In a possible implementation, after the determining of the gateway data corresponding to each of the gateway data elements in the target data source as the target data corresponding to the gateway data element for each of the gateway data elements in the gateway management platform, the method further includes:

[0033] For each of the gateway devices, an image of a nameplate of the gateway device is acquired;

[0034] Device information in the nameplate image is recognized by using image recognition technology, the device information including a plurality of gateway data elements and actual data corresponding to each of the gateway data elements;

[0035] For each of the gateway data elements of each of the gateway devices, the actual data is matched with the target data;

[0036] In a case where the actual data is inconsistent with the target data, the target data in the gateway management platform is replaced with the actual data.

[0037] In a second aspect, an embodiment of the present application provides a data processing apparatus, which comprises:

[0038] A first acquisition module is configured to acquire a plurality of data sources of gateway data in a gateway management platform, the gateway management platform being configured to manage a plurality of gateway devices, each of the gateway devices corresponding to a plurality of gateway data elements, the gateway data elements representing device information of the gateway devices;

[0039] A scoring module is configured to, for each of the gateway data elements, score the plurality of data sources based on data quality of the plurality of data sources by using a random forest model, to obtain a first score of each of the data sources;

[0040] A first determination module is configured to, for each of the gateway data elements, determine, in the plurality of first scores, a data source with the highest first score as a target data source corresponding to the gateway data element;

[0041] A second determination module is configured to, in the gateway management platform, for each of the gateway data elements of each of the gateway devices, determine gateway data in the target data source as target data corresponding to the gateway data element.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a processor and a memory storing computer program instructions;

[0043] The processor implements the method in any possible implementation method of the first aspect when executing the computer program instructions.

[0044] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method in any possible implementation method of the first aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, instructions in the computer program product being executed by a processor of an electronic device to cause the electronic device to perform the method in any possible implementation of the method in the first aspect.

[0046] The data processing method, apparatus, device, computer readable storage medium and computer program product provided in the embodiments of the present application can score the multiple data sources based on the data quality of the multiple data sources by using the random forest model for each gateway data element, obtain the first score of each data source, and determine the data source with the highest first score as the target data source corresponding to the gateway data element in the multiple first scores, so that the target data source with the highest data quality can be selected for each gateway data element in the multiple data sources. In this way, the gateway data in the target data source can be determined as the target data corresponding to the gateway data element for each gateway data element of each gateway device in the gateway management platform, so that the data quality of each gateway data element of each gateway device in the gateway management platform can be improved. That is, the data quality of the gateway data in the gateway management platform can be improved by the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced below. Those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0048] Figure 1 FIG. 1 is a flow diagram of a data processing method provided by an embodiment of the present application;

[0049] Figure 2 FIG. 2 is a schematic diagram of determining a target data source based on a random forest model provided by an embodiment of the present application;

[0050] Figure 3 FIG. 3 is a schematic diagram of extracting device information of a gateway device based on an image recognition technology provided by an embodiment of the present application;

[0051] Figure 4 FIG. 4 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0052] Figure 5 FIG. 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] The features and exemplary embodiments of the various aspects of the present application will be described in detail below with reference to the drawings. For the purpose of clarity, not all of the individual features of the application are described in detail herein. The specific embodiments described herein are intended to explain the best mode of the application, and are not intended to limit the scope of the application. The present application can be implemented without some of the specific details, which are not necessary for providing an understanding of the present application. The following description of the embodiments is merely provided to give a better understanding of the present application.

[0054] It should be noted that the relative terms such as first and second, etc., are used herein only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0055] As described in the background section, if the gateway data in the gateway management platform is inconsistent with the actual gateway data, the gateway binding will fail. In addition to this, the gateway detail page will also be displayed in the gateway management interface corresponding to the gateway management function, which is used for the user to manage and control the gateway. However, if the gateway data in the gateway management platform is inconsistent with the actual gateway data, the accuracy of the data displayed in the gateway detail page will be low, which will further affect the user experience of the gateway management function.

[0056] Therefore, in order to solve the problems in the prior art, the embodiments of the present application provide a data processing method, device, equipment, computer readable storage medium and computer program product.

[0057] The data processing method provided by the embodiments of the present application will be introduced below.

[0058] Figure 1 A flowchart of a data processing method provided by the embodiments of the present application is shown. As shown in Figure 1 The data processing method provided by the embodiments of the present application includes the following steps:

[0059] S110, acquire a plurality of data sources of gateway data in a gateway management platform, the gateway management platform being configured to manage a plurality of gateway devices, each of the gateway devices corresponding to a plurality of gateway data elements, each of the gateway data elements representing device information of the gateway device;

[0060] S120, for each of the gateway data elements, score the plurality of data sources based on data quality of the plurality of data sources by using a random forest model, to obtain a first score of each of the data sources;

[0061] S130, for each of the gateway data elements, determine, from the plurality of first scores, a data source with the highest first score as a target data source corresponding to the gateway data element;

[0062] S140, in the gateway management platform, for each of the gateway data elements of each of the gateway devices, determine gateway data in the target data source as target data corresponding to the gateway data element.

[0063] The data processing method according to the embodiments of the present application can select, for each of the gateway data elements, a target data source with the highest data quality from the plurality of data sources by using the random forest model to score the plurality of data sources based on data quality of the plurality of data sources, and obtaining a first score of each of the data sources, and determining, from the plurality of first scores, a data source with the highest first score as a target data source corresponding to the gateway data element. In this way, by determining, in the gateway management platform, for each of the gateway data elements of each of the gateway devices, gateway data in the target data source as target data corresponding to the gateway data element, the data quality of each of the gateway data elements of each of the gateway devices in the gateway management platform can be improved. That is, the data quality of the gateway data in the gateway management platform can be improved by the embodiments of the present application.

[0064] The specific implementation of each of the above steps will be described below.

[0065] In some embodiments, in S110, the gateway management platform can store device information corresponding to a plurality of gateway devices respectively. Each of the gateway devices can correspond to a plurality of gateway data elements. If the gateway data is stored in the form of a data table, the gateway data element can be a field name. The plurality of gateway data elements can include device model, manufacturer, type, medium access control (MAC) address, product serial number (SN), etc. In addition, each of the gateway data elements of each of the gateway devices can correspond to a gateway data.

[0066] In addition, the multiple data sources can include data collected by the soft probe in the running process of the gateway device, data reported by the provincial home platform, data in the third-party data platform, and historical data already in the gateway management platform. The multiple gateway data elements corresponding to the data in different data sources can be completely the same or partially the same, which is not limited herein.

[0067] As an example, the multiple data sources of the gateway data in the gateway management platform can be offline files containing gateway data obtained from multiple different data sources.

[0068] In some embodiments, in S120 and S130, after obtaining the multiple data sources, in order to improve the data quality of the gateway data in the gateway management platform, for each gateway data element, the multiple data sources can be selected and processed to determine the target data source with the optimal data quality. Each gateway data element can correspond to a target data source. The target data sources corresponding to different gateway data elements can be the same or different.

[0069] In addition, the random forest model can be used to determine the target data source with the optimal data quality from the multiple data sources. Specifically, the random forest model can include multiple decision trees. For each gateway data element, the decision trees can score the multiple data sources based on the data quality of the multiple data sources to obtain the score of each data source, and determine the data source with the highest score as the second data source of each gateway data element. The multiple decision trees can correspond to multiple second data sources. After obtaining the multiple second data sources, the number of second data sources can be counted using the voting method, and the first score of each data source can be obtained. The second data source can be any one of the multiple data sources. The number of second data sources can be no more than the number of data sources.

[0070] For example, if there are four data sources A, B, C, and D, the random forest model includes five decision trees, and three of the decision trees correspond to the second data source A and two of the decision trees correspond to the second data source B, then the score of the data source A can be determined as 3 / 5, the score of the data source B can be determined as 2 / 5, and the scores of the data sources C and D can be determined as 0. Since the score of the data source A is the highest, the data source A can be determined as the target data source corresponding to the gateway data element.

[0071] In addition, the random forest model for determining the target data source can include a plurality of random forest models corresponding to a plurality of data types respectively. Each data type can correspond to a random forest model. After determining the target data type of the gateway data element, the random forest model corresponding to the target data type can be used to determine the target data source with the highest data quality from the plurality of data sources, so as to improve the accuracy of the target data source.

[0072] Based on this, in order to further improve the accuracy of the target data source, in some embodiments, S120 can specifically include:

[0073] determining a target data type corresponding to the gateway data element from a plurality of data types;

[0074] scoring the plurality of data sources based on data quality of the plurality of data sources by using a random forest model corresponding to each data type respectively, to obtain a second score of each data source corresponding to each random forest model;

[0075] determining a weight corresponding to each second score based on the target data type;

[0076] performing weighted summation on the plurality of second scores according to the plurality of weights, to obtain a first score of each data source.

[0077] Here, due to the particularity of the gateway data, the plurality of gateway data elements can be divided into a plurality of data types according to the attribute characteristics of the gateway data elements. Among them, the gateway data element and the data type can be a many-to-one relationship. That is, each data type can correspond to a plurality of gateway data elements. Each gateway data element can correspond to a data type. In addition, the plurality of data types can include a format type, a meaning type and a value type.

[0078] For example, for the gateway data element "MAC", the main concern is the format attribute of "MAC", therefore, the data type of the gateway data element "MAC" can be the format type. Among them, the format attribute can include the length, format, case of English letters, etc. of the gateway data corresponding to the gateway data element.

[0079] For the gateway data element "installation address", the main concern is the meaning attribute of "MAC", therefore, the data type of the gateway data element "installation address" can be the meaning type. Among them, the meaning attribute can include whether it really exists, the integrity of the content, etc.

[0080] For the gateway data element "broadband account", the main concern is the value attribute of "broadband account", therefore, the data type of the gateway data element "broadband account" can be the value type. Among them, the value attribute can include whether the broadband account matches the broadband level, the invalidity of the content, etc.

[0081] As an example, the plurality of data types respectively corresponding random forest models can include a first random forest model corresponding to the format type, a second random forest model corresponding to the meaning type, and a third random forest model corresponding to the value type. The second score can include a first sub-score of each data source corresponding to the first random forest, a second sub-score of each data source corresponding to the second random forest, and a third sub-score of each data source corresponding to the third random forest.

[0082] Based on this, the above scoring of the plurality of data sources based on the data quality of the plurality of data sources by using the random forest model corresponding to the plurality of data types, to obtain the second score of each data source corresponding to each random forest model, can specifically include:

[0083] scoring of the plurality of data sources based on the data quality of the plurality of data sources by using the first random forest model, to obtain the first sub-score of each data source;

[0084] scoring of the plurality of data sources based on the data quality of the plurality of data sources by using the second random forest model, to obtain the second sub-score of each data source;

[0085] scoring of the plurality of data sources based on the data quality of the plurality of data sources by using the third random forest model, to obtain the third sub-score of each data source.

[0086] That is, for each gateway data element, each random forest model can output scores corresponding to the plurality of data sources respectively (including the first sub-score, the second sub-score and the third sub-score). For each data source, the final score (i.e. the first score) can be determined by a dynamic standard decision strategy. The formula corresponding to the dynamic standard decision strategy can be as follows:

[0087]

[0088] In the above formula, p i may be the prediction value of the i-th random forest, w i may be the weight corresponding to the i-th random forest.

[0089] Specifically, the distribution criteria of the weights of different gateway data elements can be dynamically changed. The distribution criteria can be based on the content category, i.e., the data type, carried by the data element. That is, the weights of the plurality of second scores corresponding to different data types can be different. In the case where the target data type is determined, the weights of the plurality of second scores can be determined. The weights of the plurality of first sub-scores can be the same, the weights of the plurality of second sub-scores can be the same, and the weights of the plurality of third sub-scores can be the same. For the same data source, the sum of the weights of the first sub-score, the second sub-score, and the third sub-score can be 1.

[0090] For example, for the gateway data element "MAC", the format can be the most important measurement index, and the meaning and value have little reference value. Therefore, the weights of the results output by the three random forest models can be 0.9, 0.05, and 0.05, respectively; and for the gateway data element "installation address", the meaning can be the most important measurement index, the value is secondary, and the format has little reference value. Therefore, the weights of the results output by the three random forest models can be 0.05, 0.6, and 0.35, respectively. Other different gateway data elements can also adopt similar distribution criteria. The weight distribution coefficient can be not more than 0.9 at the highest and not less than 0.05 at the lowest, ensuring that the output results of each random forest model have an impact on the final result and reducing the inaccuracy caused by a single standard.

[0091] In this way, by determining the target data source based on the above dynamic standard decision strategy, more flexible and accurate data source quality rating of multi-source data can be achieved, and the accuracy of the target data source is further improved.

[0092] In addition, each random forest can include a plurality of decision trees. Based on this, in some embodiments, the above scoring of the plurality of data sources based on the data quality of the plurality of data sources by using the random forest model corresponding to each data type to obtain the second score of each data source corresponding to each random forest model can specifically include:

[0093] For each data type, each decision tree in the random forest model is used to score the plurality of data sources based on the data quality of the plurality of data sources to obtain a third score of each data source;

[0094] For each decision tree of each data type, the data source with the highest third score in the plurality of third scores is determined as the first data source to obtain the first data source corresponding to each decision tree;

[0095] For each data type, the second score of each data source corresponding to each random forest model is determined according to the number of the plurality of first data sources by using a voting method.

[0096] Here, since one data type can correspond to one random forest model. Therefore, for each random forest model, the second score of each data source can be calculated in the above manner. In addition, the first data source can be the most reliable data source determined by the decision tree. If m decision trees are included in the random forest, m first data sources can be obtained. Among them, the m first data sources can be completely the same, or partially the same. In each random forest model, after obtaining the m first data sources corresponding to the m decision trees respectively, a voting method can be used for selection, and the data source with the most votes among the m first data sources is determined as the target data source.

[0097] For example, if there are 4 data sources A, B, C, and D, the first random forest model includes 5 decision trees, and 3 of the decision trees correspond to the first data source A, and 2 of the decision trees correspond to the first data source B, then the score of the data source A can be determined as 3 / 5, the score of the data source B can be determined as 2 / 5, and the scores of the data source C and the data source D are both 0. Since the score of the data source A is the highest, the data source A can be determined as the target data source corresponding to the gateway data element.

[0098] Based on this, in some embodiments, before the above scoring of the plurality of data sources based on the data quality of the plurality of data sources using the random forest model corresponding to each data type, it can further include:

[0099] Obtaining a plurality of training subsets corresponding to each data type;

[0100] For each data type, training the initial decision tree using the plurality of training subsets to obtain a plurality of decision trees;

[0101] Determining a random forest model according to the plurality of decision trees corresponding to each data type, to obtain a random forest model corresponding to each data type.

[0102] Here, each data type can correspond to a plurality of training subsets, and each training subset can train a decision tree. For each data type, the decision trees trained by the plurality of training subsets can form a random forest model.

[0103] In addition, each training subset can include a plurality of sample data and sample labels corresponding thereto. Each sample data can include C features. For each feature a, an initial decision tree can be constructed according to the information gain of the feature. The feature with the maximum information gain can be used as the root node of the decision tree. The calculation process of the information gain can be as follows:

[0104] gain(D, a) = H(D) - H(D|a)

[0105] The meaning of the above formula can be to calculate the information entropy of the data set D and the conditional entropy of D under the given condition of the feature a. The greater the difference is, the greater the "influence" of the feature value in all feature sets can be represented. Therefore, the feature with the maximum information gain is suitable as the root node. Then, when each tree node needs to be split, c features can be randomly selected from C features, and the condition c << C is met, and the process of calculating the information gain is repeatedly executed to obtain the next node.

[0106] After the initial decision tree is constructed, the initial decision tree can be trained by using multiple training subsets to obtain multiple decision trees.

[0107] Based on this, in some embodiments, the above obtaining multiple training subsets corresponding to each data type can specifically include:

[0108] Obtaining multiple sample data corresponding to each data type and sample labels corresponding to each sample data;

[0109] Determining feature vectors respectively corresponding to the multiple sample data based on type characteristics of each data type;

[0110] Constructing training subsets according to the feature vectors respectively corresponding to the multiple sample data and the sample labels to obtain one training subset corresponding to each data type, and returning to execute the obtaining of the multiple sample data corresponding to each data type and the sample labels corresponding to each sample data until the multiple training subsets corresponding to each data type are obtained.

[0111] Here, each data type can correspond to a sample data set. The sample data set can include multiple gateway data elements or one gateway data element, which is not limited here. In addition, each gateway data element can correspond to multiple gateway data.

[0112] As an example, when obtaining a training subset corresponding to a certain data type, a gateway data can be randomly selected from a sample data set corresponding to the data type, and a feature vector and a sample label corresponding to the gateway data are determined. Then, the gateway data is put back into the sample data set, and a gateway data is randomly selected again, and the above process is repeated. If it is set in advance that n training data are included in each training subset, by repeating the above steps, a gateway data is randomly taken out from the sample data set with replacement each time, and then the feature vector and the sample label of the gateway data are determined, until n times of sampling, that is, a training subset D can be obtained. The above process of obtaining the training subset D is repeated again, and if it is repeated m times, m training subsets can be obtained.

[0113] In addition, each data type can correspond to multiple type characteristics. For example, the multiple type characteristics corresponding to the format type can include the length, format, and case of English letters of the gateway data corresponding to the gateway data element. The multiple type characteristics corresponding to the meaning type can include whether the content is actually present and the integrity of the content. The multiple type characteristics corresponding to the value type can include whether the broadband account matches the broadband level and the timeliness of the content.

[0114] As an example, if a certain data type includes C type characteristics, for each type characteristic, if the sample data satisfies the type characteristic, the characteristic can be encoded as 1, and if the sample data does not satisfy the type characteristic, the characteristic can be encoded as 0. Repeating the above process can obtain the feature vector corresponding to the sample data. The form of the feature vector can be, for example, [1, 0, …, 0, 0]. In addition, the sample label corresponding to the feature vector can be determined by the following method: for the gateway data element corresponding to the sample data, the feature value scores of different data sources are calculated in turn according to the format standard, the meaning standard, and the value attribute standard, and the last data source with the highest score is taken as the label of the training set.

[0115] Based on this, in order to determine the feature vector corresponding to the sample data, in some embodiments, before the above determining the feature vector corresponding to each data type, the method can further include:

[0116] Constructing a feature set based on the type characteristics of each data type.

[0117] Based on this, the above determining the feature vector corresponding to each data type, can specifically include:

[0118] Determining the feature vector corresponding to each data type based on the feature set of each data type.

[0119] Here, for each gateway data element, a feature value set S can be established according to the format attribute, the meaning attribute, and the value attribute, the set size is C, and the feature values are divided into three standards: format standard, meaning standard, and value standard, denoted as S1, S2, and S3. Among them, S1 can include format type characteristics such as data element length, data element format, and whether the case of English letters is unified; S2 can include meaning type characteristics such as whether it is not empty, whether it is actually present, and content integrity; and S3 can include value type characteristics such as whether the version number is in the historical library, whether the broadband account matches the broadband level, and the timeliness of the content.

[0120] Thus, after obtaining a sample data, the feature value of the sample data can be obtained according to the features in the feature value set, and the form of the obtained feature vector can be, for example, [1, 0, …, 0, 0]. Wherein, 1 represents satisfying the feature, and 0 represents not satisfying the feature.

[0121] Based on the above embodiments, in the embodiments of the present application, the schematic diagram for determining the target data source based on the random forest model can be as shown in Figure 2 In Figure 2 , the training process of the random forest model corresponding to each of the three data types is included, and the application process of determining the target data source based on the three random forest models is included.

[0122] In some embodiments, in S140, after determining the target data source, the gateway data in the target data source corresponding to the gateway device and the gateway data element can be determined as the target data in the gateway management platform corresponding to the gateway device and the gateway data element. For example, if it is determined that the target data source of the gateway data element "MAC" is the data reported by the provincial home platform, the "MAC" corresponding to each of the plurality of gateway devices in the gateway management platform can be sourced from the data reported by the provincial home platform. The process of determining the target data can be the process of data governance based on big data.

[0123] Based on this, in order to further improve the data quality of the gateway data in the gateway management platform, in some embodiments, after the above S140, the following steps can be further included:

[0124] For each gateway device, obtain the nameplate image of the gateway device;

[0125] Recognize the device information in the nameplate image by using image recognition technology, and the device information includes a plurality of gateway data elements and actual data corresponding to each gateway data element;

[0126] For each gateway data element of each gateway device, match the actual data with the target data;

[0127] In the case where the actual data is inconsistent with the target data, the target data in the gateway management platform is replaced by the actual data.

[0128] Here, the nameplate image of the gateway device can be a real image obtained by taking a photo of the gateway device. In addition, the image recognition technology can be, for example, an optical character recognition (OCR) technology.

[0129] As an example, after obtaining the tag image, the effective text information on the tag image can be obtained by performing text recognition on the image content by using an OCR image recognition technology. After screening, information related to the device can be selected and output, and the device information can include, for example, a device model, a MAC, an SN, a manufacturer, a service set identifier (SSID), and the like. An example of a method for extracting device information of a gateway device based on an image recognition technology provided in this application can be as shown in FIG. 8. As shown in FIG. 9, the device information can include a plurality of gateway data elements and actual data corresponding to each gateway data element. After obtaining the actual data, the actual data can be used to correct the target data in the gateway management platform. Specifically, for each gateway data element of each gateway device, the actual data can be matched with the target data. If the actual data is consistent with the target data, the target data in the gateway management platform can be determined as the final data. If the actual data is inconsistent with the target data, the target data in the gateway management platform can be replaced with the actual data, that is, the actual data is determined as the final data. Figure 3 Figure 3 As shown in FIG. 9, the device information can include a plurality of gateway data elements and actual data corresponding to each gateway data element. After obtaining the actual data, the actual data can be used to correct the target data in the gateway management platform. Specifically, for each gateway data element of each gateway device, the actual data can be matched with the target data. If the actual data is consistent with the target data, the target data in the gateway management platform can be determined as the final data. If the actual data is inconsistent with the target data, the target data in the gateway management platform can be replaced with the actual data, that is, the actual data is determined as the final data.

[0130] For a certain gateway device, if the target data corresponding to the gateway data element "SN" is "CMHIABCD1234" and the actual data is "CMHIABCD4321", the data finally stored in the gateway management platform is the actual data.

[0131] In this way, by using the actual data to correct the target data in the gateway management platform, the data quality of the gateway data in the gateway management platform can be further improved.

[0132] In addition, for data that cannot be obtained through big data governance, the device information recognized by the image can be used for supplementation. For example, in the big data governance process, the SSID can not be included in multiple data sources, but the device information recognized by the image includes the SSID. Therefore, the image recognition can be supplemented to the gateway management platform, and the data quality of the gateway data in the gateway management platform can be further improved.

[0133] In this way, by using the actual data to correct the target data in the gateway management platform, the data quality of the gateway data in the gateway management platform can be further improved.

[0134] ​In the random forest algorithm optimization part, the traditional random forest algorithm randomly selects feature values from a unified feature value library to generate a sub-tree for prediction. In this scheme, different types of feature values are classified to form different feature value libraries. When using the random forest algorithm, each feature value library corresponds to an optimal data source, and the data source quality is evaluated in multiple dimensions and targeted.

[0135] In the dynamic standard decision part, the optimal data source output by each feature value library dynamically allocates the most appropriate weight distribution coefficient for different data elements, and the evaluation results are weighted and added. Finally, the quality rating score of all data sources under the target data element is obtained, and the data element carried by the data source with the highest final selection result score is selected as the final storage result, improving the flexibility of governance and the accuracy of the result.

[0136] Compared with the random forest algorithm of the prior art, which only uses a unified feature value library, this scheme separately establishes a feature value library according to the feature value category, and the prediction result using the random forest algorithm is more targeted and reasonable. Compared with the unified standard governance data element of the prior art, this scheme uses a dynamic standard decision strategy for different data elements, making the governance model more flexible and the output result more accurate.

[0137] In the data correction part, the OCR image recognition technology is innovatively used in the gateway data governance field to analyze the effective data on the gateway nameplate, correct the gateway data in the database, and supplement the missing fields. The data correction method is expanded from traditional stream message processing to image recognition processing.

[0138] Therefore, by using the random forest model to determine the target data source and using the image recognition technology to correct and supplement the target data, the data quality of the gateway data in the gateway management platform can be effectively improved. The problem of gateway binding failure caused by the inconsistency between the gateway data and the actual data in the process of using a specific application program to use the gateway management and control is solved, and the accuracy of the displayed data in the gateway detail page corresponding to the gateway management and control function is improved.

[0139] Based on the data processing method provided in the above embodiments, the application also provides a specific implementation of a data processing device. Please refer to the following embodiments.

[0140] As shown in Figure 4 The data processing device 400 provided by the embodiments of the application includes the following modules:

[0141] The first acquisition module 410 is configured to acquire a plurality of data sources of gateway data in a gateway management platform, the gateway management platform being configured to manage a plurality of gateway devices, each gateway device corresponding to a plurality of gateway data elements, the gateway data elements representing device information of the gateway devices;

[0142] The scoring module 420 is configured to score, for each gateway data element, the multiple data sources based on data quality of the multiple data sources by using the random forest model, to obtain a first score of each data source.

[0143] The first determining module 430 is configured to determine, for each gateway data element, a data source with the highest first score in the multiple first scores as a target data source corresponding to the gateway data element.

[0144] The second determining module 440 is configured to determine, in the gateway management platform, for each gateway data element of each gateway device, gateway data in the target data source as target data corresponding to the gateway data element.

[0145] The data processing apparatus 400 is described in detail as follows.

[0146] In some embodiments, the random forest includes a random forest corresponding to each data type. Based on this, the scoring module 420 can specifically include:

[0147] The first determining sub-module is configured to determine, in the multiple data types, a target data type corresponding to the gateway data element.

[0148] The scoring sub-module is configured to score, for each data type, the multiple data sources based on data quality of the multiple data sources by using the random forest model corresponding to the data type, to obtain a second score of each data source corresponding to each random forest model.

[0149] The second determining sub-module is configured to determine, based on the target data type, a weight corresponding to each second score.

[0150] The summing sub-module is configured to weight and sum the multiple second scores according to the multiple weights, to obtain the first score of each data source.

[0151] In some embodiments, each random forest model includes multiple decision trees. Based on this, the scoring sub-module can specifically include:

[0152] The scoring unit is configured to score, for each data type, the multiple data sources based on data quality of the multiple data sources by using each decision tree in the random forest model, to obtain a third score of each data source.

[0153] The first determining unit is configured to determine, for each decision tree of each data type, a data source with the highest third score in the multiple third scores as a first data source, to obtain a first data source corresponding to each decision tree.

[0154] The second determining unit is configured to determine, for each data type, a second score of each data source corresponding to each random forest model according to a quantity of the plurality of first data sources by using a voting method.

[0155] In some embodiments, the scoring module 420 can further include:

[0156] The obtaining submodule is configured to obtain a plurality of training subsets corresponding to each data type before scoring the plurality of data sources based on data quality of the plurality of data sources by using the random forest model corresponding to each data type.

[0157] The training submodule is configured to train the initial decision tree by using the plurality of training subsets respectively for each data type to obtain a plurality of decision trees.

[0158] The third determining submodule is configured to determine the random forest model according to the plurality of decision trees corresponding to each data type to obtain the random forest model corresponding to each data type.

[0159] In some embodiments, the obtaining submodule can include:

[0160] The obtaining unit is configured to obtain a plurality of sample data corresponding to each data type and a sample label corresponding to each sample data.

[0161] The third determining unit is configured to determine a feature vector corresponding to each sample data based on a type feature of each data type.

[0162] The execution unit is configured to construct a training subset according to the feature vector corresponding to each sample data and the sample label to obtain one training subset corresponding to each data type, and return to execute the obtaining of the plurality of sample data corresponding to each data type and the sample label corresponding to each sample data until the plurality of training subsets corresponding to each data type are obtained.

[0163] In some embodiments, the obtaining submodule can further include:

[0164] The construction unit is configured to construct a feature set based on the type feature of each data type before determining the feature vector corresponding to each sample data based on the type feature of each data type.

[0165] The fourth determining unit is configured to determine the feature vector corresponding to each sample data based on the type feature of each data type, including:

[0166] The fifth determining unit is configured to determine the feature vector corresponding to each sample data based on the feature set of each data type.

[0167] In some embodiments, the data processing apparatus 400 can further include:

[0168] a second obtaining module configured to obtain a nameplate image of the gateway device for each gateway device;

[0169] a recognition module configured to recognize device information in the nameplate image by using an image recognition technology, the device information including a plurality of gateway data elements and actual data corresponding to each gateway data element;

[0170] a matching module configured to match the actual data with target data for each gateway data element of each gateway device;

[0171] a replacement module configured to replace the target data in the gateway management platform with the actual data in a case where the actual data is inconsistent with the target data.

[0172] The data processing apparatus of the embodiments of the present application can score a plurality of data sources based on data quality of the plurality of data sources by using a random forest model for each gateway data element, obtain a first score of each data source, and determine a data source with the highest first score in the plurality of first scores as a target data source corresponding to the gateway data element, so as to select a target data source with the highest data quality for each gateway data element from the plurality of data sources. In this way, the gateway data in the target data source is determined as target data corresponding to the gateway data element for each gateway data element of each gateway device in the gateway management platform, so as to improve the data quality of each gateway data element of each gateway device in the gateway management platform. That is, the data quality of the gateway data in the gateway management platform can be improved by the embodiments of the present application.

[0173] Based on the data processing method provided in the above embodiments, the embodiments of the present application further provide a specific implementation of an electronic device. Figure 5 A schematic diagram of an electronic device 500 provided by the embodiments of the present application is shown.

[0174] The electronic device 500 can include a processor 510 and a memory 520 storing computer program instructions.

[0175] Specifically, the processor 510 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits that implement the embodiments of the present application.

[0176] The memory 520 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 520 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 520 can include removable or non-removable (or fixed) media, where appropriate. The memory 520 can be internal or external to the electronic device 500. In particular embodiments, the memory 520 is non-volatile, solid-state memory.

[0177] The memory can include read-only memory (ROM), random-access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to access the data and / or instructions as described with reference to the methods according to the first aspect of the application.

[0178] The processor 510 implements any of the data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 520.

[0179] In one example, the electronic device 500 can further include a communication interface 530 and a bus 540. Wherein, as shown, the processor 510, the memory 520, the communication interface 530 are connected through the bus 540 and complete the communication between each other. Figure 5

[0180] The communication interface 530 is mainly used to realize the communication between each module, device, unit and / or equipment in the embodiments of the application.

[0181] ​Bus 540 includes hardware, software, or both, to couple electronic devices to each other in a manner that allows information to be passed between or among them. The bus can include, for example, an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 540 can include one or more buses. Although the example embodiments described and illustrated herein relate to a particular bus, the application contemplates any suitable bus or interconnect.

[0182] By way of example, electronic device 500 can be a cell phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc.

[0183] The electronic device can execute the data processing method in the embodiments of the application, thereby realizing the data processing method and apparatus described in combination Figures 1 to 4 with the data processing method and apparatus.

[0184] In addition, in combination with the data processing method in the above embodiments, the embodiments of the application can provide a computer-readable storage medium to implement. The computer-readable storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any one of the data processing methods in the above embodiments.

[0185] It needs to be made clear that the application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the application.

[0186] The functions noted in the description of the structural block diagrams above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, functional cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transport information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0187] It is also important to note that the examples described herein can be implemented in a variety of systems, including and / or incorporating software, firmware, hardware, and / or circuitry. Also, the examples described herein are meant to be examples and embodiments of the application only and various changes, modifications and improvements, in addition to those not specifically described, can be made. For example, features described herein can be combined in any combination desired.

[0188] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. Alternatively, computer program implemented steps can be implemented by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and computer hardware. Those skilled in the art will recognize that the examples described herein are not limited to a computer or other programmable data processing apparatus, and that steps of a computer program can be performed by software, hardware, and / or firmware, and that a computer program can be written in any suitable language of instruction.

[0189] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A data processing method, characterized in that, include: Multiple data sources are obtained from the gateway management platform, which is used to manage multiple gateway devices. Each gateway device corresponds to multiple gateway data elements, and the gateway data elements represent the device information of the gateway device. For each gateway data element, a random forest model is used to score the multiple data sources based on their data quality, resulting in a first score for each data source. For each gateway data element, among multiple first scores, the data source with the highest first score is determined as the target data source corresponding to the gateway data element; In the gateway management platform, for each gateway data element of each gateway device, the gateway data in the target data source is determined as the target data corresponding to the gateway data element; For each of the gateway devices, obtain the nameplate image of the gateway device; The device information in the nameplate image is identified using image recognition technology. The device information includes multiple gateway data elements and the actual data corresponding to each gateway data element. For each gateway data element of each gateway device, the actual data is matched with the target data; If the actual data and the target data are inconsistent, the target data in the gateway management platform will be replaced with the actual data.

2. The method according to claim 1, characterized in that, The random forest includes random forests corresponding to multiple data types. The step of using the random forest model to score the multiple data sources based on their data quality, obtaining a first score for each data source, includes: Determine the target data type corresponding to the gateway data element from among the multiple data types; Using random forest models corresponding to the multiple data types respectively, the multiple data sources are scored based on their data quality to obtain a second score for each data source corresponding to each random forest model; Based on the target data type, determine the weights corresponding to the multiple second scores respectively; The second scores are weighted and summed according to the weights to obtain a first score for each data source.

3. The method according to claim 2, characterized in that, Each of the random forest models includes multiple decision trees. The step of using the random forest models corresponding to the multiple data types to score the multiple data sources based on their data quality, and obtaining a second score for each data source corresponding to each random forest model, includes: For each data type, each decision tree in the random forest model is used to score the multiple data sources based on their data quality, resulting in a third score for each data source. For each decision tree of each data type, among a plurality of third scores, the data source with the highest third score is determined as the first data source, thereby obtaining the first data source corresponding to each of the plurality of decision trees; For each data type, a second score for each data source corresponding to each random forest model is determined using a voting method based on the number of multiple first data sources.

4. The method according to claim 3, characterized in that, Before scoring the multiple data sources based on their data quality using random forest models corresponding to the multiple data types, the method further includes: Obtain multiple training subsets corresponding to each of the data types; For each of the aforementioned data types, the initial decision tree is trained using the plurality of training subsets to obtain the plurality of decision trees; A random forest model is determined based on the multiple decision trees corresponding to each data type, thus obtaining random forest models corresponding to the multiple data types respectively.

5. The method according to claim 4, characterized in that, The step of obtaining multiple training subsets corresponding to each of the data types includes: Obtain multiple sample data corresponding to each of the data types and sample labels corresponding to each of the sample data; Based on the type characteristics of each of the data types, determine the feature vectors corresponding to the multiple sample data respectively; Based on the feature vectors and sample labels corresponding to the multiple sample data respectively, a training subset is constructed to obtain a training subset corresponding to each data type, and the process of obtaining multiple sample data corresponding to each data type and the sample labels corresponding to each sample data is repeated until the multiple training subsets corresponding to each data type are obtained.

6. The method according to claim 5, characterized in that, Before determining the feature vectors corresponding to the plurality of sample data based on the type characteristics of each of the data types, the method further includes: Construct a feature set based on the type characteristics of each of the data types; The step of determining the feature vectors corresponding to the plurality of sample data based on the type characteristics of each of the data types includes: Based on the feature set of each data type, the feature vectors corresponding to the plurality of sample data are determined respectively.

7. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire multiple data sources of gateway data in the gateway management platform. The gateway management platform is used to manage multiple gateway devices. Each gateway device corresponds to multiple gateway data elements, and the gateway data elements represent the device information of the gateway device. The scoring module is used to score the multiple data sources based on the data quality of the multiple data sources for each gateway data element using a random forest model, and obtain a first score for each data source. The first determining module is used to determine, for each gateway data element, the data source with the highest first score among a plurality of first scores as the target data source corresponding to the gateway data element; The second determining module is used in the gateway management platform to determine the gateway data in the target data source as the target data corresponding to the gateway data element for each gateway data element of each gateway device; The second acquisition module is used to acquire the nameplate image of each of the gateway devices. The identification module is used to identify the device information in the nameplate image using image recognition technology. The device information includes multiple gateway data elements and the actual data corresponding to each gateway data element. The matching module is used to match the actual data with the target data for each gateway data element of each gateway device; The replacement module is used to replace the target data in the gateway management platform with the actual data when the actual data is inconsistent with the target data.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data processing method as described in any one of claims 1-6.

Images