Data processing method and apparatus, electronic device, and computer program product

Abstract

The application relates to the technical field of data processing, and provides a data processing method and device, electronic equipment and a computer program product. The method comprises the following steps: determining a characteristic attribute of to-be-processed data; classifying the to-be-processed data according to the characteristic attribute; determining a classification data model according to a classification result and a first knowledge graph, so as to construct a second knowledge graph according to the classification data model; the first knowledge graph is determined based on individual big data, and the second knowledge graph is a knowledge graph of product application; and updating a data model and data correlation information according to the classification result, the classification data model and the second knowledge graph. Through data classification and knowledge graph processing of to-be-processed data, the application solves the problems of low data use maintenance efficiency, high cost and high data redundancy, and improves the data processing efficiency and use value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and device, electronic equipment and computer program product. BACKGROUND

[0002] At present, with the further deepening of big data commercialization, the operator big data, on the basis of information collection, classification, summarization, analysis and other processing of each user, combined with industry information, constructs product application models such as tourism scenic spot analysis, traffic application analysis, and job and residence commuting analysis, and makes corresponding product recommendations to customers, which shows the value and ability of data. The basis of these data value applications is data classification.

[0003] Data classification is the premise of data management and the basis of data value utilization. Traditional operator data classification generally classifies data through manual or simple classification algorithms, which has the problems of high labor cost, high maintenance cost, long processing time, low classification efficiency, incomplete and inaccurate classification data, and various categories with cross, nesting and redundancy. Therefore, how to process data is a problem to be solved at present. SUMMARY

[0004] The embodiments of the present application provide a data processing method, device, electronic equipment and computer program product to solve the problem of data processing.

[0005] In a first aspect, the embodiments of the present application provide a data processing method, comprising:

[0006] determining the characteristic attribute of the to-be-processed data;

[0007] classifying the to-be-processed data according to the characteristic attribute;

[0008] determining a classification data model according to the classification result and a first knowledge graph, so as to construct a second knowledge graph according to the classification data model; the first knowledge graph is determined based on individual big data, and the second knowledge graph is a knowledge graph of product application;

[0009] updating the data model and data association information according to the classification result, the classification data model and the second knowledge graph.

[0010] In one embodiment, the determining of the characteristic attribute of the to-be-processed data comprises:

[0011] constructing a training set from the to-be-processed data to perform feature value discretization processing on the to-be-processed data in the training set;

[0012] The feature value difference processing is performed on the processed data after the feature value discretization processing, and the feature value dimension reduction processing is performed on the processed data after the feature value difference processing.

[0013] The decision tree is constructed according to the processed data after the feature value dimension reduction processing, and the decision tree is sampled according to the one-side sampling mode of the gradient.

[0014] The feature value merging is performed on the plurality of decision trees sampled, and the feature attribute of the processed data is determined according to the decision tree after the feature value merging.

[0015] In one embodiment, the feature value dimension reduction processing is performed on the processed data after the feature value difference processing, comprising:

[0016] The weight graph is constructed according to the processed data after the feature value difference processing.

[0017] The feature values in the weight graph are sorted in descending order, and the feature values in the weight graph are fused and bound according to the sorting result.

[0018] In one embodiment, the classification of the processed data according to the feature attribute comprises:

[0019] The feature vector of each of the processed data is determined according to the feature attribute.

[0020] The cosine similarity of each of the feature vectors and the target vector is determined, and the neighbor sample set is determined according to each of the cosine similarity.

[0021] The feature category to which the feature value with the maximum cosine similarity belongs is determined according to the neighbor sample set.

[0022] The processed data is classified according to the frequency of occurrence of each of the feature categories.

[0023] In one embodiment, the classification data model is determined according to the classification result and the first knowledge graph, comprising:

[0024] The association information of each of the processed data is determined according to the classification result and the first knowledge graph.

[0025] The classification data model is determined according to the association information of each of the processed data.

[0026] In one embodiment, the second knowledge graph is constructed according to the classification data model, comprising:

[0027] The association information of the data model and the data capability in the classification data model is determined, and the classification data model comprises a plurality of data models.

[0028] construct the second knowledge graph according to the association information between the data model and the data capability.

[0029] In one embodiment, the updating of the data model and the data association information according to the classification result, the classification data model and the second knowledge graph comprises:

[0030] determining the association information between the entity and the concept in the big data application layer according to the classification result, the classification data model and the second knowledge graph;

[0031] updating the data model and the data association information according to the association information between the entity and the concept.

[0032] In a second aspect, the embodiments of the present application provide a data processing apparatus, comprising:

[0033] a feature attribute determining module configured to determine the feature attribute of the to-be-processed data;

[0034] a classification module configured to classify the to-be-processed data according to the feature attribute;

[0035] a knowledge graph constructing module configured to determine a classification data model according to the classification result and a first knowledge graph, and to construct a second knowledge graph according to the classification data model; the first knowledge graph is determined based on individual big data, and the second knowledge graph is a knowledge graph of product application;

[0036] an updating module configured to update the data model and the data association information according to the classification result, the classification data model and the second knowledge graph.

[0037] In a third aspect, the embodiments of the present application provide an electronic device, comprising a processor and a memory storing a computer program, wherein the processor implements the steps of the data processing method of the first aspect when executing the program.

[0038] In a fourth aspect, the embodiments of the present application provide a computer program product, comprising a computer program, wherein the computer program implements the steps of the data processing method of the first aspect when executed by a processor.

[0039] The data processing method, device, electronic device and computer program product provided by the embodiments of the present application determine the feature attribute of the to-be-processed data, classify the to-be-processed data according to the feature attribute, determine a classification data model according to the classification result and a first knowledge graph, construct a second knowledge graph according to the classification data model, the first knowledge graph is determined based on individual big data, and the second knowledge graph is a knowledge graph of product application, and update the data model and data association information according to the classification result, the classification data model and the second knowledge graph. The present application solves the problems of low data use and maintenance efficiency, high cost and high data redundancy by classifying the to-be-processed data and constructing a knowledge graph, and improves the data processing efficiency and use value. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0041] Figure 1 is a flowchart of the data processing method provided by the embodiments of the present application;

[0042] Figure 2 is a schematic diagram of constructing a decision tree with depth limit based on a leaf-wise method provided by the embodiments of the present application;

[0043] Figure 3 is a feature sorting schematic diagram provided by the embodiments of the present application;

[0044] Figure 4 is a data classification schematic diagram provided by the embodiments of the present application;

[0045] Figure 5 is a structural schematic diagram of the data processing device provided by the embodiments of the present application;

[0046] Figure 6 is a structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present application. Referring to Figure 1 The present embodiment provides a data processing method, which can include:

[0049] Step 100, determining the characteristic attributes of the data to be processed;

[0050] To solve the problems of large storage occupation, slow algorithm speed and much manual intervention in feature importance sorting and feature selection processing in the data classification process, the present embodiment introduces a LightGBM (Light Gradient Boosting Machine, light gradient boosting machine learning) algorithm, and sorts the feature attributes by information gain and extracts the feature attributes.

[0051] Specifically, data from each production system such as a business system, a billing system and a network system is collected, and the collected data is cleaned, converted, processed and stored to form user data such as user basic information, communication behavior and payment behavior, which is used as the basic data for subsequent automatic classification of operator big data, big data application and data value-added, i.e. basic data.

[0052] For example, the user data of the location information obtained by cleaning and converting the source data is shown in Table 1 and Table 2:

[0053] Table 1

[0054]

[0055]

[0056] Table 2

[0057]

[0058] The characteristic attributes of the basic data (i.e. the data to be processed) collected based on the LightGBM algorithm, and the processing steps include:

[0059] 1) The continuous or accurate feature values are divided into a series of discrete values, based on which the storage space and the calculation amount are reduced, and the algorithm running speed is improved.

[0060] 2) The feature values are processed by difference, in the actual tree construction process, the small leaf nodes of the histogram are calculated first, and then the large sibling nodes of the histogram are directly obtained by using the histogram difference, based on which the calculation cost is reduced.

[0061] 3) Because the data volume of the operator is very large, high-dimensional data storage consumes space and time. Therefore, by fusing and binding the eigenvalues, the time complexity of histogram construction can be reduced.

[0062] 4) When traversing the decision tree learning, find the leaf with the largest split gain among all current leaves to split, which can obtain better accuracy under the same number of splits, and limit the depth of the tree to avoid overfitting.

[0063] 5) In the process of using, the sample points with large gradient are retained, and the sample points with small gradient are randomly sampled in proportion. In this way, the learning time of the model can be reduced without changing the data distribution and losing the accuracy of the learning machine.

[0064] 6) In the process of parallel learning of multiple learning machines, a voting parallelization learning method is used for parallel learning of multiple learning machines according to the large data volume of the operator.

[0065] Based on the above steps, the importance of the feature attributes of the to-be-processed data is sorted and the feature attributes are extracted.

[0066] Step 200, classifying the to-be-processed data according to the feature attributes;

[0067] To solve the problems of low efficiency of manual classification, cross-nested redundancy of categories, and the like, the KNN (K-Nearest Neighbor) algorithm is introduced to automatically classify the data. Specifically, the feature data obtained based on the LightGBM algorithm is normalized and arranged into a feature vector, then the cosine similarity between the to-be-classified sample and each sample in the data set is calculated, and the classification corresponding to the top k cosine similarities is selected. After automatic classification, manual category evaluation is performed, thereby improving the classification speed and accuracy.

[0068] Step 300, determining a classification data model according to the classification result and the first knowledge graph, so as to construct a second knowledge graph according to the classification data model;

[0069] To solve the problem of poor visualization of the relationship between data, and to prevent cross-redundancy storage of data in the data application design stage, the application embodiment mines the association information between data, constructs an individual big data knowledge graph, and establishes a classification data model of product application and a big data knowledge graph of product application.

[0070] Specifically, according to the concepts, entities, attributes, and inter-entity correlation information, a big data knowledge graph of an individual is constructed to form knowledge correlation; then, based on the automatically classified data, a classified data model of product application is established; and based on the classified data model, according to the correlation information of the data model and data capability, a big data knowledge graph of product application is constructed.

[0071] It can be understood that the first knowledge graph is determined based on big data of an individual, that is, a big data knowledge graph of an individual; and the second knowledge graph is a knowledge graph of product application, that is, a big data knowledge graph of product application.

[0072] At step 400, the data model and data correlation information are updated according to the classification result, the classified data model, and the second knowledge graph.

[0073] To solve the knowledge reuse and data value-added problems, the embodiments of the present application, based on the classified data model of product application and the big data knowledge graph, combine the new business environment to reason and deduce the knowledge graph, realize new application and new value of data in the new environment without increasing data redundancy.

[0074] Specifically, based on the automatic classification result of data, the classified data model of product application, and the big data knowledge graph of product application (that is, the second knowledge graph), the knowledge graph is reasoned and deduced in combination with the new business environment to improve the value of data application, construct a new data model and new data correlation information, reuse knowledge, and create new knowledge, new application, and new data value.

[0075] The data processing method provided by the embodiments of the present application introduces the LightGBM algorithm to calculate the information gain of the feature attribute, sort the importance of the feature attribute, and extract the feature attribute; introduces the KNN algorithm to automatically classify the data according to the feature attribute and the needs of the operator big data application; mines the correlation information between the data, constructs a big data knowledge graph of an individual, establishes a classified data model of product application and a big data knowledge graph of product application; based on the classified data model of product application and the big data knowledge graph of product application, combines the new business environment to reason and deduce the knowledge graph, realizes knowledge reuse and data value-added. The embodiments of the present application realize effective classified storage and full use of data resources through automatic classification and value-added application of data, can reuse knowledge and add value to data in a new application environment, effectively reduces the labor cost and machine cost, solves the problems of low efficiency and high cost of data use and maintenance, and high data redundancy of operators, and improves the value of data use.

[0076] In one embodiment, the determining of the characteristic attribute of the to-be-processed data comprises: constructing a training set from the to-be-processed data, performing feature value discretization processing on the to-be-processed data in the training set, performing feature value difference processing on the to-be-processed data after the feature value discretization processing, performing feature value dimension reduction processing on the to-be-processed data after the feature value difference processing, constructing a decision tree according to the to-be-processed data after the feature value dimension reduction processing, sampling the decision tree according to a one-sided sampling mode of a gradient, and determining the characteristic attribute of the to-be-processed data according to the decision tree after feature value merging.

[0077] The LightGBM algorithm is a sorting-based decision tree algorithm, which selects a suitable split point by calculating information gain. The greater the information gain, the more important the corresponding characteristic attribute, which should be selected. The LightGBM algorithm process is as follows:

[0078] 1) Extract the characteristic attribute, and construct a test set and a training set.

[0079] First, the characteristic attribute is extracted from the basic data obtained by cleaning and conversion, and a test set and a training set are constructed. Then, the training set is trained by using the LightGBM algorithm. For example, the basic data (i.e. to-be-processed data) such as user basic information, call behavior, online behavior, location information, payment information and mobile phone usage information is extracted for feature attribute extraction to obtain a feature attribute set {A1, A2,..., An}. Then, the feature attribute set is divided into a test set and a training set according to a set proportion (such as 3:7 or 4:6).

[0080] 2) Feature value discretization processing.

[0081] Each continuous or accurate feature value in the training set is divided into a series of discrete values (also known as a histogram algorithm). Based on this, it is not necessary to sort according to each feature value or to compare all different feature values one by one, which reduces the amount of calculation and saves time. The specific process is as follows:

[0082] The continuous floating point feature value is discretized into k integers, and a histogram with a width of k is constructed to count the information. In this way, when traversing the data, only the discretized feature values (k integer values) need to be traversed to find the best split point.

[0083] The above method does not require additional storage space, has less memory occupation, and the calculation cost is reduced from 0 (feature value domain * feature number) to 0 (k * feature number). The feature value domain is much larger than k. For example, the communication duration range is 0-20000 minutes, and k=256. Therefore, the running time is one hundredth of the original.

[0084] 3) Feature value difference processing.

[0085] After the eigenvalue discretization processing, the algorithm processing speed is further improved by difference processing. Specifically, in the binary tree, the eigenvalue of a leaf node can be directly obtained by the difference between the eigenvalue of the parent node and the eigenvalue of the sibling node. Based on this, in the process of constructing the binary tree, the small leaf nodes of the histogram can be calculated first, and then the large sibling nodes of the histogram are obtained by using the histogram difference, so as to reduce the calculation cost and improve the algorithm processing efficiency.

[0086] 4) Eigenvalue dimensionality reduction processing.

[0087] Since the data volume of the operator is very large, the corresponding feature data belongs to high-dimensional data, which is very sparse in storage, consuming space and time. Therefore, the eigenvalue fusion binding is needed to reduce the number of eigenvalues, and the time complexity of constructing the histogram is changed from 0 (eigenvalue domain * feature number) to 0 (eigenvalue domain * binding number).

[0088] 5) leaf-wise method for constructing decision tree with depth limit.

[0089] After the eigenvalue dimensionality reduction processing, the decision tree learner is constructed. When traversing the data, since the split gain of some leaves is low, search and split are not needed. At this time, the leaf with the largest split gain is found from all the current leaves, and then split is performed (leaf-wise method). Such a cycle can reduce more errors and improve accuracy under the same number of splits, as shown in Figure 2 .

[0090] It should be noted that when the sample size is small, this processing method may cause overfitting. Therefore, in specific processing, the max_depth parameter is used to limit the depth of the tree to avoid overfitting.

[0091] 6) One-sided sampling method based on gradient.

[0092] After the decision tree is constructed, the corresponding information gain is calculated. Since the sample points with large gradients play a major role in the calculation of information gain, the sample points with large gradients will contribute more information gain. Therefore, when sampling, the sample points with large gradients are retained, and the sample points with small gradients are randomly sampled in proportion. The specific processing is as follows:

[0093] a. Input training data, iteration step d, sampling rate a of large gradient data, sampling rate b of small gradient data, loss function and learner (generally decision tree);

[0094] b. Sort the sample data in descending order according to the absolute value of the gradient of the sample point;

[0095] c. Select the top a*100% of the sorted results to generate a subset of large gradient sample points;

[0096] d. For the remaining sample set (1-a)*100% of the samples, randomly select b*(1-a)*100% of the sample points to generate a set of small gradient sample points;

[0097] e. Select small gradient samples, multiply them by weight coefficients, and then merge them with large gradient samples;

[0098] f. Generate a new weak learner, learn based on the new weak learner, and determine whether it converges. If it converges, end the process and output the trained strong learner. If it does not converge, determine whether the number of iterations exceeds the set number. If it does, end the process and output the trained strong learner. If it does not exceed the set number, repeat step b.

[0099] The algorithm described above can significantly reduce the model learning time without changing the data distribution or sacrificing the learner's accuracy.

[0100] 7) Parallel learning and merging of feature values.

[0101] Through the above six steps, a single decision tree learner has been built. Due to the large amount of operator data and the many features, multiple decision tree learners are needed to learn in parallel and merge relevant feature values.

[0102] It should be noted that feature parallelization learning is suitable for scenarios with small amounts of data but many features, while data parallelization learning is suitable for scenarios with large amounts of data but few features. Therefore, considering the characteristics of operator big data, this application's embodiment adopts voting parallelization learning.

[0103] Specifically, the optimal local feature is found through local voting, and then the globally advantageous feature is selected through global voting. When merging histograms, only the selected features are merged, thereby reducing communication and improving performance.

[0104] For example, such as Figure 3 As shown, taking location-related communication user behavior as an example, the LightGBM algorithm described above can extract features of location-related communication user behavior and score and rank these features.

[0105] The embodiments of the present application introduce the LightGBM algorithm to sort and select the feature attributes of the operator big data, solve the problems of occupying much storage, slow algorithm speed and much manual intervention in feature importance sorting and feature selection processing in the data classification process, improve the data classification efficiency, and save the storage space.

[0106] In one embodiment, the feature value difference processing is performed on the processed data, and the feature value dimension reduction processing is performed, including: constructing a weight graph according to the feature value difference processing of the processed data; and performing descending order sorting on the feature values in the weight graph to fuse and bind the feature values in the weight graph according to the sorting result. Specifically, the process of fusing and binding the feature values is as follows: (input: feature set F, maximum conflict number K; output: feature binding set bundles;)

[0107] (1) Construct a graph with weights (i.e., a weight graph), and the weights correspond to the total conflicts between features;

[0108] (2) Perform descending order sorting on the feature values by the degrees of the features in the weight graph;

[0109] (3) Check each feature in the ordered list, and assign it to an existing feature binding bundling with small conflicts, or create a new feature binding bundling.

[0110] Repeat the above steps until the final feature binding set bundles is formed.

[0111] The embodiments of the present application fuse and bind the feature values, save the storage space and time, and improve the algorithm efficiency.

[0112] In one embodiment, the classification of the processed data according to the feature attributes includes: determining the feature vectors of each of the processed data according to the feature attributes; determining the cosine similarity of each of the feature vectors and a target vector, and determining a neighbor sample set according to each of the cosine similarities; determining the feature categories to which the K feature values with the largest cosine similarities belong according to the neighbor sample set; and classifying the processed data according to the frequency of occurrence of each of the feature categories.

[0113] Through the KNN algorithm, the data is automatically classified, and the specific steps are as follows:

[0114] 1) Data preparation: normalize the feature data obtained based on the LightGBM algorithm, and arrange the feature data into feature vectors, for example, taking cell call data as an example, the standardized feature vector obtained is as shown in Table 3:

[0115] Table 3

[0116]

[0117] 2) Calculate distance: calculate the cosine similarity of the feature vector of the data to be classified and each vector in the library of class feature vectors, wherein the sample to be classified is r={r1,...r N}, each sample in the data set with a class is {r k1 ,...,r ki ,...,r km}, and the cosine similarity of the sample to be classified and each sample in the data set is defined as:

[0118]

[0119] 3) Find neighbors: sort according to the size of the cosine similarity to form a neighbor sample set, select the top k cosine similarity maximum feature values corresponding to the classification, and count the frequency of each belonging classification, for example, taking the cell call data as an example, P 101 working population=78%; P 102 working population=27%; P 101 residential population=22%; P 102 residential population=73%;

[0120] 4) Decision classification: select the highest frequency class as the class of the test data;

[0121] Repeat the above steps until all data is classified.

[0122] For example, taking the cell call data as an example, as shown in Figure 4 , users with more night call duration will be classified as residential users, and home broadband products can be recommended for such users; users with more daytime call duration will be classified as business users, and commercial communication products can be recommended for such users.

[0123] The embodiments of the application normalize the feature data obtained by the LightGBM algorithm through the KNN algorithm, and arrange the feature data into a feature vector. The cosine similarity of the feature vector of the data to be classified and each vector in the library of class feature vectors is calculated. The neighbor sample set is formed by sorting according to the size of the cosine similarity. The top k cosine similarity maximum feature values corresponding to the classification are selected, and the frequency of each belonging classification is counted. The highest frequency class is selected as the class of the test data. Based on this, automatic classification of data is realized, and the classification efficiency of data is improved.

[0124] In an embodiment, the determining the classification data model according to the classification result and the first knowledge graph comprises: determining, according to the classification result and the first knowledge graph, associated information of each type of the to-be-processed data; and determining the classification data model according to the associated information of each type of the to-be-processed data.

[0125] Specifically, based on the data classified automatically, a knowledge graph is constructed, a mapping between the knowledge graph and the field classification is established, knowledge association is formed, and meanwhile, associated information between data is mined, and a classification data model at a product application level is established, as follows:

[0126] The individual big data knowledge graph (i.e., the first knowledge graph): nodes of the knowledge graph are obtained from hierarchical relationships of metadata, and the individual operator big data knowledge graph is constructed and displayed in a hierarchical layout. For example, taking location information as an example, a concept corresponding to a location can be a scenic spot, an office area, a transportation hub, or a residence; an entity of the office area can be Global Mansion, an entity of the residence can be a Kingkey Village, and an entity of the user can be Wang Ming; the entity corresponds to its corresponding attribute characteristics, such as entity attribute characteristics of the residence including the number of households, the number of buildings, the number of mobile users, and the number of mobile broadband, etc.; and entity attribute characteristics of the user include the date of birth, the phone number, the gender, and the age, etc.

[0127] The classification data model at the product application level: based on the data classified automatically, combined with the individual big data knowledge graph, the associated information between data is mined through classification and knowledge fusion of concepts, entities, and attributes, and a classification data model at a product application level is established, wherein the classification data model includes a plurality of data models, as shown in Table 4:

[0128] Table 4

[0129]

[0130] The embodiments of the present application mine the associated information between data, establish a classification data model at a product application level, and based on this, improve the visualization between data.

[0131] In an embodiment, the constructing the second knowledge graph according to the classification data model comprises: determining associated information between a data model and a data capability in the classification data model; and constructing the second knowledge graph according to the associated information between the data model and the data capability.

[0132] On the basis of the classified data model, a big data knowledge graph for product application is constructed according to the association information of the data model and the data capability. For example, the data of the location information category can construct corresponding tourism scenic spot analysis models, traffic application analysis models, and job and residence commuting analysis models, and in combination with the data capability, corresponding scenic spot recommendation, traffic travel recommendation, and family broadband product recommendation are performed on the customer, and the value and capability of the data are intuitively presented in the manner of the knowledge graph.

[0133] The embodiment of the application constructs a big data knowledge graph for product application on the basis of the classified data model according to the association information of the data model and the data capability, and based on this, the value and capability of the data can be intuitively presented, and the visualization between data is improved.

[0134] In one embodiment, the updating of the data model and the data association information according to the classification result, the classified data model, and the second knowledge graph comprises: determining the association information between entities and concepts in the big data application layer according to the classification result, the classified data model, and the second knowledge graph; and updating the data model and the data association information according to the association information between the entities and the concepts.

[0135] Based on the data automatic classification technology and the classified data model of product application, not only the application value of the data of the operator can be improved, but also new knowledge, new application, and new data value can be created through the reasoning and deduction of the knowledge graph.

[0136] The various entities and concepts in the big data application layer and the association information between the entities and the concepts change over time, and new business, new models, and new association information are generated. For example, the big data knowledge graph of the location information can deduce and fuse new applications such as the travel card application according to the location data of the user moving between regions, and in combination with the current visitor data, the risk area stopover warning application and the travel application can be formed. It can be understood that the entity is essentially the association information of the user and the location attribute, but new data application and new data value are generated because of the new environment.

[0137] The embodiment of the application is based on the data automatic classification result, the classified data model of product application, and the big data knowledge graph of product application, improves the data application value, combines the new business environment to reason and deduce the knowledge graph, constructs new data models and new data association information, reuses knowledge, creates new knowledge, new application, and new data value, and improves the data use value.

[0138] The data processing apparatus provided by the embodiment of the application is described below, and the data processing apparatus described below can be correspondingly referred to the data processing method described above.

[0139] Reference Figure 5 , Figure 5 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application. The data processing apparatus provided by the embodiment of the present application comprises a feature attribute determination module 501, a classification module 502, a knowledge graph construction module 503 and an updating module 504.

[0140] The feature attribute determination module 501 is configured to determine the feature attribute of the to-be-processed data.

[0141] The classification module 502 is configured to classify the to-be-processed data according to the feature attribute.

[0142] The knowledge graph construction module 503 is configured to determine a classification data model according to a classification result and a first knowledge graph, so as to construct a second knowledge graph according to the classification data model. The first knowledge graph is determined based on individual big data, and the second knowledge graph is a knowledge graph of product application.

[0143] The updating module 504 is configured to update a data model and data correlation information according to the classification result, the classification data model and the second knowledge graph.

[0144] The data processing apparatus provided by the embodiment of the present application introduces the LightGBM algorithm to calculate the information gain of the feature attribute, to sort the importance of the feature attribute and to extract the feature attribute. The KNN algorithm is introduced to automatically classify the data according to the feature attribute and the needs of the operator big data application. The correlation information between the data is mined to construct the individual big data knowledge graph, to establish the classification data model of product application and the big data knowledge graph of product application. Based on the classification data model of product application and the big data knowledge graph of product application, the knowledge graph is reasoned and deduced in combination with the new business environment, to realize knowledge reuse and data value-added. Through the automatic classification of data and the value-added application of data, the embodiment of the present application realizes effective classification storage and full use of data resources, and can realize knowledge reuse and data value-added in a new application environment. The embodiment of the present application effectively reduces the labor cost and the machine cost, solves the problems of low efficiency and high cost in the use and maintenance of operator data, high data redundancy and the like, and improves the data use value.

[0145] In one embodiment, the feature attribute determination module 501 is specifically configured to:

[0146] construct a training set from the to-be-processed data, to perform feature value discretization processing on the to-be-processed data in the training set;

[0147] perform feature value difference processing on the to-be-processed data after the feature value discretization processing, to perform feature value dimension reduction processing on the to-be-processed data after the feature value difference processing.

[0148] constructing a decision tree according to the processed data after the eigenvalue dimension reduction, and sampling the decision tree according to a one-side sampling manner of the gradient;

[0149] merging the multiple decision trees obtained by sampling according to eigenvalues, to determine the feature attribute of the processed data according to the decision tree after the eigenvalue merging.

[0150] In one embodiment, the feature attribute determination module 501 is specifically configured to:

[0151] constructing a weight graph according to the processed data after the eigenvalue difference processing;

[0152] sorting the eigenvalues in the weight graph in descending order, and fusing and binding the eigenvalues in the weight graph according to the sorting result.

[0153] In one embodiment, the classification module 502 is specifically configured to:

[0154] determining the feature vector of each of the processed data according to the feature attribute;

[0155] determining the cosine similarity of each of the feature vectors and a target vector, and determining a neighbor sample set according to each of the cosine similarities;

[0156] determining the feature categories to which the eigenvalues with the maximum cosine similarity respectively belong according to the neighbor sample set;

[0157] classifying the processed data according to the frequency of occurrence of each of the feature categories.

[0158] In one embodiment, the knowledge graph construction module 503 is specifically configured to:

[0159] determining the associated information of each of the processed data according to the classification result and the first knowledge graph;

[0160] determining the classification data model according to the associated information of each of the processed data.

[0161] In one embodiment, the knowledge graph construction module 503 is specifically configured to:

[0162] determining the associated information of a data model and a data capability in the classification data model; the classification data model includes multiple data models;

[0163] constructing the second knowledge graph according to the associated information of the data model and the data capability.

[0164] In one embodiment, the updating module 504 is specifically configured to:

[0165] According to the classification result, the classification data model and the second knowledge graph, determine the association information between entities and concepts in the big data application layer;

[0166] According to the association information between the entity and the concept, update the data model and the data association information.

[0167] Figure 6 An entity structure diagram of an electronic device is shown, as Figure 6 The electronic device can include a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the computer program in the memory 630 to execute the steps of the data processing method, for example, including:

[0168] Determine the characteristic attribute of the data to be processed;

[0169] Classify the data to be processed according to the characteristic attribute;

[0170] Determine a classification data model according to the classification result and a first knowledge graph, to construct a second knowledge graph according to the classification data model; the first knowledge graph is determined based on individual big data, and the second knowledge graph is a knowledge graph of product application;

[0171] According to the classification result, the classification data model and the second knowledge graph, update the data model and the data association information.

[0172] In addition, the logical instructions in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various program code storage media.

[0173] In another aspect, the embodiments of the present application also provide a computer program product, which comprises a computer program, the computer program can be stored in a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the steps of the data processing method provided by the above embodiments, for example, comprising:

[0174] determining a characteristic attribute of the to-be-processed data;

[0175] classifying the to-be-processed data according to the characteristic attribute;

[0176] determining a classification data model according to the classification result and a first knowledge graph, so as to construct a second knowledge graph according to the classification data model; the first knowledge graph is determined based on big data of individuals, and the second knowledge graph is a knowledge graph of product application;

[0177] updating the data model and the data correlation information according to the classification result, the classification data model and the second knowledge graph.

[0178] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0179] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some parts of the embodiment.

[0180] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: Collecting data to be processed from an operator network system, wherein the data to be processed comprises user communication behavior information, user location information and payment behavior data; Determining feature attributes of the data to be processed; Classifying the data to be processed according to the feature attributes; Determining a classification data model according to the classification result and a first knowledge graph, so as to construct a second knowledge graph according to the classification data model; The first knowledge graph is determined based on individual big data, and the second knowledge graph is a product application knowledge graph; the classification data model of product application comprises a tourism scenic spot analysis model, a population statistical insight analysis model, a vertical industry APP use analysis model, a job and residence commuting analysis model, a grid risk portrait analysis model and a regional population real-time situation awareness model; Updating the data model and data correlation information according to the classification result, the classification data model and the second knowledge graph; The method comprises the following steps: Constructing a training set from the data to be processed, so as to perform feature value discretization processing on the data to be processed in the training set; Performing feature value difference processing on the data to be processed after feature value discretization processing, so as to perform feature value dimension reduction processing on the data to be processed after feature value difference processing; Constructing a decision tree according to the data to be processed after feature value dimension reduction processing, so as to sample the decision tree according to a one-sided sampling mode of gradient; Merging feature values of a plurality of decision trees obtained by sampling, so as to determine the feature attributes of the data to be processed according to the decision tree after feature value merging.

2. The data processing method of claim 1, wherein, The method comprises the following steps: Constructing a weight graph according to the data to be processed after feature value difference processing; Performing descending order sorting on feature values in the weight graph, so as to perform fusion binding on the feature values in the weight graph according to the sorting result.

3. The data processing method of claim 1, wherein, The method comprises the following steps: Determining a feature vector of each data to be processed according to the feature attributes; Determining a cosine similarity between each feature vector and a target vector, and determining a neighbor sample set according to each cosine similarity; Determining feature categories to which K feature values with the largest cosine similarities respectively belong according to the neighbor sample set; Classifying the data to be processed according to the frequency of occurrence of each feature category.

4. The data processing method of claim 1, wherein, The method comprises the following steps: Determining correlation information of each type of data to be processed according to the classification result and the first knowledge graph; Determining the classification data model according to the correlation information of each type of data to be processed.

5. The data processing method of claim 1, wherein, The method comprises the following steps: Determining correlation information between a data model and a data capability in the classification data model; the classification data model comprises a plurality of data models; Constructing the second knowledge graph according to the correlation information between the data model and the data capability.

6. The data processing method of claim 1, wherein, The method comprises the following steps: According to the classification result, the classification data model, and the second knowledge graph, determine the association information between entities and concepts in a big data application layer; According to the association information between the entities and the concepts, update the data model and the data association information.

7. A data processing apparatus, characterized by, Comprise: A collection module configured to collect to-be-processed data from an operator network system, the to-be-processed data including user communication behavior information, user location information, and payment behavior data; A feature attribute determination module configured to determine feature attributes of the to-be-processed data; A classification module configured to classify the to-be-processed data according to the feature attributes; A knowledge graph construction module configured to determine a classification data model according to a classification result and a first knowledge graph, and to construct a second knowledge graph according to the classification data model; the first knowledge graph is determined based on individual big data, and the second knowledge graph is a product application knowledge graph; the classification data model of the product application includes a tourism scenic spot analysis model, a population statistics insight analysis model, a vertical industry APP usage analysis model, a job-housing commuting analysis model, a grid risk portrait analysis model, and a regional crowd real-time situation awareness model; An update module configured to update a data model and data association information according to the classification result, the classification data model, and the second knowledge graph; The feature attribute determination module is further configured to construct a training set from the to-be-processed data, to perform feature value discretization processing on the to-be-processed data in the training set, to perform feature value difference processing on the to-be-processed data after the feature value discretization processing, and to perform feature value dimension reduction processing on the to-be-processed data after the feature value difference processing. According to the to-be-processed data after the feature value dimension reduction processing, construct a decision tree, sample the decision tree according to a one-sided sampling method of a gradient, and determine feature attributes of the to-be-processed data according to the decision tree after feature value merging. The processor executes the computer program to realize the steps of the data processing method of any one of claims 1 to 6. The computer program is executed by the processor to realize the steps of the data processing method of any one of claims 1 to 6.

8. An electronic device comprising a processor and a memory having a computer program stored therein, characterized in that, ​ 9. A computer program product comprising a computer program, characterized in that, ​

Images