Intelligent data asset management and optimal distribution method and system

Abstract

The invention provides an intelligent data asset management and optimal distribution method and system, and the method comprises the steps: carrying out the coding of mobile user data, carrying out the data cleaning of the coded mobile user data, and obtaining the cleaned mobile user data; calculating a weight value of each word in the mobile user data after data cleaning; converting the mobile user data after data cleaning into word vectors, and endowing each word vector with a corresponding weight value to form a training sample; inputting the training sample into a neural network for training to obtain a data asset distribution model; and completing the distribution of the target mobile user data by using the data asset distribution model. When the training sample is constructed, in order to improve the feature extraction capability of the neural network, the weight of each word is defined in combination with the position information of the words and the TF-IDF information, so that the neural network model learns the features of the words with high weight values, and the classification process of different texts is better completed.

Description

Technical Field

[0001] This invention belongs to the field of data asset management technology, and more specifically, relates to an intelligent data asset management and optimal allocation method and system. Background Technology

[0002] With the rapid development of information technology, enterprises and organizations have accumulated a large amount of data assets. These data assets contain rich information and value, but due to improper management and allocation, their potential is often not fully realized. To address this issue, data asset management and allocation methods have emerged. However, traditional data asset management and allocation methods typically extract metadata information from the corresponding data assets. Then, enterprise managers assign corresponding tags to different types of data based on the extracted metadata information and store the corresponding data assets in a centralized data management system based on the tag type. It is evident that existing data asset management and allocation methods rely solely on enterprise managers to assign corresponding tags to different types of data, which is not only inefficient but also prone to unreasonable allocation of data resources, reducing the utilization efficiency and value of data assets. Summary of the Invention

[0003] To address the aforementioned problems, the present invention aims to provide an intelligent data asset management and optimal allocation method and system.

[0004] A method for intelligent data asset management and optimal allocation includes:

[0005] Step 1: Obtain mobile user data;

[0006] Step 2: Encode the mobile user data to obtain the encoded mobile user data;

[0007] Step 3: Perform data cleaning on the encoded mobile user data to obtain cleaned mobile user data;

[0008] Step 4: Calculate the weight value of each word in the cleaned mobile user data;

[0009] Step 5: Convert the cleaned mobile user data into word vectors, and assign corresponding weight values ​​to each word vector to form training samples;

[0010] Step 6: Input the training samples into the neural network for training to obtain the data asset allocation model;

[0011] Step 7: Use the data asset allocation model to complete the allocation of target mobile user data.

[0012] Preferably, step 2: encoding the mobile user data to obtain encoded mobile user data includes:

[0013] Mobile user data is encoded using the Unicode encoding method to obtain encoded mobile user data; each piece of data in the encoded mobile user data has a set of m character elements.

[0014] Preferably, step 3: performing data cleaning on the encoded mobile user data to obtain cleaned mobile user data includes:

[0015] Step 3.1: Calculate the feature value of each data point in the encoded mobile user data;

[0016] Step 3.2: Normalize the feature values ​​of each data point to obtain normalized data feature values;

[0017] Step 3.3: Remove the mobile user data corresponding to the data feature values ​​located in the outlier range as outliers to obtain the cleaned mobile user data.

[0018] Preferably, step 3.1: calculating the feature value of each data item in the encoded mobile user data includes:

[0019] Formula used:

[0020]

[0021] Calculate the feature value of each data point in the encoded mobile user data; where H(X) represents the feature value of the Xth data point, p i This represents the probability that the i-th character element appears in the corresponding set.

[0022] Preferably, step 4: calculating the weight value of each word in the cleaned mobile user data, includes:

[0023] Step 4.1: Quantify the location information of each word in the cleaned mobile user data to obtain the location information statistical formula;

[0024] Step 4.2: Obtain the TF-IDF value of each word in the cleaned mobile user data;

[0025] Step 4.3: Based on the principle of normal distribution, use TF-IDF values ​​to optimize the location information statistical formula to obtain the weight value of each word.

[0026] Preferably, the location information statistical formula is:

[0027]

[0028] Where λ represents the adjustable parameter, f iIndicates the position of the first occurrence of word i in the corresponding text, n j First(i,j) represents the total number of words in text j, and First(i,j) represents the position information of word i.

[0029] Preferably, step 4.3: Optimizing the location information statistical formula using TF-IDF values ​​based on the principle of normal distribution to obtain the weight value of each word, including:

[0030] Formula used:

[0031]

[0032] LTFIDF(i,j)=TF-IDF(i,j)×Local(i,j)

[0033] The weight value of each word is obtained; where LTFIDF(i,j) represents the weight value of the i-th word in text j, and TF-IDF(i,j) represents the TF-IDF value of word i.

[0034] Preferably, step 5: converting the cleaned mobile user data into word vectors, and assigning a corresponding weight value to each word vector to form training samples, includes:

[0035] The Word2vec model is used to convert each word in the cleaned mobile user data into a word vector. Then use the formula The corresponding word vectors The training samples are formed by combining the weight values ​​of the words.

[0036] This invention also provides an intelligent data asset management and optimal allocation system, comprising:

[0037] The mobile user data acquisition module is used to acquire mobile user data;

[0038] The encoding module is used to encode mobile user data to obtain encoded mobile user data;

[0039] The data cleaning module is used to clean the encoded mobile user data to obtain cleaned mobile user data.

[0040] The weight value calculation module is used to calculate the weight value of each word in the cleaned mobile user data;

[0041] The training sample construction module is used to convert the cleaned mobile user data into word vectors and assign corresponding weight values ​​to each word vector to form training samples;

[0042] The training module is used to input the training samples into the neural network for training to obtain a data asset allocation model;

[0043] The automatic allocation module is used to allocate target mobile user data using a data asset allocation model.

[0044] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps in the above-described intelligent data asset management and optimal allocation method.

[0045] The beneficial effects of the intelligent data asset management and optimal allocation method and system provided by this invention are as follows: Compared with the prior art, when constructing training samples, this invention combines word position information and TF-IDF information to define the weight of each word in order to improve the feature extraction capability of the neural network. This enables the neural network model to learn the features of words with high weight values, thereby better completing the classification process of different texts. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating an intelligent data asset management and optimal allocation method provided in this embodiment of the invention;

[0048] Figure 2 A diagram of a convolutional neural network structure provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of an intelligent data asset management and optimal allocation system provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0051] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0052] Please see Figure 1-2 A method for intelligent data asset management and optimal allocation includes the following steps:

[0053] Step 1: Obtain mobile user data;

[0054] Step 2: Encode the mobile user data to obtain the encoded mobile user data;

[0055] In practical applications, this invention can use the UCX data tool to collect mobile user data, and then encode the collected mobile user data into an encoded mobile user data set A using Unicode. The encoded mobile user data set A contains n data entries, and each encoded data entry has a set [x1, x2, ..., xm] consisting of m character elements. m ].

[0056] Step 3: Perform data cleaning on the encoded mobile user data to obtain cleaned mobile user data;

[0057] Furthermore, step 3 includes:

[0058] Step 3.1: Calculate the feature value of each data point in the encoded mobile user data;

[0059] Step 3.2: Normalize the feature values ​​of each data point to obtain normalized data feature values;

[0060] Step 3.3: Remove the mobile user data corresponding to the data feature values ​​located in the outlier range as outliers to obtain the cleaned mobile user data.

[0061] In practical applications, mobile user data may contain various problems, such as missing data, errors, and outliers. Data cleaning can help identify and correct these problems, improve data accuracy and consistency, and thus improve data quality.

[0062] The encoded mobile user data set A contains n data entries, and each encoded data entry has a set [x1, x2, ..., xm] consisting of m character elements. m If the value is ], then the corresponding feature (entropy) value is:

[0063]

[0064] Where, p i This represents the probability that the i-th character element appears in the corresponding character element set;

[0065] Each piece of encoded data has a corresponding feature value, based on which a mobile user data feature matrix can be constructed:

[0066]

[0067] Where, x nmThis represents the m-th character element in the n-th data item.

[0068] After calculating the feature value of each data point in the mobile user data feature matrix, a normalization formula is also needed:

[0069]

[0070] Normalize the feature values ​​of each data point to obtain normalized data feature values; where x′ pi x represents the normalized feature values ​​of the data. pi Let X be the i-th original data feature value, min{X} be the minimum value in the set of data feature values, and max{X} be the maximum value in the set of data feature values.

[0071] After obtaining the normalized data feature values, the corresponding outlier range can be determined based on the feature values ​​of the abnormal data, and the data corresponding to the feature values ​​of the data located in the outlier range can be removed as outliers.

[0072] This invention helps identify and correct anomalies in mobile user data by cleaning it, improving data integrity and consistency, facilitating the understanding of user needs and behaviors, thereby optimizing products and services and enhancing user experience.

[0073] Step 4: Calculate the weight value of each word in the cleaned mobile user data;

[0074] Furthermore, step 4 includes:

[0075] Step 4.1: Quantify the location information of each word in the cleaned mobile user data to obtain the location information statistical formula;

[0076] Step 4.2: Obtain the TF-IDF value of each word in the cleaned mobile user data;

[0077] Step 4.3: Based on the principle of normal distribution, the TF-IDF value is used to optimize the location information statistical formula to obtain the weight value of each word;

[0078] Step 5: Convert the cleaned mobile user data into word vectors, and assign corresponding weight values ​​to each word vector to form training samples;

[0079] The present invention will further explain steps 4-5 below with reference to specific embodiments:

[0080] First, a portion of the cleaned mobile user data is manually assigned category labels to form the initial training samples.

[0081] Typically, keywords best reflect the meaning or category of text, and they usually appear at the beginning of the text. Therefore, keyword position information is also an important feature. Based on this, this application uses the principle of normal distribution to calculate the weight value of each word and uses the weight value to quantify the keywords. The specific process is as follows:

[0082] (1) Statistical analysis of word position information:

[0083]

[0084] Where λ represents the adjustable parameter, f i Indicates the position of the first occurrence of word i in the corresponding text, n j This represents the total number of words in text j.

[0085] (2) Calculate the TF-IDF value of word i:

[0086] TF-IDF(i,j) = TF*IDF

[0087] In the formula, TF represents term frequency and IDF represents inverse document frequency.

[0088] The weight values ​​of words are obtained by optimizing the above formula based on the principle of normal distribution:

[0089]

[0090] LTFIDF(i,j)=TF-IDF(i,j)×Local(i,j)

[0091] LTFIDF(i,j) represents the weight value of the i-th word in text j, and TF-IDF(i,j) represents the TF-IDF value of word i.

[0092] (3) Use the Word2vec model to convert each word in the cleaned mobile user data into a word vector. Then use the formula The corresponding word vectors The input samples for the convolutional neural network are formed by combining the weight values ​​of the words.

[0093] Step 6: Input the training samples into the neural network for training to obtain the data asset allocation model;

[0094] (1) The sample is input into the convolutional layer, convolved, and then substituted into the activation function. In the convolutional layer, the activation function maps the input signal to a new output space by performing a nonlinear transformation on the neuron's output, introducing nonlinear characteristics in the process, enabling the neural network to learn nonlinear relationships. Therefore, the activation function can help the neural network learn more complex features, improving the neural network's expressive power and performance. The activation function chosen in this invention is the Tanh function.

[0095] (2) The results after convolution are input into the pooling layer for downsampling to reduce the data size. Finally, the pooled data is input into the fully connected layer to output the classification results.

[0096] To train the above convolutional neural network model, this invention uses cross-entropy as the loss function, as shown in the following equation:

[0097]

[0098] In the formula, y j ' represents the category value of the j-th text predicted by the neural network model, y j Let represent the true category value of the j-th text, and n represent the total number of categories of text.

[0099] (3) By continuously updating the network parameters of the convolutional neural network model through the backpropagation algorithm to minimize the loss function, the trained data asset allocation model can be obtained. Using this trained data asset allocation model, the categories of subsequently collected mobile user data can be automatically classified.

[0100] Step 7: Use the data asset allocation model to complete the allocation of target mobile user data.

[0101] In order to improve the feature extraction capability of the convolutional neural network model during training, this invention combines word position information and TF-IDF information to define the weight of each word. This enables the convolutional neural network model to learn the features of words (keywords) with high weight values, thereby better completing the classification process of different texts.

[0102] Please see Figure 3 The present invention also provides an intelligent data asset management and optimal allocation system, comprising:

[0103] The mobile user data acquisition module is used to acquire mobile user data;

[0104] The encoding module is used to encode mobile user data to obtain encoded mobile user data;

[0105] The data cleaning module is used to clean the encoded mobile user data to obtain cleaned mobile user data.

[0106] The weight value calculation module is used to calculate the weight value of each word in the cleaned mobile user data;

[0107] The training sample construction module is used to convert the cleaned mobile user data into word vectors and assign corresponding weight values ​​to each word vector to form training samples;

[0108] The training module is used to input the training samples into the neural network for training to obtain a data asset allocation model;

[0109] The automatic allocation module is used to allocate target mobile user data using a data asset allocation model.

[0110] Compared with the prior art, the beneficial effects of the intelligent data asset management and optimal allocation system provided by the present invention are the same as the beneficial effects of the intelligent data asset management and optimal allocation method described in the above technical solution, and will not be repeated here.

[0111] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps in the above-described intelligent data asset management and optimal allocation method.

[0112] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the intelligent data asset management and optimal allocation method described in the above technical solution, and will not be repeated here.

[0113] This invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are connected via the bus. When the computer program is executed by the processor, it implements the steps in the aforementioned intelligent data asset management and optimal allocation method. Compared with the prior art, the beneficial effects of the electronic device provided by this invention are the same as those of the intelligent data asset management and optimal allocation method described above, and will not be repeated here.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent data asset management and optimal allocation, characterized in that, include: Step 1: Obtain mobile user data; Step 2: Encode the mobile user data to obtain the encoded mobile user data; Step 3: Perform data cleaning on the encoded mobile user data to obtain cleaned mobile user data; Step 4: Calculate the weight value of each word in the cleaned mobile user data; Step 5: Convert the cleaned mobile user data into word vectors, and assign corresponding weight values ​​to each word vector to form training samples; Step 6: Input the training samples into the neural network for training to obtain the data asset allocation model; Step 7: Use the data asset allocation model to complete the allocation of target mobile user data.

2. The intelligent data asset management and optimal allocation method as described in claim 1, characterized in that, Step 2: Encoding the mobile user data to obtain encoded mobile user data, including: Mobile user data is encoded using the Unicode encoding method to obtain encoded mobile user data; each piece of data in the encoded mobile user data has a set of m character elements.

3. The intelligent data asset management and optimal allocation method as described in claim 2, characterized in that, Step 3: Perform data cleaning on the encoded mobile user data to obtain cleaned mobile user data, including: Step 3.1: Calculate the feature value of each data point in the encoded mobile user data; Step 3.2: Normalize the feature values ​​of each data point to obtain normalized data feature values; Step 3.3: Remove the mobile user data corresponding to the data feature values ​​located in the outlier range as outliers to obtain the cleaned mobile user data.

4. The intelligent data asset management and optimal allocation method as described in claim 3, characterized in that, Step 3.1: Calculate the feature value of each data item in the encoded mobile user data, including: Formula used: Calculate the feature value of each data point in the encoded mobile user data; where H(X) represents the feature value of the Xth data point, p i This represents the probability that the i-th character element appears in the corresponding set.

5. The intelligent data asset management and optimal allocation method as described in claim 4, characterized in that, Step 4: Calculate the weight value of each word in the cleaned mobile user data, including: Step 4.1: Quantify the location information of each word in the cleaned mobile user data to obtain the location information statistical formula; Step 4.2: Obtain the TF-IDF value of each word in the cleaned mobile user data; Step 4.3: Based on the principle of normal distribution, use TF-IDF values ​​to optimize the location information statistical formula to obtain the weight value of each word.

6. The intelligent data asset management and optimal allocation method as described in claim 5, characterized in that, The formula for calculating the location information is: Where λ represents the adjustable parameter, f i Indicates the position of the first occurrence of word i in the corresponding text, n j First(i,j) represents the total number of words in text j, and First(i,j) represents the position information of word i.

7. The intelligent data asset management and optimal allocation method as described in claim 6, characterized in that, Step 4.3: Based on the principle of normal distribution, the TF-IDF value is used to optimize the location information statistical formula to obtain the weight value of each word, including: Formula used: LTFIDF(i,j)=TF-IDF(i,j)×Local(i,j) The weight value of each word is obtained; where LTFIDF(i,j) represents the weight value of the i-th word in text j, and TF-IDF(i,j) represents the TF-IDF value of word i.

8. The intelligent data asset management and optimal allocation method as described in claim 7, characterized in that, Step 5: Convert the cleaned mobile user data into word vectors, and assign corresponding weight values ​​to each word vector to form training samples, including: The Word2vec model is used to convert each word in the cleaned mobile user data into a word vector. Then use the formula The corresponding word vectors The training samples are formed by combining the weight values ​​of the words.

9. A smart data asset management and optimal allocation system, characterized in that, include: The mobile user data acquisition module is used to acquire mobile user data; The encoding module is used to encode mobile user data to obtain encoded mobile user data; The data cleaning module is used to clean the encoded mobile user data to obtain cleaned mobile user data. The weight value calculation module is used to calculate the weight value of each word in the cleaned mobile user data; The training sample construction module is used to convert the cleaned mobile user data into word vectors and assign corresponding weight values ​​to each word vector to form training samples; The training module is used to input the training samples into the neural network for training to obtain a data asset allocation model; The automatic allocation module is used to complete the allocation of target mobile user data using the data asset allocation model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent data asset management and optimal allocation method as described in any one of claims 1-8.

Images