Data processing method and device, equipment, storage medium and product
By calculating the degree of correlation and principal component analysis, textual data in structured data is quantified, solving the problem that word embedding models cannot convert, achieving efficient and accurate text feature conversion, and reducing computational costs.
Patent Information
- Application Number
- CN202511727156.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-02-17
AI Technical Summary
In existing technologies, word embedding models cannot accurately convert missing words or phrases in structured data into text features, resulting in semantic information loss in machine learning and deep learning models in practical applications, and the training cost of word embedding models is high.
By calculating the correlation between textual and numerical data in structured data, the numerical data is updated, and the textual data is mapped to a vector space composed of principal components. The orthogonal mapping points of key features are used for numericalization, and a vector space is constructed to fuse textual features.
It improves the accuracy of converting structured data into text features, avoids the loss of semantic information, and reduces computational costs, making it suitable for large-scale corpus applications.
Smart Images
Figure CN121542722A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data processing technology, and in particular to a data processing method, apparatus, device, storage medium and product. Background Technology
[0002] In recent years, with the widespread application of deep learning in the field of natural language processing, how to map discrete textual information in structured data (such as two-dimensional data tables) to a computable and operable feature space has become a key technology.
[0003] In existing technologies, word embedding models are typically used to quantify words or phrases in structured data into text features, enabling machine learning and deep learning models to understand and process text. By learning on large-scale corpora, word embedding models transform words or phrases into text features, allowing machine learning and deep learning models to directly capture the semantic and syntactic relationships of words in the feature space, thus significantly improving the performance of tasks such as text classification, machine translation, and question answering systems. However, when using word embedding models to transform structured data, words or phrases that are scarce or completely absent in the training data of the word embedding models cannot be accurately converted into text features. This results in semantic information loss in the text features used by machine learning and deep learning models in practical applications. Summary of the Invention
[0004] This application provides a data processing method, apparatus, device, storage medium, and product that can accurately convert structured data into text features.
[0005] In a first aspect, embodiments of this application provide a data processing method, including: Obtain structured data, which includes at least one column of first text data and at least one column of first numerical data; Select the first target text data from the first text data that meets the preset conditions in the first target column; Calculate the first degree of correlation between the first target text data and the first numerical data in each row; Update the first numerical data using the first degree of correlation to obtain the second numerical data; Map the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data. The third numerical data includes the coordinate values of the orthogonal mapping points of the n principal components, where n is the first preset threshold. The first target text data is updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the second text data. The second textual data and the first numerical data are merged to obtain the first textual feature.
[0006] In one possible implementation, each column of first text data includes multiple sets of text fields, each set of text fields being identical; selecting first target text data of a first target column that meets preset conditions from the first text data includes: Calculate the relative entropy of the first text data in each column. The relative entropy represents the amount of information loss when the numerical distribution of one set of text fields in a set of multiple text fields is approximated by the numerical distribution of another set of text fields. From the first text data, select the first target text data of the first target column whose relative entropy meets the preset conditions.
[0007] In one possible implementation, updating the first numerical data using a first correlation level to obtain the second numerical data includes: The first numerical data is updated by multiplying the first correlation level with the first numerical data to obtain the second numerical data.
[0008] In one possible implementation, the dimension of each row of first numerical data is greater than a preset dimension; the second numerical data corresponding to each row of target text data in the first target text data is mapped to a vector space composed of n principal components to obtain third numerical data, including: Map the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of m principal components to obtain the fourth numerical data corresponding to each line of target text data in the first target text data. The fourth numerical data includes the coordinate values of the orthogonal mapping points of the m principal components, where m is the second preset threshold. From the fourth numerical data, the fifth numerical data is selected according to the preset filtering rules. The fifth numerical data includes the coordinate values of the orthogonal mapping points of the k principal components. Map the fifth numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data; Where m>k>n.
[0009] In one possible implementation, before mapping the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data, the method further includes: Receive user input for a first preset threshold and a second preset threshold.
[0010] In one possible implementation, the first target text data is updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the second text data, including: The first target text data is updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the third text data. The third text data is normalized to obtain the second text data.
[0011] In one possible implementation, after merging the second textual data and the first numerical data to obtain the first textual feature, the method further includes: When the structured data includes multiple columns of first text data, select the second target text data of the second target column that meets the preset conditions from the text data other than the first target text data in the first text data; Calculate the second degree of correlation between the second target text data and the first text feature for each row; The first text feature is updated using the second degree of correlation to obtain the sixth numerical data. Map the sixth numerical data corresponding to each line of target text data in the second target text data to a vector space composed of n principal components to obtain the seventh numerical data. The seventh numerical data includes the coordinate values of the orthogonal mapping points of the n principal components. The second target text data is updated with the coordinates of the orthogonal mapping points of the first principal component in the seventh numerical data to obtain the fourth text data. The fourth text data and the first text feature are combined to obtain the second text feature.
[0012] Secondly, embodiments of this application provide a data processing apparatus, including: The acquisition module is used to acquire structured data, which includes at least one column of first text data and at least one column of first numerical data. The selection module is used to select the first target text data that meets the preset conditions from the first text data; The calculation module is used to calculate the first correlation degree between the first target text data and the first numerical data in each row; The update module is used to update the first numerical data using the first degree of correlation to obtain the second numerical data; The mapping module is used to map the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data. The third numerical data includes the coordinate values of the orthogonal mapping points of the n principal components, where n is a first preset threshold. The update module is used to update the first target text data with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data, so as to obtain the second text data. The merging module is used to merge the second text data and the first numerical data to obtain the first text feature.
[0013] Thirdly, embodiments of this application provide an electronic device, the device comprising: A processor and a memory storing computer program instructions; a method for processing data that implements any of the above when the processor executes the computer program instructions.
[0014] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, a data processing method that implements any of the above-mentioned methods is provided.
[0015] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform any of the above-mentioned data processing methods.
[0016] The data processing method, apparatus, device, storage medium, and product of this application embodiment acquire structured data, the structured data including at least one column of first text data and at least one column of first numerical data; select first target text data of a first target column that meets preset conditions from the first text data; calculate the first correlation degree of each row of first target text data and first numerical data; update the first numerical data using the first correlation degree to obtain second numerical data; map the second numerical data corresponding to each row of target text data in the first target text data to a vector space composed of n principal components to obtain third numerical data, the third numerical data including the coordinate values of the orthogonal mapping points of the n principal components, where n is a first preset threshold; update the first target text data with the coordinate values of the orthogonal mapping points of the first principal components in the third numerical data to obtain second text data; merge the second text data and the first numerical data to obtain a first text feature.
[0017] This application first selects the first target text data in the first target column and calculates its first correlation degree with the first numerical data. The first numerical data is then updated using this first correlation degree to obtain the second numerical data. The text data is then integrated into the original numerical data based on the correlation degree. A vector space is then constructed, which itself has strong interpretability for the text data. In the vector space composed of n principal components, the coordinates of the first principal component (i.e., the orthogonal mapping points of the key features) are selected. The first target text data is then updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the second text data. This second text data is numerically converted data. In other words, by using the numerical data inherent in the structured data as a base point, the text data is numerically converted, retaining key features and not relying on the model's training data. This improves the accuracy of converting structured data into text features, thereby avoiding the loss of semantic information in text features used by machine learning and deep learning models in practical applications. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic flowchart of a data processing method provided in one embodiment of this application; Figure 2 This is a flowchart illustrating a data processing method provided in another embodiment of this application; Figure 3 This is a flowchart illustrating a data processing method provided in another embodiment of this application; Figure 4 This is a schematic flowchart of a data processing method provided in another embodiment of this application; Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0022] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.
[0023] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.
[0024] In recent years, with the widespread application of deep learning in the field of natural language processing, a key technology has emerged: how to map discrete textual information in structured data (such as two-dimensional data tables) to a computable and manipulable feature space. The quantification of text data is a crucial step in Natural Language Processing (NLP), transforming text into a form that computers can understand and process. Quantification enables machine learning models to process text and plays a vital role in tasks such as sentiment analysis, text classification, information retrieval, and machine translation.
[0025] In existing technologies, word embedding models are typically used to quantify words or phrases in structured data into text features, enabling machine learning and deep learning models to understand and process text. By learning on large-scale corpora, word embedding models transform words or phrases into text features, allowing machine learning models to directly capture the semantic and syntactic relationships of words in the feature space, thus significantly improving the performance of tasks such as text classification, machine translation, and question answering systems. However, when using word embedding models to transform structured data, if words or phrases are scarce or completely absent in the training data, the model's performance degrades, failing to accurately convert them into text features. This results in semantically missing text features used by machine learning and deep learning models in practical applications.
[0026] Moreover, using word embedding models requires storing a vocabulary of relationships between words or phrases and text features. For very large vocabularies, computing and storing word embedding matrices can require a lot of resources, especially when training on large-scale corpora, which is costly.
[0027] To address the problems of existing technologies, embodiments of this application provide a data processing method, apparatus, device, storage medium, and product. The data processing method provided in this application first selects a first target textual data from a first target column and calculates its first correlation degree with first numerical data. The first numerical data is then updated using the first correlation degree to obtain second numerical data. The textual data is then fused into the original numerical data based on the correlation degree. A vector space is then constructed, which itself has strong interpretability for the textual data. In the vector space composed of n principal components, the coordinates of the first principal component (i.e., the orthogonal mapping points of key features) are selected. The first target textual data is updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain second textual data. This second textual data is numerically converted data. That is, by using the numerical data inherent in the structured data as a base point, the textual data is numerically converted, retaining key features and not relying on model training data. This improves the accuracy of converting structured data into textual features, thereby avoiding the loss of semantic information in textual features used by machine learning and deep learning models in practical applications.
[0028] The data processing method provided in the embodiments of this application will be described in detail below.
[0029] like Figure 1 As shown, the data processing method provided in this application embodiment includes the following steps S110 to S170.
[0030] S110. Obtain structured data, which includes at least one column of first text data and at least one column of first numerical data.
[0031] In some embodiments, structured data in the form of a two-dimensional table is obtained, the structured data including at least one column of text data and at least one column of numerical data.
[0032] In one example, a portion of the structured data is shown in Table 1. All columns except the "Name" column are used for machine learning and deep learning model processing. Therefore, the text in multiple data entries from all columns except the "Name" column can be converted into numerical values. It is understood that each row corresponding to "Name" represents one data entry.
[0033] Table 1 For example, a single data entry corresponding to "Name 1" can be represented as a = (b, c). Structured data can then be represented as A = (B, C). Here, B represents text data consisting of multiple b's, and C represents numerical data consisting of c's.
[0034] In some embodiments, initial structured data is obtained, and the numerical data in the initial structured data is normalized to eliminate the influence of magnitude, thereby obtaining structured data.
[0035] S120. Select the first target text data that meets the preset conditions from the first text data.
[0036] Among them, the preset conditions can be conditions set in advance according to requirements.
[0037] In some embodiments, the preset condition is random data selection. Selecting first target text data from the first text data that satisfies the preset condition for a first target column includes: Randomly select the first target text data from the first target column of the first text data.
[0038] In some embodiments, the preset condition is to select the data with the highest importance. Selecting first target text data from the first text data that meets the preset condition for a first target column includes: Obtain the importance level of the first text data in each column; from the first text data, select the first target text data of the first target column with the highest importance.
[0039] In some embodiments, the preset condition is to select the data with the highest relative entropy. Selecting first target text data from the first text data that satisfies the preset condition for a first target column includes: Calculate the relative entropy of the first text data in each column; from the first text data, select the first target text data of the first target column with the largest relative entropy.
[0040] In one example, from the first text data shown in Table 1, select the first target text data in the "Education Level" column that meets the preset conditions.
[0041] S130. Calculate the first correlation degree between the first target text data and the first numerical data in each row.
[0042] In some embodiments, the correlation coefficients of each row of first target text data and the corresponding first numerical data are calculated to obtain the first degree of association between each row of first target text data and the first numerical data.
[0043] In one example, the first degree of association between “primary school” in the first row and “178”, “75”, “4500” and “47” in the first row is calculated, as shown in Table 1.
[0044] In another example, the first degree of association between “junior high school” in the second row and “165”, “61”, “6000” and “52” in the second row is calculated, as shown in Table 1.
[0045] It is understandable that each first-valued data point has a corresponding first-degree of correlation.
[0046] S140. Update the first numerical data using the first correlation degree to obtain the second numerical data.
[0047] In one embodiment, each first numerical data point has a corresponding first correlation degree. The first numerical data point is updated using the first correlation degree corresponding to each first numerical data point. After all the first numerical data points have been updated, the second numerical data point C' is obtained.
[0048] In some embodiments, first numerical data is updated using a first correlation degree to obtain target numerical data. The target numerical data is then normalized to obtain second numerical data.
[0049] S150. Map the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data. The third numerical data includes the coordinate values of the orthogonal mapping points of the n principal components, where n is the first preset threshold.
[0050] In some embodiments, principal component analysis is performed on the second numerical data C'. The second numerical data corresponding to each line of target text data in the first target text data is mapped to a vector space composed of n principal components to obtain the third numerical data corresponding to each line. The third numerical data includes the coordinate values of the orthogonal mapping points of the n principal components, where n is a first preset threshold.
[0051] In one example, the second numerical data corresponding to each line of the first target text data is mapped to a vector space composed of two principal components, resulting in the third numerical data PCA1 and PCA2 for each line. Here, PCA1 represents the first principal component.
[0052] S160. Update the first target text data with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the second text data.
[0053] In some embodiments, the first target text data is updated line by line with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the second text data.
[0054] In one example, the first row of first target text data is updated to PCA1 (third numeric data) of the first row, the second row of first target text data is updated to PCA1 (third numeric data) of the second row, and so on, until the last row of first target text data is updated to obtain the second text data.
[0055] It is understandable that the second text data is the first target text data after being numericalized.
[0056] It should be noted that by using principal component analysis to reduce the dimensionality of the second numerical data and then selecting the key feature PCA1, computational resources can be saved and computational costs reduced.
[0057] S170. Merge the second text data and the first numerical data to obtain the first text feature.
[0058] In some embodiments, the second textual data and the first numerical data are merged, the numericalized first target textual data is written into the structured data, and the first target textual data is deleted. The structured data includes the new numerical data, namely the first textual feature, and textual data other than the first target textual data.
[0059] This application first selects the first target text data of the first target column and calculates its first correlation degree with the first numerical data. The first numerical data is then updated using the first correlation degree to obtain the second numerical data. The text data is then fused into the original numerical data based on the correlation degree. A vector space is then constructed, which itself has strong interpretability for the text data. In the vector space composed of n principal components, the coordinates of the first principal component (i.e., the orthogonal mapping points of the key features) are selected. The first target text data is updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the second text data. This second text data is numerically converted data. That is, by using the numerical data inherent in the structured data as a base point, the text data is numerically converted, retaining key features and not relying on the model's training data. This improves the accuracy of converting structured data into text features, thereby avoiding the loss of semantic information in text features used by machine learning and deep learning models in practical applications.
[0060] In some embodiments, each column of first text data includes multiple sets of text fields, and each set of text fields is identical; for example... Figure 2 As shown, the above S120 may include S121 to S122.
[0061] S121. Calculate the relative entropy of the first text data in each column. The relative entropy represents the amount of information loss when the numerical distribution of one set of text fields in a set of multiple text fields is approximated by the numerical distribution of another set of text fields.
[0062] In some embodiments, the inter-group divergence of each column of first text data in the numerical space is calculated to obtain the relative entropy of each column of first text data. Specifically, each column of first text data includes multiple sets of text fields, and each set of text fields is identical. The sum of squared Mahalanobis distances for different text fields in each column is calculated to further determine the relative entropy of each column of first text data. Relative entropy characterizes the amount of information loss that occurs when the numerical distribution of one set of text fields in multiple sets of text fields is approximated by the numerical distribution of another set of text fields.
[0063] S122. Select the first target text data of the first target column from the first text data that satisfies the preset condition for relative entropy.
[0064] In some embodiments, the preset condition is to select the data with the highest relative entropy. From at least one column of first text data, the first target text data of the first target column with the highest relative entropy is selected.
[0065] This application embodiment uses relative entropy to characterize the numerical distribution of one set of text fields among multiple sets of text fields. When the numerical distribution of another set of text fields is used to approximate the distribution, the resulting information loss is reduced. This allows for more efficient capture of core difference information. Based on the relative entropy, the first target text data is selected for priority numericalization, reducing information loss and redundancy in subsequent analysis. In other words, it prioritizes high-value differences without losing core information.
[0066] In some embodiments, S140 may include: The first numerical data is updated by multiplying the first correlation level with the first numerical data to obtain the second numerical data.
[0067] In some embodiments, the first correlation degree is the correlation coefficient between the first target text data and the first numerical data, and each first numerical data has a corresponding first correlation degree. The product of each first numerical data and its corresponding first correlation degree is calculated, and the first numerical data is updated using the product until all first numerical data are updated, resulting in the second numerical data.
[0068] In one example, for the normalized value of "178" in the first row of the "Height" column in Table 1, the first correlation between "Elementary School" and the normalized value of "178" in the first row is used to update the product of the normalized value of "178" and the first row of the "Height" column to obtain the second numerical data in the first row of the "Height" column.
[0069] In another example, for the normalized value of "165" in the second row of the "Height" column in Table 1, the first correlation between "Junior High School" and the normalized value of "165" in the second row is used to update the product of the normalized value of "165" and the value of "Junior High School" in the second row, so as to obtain the second numerical data of the second row of the "Height" column.
[0070] In this embodiment, the first correlation degree between the first target text data and the first numerical data is multiplied by the first numerical data to form the second numerical data. The second numerical data is used to replace the first numerical data for further numericalization. This retains the characteristics of the original numerical data while adding the characteristics of the first target text data. Without losing the original numerical information, the core text information is preserved, which can improve the accuracy of converting structured data into text features.
[0071] In some embodiments, the dimension of the first numerical data in each row is greater than a preset dimension; for example... Figure 3 As shown, S150 may include S151 to S153.
[0072] S151. Map the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of m principal components to obtain the fourth numerical data corresponding to each line of target text data in the first target text data. The fourth numerical data includes the coordinate values of the orthogonal mapping points of the m principal components, where m is the second preset threshold.
[0073] In some embodiments, when the dimension of each row of first numerical data is greater than a preset dimension, dimensionality reduction of the data is performed in two principal component analyses. First, the second numerical data corresponding to each row of target text data in the first target text data is mapped to a vector space composed of m principal components to obtain the fourth numerical data corresponding to each row of target text data in the first target text data. The fourth numerical data includes the coordinate values of the orthogonal mapping points of the m principal components, where m is a second preset threshold.
[0074] It is understandable that each first numerical data point has a corresponding second numerical data point, and the first and second numerical data points have the same dimension.
[0075] S152. From the fourth numerical data, select the fifth numerical data according to the preset filtering rules. The fifth numerical data includes the coordinate values of the orthogonal mapping points of the k principal components.
[0076] In some embodiments, each row of second numerical data is first subjected to principal component analysis for dimensionality reduction, resulting in fourth numerical data corresponding to each row of target text data in the first target text data. From the fourth numerical data, fifth numerical data is selected according to a preset filtering rule. The fifth numerical data includes the coordinate values of the orthogonal mapping points of k principal components. The preset filtering rule is set in advance by the user according to their needs.
[0077] In one example, the preset filtering rule is to select the coordinates of the orthogonal mapping points of k principal components from the coordinates of the orthogonal mapping points of m principal components.
[0078] It is understandable that m > k.
[0079] S153. Map the fifth numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data.
[0080] Where m>k>n.
[0081] In some embodiments, a second principal component analysis is performed on the fifth numerical data selected according to a preset filtering rule, mapping the fifth numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data.
[0082] In some embodiments, when the dimension of the first numerical data in each row is greater than the preset dimension, Principal Component Analysis-Least Absolute Shrinkage and Selection Operator-Principal Component Analysis (PCA-LASSO-PCA) is used to perform dimensionality reduction twice on the first numerical data in each row.
[0083] For ultra-high-dimensional data, this application embodiment can use principal component analysis twice consecutively to gradually reduce dimensionality. Using principal component analysis twice consecutively on ultra-high-dimensional data can achieve deeper dimensionality reduction and noise reduction, significantly improving computational efficiency.
[0084] In some embodiments, prior to S150 above, the method may further include: Receive user input for a first preset threshold and a second preset threshold.
[0085] The first preset threshold is less than the second preset threshold.
[0086] Understandably, for data that requires high-dimensional numericalization or higher interpretability, a threshold can be selected according to the required interpretability to determine the required principal component dimension, namely the first preset threshold and the second preset threshold.
[0087] In this embodiment of the application, users can set the dimensions of the two principal component analyses according to their actual needs, thereby improving the user experience.
[0088] In some embodiments, S160 may include: The first target text data is updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the third text data. The third text data is normalized to obtain the second text data.
[0089] In this embodiment, the first target text data after numericalization is normalized to eliminate the influence of magnitude, making it easier for machine learning and deep learning models to use.
[0090] It should be noted that when the structured data includes multiple columns of first-text data, the first-text data is quantified column by column.
[0091] In some embodiments, such as Figure 4 As shown, after S170 above, the method may also include S210 to S260.
[0092] S210. When the structured data includes multiple columns of first text data, select the second target text data of the second target column that meets the preset conditions from the text data other than the first target text data in the first text data.
[0093] In some embodiments, the preset condition is random data selection. When the structured data includes multiple columns of first text data, second target text data of the second target column is randomly selected from the text data excluding the first target text data.
[0094] In some embodiments, when the structured data includes multiple columns of first text data, for the text data other than the first target text data in the first text data, the second target text data of the second target column whose relative entropy meets the preset condition is selected according to the relative entropy of each column of text data.
[0095] In some embodiments, when the structured data includes multiple columns of first text data, for the text data other than the first target text data in the first text data, the second target text data of the second target column with the largest relative entropy is selected according to the relative entropy of each column of text data.
[0096] In one example, from the first text data shown in Table 1, a second target text data column that meets preset conditions is selected.
[0097] S220. Calculate the second degree of correlation between the second target text data and the first text feature in each row.
[0098] In some embodiments, the correlation coefficients of each row of second target text data and the corresponding first text feature are calculated to obtain the second degree of association between each row of second target text data and the first text feature.
[0099] In one example, the second degree of association between “poor” in the first row as shown in Table 1 and “178”, “75”, “4500” and “47” in the first row is calculated.
[0100] In another example, the second degree of association between “poor” in the second row as shown in Table 1 and “165”, “61”, “6000” and “52” in the second row is calculated.
[0101] It is understandable that each first text feature has a corresponding second degree of association.
[0102] S230. Update the first text feature using the second degree of correlation to obtain the sixth numerical data.
[0103] In one embodiment, each first text feature has a corresponding second correlation degree. The first text feature is updated using the second correlation degree corresponding to each first text feature. After all first text features are updated, the sixth numerical data is obtained.
[0104] In some embodiments, the first text features are updated using a second degree of correlation to obtain target numerical data. The target numerical data is then normalized to obtain sixth numerical data.
[0105] In some embodiments, the first text feature is updated by multiplying the second degree of association with the first text feature to obtain the sixth numerical data.
[0106] S240. Map the sixth numerical data corresponding to each line of target text data in the second target text data to a vector space composed of n principal components to obtain the seventh numerical data. The seventh numerical data includes the coordinate values of the orthogonal mapping points of the n principal components.
[0107] In one example, the sixth numerical data corresponding to each line of the second target text data is mapped to a vector space composed of two principal components, resulting in the seventh numerical data PCA1 and PCA2 for each line. Here, PCA1 represents the first principal component.
[0108] S250. Update the second target text data with the coordinates of the orthogonal mapping points of the first principal component in the seventh numerical data to obtain the fourth text data.
[0109] In some embodiments, the second target text data is updated line by line with the coordinates of the orthogonal mapping points of the first principal component in the seventh numerical data to obtain the fourth text data.
[0110] In one example, the first row of second target text data is updated to PCA1 (seventh numeric data) of the first row, the second row of second target text data is updated to PCA1 (seventh numeric data) of the second row, and so on, until the last row of second target text data is updated, resulting in the fourth text data.
[0111] It is understandable that the fourth type of text data is the second target text data after being numericalized.
[0112] S260. Merge the fourth text data and the first text feature to obtain the second text feature.
[0113] In some embodiments, the fourth text data and the first text feature are merged, the numericalized second target text data is written into the structured data, and the second target text data is deleted. The structured data includes the new numerical data, namely the second text feature, and text data other than the first target text data and the second target text data.
[0114] This application's embodiments can continuously quantify multiple columns of textual data to achieve complete quantization of structured data, facilitating its use in machine learning and deep learning models. Furthermore, quantization is performed column-by-column, avoiding data corruption.
[0115] The text numericalization method provided in this application improves the accuracy of converting structured data into text features, thereby avoiding the loss of semantic information in text features used by machine learning and deep learning models in practical applications. For example, a doctoral degree is strongly correlated with high income (r=0.82), and after numericalization, doctoral degree = 1.38, while bachelor's degree = -1.34. Moreover, the embodiments provided in this application do not have excessively high requirements for training corpora, do not generate computationally expensive embedding matrices, and avoid the problem that the distance characteristics between vectors change as the vector dimension increases, making the similarity measurement inaccurate. At the same time, it can also preserve the word order information of textual data and focus on contextual information.
[0116] Based on the data processing method provided in the above embodiments, this application also provides specific implementations of the data processing apparatus. Please refer to the following embodiments.
[0117] See Figure 5 The data processing apparatus 300 provided in this application embodiment includes: The acquisition module 310 is used to acquire structured data, which includes at least one column of first text data and at least one column of first numerical data. The selection module 320 is used to select first target text data that meets preset conditions from the first text data; Calculation module 330 is used to calculate the first correlation degree between the first target text data and the first numerical data in each row; Update module 340 is used to update the first numerical data using the first correlation degree to obtain the second numerical data; The mapping module 350 is used to map the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data. The third numerical data includes the coordinate values of the orthogonal mapping points of the n principal components, where n is a first preset threshold. The update module 340 is used to update the first target text data to the coordinate values of the orthogonal mapping points of the first principal component in the third numerical data, so as to obtain the second text data. The merging module 360 is used to merge the second text data and the first numerical data to obtain the first text feature.
[0118] In some embodiments, each column of first text data includes multiple sets of text fields, and each set of text fields is identical; the selection module 320 can specifically be used for: Calculate the relative entropy of the first text data in each column. The relative entropy represents the amount of information loss when the numerical distribution of one set of text fields in a set of multiple text fields is approximated by the numerical distribution of another set of text fields. From the first text data, select the first target text data of the first target column whose relative entropy meets the preset conditions.
[0119] In some embodiments, the update module 340 may specifically be used for: The first numerical data is updated by multiplying the first correlation level with the first numerical data to obtain the second numerical data.
[0120] In some embodiments, the dimension of each row of first numerical data is greater than a preset dimension; the mapping module 350 can specifically be used for: Map the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of m principal components to obtain the fourth numerical data corresponding to each line of target text data in the first target text data. The fourth numerical data includes the coordinate values of the orthogonal mapping points of the m principal components, where m is the second preset threshold. From the fourth numerical data, the fifth numerical data is selected according to the preset filtering rules. The fifth numerical data includes the coordinate values of the orthogonal mapping points of the k principal components. Map the fifth numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data; Where m>k>n.
[0121] In some embodiments, the device 300 may further include: The receiving module is used to receive a first preset threshold and a second preset threshold input by the user before mapping the second numerical data corresponding to each line of target text data in the first target text data to a vector space composed of n principal components to obtain the third numerical data.
[0122] In some embodiments, the update module 340 may specifically be used for: The first target text data is updated with the coordinates of the orthogonal mapping points of the first principal component in the third numerical data to obtain the third text data. The third text data is normalized to obtain the second text data.
[0123] In some embodiments, the device 300 may further include: The selection module 320 is used to select, after merging the second text data and the first numerical data to obtain the first text feature, and when the structured data includes multiple columns of first text data, select the second target text data that meets the preset conditions from the text data of the second target column other than the first target text data in the first text data. Calculation module 330 is used to calculate the second degree of correlation between each row of second target text data and the first text feature; Update module 340 is used to update the first text features using the second degree of correlation to obtain the sixth numerical data; The mapping module 350 is used to map the sixth numerical data corresponding to each line of target text data in the second target text data to a vector space composed of n principal components to obtain the seventh numerical data. The seventh numerical data includes the coordinate values of the orthogonal mapping points of the n principal components. Update module 340 is used to update the second target text data to the coordinate values of the orthogonal mapping points of the first principal component in the seventh numerical data, so as to obtain the fourth text data; The merging module 360 is used to merge the fourth text data and the first text feature to obtain the second text feature.
[0124] Each module of the data processing apparatus provided in this application embodiment can realize the functions of each step of the data processing method provided above, and can achieve its corresponding technical effects. For the sake of brevity, it will not be described in detail here.
[0125] Based on the same inventive concept, embodiments of this application also provide an electronic device.
[0126] Figure 6 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.
[0127] An electronic device may include a processor 401 and a memory 402 storing computer program instructions.
[0128] Specifically, the processor 401 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0129] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory.
[0130] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.
[0131] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the data processing methods in the above embodiments.
[0132] In one example, the electronic device may also include a communication interface 403 and a bus 410. For example, Figure 6 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 410 and complete communication with each other.
[0133] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0134] Bus 410 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X, PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application contemplates any suitable bus or interconnection. The electronic device can perform the data processing methods described in the embodiments of the present invention, thereby implementing the data processing methods described above.
[0135] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processing methods in the above embodiments.
[0136] This application also provides a computer program product, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to perform various processes implementing any of the above-described data processing method embodiments.
[0137] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0138] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0139] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0140] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0141] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method of processing data, characterized by, The method comprises the following steps: acquiring structured data, the structured data comprising at least one column of first text type data and at least one column of first numerical type data; selecting, from the first text type data, first target text type data of a first target column that meets a preset condition; calculating a first correlation degree of each row of the first target text type data and the first numerical type data respectively; updating the first numerical type data by using the first correlation degree to obtain second numerical type data; mapping the second numerical type data corresponding to each row of target text type data in the first target text type data to a vector space formed by n principal components to obtain third numerical type data, the third numerical type data comprising coordinate values of orthogonal mapping points of the n principal components, and n being a first preset threshold; updating the first target text type data to coordinate values of an orthogonal mapping point of a first principal component in the third numerical type data to obtain second text type data; merging the second text type data and the first numerical type data to obtain first text features.
2. The method of claim 1, wherein, Each column of the first text type data comprises a plurality of groups of text fields, and each group of text fields is identical; the selecting, from the first text type data, first target text type data of a first target column that meets a preset condition comprises: calculating a relative entropy of each column of the first text type data, the relative entropy representing an information loss amount generated when a numerical value distribution of a group of text fields is approximated by a numerical value distribution of another group of text fields; selecting, from the first text type data, first target text type data of a first target column that meets a preset condition.
3. The method of claim 1, wherein, The updating the first numerical type data by using the first correlation degree to obtain second numerical type data comprises: updating the first numerical type data by using a product of the first correlation degree and the first numerical type data to obtain second numerical type data.
4. The method of claim 1, wherein, The dimension of each row of the first numerical type data is greater than a preset dimension; the mapping the second numerical type data corresponding to each row of target text type data in the first target text type data to a vector space formed by n principal components to obtain third numerical type data comprises: mapping the second numerical type data corresponding to each row of target text type data in the first target text type data to a vector space formed by m principal components to obtain fourth numerical type data corresponding to each row of target text type data in the first target text type data, the fourth numerical type data comprising coordinate values of orthogonal mapping points of the m principal components, and m being a second preset threshold; selecting, from the fourth numerical type data, fifth numerical type data according to a preset screening rule, the fifth numerical type data comprising coordinate values of orthogonal mapping points of k principal components; mapping the fifth numerical type data corresponding to each row of target text type data in the first target text type data to a vector space formed by n principal components to obtain third numerical type data; wherein m > k > n.
5. The method of claim 4, wherein, Before the mapping the second numerical type data corresponding to each row of target text type data in the first target text type data to a vector space formed by n principal components to obtain third numerical type data, the method further comprises: The first preset threshold and the second preset threshold are received.
6. The method of claim 1, wherein, The first target text type data is updated to the coordinate value of the orthogonal mapping point of the first principal component in the third numerical type data to obtain second text type data. The first target text type data is updated to the coordinate value of the orthogonal mapping point of the first principal component in the third numerical type data to obtain third text type data. The third text type data is normalized to obtain second text type data.
7. The method of claim 1, wherein, After the second text type data and the first numerical type data are merged to obtain first text features, the method further includes: In the case that the structured data includes multiple columns of first text type data, second target text type data of a second target column that meets the preset condition is selected from text type data other than the first target text type data in the first text type data; The second correlation degree of each row of the second target text type data and the first text features is calculated respectively; The first text features are updated using the second correlation degree to obtain sixth numerical type data; The sixth numerical type data corresponding to each row of target text type data in the second target text type data is mapped to a vector space formed by n principal components to obtain seventh numerical type data, the seventh numerical type data including coordinate values of orthogonal mapping points of n principal components; The second target text type data is updated to the coordinate value of the orthogonal mapping point of the first principal component in the seventh numerical type data to obtain fourth text type data; The fourth text type data and the first text features are merged to obtain second text features.
8. A data processing device, characterized by comprising: It includes: An acquisition module is configured to acquire structured data, the structured data including at least one column of first text type data and at least one column of first numerical type data; A selection module is configured to select first target text type data of a first target column that meets a preset condition from the first text type data; A calculation module is configured to calculate a first correlation degree of each row of the first target text type data and the first numerical type data respectively; An update module is configured to update the first numerical type data using the first correlation degree to obtain second numerical type data; A mapping module is configured to map second numerical type data corresponding to each row of target text type data in the first target text type data to a vector space formed by n principal components to obtain third numerical type data, the third numerical type data including coordinate values of orthogonal mapping points of n principal components, n being a first preset threshold; The update module is configured to update the first target text type data to the coordinate value of the orthogonal mapping point of the first principal component in the third numerical type data to obtain second text type data; A merging module is configured to merge the second text type data and the first numerical type data to obtain first text features.
9. An electronic device, comprising: The device includes a processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the data processing method of any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the data processing method in any one of claims 1-7.
11. A computer program product, characterised in that, The instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device can execute the data processing method in any one of claims 1-7.