Data Comparison and Analysis Methods and Devices Based on Big Data

Abstract

This invention provides a data comparison and analysis method and apparatus based on big data. The method includes: acquiring various data information; the data information includes data content and data attributes; the data attributes include data space usage; determining the corresponding storage medium based on the data space usage, and standardizing the data format in the storage medium to obtain standardized data; processing the data content of the standardized data to determine user needs, and extracting multiple keywords based on user needs; comparing each keyword with a preset keyword index set, and when a keyword is in the keyword index set, determining the classification type for classifying the data based on the keyword index; determining a classifier based on the classification type; and classifying the data content based on the classifier to obtain the output result.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data comparison and analysis method and apparatus based on big data. Background Technology

[0002] In the current technology, most SMS companies focus on the development of SMS business and neglect the analytical value of customer SMS content. Little do they know that the analytical value of SMS content plays an important role in the accurate delivery and marketing of SMS messages. Therefore, how to analyze SMS content to facilitate accurate SMS delivery has become an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a data comparison and analysis method and apparatus based on big data to solve the problems existing in the prior art.

[0004] In a first aspect, the present invention provides a data comparison and analysis method based on big data, the method comprising:

[0005] Acquire various types of data information; the data information includes data content and data attributes; the data attributes include data space usage.

[0006] Based on the space occupied by the data, the corresponding storage medium is determined, and the data format of the data is standardized in the storage medium to obtain standardized data;

[0007] The data content of the standardized data is processed to determine user needs, and multiple keywords are extracted based on the user needs.

[0008] Each keyword is compared with a preset keyword index set. When a keyword is in the keyword index set, the classification type for classifying the data is determined based on the keyword index.

[0009] Determine the classifier based on the classification type;

[0010] The data content is classified according to the classifier to obtain the output result.

[0011] In one possible implementation, the data attribute further includes: data source; the data source includes SMS data, e-commerce data, mini-program browsing data, and website browsing data; determining the corresponding storage medium based on the data's space occupancy specifically includes:

[0012] Based on the data source, determine the corresponding classification interval for the storage medium.

[0013] In one possible implementation, before determining the corresponding storage medium based on the data occupancy space, the method further includes:

[0014] Incremental data synchronization is initiated when the data occupies no more than a preset threshold.

[0015] Import the incremental data into a temporary table; the temporary table is named as the first name.

[0016] Determine if the incremental data import is complete;

[0017] Once the incremental data import is complete, the data other than the incremental data in the formal table will be imported into the temporary table.

[0018] Update the name of the original table to the second name;

[0019] Update the first name to the second name, and delete the second name from the official table.

[0020] In one possible implementation, the classification type includes binary classification, multi-class classification, multi-label classification, and imbalanced classification.

[0021] In one possible implementation, determining the classification type for classifying the data based on the keyword index specifically includes:

[0022] Determine the keyword index corresponding to the keyword; and determine the score of the category type corresponding to the keyword index; wherein, each keyword index corresponds to multiple category types, and each category type has a category type weight value;

[0023] Calculate the total weight value of all keyword categories based on the weight value of each category category corresponding to each keyword.

[0024] The classification type is determined based on the total weight value.

[0025] In one possible implementation, determining the classifier based on the classification type specifically includes:

[0026] Get the first weight value of each classifier corresponding to the current classification type;

[0027] Determine the second weight value for each classifier based on the data space occupied;

[0028] The classifier is determined based on the first weight value and the second weight value.

[0029] In one possible implementation, the classifier includes: Naive Bayes classifier, support vector machine, logistic regression, and decision tree classification algorithm.

[0030] Secondly, the present invention provides a data comparison and analysis device based on big data, the device comprising:

[0031] The acquisition module is used to acquire various types of data information; the data information includes data content and data attributes; the data attributes include the space occupied by the data.

[0032] The first determining module is used to determine the corresponding storage medium based on the space occupied by the data, and to perform unified processing on the data format of the data in the storage medium to obtain standardized data;

[0033] The first processing module is used to process the data content of the standardized data, determine user needs, and extract multiple keywords based on the user needs.

[0034] A comparison module is used to compare each keyword with a preset keyword index set;

[0035] The second determining module is used to determine the classification type of the data based on the keyword index when the keyword is in the keyword index set.

[0036] The third determining module is used to determine a classifier based on the classification type;

[0037] The second processing module is used to classify the data content according to the classifier and obtain the output result.

[0038] Thirdly, the present invention provides a chip system including a processor coupled to a memory, the memory storing program instructions, wherein when the program instructions stored in the memory are executed by the processor, the data comparison and analysis method based on big data as described in any one of the first aspects is implemented.

[0039] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, the computer program being executed by a processor using the data comparison and analysis method based on big data as described in any one of the first aspects.

[0040] By applying the big data-based data comparison and analysis method provided in this embodiment of the invention, the data content can be analyzed to determine the classification type, and then the classifier can be determined, thereby determining the target audience for the data content and achieving accurate identification of the user target. Attached Figure Description

[0041] Figure 1This is a schematic diagram of the data comparison and analysis method based on big data provided in Embodiment 1 of the present invention;

[0042] Figure 2 This is a schematic diagram of the data comparison and analysis device based on big data provided in Embodiment 2 of the present invention;

[0043] Figure 3 This is a schematic diagram of the chip system structure provided in Embodiment 3 of the present invention;

[0044] Figure 4 This is a schematic diagram of the computer storage medium structure provided in Embodiment 4 of the present invention. Detailed Implementation

[0045] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] Example 1

[0048] Figure 1 This is a schematic flowchart of a data comparison and analysis method based on big data provided in Embodiment 1 of the present invention. This method is applied in a system between an enterprise and a target audience. The system processes the data to determine the optimal target audience. This application uses the analysis of SMS messages as an example to illustrate the data comparison method based on big data of this application. Figure 1 As shown, this application includes the following steps:

[0049] Step 110: Obtain various data information; the data information includes data content and data attributes; the data attributes include data space usage.

[0050] Specifically, it can obtain a large number of SMS messages, including customer ID, SMS content, and SMS attributes, such as the storage space occupied by the SMS messages.

[0051] Step 120: Based on the space occupied by the data, determine the corresponding storage medium, and perform data format unification processing on the storage medium to obtain standardized data;

[0052] Specifically, when the number of SMS messages is less than or equal to the threshold, the SMS messages are imported using the following method:

[0053] First, when the space occupied by the data is no greater than a preset threshold, incremental data synchronization is initiated;

[0054] Before executing this method, the SMS content is first checked to determine if it already exists in the database. If the SMS content exists in the database, the memory space storing that SMS content is directly accessed. If the SMS content only partially exists in the database, incremental synchronization can be performed, synchronizing only the parts of the SMS content that are not in the database.

[0055] Secondly, the incremental data is imported into a temporary table; the table name of the temporary table is the first name;

[0056] Specifically, the incremental content from the SMS message is imported into a temporary table, and a first name is added to the temporary table.

[0057] Next, check if the incremental data import is complete;

[0058] Next, once the incremental data import is complete, the data other than the incremental data in the formal table will be imported into the temporary table;

[0059] When the import is complete, the data in the SMS message content that is identical to the incremental data stored in the database before will be imported into a temporary table.

[0060] Next, update the table name of the original table to the second name;

[0061] Finally, delete the second name in the official table and update the first name to the second name.

[0062] Specifically, the formal table is deleted, and the first name of the temporary table is updated to the second name, thereby storing the SMS information. When a new SMS information is received in the future, the new SMS information will be directly imported incrementally along with the current SMS information.

[0063] Specifically, the storage media in this application can be understood as MySQL, DB2, SQL Server, and Oracle. Each storage medium has requirements for the space occupied by the data. Since the data format in each storage medium is different, storage format conversion can be performed to adapt to different storage media.

[0064] Step 130: Process the data content of the standardized data, determine user needs, and extract multiple keywords based on the user needs;

[0065] Specifically, in this application, a keyword index set can be established based on historical data, such as SMS information, and big data analysis. In the keyword index set, it is convenient to query whether each keyword matches the keyword index.

[0066] This application processes standardized data content. When the data content is video or audio, it can be processed into text content. Then, using content analysis, the text content is analyzed to statistically analyze the word frequencies of words or phrases, obtaining keywords. Furthermore, using text analysis, the text content is analyzed and keywords are obtained based on a historical high-frequency feature word selection algorithm. This yields multiple keywords for the advertising information, and each keyword is subsequently processed sequentially.

[0067] Step 140: Compare each keyword with a preset keyword index set. When the keyword is in the keyword index set, determine the classification type for classifying the data based on the keyword index.

[0068] Specifically, first, the type of classification task needs to be identified. This can be achieved by training a classifier on historical big data. For historical big data, after data analysis, multiple keywords can be extracted. For example, for multiple SMS messages, words appearing more than a preset threshold can be identified and used as keywords. Then, the weight values ​​for the corresponding classification types of these keywords are calculated. These classification types include binary classification, multi-class classification, multi-label classification, and imbalanced classification. Thus, each keyword has multiple classification types and weights for each type.

[0069] Binary classification primarily categorizes a dataset into two main classes, meaning the output variable has only two values. For example, it might label a customer as either a "Mi Fan" or a "non-Mi Fan."

[0070] Multivariate classification primarily divides data into multiple categories, and the output variable also has multiple values. For example, the classic Iris dataset classifies flowers into three types based on petal length, sepal length, petal width, and sepal width.

[0071] Multi-label classification is a special type of classification task where each instance in the dataset has multiple output variables; that is, an instance can have multiple labels. For example, in image classification, a single image may contain multiple objects, which can be labeled using algorithms, such as buses, traffic lights, and zebra crossings.

[0072] Imbalanced classification refers to classification problems where instances in a dataset exhibit a biased or skewed distribution, meaning that one class of input variables is more frequent than others. An example is detecting fraudulent transactions using credit cards in a transaction dataset.

[0073] Subsequently, upon receiving the data, after processing in steps 110-130, keywords are obtained, and the classification type is determined based on the keywords, as follows:

[0074] First, determine the keyword index corresponding to the keyword; and determine the score of the category type corresponding to the keyword index; wherein, each keyword index corresponds to multiple category types, and each category type has a category type weight value;

[0075] Secondly, based on the weight value of the category type corresponding to each keyword, calculate the total weight value of the category types corresponding to all keywords;

[0076] Finally, the classification type is determined based on the total weight value.

[0077] In this application, each keyword has a corresponding weight for each category type. The weight of each keyword can be determined based on historical SMS information. For example, if the extracted keywords are "phone number," "judgment," and "Mi Fan," a query reveals that "phone number" has four category types: binary, multi-category, multi-tag, and unbalanced. Each category type has a corresponding weight value; for example, the weight of binary category is 0.7, multi-category category is 0.5, multi-tag category is 0.3, and unbalanced category is 0.2. "Judgment" and "Mi Fan" also have weight values ​​for their respective category types. These weight values ​​are summed, and the category type with the highest weight value is selected as the current category type. For example, if binary category has the highest weight value, then binary category is selected.

[0078] Step 150: Determine the classifier based on the classification type;

[0079] Before performing step 150, this application needs to construct a classifier that can classify big data, thereby enabling better profiling and comparison of tasks to output results that better meet customer expectations.

[0080] The classifiers include Naive Bayes classifier, support vector machine, logistic regression, and decision tree classification algorithms.

[0081] The method for determining a classifier based on these classification types is explained below:

[0082] During classification, a classifier can be determined by comprehensively considering the characteristics of the data and the classification type. The data characteristics include the size of the dataset and the number of features. Each classifier has three attributes: prediction accuracy, training time, and linear relationship with the dataset.

[0083] The size of the dataset can be used to set the weight values ​​of the four classifiers for each dataset based on the space occupied by the dataset. For example, when the space occupied by the data to be processed is no greater than the preset first space threshold, the weight value of the Naive Bayes algorithm is 0.8, the weight value of the Support Vector Machine is 0.6, the weight value of the Logistic Regression algorithm is 0.5, and the weight value of the Decision Tree classification algorithm is 0.2. This establishes the correspondence between the interval in which each data occupies space and the weight value of the corresponding classifier.

[0084] The number of features: Keywords extracted from the data may include unnecessary keywords. By identifying keywords that are hit in the keyword index, we can determine whether a keyword is necessary or unnecessary. We can also count the number and percentage of hit keywords to facilitate subsequent analysis and processing. For example, we can set corresponding weight values ​​based on the range of the percentage.

[0085] Prediction accuracy: Each classifier has a numerical value used to label the prediction accuracy of that classifier, so that the prediction accuracy of each classifier can be labeled. As for how to set the prediction accuracy of each classifier, it can be set based on experience.

[0086] Training time: Each classifier has a value used to mark the training time of that classifier, thus marking the training duration of each classifier. As for how to set the training time for each classifier, it can be set based on experience.

[0087] The linear relationship of the dataset, that is, the linear coefficient between the keyword and the output result, can be obtained by processing historical data information through big data services. The correlation coefficient between each keyword and multiple output results can be obtained, and the correlation coefficient and the output result can be stored accordingly. For example, the correlation coefficient can be stored in descending order, along with the output result under the keyword and the correlation coefficient between the keyword and the output result.

[0088] Thus, multiple classifiers are constructed, each with attributes. For each classification type, each classification type can include at least one classifier. For example, if the classification type is binary classification, the classifiers can include a Naive Bayes classifier and a Support Vector Machine. Each classifier has a corresponding first weight value, and the classifier corresponding to each classification type, along with its first weight value, is stored. As for how to determine the first weight value of the classifier corresponding to the classification type, it can be set based on experience from historical classification, and this application will not elaborate on this further.

[0089] Once the correspondence between classification types and classifiers is established, step 150 includes the following:

[0090] Get the first weight value of each classifier corresponding to the current classification type;

[0091] The query retrieves the first weight value of each classifier corresponding to the current classification type.

[0092] Determine the second weight value for each classifier based on the data space occupied;

[0093] This involves acquiring the space occupied by the data information and querying the weight value corresponding to the space occupied by the data information. Correspondingly, another weight value can be determined based on the percentage of keyword hits. Thus, the weight value of the occupied space and the weight value of the percentage of keyword hits are superimposed to obtain the second weight value.

[0094] The classifier is determined based on the first weight value and the second weight value.

[0095] Specifically, the first and second weight values ​​are superimposed to obtain the final weight value. The final weight values ​​are then sorted, and the classifier corresponding to the largest weight value is taken as the final classifier.

[0096] Step 160: Classify the data content according to the classifier to obtain the output result.

[0097] Specifically, the data is classified according to the final determined classifier to obtain the output result.

[0098] In this application, the data information is the customer's demand information. After being classified by a classifier, the corresponding output results can be obtained.

[0099] For example, if a phone number is received from a client terminal and the goal is to determine whether the phone number belongs to a Xiaomi customer, the classifier will output whether the phone number belongs to a Xiaomi customer or not.

[0100] By applying the big data-based data comparison and analysis method provided in this embodiment of the invention, the data content can be analyzed to determine the classification type, and then the classifier can be determined. The data information can then be classified and processed according to the classifier, and finally the output result of the data content can be determined, thereby achieving accurate determination of the classification of data information.

[0101] Example 2

[0102] Embodiment 2 of the present invention provides a data comparison and analysis device based on big data, such as... Figure 2 As shown, the device includes: an acquisition module 210, a first determination module 220, a first processing module 230, a comparison module 240, a second determination module 250, a third determination module 260, and a second processing module 270.

[0103] The acquisition module 210 is used to acquire various data information; the data information includes data content and data attributes; the data attributes include data space occupied.

[0104] The first determining module 220 is used to determine the corresponding storage medium based on the space occupied by the data, and to perform unified processing on the data format of the data in the storage medium to obtain standardized data;

[0105] The first processing module 230 is used to process the data content of the standardized data, determine user needs, and extract multiple keywords based on the user needs;

[0106] The comparison module 240 is used to compare each keyword with a preset keyword index set;

[0107] The second determining module 250 is used to determine the classification type for classifying the data based on the keyword index when the keyword is in the keyword index set;

[0108] The third determining module 260 is used to determine a classifier based on the classification type;

[0109] The second processing module 270 is used to classify the data content according to the classifier and obtain the output result.

[0110] The data comparison and analysis device based on big data provided in Embodiment 2 of the present invention can execute the method steps in Embodiment 1 of the above method. Its implementation principle and technical effect are similar, and will not be described again here.

[0111] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a determination module can be a separate processing element, or it can be integrated into a chip within the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0112] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0113] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The aforementioned computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0114] Example 3

[0115] Embodiment 3 of the present invention provides a chip system, such as Figure 3As shown, it includes a processor and a coupled memory. The memory stores program instructions. When the program instructions stored in the memory are executed by the processor, any of the big data-based data comparison and analysis methods provided in Embodiment 1 are implemented.

[0116] Example 4

[0117] Embodiment 4 of the present invention provides a computer-readable storage medium, such as... Figure 4 As shown, it includes a program or instructions that, when run on a computer, implement any of the big data-based data comparison and analysis methods provided in Embodiment 1.

[0118] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0119] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0120] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data comparison and analysis method based on big data, characterized in that... The method includes: Acquire various types of data information; the data information includes data content and data attributes; the data attributes include data space usage. Based on the space occupied by the data, the corresponding storage medium is determined, and the data format of the data is standardized in the storage medium to obtain standardized data; The data content of the standardized data is processed to determine user needs, and multiple keywords are extracted based on the user needs. Each keyword is compared with a preset keyword index set. When a keyword is in the keyword index set, the classification type for classifying the data is determined based on the keyword index. The classification type includes binary classification, multi-variable classification, multi-label classification, and imbalanced classification. Determine the classifier based on the classification type; The data content is classified according to the classifier to obtain the output result; Specifically, determining the classification type for data based on the keyword index includes: Determine the keyword index corresponding to the keyword; and determine the score of the category type corresponding to the keyword index; wherein, each keyword index corresponds to multiple category types, and each category type has a category type weight value; Calculate the total weight value of all keyword categories based on the weight value of each category category corresponding to each keyword. The classification type is determined based on the total weight value.

2. The method according to claim 1, characterized in that, The data attributes further include: data source; the data source includes SMS data, e-commerce data, mini-program browsing data, and website browsing data; determining the corresponding storage medium based on the data's space usage specifically includes: Based on the data source, determine the corresponding classification interval for the storage medium.

3. The method according to claim 1, characterized in that, Before determining the corresponding storage medium based on the data occupancy space, the method further includes: Incremental data synchronization is initiated when the space occupied by the data is not greater than a preset threshold. Import the incremental data into a temporary table; the temporary table is named as the first name. Determine if the incremental data import is complete; Once the incremental data import is complete, the data other than the incremental data in the formal table will be imported into the temporary table. Update the name of the original table to the second name; Update the first name to the second name, and delete the second name from the official table.

4. The method according to claim 1, characterized in that, The step of determining the classifier based on the classification type specifically includes: Get the first weight value of each classifier corresponding to the current classification type; Determine the second weight value for each classifier based on the data space occupied; The classifier is determined based on the first weight value and the second weight value.

5. The method according to claim 4, characterized in that, The classifiers include: Naive Bayes classifier, support vector machine, logistic regression, and decision tree classification algorithm.

6. A data comparison and analysis device based on big data, characterized in that... The device includes: The acquisition module is used to acquire various types of data information; the data information includes data content and data attributes; the data attributes include the space occupied by the data. The first determining module is used to determine the corresponding storage medium based on the space occupied by the data, and to perform unified processing on the data format of the data in the storage medium to obtain standardized data; The first processing module is used to process the data content of the standardized data, determine user needs, and extract multiple keywords based on the user needs. A comparison module is used to compare each keyword with a preset keyword index set; The second determining module is used to determine the classification type of the data based on the keyword index when the keyword is in the keyword index set; wherein the classification type includes binary classification, multi-variable classification, multi-label classification, and imbalanced classification. The third determining module is used to determine a classifier based on the classification type; The second processing module is used to classify the data content according to the classifier and obtain the output result. Specifically, the second determining module determines the classification type for classifying the data based on the keyword index, including: Determine the keyword index corresponding to the keyword; and determine the score of the category type corresponding to the keyword index; wherein, each keyword index corresponds to multiple category types, and each category type has a category type weight value; Calculate the total weight value of all keyword categories based on the weight value of each category category corresponding to each keyword. The classification type is determined based on the total weight value.

7. A chip system, characterized in that, The chip system includes a processor coupled to a memory, which stores program instructions. When the program instructions stored in the memory are executed by the processor, the data comparison and analysis method based on big data as described in any one of claims 1-5 is implemented.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor according to any one of claims 1-5, the data comparison and analysis method based on big data.

Images