Data cleaning system
By designing a data cleaning system, using multi-step cleaning and improving particle swarm algorithms, the problem of lengthy data processing in the existing technology is solved, and the rapid formation of user portraits and improvement of marketing success rate is achieved.
Patent Information
- Application Number
- CN202411016199.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, data processing steps are lengthy and cumbersome, resulting in a large amount of time spent on data processing, which is not conducive to the improvement of marketing efficiency and success rate.
A data cleaning system was designed, including a label division module, a template building module, a data collection module, a cleaning and storage module and a timing maintenance module. Through multi-step cleaning and improving the particle swarm algorithm, data matching and storage processing are quickly formed by quickly forming users. portrait.
It achieves rapid acquisition of accurate SMS data and useful information, improves marketing accuracy and speed, and improves marketing success rate.
Smart Images

Figure CN119071743A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a data cleaning system. Background Art
[0002] As people's lives become more convenient, the communication between people is getting closer and closer, which in turn generates a large amount of SMS data. These SMS data come from various networks, enterprises, etc., and contain the real intentions and preferences of the data producers. Therefore, it is urgent to mine these SMS data. We have a large amount of SMS data. We need to clean up the useful information from these data and form user portraits, so as to facilitate precision marketing in combination with user portraits in the later stage to improve the marketing success rate.
[0003] Prior art, such as the invention with publication number CN110009416A, discloses a system based on big data cleaning and AI precision marketing, which only removes dirty, false, fake, and waste data to obtain cleaned data, and then performs further multi-step operations such as data analysis, matching, and data mining to obtain user portraits for precision marketing. Through the above operations, accurate user portraits or group labels can be obtained, and the processed data has high accuracy, making the subsequent marketing services more personalized and humane. However, the data processing steps are lengthy and cumbersome, resulting in more time-consuming data processing, which is not conducive to improving marketing efficiency.
[0004] In order to solve the above problems, the present invention proposes a data cleaning system. Summary of the invention
[0005] The purpose of the present invention is to provide a data cleaning system to solve the problems raised in the background technology:
[0006] The steps of data processing in the prior art are lengthy and cumbersome, resulting in a lot of time spent on data processing, which is not conducive to improving marketing efficiency and success rate.
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] A data cleaning system, comprising:
[0009] Label division module: used to divide SMS data into different levels;
[0010] Template creation module: used to create SMS templates for matching SMS content;
[0011] Data collection module: used to collect SMS data of each user according to time intervals;
[0012] Cleaning and warehousing module: used to clean SMS data, match SMS data with SMS templates and process them into the warehouse;
[0013] Scheduled maintenance module: used to regularly update and maintain user time interval labels.
[0014] Preferably, the label division module divides the SMS data into industry-level labels, platform-level labels and event-level labels.
[0015] Preferably, the data corresponding to the industry-level tag is stored in a main table, and the data corresponding to the platform-level tag is stored in an industry subtable.
[0016] Preferably, the cleaning and warehousing module includes a data cleaning module, a data matching module and a data warehousing module;
[0017] The data cleaning module is used to clean the SMS data;
[0018] The data matching module is used to match the cleaned SMS data with the SMS template and add the corresponding event tag to the user corresponding to the SMS;
[0019] The data storage module is used to perform data storage processing according to the matching results of the data matching module.
[0020] Preferably, the data cleaning module includes a first cleaning module, a second cleaning module and a third cleaning module;
[0021] The first cleaning module is used to perform preliminary cleaning processing on the collected SMS data by removing duplicate values and filling missing values;
[0022] The second cleaning module is used to perform further abnormal data processing on the preliminary cleaning processing result;
[0023] The third cleaning module is used to mine and clean the SMS data based on the cleaning result of the second cleaning module based on a data mining algorithm.
[0024] Preferably, the data mining algorithm of the third cleaning module is as follows:
[0025] Set the condition set:
[0026] P = {p 1 ,p 2 ,…,p P}
[0027] Among them, P represents the condition set in the data mining process; p 1 ,p 2 ,…,p P represents the first, second, ..., Pth condition set in the data mining process; |P| represents all the conditions set in the data mining process;
[0028] According to the set of data mining conditions, the mining solution set that provides accurate results for this mining is further set as follows:
[0029] F={f 1 ,f 2 ,…,f F}
[0030] Where F represents the set of solutions in the data mining process; f 1 ,f 2 ,…,f F represents the first, second, ..., F|th scheme set in the data mining process; |F| represents all schemes set in the data mining process;
[0031] The judgment matrix in the mining process is calculated based on the condition set and solution set as follows:
[0032]
[0033] Among them, W represents the set of solutions in the data mining process; represents the judgment basis formed according to the |P|th condition and the |F|th solution in the judgment matrix;
[0034] The weight conditions for judgment are met:
[0035]
[0036] Among them, w a represents the ath weight coefficient, |Q is the number of weight coefficients;
[0037] The weights of each judgment basis are obtained by optimizing based on the improved particle swarm algorithm:
[0038] v ij (t) = ωv ij (t)+c1r1(p ij (t)-x ij (t))+c2r2(g ij (t)-x ij (t))
[0039] x ij (t+1)=x ij (t)+v ij (t+1)
[0040] Where i = 1, 2, ..., n represents the number of particles; j = 1, 2, ..., m represents the dimension of the search space; t represents the number of iterations; v ij represents the velocity of particle i in the jth dimension; x ijrepresents the position of particle i in the jth dimension; ω represents the inertia weight, which is a constant; c1 and c2 represent learning factors; r1 and r2 represent random numbers between [0,1]; p ij represents the optimal position of particle i in the jth dimension; g ij represents the best position of all particles i in the entire group in the jth dimension;
[0041] Based on genetic operations, the speed and position distribution of particles i and k are cross-operated as follows:
[0042]
[0043] Among them, α and β represent random values between [0,1]; v kj (t) represents the velocity of particle k in the jth dimension; x kj (t) represents the position of particle k in the jth dimension;
[0044] Update individual extreme values and group global extreme values and repeat iterations until the optimal weight parameters are obtained;
[0045] The judgment matrix obtained based on the optimization is normalized and sorted from large to small, and the sorting information is selected based on the sorting result to obtain the cleaned data.
[0046] Preferably, the data matching module also performs optimal matching between SMS data and labels at each level based on an improved particle swarm algorithm.
[0047] Compared with the prior art, the present invention provides a data cleaning system with the following beneficial effects:
[0048] The present invention obtains accurate SMS data based on a multi-step data cleaning operation, and further cleans the data through the conditions of corresponding tags, accurately obtains corresponding useful information, quickly forms user portraits, and improves the accuracy and speed of subsequent marketing combined with user portraits, thereby improving the marketing success rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 This is a system block diagram mentioned in Embodiment 1 of the present invention;
[0050] Figure 2 This is a schematic diagram of the classification accuracy of different user labels mentioned in Example 2 of the present invention.
[0051] 100. Label division module; 200. Template establishment module; 300. Data collection module; 400. Cleaning and warehousing module; 410. Data cleaning module; 420. Data matching module; 430. Data warehousing module; 411. First cleaning module; 412. Second cleaning module; 413. Third cleaning module; 500. Scheduled maintenance module. DETAILED DESCRIPTION
[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0053] The present invention obtains accurate SMS data based on a multi-step data cleaning operation, and further cleans the data through the corresponding label conditions, accurately obtains corresponding useful information, quickly forms a user portrait, and improves the accuracy and speed of marketing combined with the user portrait in the later stage, so as to improve the marketing success rate. Specifically, it includes the following contents.
[0054] Embodiment 1:
[0055] See also Figure 1 , the present invention provides a data cleaning system, comprising:
[0056] The label division module 100 is used to divide the SMS data into different levels; it can maintain a set of systematic labels, the first level is divided according to the industry, the second level is divided according to the platform, and the third level is divided according to the event. Among them, the industry-level label data is stored in the main table, and the platform-level label data is stored in the industry word table.
[0057] Template creation module 200: used to create a text message template for matching text message content;
[0058] Create SMS templates to match SMS content and add corresponding event tags to users.
[0059] Data collection module 300: used to collect SMS data of each user;
[0060] Cleaning and warehousing module 400: used to clean the SMS data, match the SMS data with the SMS template and process the SMS data into the warehouse;
[0061] The first cleaning module 411 and the second cleaning module 412 in the data cleaning module perform multi-step cleaning on the collected SMS data of each user, achieving preliminary cleaning of the SMS data, eliminating duplicate, abnormal and useless data, and facilitating further extraction of data in the later stage to obtain useful information. Then, through the mining and cleaning of the third cleaning module 413, useful information in the SMS data can be extracted according to different label features or attributes of the SMS data to eliminate redundant information. The data matching module 420 is used to match the SMS template, and the corresponding time tag is added to the user. Then, the data storage module 430 performs the storage processing of the data, so as to form a systematic and comprehensive user portrait, so as to combine the user portrait for precision marketing in the later stage of marketing.
[0062] Scheduled maintenance module 500: used for scheduled maintenance of user's time interval tags.
[0063] By regularly maintaining user tags, it is possible to easily distinguish the user's current status. For example, a registered user will no longer be a novice user after three or seven days and will no longer need novice guidance or recommendations.
[0064] Embodiment 2:
[0065] The label division module 100 first divides different industry labels, such as agriculture, forestry, animal husbandry, fishery, mining, manufacturing, electricity, heat, gas and water production and supply, construction, wholesale and retail, transportation, warehousing and postal services, accommodation and catering, information transmission, software and information technology services, finance, real estate, leasing and business services, feasibility research and technical services, water conservancy, environment and public facilities management, resident services, repair and other services, education, health and social work, culture, sports and entertainment, public management, social security and social organizations, international organizers, 19 categories and their 97 major industry categories. Take the ten platforms of the financial industry divided by the label division module 100 as an example, including Du Xiaoman platform, Anyihua platform, Dishuidai platform, 360 IOU platform, Ctrip platform, Xiaohua wallet platform, Jirong platform, Paipaidai platform, Yangqiancan platform and Fenqi Le platform. The label division module 100 divides the events in the platform into registration events, loan events, repayment events and security events. Each event can be divided into labels such as t1, t3, t7, m1, m3, m6, m12, etc., which respectively indicate whether there is such an event within a certain time interval.
[0066] Based on the user SMS data collected by the data collection module 300 and the three cleaning operations of the data cleaning module 410 in the cleaning and warehousing module 400, useful information is obtained, and the data matching module 420 matches the obtained useful information with the labels of each level, and finally the data warehousing module 430 performs warehousing and storage processing. The scheduled maintenance module 500 regularly updates the user labels stored in the warehouse. For example, if the update time limit of the event corresponding to the label t1 is 3 days, then after 3 days, the event corresponding to the label t1 is updated to "No". If the update time limit of the event corresponding to the event t7 is 7 days, then after 7 days, the event corresponding to the label t7 is updated to "No".
[0067] The different schemes in the scheme set are extracted correspondingly to each label condition, and the feature weights of each feature condition are obtained based on the improved particle swarm algorithm. The data label judgment matrix of each user is obtained and a user label accuracy bar chart is drawn. The accuracy level is divided into 0 points or more - very accurate, 80-90 points or less - very accurate, 70-80 points or less - accurate, 60-70 points or less - general, and 60 points or less - inaccurate. For details, please refer to Figure 2 As can be seen from the figure, user 1 can be divided into tags t3, t3, m1, m3 and m6; user 2 can be divided into tags t3, t7, m3, m6 and m12; user 3 can be divided into tags t1, t3, t7, m3 and m6. The scheduled maintenance module 500 then performs regular updates and maintenance based on the time limits corresponding to each tag. For example, when a registered user borrows money for the first time and there is no interest in the first month, the tag corresponding to the interest will be updated after one month.
[0068] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A data cleaning system, characterized in that: include: Label division module (100): used to divide SMS data into different levels; Template creation module (200): used to create a short message template for matching short message content; Data collection module (300): used to collect SMS data of each user according to time intervals; Cleaning and warehousing module (400): used to clean the SMS data, match the SMS data with the SMS template and process the SMS data into the warehouse; The scheduled maintenance module (500) is used to regularly update and maintain the time interval labels of users.
2. A data cleaning system according to claim 1, characterized in that: The label classification module (100) classifies the SMS data into industry-level labels, platform-level labels and event-level labels.
3. A data cleaning system according to claim 2, characterized in that: The data corresponding to the industry-level tag is stored in the main table, and the data corresponding to the platform-level tag is stored in the industry sub-table.
4. A data cleaning system according to claim 1, characterized in that: The cleaning and warehousing module (400) comprises a data cleaning module (410), a data matching module (420) and a data warehousing module (430); The data cleaning module (410) is used to clean the SMS data; The data matching module (420) is used to match the cleaned SMS data with the SMS template, and add a corresponding event tag to the user corresponding to the SMS; The data storage module (430) is used to perform data storage processing according to the matching result of the data matching module (420).
5. A data cleaning system according to claim 4, characterized in that: The data cleaning module (410) comprises a first cleaning module (411), a second cleaning module (412) and a third cleaning module (413); The first cleaning module (411) is used to perform preliminary cleaning processing on the collected SMS data by removing duplicate values and filling missing values; The second cleaning module (412) is used to further process abnormal data on the preliminary cleaning processing result; The third cleaning module (413) is used to perform mining and cleaning of SMS data on the cleaning result of the second cleaning module (412) based on a data mining algorithm.
6. A data cleaning system according to claim 5, characterized in that: The data mining algorithm of the third cleaning module (413) is specifically as follows: Set the condition set: P={p 1 ,p 2 ,…,p P } Among them, P represents the condition set in the data mining process; p 1 ,p 2 ,…,p P represents the first, second, ..., Pth condition set in the data mining process; |P| represents all the conditions set in the data mining process; According to the set of data mining conditions, the mining solution set that provides accurate results for this mining is further set as follows: F={f 1 ,f 2 ,…,f F } Where F represents the set of solutions in the data mining process; f 1 ,f 2 ,…,f F represents the first, second, ..., F|th scheme set in the data mining process; |F| represents all schemes set in the data mining process; The judgment matrix in the mining process is calculated based on the condition set and solution set as follows: Among them, W represents the set of solutions in the data mining process; represents the judgment basis formed according to the |P|th condition and the |F|th solution in the judgment matrix; The weight conditions for judgment are met: Among them, w a represents the ath weight coefficient, |Q is the number of weight coefficients; The weights of each judgment basis are obtained by optimizing based on the improved particle swarm algorithm: v ij (t)=ωv ij (t)+c1r1(p ij (t)-x ij (t))+c2r2(g ij (t)-x ij (t)) x ij (t+1)=x ij (t)+v ij (t+1) Where i = 1, 2, ..., n represents the number of particles; j = 1, 2, ..., m represents the dimension of the search space; t represents the number of iterations; v ij represents the velocity of particle i in the jth dimension; x ij represents the position of particle i in the jth dimension; ω represents the inertia weight, which is a constant; c1 and c2 represent learning factors; r1 and r2 represent random numbers between [0,1]; p ij represents the optimal position of particle i in the jth dimension; g ij represents the best position of all particles i in the entire group in the jth dimension; Based on genetic operations, the speed and position distribution of particles i and k are cross-operated as follows: Among them, α and β represent random values between [0,1]; v kj (t) represents the velocity of particle k in the jth dimension; x kj (t) represents the position of particle k in the jth dimension; Update individual extreme values and group global extreme values and repeat iterations until the optimal weight parameters are obtained; The judgment matrix obtained based on the optimization is normalized and sorted from large to small, and the sorting information is selected based on the sorting result to obtain the cleaned data.
7. A data cleaning system according to claim 6, characterized in that: The data matching module (420) also performs optimal matching between SMS data and labels at each level based on an improved particle swarm algorithm.
Citation Information
Patent Citations
System based on big data cleaning and AI precision marketing
CN110009416A
