A dual-mode data processing method for offline and real-time big data platform

Abstract

The present invention relates to the field of big data processing technology, and in particular to a big data platform offline and real-time dual-mode data processing method, including: obtaining initial real-time data, preliminary real-time data and historical data; obtaining the initial support of each course according to the total number of times each course is purchased in the initial real-time data; dividing the preliminary real-time data and the historical data into several periods; obtaining the final support of each course according to the total number of times each course is purchased in each period in the preliminary real-time data and the historical data and the initial support of each course; obtaining the course support threshold in each category according to the number of courses contained in each category and the volatility of the course support in each category, thereby obtaining the input of the FP tree model and finally obtaining the data mining result. The present invention aims to solve the problem that the course support obtained by the existing method is inaccurate, and thus the association rules of the courses are inaccurate.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to an offline and real-time dual-mode data processing method for a big data platform. Background Art

[0002] The big data platform project is mainly composed of two parts: offline data platform and real-time data platform. Among them, the offline data platform mainly handles the needs that have no or low requirements for data real-time; while the real-time data platform mainly handles the needs that have high requirements for data real-time. Through the offline + real-time data processing mode, different data processing modes can be selected for different needs, making the data processing results more realistic. For example, you can first use offline processing methods to analyze and model historical data, and then combine the modeling results with real-time data to make real-time decisions and recommendations.

[0003] In the process of personalized online course recommendations and inventory optimization of supplementary teaching materials on the electronic online course platform, the real-time system monitors students' purchase records, browsing records and course inventory levels, and the FP-growth algorithm is used on the offline data platform to analyze the course purchase records, obtain the association rules between each course and other courses, and make course recommendations based on the association rules between each course and other courses.

[0004] However, when students purchase courses on online course platforms, they are often affected by holidays and exams, so that the course purchase records are greatly affected by external factors, which causes the support of the courses obtained based on short-term data to be overestimated or underestimated, making the FP tree model inconsistent with the actual performance of the course products, resulting in inaccurate association rules between each course and other courses, which in turn causes inaccurate course recommendation plans. Summary of the invention

[0005] The present invention provides a big data platform offline and real-time dual-mode data processing method to solve the existing problems.

[0006] The present invention provides a big data platform offline and real-time dual-mode data processing method using the following technical solutions:

[0007] The present invention proposes a method for processing data in an offline and real-time dual-mode manner on a big data platform, the method comprising the following steps:

[0008] According to the course purchase records, initial real-time data, preliminary real-time data and historical data are obtained; each of the data includes a course purchase record of a collection period; the course purchase record includes the time when each course was purchased each time and the category to which each course belongs;

[0009] According to the total number of purchases of each course and the category to which it belongs in the initial real-time data, the occurrence frequency of each course, the occurrence frequency of courses in each category, and the occurrence frequency of each course in its category are obtained, and then the initial support degree of each course is obtained;

[0010] According to the collection time of the initial real-time data, the preliminary real-time data and the historical data are divided into several periods, and the corresponding period of the initial real-time data is obtained, and the corresponding period is the period in the historical data; according to the total number of times each course is purchased in each period in the preliminary real-time data and the historical data, the preliminary support of each course in each period of the preliminary real-time data and the preliminary support of each course in the corresponding period of the initial real-time data are obtained; according to the difference between the preliminary support of each course in each period of the preliminary real-time data and the initial support of each course, the volatility of the support of each course and the volatility of the support of courses in each category are obtained; according to the volatility of the support of each course, the volatility of the support of courses in each category, the initial support of each course, the preliminary support of each course and the preliminary support of each course in the corresponding period of the initial real-time data, the final support of each course is obtained;

[0011] According to the number of courses in each category, the volatility of the support of courses in each category, and the total number of times all courses in each category were purchased in the initial real-time data, the support threshold of courses in each category is obtained, and then the input of the FP tree model is obtained;

[0012] According to the input of FP tree model, data mining results are obtained, and then course recommendations are made.

[0013] Furthermore, the specific method of obtaining the initial real-time data, preliminary real-time data and historical data according to the course purchase record is as follows:

[0014] The time when the data is obtained is recorded as the current time, and the time before the current time is recorded as the current time. The course purchase records within the past month are used as the initial real-time data, and the course purchase records within the year before the current time are used as the preliminary real-time data; Course purchase records within the year as historical data;

[0015] The months corresponding to the initial real-time data are recorded as the time corresponding to the initial real-time data;

[0016] in, is the preset initial real-time data collection duration value, It is the preset duration of historical data collection.

[0017] Furthermore, the method of obtaining the occurrence frequency of each course, the occurrence frequency of courses in each category, and the occurrence frequency of each course in its category according to the total number of times each course is purchased and the category to which it belongs in the initial real-time data includes the following specific methods:

[0018]

[0019]

[0020]

[0021] In the formula, Indicates The frequency of occurrence of courses in the category, Indicates the total number of times all courses were purchased in the initial real-time data. Indicates the initial real-time data The total number of times courses in the category were purchased, Indicates The first The frequency of occurrence of courses, Indicates the initial real-time data The first The total number of times the course was purchased, Indicates The first The course is in The frequency of occurrence within a category.

[0022] Furthermore, the specific method of obtaining the initial support of each course includes:

[0023]

[0024] In the formula, Indicates The first Initial support for the course, Indicates The frequency of occurrence of courses in the category, Indicates The first The frequency of occurrence of courses, Indicates The first The course is in The frequency of occurrence within a category, Represents the normalization function.

[0025] Furthermore, the preliminary real-time data and the historical data are divided into several periods according to the collection time of the initial real-time data, and the corresponding period of the initial real-time data is obtained, including the specific method of:

[0026] Will Months is taken as the collection duration of a cycle, and the preliminary real-time data is divided into multiple cycles to obtain the total number of purchases of each course in each cycle in the preliminary real-time data;

[0027] Will Months is taken as the collection duration of a cycle, and the historical data is divided into several cycles to obtain the total number of times each course was purchased in each cycle in the historical data;

[0028] The months corresponding to each cycle in the historical data are recorded as the time corresponding to each cycle in the historical data;

[0029] The period in the historical data that is the same as the time corresponding to the initial real-time data is recorded as the corresponding period of the initial real-time data;

[0030] According to the time difference between the purchase time of all courses in each period and the current time, the corresponding periods of the initial real-time data are sorted from large to small.

[0031] Furthermore, the method of obtaining the volatility of the support of each course and the volatility of the support of courses in each category based on the difference between the preliminary support of each course in each cycle of the preliminary real-time data and the initial support of each course includes the following specific methods:

[0032]

[0033] In the formula, Indicates The first The volatility of the support for each course, Indicates The first Initial support for the course, Indicates the first In the cycle The first Initial support for the course, Indicates the number of cycles contained in the preliminary real-time data. Indicates the collection duration of the initial real-time data. represents the absolute value function;

[0034] Based on the volatility of support for all courses in the same category, we can obtain the volatility of support for courses in each category.

[0035] Furthermore, the method of obtaining the volatility of the support for courses in each category includes:

[0036]

[0037] In the formula, Indicates The volatility of course support within a category, Indicates The number of courses contained in the category.

[0038] Furthermore, the method of obtaining the final support of each course according to the volatility of the support of each course, the volatility of the support of courses in each category, the initial support of each course, the preliminary support of each course, and the preliminary support of each course in the corresponding period of the initial real-time data includes the following specific methods:

[0039]

[0040] In the formula, Indicates The first The final support level of each course, Indicates the first In the cycle The first Initial support for the course, Indicates the number of cycles contained in the preliminary real-time data. Indicates The first Initial support for the course, Indicates The first The volatility of the support for each course, Indicates The volatility of course support within a category, Indicates the collection time of historical data. Indicates the historical data The order value of the corresponding period , The first The corresponding cycle The first Initial support for the course, represents the symbolic function, Represents the normalization function.

[0041] Furthermore, the course support threshold in each category is obtained according to the number of courses in each category, the volatility of the course support in each category, and the total number of times all courses in each category are purchased in the initial real-time data, including the specific method of:

[0042]

[0043] In the formula, Indicates The course support threshold within the category, represents the preset support threshold, Indicates The number of courses in each category, Indicates The volatility of course support within a category, Indicates The total number of times all courses in the category were purchased in the initial real-time data, Indicates The total number of times all courses in the category were purchased during all periods in the preliminary real-time data, Indicates the number of cycles contained in the preliminary real-time data. is an exponential function with a natural constant as base, represents the normalization function, Represents a symbolic function.

[0044] Furthermore, the specific method of obtaining the input of the FP tree model includes:

[0045] The final support of the course is greater than the The first threshold of course support within the category The courses in each category are used as an input to the FP-tree model.

[0046] The beneficial effects of the technical solution of the present invention are as follows: according to the purchase records of the courses, initial real-time data, preliminary real-time data and historical data are obtained; when the initial support of each course is obtained according to the total number of times each course is purchased in the initial real-time data, not only the frequency of occurrence of each course but also the frequency of occurrence of each course in its category are used to pave the way for obtaining the final support of each course; when obtaining the final support of each course, the initial support of each course is adjusted according to the historical data, so that the adjusted initial support is more universal, and the initial support of each course is adjusted according to the historical data. The preliminary support of the course in each cycle of preliminary real-time data determines whether the initial support of the course is too large or too small. The initial support of the course with a large initial support is adjusted downward, and the initial support of the course with a small initial support is adjusted upward, so that the final initial support of each course is closer to the actual situation. When obtaining the course support threshold in each category, the course support in each category is adjusted based on the fact that during the peak season, due to the increase in the overall sales of course products, uncommon course product combinations may also appear frequently, so that more association rules are obtained, thereby increasing the sales of the course. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a flowchart of the steps of a big data platform offline and real-time dual-mode data processing method of the present invention. DETAILED DESCRIPTION

[0049] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a big data platform offline and real-time dual-mode data processing method proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0050] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0051] The specific scheme of the offline and real-time dual-mode data processing method for a big data platform provided by the present invention is described in detail below with reference to the accompanying drawings.

[0052] See also Figure 1 , which shows a flowchart of a method for processing offline and real-time dual-mode data on a big data platform provided by an embodiment of the present invention, the method comprising the following steps:

[0053] Step S001: Obtain initial real-time data, preliminary real-time data and historical data based on the course purchase record.

[0054] The purpose of this embodiment is to analyze the course purchase records and obtain the course recommendation plan of the online course platform.

[0055] Specifically, when obtaining data from the database of the online course platform, the time when the data is obtained is recorded as the current time, and the time before the current time is recorded as the current time. The course purchase records within the past month are used as the initial real-time data, and the course purchase records within the year before the current time are used as the preliminary real-time data. The course purchase records within the year are used as historical data. The course purchase records are the time when each course is purchased and the category of each course. The categories of the courses are as follows: For example, English reading courses and English listening courses both belong to the English category.

[0056] Furthermore, because the current time before The course purchase records within the month are taken as the initial real-time data, that is, the initial real-time data corresponds to a number of months, and the number of months corresponding to the initial real-time data is recorded as the time corresponding to the initial real-time data. , in the case of 12 being satisfied, Under the premise of being divisible, other values ​​can be set in other implementations. The collection time length value of historical data preset in this embodiment , in satisfying Under the premise of , other implementations can be set to other values. The collection time value of the preliminary real-time data is 1. Among them, the collection time of the initial real-time data is within the collection time of the preliminary real-time data, and the collection time of the preliminary real-time data is within the collection time of the historical data.

[0057] At this point, the initial real-time data, preliminary real-time data, historical data, and the time corresponding to the initial real-time data are obtained.

[0058] Step S002: Based on the total number of times each course is purchased and the category to which it belongs in the initial real-time data, the occurrence frequency of each course, the occurrence frequency of courses in each category, and the occurrence frequency of each course in its category are obtained, and then the initial support for each course is obtained.

[0059] It should be noted that when analyzing the course purchase records and obtaining the course recommendation plan, it is necessary to obtain the initial support of each course based on the initial real-time data, and then correct the initial support of each course based on the historical data to obtain the final support of each course. Based on the final support of each course, the course recommendation plan is obtained. Therefore, the initial support of each course is calculated based on the initial real-time data.

[0060] It should be further explained that the support of each course reflects the purchase frequency of each course. And the purchase frequency of each course refers to the purchase frequency of each course in the category to which the course belongs and the purchase frequency of each course in all courses. Therefore, the initial support of each course is obtained based on the number of purchases of each course in the initial real-time data.

[0061] Specifically, the The first The initial support of a course is recorded as , the specific calculation formula is as follows:

[0062]

[0063]

[0064]

[0065]

[0066] In the formula, Indicates The frequency of occurrence of courses in the category, Indicates the total number of times all courses were purchased in the initial real-time data. Indicates the initial real-time data The total number of times courses in the category were purchased, Indicates The first The frequency of occurrence of courses, Indicates the initial real-time data The first The total number of times the course was purchased, Indicates The first The course is in The frequency of occurrence within a category, Indicates The first Initial support for the course, Represents a linear normalization function, whose normalized object is all courses .

[0067] It should be noted that It means the of the categories The course is in The greater the value, the more important the category. of the categories The course is in The more important the category, the The first The course is in The frequency of occurrence within the category The first The greater the impact of the initial support for a course; The smaller the value of of the categories The course is in The less important the category, the The first The frequency of occurrence of the course The first The greater the impact of the initial support for a course.

[0068] At this point, the initial support for each course is obtained.

[0069] Step S003: According to the collection time of the initial real-time data, the preliminary real-time data and the historical data are divided into several periods, and the corresponding period of the initial real-time data is obtained; according to the total number of times each course is purchased in each period in the preliminary real-time data and the historical data, the preliminary support of each course in each period of the preliminary real-time data and the preliminary support of each course in the corresponding period of the initial real-time data are obtained; according to the difference between the preliminary support of each course in each period of the preliminary real-time data and the initial support of each course, the volatility of the support of each course and the volatility of the support of courses in each category are obtained; according to the volatility of the support of each course, the volatility of the support of courses in each category, the initial support of each course, the preliminary support of each course and the preliminary support of each course in the corresponding period of the initial real-time data, the final support of each course is obtained.

[0070] It should be noted that when students purchase online courses on online course platforms, they may be affected by holidays and exams, so the total number of purchases of the same course at different times may be different, which may overestimate or underestimate the support of the course obtained based on short-term data. Therefore, based on historical data and preliminary real-time data, the initial support of each course is corrected to obtain the final support of the course. In addition, when obtaining the final support of the course, the preliminary real-time data and the historical initial real-time data are divided into multiple periods according to the collection time of the initial real-time data.

[0071] It should be further explained that the holiday distribution and examination distribution of each year are similar to those of other years, which makes the total number of purchases of the same course in the same month in different years similar. Therefore, the initial support of the course in the initial real-time data is corrected based on historical data, making the final support of the course more universal.

[0072] It should be further explained that the distribution of holidays and exams in each year is similar to that in other years, which makes the total number of purchases of the same course in the same month in different years similar. Therefore, the volatility of the support of each course is obtained based on the preliminary real-time data. The final support of each course is obtained based on the volatility of the support of each course and the correction value of the initial support of the course.

[0073] Specifically, Months is taken as the collection duration of a cycle, and then the preliminary real-time data is divided into multiple cycles to obtain the total number of times each course is purchased in each cycle in the preliminary real-time data. Among them, the purchase time of each course in each cycle is different from the purchase time of each course in other cycles.

[0074] Furthermore, the same division method is used to divide the historical data into several periods, and the total number of times each course is purchased in each period of the historical data is obtained. The several months corresponding to each period in the historical data are recorded as the time corresponding to each period in the historical data. The period in the historical data that is the same as the time corresponding to the initial real-time data is recorded as the corresponding period of the initial real-time data. According to the time difference between the purchase time of all courses in each period and the current time, the corresponding periods of the initial real-time data are sorted from large to small.

[0075] Furthermore, according to the method for obtaining the initial support of each course, the preliminary support of each course in each period in the preliminary real-time data and the preliminary support of each course in the corresponding period of the initial real-time data in the historical data are obtained, and the final support of each course is obtained.

[0076] Specifically, the The first The final support of a course is recorded as , the specific calculation formula is as follows:

[0077]

[0078]

[0079] In the formula, Indicates The first The volatility of the support for each course, Indicates The first Initial support for the course, Indicates the first In the cycle The first Initial support for the course, Indicates the number of cycles contained in the preliminary real-time data. Indicates the collection duration of the initial real-time data. Indicates The volatility of course support within a category, Indicates The number of courses in each category, represents the absolute value function.

[0080] It should be noted that The larger the value, the The first The more the initial support for a course is affected by external factors, the more The first The greater the degree of adjustment of the support for a course.

[0081]

[0082] In the formula, Indicates The first The final support level of each course, Indicates the collection time of historical data. The first The corresponding cycle The first Initial support for the course, Indicates the historical data The order value of the corresponding period , Represents a sign function. When the input value is greater than 0, the output is 1. When the input value is less than 0, the output is -1. When the input value is equal to 0, the output is 0. Represents a linear normalization function, and the normalized object is all courses in all categories. .

[0083] It should be noted that Reflects the The first Is the initial support of a course higher or lower than the actual support? If its value is greater than 0, it means that the initial support of the course is higher or lower than the actual support. The first The initial support of the course is lower than the actual support, that is, the first The first The initial support for each course is expanded; when , explain The first The initial support of the course is higher than the actual support, that is, the The first The initial support of each course is reduced; Indicates the historical data The order value of the corresponding period The larger the value, the faster the initial real-time data The smaller the time difference between the first corresponding period and the initial real-time data, the smaller the time difference between the first corresponding period and the initial real-time data. The corresponding cycle The first The greater the impact of the initial support for each course on the final support for each course.

[0084] At this point, the final support level for each course is obtained.

[0085] Step S004: Based on the number of courses in each category, the volatility of the course support in each category, and the total number of times all courses in each category were purchased in the initial real-time data, the course support threshold in each category is obtained, and then the input of the FP tree model is obtained.

[0086] It should be noted that, before constructing the FP tree model, a support threshold is preset, and the courses whose final support is greater than the support threshold are used as the input of the FP tree model. Therefore, a support threshold is preset. In addition, the support of a course is affected differently by external factors such as holidays and exams, so that the final support of the same course at different times is different, making it impossible to correct the support of courses at different times through a fixed support threshold, so the support threshold is adaptively adjusted.

[0087] It should be further explained that in the peak season, due to the increase in the overall sales of course products, uncommon course product combinations may also appear frequently. Therefore, in order to capture more association rules, the support threshold is lowered in the peak season. In addition, the peak and off-peak seasons of courses are mainly determined based on the sales of course products. Therefore, the support threshold is adjusted based on the overall sales of course products.

[0088] It should be further explained that, although lowering the support threshold can help the algorithm obtain more association rules, it will also increase the amount of calculation. Therefore, when adjusting the support according to the overall sales volume of course products, the support threshold is adjusted in combination with the number of courses contained in each category to obtain the course support threshold in each category. Also, when adjusting the support threshold according to the number of courses contained in a category, if the current category contains more course products, increasing the support threshold can reduce more candidate item sets, thereby reducing the amount of calculation and making the algorithm run faster. Therefore, based on the number of courses in each category and the number of courses in the initial real-time data, the support threshold is adjusted accordingly. The total number of courses purchased in the first category and the total number of courses purchased in each category in each cycle of the preliminary real-time data are obtained. The course support threshold within the category.

[0089] Specifically, the The course support threshold within a category is recorded as , the specific calculation formula is as follows:

[0090]

[0091] In the formula, Indicates The course support threshold within the category, represents the preset support threshold. The preset support threshold in this embodiment is , this is used as an example for description, and other implementations may be set to other values, Indicates The number of courses in each category, Indicates The volatility of course support within a category, Indicates The total number of times all courses in the category were purchased in the initial real-time data, Indicates The total number of times all courses in the category were purchased during all periods in the preliminary real-time data, Indicates the number of cycles contained in the preliminary real-time data. is an exponential function with a natural constant as the base. The model shows an inverse proportional relationship. As the input of the model, the implementer can set the inverse proportional function according to the actual situation; represents a linear normalization function. The normalization interval of this embodiment is [ ]; Represents a sign function. When the input value is greater than 0, the output is 1. When the input value is less than 0, the output is -1. When the input value is equal to 0, the output is 0.

[0092] It should be noted that The larger the value, the The more courses a category contains, the larger the The course support threshold within a category can reduce the amount of calculation even more; The value of is less than 0, indicating that the time corresponding to the initial real-time data is The peak season for sales of courses within a category; Indicates The greater the volatility of the course support within a category, the greater the value, which indicates that uncommon course product combinations are more likely to appear frequently during peak seasons. That is, lowering the course support threshold can capture more association rules.

[0093] Furthermore, the final support of the course is greater than the first The first threshold of course support within the category The courses in each category are used as an input to the FP-tree model.

[0094] At this point, the course support threshold in each category and the input of the FP-tree model are obtained.

[0095] Step S005: According to the input of the FP tree model, data mining results are obtained, and then course recommendations are made.

[0096] Specifically, according to the courses as the input of the FP tree model, the categories and the header table under each category are constructed, and the category FP tree model and the course FP tree in each category are further obtained. Among them, constructing the header table and constructing the FP tree model are both well-known technologies, which will not be repeated in this embodiment.

[0097] Furthermore, the FP-Growth algorithm is used to mine the frequent item sets of each category FP tree and the course FP tree within each category, and the course association rules under each FP tree model are obtained. The category association rules under the category FP tree model and the course association rules under the course FP tree model are combined and analyzed to obtain the association rules between courses. Finally, the association rules for students when purchasing course products are obtained. Among them, the FP-Growth algorithm, course association rules, and the combined analysis of course association rules are all well-known technologies, which will not be repeated in this embodiment.

[0098] Furthermore, according to the association rules when students purchase course products, combined with their real-time browsing and purchasing behaviors, courses are recommended to students. The integration between the mining results of offline data and the initial real-time data processing platform is ensured to achieve seamless data flow and complete the processing of offline and real-time dual-mode data based on the big data platform.

[0099] At this point, this embodiment is completed.

[0100] To sum up, in the embodiments of the present invention, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing data in dual modes of offline and real-time on a big data platform, characterized in that: The method comprises the following steps: According to the course purchase records, initial real-time data, preliminary real-time data and historical data are obtained; each of the data includes a course purchase record of a collection period; the course purchase record includes the time when each course was purchased each time and the category to which each course belongs; According to the total number of purchases of each course and the category to which it belongs in the initial real-time data, the occurrence frequency of each course, the occurrence frequency of courses in each category, and the occurrence frequency of each course in its category are obtained, and then the initial support degree of each course is obtained; According to the collection time of the initial real-time data, the preliminary real-time data and the historical data are divided into several periods, and the corresponding period of the initial real-time data is obtained, and the corresponding period is the period in the historical data; according to the total number of times each course is purchased in each period in the preliminary real-time data and the historical data, the preliminary support of each course in each period of the preliminary real-time data and the preliminary support of each course in the corresponding period of the initial real-time data are obtained; according to the difference between the preliminary support of each course in each period of the preliminary real-time data and the initial support of each course, the volatility of the support of each course and the volatility of the support of courses in each category are obtained; according to the volatility of the support of each course, the volatility of the support of courses in each category, the initial support of each course, the preliminary support of each course and the preliminary support of each course in the corresponding period of the initial real-time data, the final support of each course is obtained; According to the number of courses in each category, the volatility of the support of courses in each category, and the total number of times all courses in each category were purchased in the initial real-time data, the support threshold of courses in each category is obtained, and then the input of the FP tree model is obtained; According to the input of FP tree model, data mining results are obtained, and then course recommendations are made.

2. The method for processing data in a dual mode of offline and real-time on a big data platform according to claim 1, characterized in that: The specific method of obtaining the initial real-time data, preliminary real-time data and historical data according to the course purchase record is as follows: The time when the data is obtained is recorded as the current time, and the time before the current time is recorded as the current time. The course purchase records within the past month are used as the initial real-time data, and the course purchase records within the year before the current time are used as the preliminary real-time data; Course purchase records within the year as historical data; The months corresponding to the initial real-time data are recorded as the time corresponding to the initial real-time data; in, is the preset initial real-time data collection duration value, It is the preset duration of historical data collection.

3. The method for processing data in a dual mode of offline and real-time on a big data platform according to claim 1, characterized in that: The method of obtaining the occurrence frequency of each course, the occurrence frequency of courses in each category, and the occurrence frequency of each course in its category according to the total number of times each course is purchased and the category to which it belongs in the initial real-time data includes the following specific methods: In the formula, Indicates The frequency of occurrence of courses in the category, Indicates the total number of times all courses were purchased in the initial real-time data. Indicates the initial real-time data The total number of times courses in the category were purchased, Indicates The first The frequency of occurrence of courses, Indicates the initial real-time data The first The total number of times the course was purchased, Indicates The first The course is in The frequency of occurrence within a category.

4. The method for processing data in a dual mode of offline and real-time on a big data platform according to claim 3, characterized in that: The specific method for obtaining the initial support of each course is as follows: In the formula, Indicates The first Initial support for the course, Indicates The frequency of occurrence of courses in the category, Indicates The first The frequency of occurrence of courses, Indicates The first The course is in The frequency of occurrence within a category, Represents the normalization function.

5. The method for processing data in a dual mode of offline and real-time on a big data platform according to claim 2, characterized in that: The method of dividing the preliminary real-time data and the historical data into several periods according to the collection time of the initial real-time data and obtaining the corresponding period of the initial real-time data includes: Will Months is taken as the collection duration of a cycle, and the preliminary real-time data is divided into multiple cycles to obtain the total number of purchases of each course in each cycle in the preliminary real-time data; Will Months is taken as the collection duration of a cycle, and the historical data is divided into several cycles to obtain the total number of times each course was purchased in each cycle in the historical data; The months corresponding to each cycle in the historical data are recorded as the time corresponding to each cycle in the historical data; The period in the historical data that is the same as the time corresponding to the initial real-time data is recorded as the corresponding period of the initial real-time data; According to the time difference between the purchase time of all courses in each period and the current time, the corresponding periods of the initial real-time data are sorted from large to small.

6. The method for processing data in a dual mode of offline and real-time on a big data platform according to claim 1, characterized in that: The method of obtaining the volatility of the support of each course and the volatility of the support of courses in each category based on the difference between the preliminary support of each course and the initial support of each course in each cycle of the preliminary real-time data includes the following specific methods: In the formula, Indicates The first The volatility of the support for each course, Indicates The first Initial support for the course, Indicates the first In the cycle The first Initial support for the course, Indicates the number of cycles contained in the preliminary real-time data. Indicates the collection duration of the initial real-time data. represents the absolute value function; Based on the volatility of support for all courses in the same category, we can obtain the volatility of support for courses in each category.

7. The method for processing data in dual modes of offline and real-time on a big data platform according to claim 6, characterized in that: The specific method for obtaining the volatility of course support in each category is as follows: In the formula, Indicates The volatility of course support within a category, Indicates The number of courses contained in the category.

8. The method for processing data in a dual mode of offline and real-time on a big data platform according to claim 1, characterized in that: The method of obtaining the final support of each course according to the volatility of the support of each course, the volatility of the support of courses in each category, the initial support of each course, the preliminary support of each course and the preliminary support of each course in the corresponding period of the initial real-time data includes the following specific methods: In the formula, Indicates The first The final support level of each course, Indicates the first In the cycle The first Initial support for the course, Indicates the number of cycles contained in the preliminary real-time data. Indicates The first Initial support for the course, Indicates The first The volatility of the support for each course, Indicates The volatility of course support within a category, Indicates the collection time of historical data. Indicates the historical data The order value of the corresponding period , The initial real-time data The corresponding cycle The first Initial support for the course, represents the symbolic function, Represents the normalization function.

9. The method for processing data in dual modes of offline and real-time on a big data platform according to claim 7, characterized in that: The method of obtaining the course support threshold in each category according to the number of courses in each category, the volatility of the course support in each category, and the total number of times all courses in each category are purchased in the initial real-time data includes the following specific methods: In the formula, Indicates The course support threshold within the category, represents the preset support threshold, Indicates The number of courses in each category, Indicates The volatility of course support within a category, Indicates The total number of times all courses in the category were purchased in the initial real-time data, Indicates The total number of times all courses in the category were purchased during all periods in the preliminary real-time data, Indicates the number of cycles contained in the preliminary real-time data. is an exponential function with a natural constant as base, represents the normalization function, Represents a symbolic function.

10. The method for processing data in dual modes of offline and real-time on a big data platform according to claim 9, characterized in that: The specific method of obtaining the input of the FP tree model includes: The final support of the course is greater than the The first threshold of course support within the category The courses in each category are used as an input to the FP-tree model.

Images