Dynamic model establishment method based on big data technology

Abstract

The invention relates to the technical field of big data application, in particular to a dynamic model establishment method based on a big data technology. Comprising the steps that target types are set, and the target types comprise a user target, a crowd target with association attributes, a single enterprise target, an enterprise group target with association attributes and a commodity target; collecting target dynamic related data in real time by using a system log and a system database of the big data platform; classifying and marking all the obtained data, and clustering the multi-target multi-class data according to the types of the obtained multiple pieces of target data; and a weighted linear regression ensemble learning method of machine learning is utilized to learn data of a cluster where the multi-target common interest feature points are located, and establishment of a dynamic interest model is completed. According to the invention, the cost of the current big data service can be effectively reduced, and services with various requirements can be provided for more small and medium-sized enterprises.

Description

Technical Field

[0001] The present invention relates to the field of big data application technology, and in particular to a dynamic model building method based on big data technology. Background Art

[0002] With the rapid development of the Internet, people have more and more ways to obtain information, but the various information on the Internet is extremely redundant and has far exceeded the limits that individuals and general enterprises can accept and process. Therefore, big data technology came into being. At present, for their own interests or development needs, both individuals and enterprises need the support of various data. Therefore, there are many data acquisition methods for individuals or enterprises, and the corresponding companies providing data services are also developing and increasing rapidly. In the increasingly fierce competition environment, a variety of service classifications have emerged, including high-end services that provide precise services and low-end services that provide general accuracy, each with its own market. Generally speaking, high-end services are more favored by companies and require larger investments, but in fact, there are fewer companies that need high-end services, and more companies need lower prices. Therefore, how to improve the cost-effectiveness of big data services is more important.

[0003] The more popular user interest recommendation service, for which various product companies can selectively push advertisements, is often provided by collecting user operations, historical browsing records, etc. However, in reality, user interests are very changeable. If you want to accurately recommend specific users, you usually have to collect multi-faceted data on the individual user. In practice, this is difficult to generate real benefits unless it is a special case or when facing large customers. For individual users, the cost may be relatively high.

[0004] In fact, many users have common interests, and these common users can be linked together. Of course, the same goes for enterprises. The linked users or enterprises can be treated as the same type of targets, and then data can be collected uniformly, models can be established based on needs, and then data services can be provided. Although this reduces accuracy, it also reduces costs. For most users who do not need too precise services, this is an effective way to reduce costs. Summary of the invention

[0005] The present invention belongs to the field of big data application technology, and relates to a method for establishing a dynamic model based on big data technology. The present invention mainly includes setting target types, and the target types include user targets, crowd targets with associated attributes, single enterprise targets, enterprise group targets with associated attributes, and commodity targets; using the system log and system database of the big data platform to collect target dynamic related data in real time; classifying and marking all acquired data, and clustering multiple categories of data of multiple targets according to the categories of multiple acquired target data; using the weighted linear regression integrated learning method of machine learning, learning the cluster data where the common interest feature points of multiple targets are located, and completing the establishment of a dynamic interest model. The beneficial effect of the present invention is that it can effectively reduce the cost of current big data services, and can provide services with multiple needs for more small and medium-sized enterprises.

[0006] To solve the shortcomings of the existing technology, a dynamic model building method based on big data technology is provided, which includes the following steps: S1. Setting target types, wherein the target types include user targets, group targets with associated attributes, single enterprise targets, enterprise group targets with associated attributes, and product targets; S2, data collection, according to the target set in step S1, using the system log and system database of the big data platform, real-time collection of target dynamic related data, the target dynamic related data including target public data, target operation data, target dynamic related data and target flow data; S3, according to the data collected in step S2, all the acquired data are classified and marked, and according to the categories of the acquired multiple target data, the multiple categories of data of multiple targets are clustered, that is, a similarity threshold is set, and the categories less than the threshold are judged to have the same attributes; according to the clustering results, the same categories among the multiple targets are found, and the same categories are set as the common interest feature points of the multiple targets; S4. Dynamic interest model is established. The weighted linear regression ensemble learning method of machine learning is used to learn the cluster data of the common interest feature points of multiple targets, and the effect of different feature attribute data on the common interest feature points of multiple targets is analyzed to obtain the interest value prediction function of the target at different interest points. Then, combined with the current data of the target, the values ​​of the common interest feature points of multiple targets are calculated and sorted to form an interest point vector and complete the establishment of the dynamic interest model.

[0007] In step S2, after collecting the data, the data is preprocessed in the following manner: Through data cleaning and data standardization, the collected data is supplemented with missing values, smoothed with noise data, and digitized to achieve the goals of data format standardization, error correction, removal of duplicate data, and data computability processing; through data integration, data from multiple data sources are integrated into one data set; Write the integrated data set into a matrix format, use each target dynamic related data as row data, and each target dynamic related data collected as column data. For the constructed matrix data, the data type required for each target is different, and further screening of the data is also included, using row screening and column screening respectively; The row screening method is to set a threshold value and delete the values ​​less than the threshold value, thereby filtering out the data with the lowest relevance. The column screening method is to use an evaluation function, set weights of different target types for each type of data for different types of targets, evaluate and calculate the collected data according to different weights, and obtain a valid data range, retain the data within the range, and delete the data beyond the range.

[0008] The specific method of step S3 is: S31, from the obtained data, select part of the data near the threshold data as the initial cluster center; S32, using a clustering algorithm to cluster similar data in the acquired multi-target data, that is, similar data of multiple targets are used to jointly form a cluster; S33. According to the clustering result, the feature points with the highest frequency of occurrence in the clusters are selected as the common interest feature points of the multiple targets.

[0009] The specific method of step S4 is: S41, obtaining user cluster data: according to the result of step S3, obtaining the user's interest value prediction function by performing machine learning on the cluster data, and further calculating the user's interest value for the point of interest; S42, using a weighted linear regression ensemble learning method of machine learning to analyze and process the training set data, thereby obtaining a prediction function of the interest value of the target; S43, using the interest value prediction function obtained by machine learning and combining the current data of the target to calculate the interest values ​​of the multiple targets at the interest points; S44. Various points of interest and their interest values ​​obtained by analyzing the target data are represented by a spatial vector model, thereby completing the model building process.

[0010] It also includes: updating the established dynamic model, setting different update cycles according to the selected multi-objective type, and starting the update cycle after the dynamic model is obtained. After the update time is reached, the dynamic model establishment steps are repeated to re-establish the model.

[0011] The present invention mainly includes setting target types, including user targets, crowd targets with associated attributes, single enterprise targets, enterprise group targets with associated attributes, and commodity targets; using the system log and system database of the big data platform to collect target dynamic related data in real time; classifying and marking all acquired data, clustering multiple types of data of multiple targets according to the categories of multiple acquired target data; using the weighted linear regression integrated learning method of machine learning to learn the cluster data of the common interest feature points of multiple targets, and complete the establishment of dynamic interest model. The present invention can effectively reduce the cost of current big data services, and can provide services with multiple needs for more small and medium-sized enterprises. DETAILED DESCRIPTION

[0012] In view of the above problems, the present invention proposes a dynamic model building method based on big data technology.

[0013] The technical solution of the present invention is: a method for establishing a dynamic model based on big data technology, comprising the following steps: S1. Setting target types, wherein the target types include user targets, group targets with associated attributes, single enterprise targets, enterprise group targets with associated attributes, and product targets; User goals are specifically user behavior data, user survey data, and market share data. By collecting user behavior data, such as browsing history, purchase history, search behavior, etc., we can understand the user's interests, preferences, and behavioral characteristics; at the same time, conducting user surveys can obtain users' needs, evaluations, and suggestions for products or services. In addition, market share data can provide information about the product's performance in the market. Based on these data, we can analyze user goals and optimize product design and marketing strategies; The crowd targets with associated attributes are specifically user portrait data, social network analysis data, and associated attribute data. Through user portrait data, we can understand the associated attributes of the crowd, such as age, gender, region, interests and hobbies. At the same time, social network analysis can reveal the relationship and communication characteristics between people. Furthermore, combined with the associated attribute data, the crowd can be segmented and the crowd targets with associated attributes can be identified. Through these analyses, market positioning, advertising and user maintenance can be carried out in a targeted manner.

[0014] The specific single enterprise goal is enterprise financial data, market research data, and competitor data. By analyzing the enterprise financial data, we can understand the enterprise's revenue, profit, cash flow, etc., and judge the enterprise's operating conditions; at the same time, market research data and competitor data can provide information such as industry trends, market share and competitiveness. Combining these data, we can analyze the single enterprise goal and provide support for the enterprise to formulate development strategies and business strategies.

[0015] The enterprise group targets with associated attributes are specifically enterprise credit data, industry reports, and market research data. By analyzing enterprise credit data, industry reports, and market research data, we can understand the characteristics of the enterprise group and industry trends; these data can provide information such as the credit status, market size, and growth rate of the enterprise group; based on these data, we can set goals for the enterprise group and provide suggestions and guidance on market opportunities, investment directions, etc.; The specific product goals are sales data, market research data, and competitive product analysis data. By analyzing sales data, market research data, and competitive product analysis data, you can understand the market performance, competition, and potential opportunities of the product. Sales data can provide information such as sales, sales volume, and market share of the product. Market research data can understand user demand and feedback for the product. Competitive product analysis data can reveal the product features and market strategies of competitors. Based on these data, you can set product goals and formulate product positioning, marketing, and sales strategies.

[0016] S2. Data collection: According to the target set in step S1, the system log and system database of the big data platform are used. The big data platform adopts Spark, which is a fast and general big data processing engine developed by the Apache Software Foundation. The system database adopts MySQL, which is an open source relational database management system developed by the Swedish MySQL AB company, to collect target dynamic related data in real time. The target dynamic related data includes target public data, target operation data, target dynamic related data and target flow data; The target public data specifically refers to social media data, user comment data, and user survey data. By collecting social media data, such as user posts, reposts, and comments on Weibo, WeChat public accounts, and forums, we can understand the public's discussions and evaluations of the target. User comment data can reveal users' feedback, opinions, and needs for the target; at the same time, conducting user surveys can obtain the public's attitudes, opinions, and behavioral habits towards the target. Based on these data, we can analyze the target's influence, reputation, and demand characteristics among the public; Target operation data specifically includes user behavior data, operation log data, and click record data. By collecting user behavior data, such as browsing records, click records, purchase records, etc., we can understand the user's interest, preference, and behavior characteristics towards the target. At the same time, operation log data can record the user's operation steps and behavior path when using the target. Based on this data, we can analyze indicators such as the target's usage, function preferences, and user experience. The target dynamic association data specifically includes social network data, association attribute data, and keyword analysis data. Through social network data and association attribute data, the association relationship and mutual influence between the target and other entities can be revealed; for example, the attention, dissemination characteristics and influence of the target in the social network can be analyzed; at the same time, the keyword analysis data can be used to understand the public's attention and discussion heat on the target-related topics; based on these data, the social influence, relevance and topic heat of the target can be analyzed; The target flow data specifically includes transaction data, logistics data, and conversion data. By collecting transaction data, logistics data, and conversion data, we can understand the flow of the target in the sales and supply chain links; transaction data can provide information such as the target's sales, sales volume, and market share, logistics data can track the target's inventory, distribution, and logistics routes, and conversion data can record the user's conversion process from browsing, attention to purchase; based on these data, we can analyze indicators such as the target's sales, supply chain efficiency, and user purchasing behavior; S3, according to the data collected in step S2, all the acquired data are classified and marked, and according to the categories of the acquired multiple target data, the multiple categories of data of multiple targets are clustered, that is, a similarity threshold is set, and the categories less than the threshold are judged to have the same attributes; according to the clustering results, the same categories among the multiple targets are found, and the same categories are set as the common interest feature points of the multiple targets; S4. Dynamic interest model establishment: The weighted linear regression ensemble learning method of machine learning is used to learn the cluster data of the common interest feature points of multiple targets, and the effects of different feature attribute data on the common interest feature points of multiple targets are analyzed to obtain the interest value prediction function of the target at different interest points. Then, combined with the current data of the target, the values ​​of the common interest feature points of multiple targets are calculated and sorted to form an interest point vector and complete the establishment of a dynamic interest model.

[0017] Furthermore, in step S2, after collecting the data, the data is preprocessed in the following manner: Through data cleaning and data standardization, the collected data are supplemented with missing values, smoothed noise data, and digitized to achieve the goals of data format standardization, error correction, removal of duplicate data, and data computational processing; through data integration, data from multiple data sources are integrated into one data set. The integrated data set is written in matrix format, with each target dynamic related data as row data and each target dynamic related data collected as column data. For the constructed matrix data, because the data type required for each target is different, it also includes further data screening, where row screening and column screening are used respectively; The row screening method is to set a threshold value and delete the values ​​less than the threshold value, thereby filtering out the data with the lowest relevance. The column screening method is to use an evaluation function, set weights of different target types for each type of data for different types of targets, evaluate and calculate the collected data according to different weights, and obtain a valid data range, retain the data within the range, and delete the data beyond the range.

[0018] Furthermore, the specific method of step S3 is: S31, from the obtained data, select part of the data near the threshold data as the initial cluster center; S32, using a clustering algorithm to cluster similar data in the acquired multi-target data together, that is, similar data of multiple targets are used to jointly form a cluster, and the clustering algorithm generally adopts a K-means algorithm; S33. According to the clustering result, the feature points with the highest frequency of occurrence in the clusters are selected as the common interest feature points of the multiple targets.

[0019] Furthermore, the specific method of step S4 is: S41, obtaining user cluster data: according to the result of step S3, obtaining the user's interest value prediction function by performing machine learning on the cluster data, and further calculating the user's interest value for the point of interest. The interest value prediction function uses the word frequency-inverse document frequency method to calculate the similarity of the content and predict the user's interest level therein; S42, using a weighted linear regression ensemble learning method of machine learning to analyze and process the training set data, thereby obtaining a prediction function of the interest value of the target; S43, using the interest value prediction function obtained by machine learning and combining the current data of the target to calculate the interest values ​​of the multiple targets at the interest points; S44. Various points of interest and their interest values ​​obtained by analyzing the target data are represented by a spatial vector model to complete the model building process. The spatial vector model generally adopts a TF-IDF model.

[0020] Furthermore, it also includes: Update the established dynamic model: according to the selected multi-objective type, set different update cycles respectively. After the dynamic model is obtained, the update cycle starts timing. When the update time is reached, repeat the steps of establishing the dynamic model to rebuild the model.

[0021] The beneficial effect of the present invention is that it can effectively reduce the cost of current big data services and can provide services that meet various needs for more small and medium-sized enterprises.

[0022] The technical solution of the present invention is described in detail below: The present invention mainly comprises the following steps: 1. Set the target type, which includes user target, crowd target with associated attributes, single enterprise target, enterprise group target with associated attributes, and product target; 2. Data collection: According to the target set in step S1, the system log and system database of the big data platform are used to collect target dynamic related data in real time, wherein the target dynamic related data includes target public data, target operation data, target dynamic related data and target flow data; After collecting the data, the data is also preprocessed in the following ways: Through data cleaning and data standardization, the collected data is supplemented with missing values, smoothed with noise data, and digitized to achieve the goals of data format standardization, error correction, removal of duplicate data, and data computability processing; through data integration, data from multiple data sources are integrated into one data set; Write the integrated data set into a matrix format, with each target dynamic related data as row data and each target dynamic related data collected as column data. For the constructed matrix data, because the data type required for each target is different, further data screening is also included, and row screening and column screening are used here respectively; The row screening method is to set a threshold value and delete the values ​​less than the threshold value, thereby filtering out the data with the lowest relevance. The column screening method is to use an evaluation function, set weights of different target types for each type of data for different types of targets, evaluate and calculate the collected data according to different weights, and obtain a valid data range, retain the data within the range, and delete the data beyond the range.

[0023] 3. All acquired data are classified and marked, and the multi-category data of multiple targets are clustered according to the categories of the acquired multiple target data, that is, a similarity threshold is set, and the categories less than the threshold are judged to have the same attributes; according to the clustering results, the same categories among multiple targets are found, and the same categories are set as the common interest feature points of multiple targets; The specific method is: 1) From the acquired data, select some data near the threshold data as the initial clustering center; 2) Using clustering algorithms, similar data in the acquired multi-target data are clustered together, that is, similar data of multiple targets are used to jointly form a cluster; 3) According to the clustering results, select the most frequently appearing feature points in the cluster as the common interest feature points of multiple targets.

[0024] 4. Dynamic interest model establishment: Using the weighted linear regression ensemble learning method of machine learning, the cluster data of the common interest feature points of multiple targets are learned, and the effects of different feature attribute data on the common interest feature points of multiple targets are analyzed to obtain the interest value prediction function of the target at different interest points. Then, combined with the current data of the target, the values ​​of the common interest feature points of multiple targets are calculated and sorted to form an interest point vector and complete the establishment of a dynamic interest model. The specific method is: 1) Obtaining user cluster data: According to the result of step S3, the user's interest value prediction function is obtained by performing machine learning on the cluster data, and the user's interest value for the point of interest is further calculated; 2) Use the weighted linear regression ensemble learning method of machine learning to analyze and process the training set data to obtain the target interest value prediction function; 3) Using the interest value prediction function obtained by machine learning and combining it with the current data of the target, the interest values ​​of multiple targets at the points of interest are calculated; 4) The various points of interest and their interest values ​​obtained by analyzing the target data are represented using a spatial vector model to complete the model building process.

[0025] 1. Development of big data technology: With the rapid development of big data technology, the ability to collect, store and process data has been greatly improved. This provides a technical basis for the implementation of this patent, making it possible to collect, process and analyze large-scale data in real time.

[0026] 2. Mature application of machine learning algorithms: Machine learning algorithms have been widely used in many fields and have achieved remarkable results. This patent uses the weighted linear regression ensemble learning method of machine learning to learn and analyze data, further improving the accuracy and effectiveness of the model. Feasibility of data processing and preprocessing: The data preprocessing methods mentioned in step S2, including data cleaning, standardization, integration, etc., are all mature technologies in the current data processing field. These technologies can ensure the accuracy and consistency of the data and provide a reliable data foundation for subsequent model establishment.

[0027] 3. Application of clustering algorithm: Clustering algorithm is one of the commonly used algorithms in data mining and machine learning, which can cluster similar data together. The clustering method in step S3 clusters multi-class data of multiple targets by setting a similarity threshold, finds the common features between multiple targets, and provides key information for the establishment of a dynamic interest model.

[0028] 4. Dynamic model update mechanism: The patent mentions updating the established dynamic model and setting different update cycles according to different target types. This update mechanism can ensure that the model keeps pace with the times, adapts to changing data and target requirements, and maintains the accuracy and effectiveness of the model.

[0029] In summary, the method proposed in this patent is supported by mature technologies and applications in big data technology, machine learning algorithms, data processing and preprocessing technology, clustering algorithms, and dynamic model update mechanisms, so this patent is feasible.

Claims

1. A method for establishing a dynamic model based on big data technology, characterized in that: The following steps are involved: S1. Setting target types, wherein the target types include user targets, group targets with associated attributes, single enterprise targets, enterprise group targets with associated attributes, and product targets; S2, data collection, according to the target set in step S1, using the system log and system database of the big data platform, real-time collection of target dynamic related data, the target dynamic related data including target public data, target operation data, target dynamic related data and target flow data; S3, according to the data collected in step S2, all the acquired data are classified and marked, and according to the categories of the acquired multiple target data, the multiple categories of data of multiple targets are clustered, that is, a similarity threshold is set, and the categories less than the threshold are judged to have the same attributes; according to the clustering results, the same categories among the multiple targets are found, and the same categories are set as the common interest feature points of the multiple targets; S4. Dynamic interest model is established. The weighted linear regression ensemble learning method of machine learning is used to learn the cluster data of the common interest feature points of multiple targets, and the effect of different feature attribute data on the common interest feature points of multiple targets is analyzed to obtain the interest value prediction function of the target at different interest points. Then, combined with the current data of the target, the values ​​of the common interest feature points of multiple targets are calculated and sorted to form an interest point vector and complete the establishment of the dynamic interest model.

2. The method for establishing a dynamic model based on big data technology according to claim 1, characterized in that: In step S2, after collecting the data, the data is preprocessed in the following manner: Through data cleaning and data standardization, the collected data is supplemented with missing values, smoothed with noise data, and digitized to achieve the goals of data format standardization, error correction, removal of duplicate data, and data computability processing; through data integration, data from multiple data sources are integrated into one data set; Write the integrated data set into a matrix format, use each target dynamic related data as row data, and each target dynamic related data collected as column data. For the constructed matrix data, the data type required for each target is different, and further screening of the data is also included, using row screening and column screening respectively; The row screening method is to set a threshold value and delete the values ​​less than the threshold value, thereby filtering out the data with the lowest relevance. The column screening method is to use an evaluation function, set weights of different target types for each type of data for different types of targets, evaluate and calculate the collected data according to different weights, and obtain a valid data range, retain the data within the range, and delete the data beyond the range.

3. The method for establishing a dynamic model based on big data technology according to claim 2, characterized in that: The specific method of step S3 is: S31, from the obtained data, select part of the data near the threshold data as the initial cluster center; S32, using a clustering algorithm to cluster similar data in the acquired multi-target data, that is, similar data of multiple targets are used to jointly form a cluster; S33. According to the clustering result, the feature points with the highest frequency of occurrence in the clusters are selected as the common interest feature points of the multiple targets.

4. The method for establishing a dynamic model based on big data technology according to claim 3, characterized in that: The specific method of step S4 is: S41, obtaining user cluster data: according to the result of step S3, obtaining the user's interest value prediction function by performing machine learning on the cluster data, and further calculating the user's interest value for the point of interest; S42, using a weighted linear regression ensemble learning method of machine learning to analyze and process the training set data, thereby obtaining a prediction function of the interest value of the target; S43, using the interest value prediction function obtained by machine learning and combining the current data of the target to calculate the interest values ​​of the multiple targets at the interest points; S44. Various points of interest and their interest values ​​obtained by analyzing the target data are represented by a spatial vector model, thereby completing the model building process.

5. The method for establishing a dynamic model based on big data technology according to claim 4, characterized in that: Also includes: The established dynamic model is updated. Different update cycles are set according to the selected multi-objective type. After the dynamic model is obtained, the update cycle starts timing. When the update time is reached, the dynamic model establishment steps are repeated to rebuild the model.