Information consultation service system based on big data analysis and acquisition
By processing data using the Apache Hadoop and Apache Spark frameworks, combined with data mining and machine learning, user profiles are constructed and a time decay factor is introduced. This solves the problems of low data processing efficiency and insufficient recommendation accuracy in existing information consulting systems, and realizes personalized, real-time information recommendation services.
Patent Information
- Application Number
- CN202510907066.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-02
- Publication Date
- 2025-10-31
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing information consulting systems suffer from low data processing efficiency, their recommendation algorithms rely on historical data and fail to capture users' real-time needs in a timely manner, and their user profiles lack in-depth analysis, resulting in low accuracy and relevance of recommendation results.
Data collection, processing, and storage are performed using the Apache Hadoop and Apache Spark distributed computing frameworks. Combined with data mining and machine learning algorithms, user profiling and personalized recommendation algorithms are used. Collaborative filtering and matrix factorization techniques are employed, and time decay factors and popular information penalty factors are introduced to adjust the recommendation results in real time.
It improves data processing efficiency and recommendation accuracy, enabling real-time capture of changes in user needs and providing personalized and accurate information consulting services, avoiding outdated or overly simplistic recommendation results.
Smart Images
Figure CN120873255A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information consulting service technology, and in particular to an information consulting service system based on big data analysis and collection. Background Technology
[0002] With the rapid development of the internet and big data technologies, various information consulting service systems have emerged. These systems collect user data and external information sources, and use data analysis techniques to provide users with personalized information recommendations and services. Information consulting systems are widely used in e-commerce, social media, news recommendations, and other fields, and can provide customized information based on users' behavior, interests, and needs.
[0003] While some information consulting systems in the present technology have adopted big data technology and recommendation algorithms, the following problems still exist: On the one hand, the data collection and processing process is inefficient, and the system cannot process raw data in a timely manner; on the other hand, existing recommendation algorithms rely too heavily on historical data and fail to fully capture users' real-time needs and changes in interests, resulting in low accuracy and relevance of recommendation results; furthermore, many systems lack in-depth analysis when building user profiles, failing to accurately reflect users' potential needs. Therefore, how to improve data processing efficiency and enhance the personalized accuracy of recommendation algorithms has become a key issue that urgently needs to be addressed in the present technology.
[0004] To address the aforementioned issues, this invention provides an information consulting service system based on big data analysis and collection. By using data mining and machine learning algorithms, it can more accurately capture user needs and improve the accuracy of personalized recommendations. Summary of the Invention
[0005] To address the above problems, this invention provides an information consulting service system based on big data analysis and collection, thereby solving the shortcomings of existing technologies in terms of data processing efficiency, accuracy of personalized recommendations, and prediction of user needs.
[0006] To address the aforementioned technical problems, the present invention provides the following technical solution: an information consulting service system based on big data analysis and collection, comprising: The data acquisition module is responsible for acquiring raw data. The raw data includes user data and external data; The user data includes the user's personal information and behavioral data. The personal information includes age, gender, and location. The behavioral data includes browsing history, search records, and click behavior. The external data sources include news websites, forums, and social media platforms, which are used to supplement user needs.
[0007] A data processing and storage module, wherein the data processing and storage module processes and stores raw data through distributed computing framework technology; The distributed computing framework technology mentioned above uses two distributed data processing frameworks: Apache Hadoop and Apache Spark. The Apache Hadoop provides distributed storage and batch processing capabilities, making it suitable for large-scale offline computing tasks. The Apache Spark uses in-memory computing to accelerate data processing speed, making it suitable for real-time computing and machine learning tasks. The Apache Spark is integrated with Apache Hadoop, leveraging Apache Hadoop's storage and resource management capabilities.
[0008] The Apache Hadoop and Apache Spark technologies provide distributed computing capabilities, enabling deduplication, outlier detection, and noise reduction operations in the context of big data. The specific operations of Apache Hadoop and Apache Spark in the deduplication process are as follows: Apache Hadoop's MapReduce framework distributes data across multiple computing nodes. In the Map phase, the data is split and processed in parallel, and in the Reduce phase, identical data is merged, keeping only one copy to achieve deduplication. Apache Spark uses a resilient distributed dataset to represent data and removes duplicate data using the distinct() operation; The specific operations of Apache Hadoop and Apache Spark in outlier detection are as follows: Apache Hadoop's MapReduce is used to perform basic statistical analysis, identify outliers in the data, and use statistical methods to discover and label outlier data points by calculating the mean and standard deviation. Apache Spark uses Z-Score or IQR statistical methods to handle outlier detection tasks in memory, dynamically adjusting outliers in the data stream. The specific operations of Apache Hadoop and Apache Spark in the noise reduction process are as follows: Apache Hadoop removes noisy data through parallelized data cleaning algorithms; Apache Spark removes noise from text data by performing stop word removal and stemming, and removes noise from time-series data by using smoothing filtering algorithms.
[0009] The data analysis and mining module uses data mining techniques to extract effective information from raw data, and further analyzes user data to build user profiles based on data mining techniques. The data mining technology uses K-means clustering to classify user data and identify similar user groups; it uses classification algorithms to predict and categorize user behavior, predicting users' information preferences based on their historical behavior; and it uses A priori analysis to discover hidden correlations in the data and identify behavioral relationships between users. The user profile is established through comprehensive analysis of users' historical behavior, personal information, and interest preference data to create a deep understanding of users' personalized needs; the user profile includes the following: Basic information: age, gender, and geographic location; Behavioral characteristics: browsing history and click behavior; Interests and preferences: reading time and number of clicks; Potential demand forecasting: Based on users' historical behavior, and combined with machine learning algorithms, models are used to predict users' future needs.
[0010] The user profile is modeled by combining clustering and classification analysis. The system forms corresponding categories by capturing users' interests and preferences, identifies users' continuous interest in information through classification analysis, constructs feature engineering to predict their future needs, and makes personalized recommendations. The interest categories are defined by extracting explicit and implicit user feedback. Explicit feedback includes user satisfaction, while implicit feedback includes browsing time, click behavior, favorites, and purchases. User satisfaction includes likes and shares.
[0011] The personalized recommendation module provides information consulting services to users based on data analysis results and through personalized recommendation algorithms.
[0012] Based on the user profile, a user-information matrix is constructed, which includes a data table of user satisfaction with information. Information is recommended to users through similarity calculation. The similarity calculation determines the relevance between users. The similarity is calculated using the Pearson correlation coefficient to identify the user group most similar to the target user, as shown in the formula:
[0013] in, It is information With information Similarity between them It is user u's rating of information i. It is the average rating of user u. It is user v's rating of information i. It is the average rating of user v. It is a set of information that users u and v jointly rate; After calculating the similarity matrix between users and information, the system generates preliminary recommendations based on user-based collaborative filtering and information-based collaborative filtering.
[0014] The user-based collaborative filtering identifies other users similar to the target user and recommends information about the preferences of these similar users, as shown in the following formula:
[0015] in, User Information Predicted score Is with The set of most similar users It is information With information Similarity between them User Information Predicted score and User and Average rating; The information-based collaborative filtering recommends other information similar to those that users have historically liked, as shown in the following formula:
[0016] in, User Information Predicted score User Average rating Is with The set of most similar information It is information With information Similarity between them User Information Predicted score; The recommendation results are further adjusted through matrix factorization. The core objective of matrix factorization is to split the original user-information rating matrix into several low-dimensional latent factor matrices, thereby reducing the sparsity of the data and discovering potential interest patterns. The rating matrix is decomposed using SVD to obtain the product of three matrices, which helps to identify the hidden latent features between users and information, as shown in the following formula:
[0017] in, It is a user-information rating matrix. It is a user latent factor matrix. It is a diagonal matrix containing singular values. It is the information latent factor matrix; Based on SVD, user bias and information bias are added to solve the bias problem in recommendation systems, as shown in the following formula:
[0018] in, User Information Predicted score It is the global average score. and These are the bias terms for user u and information i, respectively. and These are the latent factors for user u and information i, respectively.
[0019] Considering that user interests change over time, a time decay factor is used to reduce the impact of outdated data on recommendation results. By adjusting the recommendation system's emphasis on historical data and adjusting weights based on the user's interaction time with information, the system focuses more on the user's current needs and interests, while ignoring the influence of past behavior, and reflects the user's latest interest changes in real time, as shown in the formula:
[0020] in, It is the time decay factor. It is the attenuation coefficient. It is the current time. It's the scoring time; To avoid over-recommending popular information, a popular information penalty factor is introduced. This factor reduces the recommendation weight of popular information to prevent the recommendation results from becoming too homogeneous, as shown in the formula:
[0021] in, It is a punishment factor for information. It refers to the frequency of occurrence of information h. It is the weight of the penalty factor.
[0022] Compared with the prior art, the beneficial effects of the present invention are: This invention utilizes big data technology for data collection, processing, and storage, collecting large amounts of data from multiple information sources and processing them efficiently to ensure the comprehensiveness and high quality of the data.
[0023] This invention provides users with accurate consulting services through personalized recommendation algorithms based on the analysis of user behavior data, interests and preferences, and historical records. It also optimizes user experience and improves the relevance and accuracy of services through machine learning and data mining techniques.
[0024] This invention employs advanced data analysis and mining methods to extract effective information from raw data and establish user profiles, providing strong support for user demand prediction and recommendations.
[0025] This invention comprehensively analyzes users' personal information, behavioral characteristics, and interests, combining clustering and classification analysis to capture changes in user needs in real time and dynamically adjust recommendation results to better meet users' immediate needs.
[0026] This invention leverages the Apache Hadoop and Apache Spark distributed computing frameworks to efficiently perform data cleaning, deduplication, outlier detection, and noise reduction, ensuring the accuracy and speed of data processing.
[0027] This invention avoids overly simplistic or outdated recommendation results by introducing a time decay factor and a popular information penalty factor, ensuring that the recommendation system can better reflect the user's current interests and needs. Attached Figure Description
[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.
[0029] Figure 1 This is the system architecture diagram of the present invention. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to describe selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] Please refer to Figure 1 This is a schematic diagram of an information consulting service system based on big data analysis and collection provided in an embodiment of the present invention, including: The data acquisition module is responsible for acquiring raw data from various information sources. The raw data includes user data and external data. User personal information includes age, gender, and geographic location; behavioral data includes user browsing history, search records, and click behavior. User data is collected through web interfaces or mobile applications.
[0032] External data is obtained by crawling external systems to supplement user needs. External data comes from news websites, forums, and social platforms. External data crawling uses web crawling technology, and external data is crawled regularly to ensure the timeliness and comprehensiveness of the data.
[0033] A data processing and storage module, wherein the data processing and storage module processes and stores raw data through distributed computing framework technology; The distributed computing framework technology mentioned above uses two distributed data processing frameworks: Apache Hadoop and Apache Spark. The Apache Hadoop provides distributed storage and batch processing capabilities, making it suitable for large-scale offline computing tasks. The Apache Spark uses in-memory computing to accelerate data processing speed, making it suitable for real-time computing and machine learning tasks. The integration of Apache Spark with Apache Hadoop leverages Apache Hadoop's storage and resource management capabilities to improve big data computing efficiency.
[0034] The Apache Hadoop and Apache Spark technologies provide distributed computing capabilities, enabling deduplication, outlier detection, and noise reduction operations in the context of big data. The specific operations of Apache Hadoop and Apache Spark in the deduplication process are as follows: Apache Hadoop's MapReduce framework distributes data across multiple computing nodes. In the Map phase, the data is split and processed in parallel, and in the Reduce phase, identical data is merged, keeping only one copy to achieve deduplication. Apache Spark uses a resilient distributed dataset to represent data and removes duplicate data using the distinct() operation; The specific operations of Apache Hadoop and Apache Spark in outlier detection are as follows: Apache Hadoop's MapReduce is used to perform basic statistical analysis, identify outliers in the data, and use statistical methods to discover and label outlier data points by calculating the mean and standard deviation. Apache Spark uses Z-Score or IQR statistical methods to handle outlier detection tasks in memory, dynamically adjusting outliers in the data stream. The specific operations of Apache Hadoop and Apache Spark in the noise reduction process are as follows: Apache Hadoop removes noisy data through parallelized data cleaning algorithms; Apache Spark removes noise from text data by performing stop word removal and stemming, and removes noise from time-series data by using smoothing filtering algorithms.
[0035] The data analysis and mining module uses data mining techniques to extract effective information from raw data, and further analyzes user data to build user profiles based on data mining techniques. The data mining technology uses K-means clustering to classify user data and identify similar user groups; it uses classification algorithms to predict and categorize user behavior, predicting users' information preferences based on their historical behavior; and it uses A priori analysis to discover hidden correlations in the data and identify behavioral relationships between users. The user profile is established through comprehensive analysis of users' historical behavior, personal information, and interest preference data to create a deep understanding of users' personalized needs; the user profile includes the following: Basic information: age, gender, and geographic location; Behavioral characteristics: browsing history and click behavior; Interests and Preferences: By analyzing user behavior data, we can uncover users' interests and preferences and identify trends in user interest changes over a specific time period. Potential demand forecasting: Based on users' historical behavior, and combined with machine learning algorithms, models are used to predict users' future needs.
[0036] The user profile is modeled by combining clustering and classification analysis. The system forms corresponding categories by capturing users' interests and preferences, identifies users' continuous interest in information through classification analysis, constructs feature engineering to predict their future needs, and makes personalized recommendations. The interest categories are defined by extracting explicit and implicit user feedback. Explicit feedback includes user satisfaction, while implicit feedback includes browsing time, click behavior, favorites, and purchases. User satisfaction includes likes and shares.
[0037] The personalized recommendation module provides information consulting services to users based on data analysis results and through personalized recommendation algorithms.
[0038] Based on the user profile, a user-information matrix is constructed, which includes a data table of user satisfaction with information. Information is recommended to users through similarity calculation. The similarity calculation determines the relevance between users. The similarity is calculated using the Pearson correlation coefficient to identify the user group most similar to the target user, as shown in the formula:
[0039] in, It is information With information Similarity between them It is user u's rating of information i. It is the average rating of user u. It is user v's rating of information i. It is the average rating of user v. It is a set of information that users u and v jointly rate; After calculating the similarity matrix between users and information, the system combines user-based collaborative filtering and information-based collaborative filtering to generate preliminary recommendations.
[0040] The user-based collaborative filtering identifies other users similar to the target user and recommends information about the preferences of these similar users, as shown in the following formula:
[0041] in, User Information Predicted score Is with The set of most similar users It is information With information Similarity between them User Information Predicted score and User and Average rating; The information-based collaborative filtering recommends other information similar to the information that a user has liked in the past by identifying such information, as shown in the following formula:
[0042] in, User Information Predicted score User Average rating Is with The set of most similar information It is information With information Similarity between them User Information Predicted score; The recommendation results are further adjusted through matrix factorization. The core objective of matrix factorization is to split the original user-information rating matrix into several low-dimensional latent factor matrices, thereby reducing the sparsity of the data and discovering potential interest patterns. The rating matrix is decomposed using SVD to obtain the product of three matrices, which helps to identify the hidden latent features between users and information, as shown in the following formula:
[0043] in, It is a user-information rating matrix. It is a user latent factor matrix. It is a diagonal matrix containing singular values. It is the information latent factor matrix; Based on SVD, user bias and information bias are added to solve the bias problem in recommendation systems, as shown in the following formula:
[0044] in, User Information Predicted score It is the global average score. and These are the bias terms for user u and information i, respectively. and These are the latent factors for user u and information i, respectively.
[0045] Considering that user interests change over time, a time decay factor is used to reduce the impact of outdated data on recommendation results. By adjusting the recommendation system's emphasis on historical data and adjusting weights based on the user's interaction time with information, the system focuses more on the user's current needs and interests, while ignoring the influence of past behavior, and reflects the user's latest interest changes in real time, as shown in the formula:
[0046] in, It is the time decay factor. It is the attenuation coefficient. It is the current time. It's the scoring time; To avoid over-recommending popular information, a popular information penalty factor is introduced. This factor reduces the recommendation weight of popular information to prevent the recommendation results from becoming too homogeneous, as shown in the formula:
[0047] in, It is a punishment factor for information. It refers to the frequency of occurrence of information h. It is the weight of the penalty factor.
[0048] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations will be apparent to those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An information consulting service system based on big data analysis and data collection, characterized in that, include: The data acquisition module is responsible for acquiring raw data. A data processing and storage module, wherein the data processing and storage module processes and stores raw data through distributed computing framework technology; The data analysis and mining module uses data mining techniques to extract effective information from raw data, and further analyzes user data to build user profiles based on data mining techniques. The personalized recommendation module provides information consulting services to users based on data analysis results and through personalized recommendation algorithms.
2. The information consulting service system based on big data analysis and data collection according to claim 1, characterized in that: The raw data includes user data and external data; The user data includes the user's personal information and behavioral data. The personal information includes age, gender, and location. The behavioral data includes browsing history, search records, and click behavior. The external data sources include news websites, forums, and social media platforms, which are used to supplement user needs.
3. The information consulting service system based on big data analysis and data collection according to claim 1, characterized in that: The distributed computing framework technology mentioned above uses two distributed data processing frameworks: Apache Hadoop and Apache Spark. The Apache Hadoop provides distributed storage and batch processing capabilities, making it suitable for large-scale offline computing tasks. The Apache Spark uses in-memory computing to accelerate data processing speed, making it suitable for real-time computing and machine learning tasks. The Apache Spark is integrated with Apache Hadoop, leveraging Apache Hadoop's storage and resource management capabilities.
4. The information consulting service system based on big data analysis and collection according to claim 3, characterized in that: The Apache Hadoop and Apache Spark technologies provide distributed computing capabilities, enabling deduplication, outlier detection, and noise reduction operations in the context of big data. The specific operations of Apache Hadoop and Apache Spark in the deduplication process are as follows: Apache Hadoop's MapReduce framework distributes data across multiple computing nodes. In the Map phase, the data is split and processed in parallel, and in the Reduce phase, identical data is merged, keeping only one copy to achieve deduplication. Apache Spark uses a resilient distributed dataset to represent data and removes duplicate data using the distinct() operation; The specific operations of Apache Hadoop and Apache Spark in outlier detection are as follows: Apache Hadoop's MapReduce is used to perform basic statistical analysis, identify outliers in the data, and use statistical methods to discover and label outlier data points by calculating the mean and standard deviation. Apache Spark uses Z-Score or IQR statistical methods to handle outlier detection tasks in memory, dynamically adjusting outliers in the data stream. The specific operations of Apache Hadoop and Apache Spark in the noise reduction process are as follows: Apache Hadoop removes noisy data through parallelized data cleaning algorithms; Apache Spark removes noise from text data by performing stop word removal and stemming, and removes noise from time-series data by using smoothing filtering algorithms.
5. The information consulting service system based on big data analysis and data collection according to claim 1, characterized in that: The data mining technology uses the K-means clustering algorithm to classify user data and identify similar user groups; it also uses a classification algorithm to predict and classify user behavior, and predicts the degree of user preference for information based on the user's historical behavior. By using prior data analysis, hidden correlations can be discovered, revealing behavioral relationships between users. The user profile is established through comprehensive analysis of users' historical behavior, personal information, and interest preference data to create a deep understanding of users' personalized needs; the user profile includes the following: Basic information: age, gender, and geographic location; Behavioral characteristics: browsing history and click behavior; Interests and preferences: reading time and number of clicks; Potential demand forecasting: Based on users' historical behavior, and combined with machine learning algorithms, models are used to predict users' future needs.
6. The information consulting service system based on big data analysis and data collection according to claim 5, characterized in that: The user profile is modeled by combining clustering and classification analysis. The system forms corresponding categories by capturing users' interests and preferences, identifies users' continuous interest in information through classification analysis, constructs feature engineering to predict their future needs, and makes personalized recommendations. The interest categories are defined by extracting explicit and implicit user feedback. Explicit feedback includes user satisfaction, while implicit feedback includes browsing time, click behavior, favorites, and purchases. User satisfaction includes likes and shares.
7. The information consulting service system based on big data analysis and data collection according to claim 6, characterized in that: Based on the user profile, a user-information matrix is constructed, which includes a data table of user satisfaction with information. Information is recommended to users through similarity calculation. The similarity calculation determines the relevance between users. The similarity is calculated using the Pearson correlation coefficient to identify the user group most similar to the target user, as shown in the formula: ； in, It is information With information Similarity between them It is user u's rating of information i. It is the average rating of user u. It is user v's rating of information i. It is the average rating of user v. It is a set of information that users u and v jointly rate; After calculating the similarity matrix between users and information, the system generates preliminary recommendations based on user-based collaborative filtering and information-based collaborative filtering.
8. The information consulting service system based on big data analysis and data collection according to claim 7, characterized in that: The user-based collaborative filtering identifies other users similar to the target user and recommends information about the preferences of these similar users, as shown in the following formula: ； in, User Information Predicted score Is with The set of most similar users It is information With information Similarity between them User Information Predicted score and User and Average rating; The information-based collaborative filtering recommends other information similar to those that users have historically liked, as shown in the following formula: ； in, User Information Predicted score User Average rating Is with The set of most similar information It is information With information Similarity between them User Information Predicted score; The recommendation results are further adjusted through matrix factorization. The core objective of matrix factorization is to split the original user-information rating matrix into several low-dimensional latent factor matrices, thereby reducing the sparsity of the data and discovering potential interest patterns. The rating matrix is decomposed using SVD to obtain the product of three matrices, which helps to identify the hidden latent features between users and information, as shown in the following formula: ； in, It is a user-information rating matrix. It is a user latent factor matrix. It is a diagonal matrix containing singular values. It is the information latent factor matrix; Based on SVD, user bias and information bias are added to solve the bias problem in recommendation systems, as shown in the following formula: ； in, User Information Predicted score It is the global average score. and These are the bias terms for user u and information i, respectively. and These are the latent factors for user u and information i, respectively.
9. The information consulting service system based on big data analysis and data collection according to claim 8, characterized in that: Considering that user interests change over time, a time decay factor is used to reduce the impact of outdated data on recommendation results. By adjusting the recommendation system's emphasis on historical data and adjusting weights based on the user's interaction time with information, the system focuses more on the user's current needs and interests, while ignoring the influence of past behavior, and reflects the user's latest interest changes in real time, as shown in the formula: ； in, It is the time decay factor. It is the attenuation coefficient. It is the current time. It's the scoring time; To avoid over-recommending popular information, a popular information penalty factor is introduced. This factor reduces the recommendation weight of popular information to prevent the recommendation results from becoming too homogeneous, as shown in the formula: ； in, It is a punishment factor for information. It refers to the frequency of occurrence of information h. It is the weight of the penalty factor.