Information processing method and system based on big data mining
By leveraging big data mining technology, the shortcomings of traditional recommendation systems in processing multi-source heterogeneous data and handling dynamic changes in user interests have been addressed. This has enabled highly accurate and efficient personalized recommendations, solved the cold start problem, and improved the system's response speed and scalability.
Patent Information
- Application Number
- CN202511062897.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-11-18
AI Technical Summary
Traditional recommendation systems suffer from inaccurate feature extraction when processing multi-source heterogeneous data, static user profiles cannot adapt to dynamic changes in interests, and a single recommendation algorithm struggles to balance accuracy, diversity, and cold start requirements.
An information processing system based on big data mining is adopted, including data preprocessing, dynamic user profile construction, multimodal content understanding, hybrid recommendation and distributed storage. Combined with dynamic decay model, sliding window technology and online learning mechanism, it can realize real-time accurate modeling and efficient recommendation of multi-source data.
It significantly improves the accuracy and real-time response capability of the recommendation system, increasing the recommendation accuracy by more than 40%, and the coverage of long-tail content to 85%, while maintaining high response speed and scalability in high-concurrency scenarios.
Smart Images
Figure CN120974019A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data mining and information processing technology, specifically to an information processing method and system based on big data mining. Background Technology
[0002] With the rapid development of the Internet and mobile terminals and the popularization of smart terminals, massive amounts of user behavior data, content data and contextual information are constantly accumulating, and various online platforms generate massive amounts of user behavior data and content data every day.
[0003] Traditional recommendation systems currently have certain shortcomings when processing this data. For example, they lack the ability to process multi-source heterogeneous data, resulting in inaccurate feature extraction. Static user profiles cannot adapt to dynamic changes in interests, and a single recommendation algorithm cannot simultaneously meet the requirements of accuracy, diversity, and cold start. These shortcomings lead to certain drawbacks in their use. To address these issues, we propose an information processing method and system based on big data mining. Summary of the Invention
[0004] The purpose of this invention is to provide an information processing method and system based on big data mining to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: an information processing system based on big data mining, the information processing system based on big data mining comprising:
[0006] The data preprocessing module is configured to perform data cleaning, outlier detection, and multi-source data entity alignment.
[0007] The user profile building module uses a dynamic decay model to process historical user behavior data and integrates explicit attributes and implicit features to build a multi-dimensional user profile.
[0008] The content understanding engine extracts semantic features from text, images, and videos using multimodal analysis technology and builds a content association network based on a domain knowledge graph.
[0009] The hybrid recommendation module integrates collaborative filtering algorithms, content similarity calculation, and deep learning models to generate preliminary recommendation results.
[0010] Context-aware adapters adjust recommendation results in real time by combining spatiotemporal context information;
[0011] The feedback optimization module continuously updates the recommendation model parameters through an online learning mechanism. Through the collaborative work of multiple modules, a complete closed loop is achieved from raw data collection to personalized recommendation output, significantly improving the accuracy and real-time response capability of the recommendation system.
[0012] Preferably, the user profile building module includes:
[0013] The behavioral feature extraction unit uses sliding window technology to analyze the user's recent interaction sequence.
[0014] The social network analysis unit mines user community characteristics based on graph computing algorithms.
[0015] The interest decay calculation unit applies an exponential decay function to process historical behavior data. Through a triple feature extraction mechanism, it accurately captures short-term interest fluctuations and long-term preference characteristics of users, improving the timeliness of profile updates by more than 40%.
[0016] Preferably, the content understanding engine includes: a multimodal feature extractor, a knowledge graph construction unit, and a cross-modal alignment module. The multimodal feature extractor includes a CNN image analysis module and an LSTM text understanding module. The knowledge graph construction unit establishes a domain ontology library through entity recognition and relation extraction technologies. The cross-modal alignment module maps features from different modalities to a unified semantic space. Through cross-modal feature alignment technology, the semantic gap problem of heterogeneous data is solved, and the accuracy of content similarity calculation is improved by 35%.
[0017] Preferably, the hybrid recommendation module adopts a cascaded architecture, which includes a recall layer, a ranking layer, and a strategy layer;
[0018] The recall layer runs item-based collaborative filtering and content-based similarity retrieval in parallel. The ranking layer uses a deep neural network model to refine the ranking of the recall results. The strategy layer is responsible for applying diversity control algorithms and cold start handling rules. Through this three-level processing architecture, the coverage of long-tail content is increased to over 85% while ensuring recommendation relevance, effectively solving the cold start problem.
[0019] Preferably, the information processing system based on big data mining further includes a distributed storage module. The distributed storage module is used to persistently store user profile data, content feature vectors, and recommendation model parameters. The storage module adopts a columnar storage structure to organize user behavior data and indexes content features through a vector database. By combining columnar storage with a vector database, the query efficiency of large-scale feature data is improved by 60%, while the storage space occupancy is reduced by 30%.
[0020] Preferably, the distributed storage module consists of a real-time writing submodule, a cache acceleration submodule, and a data sharding submodule;
[0021] The real-time writing submodule uses a log structured merge tree (LSM-Tree) structure to handle high-frequency writes;
[0022] The caching acceleration submodule uses the LRU-K algorithm to manage the memory cache of hot data;
[0023] The data sharding submodule uses a consistent hashing algorithm to dynamically expand storage nodes. Through a three-level storage optimization design, the system can still maintain a 95% request response time of less than 50ms under a million-level QPS pressure.
[0024] An information processing method includes the following specific steps:
[0025] S1. Collect and preprocess user interaction data in real time to generate standardized behavior logs;
[0026] S2. Dynamically update user profiles and calculate the timeliness weight of interest tags;
[0027] S3. Perform multimodal feature extraction and knowledge graph association analysis on the candidate content;
[0028] S4. Generate an initial recommendation list using a hybrid recommendation algorithm;
[0029] S5. Adjust the ranking of recommendation results based on current context parameters;
[0030] S6. Collect user feedback data and update model parameters online.
[0031] Preferably, the dynamic update in S2 includes:
[0032] S2.1 Assign linearly increasing weights to users' recent behaviors;
[0033] S2.2 Apply the exponential decay function to historical behavior data;
[0034] S2.3 When interest drift is detected, profile reconstruction is triggered. Through a dual weight adjustment mechanism, the detection sensitivity of user interest changes is increased by 50%, and the profile update delay is controlled within 5 minutes.
[0035] Preferably, in step S4, an improved Item2Vec algorithm is used to solve the data sparsity problem, an attention mechanism is used to fuse multi-source features, and adversarial training is used to improve the robustness of the model. Through the optimized combination of these three algorithms, the recommendation accuracy can still be maintained at more than 78% in the data sparse scenario, and the model's anti-interference ability is improved by 3 times.
[0036] Compared with existing technologies, the present invention provides an information processing method and system based on big data mining, which has the following beneficial effects:
[0037] 1. This information processing method and system based on big data mining adopts multimodal data fusion and dynamic profile updating. By integrating heterogeneous data from multiple sources such as text, images, and videos, and combining dynamic decay models and sliding window technology, it achieves real-time and accurate modeling of user interests, improving recommendation accuracy by more than 40%, and effectively solving the problem that traditional static profiles cannot capture dynamic changes in user interests.
[0038] 2. This information processing method and system based on big data mining, through an intelligent hybrid recommendation architecture, adopts a cascaded hybrid recommendation framework, and integrates collaborative filtering, content recommendation and deep learning algorithms. While ensuring recommendation accuracy, it significantly improves the coverage rate of long-tail content to 85%, and effectively solves the cold start problem for new users and new items.
[0039] 3. This information processing method and system based on big data mining, with an architecture design based on distributed storage and vector database, supports high-concurrency processing of 100,000+ user behavior events per second. It achieves real-time model updates through an online learning mechanism, enabling the system to maintain high response speed while increasing click-through rate by 28.6%, and has good horizontal scalability. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of the information processing system of the present invention;
[0041] Figure 2 This is a schematic diagram of the information processing method steps of the present invention. Detailed Implementation
[0042] like Figure 1 As shown, the present invention provides a technical solution: an information processing system based on big data mining, the information processing system based on big data mining comprising:
[0043] The data preprocessing module is configured to perform data cleaning, outlier detection, and multi-source data entity alignment.
[0044] The user profile building module uses a dynamic decay model to process historical user behavior data and integrates explicit attributes and implicit features to build a multi-dimensional user profile.
[0045] The content understanding engine extracts semantic features from text, images, and videos using multimodal analysis technology and builds a content association network based on a domain knowledge graph.
[0046] The hybrid recommendation module integrates collaborative filtering algorithms, content similarity calculation, and deep learning models to generate preliminary recommendation results.
[0047] Context-aware adapters adjust recommendation results in real time by combining spatiotemporal context information;
[0048] The feedback optimization module continuously updates the recommendation model parameters through an online learning mechanism. Through the collaborative work of multiple modules, a complete closed loop is achieved from raw data collection to personalized recommendation output, significantly improving the accuracy and real-time response capability of the recommendation system.
[0049] The user profile building module includes:
[0050] The behavioral feature extraction unit uses sliding window technology to analyze the user's recent interaction sequence.
[0051] The social network analysis unit mines user community characteristics based on graph computing algorithms.
[0052] The interest decay calculation unit applies an exponential decay function to process historical behavior data. Through a triple feature extraction mechanism, it accurately captures short-term interest fluctuations and long-term preference characteristics of users, improving the timeliness of profile updates by more than 40%.
[0053] The content understanding engine includes a multimodal feature extractor, a knowledge graph construction unit, and a cross-modal alignment module. The multimodal feature extractor includes a CNN image analysis module and an LSTM text understanding module. The knowledge graph construction unit establishes a domain ontology library through entity recognition and relation extraction technologies. The cross-modal alignment module maps features from different modalities to a unified semantic space. Through cross-modal feature alignment technology, it solves the semantic gap problem of heterogeneous data and improves the accuracy of content similarity calculation by 35%.
[0054] The hybrid recommendation module adopts a cascaded architecture, which includes a recall layer, a ranking layer, and a strategy layer;
[0055] The recall layer runs item-based collaborative filtering and content-based similarity retrieval in parallel. The ranking layer uses a deep neural network model to refine the ranking of the recall results. The strategy layer is responsible for applying diversity control algorithms and cold start handling rules. Through this three-level processing architecture, the coverage of long-tail content is increased to over 85% while ensuring recommendation relevance, effectively solving the cold start problem.
[0056] The information processing system based on big data mining also includes a distributed storage module. The distributed storage module is used to persistently store user profile data, content feature vectors and recommendation model parameters. The storage module uses a columnar storage structure to organize user behavior data and indexes content features through a vector database. By combining columnar storage with a vector database, the query efficiency of large-scale feature data is improved by 60%, while the storage space usage is reduced by 30%.
[0057] The distributed storage module consists of a real-time write submodule, a cache acceleration submodule, and a data sharding submodule;
[0058] The real-time writing submodule uses a log structured merge tree (LSM-Tree) structure to handle high-frequency writes;
[0059] The caching acceleration submodule uses the LRU-K algorithm to manage the memory cache of hot data;
[0060] The data sharding submodule uses a consistent hashing algorithm to dynamically expand storage nodes. Through a three-level storage optimization design, the system can still maintain a 95% request response time of less than 50ms under a million-level QPS pressure.
[0061] like Figure 2 As shown, an information processing method includes the following specific steps:
[0062] S1. Collect and preprocess user interaction data in real time to generate standardized behavior logs;
[0063] S2. Dynamically update user profiles and calculate the timeliness weight of interest tags, where dynamic updates include:
[0064] S2.1 Assign linearly increasing weights to users' recent behaviors;
[0065] S2.2 Apply the exponential decay function to historical behavior data;
[0066] S2.3 When interest drift is detected, profile reconstruction is triggered. Through a dual weight adjustment mechanism, the detection sensitivity of user interest changes is increased by 50%, and the profile update delay is controlled within 5 minutes.
[0067] S3. Perform multimodal feature extraction and knowledge graph association analysis on the candidate content;
[0068] S4. An initial recommendation list is generated through a hybrid recommendation algorithm. The improved Item2Vec algorithm is used to solve the data sparsity problem, the attention mechanism is used to fuse multi-source features, and the robustness of the model is improved through adversarial training. Through the optimization and combination of the three algorithms, the recommendation accuracy can still be maintained at more than 78% in the data sparse scenario, and the model’s anti-interference ability is improved by 3 times.
[0069] S5. Adjust the ranking of recommendation results based on current context parameters;
[0070] S6. Collect user feedback data and update model parameters online.
[0071] The present invention has been described in detail above. However, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, any modifications or improvements that do not depart from the spirit of the present invention are within the scope of protection of the present invention.
Claims
1. An information processing system based on big data mining, characterized in that: The information processing system based on big data mining includes: The data preprocessing module is configured to perform data cleaning, outlier detection, and multi-source data entity alignment. The user profile building module uses a dynamic decay model to process historical user behavior data and integrates explicit attributes and implicit features to build a multi-dimensional user profile. The content understanding engine extracts semantic features from text, images, and videos using multimodal analysis technology and builds a content association network based on a domain knowledge graph. The hybrid recommendation module integrates collaborative filtering algorithms, content similarity calculation, and deep learning models to generate preliminary recommendation results. Context-aware adapters adjust recommendation results in real time by combining spatiotemporal context information; The feedback optimization module continuously updates the recommendation model parameters through an online learning mechanism.
2. The information processing system based on big data mining according to claim 1, characterized in that: The user profile building module includes: The behavioral feature extraction unit uses sliding window technology to analyze the user's recent interaction sequence. The social network analysis unit mines user community characteristics based on graph computing algorithms. The interest decay calculation unit applies an exponential decay function to process historical behavior data.
3. The information processing system based on big data mining according to claim 1, characterized in that: The content understanding engine includes a multimodal feature extractor, a knowledge graph construction unit, and a cross-modal alignment module. The multimodal feature extractor includes a CNN image analysis module and an LSTM text understanding module. The knowledge graph construction unit establishes a domain ontology library through entity recognition and relation extraction technologies. The cross-modal alignment module maps features from different modalities to a unified semantic space.
4. The information processing system based on big data mining according to claim 1, characterized in that: The hybrid recommendation module adopts a cascaded architecture, which includes a recall layer, a ranking layer, and a strategy layer. The recall layer runs item-based collaborative filtering and content-based similarity retrieval in parallel, the ranking layer refines the ranking of the recall results using a deep neural network model, and the strategy layer is responsible for applying diversity control algorithms and cold start processing rules.
5. The information processing system based on big data mining according to claim 1, characterized in that: The information processing system based on big data mining also includes a distributed storage module, which is used to persistently store user profile data, content feature vectors, and recommendation model parameters. The storage module organizes user behavior data using a columnar storage structure and indexes content features through a vector database.
6. The information processing system based on big data mining according to claim 5, characterized in that: The distributed storage module consists of a real-time writing submodule, a cache acceleration submodule, and a data sharding submodule. The real-time writing submodule uses a log structured merge tree (LSM-Tree) structure to handle high-frequency writes; The caching acceleration submodule uses the LRU-K algorithm to manage the memory cache of hot data; The data sharding submodule uses a consistent hashing algorithm to dynamically expand storage nodes.
7. An information processing method, characterized in that: The information processing method, based on the big data mining-based information processing system according to any one of claims 1-6, includes the following specific steps: S1. Collect and preprocess user interaction data in real time to generate standardized behavior logs; S2. Dynamically update user profiles and calculate the timeliness weight of interest tags; S3. Perform multimodal feature extraction and knowledge graph association analysis on the candidate content; S4. Generate an initial recommendation list using a hybrid recommendation algorithm; S5. Adjust the ranking of recommendation results based on current context parameters; S6. Collect user feedback data and update model parameters online.
8. The information processing method according to claim 7, characterized in that: The dynamic update in S2 includes: S2.1 Assign linearly increasing weights to users' recent behaviors; S2.2 Apply the exponential decay function to historical behavior data; S2.3 Trigger profile reconstruction when interest drift is detected.
9. The information processing method according to claim 7, characterized in that: The S4 algorithm uses an improved Item2Vec algorithm to address the data sparsity problem, employs an attention mechanism to fuse multi-source features, and enhances model robustness through adversarial training.