Personalized recommendation method and system for hot events

Abstract

The invention provides a personalized recommendation method and system for hot events, and relates to the technical field of big data analysis, and the method comprises the steps: collecting multi-source real-time data; preprocessing the multi-source real-time data to obtain multi-source real-time preprocessed data; performing weighted fusion on the multi-source real-time preprocessing data to obtain a weighted fusion score; generating a hotspot event based on the weighted fusion score; and retrieving the hot events to generate personalized recommendation contents meeting user requirements. According to the invention, real-time and intelligent generation of the hotspot event is effectively realized, personalized hotspot event recommendation is provided for the user, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a personalized recommendation method and system for hot events. Background Art

[0002] With the rapid development of Internet technology, the speed and breadth of information dissemination have reached unprecedented levels. Online hot events have frequently emerged, becoming an important force influencing public opinion and guiding public attention. Therefore, intelligent discovery and analysis technology of hot events is particularly important. It not only concerns the timeliness of information dissemination, but also directly affects people's accurate grasp of and effective response to public opinion trends.

[0003] The prior art discloses hotspot prediction through the collection and analysis of historical data. This method usually first determines the theme of a hot event, and then conducts detailed analysis and mining in the historical data. This logic of first determining the theme and then analyzing leads to a natural lag in the system's capture of hot events, making it difficult to achieve true real-time generation of hot events. In today's rapidly changing public opinion, users expect to obtain event information immediately, but due to reliance on historical data, they often miss the opportunity to monitor and respond. At the same time, the life cycle of hot events is shortened, and post-analysis methods are difficult to identify potential hotspots in a timely manner. The system's response speed cannot keep up with the speed of information dissemination, making it difficult to meet real-time monitoring needs. Existing hot event analysis technology also uses preset algorithms and model parameters for hotspot prediction and analysis.

[0004] The prior art also discloses hotspot prediction and analysis through preset algorithms and model parameters, and predicts future hotspot trends by pattern matching and time series analysis of historical data. Although this method is effective in dealing with conventional hotspots, it often appears passive when facing emerging and sudden hot events; especially when dealing with brand-new events that have never occurred before, due to the lack of corresponding historical data support, it is difficult for existing models to make accurate predictions quickly. This dependence on past data makes it impossible for the system to adjust and respond in time when faced with a rapidly changing public opinion environment, resulting in the accuracy of the prediction being affected; in addition, in terms of intelligent recommendations, existing algorithms and models usually rely on fixed rules and parameters, and are difficult to dynamically adjust according to the user's industry, company background or specific needs, which results in the recommended hot events may not fully match the user's actual interests or needs, reducing the effectiveness of the recommendation and user experience.

[0005] Therefore, how to generate hot events in real time and intelligently to provide personalized hot event recommendations is an extremely important technical problem to be solved. Summary of the invention

[0006] In order to solve the problem of how the above-mentioned prior art can generate hot events in real time and intelligently to provide personalized hot event recommendations, the present invention proposes a personalized recommendation method and system for hot events, which effectively realizes the real-time and intelligent generation of hot events, provides users with personalized hot event recommendations, and improves user experience.

[0007] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0008] A personalized recommendation method for hot events includes the following steps:

[0009] S1, collect real-time data from multiple sources;

[0010] S2, preprocessing the multi-source real-time data to obtain multi-source real-time preprocessed data;

[0011] S3, performing weighted fusion on the multi-source real-time pre-processed data to obtain a weighted fusion score;

[0012] S4, generating hot events based on the weighted fusion score;

[0013] S5. Search the hot events and generate personalized recommendation content that meets user needs.

[0014] Preferably, the multi-source real-time data includes news website data, social media data, authoritative list data and industry-specific data.

[0015] Preferably, the preprocessing of the multi-source real-time data includes:

[0016] S21, converting the formats of the multi-source real-time data into a unified standard format;

[0017] S22, performing text recognition on multi-source real-time data in a unified standard format to obtain text data;

[0018] S23, performing data cleaning on the text data to obtain cleaned data;

[0019] S24, performing denoising processing on the cleaned data to obtain denoised data;

[0020] S25, using the denoised data as input of a preset large language model, and outputting a high-dimensional semantic vector of the text;

[0021] S26. Generate sentiment information data of the text based on the high-dimensional semantic vector, wherein the sentiment information data is multi-source real-time preprocessed data.

[0022] Preferably, the weighted fusion of the multi-source real-time pre-processed data includes:

[0023] S31, perform authoritative weighted calculation on the multi-source real-time preprocessed data to obtain the multi-source real-time data corresponding to the multi-source real-time preprocessed data

[0024] The authoritative score of the authoritative score is calculated as follows:

[0025]

[0026] Among them, W authority represents the authority score, N represents the number of multi-source real-time data corresponding to the multi-source real-time preprocessed data, and w i represents the weight of the i-th multi-source real-time preprocessed data, D i represents the i-th multi-source real-time preprocessed data;

[0027] S32, weighting the freshness of the multi-source real-time preprocessed data to obtain the freshness score of the multi-source real-time data corresponding to the multi-source real-time preprocessed data. The calculation expression of the freshness score is as follows:

[0028] W fresh =e -λΔt

[0029] Among them, W fresh represents the freshness score, λ represents the decay factor, and Δt represents the time interval for real-time preprocessing of multi-source data;

[0030] S33, weighted fusion is performed on the authority score and the freshness score to obtain a weighted fusion score, and the calculation expression of the weighted fusion score is as follows:

[0031]

[0032] Among them, S total Represents the weighted fusion score.

[0033] Preferably, generating a hot event based on the weighted fusion score includes: comparing the weighted fusion score with a hot event judgment threshold score T, and judging whether the weighted fusion score exceeds the hot event judgment threshold score T; if so, judging that the multi-source real-time data corresponding to the weighted fusion score is a hot event; if not, judging that the multi-source real-time data corresponding to the weighted fusion score is a common event.

[0034] Preferably, the hot event judgment threshold score T is a dynamic threshold score.

[0035] Preferably, searching the hot events to generate personalized recommended content that meets user needs includes:

[0036] S51, constructing an industry knowledge base for the user, training the industry knowledge base, and obtaining a trained industry knowledge base;

[0037] S52, obtaining the industry background of the user, and retrieving industry events related to the industry background of the user from the industry knowledge base according to the industry background of the user;

[0038] S53, matching the industry event with the hot event to obtain a hot event that matches a qualified condition;

[0039] S54: Generate a hot event recommendation list based on the hot events matched by cosine similarity, and use the hot event recommendation list as personalized recommendation content for the user.

[0040] Preferably, the industry events are matched with the hot events to obtain hot events that meet the matching conditions, including:

[0041] S531. Retrieving an industry feature vector of an industry event related to the user's industry background from the industry knowledge base;

[0042] S532. Perform cosine similarity matching on the industry feature vector and the vector of the hot event to obtain the hot events that meet the cosine similarity matching condition.

[0043] Preferably, the method further includes the following steps: monitoring and warning public opinion on the personalized recommended content.

[0044] The present invention also proposes a personalized recommendation system for hot events, including:

[0045] Acquisition module, used to collect real-time data from multiple sources;

[0046] A data processing module, used for preprocessing the multi-source real-time data to obtain multi-source real-time preprocessed data;

[0047] A weighted fusion algorithm module performs weighted fusion on the multi-source real-time pre-processed data to obtain a weighted fusion score;

[0048] An event generation module generates hot events based on the weighted fusion score;

[0049] The personalized recommendation module searches for the hot events and generates personalized recommendation content for users.

[0050] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0051] The present invention proposes a personalized recommendation method and system for hot events. First, multi-source real-time data is preprocessed, and the multi-source real-time preprocessed data is weightedly fused. The purpose is to perform real-time collection and analysis through this real-time data stream processing framework, break the existing logical limitation of first determining the topic and then analyzing, and realize the real-time generation of hot events; and introduce multi-source real-time data as an auxiliary reference. In the process of real-time hot event generation, the dynamic nature of multi-source real-time preprocessed data and the authority of multi-source real-time data are used to quickly identify and generate hot events in the early stage of the event. This process ensures that the capture of hot events is not only fast, but also has high accuracy and authority; further, hot events are retrieved to generate personalized recommendation content that meets user needs. Not only can conventional hot event generation be processed, but also recommendation results can be dynamically optimized according to user needs to ensure that the recommended hot events are highly consistent with user needs, and more accurate hot event recommendations are achieved, thereby improving the effectiveness of recommendations and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram showing a process flow of a personalized recommendation method for hot events proposed in an embodiment of the present invention;

[0053] Figure 2 A schematic diagram showing another process flow of a personalized recommendation method for hot events proposed in an embodiment of the present invention;

[0054] Figure 3 A flowchart showing preprocessing of multi-source real-time data proposed in an embodiment of the present invention;

[0055] Figure 4 A flowchart showing the process of generating personalized recommendation content for a user proposed in an embodiment of the present invention;

[0056] Figure 5 A flowchart of an event matching process proposed in an embodiment of the present invention is shown;

[0057] Figure 6 A structural diagram of a personalized recommendation system for hot events proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;

[0059] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;

[0060] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0061] Example 1

[0062] like Figure 1 and Figure 2 As shown, this embodiment proposes a personalized recommendation method for hot events, including the following steps:

[0063] S1, collect real-time data from multiple sources;

[0064] In S1, multi-source real-time data is collected through web crawlers and API interface calls, and the data capture frequency is optimized through scheduling and load balancing mechanisms to ensure real-time performance; the multi-source real-time data includes news website data, social media data, authoritative list data, and industry-specific data;

[0065] News website data includes news reports and comment data, and extracts key text information related to events; social media data includes hot topics and user discussion data on social platforms; authoritative list data refers to the authoritative list data of major websites, including hot search lists, news rankings, etc. Authoritative list data plays an important role in the authoritative evaluation of hot events; industry-specific data targets specific industries, including specific data sources in the fields of finance and technology, in order to capture important dynamics within the industry; types of multi-source real-time data include text, images, and videos.

[0066] S2, preprocessing the multi-source real-time data to obtain multi-source real-time preprocessed data;

[0067] In S2, see Figure 3 In order to ensure data quality and consistency, the multi-source real-time data is pre-processed, including:

[0068] S21, converting the formats of the multi-source real-time data into a unified standard format;

[0069] In S21, multi-source real-time data in different formats (such as JSON, XML, etc.) are converted into a unified standard format to ensure that all input data is processed in a unified standard format to provide consistency for subsequent analysis.

[0070] S22, performing text recognition on multi-source real-time data in a unified standard format to obtain text data;

[0071] In S22, the system uses OCR technology to perform text recognition on multi-source real-time data in a unified standard format, such as identifying pictures in articles and pictures generated by video frame cutting, and extracting their content as text data. This step ensures that the text information in the image is combined with other parts of the article to provide complete text data.

[0072] S23, performing data cleaning on the text data to obtain cleaned data;

[0073] In S23, after completing the format conversion and OCR recognition, the text data needs to be cleaned to remove invalid, repeated and obvious noise data. Data cleaning ensures the integrity and accuracy of the data and provides high-quality data for denoising processing.

[0074] S24, performing denoising processing on the cleaned data to obtain denoised data;

[0075] In S24, a denoising algorithm is used to denoise the cleaned data, and the remaining low-quality or irrelevant data is filtered out to further improve the overall quality of the data; the denoising process makes the data more reliable and provides a clean data source for subsequent analysis steps.

[0076] S25, using the denoised data as input of a preset large language model, and outputting a high-dimensional semantic vector of the text;

[0077] In S25, the title, text and OCR recognition results of the article in the denoised data are used as inputs of the preset large language model LLM, and the high-dimensional semantic vector of the text is output through the large language model LLM; the high-dimensional semantic vector represents the semantic features of the text, which is convenient for subsequent similarity calculation and semantic analysis; the generation of the high-dimensional semantic vector captures the key semantic structure in the article and provides a semantic basis for subsequent sentiment judgment.

[0078] S26. Generate sentiment information data of the text based on the high-dimensional semantic vector, wherein the sentiment information data is multi-source real-time preprocessed data.

[0079] In S26, the Blue Eagle universal sentiment algorithm is used to perform sentiment analysis on the text content, and the high-dimensional semantic vector is used as the input of the Blue Eagle universal sentiment algorithm. The Blue Eagle universal sentiment algorithm is used to identify the main objects (such as people, events or organizations) in the text, and the sentiment information data of the main objects is determined. The sentiment information data includes sentiment tendency (such as positive, negative or neutral) and sentiment intensity. By combining the sentiment information data with the high-dimensional semantic vector representation, the sentiment characteristics of the text can be more comprehensively understood, and the overall sentiment attitude and public opinion orientation of the article can be obtained.

[0080] During the preprocessing process, parallel computing and batch processing technologies are used. Parallel computing technology is used to accelerate the data processing process; batch processing technology is used to efficiently process large-scale data and reduce delays.

[0081] Parallel computing and batch processing technology: In the preprocessing stage, parallel computing technology is used to accelerate the data processing process. Batch processing technology is used to efficiently process large-scale data and reduce latency.

[0082] S3, performing weighted fusion on the multi-source real-time pre-processed data to obtain a weighted fusion score;

[0083] In S3, in order to generate accurate and authoritative hot events, a multi-level weighted algorithm is designed to calculate the authority, freshness and importance of the data; the weighted fusion of the multi-source real-time pre-processed data includes:

[0084] S31, performing an authority weighted calculation on the multi-source real-time preprocessed data, assigning an authority weight to the multi-source real-time preprocessed data, and obtaining an authority score of the multi-source real-time data corresponding to the multi-source real-time preprocessed data. The calculation expression of the authority score is as follows:

[0085]

[0086] Among them, W authority represents the authority score, N represents the number of multi-source real-time data corresponding to the multi-source real-time preprocessed data, and w i represents the weight of the i-th multi-source real-time preprocessed data, D i represents the i-th multi-source real-time pre-processed data; data with high authority scores will have a greater impact on the generation of hot events.

[0087] S32, weighting the freshness of the multi-source real-time preprocessed data to obtain the freshness score of the multi-source real-time data corresponding to the multi-source real-time preprocessed data. The calculation expression of the freshness score is as follows:

[0088] W fresh =e -λΔt

[0089] Among them, W fresh represents the freshness score, λ represents the decay factor, which controls the rate at which freshness decreases over time, and Δt represents the time interval for real-time preprocessing of data from multiple sources. Data with a high freshness score will receive a higher weight to ensure its importance in generating hot events.

[0090] S33, weighted fusion is performed on the authority score and the freshness score to obtain a weighted fusion score, so as to ensure the priority of the latest data in hotspot generation. The calculation expression of the weighted fusion score is as follows:

[0091]

[0092] Among them, S total Represents the weighted fusion score, which indicates the weight of the data in the generation of hot events. The weighted fusion score will be used for subsequent hot event generation and analysis.

[0093] S4, generating hot events based on the weighted fusion score;

[0094] In S4, the hot event is generated based on the weighted fusion score, including: comparing the weighted fusion score with the hot event judgment threshold score T, judging whether the weighted fusion score exceeds the hot event judgment threshold score T, if so, judging that the multi-source real-time data corresponding to the weighted fusion score is a hot event; if not, judging that the multi-source real-time data corresponding to the weighted fusion score is a common event.

[0095] S5. Search the hot events and generate personalized recommendation content that meets user needs.

[0096] The hot event judgment threshold score T is a dynamic threshold score, which is used to determine whether the data has hot potential. When the weighted fusion score exceeds the dynamic threshold score, the multi-source real-time data corresponding to the weighted fusion score is considered to be a hot event. The dynamic threshold score will be automatically adjusted according to the current public opinion dynamics, context and content characteristics to adapt to different data sources and time periods, including:

[0097] Real-time adjustments based on public opinion dynamics include: automatically adjusting the dynamic threshold score according to fluctuations when monitoring public opinion dynamics in real time; when public opinion fluctuates greatly, such as emergencies or major news, the weighted fusion score in the data source is significantly increased, which will lower the dynamic threshold score and quickly capture related hot events; and when public opinion is stable, the system will increase the dynamic threshold score to avoid interference from noise data and ensure that identified events are more accurate.

[0098] Adjustments based on context and content features include: adjusting the dynamic threshold score according to event type and content features. For example, entertainment events spread quickly and fluctuate greatly, so the system sets a lower dynamic threshold score to ensure that rapidly developing events are captured; while technology events are relatively stable, the system increases the dynamic threshold score to ensure that only truly important events are captured. In addition, further adjustments are made based on emotional intensity and topic popularity. In particular, when negative emotions are detected to be dominant, the dynamic threshold score will be lowered to promptly identify events that may trigger public opinion and improve the sensitivity of monitoring.

[0099] The method further comprises the following steps:

[0100] S6. Public opinion monitoring and early warning are performed on the personalized recommended content. While generating events and event features in real time, sentiment analysis and dynamic threshold adjustment can be used to quickly identify and respond to emergencies dominated by negative emotions. This provides higher public opinion monitoring sensitivity and early warning capabilities in sudden public crises or major news events.

[0101] In this embodiment, multi-source real-time data is first preprocessed and weighted fused. The purpose is to perform real-time collection and analysis through this real-time data stream processing framework, break the existing logical limitation of first determining the topic and then analyzing it, and realize the real-time hot event generation; and introduce multi-source real-time data as an auxiliary reference. In the process of real-time hot event generation, the dynamic nature of multi-source real-time preprocessed data and the authority of multi-source real-time data are used to quickly identify and generate hot events in the early stage of the event. This process ensures that the capture of hot events is not only fast, but also has high accuracy and authority; further, hot events are retrieved to generate personalized recommendation content that meets user needs. It can not only handle conventional hot event generation, but also dynamically optimize recommendation results according to user needs, ensure that the recommended hot events are highly consistent with user needs, achieve more accurate hot event recommendations, and improve the effectiveness of recommendations and user satisfaction.

[0102] Example 2

[0103] See also Figure 4 S5 retrieves the hot events and generates personalized recommendation content that meets user needs, including:

[0104] S51, constructing an industry knowledge base for the user, training the industry knowledge base, and obtaining a trained industry knowledge base;

[0105] In S51, key data in the industry are continuously collected and updated to ensure that the industry knowledge base of the user can reflect the latest industry trends and events, which ensures the real-time and representativeness of the industry knowledge base and makes the recommendation more accurate; the industry knowledge base of the user is constructed, including:

[0106] Data Collection: Collect large amounts of text data related to specific industries and companies from sources such as industry reports, technical documents, market analysis, company news, etc.

[0107] Data preprocessing: The collected text data is cleaned, segmented, and standardized to ensure data quality and consistency.

[0108] Fine-tuning the retriever using the RAG model

[0109] Fine-tuning objective: Fine-tune the retrieval side of the RAG model by using industry-related data so that it can more accurately identify content relevant to the industry and company context.

[0110] The training of the industry knowledge base includes:

[0111] Deep learning models are used to train industry knowledge bases. By analyzing the relationships between different events in the industry, trend evolution and other information, the generalization ability and accuracy of the models are improved. Specifically, the following are included:

[0112] Deep learning model pre-training: Use industry data to pre-train deep learning models to ensure that the deep learning models can grasp key knowledge and event patterns in the industry.

[0113] Regular fine-tuning: As the industry and public opinion change, deep learning models will be regularly fine-tuned to adapt to new hot spots and industry changes.

[0114] Labeling of industry knowledge base: By labeling the data in the industry knowledge base, the deep learning model can identify different types of hot events and their related content, ensuring the high relevance of recommended content.

[0115] S52, obtaining the industry background of the user, and retrieving industry events related to the industry background of the user from the industry knowledge base according to the industry background of the user;

[0116] When a user requests a recommendation for a hot event, industry events related to the user's industry background are retrieved from the industry knowledge base through the following steps:

[0117] Preliminary analysis of industry and company background includes: first analyzing the user's industry background and company-related information to ensure that the recommended hot events are highly relevant to the user's business needs. This analysis step combines the industry knowledge base and company background to accurately screen out hot events that match the user's interests and needs.

[0118] Information retrieval and RAG model application: Based on the results of the preliminary analysis, the system uses the RAG model to generate and recommend hot events. The RAG model extracts events related to the user's background from the industry knowledge base through information retrieval technology, and dynamically generates them based on the current real-time data, thereby providing personalized hot event recommendations that meet the user's interests.

[0119] S53, matching the industry event with the hot event to obtain a hot event that matches a qualified condition;

[0120] In S53, see Figure 5 , the industry events are matched with the hot events, and the hot events that meet the matching conditions include:

[0121] S531. Retrieving an industry feature vector of an industry event related to the user's industry background from the industry knowledge base;

[0122] S532. Perform cosine similarity matching on the industry feature vector and the vector of the hot event to obtain the hot events that meet the cosine similarity matching condition; since the recommendation requirement is relatively high, the cosine similarity is selected as 0.85.

[0123] S54: Generate a hot event recommendation list according to the hot events matched by cosine similarity, and use the hot event recommendation list as the personalized recommendation content.

[0124] In this embodiment, in order to ensure high efficiency and real-time performance in a large-scale data environment, the following optimization measures are adopted:

[0125] Parallel computing is adopted and parallel computing technology is used, especially in weighted calculation and cluster analysis of multi-source real-time pre-processed data. The computing efficiency is greatly improved through multi-threading or distributed computing framework to ensure high efficiency when processing large amounts of data.

[0126] Real-time stream processing uses real-time stream processing technology to manage the real-time inflow and processing of data, ensuring that hot events can be generated at the moment the data arrives. Real-time stream processing technology enables the system to respond quickly, generate hot events in real time and make recommendations. In short, through the real-time collection and weighted processing of multi-source real-time data, the design of intelligent hotspot generation and personalized recommendation, this embodiment effectively solves the shortcomings of the existing technology in terms of real-time and intelligent generation of hotspots, and can efficiently and accurately identify and recommend hot events, providing users with more efficient and authoritative public opinion monitoring and analysis services.

[0127] Example 3

[0128] This embodiment further illustrates the effectiveness of the personalized recommendation method for hot events proposed by the present invention through a specific embodiment.

[0129] Step 1. Implementation of data acquisition and preprocessing module

[0130] In Step 1, the collection of multi-source real-time data includes:

[0131] Data collection from news websites: We use our own crawler framework to collect data (such as news titles, text, and comments) from major domestic and foreign news websites every 30 minutes through a scheduled task module. The crawler framework combines API calls to obtain structured data to ensure data integrity and timeliness.

[0132] Social media data collection: Through the self-developed crawler framework, we integrate the data interfaces of social media platforms such as Twitter API and Weibo API, and use the OAuth authentication mechanism to ensure real-time data capture. The data collection frequency is set to minutes to capture the latest user discussions and hot topics.

[0133] Obtaining authoritative list data: Regularly (every 10 minutes) use the self-developed crawler framework to obtain authoritative list data from major websites, such as Google Trends, Baidu Hot Search List, and Weibo Hot Search List.

[0134] Industry-specific data collection: For the fields of finance, technology, etc., we use our self-developed crawler framework to collect data from websites in specific industries and obtain dynamic information in the industry in real time, including stock quotes, industry news, etc.

[0135] All collected data will be stored in the Elasticsearch full database to ensure efficient query and analysis of the data.

[0136] Step 2. Data preprocessing, including:

[0137] Format conversion: First, the collected data in different formats (such as JSON, XML, etc.) are uniformly converted into DataFrame format. This process is completed using Python's Pandas library to ensure that all data is converted to a standard format and stored in the Elasticsearch full database to facilitate subsequent retrieval and analysis operations. Format conversion is the first step in data processing, providing a unified input format for subsequent data cleaning and denoising.

[0138] OCR recognition: Before data cleaning and denoising, OCR technology is used to identify the images in the article and the images generated by video frame cutting, and their content is extracted as text data. This step ensures that the text information in the image can be combined with the rest of the article, ensuring that a complete data set is provided for subsequent processing.

[0139] Data cleaning: After format conversion and OCR recognition are completed, data cleaning begins. Regular expressions are used to remove HTML tags, invalid characters, and duplicate data to retain key information of the data. For text data, stop words are also deleted and part-of-speech tagging is performed; for image and video data, OpenCV is used for image cleaning. This step ensures the integrity and accuracy of the data and provides high-quality data for subsequent denoising processing.

[0140] Denoising: After data cleaning, the remaining data is further processed using a denoising algorithm. For text data, low-quality, irrelevant information is removed; for image data, OpenCV is used for further denoising to ensure that the image data quality is high enough for subsequent analysis. Denoising ensures that the data source is clean and high-quality, laying a solid foundation for subsequent vector generation and sentiment judgment.

[0141] Vector generation: After cleaning and denoising, a high-dimensional semantic vector is generated using a pre-trained language model based on the title, text, and OCR results of the article in the denoised data. Vector generation captures the key semantic structure in the article and provides a semantic basis for sentiment judgment. The generated semantic vector can be used for subsequent similarity calculation and semantic analysis.

[0142] Sentiment judgment: After generating the semantic vector, the Blue Eagle Universal Sentiment Algorithm is used to perform sentiment analysis on the text content. The algorithm combines the results of the semantic vector to identify the main objects in the text (such as people, events, or organizations) and judge the sentiment tendency (such as positive, negative, or neutral) and sentiment intensity of these objects. Sentiment judgment provides the overall sentiment characteristics and public opinion orientation of the text.

[0143] Step 3. Implementation of multi-level weighted algorithm module, including:

[0144] 3.1 Extract events from real-time data: Cluster the fingerprints of multi-source real-time data in the database at set time intervals, extract the fingerprints with the highest statistical values ​​of the clustering results, extract the basic vectors of the articles corresponding to these fingerprints, and find all articles whose vector differences are within the set threshold range. The set threshold is determined according to the actual application scenario, and vectors with a cosine similarity greater than 0.85 are considered similar.

[0145] 3.2. Extract events from authoritative lists: Regularly collect the latest popular article data from major authoritative lists, and generate vector representations based on the article title, text, and OCR results.

[0146] 3.3 Weighted calculation of event hotspot values: The heat scoring model configures an initial weight for each data source, and the weight is dynamically adjusted based on the historical accuracy and reliability of the data source.

[0147]

[0148] Among them, D1 is a news website, and the weight of W1 is 0.3; D2 is a social media, and the weight of W2 is 0.2; D3 is an authoritative list, and the weight of W3 is 0.5;

[0149] 3.4 Data freshness weighting: The time decay model weights the freshness of real-time data to ensure that the latest data has priority in hotspot generation. The freshness weight score W fresh Decreases over time, the formula is as follows:

[0150] W fresh =e -λΔt

[0151] 3.5 Parameter settings include: the timestamp Δt of the data will be converted and used to calculate the freshness weight, and the decay factor λ is set between 0.1-0.5, which is dynamically adjusted according to different data types; the Δt of the news website is the timestamp of the earliest article published within the event range, and λ is set to 0.2; the Δt of the social media is the timestamp of the earliest article published within the event range, and λ is set to 0.5; the Δt of the authoritative list is the timestamp of the hot event article, and λ is set to 0.1

[0152] 3.6 Comprehensive score calculation: Pandas library is used to implement comprehensive calculation of weighted scores. The authority score and freshness score of the data are multiplied and summarized to get the final score:

[0153]

[0154] These calculation results are saved in the database for subsequent hot event generation modules to call.

[0155] Step 4. Realization of real-time hot event generation, including:

[0156] 4.1. Threshold setting and dynamic adjustment: Assess each event’s potential as a hotspot by assigning a comprehensive score to it. In order to determine which events can be considered hotspots, a dynamic threshold score is set and adjusted in real time based on the current public opinion dynamics and event characteristics.

[0157] Initial threshold setting: Set the dynamic threshold score to 0.7, which is calculated based on the median value of the historical data distribution. The dynamic threshold score is used as a baseline at startup to judge the hotspot potential of an event.

[0158] Sliding window monitoring and adjustment: The sliding window method (window size is 1 hour) is used to monitor multi-source real-time data. The weighted fusion scores of all events are compared with the dynamic threshold scores, and the events whose weighted fusion scores exceed the dynamic threshold scores are extracted and marked as potential hot events.

[0159] Public opinion fluctuation detection: The event data in the sliding window is stored and updated in real time through the Queue library in Python. When a large public opinion fluctuation is detected (such as an emergency or major news), the weighted fusion score increases significantly, and the dynamic threshold score is automatically reduced to capture more relevant hot events in a timely manner.

[0160] Adjustment during the stable period: When public opinion is relatively stable, the dynamic threshold score is appropriately increased to filter out low-heat events and ensure that the generated hot events have higher accuracy and relevance.

[0161] Adjustment based on event characteristics: Dynamically adjust the dynamic threshold score based on the type and content characteristics of the hot event, for example:

[0162] For entertainment events, due to their fast spread and large fluctuations, a lower dynamic threshold score will be set to ensure that relevant hot spots are captured in a timely manner.

[0163] For science and technology events, because they are relatively stable and attract concentrated attention, the dynamic threshold score will be increased to ensure that only truly important events are captured.

[0164] Adjustments are also made based on the content features of the data, such as sentiment intensity and topic popularity. When an event is detected to have a high sentiment intensity, especially when it is dominated by negative emotions, the dynamic threshold score may be lowered to identify and respond to potential hot events earlier.

[0165] 4.2. Generation and storage of hot event vectors, including:

[0166] For hot events that exceed the dynamic threshold score, the fine-tuned RAG model will be used to generate feature vectors to facilitate subsequent personalized recommendations and analysis.

[0167] Feature vector generation: Using the fine-tuned RAG model, feature vector representations are generated for the extracted hot event texts. These vectors can accurately represent the characteristics of hot events, ensuring high relevance during the recommendation process.

[0168] Storage and management: The generated hot event feature vectors will be stored in the event library for the real-time recommendation module to call. The stored feature vector data can also be used for subsequent event analysis, trend tracking, and further model training and optimization.

[0169] Step 5. Implementation of intelligent hotspot generation and personalized recommendation module

[0170] 5.1 Personalized recommendation algorithm, including:

[0171] Construction of industry knowledge base

[0172] Data Collection: Collect large amounts of text data related to specific industries and companies from sources such as industry reports, technical documents, market analysis, company news, etc.

[0173] Data preprocessing: The collected text data is cleaned, segmented, and standardized to ensure data quality and consistency.

[0174] Fine-tuning the retriever using the RAG model

[0175] Fine-tuning objective: Fine-tune the retrieval side of the RAG model by using industry-related data so that it can more accurately identify content relevant to the industry and company context.

[0176] Training process:

[0177] Input: industry text data and its associated labels or descriptions.

[0178] Output: The retrieval side of the fine-tuned RAG model, capable of generating industry-specific high-quality feature vector representations.

[0179] Storage: Store the generated feature vectors in the industry knowledge base for subsequent recommendations.

[0180] 5.2 Personalized Recommendation Process

[0181] Acquisition of user industry vectors;

[0182] User configuration: The user selects his or her own industry background and retrieves feature vectors related to the industry from the industry knowledge base.

[0183] Vector matching and recommendation

[0184] Matching process: Match the user's industry feature vector with the vector of hot events through cosine similarity. In this project, due to the high recommendation requirements, a cosine similarity of 0.85 was selected.

[0185] Recommendation generation: Generate a personalized hot event recommendation list based on events that meet the cosine similarity conditions, and recommend hot events that are most relevant to the user's industry and company background.

[0186] Step 5. Parallel computing and real-time stream processing implementation, including:

[0187] 5.1 Implementation of Parallel Computing

[0188] Multithreaded computing: Use Python's Multiprocessing library to implement multithreaded processing of data weighting and hot event generation to improve computing efficiency. Parallel tasks are divided based on data sources to ensure that each data source is processed independently and efficiently.

[0189] Distributed computing framework: Apache Spark is used for distributed processing of large-scale data, and processing tasks of different data sources are assigned to different nodes in the cluster to accelerate the computing process.

[0190] 5.2 Implementation of Real-time Stream Processing

[0191] Real-time data flow management: The system uses Apache Kafka to build a data flow pipeline to achieve real-time data transmission and processing. Each data source corresponds to a Kafka topic, and the data flows to the processing module through Kafka.

[0192] Stream processing engine: Apache Flink is used as a real-time stream processing engine to perform real-time calculation and analysis on the data incoming from Kafka, ensuring that hot events can be generated and recommended to users immediately when the data arrives.

[0193] Example 4

[0194] See also Figure 6 This embodiment also proposes a personalized recommendation system for hot events, including:

[0195] Acquisition module, used to collect real-time data from multiple sources;

[0196] A data processing module, used for preprocessing the multi-source real-time data to obtain multi-source real-time preprocessed data;

[0197] A weighted fusion algorithm module performs weighted fusion on the multi-source real-time pre-processed data to obtain a weighted fusion score;

[0198] An event generation module generates hot events based on the weighted fusion score;

[0199] The personalized recommendation module searches for the hot events and generates personalized recommendation content for users.

[0200] In this embodiment, multi-source real-time data is first preprocessed and weighted fused. The purpose is to perform real-time collection and analysis through this real-time data stream processing framework, break the existing logical limitation of first determining the topic and then analyzing it, and realize the real-time hot event generation; and introduce multi-source real-time data as an auxiliary reference. In the process of real-time hot event generation, the dynamic nature of multi-source real-time preprocessed data and the authority of multi-source real-time data are used to quickly identify and generate hot events in the early stage of the event. This process ensures that the capture of hot events is not only fast, but also has high accuracy and authority; further, hot events are retrieved to generate personalized recommendation content that meets user needs. It can not only handle conventional hot event generation, but also dynamically optimize recommendation results according to user needs, ensure that the recommended hot events are highly consistent with user needs, achieve more accurate hot event recommendations, and improve the effectiveness of recommendations and user satisfaction.

[0201] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A personalized recommendation method for hot events, characterized in that: The following steps are involved: S1, collect real-time data from multiple sources; S2, preprocessing the multi-source real-time data to obtain multi-source real-time preprocessed data; S3, performing weighted fusion on the multi-source real-time pre-processed data to obtain a weighted fusion score; S4, generating hot events based on the weighted fusion score; S5. Search the hot events and generate personalized recommendation content that meets user needs.

2. The personalized recommendation method for hot events according to claim 1, characterized in that: Multi-source real-time data is collected through web crawlers and API interface calls, and the multi-source real-time data includes news website data, social media data, authoritative list data and industry-specific data.

3. The personalized recommendation method for hot events according to claim 1, characterized in that: The preprocessing of the multi-source real-time data includes: S21, converting the formats of the multi-source real-time data into a unified standard format; S22, performing text recognition on multi-source real-time data in a unified standard format to obtain text data; S23, performing data cleaning on the text data to obtain cleaned data; S24, performing denoising processing on the cleaned data to obtain denoised data; S25, using the denoised data as input of a preset large language model, and outputting a high-dimensional semantic vector of the text; S26. Generate sentiment information data of the text based on the high-dimensional semantic vector, wherein the sentiment information data is multi-source real-time preprocessed data.

4. The personalized recommendation method for hot events according to claim 1, characterized in that: The weighted fusion of the multi-source real-time pre-processed data includes: S31, perform an authority weighted calculation on the multi-source real-time preprocessed data to obtain the authority score of the multi-source real-time data corresponding to the multi-source real-time preprocessed data. The calculation expression of the authority score is as follows: Among them, W authority represents the authority score, N represents the number of multi-source real-time data corresponding to the multi-source real-time preprocessed data, and w i represents the weight of the i-th multi-source real-time preprocessed data, D i represents the i-th multi-source real-time preprocessed data; S32, weighting the freshness of the multi-source real-time preprocessed data to obtain the freshness score of the multi-source real-time data corresponding to the multi-source real-time preprocessed data. The calculation expression of the freshness score is as follows: W fresh =e -λΔt Among them, W fresh represents the freshness score, λ represents the decay factor, and Δt represents the time interval for real-time preprocessing of multi-source data; S33, weighted fusion is performed on the authority score and the freshness score to obtain a weighted fusion score, and the calculation expression of the weighted fusion score is as follows: Among them, S total Represents the weighted fusion score.

5. The personalized recommendation method for hot events according to claim 1, characterized in that: The generating of hot events based on the weighted fusion score includes: comparing the weighted fusion score with a hot event judgment threshold score T, judging whether the weighted fusion score exceeds the hot event judgment threshold score T, and if so, judging that the multi-source real-time data corresponding to the weighted fusion score is a hot event; if not, judging that the multi-source real-time data corresponding to the weighted fusion score is a common event.

6. The personalized recommendation method for hot events according to claim 5, characterized in that: The hot event judgment threshold score T is a dynamic threshold score.

7. The personalized recommendation method for hot events according to claim 1, characterized in that: The step of searching the hot events and generating personalized recommended content that meets user needs includes: S51, constructing an industry knowledge base for the user, training the industry knowledge base, and obtaining a trained industry knowledge base; S52, obtaining the industry background of the user, and retrieving industry events related to the industry background of the user from the industry knowledge base according to the industry background of the user; S53, matching the industry event with the hot event to obtain a hot event that matches a qualified condition; S54: Generate a hot event recommendation list according to the hot events matched by cosine similarity, and use the hot event recommendation list as the personalized recommendation content.

8. The personalized recommendation method for hot events according to claim 7, characterized in that: The industry event is matched with the hot event to obtain hot events that meet the matching conditions, including: S431. Retrieving an industry feature vector of an industry event related to the user's industry background from the industry knowledge base; S432. Perform cosine similarity matching on the industry feature vector and the vector of the hot event to obtain the hot events that meet the cosine similarity matching condition.

9. The personalized recommendation method for hot events according to any one of claims 1 to 8, characterized in that: The following steps are also included: Conduct public opinion monitoring and early warning on the personalized recommended content.

10. A personalized recommendation system for hot events, characterized in that: include: Acquisition module, used to collect real-time data from multiple sources; A data processing module, used for preprocessing the multi-source real-time data to obtain multi-source real-time preprocessed data; A weighted fusion algorithm module performs weighted fusion on the multi-source real-time pre-processed data to obtain a weighted fusion score; An event generation module generates hot events based on the weighted fusion score; The personalized recommendation module searches for the hot events and generates personalized recommendation content for users.

Images