A visual monitoring system for big data management

Abstract

The present invention provides a visualization monitoring system for big data management, and relates to the technical field of big data management. The system comprises: a data acquisition module; a data preprocessing module; a view data module, which is used to convert a data set into a plurality of view data according to a preset feature dimension; a preliminary correlation calculation module, which is used to calculate the preliminary correlation of data items i and j in the view data; a final correlation calculation module, which is used to optimize the preliminary correlation to obtain the adjusted correlation of data items i and j, and calculate the final correlation of data items i and j according to the adjusted correlation of each feature dimension; a data screening module, which is used to compare and screen the final correlation with a preset threshold to obtain a screened data set; a data stream processing module, which is used to obtain real-time data, and adjust the parameters of the data screening module according to the change of the data stream; and a visualization display module. The present invention improves the accurate analysis of public opinion data.

Description

Technical Field

[0001] The present invention relates to the technical field of big data management, and in particular to a visual monitoring system for big data management. Background Art

[0002] With the development of the Internet and social media, the speed and scope of online information dissemination have greatly increased, especially at sensitive moments such as major social events, corporate crises, and government policy releases, online public opinion has shown the characteristics of rapid dissemination. Timely monitoring and effective analysis of public opinion are of great significance to relevant institutions such as government, enterprises, and the media, which can help these institutions to grasp the dynamics of public opinion in a timely manner, respond quickly, and take corresponding measures. However, public opinion data usually comes from diverse sources and is large in scale, involving multiple information sources such as news reports, social media, and forum comments. Data generated by different information sources have different levels of structure and content types, which significantly increases the complexity of public opinion data.

[0003] Existing public opinion monitoring and analysis systems mainly screen and process data through keyword searches, simple sentiment analysis, etc., but these methods often fail to fully explore the potential correlations between data, and are also difficult to cope with the dynamic changes in public opinion hotspots. Especially in the face of multi-source data, the existing system lacks flexibility and adaptability, and cannot make accurate judgments on public opinion hotspots in a timely manner. At the same time, there may be a large amount of irrelevant or noisy information in public opinion data. How to improve the accuracy of public opinion analysis while ensuring data processing efficiency has become a problem that needs to be solved urgently in existing technologies. Summary of the invention

[0004] The purpose of the present invention is to provide a visual monitoring system for big data management, aiming to solve the problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A visual monitoring system for big data management, the system comprising:

[0007] A data acquisition module, used to collect and generate original data sets from multiple data sources, wherein the original data sets include data items of multiple dimensions;

[0008] The data preprocessing module is used to preprocess the original data set to generate a data set. The preprocessing operations include data cleaning, format conversion, missing data filling and noise filtering;

[0009] A view data module, used for converting a data set into a plurality of view data according to a preset feature dimension;

[0010] A preliminary correlation calculation module, used to calculate the preliminary correlation between data items i and j in the view data;

[0011] The final correlation calculation module is used to optimize the preliminary correlation to obtain the adjusted correlation between data items i and j, and calculate the final correlation between data items i and j according to the adjusted correlation of each feature dimension;

[0012] A data screening module is used to screen the data set according to the comparison between the final correlation degree and the preset threshold value to obtain a screened data set;

[0013] The data stream processing module is used to obtain the real-time data of the data acquisition module and the data screening module, and adjust the parameters of the data screening module according to the changes in the data stream to obtain the real-time screening data set and data stream trend results;

[0014] The visualization module is used to display the real-time filtered data sets and data flow trend results in a graphical manner.

[0015] Preferably, the adjustment correlation is calculated by the following formula:

[0016] ×h( ),

[0017] in, is a feature function, which represents the screening condition of the kth feature dimension Preliminary correlation between data items i and j in the view data The adjusted correlation after processing, is the preliminary correlation between data items i and j in the vth feature dimension, v represents the index of the feature value in the kth feature dimension, m is the number of feature dimensions, is the adjustment function, n is the number of data items in the view data, is the influence function of the kth feature dimension, is the filtering condition for the kth feature dimension.

[0018] Preferably, the adjustment function The calculation formula is .

[0019] Preferably, the influence function The calculation formula is ,in, is the adjustment coefficient, for Numerical index in the k-th feature dimension.

[0020] Preferably, the final correlation degree is calculated by the following calculation formula:

[0021] ,in, represents the final association between data items i and j in the view data, is the weight of the kth feature dimension.

[0022] Preferably, the data stream processing module includes a data stream change rate calculation unit, which is used to calculate the change rate of the screening data set when a data stream change is detected; the change rate of the screening data set is calculated by the following calculation formula:

[0023] ,

[0024] Among them, F(t) is the screening data set at time t, Indicates the time interval The rate of change within the filtered data set.

[0025] Preferably, the data stream processing module further includes a parameter adjustment unit, which is used to combine the historical data and the current real-time data stream and adjust the processing parameters in the screening data set through a second calculation formula; the second calculation formula is as follows:

[0026] ,

[0027] Where P(t) is the processing parameter at time t, is the learning rate, which is used to control the amplitude of parameter adjustment. is the adjustment coefficient, which is used to balance the impact of historical data on current adjustments. It is a nonlinear function used to further adjust the processing parameters according to the changing trend of the screening data set.

[0028] Preferably, the nonlinear function The calculation formula is as follows:

[0029] ,in, is the adjustment coefficient, is the sensitivity adjustment factor.

[0030] Preferably, the visualization display module includes a plurality of layout optimization units, and the layout optimization unit selects a display layout by analyzing the priority of the view data in the real-time screening data set; the priority of the view data is calculated by the following formula:

[0031] ,

[0032] in, Indicates the priority of the rth view data, is the weight of the qth data item in the view data, is the eigenvalue of the qth data item in the view data, is the layout function, is the total number of data items.

[0033] Preferably, the layout function The calculation formula is as follows:

[0034] ,

[0035] in, is the mean of the view data, is the adjustment coefficient.

[0036] The above solution of the present invention includes at least the following beneficial effects:

[0037] The public opinion data acquisition module can collect multi-source data from multiple channels, including social media, news websites, forums and other public opinion data sources, to ensure the comprehensiveness of online public opinion information. Through the preprocessing module, the acquired data is cleaned, noise filtered and missing values ​​filled, which can effectively improve the quality of the data and provide a high-quality data foundation for subsequent analysis. The view data module classifies the public opinion data according to its characteristic dimensions and converts the data into view data for visual display, so that analysts can more intuitively grasp the changes in public opinion.

[0038] The preliminary correlation calculation module calculates the preliminary correlation between different public opinion topics and keywords through a specific algorithm, providing a basic reference for public opinion analysis. The final correlation calculation module optimizes the preliminary correlation to ensure that the analysis system can accurately identify the potential correlation in the public opinion information and generate accurate correlation results. The data screening module screens a large amount of public opinion data based on the final correlation to ensure that the analysis focus can be focused on the response to current events or crises. The data stream processing module monitors the real-time changes of public opinion data, and adjusts the data screening results based on real-time public opinion feedback information to ensure that the system can quickly respond to public opinion hotspots. The visualization display module finally displays the screened public opinion information in the form of charts and trend charts, so that analysts can intuitively understand the development trends and changes of public opinion. The system can achieve efficient public opinion monitoring, analysis and feedback processing in a complex and dynamic public opinion environment, providing decision makers with real-time public opinion insights and judgment basis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a visual monitoring system for big data management provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] like Figure 1 FIG. 1 is a flowchart of a visual monitoring system for big data management provided by an embodiment of the present invention, wherein the system includes:

[0042] A data acquisition module, used to collect and generate original data sets from multiple data sources, wherein the original data sets include data items of multiple dimensions;

[0043] The data preprocessing module is used to preprocess the original data set to generate a data set. The preprocessing operations include data cleaning, format conversion, missing data filling and noise filtering;

[0044] A view data module, used for converting a data set into a plurality of view data according to a preset feature dimension;

[0045] A preliminary correlation calculation module, used to calculate the preliminary correlation between data items i and j in the view data;

[0046] The final correlation calculation module is used to optimize the preliminary correlation to obtain the adjusted correlation between data items i and j, and calculate the final correlation between data items i and j according to the adjusted correlation of each feature dimension;

[0047] A data screening module is used to screen the data set according to the comparison between the final correlation degree and the preset threshold value to obtain a screened data set;

[0048] The data stream processing module is used to obtain the real-time data of the data acquisition module and the data screening module, and adjust the parameters of the data screening module according to the changes in the data stream to obtain the real-time screening data set and data stream trend results;

[0049] The visualization display module is used to display the real-time screening data sets and data flow trend results in a graphical manner. The display module supports user-defined configuration options and provides a variety of data display formats to meet the different needs of users when monitoring and analyzing big data.

[0050] In an embodiment of the present invention, the acquisition module of public opinion data can collect multi-source data from multiple channels, including public opinion data sources such as social media, news websites, and forums, to ensure the comprehensiveness of network public opinion information. Through the preprocessing module, the acquired data is cleaned, noise filtered, and missing values ​​are filled, which can effectively improve the quality of the data and provide a high-quality data basis for subsequent analysis. The view data module classifies the public opinion data according to the characteristic dimensions of the public opinion data and converts the data into view data for visual display, so that analysts can more intuitively grasp the changes in public opinion. The preliminary correlation calculation module calculates the preliminary correlation between different public opinion topics and keywords through a specific algorithm to provide a basic reference for public opinion analysis. The final correlation calculation module optimizes the preliminary correlation to ensure that the analysis system can accurately identify the potential correlation in the public opinion information and generate accurate correlation results. The data screening module screens a large amount of public opinion data in combination with the final correlation to ensure that the analysis focus can focus on the response to the current event or crisis. The data stream processing module monitors the real-time changes of the public opinion data, and adjusts the data screening results in combination with the real-time public opinion feedback information to ensure that the system can quickly respond to public opinion hotspots. The visualization module will eventually display the filtered public opinion information in the form of charts and trend charts, so that analysts can intuitively understand the development trends and changes of public opinion. The system can achieve efficient public opinion monitoring, analysis and feedback processing in a complex and dynamic public opinion environment, providing decision makers with real-time public opinion insights and judgment basis.

[0051] In one case of this embodiment, the data acquisition module is responsible for collecting raw data from multiple data sources. The data source can be social media, sensor networks, databases, etc. For example, in a public opinion monitoring system, the data acquisition module collects comments and news updates posted by users from social platforms such as Twitter and Weibo in real time.

[0052] In one case of this embodiment, the data preprocessing module functions to clean and process raw data, unify data formats, and handle missing values ​​or abnormal data. For example, the data preprocessing module unifies the timestamp formats from different data sources into the ISO standard format and removes duplicate entries and blank data in the data.

[0053] In one case of this embodiment, the view data is a subset of the data set divided according to different feature dimensions. For example, the system can generate multiple view data based on user comments according to dimensions such as time, region, and sentiment tendency, so as to perform subsequent multi-dimensional analysis and monitoring.

[0054] The following are the roles of view data in public opinion analysis and other big data management scenarios:

[0055] Simplify the representation of multidimensional data: The main role of view data is to transform complex multidimensional data into an actionable, simplified view. This transformation helps users focus on data of certain feature dimensions or features, rather than processing the entire complex original data set at the same time.

[0056] Example: In public opinion analysis, the original data may contain multiple dimensions such as time, location, emotion, keywords, etc. Through view data, you can generate views about a specific time period or a specific topic, allowing users to analyze the public opinion situation of these dimensions more focused.

[0057] Optimizing data processing efficiency: The generation of view data helps improve the efficiency of data processing, especially when the original data set is very large. By pre-screening and transforming the generated view data, subsequent analysis and processing only needs to be based on these smaller data sets, greatly reducing the computational complexity.

[0058] Example: When processing massive amounts of social media data, the generated view data can contain only specific keywords and sentiment analysis results, avoiding repeated calculations of the entire dataset, thereby improving processing efficiency.

[0059] Support multi-dimensional analysis: View data allows data to be analyzed from multiple dimensions at the same time. For example, different views can be generated for different user behaviors (likes, comments, reposts, etc.), reflecting the impact of these behaviors on public opinion. When visualizing data, view data can be displayed from different dimensions according to user needs, so as to facilitate multi-dimensional comparison and analysis.

[0060] Example: In public opinion analysis, multiple views such as sentiment distribution, topic popularity, and user activity can be generated to help analysts comprehensively judge public opinion trends from multiple dimensions.

[0061] Improve data visualization: View data is particularly important in visualization, as it can provide clearer analysis results and data insights. Each view data can display data information of different dimensions or features separately, so that complex data structures can be presented in a graphical way, which is convenient for users to understand and analyze.

[0062] Example: In the public opinion monitoring system, public opinion sentiment views (proportion charts of positive, negative, and neutral sentiments), topic word cloud views (hot keywords), and time trend charts (public opinion development trends) can be generated. These views help users understand data changes in a more intuitive way.

[0063] Enhanced flexibility in data filtering and screening: View data can be used as the basic unit for filtering and screening, allowing users to flexibly filter and analyze data according to different needs. Users can set filtering conditions based on data items in the view to generate new filtered data sets that meet specific conditions.

[0064] Example: In public opinion analysis, users can filter out public opinion information with negative sentiment through specific view data, or filter out the speeches of highly influential users, so as to locate key public opinion content more quickly.

[0065] Support dynamic adjustment and real-time feedback: View data can be dynamically updated according to changes in real-time data streams, which enables the system to respond to changes in new data in a timely manner and adjust view content in real time. This is especially important for public opinion monitoring, because public opinion changes rapidly and the system needs to be able to dynamically display the latest public opinion information.

[0066] Example: When a hot topic breaks out, the view data can update the discussion heat of key topics in real time, show changes in user participation, and adjust sentiment analysis results in real time based on the arrival of new data.

[0067] Provide multiple data aggregation forms: View data can summarize and generalize complex data through aggregation and summary. For example, data can be aggregated by time period, geographic location, user group, etc., to help users analyze data at a higher level.

[0068] Example: In public opinion analysis, you can generate a regional view to show the distribution of public opinion in different regions, or generate a time series view to show the trend of public opinion over time.

[0069] Support customized analysis for specific scenarios: Different view data can be customized according to specific application scenarios. For example, in crisis public relations management, you can customize a dedicated view of negative public opinion, focusing on showing the changing trends of negative comments and negative emotions, to provide support for management decisions.

[0070] Example: When monitoring a brand crisis, the system can generate a dedicated negative public opinion view, displaying negative news or user complaints related to the brand in real time, helping the public relations team respond quickly.

[0071] These functions are particularly suitable for public opinion analysis, helping users to deeply explore and understand public opinion data from multiple perspectives, different dimensions, and dynamic changes, providing strong support for decision-making and response.

[0072] As a preferred embodiment of the present invention, the adjustment correlation degree is calculated by the following calculation formula:

[0073] ×h( ),

[0074] in, is a feature function, which represents the screening condition of the kth feature dimension Preliminary correlation between data items i and j in the view data The adjusted correlation after processing, is the preliminary correlation between data items i and j in the vth feature dimension, v represents the index of the feature value in the kth feature dimension, m is the number of feature dimensions, is the adjustment function, n is the number of data items in the view data, is the influence function of the kth feature dimension, is the filtering condition for the kth feature dimension.

[0075] In an embodiment of the present invention, the formula can accurately calculate the correlation between public opinion topics. By calculating the data feature dimensions of different public opinion topics, the system can identify the potential connection between different public opinion information. By introducing the adjustment function and the influence function to dynamically adjust the preliminary correlation between data items, it is possible to optimize the data screening results more flexibly and accurately when processing large-scale, multi-dimensional data. To ensure that the potential relationship between different public opinion data can be fully mined. This formula can accurately evaluate the correlation between events or hot spots, improve the depth and breadth of public opinion analysis, and ensure that the analysis results are more valuable for reference. The adjustment function corrects the preliminary correlation between data items based on the number of view data items and the complexity of correlation calculation to avoid calculation errors caused by differences in data scale. The influence function weights the correlation of specific dimensions according to the importance of different feature dimensions to ensure that the influence of specific dimensions is properly reflected when calculating the final correlation. This mechanism improves the adaptability of the system when facing complex data sets, and can flexibly adjust the data screening strategy according to actual conditions, avoiding the defect that a single algorithm in the prior art cannot adapt to diversified data features.

[0076] In one case of this embodiment, the preliminary correlation The calculation formula is as follows:

[0077] ,

[0078] in, and are the eigenvalues ​​of data items i and j on the vth feature dimension, and is the text feature vector of data items i and j. By dividing the data items into three types according to their data types, different processing methods are used to calculate the preliminary correlation between data items i and j, so as to express the relationship between data items i and j more accurately and realistically.

[0079] In one case of this embodiment, the adjustment function is used to modify the correlation between data items according to the size of the data set. For example, when processing public opinion analysis, if the number of comments in a certain period of time suddenly increases, the adjustment function can smoothly adjust the correlation according to the data volume to avoid the system from over-responding to sudden data. It can optimize the changes in the amount of public opinion data. Adjustment function The calculation formula is . This formula can reduce the deviation caused by the uneven distribution of public opinion data by logarithmically adjusting the amount of public opinion data. For some hot public opinions, the amount of data may increase rapidly in a short period of time. At this time, by introducing the logarithmic function, the system can stably handle the data changes of such emergencies and ensure the accuracy of the correlation calculation. This formula can ensure the accuracy and consistency of public opinion analysis results when dealing with public opinion emergencies such as hot events and large-scale online discussions, especially in scenarios where the degree of online discussion changes dramatically, and has high robustness.

[0080] In one case of this embodiment, the influence function is used to control the contribution of different feature dimensions to the adjustment of the correlation. For example, in user behavior analysis, the dimension of user activity may be more important than the user registration time. The influence function can be adjusted by weighting to make the user activity contribute more to the correlation. The calculation formula is ,in, is the adjustment coefficient, for The numerical index of the kth feature dimension. It is used to flexibly adjust the feature dimension of public opinion data. This function adjusts the coefficient , dynamically adjust the influence of different dimensional features in adjusting the correlation calculation. This formula ensures that the system can flexibly handle the influence of feature dimensions according to the changing characteristics of public opinion data, such as the weights of different dimensions (such as news reports, user comments, etc.). Through the influence function, the system can better adapt to complex public opinion data scenarios, ensuring that the accuracy and flexibility of data are improved when analyzing complex, multi-dimensional public opinion events.

[0081] In the public opinion analysis system, different data sources (such as social media, news reports, forums, comments, etc.) often have different characteristic dimensions. Each characteristic dimension can represent an attribute of public opinion data, such as time, location, public opinion sentiment (positive, negative, neutral), source platform (Weibo, WeChat, news websites, etc.), keywords, etc. Through the division of characteristic dimensions, the system can better understand and process public opinion information on different dimensions.

[0082] The following are the screening conditions for the kth feature dimension: For example:

[0083] Feature Dimension: Sentiment Analysis

[0084] Example description: Assume that we are interested in the sentiment dimension of public opinion data , the filter condition in this dimension can include public opinion data with "emotion polarity of negative". Through this filter condition, the system can filter out all comments, news or social media posts that express negative emotions. By filtering negative emotions, analysts can focus on changes in negative public opinion and take timely actions.

[0085] Filters : The polarity value of public opinion sentiment is lower than the set threshold, for example:

[0086] , here Represents the sentiment polarity score of the sentiment analysis model on the public opinion data. Data with negative sentiment will be filtered out.

[0087] Feature Dimension: Keywords

[0088] Example description: Keyword dimension Refers to keywords related to public opinion, such as "accident", "product quality", "leakage", etc. The screening conditions can be public opinion data containing certain specific keywords. Through the screening conditions, the system can identify and analyze public opinion information related to specific topics, thereby helping analysts focus on certain themes or topics.

[0089] Filters : Filter all public opinion data containing the keyword "accident", for example: ,here Represents keywords in public opinion data.

[0090] Feature dimension: time

[0091] Example description: Time dimension It is used to filter public opinion data within a specific time period. The filtering condition can be "public opinion information released in the past 24 hours". Through the filtering of this dimension, the system can focus on the latest public opinion information and help analysts to grasp the real-time dynamics of public opinion in a timely manner.

[0092] Filters : Time range: Public opinion data within the past 24 hours, for example:

[0093] ,here Indicates the timestamp of public opinion data.

[0094] Feature dimension: Geographic location

[0095] Example description: Geographic dimension It can be used to filter public opinion data in a specific area. For example, if we only care about the public opinion information of a certain city (such as area A), the system can find all public opinion data related to area A through the filtering conditions.

[0096] Filters : Public opinion data related to the geographical location "Area A", for example: ,here Represents the geographic location information in public opinion data.

[0097] Feature dimension: source of public opinion

[0098] Example description: Public opinion source dimension Used to filter public opinion information from specific source platforms. For example, by filtering only public opinion data from Weibo, the system can analyze the spread of public opinion on social media platforms.

[0099] Filters : Filter the data whose public opinion source is "Weibo", for example:

[0100] ,here Indicates the source platform of public opinion data.

[0101] From these examples, we can see that As the screening condition of the kth feature dimension, the screening conditions can be set for different feature dimensions of public opinion data (such as sentiment, keywords, time, location, and source platform, etc.). Through these screening conditions, the system can flexibly and efficiently screen out public opinion data that meets the requirements of specific dimensions, helping analysts to analyze and respond to public opinion information more accurately.

[0102] in, The value of is defined as , by performing The target contribution value is determined based on historical data. For example, in the historical data, when the number of times a keyword appears in the keyword feature dimension is greater than or equal to 10 times, the data is The target contribution value reaches the maximum. The value of is 1. The goal is to filter data that meets the feature dimension requirements.

[0103] As a preferred embodiment of the present invention, the final correlation degree is calculated by the following calculation formula:

[0104] ,

[0105] in, is the final correlation between data items i and j in the view data. This value is obtained by comprehensively considering the contributions of different feature dimensions k. It is the output of the data screening module and is used to determine whether data items i and j meet the screening conditions.

[0106] is the weight of the kth feature dimension. This weight value is used to adjust the final correlation of each feature dimension. Different feature dimensions may have different effects on the screening results, so the weights are used to To control the influence of each dimension.

[0107] In the embodiment of the present invention, the formula weights the characteristic dimensions between public opinion topics to optimize the final screening results of public opinion data. , the system can focus more on important dimensions related to public opinion hotspots or emergencies when processing public opinion data. This weight allocation formula ensures that the system can flexibly adjust the analysis strategy, effectively focus on key information when dealing with complex public opinion environments, and improve the accuracy and pertinence of public opinion analysis results. This mechanism ensures that the system can flexibly adjust according to different application scenarios when processing multi-dimensional data, so that the correlation calculation is more in line with actual needs. Compared with the fixed weight allocation method in the prior art, the system can dynamically adjust the weights according to actual data changes and needs, which further improves the flexibility and accuracy of data processing.

[0108] In one case of this embodiment, weighted calculation is used to perform differentiated processing on different dimensions in multidimensional data processing. For example, in a recommendation system, a user's interest tag may be more important than the user's geographical location. Through weighted calculation, the system can make more accurate recommendations based on the user's interests.

[0109] As another preferred embodiment of the present invention, the data stream processing module includes a data stream change rate calculation unit, which is used to calculate the change rate of the screening data set by a first calculation formula when a data stream change is detected; the first calculation formula is as follows:

[0110] ,

[0111] Among them, F(t) is the screening data set at time t, Indicates the time interval The rate of change within the filtered data set.

[0112] In the embodiment of the present invention, the formula can accurately capture the changes in the public opinion data stream and monitor the generation and spread of public opinion hotspots in real time. When the network public opinion data changes frequently, the system can detect the change trend of the data stream through the formula, so as to adjust the public opinion analysis strategy in time. By monitoring the change rate, the system can identify the key nodes in the development of public opinion, quickly respond to the public opinion crisis in the network, and provide real-time feedback and reference basis for emergency response.

[0113] As another preferred embodiment of the present invention, the data stream processing module further includes a parameter adjustment unit, which is used to combine the historical data and the current real-time data stream and adjust the processing parameters in the screening data set through a second calculation formula; the second calculation formula is as follows:

[0114] ,

[0115] Where P(t) is the processing parameter at time t, is the learning rate, which is used to control the amplitude of parameter adjustment. is the adjustment coefficient, which is used to balance the impact of historical data on current adjustments. It is a nonlinear function used to further adjust the processing parameters according to the changing trend of the screening data set.

[0116] In the embodiment of the present invention, the formula adjusts the processing parameters in real time by combining historical public opinion data and the current public opinion change rate, so that the system can adaptively respond to changes in public opinion. and adjustment coefficient The dynamic adjustment of can ensure that the system can quickly make analysis adjustments when facing sudden public opinion, avoiding analysis lags due to sudden changes in data streams. This solution improves the timeliness and accuracy of public opinion processing, enabling the system to effectively respond to public opinion emergencies and provide valuable analysis results. Compared with the lag of parameter adjustment in the existing technology, this method can track changes in data streams in real time, improving the real-time and adaptive capabilities of data processing.

[0117] In one case of this embodiment, a nonlinear function is used to process data flow changes to ensure that processing parameters can be adjusted in time according to data flow fluctuations. For example, in network traffic monitoring, when the traffic suddenly increases, the nonlinear adjustment mechanism will adjust the system's processing parameters according to the traffic fluctuations to prevent the system from crashing due to excessive traffic. The calculation formula is as follows:

[0118] ,in, is the adjustment coefficient, which is used to control the amplitude of nonlinear adjustment. is the sensitivity adjustment coefficient. This function can be obtained through the hyperbolic tangent function Realize nonlinear control of parameter adjustment. Adjustment coefficient and sensitivity factor The introduction of enables the system to smoothly adjust parameters when facing sudden changes in public opinion data, avoiding analysis fluctuations caused by over-adjustment. Through this nonlinear function, the system can accurately control public opinion data, ensure the stability and flexibility of parameter adjustment when processing a large amount of dynamic public opinion data, and avoid extreme deviations in the process of public opinion data processing.

[0119] As another preferred embodiment of the present invention, the visualization display module includes a plurality of layout optimization units, and the layout optimization unit selects a display layout by analyzing the priority of the view data in the real-time screening data set; the priority of the view data is calculated by the following formula:

[0120] ,

[0121] in, Indicates the priority of the rth view data, is the weight of the qth data item in the view data, is the eigenvalue of the qth data item in the view data, is a layout function used to evaluate the relative importance of view data in layout unit i. is the total number of data items.

[0122] In the embodiment of the present invention, the visualization display module optimizes the data display layout through the layout optimization unit, realizes the priority management of data display and the selection of the best layout, so that the system can display the most important data to the user first according to the priority of different data views, thereby improving the analysis efficiency and decision-making ability of the user; the formula proposed by the present invention By weighting the priority of public opinion data, the system can optimize the display layout according to the weight and importance of each piece of public opinion data. Prioritizing the display of important public opinion events or topics can effectively improve the work efficiency of public opinion analysts, allowing them to quickly locate the most important public opinion hotspots. This solution is suitable for displaying a large amount of complex public opinion data, ensuring that the display interface is concise and focused, and optimizing the visualization of public opinion data. This formula combines the weight of the data view and data value , and through the layout function The relative importance of view data in the layout unit is evaluated to make the data display layout more in line with the actual needs of users; compared with the traditional fixed layout method, this layout optimization mechanism can flexibly adjust the display layout, improve the relevance and intuitiveness of data display, especially in the scenario of multi-data source or multi-dimensional data display, the system can significantly improve the data display effect and user experience through intelligent layout optimization, ensuring that users can still quickly and efficiently obtain key information in a complex data environment.

[0123] As another preferred embodiment of the present invention, the layout function The calculation formula is as follows:

[0124] ,

[0125] in, It is used to appropriately scale the data values ​​to avoid large differences in values ​​affecting the display effect. is the mean of the data, used to smooth data changes. It is an adjustment coefficient used to control the response sensitivity of the layout function.

[0126] In the embodiment of the present invention, the layout optimization mechanism is to The calculation method is optimized to achieve flexible adaptation to different user needs and ensure that the system can provide the best data display effect in different scenarios; the layout function is optimized by combining formulas , the formula combines the scaling of data values ​​and the control of response sensitivity, so that the system can flexibly adjust the display layout according to the characteristics of different data values; by introducing the logarithmic scaling of data values , the system can avoid the adverse effects of large numerical differences on the display effect, ensuring the balance and consistency of data display; at the same time, the layout function is also combined with the response sensitivity control function , can be based on the data value The deviation from the mean μ adjusts the sensitivity of the layout, making the data display more in line with the user's expectations and needs; compared with the traditional static layout method, this optimization mechanism significantly improves the system's adaptability to diverse user needs, especially in the face of complex and dynamically changing data environments. The system can ensure the accuracy and intuitiveness of data display by intelligently adjusting the layout, thereby providing users with more personalized and efficient data display services, improving users' analysis experience and decision-making efficiency.

[0127] As another preferred embodiment of the present invention, the visual display module further includes a plurality of display units, and the display units are capable of generating a plurality of types of graphical display interfaces based on the priority of the view data;

[0128] Users can use the interactive functions provided by the interface to independently select and adjust the layout and format of the displayed content in order to conduct comprehensive analysis and monitoring of different types of data.

[0129] In an embodiment of the present invention, the visualization display module generates various types of graphical display interfaces through multiple display units. Users can independently select and adjust the layout and format of the display content through the interactive functions provided by the interface, which significantly improves the flexibility of data analysis and user experience; the display unit can automatically generate the most suitable data display form based on the priority of the view data, ensuring that users can quickly obtain the required information in different data analysis scenarios; at the same time, users can freely adjust the display layout and format through interactive functions according to their own needs, so that data display is more in line with personalized needs, and the efficiency and accuracy of data analysis are improved; compared with the prior art, the display module significantly improves the flexibility and user-friendliness of data display by providing a variety of graphical display interfaces and powerful user interaction functions. In particular, when faced with complex and diverse data analysis tasks, users can independently adjust the display content through intuitive interface operations to ensure the efficiency and accuracy of the data analysis process, thereby providing users with better analysis support and decision-making services, and improving the overall user experience of the data management system.

[0130] As another preferred embodiment of the present invention, the visualization display module also supports multi-dimensional data views, and can simultaneously display multiple data sets from different data sources on the same interface;

[0131] The display module integrates data from different data sources through a data fusion algorithm and presents it in a unified view, so that users can comprehensively analyze the results of various data sources in one view.

[0132] In an embodiment of the present invention, the visualization display module supports multi-dimensional data view display, can simultaneously display multiple data sets from different data sources on the same interface, and integrates the data from different data sources through a data fusion algorithm to present a unified view, ensuring that users can comprehensively analyze the results of various data sources in one view; the data fusion algorithm can match and integrate data from different sources according to the data characteristics of each data source, so that the final displayed view can clearly and accurately reflect the associations and differences between the data sources, thereby improving the comprehensiveness and accuracy of data analysis; the module reduces the operation time and complexity of users switching between different data sources through a unified display view, significantly improving the efficiency of data analysis and user experience; compared with the prior art, the visualization module supporting multi-dimensional data view display provides a more intuitive and comprehensive data analysis tool through an intelligent data fusion algorithm, especially when dealing with complex data analysis tasks across multiple data sources, users can quickly obtain and analyze key information of each data source through a unified view, thereby improving analysis efficiency, ensuring the accuracy and consistency of data analysis results, and providing users with more powerful analysis and decision support.

[0133] As another preferred embodiment of the present invention, the display unit can automatically recommend the most suitable chart type according to the user's preferences and historical operation records;

[0134] It also allows users to quickly adjust the display format based on real-time needs to better understand the data analysis results.

[0135] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A visual monitoring system for big data management, characterized in that: The system comprises: A data acquisition module, used to collect and generate original data sets from multiple data sources, wherein the original data sets include data items of multiple dimensions; The data preprocessing module is used to preprocess the original data set to generate a data set. The preprocessing operations include data cleaning, format conversion, missing data filling and noise filtering; A view data module, used for converting a data set into a plurality of view data according to a preset feature dimension; A preliminary correlation calculation module, used to calculate the preliminary correlation between data items i and j in the view data; The final correlation calculation module is used to optimize the preliminary correlation to obtain the adjusted correlation between data items i and j, and calculate the final correlation between data items i and j according to the adjusted correlation of each feature dimension; A data screening module is used to screen the data set according to the comparison between the final correlation degree and the preset threshold value to obtain a screened data set; The data stream processing module is used to obtain the real-time data of the data acquisition module and the data screening module, and adjust the parameters of the data screening module according to the changes in the data stream to obtain the real-time screening data set and data stream trend results; Visualization display module, used to display the real-time filtered data sets and data flow trend results in a graphical way; Initial correlation The calculation formula is as follows: ， in, and are the eigenvalues ​​of data items i and j on the vth feature dimension, and is the text feature vector of data items i and j; The adjustment correlation is calculated by the following formula: ×h( )， in, is a feature function, which represents the screening condition of the kth feature dimension Preliminary correlation between data items i and j in the view data The adjusted correlation after processing, is the preliminary correlation between data items i and j in the vth feature dimension, v represents the index of the feature value in the kth feature dimension, m is the number of feature dimensions, is the adjustment function, n is the number of data items in the view data, is the influence function of the kth feature dimension, is the screening condition for the kth feature dimension; Adjustment function The calculation formula is ; Influence Function The calculation formula is ,in, is the adjustment coefficient, for Numerical index in the kth feature dimension; The final correlation degree is calculated by the following calculation formula: ,in, represents the final association between data items i and j in the view data, is the weight of the kth feature dimension.

2. A visual monitoring system for big data management according to claim 1, characterized in that: The data stream processing module includes a data stream change rate calculation unit, which is used to calculate the change rate of the screening data set when a data stream change is detected; the change rate of the screening data set is calculated by the following calculation formula: ， Among them, F(t) is the screening data set at time t, Indicates the time interval The rate of change within the filtered data set.

3. A visual monitoring system for big data management according to claim 2, characterized in that: The data stream processing module also includes a parameter adjustment unit, which is used to combine historical data and current real-time data streams and adjust processing parameters in the filtered data set; the adjusted processing parameters are calculated using the following calculation formula: ， Where P(t) is the processing parameter at time t, is the learning rate, is the adjustment coefficient, is a nonlinear function.

4. A visual monitoring system for big data management according to claim 3, characterized in that: Nonlinear functions The calculation formula is as follows: ,in, is the adjustment coefficient, is the sensitivity adjustment factor.

5. A visual monitoring system for big data management according to claim 4, characterized in that: The visualization display module includes multiple layout optimization units. The layout optimization unit selects the display layout by analyzing the priority of the view data in the real-time screening data set. The priority of the view data is calculated by the following formula: ， in, Indicates the priority of the rth view data, is the weight of the qth data item in the view data, is the eigenvalue of the qth data item in the view data, is the layout function, is the total number of data items.

6. A visual monitoring system for big data management according to claim 5, characterized in that: Layout Functions The calculation formula is as follows: ， in, is the mean of the view data, is the adjustment coefficient.

Images