An information collection system based on big data
By introducing technologies such as data source selection, content quality analysis, distributed computing and load balancing in the information collection system, the delay and bottleneck problems in public opinion data processing are solved, and efficient and accurate data processing and system stability are achieved.
Patent Information
- Application Number
- CN202411276655.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-12
AI Technical Summary
The existing information collection system has transmission delays and processing bottlenecks when processing public opinion data, which cannot achieve real-time and accurate screening, and dynamic adjustment of processing tasks is difficult.
A big data-based information collection system is designed, including data source selection module, data filtering module, data preprocessing module, distributed computing module, task allocation module, load balancing module and expansion fault processing module, and automatically select data sources. , content quality analysis, data sharding, multi-node parallel processing, dynamic task allocation and load regulation, fault processing and other technologies to ensure the efficiency and stability of data processing.
It improves the speed and efficiency of data processing, ensures data quality and system load balancing, realizes efficient and accurate collection and processing of public opinion data, and maintains stable operation in a complex and changeable public opinion environment.
Smart Images

Figure CN118796950B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data collection, and in particular to an information collection system based on big data. Background Art
[0002] Big data refers to the amount of data involved that is so large that it cannot be captured, managed, processed, and organized into information that helps companies make more proactive business decisions within a reasonable time through mainstream software tools. Public opinion data refers to information such as user opinions, comments, emotions, and topic discussions collected from various online platforms. With the rapid development of the Internet, the sources of public opinion data are becoming increasingly complex, with huge amounts of data and high real-time requirements.
[0003] However, the existing information collection system has exposed a series of deficiencies in data processing when processing public opinion data. First, the centralized architecture is prone to transmission delays and processing bottlenecks in the process of large-scale data processing, which affects the real-time performance and response speed of the system. Second, it is impossible to accurately screen the importance of system goals and dynamically adjust the task of processing data. Summary of the invention
[0004] The purpose of the present invention is to provide a subject matter that aims to solve the problems mentioned in the background technology.
[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0006] An information collection system based on big data, the system comprising:
[0007] A data source selection module, configured to automatically select a data source according to a first preset indicator to obtain a first data source set, wherein the first preset indicator includes an update frequency, integrity, and credibility of the data source;
[0008] A data filtering module is used to filter the first data source set through a content quality analysis formula to obtain a second data source set; the content quality analysis formula scores the data according to content relevance, content accuracy and completeness, and then weighted sums the scores to obtain a total score, and the data is filtered according to the total score;
[0009] A data preprocessing module, used to preprocess the second data source set to generate a data set, wherein the preprocessing includes data cleaning, data standardization and data conversion;
[0010] A distributed computing module is used to process a data set in real time and generate computing result data. The distributed computing module includes multiple nodes, each of which performs processing tasks for different data in the data set. The nodes transmit and synchronize data through message queues. The distinguishing criteria for the processing tasks include the computing amount and importance of the processing tasks. Each node is allocated different computing capabilities through IoT devices.
[0011] A task allocation module, used to obtain a first task weight according to the remaining computing capacity of each node and the computing amount of the processing task, and allocate the processing task of the data segment to the node according to the first task weight;
[0012] The load balancing module is used to obtain the load balancing weight of each node according to the real-time computing load of each node and the execution efficiency of the processing task, and feed the load balancing weight back to the task allocation module to achieve dynamic adjustment of the distribution of processing tasks;
[0013] The expansion fault processing module is used to monitor the status of the node. When a node fails, the weight of the second task is calculated by the weight calculation formula and fed back to the task allocation module to allocate the task of the node to other healthy nodes.
[0014] The data storage module is used to store the calculation result data. The data storage module includes a distributed database system that can store data in a distributed manner among multiple storage nodes.
[0015] Furthermore, the data preprocessing module includes:
[0016] A data fragmentation unit, used for, before data cleaning and conversion, dividing the data of the second data source set into a plurality of data fragments according to a preset threshold value to obtain a first data set;
[0017] The preset threshold is dynamically adjusted based on the data type and data volume.
[0018] Furthermore, the size of the data segment is calculated by the following calculation formula:
[0019] ,
[0020] in, is the size of the data segment, is the total size of the second data source set, is the computing power of the node, is the preset load factor, is the standard deviation of the data segment, is the correlation factor of the data segment.
[0021] Furthermore, the relevance factor of the data segment Obtained through the following steps:
[0022] Perform feature extraction on each data segment to obtain a feature vector for each data segment;
[0023] Calculate the similarity of the feature vectors to obtain a similarity matrix;
[0024] According to the similarity matrix, the similarity of each data segment is normalized to obtain a similarity score;
[0025] The similarity scores are weighted averaged to finally obtain the relevance factor of the data fragments. .
[0026] Furthermore, the calculation formula of the similarity score is:
[0027] ,
[0028] in, and are the feature vectors of the y-th and u-th data segments, respectively. for and The dot product of and is the modulus of the eigenvector.
[0029] Furthermore, the first task weight is calculated by the following formula:
[0030] ,
[0031] in, is the first task weight, is the remaining computing capacity of the node, is the size of the data segment, To handle the computational workload of the task, is the computing power of the jth node, is the remaining computing capacity of the jth node, and m is the total number of nodes.
[0032] Furthermore, the load balancing weight is calculated by the following formula:
[0033] ,
[0034] in, is the load balancing weight; is the computational load of the node, is the execution efficiency of the corresponding processing task of the node. The execution efficiency is calculated by the ratio of the computing power of the node to the computing amount of the processing task. is the execution progress of the processing task of the kth node. The execution progress is calculated by the ratio of the remaining calculation amount to the calculation amount. is the computational load of the kth node, and m is the total number of nodes.
[0035] Furthermore, the weight of the second task is calculated by the following weight formula:
[0036] ,
[0037] in, is the second task weight, is the computing power of the failed node, is the state of the failed node, is the state exponential function of the faulty node, is the processing task importance of the faulty node, is the processing task importance of the cth node, is the state index of the cth node, is the state of the cth node, is the state exponential function of the cth node, is the computing power of the cth node, and m is the total number of nodes;
[0038] in, , is the constant of node c, is the failure-free operation time of node c.
[0039] Furthermore, the state index of the cth node The calculation formula is as follows:
[0040] ,
[0041] in, is the temperature of node c, is the historical failure rate of node c, which is calculated by the ratio of the number of historical failures to the running time. is the computational load of node c, is the remaining computing capacity of node c, , , and To adjust the weight coefficient, Z is the normalization factor.
[0042] Furthermore, the importance of the processing task of the cth node The calculation formula is as follows:
[0043] ,
[0044] in, To rate based on pre-set system goals, To process the task completion time, To handle the computational workload of the task, , and To adjust the weight coefficient.
[0045] The above solution of the present invention includes at least the following beneficial effects:
[0046] First, the system uses the data source selection module to automatically select the optimal data source based on the first preset indicators such as the update frequency, integrity and credibility of the data source, thereby ensuring the reliability and timeliness of data quality; secondly, through the data filtering module, the system can perform content quality analysis on the selected data source set, further improving the accuracy and relevance of the data. This process ensures that the basic data for subsequent data processing has high credibility and high precision; the data preprocessing module ensures the uniformity and consistency of the data in the process of cleaning, standardizing and converting the data source set, thereby providing high-quality data input for distributed computing.
[0047] The distributed computing module in the system processes different data fragments in parallel through multiple nodes, effectively improving the speed and efficiency of data processing, and realizes data transmission and synchronization through message queues, ensuring the efficiency and stability of the processing process; the system also introduces a task allocation module and a load balancing module, which dynamically adjusts the processing tasks and real-time computing load of the nodes to ensure the reasonable allocation of tasks and the load balancing of the system, thereby optimizing the computing power utilization of the overall system; in addition, the node status is monitored through the expansion fault processing module, and when a node fails, the processing tasks are intelligently allocated to other healthy nodes for continued processing, thereby improving the responsiveness of the system.
[0048] Finally, the data storage module adopts a distributed database system to realize the distributed storage of calculation result data, improving the data storage efficiency and the fault tolerance of the system; through the above technical solutions, the system has demonstrated outstanding technical advantages in improving data processing efficiency, ensuring data quality, achieving load balancing, and node fault handling. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A schematic diagram of the architecture of an information collection system based on big data provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0051] like Figure 1 As shown, an embodiment of the present invention provides a schematic diagram of the architecture of an information collection system based on big data, and the system includes:
[0052] A data source selection module, configured to automatically select a data source according to a first preset indicator to obtain a first data source set, wherein the first preset indicator includes an update frequency, integrity, and credibility of the data source;
[0053] A data filtering module, used to filter the first data source set by a content quality analysis formula to obtain a second data source set;
[0054] A data preprocessing module, used to preprocess the second data source set to generate a data set, wherein the preprocessing includes data cleaning, data standardization and data conversion;
[0055] A distributed computing module is used to process the data set in real time and generate computing result data. The distributed computing module includes multiple nodes, each of which performs processing tasks for different data in the data set. The nodes transmit and synchronize data through message queues.
[0056] The distinguishing criteria of the processing tasks include the computational load and importance of the processing tasks;
[0057] Each node is allocated different computing power through IoT devices. The computing power of a node refers to the processing power of each computing node (such as a server, computer or smart device) in the warehouse system. The computing power is usually determined by the hardware configuration of the node (such as CPU, memory, storage) and the degree of software optimization.
[0058] A task allocation module, used to obtain a first task weight according to the remaining computing capacity of each node and the computing amount of the processing task, and allocate the processing task of the data segment to the node according to the first task weight;
[0059] The load balancing module is used to obtain the load balancing weight of each node according to the real-time computing load of each node and the execution efficiency of the processing task, and feed the load balancing weight back to the task allocation module to achieve dynamic adjustment of the distribution of processing tasks;
[0060] The expansion fault processing module is used to monitor the status of the node. When a node fails, the weight of the second task is calculated by the weight calculation formula and fed back to the task allocation module to allocate the task of the node to other healthy nodes.
[0061] The data storage module is used to store the calculation result data. The data storage module includes a distributed database system that can store data in a distributed manner among multiple storage nodes.
[0062] In the embodiment of the present invention, the system automatically selects the data source through the data source selection module, and can screen out high-quality public opinion data sources from a large number of information sources according to first preset indicators such as update frequency, integrity and credibility, thereby ensuring that the data acquired by the system is timely, accurate and reliable; through the data filtering module, the system uses the content quality analysis formula to further screen and filter the data source, effectively eliminating irrelevant or low-quality public opinion data, thereby improving the accuracy and reliability of data analysis; the data preprocessing module in the system cleans, standardizes and converts the filtered data to generate a public opinion data set with a unified structure, thereby ensuring that multi-source data can be effectively integrated and processed during the analysis process; the distributed computing module is responsible for real-time processing of the public opinion data set. The system can quickly process massive public opinion information through parallel computing of multiple nodes, thereby improving the processing efficiency of the system. It ensures timely response and generation of analysis results when public opinion breaks out; the task allocation module can dynamically calculate task weights and allocate tasks according to the remaining computing power of each node and the computing amount of processing tasks, ensuring the rational use of system resources and efficient task allocation; the load balancing module performs load balancing regulation according to the real-time computing load and task execution efficiency of each node, ensuring that the system can still operate efficiently under high load conditions; the expansion fault handling module can monitor the node status in real time. When a node fails, the system calculates the second task weight and reallocates the tasks of the failed node to other healthy nodes, thereby ensuring the high availability of the system and the continuity of data processing; through the cooperation of these modules, the system can efficiently and accurately collect, analyze and process public opinion data, and ensure that the system maintains stable operation in a complex and changeable public opinion environment.
[0063] Among them, the data source selection module is the core component of the information collection system, which is responsible for automatically screening out the optimal data source from a diverse and potential set of data sources; this module evaluates the data source through a preset indicator system, and these indicators include but are not limited to the update frequency of the data source, data integrity, historical performance of the data, and the credibility of the data.
[0064] Update frequency refers to the speed and real-time nature of data updates at the data source, which can ensure that the data information obtained by the system is the latest, thereby ensuring that the system processes real-time and valuable data; data integrity refers to whether the data source provides comprehensive data content to avoid the impact of missing data on system analysis and decision-making; historical performance evaluation is based on the data source's past performance in similar scenarios to assess its continued reliability; data credibility refers to the authority and accuracy of the data source, ensuring that the data source used by the system is reliable and will not lead to erroneous or misleading results.
[0065] The module dynamically selects the optimal data source through the comprehensive scoring of these indicators, providing high-quality basic input for subsequent data processing; at the same time, the module can automatically adjust the selection criteria according to the actual needs of the system. For example, in scenarios with high real-time requirements, it gives priority to data sources with high update frequency, or in scenarios with high requirements for data accuracy, it gives priority to data sources with high credibility. The automated and intelligent design of the data source selection module greatly reduces human intervention and improves the system's autonomous decision-making ability and efficiency.
[0066] Among them, the data filtering module is an important link in the system's data processing flow, which aims to further control the quality and optimize the data obtained through the data source selection module; this module first conducts a preliminary analysis of the data content in the data source set, evaluates its relevance and accuracy to system requirements, and screens out data that is irrelevant to the task objectives or may cause noise interference; the data filtering module uses a series of algorithms and rules, such as keyword matching, pattern recognition, anomaly detection, etc., to ensure that the filtered data is highly relevant and accurate; keyword matching identifies data through preset key features and retains data fragments related to the task; pattern recognition automatically identifies data that does not conform to the pattern and eliminates it by learning the structure and characteristics of the data; anomaly detection identifies and processes outliers in the data or data that deviates from the normal range to prevent erroneous data from affecting the analysis results of the system.
[0067] During the filtering process, the data filtering module will also perform certain standardization processing on the data to ensure the consistency of the data format so that the subsequent data preprocessing module can be executed smoothly; through these steps, the data filtering module greatly improves the quality of the data and reduces the impact of data redundancy and noise on system performance, thereby providing a clean and tidy basic data set for subsequent data preprocessing and analysis.
[0068] More specifically, the content quality analysis formula can score according to content relevance, content accuracy, and completeness, and then perform weighted summation to obtain a total score, which is used as the basis for content quality analysis.
[0069] Among them, the data preprocessing module is a crucial part of the data processing flow, which is responsible for further cleaning, standardization and conversion of the data filtered out by the data filtering module to ensure the consistency and operability of the data; the cleaning process includes processing missing data, correcting erroneous data, deleting duplicate data and other operations to ensure the integrity and accuracy of the data; missing data processing usually adopts methods such as filling, deletion or interpolation, and the specific choice depends on the importance and degree of missing data; the correction of erroneous data depends on the system's rule base or historical data, and errors are identified and corrected through comparison and verification; the deletion of duplicate data ensures the uniqueness of the data set and avoids redundant calculations and result deviations caused by duplicate data; standardization processing refers to converting data from different sources or in different formats into a unified format and unit to ensure the comparability and consistency of data in subsequent processing.
[0070] Common operations of standardization processing include numerical standardization, date format unification, encoding conversion, etc.; finally, conversion processing mainly adjusts the dimension, structure or representation of data, such as converting unstructured data into structured data, or simplifying complex data types into a form that the system can directly process; through these processes, the data preprocessing module ensures high-quality data input.
[0071] In a preferred embodiment of the present invention, the data preprocessing module comprises:
[0072] The data slicing unit is used to divide the data of the second data source set into a plurality of data segments according to a preset threshold before data cleaning and conversion to obtain a first data set; the preset threshold is dynamically adjusted based on the data type and data volume; the size of the data segment is calculated by the following calculation formula:
[0073] ,
[0074] in, is the size of the data segment, is the total size of the second data source set, is the computing power of the node, is the preset load factor, is the standard deviation of the data segment, is the correlation factor of the data segment.
[0075] In an embodiment of the present invention, the data preprocessing module divides the data in the second data source set into multiple data segments according to a preset threshold value through a data segmentation unit before data cleaning and conversion; the preset threshold value of the data segmentation is dynamically adjusted based on the type and amount of data, thereby ensuring that each data segment can adapt to different processing requirements and improving the data processing efficiency of the system; by dynamically adjusting the threshold value, the module can adapt to public opinion data of different scales and ensure that resources are reasonably utilized during data processing.
[0076] Among them, the size of the data fragment is calculated through a specific formula, which comprehensively considers parameters such as the total size of the data source, the computing power of the node, and the preset load factor; through this formula, the system can reasonably allocate data fragments to different nodes to ensure the optimal utilization of computing resources; the introduction of the standard deviation and correlation factors of the data fragments can further optimize the allocation strategy and improve the system's processing efficiency and accuracy of public opinion data; the system can automatically adjust the size of the data fragment according to the processing power of different nodes, thereby improving the overall computing performance.
[0077] Among them, for the calculation formula :
[0078] Molecular part: through and The product of these two factors can reflect the "workload" required to process data of a specified size under the current computing power, which can make the size of the data fragment proportional to the computing power of the node, thereby ensuring that stronger computing nodes take on more data fragments and optimizing the computing power utilization of the system.
[0079] It is an adjustment to the preset load factor. The change of the cosine function within the range can introduce a small adjustment to the size of the data fragment. Through this adjustment, the size of the data fragment can be fine-tuned according to the load balance of the system to avoid node overload or idleness, which is conducive to dynamic load balancing.
[0080] Will and The square root of the two parts is taken after adding them together to smooth the calculation results and reduce the extreme fluctuations in the calculated values. This smoothing process helps to avoid sharding that is too large or too small and ensures the stability of data processing. Through the form of square roots, it can also prevent the data fragments from being too large due to low computing power, thereby preventing excessive node processing pressure.
[0081] Denominator partial pass: The shard size is adjusted by combining the standard deviation and correlation factor to make the data fragments more uniform during the processing process while ensuring that the correlation between data fragments remains consistent.
[0082] By combining the numerator and denominator, the size of the data segment can be calculated. .
[0083] As another preferred embodiment of the present invention, the correlation factor of the data segment is Obtained through the following steps:
[0084] Perform feature extraction on each data segment to obtain a feature vector for each data segment;
[0085] The similarity of the feature vectors is calculated to obtain a similarity matrix; the similarity calculation formula is:
[0086] ,
[0087] in, and are the feature vectors of the y-th and u-th data segments, respectively. for and The dot product of and is the modulus of the eigenvector;
[0088] According to the similarity matrix, the similarity of each data segment is normalized to obtain a similarity score;
[0089] The similarity scores are weighted averaged to finally obtain the relevance factor of the data fragments. .
[0090] In an embodiment of the present invention, the correlation factors of data segments are obtained through feature extraction, similarity calculation and normalization processing, and the system can optimize the processing according to the similarity between different data segments; through the similarity calculation of feature vectors, the system can accurately evaluate the correlation between different public opinion data segments, and adjust the processing strategy according to the correlation, thereby improving the accuracy of data processing; this process ensures that highly correlated data segments can be processed together, thereby improving the accuracy and effectiveness of the system in analyzing public opinion.
[0091] Among them, the similarity scoring formula evaluates the similarity by calculating the dot product of the feature vectors of different data fragments, and then obtains the similarity score; this formula can accurately evaluate the similarity of public opinion data fragments in the multi-dimensional feature space, thereby optimizing the data processing flow; through the similarity score, the system can more intelligently identify and associate related public opinion events, provide a data basis for subsequent public opinion analysis and prediction, and improve the accuracy and reliability of the analysis.
[0092] As another preferred embodiment of the present invention, the first task weight is calculated by the following formula:
[0093] ,
[0094] in, is the first task weight, is the remaining computing capacity of the node, is the size of the data segment, To handle the computational workload of the task, is the computing power of the jth node, is the remaining computing capacity of the jth node, and m is the total number of nodes.
[0095] In an embodiment of the present invention, through the calculation formula of the first task weight, the system can reasonably calculate the task weight of each node according to the remaining computing power of the node, the size of the data segment and the computing amount of the processing task; this formula ensures that the system takes into account the current status and task requirements of each node when allocating resources, and dynamically adjusts the priority of the task, thereby optimizing the system's computing resource allocation and improving the efficiency of public opinion processing; especially during peak periods, this formula can effectively prevent waste of resources and ensure that important tasks are given priority.
[0096] Furthermore, for the calculation formula :
[0097] Molecular part: Combined with the remaining computing power of the node and the size of the data segment , by adjusting the data segment size and task computing amount and taking the square root, to ensure that the remaining nodes with stronger computing power can take on more or more complex tasks; Reduce the impact of high-computation tasks on shard size. Increase, Approaching zero, it makes it more difficult to assign tasks to less capable nodes.
[0098] Denominator: The remaining computing power of all nodes and computing power After summing the products, a sine operation is performed to obtain the relative quantitative value of the remaining computing power of each node. By summing and squaring these values, an amplified adjustment factor is obtained, which enables the task load to be distributed fairly and efficiently among all nodes.
[0099] This formula takes into account the remaining computing power of the node, the size of the data fragment, and the computational effort of the task, and ultimately generates a weight value. , which is used to determine the priority of task allocation; the numerator ensures that the remaining nodes with strong computing power can be allocated to larger or more complex data fragments first, reducing the burden on weak nodes; the denominator uses the capabilities of all nodes to adjust the task allocation ratio, ensuring load balancing and optimal use of system resources, thereby improving the efficiency and response speed of the overall system.
[0100] As another preferred embodiment of the present invention, the load balancing weight is calculated by the following formula:
[0101] ,
[0102] in, is the load balancing weight; is the computational load of the node, is the execution efficiency of the corresponding processing task of the node. The execution efficiency is calculated by the ratio of the computing power of the node to the computing amount of the processing task. is the execution progress of the processing task of the kth node. The execution progress is calculated by the ratio of the remaining calculation amount to the calculation amount. is the computational load of the kth node, and m is the total number of nodes.
[0103] In an embodiment of the present invention, the load balancing weight is calculated through the computing load, computing power and task execution efficiency of the node, and the system can adjust the load distribution of each node in real time; through this formula, the system can dynamically control the load of each node to ensure that the entire system can still run smoothly under high load conditions; the introduction of load balancing not only optimizes resource utilization, but also enhances the system's ability to respond to sudden public opinions, ensuring that overload or resource idleness is avoided when multiple nodes are calculating in parallel.
[0104] Among them, the calculation formula :
[0105] The numerator part of the formula It combines the current load, computing power, and task execution efficiency of the node; specifically, computing power The square root of reflects the contribution of computing resources, while the task execution efficiency Adjustment is performed using an inverse sine function to ensure that the dynamic execution characteristics of the task are taken into account.
[0106] The denominator is the processing load of all nodes. and execution progress The weighted average of the ... , used to dynamically adjust task allocation in distributed systems to ensure efficient use of system resources and balanced task execution.
[0107] As another preferred embodiment of the present invention, the second task weight is calculated by the following weight formula:
[0108] ,
[0109] in, is the second task weight, is the computing power of the failed node, is the state of the failed node, is the state exponential function of the faulty node, is the processing task importance of the faulty node, is the processing task importance of the cth node, is the state index of the cth node, is the state of the cth node, is the state exponential function of the cth node, is the computing power of the cth node, and m is the total number of nodes;
[0110] in, , is the constant of node c, is the failure-free operation time of node c.
[0111] In an embodiment of the present invention, the second task weight is calculated by using parameters such as the state index function of the faulty node, the importance of processing the task, and the state index of the healthy node; this formula can quickly evaluate the importance of the task and reallocate the task when a node fails, thereby ensuring that the system can still efficiently process public opinion data in the event of a node failure; through the calculation of the state index, the system can give priority to nodes with higher health to allocate important tasks, thereby improving the fault tolerance and stability of the system.
[0112] Among them, for the calculation formula :
[0113] Molecular part Combined computing power of failed nodes , Fault Status Index and the importance of the task ; Through the combination of exponential and logarithmic functions, it is ensured that tasks can be quickly transferred to other nodes when the fault state is serious; the numerator result is processed by three square roots to further smooth the weight calculation and prevent extreme situations from causing unbalanced task distribution.
[0114] Denominator It is the weighted sum of the computing power, status, task importance and status index of all healthy nodes. This part ensures that after a failure occurs, tasks can be fairly redistributed to the most suitable nodes, minimizing the fluctuation of system performance and the risk of interruption of task execution. Formula Finally, a weight value is provided to guide the system to redistribute tasks when a failure occurs, so as to maintain the stability and efficiency of the overall system.
[0115] in, It emphasizes that when a node fails, special attention should be paid to task redistribution. Therefore, the fault state index function is usually used to amplify the impact of the failure to ensure that the faulty node can be effectively avoided in the weight distribution. and The calculation formula is the same as; It is used to measure the severity of the node's failure at the current moment, or the possibility or risk of the node being in a failure state; It is more comprehensive and smooth, reflecting the overall health status of each node in daily operation, thus helping the system optimize resource allocation.
[0116] Furthermore, the state index of the cth node The calculation formula is as follows:
[0117] ,
[0118] in, is the temperature of node c, is the historical failure rate of node c, which is calculated by the ratio of the number of historical failures to the running time. is the computational load of node c, is the remaining computing capacity of node c, , , and To adjust the weight coefficient, Z is the normalization factor.
[0119] Specifically, the node status index is calculated comprehensively through parameters such as the node's temperature, historical failure rate, computing load, and remaining computing power. The system can monitor the health status of each node in real time. The formula can issue an early warning before a problem occurs at the node and adjust the task allocation strategy in a timely manner, thereby reducing the risk of system failure. By introducing the node status index, the system can maintain efficient operation during the public opinion processing process and improve the long-term stability of the system.
[0120] Furthermore, the importance of the processing task of the cth node The calculation formula is as follows:
[0121] ,
[0122] in, To prioritize the processing tasks, Rating based on preset system goals, To process the task completion time, To handle the computational workload of the task, , and To adjust the weight coefficient.
[0123] Specifically, the importance of processing tasks is calculated through the task priority, completion time and computing amount. This formula ensures that the system can sort tasks according to the urgency and complexity of public opinion events. Through this formula, the system can reasonably allocate computing resources and give priority to tasks that are critical to public opinion monitoring and analysis, thereby improving the system's response speed and processing capabilities to emergencies and hot public opinions.
[0124] The priority of processing tasks is rated based on preset system goals. Specifically, the system goals of public opinion data processing usually include: timely capturing hot events, real-time monitoring of public opinion changes, warning of potential crises, and providing public opinion analysis reports; rating is based on preset rating standards.
[0125] To achieve these goals, the system dynamically adjusts the priority of tasks based on the urgency of different tasks, their impact on the system as a whole, and the complexity of task processing.
[0126] Priority under the goal of capturing hot events: When the system's goal is to capture and respond to hot events in a timely manner, the task priority will be set based on the speed of event propagation, social attention, and media coverage; for example, if a topic triggers a large number of user discussions on social media in a short period of time, and the emotions of participating users show an extreme trend, the system will prioritize tasks related to the topic (such as data collection, sentiment analysis, trend prediction, etc.); at this time, the priority calculation formula will take into account the user participation, topic popularity, and spread of the event, and give priority to hot topics that may cause a wide range of impacts.
[0127] Practical application: When the system detects that a social event (such as breaking news, public crisis, etc.) has sparked widespread discussion on social media, the system will set the public opinion data processing task of the event to the highest priority to ensure that the relevant data can be collected, analyzed and fed back to relevant decision makers as soon as possible.
[0128] Priority under the crisis warning goal: If the system's goal is to warn of potential public opinion crises, the priority of the task will be determined based on the risk level of the event, the negative sentiment index of public opinion, and the similarity with historical crisis events; when the system identifies that an event may evolve into a public opinion crisis, related public opinion analysis tasks (such as negative comment sentiment analysis, user behavior monitoring, etc.) will be given a higher priority.
[0129] Practical application: If a company's product encounters negative reviews and these reviews spread rapidly, the system will prioritize the public opinion analysis task of the incident. By quickly analyzing the propagation pattern, emotional tendency and influence of negative comments, the system will promptly issue crisis warnings to the company to help the company respond in advance.
[0130] Priority under the goal of generating a public opinion report: In the scenario where a public opinion analysis report needs to be generated, the system's goal is to provide a comprehensive public opinion analysis result for decision makers to refer to; the task priority setting will be based on the complexity of data processing, the time limit for generating the report, and the processing requirements for various types of data (such as text, images, and videos).
[0131] Practical application: For public opinion monitoring of a hot event, the system needs to process a large amount of text data, social media interaction data, and image and video content in a short period of time; in order to ensure the timeliness of report generation, the system will give priority to tasks such as text processing, sentiment analysis, and trend prediction, and assign lower priority to tasks that do not require high processing time, such as supplementing background information.
[0132] Priority under the social platform monitoring target: For public opinion monitoring on social platforms, the system's goal is to continuously track and analyze the spread of certain specific keywords or topics; in this case, the priority of the task will be set based on indicators such as the real-time search volume, discussion heat, and number of comments of specific keywords; when the system detects a sudden surge in discussion volume for a preset keyword, or the keyword is related to a potential hot event, the system will give priority to these keyword-related tasks, such as sentiment analysis of related comments, user engagement analysis, etc.
[0133] Practical application: When monitoring keywords related to a brand on an online social platform, the system may prioritize posts and comments with high discussion and interaction volume, analyze user sentiment, and generate public opinion reports on brand reputation in a timely manner to facilitate brand management and market strategy adjustments by companies.
[0134] Priority of responding to sudden public opinion events: In the public opinion system, sudden public opinion events often trigger the highest priority tasks. For example, the system may detect that a company’s product quality problems or a politician’s controversial remarks are widely spread in a short period of time. In order to respond to such sudden events in a timely manner, the system needs to quickly analyze a large amount of relevant data, including the propagation path of negative public opinion, the emotional tendencies of participating users, and the potential impact of the event on the brand image or public trust. The priority of these tasks will be dynamically adjusted based on the urgency of the event, the scope of social impact, and the system’s processing time limit.
[0135] Practical application: When a product suddenly causes a large number of complaints among users, the system sets the highest priority for processing public opinion data for the incident, quickly analyzes negative reviews, identifies major communication channels, generates impact assessment reports and pushes them to the company, helping it take timely public relations and remedial measures.
[0136] The system dynamically adjusts task priorities based on these factors to ensure that high-risk, high-impact public opinion tasks are handled first, improving the system's response speed and coping capabilities.
[0137] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An information collection system based on big data, characterized in that: The system comprises: A data source selection module, configured to automatically select a data source according to a first preset indicator to obtain a first data source set, wherein the first preset indicator includes an update frequency, integrity, and credibility of the data source; A data filtering module is used to filter the first data source set through a content quality analysis formula to obtain a second data source set; the content quality analysis formula scores the data according to content relevance, content accuracy and completeness, and then weighted sums the scores to obtain a total score, and the data is filtered according to the total score; A data preprocessing module, used to preprocess the second data source set to generate a data set, wherein the preprocessing includes data cleaning, data standardization and data conversion; A distributed computing module is used to process a data set in real time and generate computing result data. The distributed computing module includes multiple nodes, each of which performs processing tasks for different data in the data set. The nodes transmit and synchronize data through message queues. The distinguishing criteria for the processing tasks include the computing amount and importance of the processing tasks. Each node is allocated different computing capabilities through IoT devices. A task allocation module, used to obtain a first task weight according to the remaining computing capacity of each node and the computing amount of the processing task, and allocate the processing task of the data segment to the node according to the first task weight; The load balancing module is used to obtain the load balancing weight of each node according to the real-time computing load of each node and the execution efficiency of the processing task, and feed the load balancing weight back to the task allocation module to achieve dynamic adjustment of the distribution of processing tasks; The expansion fault processing module is used to monitor the status of the node. When a node fails, the weight of the second task is calculated by the weight calculation formula and fed back to the task allocation module to allocate the task of the node to other healthy nodes. A data storage module, used to store calculation result data, wherein the data storage module includes a distributed database system capable of distributing and storing data among multiple storage nodes; The data preprocessing module comprises: A data fragmentation unit, used for, before data cleaning and conversion, dividing the data of the second data source set into a plurality of data fragments according to a preset threshold value to obtain a first data set; The preset threshold is dynamically adjusted based on the data type and data volume; The size of the data segment is calculated by the following calculation formula: , in, is the size of the data segment, is the total size of the second data source set, is the computing power of the node, is the preset load factor, is the standard deviation of the data segment, is the relevance factor of the data segment; The first task weight is calculated by the following formula: , in, is the first task weight, is the remaining computing capacity of the node, is the size of the data segment, To handle the computational workload of the task, is the computing power of the jth node, is the remaining computing capacity of the jth node, and m is the total number of nodes; The load balancing weight is calculated by the following formula: , in, is the load balancing weight; is the computational load of the node, is the execution efficiency of the corresponding processing task of the node. The execution efficiency is calculated by the ratio of the computing power of the node to the computing amount of the processing task. is the execution progress of the processing task of the kth node. The execution progress is calculated by the ratio of the remaining calculation amount to the calculation amount. is the computational load of the kth node, and m is the total number of nodes; The second task weight is calculated by the following weight formula: , in, is the second task weight, is the computing power of the failed node, is the state of the failed node, is the state exponential function of the faulty node, is the processing task importance of the faulty node, is the processing task importance of the cth node, is the state index of the cth node, is the state of the cth node, is the state exponential function of the cth node, is the computing power of the cth node, and m is the total number of nodes; in, , is the constant of node c, is the failure-free operation time of node c.
2. The information collection system based on big data according to claim 1 is characterized in that: The relevance factor of the data segment Obtained through the following steps: Perform feature extraction on each data segment to obtain a feature vector for each data segment; Calculate the similarity of the feature vectors to obtain a similarity matrix; According to the similarity matrix, the similarity of each data segment is normalized to obtain a similarity score; The similarity scores are weighted averaged to finally obtain the relevance factor of the data fragments. .
3. The information collection system based on big data according to claim 2 is characterized in that: The calculation formula of the similarity score is: , in, and are the feature vectors of the y-th and u-th data segments, respectively. for and The dot product of and is the modulus of the eigenvector.
4. The information collection system based on big data according to claim 1 is characterized in that: The state index of the cth node The calculation formula is as follows: , in, is the temperature of node c, is the historical failure rate of node c, which is calculated by the ratio of the number of historical failures to the running time. is the computational load of node c, is the remaining computing capacity of node c, , , and To adjust the weight coefficient, Z is the normalization factor.
5. The information collection system based on big data according to claim 4 is characterized in that: The importance of the processing task of the cth node The calculation formula is as follows: , in, To prioritize the processing tasks, Rating based on preset system goals, To process the task completion time, To handle the computational workload of the task, , and To adjust the weight coefficient.
