Data media interaction system based on big data visualization

Abstract

The invention relates to a data media interaction system based on big data visualization, which belongs to the technical field of data processing, and comprises a multi-source heterogeneous data acquisition module configured to acquire structured and unstructured data in real time through a distributed crawler and an API interface; the dynamic preprocessing module comprises a streaming data cleaning engine and a cache management unit and is used for performing real-time data denoising, format conversion and cache optimization; the intelligent analysis engine is integrated with a dynamic modeling unit based on machine learning and an incidence relation self-optimization mechanism; the multi-modal interaction module has a multi-channel input analysis capability of voice, gesture and eye movement tracking; the self-adaptive visual rendering module comprises a GPU accelerated real-time rendering engine and a visual element dynamic recombination unit; and the resource intelligent scheduling module is deployed with a heterogeneous computing resource dynamic allocation algorithm and a load prediction model. According to the invention, multiple data can be collected, and the data can be integrated.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data media interaction system based on big data visualization. Background Art

[0002] With the advancement of digitalization, data media interaction systems are widely used in many fields. However, faced with the surge in data volume and the diversification of user interaction needs, existing technical bottlenecks are becoming prominent.

[0003] When it comes to data collection and integration, data sources are becoming increasingly diverse, encompassing structured, semi-structured, and unstructured data from sources like websites, social media, and sensors. Traditional data collection methods struggle to efficiently process data in diverse formats and protocols, resulting in low collection efficiency and an inability to adequately support subsequent analysis. In the news media sector, for example, journalists collect material from multiple platforms. Due to varying data formats and interfaces, this process requires significant labor and is prone to errors. Furthermore, these efforts present challenges, including poor real-time interactive performance, limited visualization, and low resource utilization. Summary of the Invention

[0004] The purpose of the present invention is to address the problems existing in the background technology and to propose a data media interaction system based on big data visualization that can collect multiple data and integrate and process the data.

[0005] The technical solution of the present invention is a data media interactive system based on big data visualization, comprising: Multi-source heterogeneous data acquisition module, configured to acquire structured and unstructured data in real time through distributed crawlers and API interfaces; Dynamic preprocessing module, including streaming data cleaning engine and cache management unit, is used to perform real-time data denoising, format conversion and cache optimization; Intelligent analysis engine, integrating dynamic modeling units based on machine learning and self-optimization mechanism of association relationships; Multimodal interaction module, capable of parsing multi-channel inputs such as voice, gestures, and eye tracking; Adaptive visualization rendering module, including GPU-accelerated real-time rendering engine and dynamic reorganization unit of visualization elements; The resource intelligent scheduling module deploys a dynamic allocation algorithm for heterogeneous computing resources and a load prediction model.

[0006] Preferably, the dynamic preprocessing module includes: a real-time data cleaning submodule, which adopts a dual filtering mechanism based on a rule engine and a neural network; and a cache management unit that implements a hierarchical storage strategy, including a hot data storage layer and a cold data compression layer.

[0007] Preferably, the intelligent analysis engine includes: a dynamic modeling unit adopting an online learning mechanism, which can automatically adjust the analysis model according to changes in data characteristics; and a correlation self-optimization mechanism including a cross-dimensional correlation calculator and a dynamic weight allocator.

[0008] Preferably, the multimodal interaction module is configured with: a multi-dialect recognition unit for voice commands that supports context-sensitive semantic understanding; a gesture recognition unit integrated with a three-dimensional spatial trajectory analysis algorithm; and an eye tracking unit including a gaze point prediction model and an interaction delay compensation mechanism.

[0009] Preferably, the adaptive visualization rendering module includes: a visualization mode dynamic switching unit that can automatically match the best display form according to the data type; and a rendering effect self-optimization unit that implements dynamic adjustment of visual parameters based on user feedback.

[0010] Preferably, the resource intelligent scheduling module includes: a dynamic allocation algorithm for heterogeneous computing resources, supporting hybrid scheduling of CPU / GPU / FPGA; and a load prediction model using a combined prediction method of time series analysis and reinforcement learning.

[0011] Preferably, it also includes: a security management module, which includes a dynamic permission management unit and a data encryption transmission channel; the dynamic permission management unit implements visual element-level access control based on user roles.

[0012] Preferably, the security management and control module also includes: a data desensitization processing unit, which is equipped with a sensitive information recognition model and a dynamic masking algorithm; and an abnormal behavior detection unit, which uses user operation pattern analysis and deep learning anomaly recognition.

[0013] Preferably, it also includes: an intelligent configuration module, including a visual template generator and a rule customization interface; the template generator is integrated with an automatic layout algorithm based on design pattern recognition.

[0014] Preferably, the system architecture adopts: a distributed microservice architecture, with loosely coupled communication between modules through an event bus; and a containerized deployment unit to support elastic expansion and fault isolation of computing resources.

[0015] Compared with the prior art, the present invention has the following beneficial technical effects: In the present invention, the multi-source heterogeneous data acquisition module utilizes distributed crawlers and API interfaces to acquire various types of data in real time, breaking the limitations of traditional acquisition, solving the problem of multi-source data integration, and providing a comprehensive data foundation for subsequent processing.

[0016] In the dynamic preprocessing module, the real-time data cleaning submodule uses a rule engine and a neural network for dual filtering to accurately remove noise and improve data quality; the cache management unit uses hierarchical storage, with the hot data layer ensuring fast access and the cold data layer saving space, optimizing the cache and improving data access efficiency.

[0017] The dynamic modeling unit of the intelligent analysis engine can learn online and automatically adjust the model according to data characteristics to ensure accurate analysis; the self-optimization mechanism of association relationships uses cross-dimensional association calculation and dynamic weight allocation to explore the potential value of data.

[0018] The multimodal interaction module integrates voice, gesture, and eye tracking. Voice recognition supports multiple dialects and contextual understanding, gesture recognition uses a three-dimensional trajectory algorithm, and eye tracking features gaze point prediction and delay compensation, providing users with a natural and convenient interactive experience.

[0019] The adaptive visualization rendering module dynamically switches visualization modes, matching the best display format based on data type; the rendering effect self-optimization unit adjusts parameters based on user feedback to help users quickly understand data.

[0020] The intelligent resource scheduling module uses a dynamic allocation algorithm for heterogeneous computing resources to achieve hybrid scheduling of CPU, GPU, and FPGA. It combines the load prediction model to allocate resources in advance, improve utilization, and ensure system stability and efficiency.

[0021] The security control module comprehensively ensures system and data security, from dynamic permission management, data encryption transmission, desensitization processing to abnormal behavior detection.

[0022] The visual template generator of the intelligent configuration module automatically generates beautiful layouts, and the rule customization interface meets the user's personalized needs.

[0023] The system adopts a distributed microservice architecture and containerized deployment. The former facilitates functional expansion and module updates, while the latter enables elastic resource expansion and fault isolation, enhancing system availability and maintainability.

[0024] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and to implement it according to the contents of the description, the following preferred embodiments of the present invention are described in detail with reference to the accompanying drawings. The specific implementation methods of the present invention are given in detail by the following embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a structural diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples provided are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. The following paragraphs describe the present invention in more detail by way of example with reference to the accompanying drawings. It should be noted that the drawings are all in a very simplified form and are not to exact scale, and are only used for the purpose of conveniently and clearly illustrating the embodiments of the present invention.

[0027] Example 1 like Figure 1 As shown, the present invention proposes a data media interaction system based on big data visualization, including: Multi-source heterogeneous data acquisition module, configured to acquire structured and unstructured data in real time through distributed crawlers and API interfaces; Dynamic preprocessing module, including streaming data cleaning engine and cache management unit, is used to perform real-time data denoising, format conversion and cache optimization; Intelligent analysis engine, integrating dynamic modeling units based on machine learning and self-optimization mechanism of association relationships; Multimodal interaction module, capable of parsing multi-channel inputs such as voice, gestures, and eye tracking; Adaptive visualization rendering module, including GPU-accelerated real-time rendering engine and dynamic reorganization unit of visualization elements; The resource intelligent scheduling module deploys a dynamic allocation algorithm for heterogeneous computing resources and a load prediction model.

[0028] The dynamic preprocessing module includes: a real-time data cleaning submodule, which adopts a dual filtering mechanism based on a rule engine and a neural network; the cache management unit implements a hierarchical storage strategy, including a hot data storage layer and a cold data compression layer. The dual filtering mechanism of the real-time data cleaning submodule can more accurately remove data noise and improve data quality; the hierarchical storage strategy of the cache management unit can reasonably store data according to the frequency of data use. The hot data storage layer ensures fast access to frequently used data, and the cold data compression layer saves storage space, thereby realizing cache optimization, providing high-quality and efficiently accessed data for subsequent data processing, and improving the overall performance of the system.

[0029] The intelligent analysis engine includes: the dynamic modeling unit adopts an online learning mechanism, which can automatically adjust the analysis model according to changes in data characteristics; the association self-optimization mechanism includes a cross-dimensional association calculator and a dynamic weight allocator. The online learning mechanism of the dynamic modeling unit enables the system to adapt to changes in data characteristics in real time, ensuring the accuracy and effectiveness of the analysis model; the association self-optimization mechanism can automatically optimize the association analysis between data, explore the potential value of data, improve the depth and breadth of data analysis, and provide more valuable information for data media interaction.

[0030] The multimodal interaction module is configured with: a multi-dialect recognition unit for voice commands, which supports context-related semantic understanding; a gesture recognition unit integrated with a three-dimensional spatial trajectory analysis algorithm; an eye tracking unit including a gaze point prediction model and an interaction delay compensation mechanism. The multimodal interaction method provides users with a more natural and convenient interaction experience, meeting the interaction needs of different users; the multi-dialect recognition unit and context-related semantic understanding improve the accuracy and applicability of voice interaction; the three-dimensional spatial trajectory analysis algorithm of the gesture recognition unit can accurately identify user gestures; the gaze point prediction model and interaction delay compensation mechanism of the eye tracking unit reduce interaction delay, improve the real-time and smoothness of interaction, and enhance the interaction effect between the user and the system.

[0031] In this embodiment, in the news media sector, to obtain diverse news data, a distributed crawler can be configured to capture data from major news websites and social media platforms. For example, the crawler can be configured to traverse web pages according to the website's page structure and link relationships using a breadth-first search (BFS) or depth-first search (DFS) algorithm, extracting structured data such as news headlines, text, and release dates, as well as unstructured data such as comments and images, using regular expressions or XPath syntax. At the same time, an API interface can be used to connect with news information platforms to obtain standardized news data provided by them, such as real-time stock data for financial news and sports scores. Regarding data collection frequency, a higher frequency can be set for hot news data, such as every 5 minutes, based on data update requirements; regular news data can be collected every hour.

[0032] In the real-time data cleaning submodule, the rule engine pre-sets various data cleaning rules. For example, the time format in news data must conform to the standard "YYYY-MM-DDHH:MM:SS" format, and any non-compliant formats are automatically converted. Sensitive words in text data are filtered using keyword matching rules. In the neural network component, a convolutional neural network (CNN) is used to identify noise in news text data. A large number of news texts, labeled with both noise and normal data, are used as training data. The model is trained to learn data features and detect and remove noise in new data in real time during the data collection process.

[0033] When the cache management unit implements a hierarchical storage strategy, it divides news data into hot data and cold data according to its access frequency. For popular news, that is, data with access volume higher than a threshold value such as 1,000 times / hour within a certain period of time, it is stored in the hot data storage layer and stored on solid-state drives (SSDs) to ensure fast reading. For unpopular news data with low access volume, it is compressed using a compression algorithm such as Gzip and stored in the cold data compression layer and stored on mechanical hard drives (HDDs) to save storage space.

[0034] When analyzing news dissemination trends, the dynamic modeling unit employs an online learning mechanism. Taking data on changes in news readership over time as an example, the model initially employs a linear regression model for analysis. As new data continuously flows in, the model parameters are updated in real time using the stochastic gradient descent (SGD) algorithm, automatically adjusting the analysis model to accommodate fluctuations in news readership. Within the self-optimization mechanism for correlations, a cross-dimensional correlation calculator calculates the correlation between different dimensions of a news story, such as keyword popularity, release date, and media source, quantifying the degree of correlation using the Pearson correlation coefficient. A dynamic weight allocator assigns weights to each dimension based on the degree of correlation. For example, when analyzing the spread of hot news, the weight of keyword popularity is increased to more accurately uncover correlations between news stories.

[0035] In the voice interaction scenario, when a user says "Search for the latest tech news from the past week," the multi-dialect recognition unit for voice commands collects the voice signal, extracts acoustic features using the MFCC algorithm, and uses an LSTM-based acoustic model for speech recognition, converting the speech into text. The BERT model then performs semantic understanding, combining contextual information such as "latest week" and "tech news" to accurately understand the user's intent and retrieve relevant news from system data.

[0036] In the news browsing interface, when the user makes an upward swipe gesture, the gesture recognition unit uses the depth camera to obtain the three-dimensional point cloud data of the gesture. After preprocessing, the gesture features are extracted using CNN. The pattern matching algorithm is used to compare with the predefined "upward swipe" gesture model to identify the user's intention, and the system executes the operation of scrolling the news list upward.

[0037] The eye tracking unit is on the news details page. The camera captures the user's eye image, locates the pupil and corneal reflection point through corneal reflection method, and uses the support vector regression (SVR) model to predict the position of the user's gaze point on the screen. When the user looks at a picture in the news, the system automatically enlarges the picture for display. The interactive delay compensation mechanism records the timestamps of each link of eye movement data collection, processing and system response, uses the dynamic time warping (DTW) algorithm to predict delays, adjusts the system response in advance, and realizes instant interaction.

[0038] When the system receives different types of news data, the dynamic visualization mode switching unit automatically determines the data type. For time-series data showing changes in news readership over time, a line chart is used for presentation; for data on the distribution of news attention in different regions, a map heat map is used. The rendering optimization unit collects user feedback on the visualization effects. For example, if a user is dissatisfied with the color contrast of a news chart, the system uses a weighted average algorithm to adjust the chart's color, font size, and other visual parameters based on the user's feedback rating to optimize the rendering effect.

[0039] During peak news seasons, system load increases. A dynamic allocation algorithm for heterogeneous computing resources allocates computing resources based on task type. For real-time analysis of news data, some compute-intensive tasks are assigned to the GPU for parallel processing. Data storage and simple query tasks are handled by the CPU. A load forecasting model uses the ARIMA model from time series analysis to predict system load trends over a period of time. Combined with the Q-learning algorithm from reinforcement learning, resource allocation strategies are adjusted in advance based on the forecast results, such as increasing GPU resources to cope with the upcoming high load.

[0040] The dynamic permission management unit assigns permissions based on user roles. For example, ordinary users can only browse public news content, editors can modify and publish news, and administrators have all permissions. Based on user roles, level-by-level access control is performed on visual elements. For example, ordinary users cannot see the news data analysis charts in the background. The data desensitization processing unit uses a sensitive information recognition model based on convolutional neural networks to identify sensitive information such as ID numbers and bank card numbers in news, and uses a dynamic masking algorithm to replace sensitive information with specific symbols. The abnormal behavior detection unit analyzes user operation patterns through a clustering analysis algorithm. When abnormal behavior such as a user frequently attempting to log in to different accounts in a short period of time is detected, the deep learning convolutional neural network model is used to identify abnormalities and take timely measures such as banning the account.

[0041] The visualization template generator generates visualization templates based on the characteristics of news data and an automatic layout algorithm based on design pattern recognition. For example, for comparative data of multiple news items, it automatically generates a bar chart template and reasonably lays out elements such as titles, coordinate axes, and legends. Users can modify the template style according to their own needs through the rule customization interface, such as adjusting the color and width of the bar chart, to achieve personalized configuration.

[0042] Example 2 like Figure 1As shown, the present invention proposes a data media interaction system based on big data visualization. Compared with the first embodiment, the adaptive visualization rendering module in this embodiment includes: a visualization mode dynamic switching unit, which can automatically match the optimal display form according to the data type; a rendering effect self-optimization unit, which implements dynamic adjustment of visual parameters based on user feedback, and the GPU-accelerated real-time rendering engine improves the rendering speed of data visualization to meet the needs of real-time interaction; the visualization mode dynamic switching unit automatically selects the appropriate display form according to the data type, which solves the problem of single visualization form; the rendering effect self-optimization unit adjusts the visual parameters based on user feedback, so that the visualization effect is more in line with user needs, and improves the user's understanding and analysis efficiency of the data.

[0043] The intelligent resource scheduling module includes: a dynamic allocation algorithm for heterogeneous computing resources, supporting hybrid scheduling of CPU / GPU / FPGA; a load prediction model that uses a combined prediction method of time series analysis and reinforcement learning. The dynamic allocation algorithm for heterogeneous computing resources can reasonably allocate different types of computing resources according to task requirements, improving resource utilization; the load prediction model accurately predicts system load, schedules and allocates resources in advance, avoids resource waste and system overload, and ensures stable operation and efficient performance of the system.

[0044] Example 3 like Figure 1 As shown, the present invention proposes a data media interaction system based on big data visualization. Compared with the first or second embodiment, this embodiment also includes: a security control module, including a dynamic permission management unit and a data encryption transmission channel; the dynamic permission management unit implements visual element-level access control based on user roles. The security control module also includes: a data desensitization processing unit, which is equipped with a sensitive information recognition model and a dynamic masking algorithm; an abnormal behavior detection unit, which adopts user operation mode analysis and deep learning anomaly recognition, and the dynamic permission management unit realizes more refined permission control to protect the security and privacy of data; the data encryption transmission channel ensures the security of data during transmission; the data desensitization processing unit processes sensitive information to prevent sensitive information leakage; the abnormal behavior detection unit can timely detect and handle abnormal operations to ensure the safe and stable operation of the system.

[0045] Example 4 like Figure 1As shown, the present invention proposes a data media interaction system based on big data visualization. Compared with embodiment one, embodiment two or embodiment three, this embodiment also includes: an intelligent configuration module, including a visualization template generator and a rule customization interface; the template generator is integrated with an automatic layout algorithm based on design pattern recognition, and the visualization template generator can automatically generate visualization templates to improve the efficiency of visualization configuration; the rule customization interface provides users with a flexible configuration method to meet the personalized needs of different users; the automatic layout algorithm makes the generated visualization template layout more reasonable and beautiful, thereby improving the user experience.

[0046] The system architecture adopts: distributed microservice architecture, with loosely coupled communication between modules through the event bus; containerized deployment units, which support elastic expansion and fault isolation of computing resources; the distributed microservice architecture makes the system have good scalability and maintainability, facilitating system function expansion and module updates; the loosely coupled communication method of the event bus improves the stability and reliability of the system; the containerized deployment unit realizes the elastic expansion of computing resources and can dynamically adjust resources according to business needs. At the same time, the fault isolation mechanism ensures that the system can still operate normally when some modules fail, thereby improving the system's availability; the distributed microservice architecture involves service registration, discovery and calling.

[0047] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any ordinary technician in this industry can smoothly implement the present invention as shown in the drawings and described above. However, any equivalent changes, modifications and evolutions made by technicians familiar with this profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the scope of protection of the technical solution of the present invention.

Claims

1. A data media interactive system based on big data visualization, characterized by: include, Multi-source heterogeneous data acquisition module, configured to acquire structured and unstructured data in real time through distributed crawlers and API interfaces; Dynamic preprocessing module, including streaming data cleaning engine and cache management unit, is used to perform real-time data denoising, format conversion and cache optimization; Intelligent analysis engine, integrating dynamic modeling units based on machine learning and self-optimization mechanism of association relationships; Multimodal interaction module, capable of parsing multi-channel inputs such as voice, gestures, and eye tracking; Adaptive visualization rendering module, including GPU-accelerated real-time rendering engine and dynamic reorganization unit of visualization elements; The resource intelligent scheduling module deploys a dynamic allocation algorithm for heterogeneous computing resources and a load prediction model.

2. The data media interactive system based on big data visualization according to claim 1, characterized in that: The dynamic preprocessing module includes: a real-time data cleaning sub-module, which adopts a dual filtering mechanism based on a rule engine and a neural network; the cache management unit implements a hierarchical storage strategy, including a hot data storage layer and a cold data compression layer.

3. The data media interactive system based on big data visualization according to claim 2, characterized in that: The intelligent analysis engine includes: the dynamic modeling unit adopts an online learning mechanism, which can automatically adjust the analysis model according to changes in data characteristics; the association relationship self-optimization mechanism includes a cross-dimensional association calculator and a dynamic weight allocator.

4. The data media interactive system based on big data visualization according to claim 3, characterized in that: The multimodal interaction module is equipped with: a multi-dialect recognition unit for voice commands that supports context-sensitive semantic understanding; a gesture recognition unit integrated with a three-dimensional spatial trajectory analysis algorithm; and an eye tracking unit that includes a gaze point prediction model and an interaction delay compensation mechanism.

5. The data media interactive system based on big data visualization according to claim 4, characterized in that: The adaptive visualization rendering module includes: a dynamic visualization mode switching unit that can automatically match the optimal display form according to the data type; and a rendering effect self-optimization unit that implements dynamic adjustment of visual parameters based on user feedback.

6. The data media interactive system based on big data visualization according to claim 5, characterized in that: The intelligent resource scheduling module includes: a dynamic allocation algorithm for heterogeneous computing resources, supporting hybrid scheduling of CPU / GPU / FPGA; and a load prediction model that uses a combined prediction method of time series analysis and reinforcement learning.

7. The data media interactive system based on big data visualization according to claim 6, characterized in that: Also includes: Security control module, including dynamic permission management unit and data encryption transmission channel; The dynamic rights management unit implements visual element-level access control based on user roles.

8. The data media interactive system based on big data visualization according to claim 7, characterized in that: The security management and control module also includes: a data desensitization processing unit, which is equipped with a sensitive information recognition model and a dynamic masking algorithm; and an abnormal behavior detection unit, which uses user operation pattern analysis and deep learning anomaly recognition.

9. The data media interactive system based on big data visualization according to claim 8, characterized in that: Also includes: The intelligent configuration module includes a visual template generator and a rule customization interface; the template generator is integrated with an automatic layout algorithm based on design pattern recognition.

10. The data media interactive system based on big data visualization according to claim 9, characterized in that: The system architecture adopts: distributed microservice architecture, with loosely coupled communication between modules through the event bus; Containerized deployment units support elastic expansion and fault isolation of computing resources.

Images