A real-time integration and visualization method and system for multivariate data

Abstract

The present invention belongs to the technical field of data visualization, and discloses a real-time integration and visualization method and system for multivariate data. The method preprocesses multivariate data, generates data fingerprints and clusters them, binds multivariate data and cloud data, monitors and optimizes computing resources in real time, and utilizes virtual reality technology to provide users with an immersive experience. The present invention collects operating data of a three-dimensional visualization platform, dynamically adjusts computing resources, optimizes the configuration and use of computing resources, improves the operating efficiency and stability of the platform, and utilizes virtual reality technology in the three-dimensional visualization platform to provide users with an immersive experience, thereby enhancing the user's sense of experience and increasing the attractiveness of the three-dimensional visualization platform.

Description

Technical Field

[0001] The present invention belongs to the technical field of data visualization, and in particular relates to a multivariate data real-time integration and visualization method and system. Background Art

[0002] With the rapid development of information technology, how to efficiently process and display multivariate data has become one of the research focuses in the field of data science and visualization technology; however, existing data integration and visualization methods still have many shortcomings.

[0003] Existing data integration methods require data cleaning and data fusion operations, which makes it difficult to process large-scale data streams in real time, resulting in large data delays. In addition, the function of dynamically adjusting computing resources provided by existing data visualization platforms is only based on fixed thresholds and cannot be adjusted according to operating conditions, resulting in low utilization efficiency of computing resources. At the same time, although existing visualization methods provide basic visualization functions in terms of user experience, they are still insufficient in providing an immersive experience and cannot be optimized and adjusted based on user feedback. Therefore, how to achieve real-time integration and efficient visualization of multivariate data while optimizing computing resource allocation has become a problem to be solved. Summary of the invention

[0004] To solve the above technical problems, the present invention provides a method and system for real-time integration and visualization of multivariate data. By collecting the operating data of a three-dimensional visualization platform and dynamically adjusting computing resources, the configuration and use of computing resources are optimized, the operating efficiency and stability of the platform are improved, and the virtual reality technology in the three-dimensional visualization platform is used to provide users with an immersive experience, thereby enhancing the user experience and increasing the attractiveness of the three-dimensional visualization platform, solving the problem of dynamic resource management in real-time integration and efficient visualization of multivariate data.

[0005] The method for real-time integration and visualization of multivariate data of the present invention comprises the following steps:

[0006] S1, obtain multi-data in the database and perform preprocessing;

[0007] S2, using the preprocessed multivariate data to generate data fingerprints, clustering the cloud data in the multivariate data using the data fingerprints, and generating classification features;

[0008] S3, using classification features to bind multivariate data and cloud data, and integrating and synchronizing the bound content to the 3D visualization platform;

[0009] S4. Monitor the actual operation of the three-dimensional visualization platform in real time and collect operation data, dynamically adjust computing resources according to the collected operation data, and optimize the configuration of computing resources;

[0010] S5. Use virtual reality technology in the three-dimensional visualization platform to provide users with an immersive experience, collect user feedback, and continuously adjust and optimize the visualization method of real-time integration of multivariate data based on the feedback information.

[0011] Furthermore, S1 is specifically:

[0012] Acquire multi-dimensional data in the database by integrating the intranet environment and accessing the upper management system;

[0013] Check and correct erroneous and missing values ​​in multivariate data, handle missing values ​​by interpolation, unify data formats, remove noisy data, and correct errors in data formats.

[0014] Furthermore, S2 is specifically:

[0015] Extracting key features of text keywords and numerical ranges from the processed multivariate data, and encoding the extracted key features into a binary string of fixed length to generate a data fingerprint;

[0016] Calculate the Hamming distance between the first and second data fingerprints, the expression is:

[0017] ;

[0018] in, represents the Hamming distance, represents the first data fingerprint, Represents the second data fingerprint, represents the Hamming distance between the first data fingerprint and the second data fingerprint, Indicates the number of binary digits contained in each data fingerprint, Represents the position index of the binary string, Indicates that the calculation starts from the first bit of the data fingerprint. Represents the value of the first data fingerprint at the position index of the binary string, Represents the value of the second data fingerprint at the position index of the binary string, Represents the difference between the first data fingerprint and the second data fingerprint in the position index of the binary string;

[0019] The cloud data are clustered based on the calculated Hamming distance, and cloud data with high similarity are classified into the same category, thereby generating classification features.

[0020] Furthermore, S3 is specifically:

[0021] Extract classification features of multivariate data from clustered multivariate data;

[0022] Match the classification features of cloud data and multivariate data, find multivariate data with the same classification features, and bind cloud data and multivariate data;

[0023] The bound content is integrated and synchronized to the 3D visualization platform by using API interfaces and message queues.

[0024] Furthermore, S4 is specifically:

[0025] Determine the performance indicators, application indicators, and system health indicators to be monitored, and deploy monitoring tools. Use ELKStack to collect and analyze the logs of the 3D visualization platform, and automatically collect operation data through SDK, and send the collected operation data to the monitoring server through a secure protocol;

[0026] Use data analysis tools to analyze the collected operation data in real time. By analyzing the CPU utilization and memory utilization indicators, find out the main factors that lead to poor application performance and set the CPU and memory utilization thresholds. The expressions are:

[0027] ,

[0028] ,

[0029] in, Indicates the preset CPU utilization threshold. Indicates the time point used to identify the running data. Indicates the CPU utilization threshold that changes over time. Indicates the average CPU utilization. Indicates the standard deviation of CPU utilization. Indicates the preset memory utilization threshold. Indicates the average memory utilization. represents the standard deviation of memory utilization, and Constants representing the sensitivity of adjusting the CPU and memory utilization thresholds respectively;

[0030] Check the usage of computing resource configuration according to the preset CPU and memory utilization thresholds, perform benchmark and stress tests on the 3D visualization platform, record the performance of the platform under different loads, regularly re-evaluate the usage of resource configuration, and continue to optimize based on the new evaluation results;

[0031] Dynamically adjust computing resources according to the new evaluation results to optimize the configuration of computing resources. The expression is:

[0032] ,

[0033] ,

[0034] in, Indicates the adjustment amount of the number of computing resources, Indicates the new number of computing resources, Indicates the amount of unneeded computing resources, represents the length of the time window, It represents the time point after the time point of running data plus the time window length. Indicates the number of computing resources at the time point after the time point of the running data plus the time window length. Indicates the number of computing resources at the time point of the current identification running data.

[0035] Furthermore, in S5, the virtual reality technology in the three-dimensional visualization platform is used to provide users with an immersive experience, specifically:

[0036] Use a laser scanner to perform 3D scanning of real scenes to obtain high-precision point cloud data. Based on the point cloud data, use WebGL's 3D modeling software to create a 3D visualization model that matches the real scene.

[0037] Prepare VR headsets, add realistic sound effects and introduce haptic feedback mechanisms to simulate the feeling of touching objects, and use an integrated physics engine to simulate object movement and collisions;

[0038] Integrate the real-time data when using virtual reality headsets into the 3D visualization platform, and use animation transition effects to show the changes in real-time data, so that users can intuitively understand the visualization trends of real-time data, thereby providing users with an immersive experience.

[0039] Furthermore, in S5, user feedback is collected, and the visualization method of real-time integration of multivariate data is continuously adjusted and optimized according to the feedback information, specifically:

[0040] Integrate online forms, emails, and social media feedback channels into the 3D visualization platform;

[0041] Regularly send user satisfaction surveys on feedback channels to collect users' evaluations and feedback on platform functions, performance, and usability. Use machine learning algorithms to identify users' behavior on the platform and analyze their usage habits and preferences for the 3D visualization platform, thereby adjusting and optimizing the platform's functional modules and improving the user interface design.

[0042] Regularly release updated versions of the 3D visualization platform to gradually improve the platform functions, and use A / B testing methods to compare the effects of the new and old versions to verify the effectiveness of the improvement measures.

[0043] The present invention also provides a multivariate data real-time integration and visualization system for implementing the multivariate data real-time integration and visualization method, the system comprising:

[0044] The data acquisition and preprocessing module is responsible for acquiring multi-dimensional data in the database and performing preprocessing;

[0045] The data fingerprint generation module is responsible for generating data fingerprints using preprocessed multivariate data, clustering cloud data in multivariate data using data fingerprints, and generating classification features;

[0046] The data binding and classification module is responsible for binding multivariate data and cloud data using classification features, and integrating and synchronizing the bound content to the 3D visualization platform;

[0047] The real-time monitoring and resource optimization module is responsible for real-time monitoring of the actual operation of the 3D visualization platform and collecting operation data. It dynamically adjusts computing resources based on the collected operation data and optimizes the configuration of computing resources.

[0048] The 3D visualization and user interaction module is responsible for using the virtual reality technology in the 3D visualization platform to provide users with an immersive experience;

[0049] The user feedback processing and optimization module is responsible for collecting user feedback and continuously adjusting and optimizing the visualization method of real-time integration of multivariate data based on the feedback information.

[0050] The beneficial effects described in the present invention are as follows: Different from the dynamic adjustment in the prior art which is mostly based on the overall load of the system, the present invention can go deep into the specific process of data processing, collect the operation data of the three-dimensional visualization platform, and dynamically adjust the computing resources, which not only optimizes the allocation and use of background data processing resources, but also combines the operation requirements of the three-dimensional visualization platform, and ensures the smoothness of user experience and the stability of platform operation by dynamically adjusting the computing resource allocation in the virtual reality technology. The virtual reality technology in the three-dimensional visualization platform is used to provide users with an immersive experience, enhance the user experience and increase the attractiveness of the three-dimensional visualization platform; the present invention also generates data fingerprints, combines the Hamming distance and introduces a dynamic adjustment mechanism, and can dynamically update the data fingerprints according to real-time data changes to perform corresponding clustering optimization, making the clustering process more flexible and efficient, reducing the redundancy of data in the processing process, and improving the computing efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method for real-time integration and visualization of multivariate data in Example 1;

[0052] Figure 2 This is a decision diagram of the real-time integration and visualization method of multivariate data in Example 1. DETAILED DESCRIPTION

[0053] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0054] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a multivariate data real-time integration and visualization method, comprising the following steps:

[0055] S1. Obtain multi-dimensional data from the database and perform preprocessing, specifically:

[0056] Acquire multi-dimensional data in the database by integrating the intranet environment and accessing the upper management system;

[0057] Multivariate data includes text data, image data, time series data, cloud data, and network data;

[0058] Intranet environment integration refers to the process of integrating various information systems and services in a local area network within an organization. The intranet environment contains multiple subsystems and services, such as database servers, application servers, storage devices, and various business systems. These systems and services are usually located behind a firewall and only authorized users are allowed to access them;

[0059] Accessing the upper management system means connecting each subsystem and service in the intranet to a centralized management system for unified management and monitoring. Accessing the upper management system will provide a unified platform, allowing administrators to centrally manage multiple subsystems and services, thereby monitoring the operating status of each system in real time, and promptly issuing alarms when abnormalities occur, recording user access behaviors and system operations, and facilitating post-audit and troubleshooting;

[0060] Check and correct the erroneous and missing values ​​in multivariate data, and handle missing values ​​by interpolation, unify data formats, remove noise data and correct errors in data formats;

[0061] Interpolation is a mathematical method for estimating unknown data values ​​between known data points. In the fields of data analysis, signal processing, and computer graphics, interpolation is widely used to fill missing data, smooth curves, and improve image quality. In the data acquisition process, data missing may occur due to sensor failure, communication interruption, etc. Interpolation can help fill these missing data.

[0062] S2. Generate data fingerprints using the preprocessed multivariate data, cluster the cloud data in the multivariate data using the data fingerprints, and generate classification features, specifically:

[0063] Extracting key features of text keywords and numerical ranges from the processed multivariate data, and encoding the extracted key features into a binary string of fixed length to generate a data fingerprint;

[0064] Data fingerprint is a compact representation method for identifying a data set, usually a fixed-length string or digital sequence. The main purpose of data fingerprint is to quickly compare a large number of data sets to determine whether these data sets are the same or similar without directly comparing the original data.

[0065] Calculate the Hamming distance between the first and second data fingerprints, the expression is:

[0066] ;

[0067] in, represents the Hamming distance, represents the first data fingerprint, Represents the second data fingerprint, represents the Hamming distance between the first data fingerprint and the second data fingerprint; Indicates the number of binary bits contained in each data fingerprint, which is used to define the length of the data fingerprint; Represents the position index of the binary string, used to traverse each binary bit, Indicates that the calculation starts from the first bit of the data fingerprint; Represents the value of the first data fingerprint at the position index of the binary string, Represents the value of the second data fingerprint at the position index of the binary string, It represents the difference between the first data fingerprint and the second data fingerprint in the position index of the binary string. By comparing the differences between the data items of the two data fingerprints, the similarity of the two data fingerprints is determined. The smaller the similarity, the more similar the two data fingerprints are.

[0068] The Hamming distance is a measure of the difference between two strings of equal length, which is defined as the number of different characters in corresponding positions of the two strings. The cloud data are clustered based on the calculated Hamming distance, and cloud data with higher similarity are classified into the same category, thereby generating classification features.

[0069] S3. Use classification features to bind multivariate data and cloud data, and integrate and synchronize the bound content to the 3D visualization platform. Specifically:

[0070] Extract classification features of multivariate data from clustered multivariate data;

[0071] Match the classification features of cloud data and multivariate data, find multivariate data with the same classification features, and bind cloud data and multivariate data;

[0072] The bound content is integrated and synchronized to the 3D visualization platform by using API interfaces and message queues.

[0073] The API interface is a set of rules that define how software components communicate with each other. It specifies how to transfer data, request services, and receive responses between different software components. The API interface is mainly divided into two types: library API and network API. The library API is a function library provided by the programming language, which is used to implement functions such as file operations and network requests. The network API refers to a service interface that exchanges data through a network protocol, and is used for communication between different components in a distributed system.

[0074] S4. Monitor the actual operation of the 3D visualization platform in real time and collect operation data. Dynamically adjust computing resources according to the collected operation data and optimize the configuration of computing resources. Specifically:

[0075] Determine the performance indicators, application indicators, and system health indicators to be monitored, and deploy monitoring tools. Use ELKStack (data open source tool combination) to collect and analyze the logs of the 3D visualization platform, and automatically collect operation data through SDK (software development kit), and send the collected operation data to the monitoring server through a secure protocol;

[0076] Use data analysis tools to analyze the collected operation data in real time. By analyzing the CPU utilization and memory utilization indicators, find out the main factors that lead to poor application performance, such as excessively high and low CPU utilization and memory utilization, and set the CPU and memory utilization thresholds. The expressions are as follows:

[0077] ,

[0078] ,

[0079] in, Indicates the preset CPU utilization threshold; Indicates the time point used to identify the running data; Indicates the CPU utilization threshold that changes over time; Indicates the average CPU utilization; Indicates the standard deviation of CPU utilization, reflecting the fluctuation of CPU utilization over a period of time; Indicates the preset memory utilization threshold. Indicates the average memory utilization, reflecting the average operating status of the 3D visualization platform over a period of time. Represents the standard deviation of memory utilization, reflecting the time point The variation range of CPU utilization and memory utilization. The larger the standard deviation, the greater the data fluctuation, and the smaller the standard deviation, the more stable the data. and The constants respectively represent the sensitivity of adjusting the CPU and memory utilization thresholds, and are used to adjust the dynamic range of the thresholds. A larger constant value will make the threshold further away from the average value, and a smaller constant value will make the threshold closer to the average value. By analyzing the running data and observing the change trends of the CPU and memory utilization, we can draw the history and memory utilization charts, and determine the constant values ​​of the sensitivity of the CPU and memory utilization thresholds according to the change trend rules reflected in the charts.

[0080] If the CPU and memory utilization thresholds are exceeded, the load performance strategy needs to be adjusted and an alarm notification needs to be sent to relevant personnel. If the CPU and memory utilization thresholds are not exceeded, the configuration of computing resources needs to be optimized and preventive checks on the 3D visualization platform need to be performed regularly.

[0081] Check the usage of computing resource configuration according to the preset CPU and memory utilization thresholds, perform benchmark and stress tests on the 3D visualization platform, record the performance of the platform under different loads, regularly re-evaluate the usage of resource configuration, and continue to optimize based on the new evaluation results;

[0082] Dynamically adjust computing resources according to the new evaluation results to optimize the configuration of computing resources. The expression is:

[0083] ,

[0084] ,

[0085] in, Indicates the amount of adjustment in the number of computing resources, calculated by To decide whether to increase or decrease computing resources, Indicates the new number of computing resources, Indicates the amount of unneeded computing resources, Indicates the length of the time window, by setting To plan when to adjust computing resources, It represents the time point after the time point of running data plus the time window length. Indicates the number of computing resources at the time point after the time point of the running data plus the time window length. Indicates the number of computing resources at the time point of the current identification running data.

[0086] S5. Use virtual reality technology in the 3D visualization platform to provide users with an immersive experience, specifically:

[0087] Use a laser scanner to perform 3D scanning of real scenes to obtain high-precision point cloud data. Based on the point cloud data, use WebGL's 3D modeling software to create a 3D visualization model that matches the real scene. Point cloud data is a 3D data representation method consisting of a series of 3D coordinate points. Each point represents a position in space and usually includes additional information, such as color, intensity, reflectivity and other attributes, which are used to describe the characteristics of each point. Point cloud data is often used to create 3D models and generate immersive 3D environments to provide users with an immersive experience. WebGL is a 3D graphics tool for rendering interactions in web browsers. It is widely used to create visualization applications for 3D models and scenes, such as architectural models, mechanical designs, topographic maps, etc. WebGL is used to implement interactive functions such as rotation and scaling of 3D models, and to optimize performance, so as to improve the smooth operation of 3D models in browsers.

[0088] Prepare virtual reality headsets, add real sound effects and introduce tactile feedback mechanisms to simulate the feeling of touching objects, and use integrated physics engines to simulate the movement and collision of objects; tactile feedback mechanisms are a technology that enhances the human-computer interaction experience by simulating tactile sensations. Tactile sensors are used to simulate the touch of objects, detect user movements and contact with virtual objects, and trigger corresponding tactile feedback so that users can feel a tactile experience similar to that in the real world in a virtual environment. Common tactile sensors include touch screens, force-sensitive resistors, optical sensors, etc.; integrated physics engines refer to simulation engines that simulate the physical laws of the real world in software applications. Physics engines can provide realistic physical effects to make virtual environments more realistic and credible. The core functions of integrated physics engines include collision detection, support joints, and particle systems. Collision detection is responsible for simulating the physical phenomena of rebound and friction after collision. Support joints achieve connections between objects by simulating complex mechanical structures. Particle systems are responsible for simulating particle behaviors such as flames, rain, snow, and smoke to create visually realistic effects;

[0089] Integrate the real-time data when using virtual reality headsets into the 3D visualization platform, and use animation transition effects to show the changes in real-time data, so that users can intuitively understand the visualization trends of real-time data, thereby providing users with an immersive experience.

[0090] Collect user feedback and continuously adjust and optimize the visualization method of real-time integration of multivariate data based on the feedback information, specifically:

[0091] Integrate online forms, emails, and social media feedback channels into the 3D visualization platform;

[0092] Regularly send user satisfaction surveys on feedback channels to collect users' evaluations and feedback on platform functions, performance, and usability. Use machine learning algorithms to identify users' behavior on the platform and analyze their usage habits and preferences for the 3D visualization platform, thereby adjusting and optimizing the platform's functional modules and improving the user interface design.

[0093] Regularly release updated versions of the 3D visualization platform to gradually improve the platform's functions, and use A / B testing methods to compare the effects of the new and old versions to verify the effectiveness of the improvement measures;

[0094] The A / B testing method is also called split testing. It is an experimental method commonly used in product development. It determines which version can better achieve the expected goal by comparing the design and functions of two or more versions. The basic principle is to randomly divide users into two or more groups, experience different versions respectively, and then conduct statistical analysis to compare the effects of different versions. This testing method makes the experimental results not affected by other external factors, so as to draw more accurate and reliable conclusions.

[0095] This embodiment also provides a multivariate data real-time integration and visualization system, including: a data acquisition and preprocessing module, responsible for acquiring multivariate data in a database and performing preprocessing;

[0096] The data fingerprint generation module is responsible for generating data fingerprints using preprocessed multivariate data, clustering cloud data in multivariate data using data fingerprints, and generating classification features;

[0097] The data binding and classification module is responsible for binding multivariate data and cloud data using classification features, and integrating and synchronizing the bound content to the 3D visualization platform;

[0098] The real-time monitoring and resource optimization module is responsible for real-time monitoring of the actual operation of the 3D visualization platform and collecting operation data. It dynamically adjusts computing resources based on the collected operation data and optimizes the configuration of computing resources.

[0099] The 3D visualization and user interaction module is responsible for using the virtual reality technology in the 3D visualization platform to provide users with an immersive experience; the user feedback processing and optimization module is responsible for collecting user feedback and continuously adjusting and optimizing the visualization method of real-time integration of multivariate data based on the feedback information.

[0100] This embodiment also provides a computer device suitable for the case of a real-time integration and visualization method for multivariate data, comprising: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the real-time integration and visualization method for multivariate data proposed in the above embodiment.

[0101] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0102] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for realizing real-time integration and visualization of multivariate data as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0103] In summary, the present invention collects the operating data of the three-dimensional visualization platform, dynamically adjusts the computing resources, optimizes the configuration and use of the computing resources, improves the platform's operating efficiency and stability, and utilizes the virtual reality technology in the three-dimensional visualization platform to provide users with an immersive experience, thereby enhancing the user's experience and increasing the attractiveness of the three-dimensional visualization platform; by generating data fingerprints, clustering the data, and generating classification features, it is possible to efficiently process massive data, reduce data redundancy during the processing process, and improve the system's computing efficiency.

[0104] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the multivariate data real-time integration and visualization method are provided.

[0105] In order to verify the effectiveness of the method of the present invention, a test environment simulating a three-dimensional visualization platform was constructed, and two different test groups were set up, namely the control group and the method group of the present invention (experimental group). This experiment demonstrated the advantages of the present invention by comparing the performance of the two groups in data processing, resource optimization and user experience.

[0106] The experimental environment is as follows:

[0107] Hardware configuration, the server uses Intel Xeon E5-2650 v4 CPU, equipped with 128GB RAM, and uses solid-state drive SSD as storage medium;

[0108] Software configuration: The operating system is Ubuntu Server 18.04 LTS, Apache Tomcat is used as the application server, MySQL is used as the database management system, and the front end uses the AngularJS framework with WebGL for 3D rendering;

[0109] Testing tools: use ELK Stack (Logstash, Kibana) for log collection and analysis, JMeter for performance testing, and Postman for API testing;

[0110] The test data simulates a multivariate data set containing text data, image data, time series data, cloud data, and network data, with a total data volume of approximately 1TB.

[0111] Example steps:

[0112] A higher-level management system is deployed in the test environment, and multivariate data is obtained through intranet environment integration; multivariate data is obtained from multiple sources, including the company's internal database server, business system, etc.

[0113] The acquired multivariate data were preprocessed, including checking and correcting erroneous and missing values, handling missing values ​​by interpolation, unifying the data format, removing noisy data, and correcting errors in the data format;

[0114] Use the Pandas library to clean and format data to ensure data consistency and availability;

[0115] Generate data fingerprints using preprocessed multivariate data, extract key features of text keywords and numerical ranges, and encode them into fixed-length binary strings to generate data fingerprints;

[0116] Calculate the Hamming distance between different data fingerprints, and cluster the cloud data based on this distance to generate classification features;

[0117] For the clustered multivariate data, extract the classification features of the multivariate data, match the classification features of the cloud data with the multivariate data, find the multivariate data with the same classification features, and bind the cloud data with the multivariate data;

[0118] Integrate and synchronize bound content to the 3D visualization platform using RESTful API interfaces and RabbitMQ message queues;

[0119] Use Postman to test the API interface and monitor the status of the message queue through the RabbitMQ management interface;

[0120] Use ELK Stack to monitor the actual operation of the 3D visualization platform in real time, and automatically collect operation data through SDK, including performance indicators such as CPU utilization and memory utilization;

[0121] Use data analysis tools to analyze the collected operating data in real time, dynamically adjust computing resources based on the analysis results, and optimize the configuration of computing resources;

[0122] According to the formula and To set the CPU and memory utilization thresholds;

[0123] Use a laser scanner to perform 3D scanning of real scenes, obtain high-precision point cloud data, and use WebGL to create a 3D visualization model that matches the real scene;

[0124] Prepare virtual reality headsets, add realistic sound effects, and introduce tactile feedback mechanisms to simulate the feeling of touching objects;

[0125] Use Unity3D integrated physics engine to simulate the movement and collision of objects, and integrate real-time data when using virtual reality headsets into the 3D visualization platform;

[0126] Use Unity3D for 3D modeling and render the 3D model in the browser via WebGL;

[0127] Integrate online forms, emails, and social media feedback channels into the 3D visualization platform;

[0128] Regularly send user satisfaction surveys, and use machine learning algorithms to identify user behavior trajectories and analyze user usage habits and preferences;

[0129] Use Python Scikit-learn library to analyze user behavior, and adjust and optimize platform function modules based on the analysis results;

[0130] Use A / B testing method to verify the effectiveness of improvement measures.

[0131] The details are shown in Table 1 below:

[0132] Table 1 Comparative analysis of multivariate data processing and resource utilization

[0133]

[0134] Among them, the static resource allocation method group is used to simulate the performance of the system under a fixed resource configuration. This method group does not involve dynamic adjustment of resources and is used to evaluate the stability and resource utilization efficiency of the system without adjustment; the existing dynamic adjustment method group is used to simulate the dynamic adjustment method based on a fixed threshold in the prior art, and dynamically adjusts computing resources according to the overall system load; and the method group of the present invention dynamically adjusts computing resources through real-time monitoring and combining dynamic needs in the data processing process, which not only optimizes the background data processing, but also combines the needs of the three-dimensional visualization platform.

[0135] By analyzing the data in Table 1, it can be seen that the data processing time of the static resource allocation method group is 120 seconds. Due to the fixed allocation of resources, it is impossible to optimize resource usage according to real-time needs, resulting in slow data processing speed; the data processing time of the existing dynamic adjustment method group is 90 seconds, which is an improvement over the static resource allocation method, but because its dynamic adjustment mechanism is based on a fixed threshold, it is impossible to make fine adjustments according to the real-time situation of data processing, so the processing speed improvement is limited; the data processing time of the method group of the present invention is 60 seconds, and the processing speed is 50% higher than that of the existing dynamic adjustment method. This is because the present invention can go deep into the specific links of the data processing process, dynamically adjust computing resources, and maximize the optimization of resource utilization efficiency. The average CPU utilization of the static resource allocation method group is 75%, and the standard deviation is 10, indicating that the CPU utilization is high and fluctuates greatly, and some resources may be wasted or idle when processing low loads; the average CPU utilization of the existing dynamic adjustment method group drops to 70%, and the standard deviation is 8, indicating that the dynamic adjustment of the prior art can optimize resource utilization to a certain extent, but still relies on a fixed threshold, which is not very ideal when dealing with fluctuations under different loads; the average CPU utilization of the method group of the present invention is 60%, and the standard deviation is 5. The dynamic adjustment mechanism is more flexible and can automatically optimize resource allocation according to the real-time load, so that the CPU utilization is more stable and excessive occupation or waste of resources is avoided. The average memory utilization of the static resource allocation method group is 80%, and the standard deviation is 15, indicating that the memory utilization is high and fluctuates greatly, and there may be resource waste or unreasonable allocation; the average memory utilization of the existing dynamic adjustment method group is reduced to 75%, and the standard deviation is 12, and the memory management is improved, but due to the limitation of the fixed threshold, the fluctuation is still obvious; the average memory utilization of the method group of the present invention is 65%, and the standard deviation is 10, indicating that the present invention is more efficient in memory resource management, combined with the changes in real-time data, and can dynamically adjust memory allocation according to actual needs, avoiding excessive occupation and fluctuation of resources.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. A method for real-time integration and visualization of multivariate data, characterized in that: The following steps are involved: S1, obtain multi-data in the database and perform preprocessing; S2, using the preprocessed multivariate data to generate data fingerprints, clustering the cloud data in the multivariate data using the data fingerprints, and generating classification features; S3, using classification features to bind multivariate data and cloud data, and integrating and synchronizing the bound content to the 3D visualization platform; S4. Monitor the actual operation of the three-dimensional visualization platform in real time and collect operation data, dynamically adjust computing resources according to the collected operation data, and optimize the configuration of computing resources; Specifically: Determine the performance indicators, application indicators, and system health indicators to be monitored, and deploy monitoring tools. Use ELK Stack to collect and analyze the logs of the 3D visualization platform, and automatically collect operation data through SDK, and send the collected operation data to the monitoring server through a secure protocol; Use data analysis tools to analyze the collected operation data in real time. By analyzing the CPU utilization and memory utilization indicators, find out the factors that lead to poor application performance and set the CPU and memory utilization thresholds. The expressions are: ， ， in, Indicates the preset CPU utilization threshold. Indicates the time point used to identify the running data. Indicates the CPU utilization threshold that changes over time. Indicates the average CPU utilization. Indicates the standard deviation of CPU utilization. Indicates the preset memory utilization threshold. Indicates the average memory utilization. represents the standard deviation of memory utilization, and Constants representing the sensitivity of adjusting the CPU and memory utilization thresholds respectively; Check the usage of computing resource configuration according to the preset CPU and memory utilization thresholds, perform benchmark and stress tests on the 3D visualization platform, record the performance of the platform under different loads, regularly re-evaluate the usage of resource configuration, and continue to optimize based on the new evaluation results; Dynamically adjust computing resources according to the new evaluation results to optimize the configuration of computing resources. The expression is: ， ， in, Indicates the adjustment amount of the number of computing resources, Indicates the new number of computing resources, Indicates the amount of unneeded computing resources, represents the length of the time window, It represents the time point after the time point of running data plus the time window length. Indicates the number of computing resources at the time point after the time point of the running data plus the time window length. Indicates the number of computing resources at the time point of the current identification running data; S5. Use virtual reality technology in the three-dimensional visualization platform to provide users with an immersive experience, collect user feedback, and continuously adjust and optimize the visualization method of real-time integration of multivariate data based on the feedback information.

2. The method for real-time integration and visualization of multivariate data according to claim 1, characterized in that: S1 is specifically: Acquire multi-dimensional data in the database by integrating the intranet environment and accessing the upper management system; Check and correct erroneous and missing values ​​in multivariate data, handle missing values ​​by interpolation, unify data formats, remove noisy data, and correct errors in data formats.

3. The method for real-time integration and visualization of multivariate data according to claim 1, characterized in that: S2 is specifically: Extracting key features of text keywords and numerical ranges from the processed multivariate data, and encoding the extracted key features into a binary string of fixed length to generate a data fingerprint; Calculate the Hamming distance between the first and second data fingerprints, the expression is: ； in, represents the Hamming distance, represents the first data fingerprint, Represents the second data fingerprint, represents the Hamming distance between the first data fingerprint and the second data fingerprint, Indicates the number of binary digits contained in each data fingerprint, Represents the position index of the binary string, Indicates that the calculation starts from the first bit of the data fingerprint. Represents the value of the first data fingerprint at the position index of the binary string, Represents the value of the second data fingerprint at the position index of the binary string, Represents the difference between the first data fingerprint and the second data fingerprint in the position index of the binary string; The cloud data are clustered based on the calculated Hamming distance, and cloud data with high similarity are classified into the same category, thereby generating classification features.

4. The method for real-time integration and visualization of multivariate data according to claim 1, characterized in that: S3 is specifically: Extract classification features of multivariate data from clustered multivariate data; Match the classification features of cloud data and multivariate data, find multivariate data with the same classification features, and bind cloud data and multivariate data; The bound content is integrated and synchronized to the 3D visualization platform by using API interfaces and message queues.

5. The method for real-time integration and visualization of multivariate data according to claim 1, characterized in that: In S5, virtual reality technology in the 3D visualization platform is used to provide users with an immersive experience, specifically: Use a laser scanner to perform 3D scanning of real scenes to obtain high-precision point cloud data; based on the point cloud data, use WebGL's 3D modeling software to create a 3D visualization model that matches the real scene; Prepare VR headsets, add realistic sound effects and introduce haptic feedback mechanisms to simulate the feeling of touching objects, and use an integrated physics engine to simulate object movement and collisions; Integrate the real-time data when using virtual reality headsets into the 3D visualization platform, use animation transition effects to show the changes in real-time data, and provide users with an immersive experience.

6. The method for real-time integration and visualization of multivariate data according to claim 5, characterized in that: Collect user feedback and continuously adjust and optimize the visualization method of real-time integration of multivariate data based on the feedback information, specifically: Integrate online forms, emails, and social media feedback channels into the 3D visualization platform; Regularly send user satisfaction surveys on feedback channels to collect users' evaluations and feedback on platform functions, performance, and usability. Use machine learning algorithms to identify users' behavior on the platform and analyze their usage habits and preferences for the 3D visualization platform, thereby adjusting and optimizing the platform's functional modules and improving the user interface design. Regularly release updated versions of the 3D visualization platform to gradually improve the platform functions, and use A / B testing methods to compare the effects of the new and old versions to verify the effectiveness of the improvement measures.

7. A multivariate data real-time integration and visualization system, characterized in that: For implementing the multivariate data real-time integration and visualization method according to any one of claims 1 to 6, the system comprises: The data acquisition and preprocessing module is responsible for acquiring multi-dimensional data in the database and performing preprocessing; The data fingerprint generation module is responsible for generating data fingerprints using preprocessed multivariate data, clustering cloud data in multivariate data using data fingerprints, and generating classification features; The data binding and classification module is responsible for binding multivariate data and cloud data using classification features, and integrating and synchronizing the bound content to the 3D visualization platform; The real-time monitoring and resource optimization module is responsible for real-time monitoring of the actual operation of the 3D visualization platform and collecting operation data. It dynamically adjusts computing resources based on the collected operation data and optimizes the configuration of computing resources. Specifically: Determine the performance indicators, application indicators, and system health indicators to be monitored, and deploy monitoring tools. Use ELK Stack to collect and analyze the logs of the 3D visualization platform, and automatically collect operation data through SDK, and send the collected operation data to the monitoring server through a secure protocol; Use data analysis tools to analyze the collected operation data in real time. By analyzing the CPU utilization and memory utilization indicators, find out the factors that lead to poor application performance and set the CPU and memory utilization thresholds. The expressions are: ， ， in, Indicates the preset CPU utilization threshold. Indicates the time point used to identify the running data. Indicates the CPU utilization threshold that changes over time. Indicates the average CPU utilization. Indicates the standard deviation of CPU utilization. Indicates the preset memory utilization threshold. Indicates the average memory utilization. represents the standard deviation of memory utilization, and Constants representing the sensitivity of adjusting the CPU and memory utilization thresholds respectively; Check the usage of computing resource configuration according to the preset CPU and memory utilization thresholds, perform benchmark and stress tests on the 3D visualization platform, record the performance of the platform under different loads, regularly re-evaluate the usage of resource configuration, and continue to optimize based on the new evaluation results; Dynamically adjust computing resources according to the new evaluation results to optimize the configuration of computing resources. The expression is: ， ， in, Indicates the adjustment amount of the number of computing resources, Indicates the new number of computing resources, Indicates the amount of unneeded computing resources, represents the length of the time window, It represents the time point after the time point of running data plus the time window length. Indicates the number of computing resources at the time point after the time point of the running data plus the time window length. Indicates the number of computing resources at the time point of the current identification running data; The 3D visualization and user interaction module is responsible for using the virtual reality technology in the 3D visualization platform to provide users with an immersive experience; The user feedback processing and optimization module is responsible for collecting user feedback and continuously adjusting and optimizing the visualization method of real-time integration of multivariate data based on the feedback information.

Images