Multimodal data visualization methods, storage media, and electronic devices

Abstract

This invention discloses a multimodal data chart visualization method, including a two-dimensional multimodal data chart displayed in a two-dimensional screen space, and a data chart displayed in a three-dimensional space in the form of a bulletin board. Essentially, the two-dimensional screen space chart is understood as a chart pasted onto a specific position on a fixed plane in three-dimensional space. The method also includes a multimodal data chart visualization method based on this chart format, enabling real-time, seamless switching between the two-dimensional and three-dimensional screen space charts. Through steps such as acquiring multimodal data from a three-dimensional model, calculating the chart position in three-dimensional space, determining occlusion relationships, automatically adjusting occlusion, perspective switching, and dynamic interpolation paths, the method achieves dynamic visualization and analysis of multimodal data charts in two-dimensional and three-dimensional scenes based on GPU texture data. This invention solves the problems of existing chart display methods being monotonous, chart position occlusion, and unnatural perspective switching, thus improving the interactivity and practicality of data visualization.

Description

Technical Field

[0001] This invention relates to data visualization technology, and in particular to a method, storage medium, and electronic device for real-time interactive switching analysis of multimodal data charts in two-dimensional screen space and three-dimensional spatial scenes. Background Technology

[0002] Of all the human sensory organs, the human eye possesses the strongest signal processing capability. Vision is the most important channel for acquiring information; over 50% of the human brain's function is dedicated to visual perception, including decoding visual information and processing visual symbols. However, the human visual processing memory is limited, often requiring external assistance during the cognitive process. Images and graphics have been the most important tools in human cognition for the past two centuries. Visualization is a cognitive view of things generated during the cognitive process, enhancing understanding and facilitating knowledge exchange. The application of visualization techniques has a long history; in the Middle Ages, people used geomagnetic maps with isolines, arrows indicating major wind directions at sea, and celestial maps. Currently, visual analytics has been developed to address the challenges of analyzing massive, high-dimensional, multi-source, and dynamic data. This is the origin of the development of data visualization. The challenges stem not only from the sheer volume, high dimensionality, diverse sources, and polymorphism of data, but more importantly from the dynamic nature of data acquisition, the noise and contradictions in data content, and the heterogeneity and heterogeneity of data relationships.

[0003] The role of data visualization lies in gaining knowledge through observation, that is, from seeing objects to acquiring knowledge. For complex, large-scale data, existing statistical analysis or data mining methods often simplify and abstract the data, hiding the true structure of the dataset. Data visualization, on the other hand, can restore or even enhance the global structure and specific details within the data. If data visualization is viewed as an artistic creation process, then it needs to achieve a balance between truth, goodness, and beauty, effectively mining, disseminating, and communicating the information, knowledge, and ideas contained within the data, and achieving a balance between design and function. In this sense, data visualization embodies the function of broadening one's understanding and gaining knowledge.

[0004] Multimodal data refers to data composed of multiple modes or sources. For example, a 3D model may contain data in multiple modes such as shape, color, and texture. Multimodal data charts are a method of visualizing multimodal data, which can display multimodal data in chart form, enabling people to understand and analyze the data from multiple perspectives.

[0005] In current data visualization technologies, multimodal data charts are generally displayed using a two-dimensional screen space. However, this display method has some limitations, such as the inability to vividly present the spatial structure of three-dimensional models and the inability to observe data from multiple angles. To overcome these problems, researchers have begun to explore integrating multimodal data charts into three-dimensional spatial scenes. For example, patent application CN113079411A discloses a multimodal data synchronous visualization system. When collecting EEG signals, synchronously recorded evoked videos and videos of the subject's facial expressions are superimposed on the EEG waveform. The recorded eye gaze positions are marked on the recorded video displaying the evoked images, displayed as fixed-size circles. This allows researchers to detect the subject's eye movements and current state simply by observing the more easily noticeable marked shapes and video image information. Patent application CN108710628A discloses a sketch-based interactive... The proposed visual analysis method for multimodal data divides the original dataset into various visual data structures and matches these structures with visual forms. Based on the mapping relationship from the original dataset to the visual data structure to the visual form, the original dataset is decomposed into multiple related information aspects. Each information aspect is presented in a view through a visual form, and combined with the layout information selected by the user, a multi-view related view is generated. The method identifies the sketch symbols drawn by the user when performing sketch selection operations in the multi-view related view according to the target analysis requirements, analyzes the meaning of the sketch gestures composed of sketch symbols, and generates a new view based on the meaning of the sketch gestures.

[0006] Existing methods for integrating multimodal data charts into 3D spatial scenes often require manual adjustment of the chart's position and orientation. This is cumbersome and makes it difficult to ensure that the chart's position and orientation are always optimal. Furthermore, when there are many charts, occlusion issues may occur. Additionally, when the viewing perspective changes, the chart's orientation may need to be manually adjusted to adapt to the new perspective, which undoubtedly increases the complexity of the operation. Existing methods typically only focus on displaying data in 2D screen space or 3D spatial scenes, lacking sufficient interactivity and practicality.

[0007] Therefore, exploring a method to enable automatic interactive switching of multimodal data charts in two-dimensional screen space and three-dimensional space scene, as well as solving problems such as chart position occlusion and unnatural perspective transitions, has become an important challenge that current data visualization technology urgently needs to address. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal data chart visualization method based on real-time interactive seamless switching between two-dimensional and three-dimensional spaces. This method involves displaying two-dimensional multimodal data charts in a two-dimensional screen space and data charts displayed in a three-dimensional space in the form of a bulletin board. In other words, the two-dimensional screen space chart can be understood as a chart pasted on a fixed plane at a specific position in three-dimensional space. The invention also provides a multimodal data chart visualization method based on this chart form, which allows for real-time interactive seamless switching between two-dimensional screen space and three-dimensional space charts.

[0009] This invention proposes a method for visualizing multimodal data charts in two-dimensional and three-dimensional spaces based on bidirectional real-time interaction, enabling users to switch and display charts in real time between two-dimensional screen space and three-dimensional spatial scenes. This method is applicable to multimodal data charts, such as those with various data attributes like shape, color, and texture, improving the dimensionality and efficiency of data analysis. The invention employs the following technical solution:

[0010] A method for visualizing multimodal data charts based on real-time interactive seamless switching between two-dimensional and three-dimensional spaces. The method involves displaying two-dimensional multimodal data charts in two-dimensional screen space and displaying data charts in three-dimensional solid space in the form of a bulletin board. In other words, the two-dimensional screen space chart is understood as a chart pasted on a fixed plane at a specific position in three-dimensional space. The method also includes a method for visualizing multimodal data charts based on this chart form, which allows for real-time interactive seamless switching between two-dimensional screen space and three-dimensional solid space charts.

[0011] During real-time interaction, steps S1 to S6 are executed to switch from a two-dimensional screen space chart to a three-dimensional space visualization and vice versa.

[0012] S1. During real-time interaction, obtain multimodal data charts, select the chart display format through real-time interaction with the user, and select the corresponding seamless switching mode for real-time interaction in two-dimensional and three-dimensional space according to the specified display format.

[0013] S2. Obtain the current position of the chart and the placement position of the target chart. First, obtain and record the starting position of the currently displayed chart in the three-dimensional space. Then, obtain and record the placement position of the target chart. The method for obtaining the placement position of the target chart includes calculating the possible placement positions of the target chart in the corresponding space based on the two-dimensional or three-dimensional space characteristics of the target chart. After calculating the possible placement positions of the target chart, determine whether the bounding box of the chart position intersects with the bounding boxes of other charts and obstructs each other. Automatically adjust the position and chart orientation to avoid obstruction.

[0014] S3. Collect and integrate multimodal data. Based on the characteristics of the multimodal data and visualization requirements, select appropriate chart types to display the data, call the rendering engine, draw dynamic multimodal data charts onto the GPU texture data, and update the pixel data in the GPU texture in real time according to the changes in the data.

[0015] S4. Based on the starting position of the chart and the target chart placement position obtained in step S2, calculate the translation and rotation dynamic interpolation path from the current position of the chart to the final target placement position of the chart. Use a variety of effective interpolation methods to calculate the path to achieve a smooth and natural switching process.

[0016] S5. During the switching process of multimodal data charts, dynamic multimodal data charts are drawn in three-dimensional space based on real-time updated GPU texture data and dynamic path interpolation data.

[0017] S6. After reaching the target position of the multimodal data chart, the chart position remains stationary, but the chart content continues to be dynamically refreshed.

[0018] Based on the above technical solutions, the present invention may also employ the following further technical solutions, or combine these further technical solutions:

[0019] In step S2, the starting position of the chart and the placement position of the target chart are obtained. First, the starting position of the currently displayed chart is obtained and recorded. Then, the placement position of the target chart is obtained. The method for obtaining the placement position of the target chart includes calculating the possible placement positions of the target chart in the corresponding space based on the two-dimensional or three-dimensional spatial characteristics of the target chart. After calculating the possible placement positions of the target chart, it is determined whether the bounding box of the chart position intersects with the bounding boxes of other charts and thus obstructs each other. The position and chart orientation are automatically adjusted to avoid obstruction. Specifically, the following steps are included:

[0020] S2.1 Obtain the current position of the chart. Based on the spatial characteristics of the chart selected by the user in real-time interaction and the chart's position in the current space, generate the current three-dimensional spatial position of the chart and record it for later use.

[0021] S2.2, Obtain and record the placement position of the target chart. The method for obtaining the placement position of the target chart includes calculating the possible placement position of the target chart in the corresponding space based on the two-dimensional or three-dimensional spatial characteristics of the target chart. After calculating the possible placement position of the target chart, it is determined whether the bounding box of the chart position intersects with the bounding boxes of other charts and thus occludes them. The position and chart orientation are automatically adjusted to avoid occlusion. This includes the following steps:

[0022] S2.2.1, Based on the two-dimensional or three-dimensional spatial characteristics of the target chart, calculate the possible placement of the target chart in the corresponding space;

[0023] S2.2.2 uses a combination of methods to determine whether the bounding box of a chart intersects with or occludes with other chart bounding boxes. One specific method is to use the separating axis theorem to detect whether collisions occur between convex polygons (such as AABB, OBB, spheres, etc.). The main idea is that if there exists a separating axis such that the projections of two bounding boxes do not overlap on the projection of that axis, then the two bounding boxes are separated in 3D space and do not intersect or occlude.

[0024] S2.2.3, if occlusion exists, the chart position is automatically adjusted until occlusion is eliminated. Simulated annealing is used for occlusion adjustment; simulated annealing is a probabilistic search algorithm used to solve global optimization problems. Its main idea comes from the simulation of the solid annealing process—after a solid is heated, particles move randomly, and then as it cools slowly, the particles gradually tend towards an ordered state.

[0025] S2.2.4 automatically adjusts the orientation of the chart target placement based on the change in viewpoint in 3D space. This is achieved using a method that integrates the camera chart viewpoint and chart occlusion relationships. Quaternions and camera control are used to realize viewpoint transitions and target placement orientation changes. Quaternions are a mathematical concept that extends complex numbers and can be used to represent rotations in 3D space. Compared to Euler angles, quaternions avoid gimbal lock problems, provide a more stable rotation representation, and can smoothly transition between two rotations through interpolation.

[0026] Preferably, if occlusion exists in step S2.2.3, the chart position is automatically adjusted until occlusion is eliminated. Simulated annealing algorithm is used for occlusion adjustment, including the following steps:

[0027] (a) Initialization: Set the initial temperature, cooling coefficient (typically less than 1 and close to 1), and termination temperature. Randomly select a reasonable initial chart location and calculate its occlusion level as the initial cost;

[0028] (b) Iterative process: Repeat the following steps as the temperature decreases from the initial temperature to the final temperature:

[0029] (ba) Perturbation: Randomly select a new location, which can be a random perturbation at the current location;

[0030] (bb) If the new location is unobstructed or the obstruction is reduced, then accept the new location; otherwise, accept the new location with a certain probability, where the probability is determined by the current temperature and the degree of obstruction between the old and new locations.

[0031] (bc) Cooling: Update the temperature, usually by multiplying it by a factor less than 1 to simulate the cooling process;

[0032] (c) End iteration: The iteration ends when the temperature drops below the termination temperature. The current solution is returned as the final solution.

[0033] Preferably, step S2.2.4 automatically changes the orientation of the chart target placement based on the perspective change in three-dimensional space. This uses a method that integrates the camera chart perspective and chart occlusion relationships, employing quaternions and camera control to achieve perspective change and target placement orientation change, including the following steps:

[0034] (a) Initialize the scene and objects: First, it is necessary to obtain the current world space position of the camera and the target icon in the 3D scene, as well as the camera's viewpoint and the target icon's orientation;

[0035] (b) Calculate the rotation quaternion: Based on the camera's viewpoint and the target chart's orientation, calculate the rotation quaternion of the target chart under the new viewpoint. Quaternion multiplication can be used to combine multiple rotation operations;

[0036] (c) Interpolation Transition: To achieve a smooth perspective transition, spherical linear interpolation can be used to interpolate between the current quaternion and the target quaternion. The interpolation process can be configured with transition time and speed as needed. This method calculates the quaternion dot product, determines the interpolation direction, calculates the interpolation angle, and applies a quaternion scheme that adjusts the interpolation effect based on time to achieve a smooth time-based angle transition.

[0037] (d) Update object orientation: Apply the interpolated quaternion to the rotation transformation of the object to update the object's orientation in real time.

[0038] In step S4, based on the starting position of the chart and the target chart placement position obtained in step S2, a dynamic interpolation path of translation and rotation from the current position of the chart to the final target placement position is calculated. Multiple effective interpolation methods are used to calculate the path to achieve a smooth and natural switching process, including the following steps:

[0039] S4.1, Define the start and end positions of the chart: Based on the start position of the chart obtained in step S2 and the target chart placement position, define the start and end positions of the chart. These two positions can be represented as two-dimensional or three-dimensional vectors;

[0040] S4.2, Define control points: Before calculating the spline shape, a set of control points needs to be defined. The shape of the spline is determined by a set of control points. At least four control points (including the start point and the end point) should be determined, but more control points are usually generated by random sampling to achieve more complex curve shapes.

[0041] S4.3, Calculating the spline curve equation: Given a set of control points and a parameter t (ranging from 0 to 1), several spline calculation schemes can be used to calculate the spline curve equation. For example, one method uses the Catmull-Rom spline interpolation method. The formula for calculating any point on the spline is as follows: C(t) = 0.5 * [(2 * P1) + (-P0 + P2) * t + (2 * P0 - 5 * P1 + 4 * P2 - P3) * t^2 + (-P0 + 3 * P1 - 3 * P2 + P3) * t^3]. Where P0, P1, P2, and P3 are four adjacent control points, t is the parameter, and C(t) is the position of a point on the Catmull-Rom spline curve calculated based on the parameter t.

[0042] S4.4, Calculate translation and rotation: Use spline curves to calculate the translation and rotation of the chart. Translation is determined directly by the points on the curve, while rotation is determined by calculating the tangent to the curve. The direction of the tangent is the direction of rotation, and the tangent is calculated using the derivative of the spline curve.

[0043] S4.5, Dynamic Interpolation: Finally, the position and rotation of the chart are dynamically changed by changing the value of parameter t. When t changes from 0 to 1, the chart will move from the starting position to the ending position along the spline curve we calculated, while rotating accordingly.

[0044] Preferably, step S5 includes the following specific steps:

[0045] S5.1, Prepare rendering parameters: Based on the dynamic path interpolation data calculated in the previous steps, including translation vectors and rotation vectors, in order to realize the translation and rotation animation effects of the chart in three-dimensional space.

[0046] S5.2 Configure camera parameters and view settings in the 3D rendering environment to observe dynamic multimodal data graphs at appropriate angles and viewing distances;

[0047] S5.3 applies the GPU texture data of the rendered dynamic multimodal data chart to a 3D chart model, maps it to a 3D planar model, and associates the texture coordinates with the model vertex information.

[0048] S5.4, Apply dynamic path interpolation data to the 3D chart model based on the previously calculated translation and rotation vectors, and generate a new transformation matrix for the 3D chart model;

[0049] S5.5 plots dynamic multimodal data graphs in three-dimensional space based on camera, view, and dynamic transformation settings.

[0050] In a second aspect of the present invention, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described multimodal data chart visualization method based on real-time interactive seamless switching in two- and three-dimensional space.

[0051] According to a second aspect of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described multimodal data chart visualization method based on real-time interactive seamless switching in two-dimensional and three-dimensional space.

[0052] The beneficial effects of this invention are as follows: Addressing the display needs of complex multimodal data associations and structures, and overcoming the limitations of existing technologies, this invention introduces a comprehensive set of technologies, including spatial acceleration structures, parallel computing, computational caching, and GPU computing, as well as an automatic mechanism for calculating the placement of charts in 2D and 3D screen space. It also automatically handles chart occlusion mechanisms, smooth perspective transitions in 3D space, and target orientation changes, thereby improving the utilization rate and interactivity of the visualization space. Furthermore, this invention introduces various spline interpolation techniques to calculate translational and rotational dynamic interpolation paths, achieving a smooth transition from 2D screen space to 3D space, enhancing the visual experience of data visualization, and providing a better real-time interactive experience for users. This method improves space utilization efficiency and interactivity, overcoming the limitations of traditional 2D charts in displaying multimodal data, thus achieving a more intuitive and efficient data visualization effect and optimizing the user experience. Experimental test results show that this invention can achieve seamless and smooth switching between 2D and 3D spaces for hundreds of multimodal data charts on a moderately powerful ordinary PC, with a system frame rate exceeding 60 frames per second, fully achieving real-time performance and meeting the performance requirements for smooth user interaction. This invention improves the efficiency and user experience of real-time interaction of multimodal data charts, and can be easily integrated into any mature and complete drawing engine. The invention has significant application potential. Attached Figure Description

[0053] Figure 1 This is a schematic flowchart of the method of the present invention;

[0054] Figure 2 This is a visualization of the multimodal data chart in three-dimensional space in an embodiment of the present invention;

[0055] Figure 3 This is a visualization of the seamless and smooth switching of multimodal data charts in two-dimensional and three-dimensional space in an embodiment of the present invention;

[0056] Figure 4This is a visualization of the multimodal data chart in two-dimensional space in an embodiment of the present invention. Detailed Implementation

[0057] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the invention is not limited to the specific embodiments disclosed below.

[0058] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0059] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0060] The flowchart of the multimodal data chart visualization method based on real-time interactive seamless switching in two- and three-dimensional space in this embodiment is as follows: Figure 1 As shown, its implementation method is specifically divided into the following 6 steps, including four steps of visualization calculation for each frame during real-time interaction and two preparatory steps before that:

[0061] (1) Define the data source: Determine the source of multimodal data; at this stage, it is necessary to determine the various sources that provide data for the model and process the data.

[0062] This embodiment supports determining applicable multimodal data sources. At this stage, it is necessary to identify the various sources that provide data to the model. These include, but are not limited to, the following types: real-time sensor data, such as data from various types of sensors including temperature sensors, pressure sensors, and humidity sensors; database information, such as relational databases and non-relational databases, which can provide historical data, device information, operating parameters, etc.; and image and video data, which can be used for tasks such as artificial intelligence recognition and computer vision.

[0063] Simultaneously, the system ensures the real-time performance of data sources. Through precise clock synchronization and low-latency communication networks, it guarantees time consistency across various data sources. Appropriate data transmission protocols are selected based on factors such as communication environment stability and transmission speed to ensure real-time data transmission. Data source reliability is guaranteed through data preprocessing. For different data types, appropriate preprocessing methods are used to clean, denoise, and interpolate the raw data, removing invalid and abnormal data. Data correction and verification involve validating the collected data and correcting and verifying any outliers or errors to ensure the accuracy of the data sources. Data backup and redundancy strategies ensure that critical data can still be obtained even if a data source fails, thus maintaining the reliability of the entire system.

[0064] (2) Design a data visualization scheme: To display multimodal data charts, design corresponding data visualization components and styles. Factors to consider include data type, chart type, color, layout, and typography. Appropriately design visualization elements to improve user experience and data analysis effectiveness.

[0065] In this embodiment, multimodal data charts will select appropriate chart types based on data characteristics and display requirements, such as line charts, bar charts, pie charts, scatter plots, and heatmaps. Color will be used appropriately, with different colors representing different categories or trends to improve data readability; at the same time, visual communication standards will be followed to avoid color discrepancies and misleading information, using a color scheme without color differences. A clear and reasonable layout will be designed to maintain balance between charts, labels, and other elements, making them easy to distinguish and read; the spatial distribution will be adjusted according to the characteristics of the data charts to ensure appropriate margins and spacing. Font, font size, line spacing, and other typographic details will be consistent to improve visual effects and reading comfort; attention will also be paid to hierarchical relationships, appropriately using bolding, italics, and other typographic techniques. Animations, dynamic effects, or real-time refreshes can be introduced as needed to make data visualization more dynamic; a user-friendly interactive interface will be designed, including chart filtering, zooming, and highlighting functions, facilitating in-depth analysis through user operations. Different device and terminal types will be considered to achieve adaptability and responsive display of visualization components in various scenarios, improving user experience. Pay attention to performance factors such as rendering speed and loading latency, and take appropriate measures to optimize the performance of visualization components and reduce user waiting time.

[0066] In the subsequent real-time interaction process, the visualization calculation for each frame requires the execution of steps S1 to S6:

[0067] S1. During real-time interaction, obtain multimodal data charts, select the chart display format through real-time interaction with the user, and select the corresponding seamless switching mode for real-time interaction in two-dimensional and three-dimensional space according to the specified display format.

[0068] In this embodiment, during real-time interaction, various user operations are captured and multimodal data charts of the 3D model are dynamically collected and optimized according to the interaction methods supported by the device. These charts are then presented to the user in real time with high-quality visualization components and styles to provide in-depth data analysis, information delivery, and real-time feedback.

[0069] S1.1, Capture User Interactions: During real-time interaction, record various user actions such as clicking, dragging, zooming, and selecting. Captured data can be used to analyze user behavior, improve data visualization, or trigger other functions.

[0070] S1.2, Real-time update of model and data charts: Based on user interaction, update the multimodal data of the manipulated objects in the 3D model to ensure the real-time transmission and processing of data, so that users can obtain timely feedback and results during the interaction process;

[0071] S1.2, Based on the chart display format selected by the user in real time and the display format of the specified target chart, select the corresponding real-time interactive seamless switching method between two-dimensional and three-dimensional spaces. For example, if the user selects a chart in two-dimensional screen space and issues a command to switch the display space, the system will select the seamless switching method from two-dimensional screen space to three-dimensional space and perform the switching operation on the target chart.

[0072] S2. Obtain the current position of the chart and the placement position of the target chart; First, obtain and record the starting position of the currently displayed chart in 3D space, then obtain and record the placement position of the target chart; The method for obtaining the placement position of the target chart includes calculating the possible placement positions of the target chart in the corresponding space based on the characteristics of the 2D or 3D space in which the target chart is located; after calculating the possible placement positions of the target chart, determining whether the bounding box of the chart position intersects with the bounding boxes of other charts and thus obscures each other, and automatically adjusting the position and chart orientation to avoid obstruction; The main steps include:

[0073] S2.1 Obtain the current position of the chart. Based on the spatial characteristics of the chart selected by the user in real-time interaction and the chart's position in the current space, generate the current three-dimensional spatial position of the chart and record it for later use.

[0074] S2.2, Obtain and record the placement position of the target chart. The method for obtaining the placement position of the target chart includes calculating the possible placement position of the target chart in the corresponding space based on the two-dimensional or three-dimensional spatial characteristics of the target chart. After calculating the possible placement position of the target chart, it is determined whether the bounding box of the chart position intersects with the bounding boxes of other charts and thus occludes them. The position and chart orientation are automatically adjusted to avoid occlusion. This includes the following steps:

[0075] S2.2.1, Based on the two-dimensional or three-dimensional spatial characteristics of the target chart, calculate the possible placement of the target chart in the corresponding space;

[0076] S2.2.2, Multiple methods are used to determine whether the bounding box of a chart intersects with or occludes with other chart bounding boxes. One specific method used in this embodiment is to use the separating axis theorem to detect collisions between convex polygons (such as AABB, OBB, spheres, etc.). The main idea is that if there exists a separating axis such that the projections of two bounding boxes do not overlap on the projection of this axis, then these two bounding boxes are separated in 3D space and do not intersect or occlude. The specific process of step S2.2.2 can be divided into the following sub-steps:

[0077] (a) Identify all potential separation axes; when processing OBB, calculate its 15 potential separation axes.

[0078] (b) Calculate the projection for each potential separation axis by first calculating the projection of the center points of A and B onto the axis, and then calculating half of the extension range of A and B on that axis.

[0079] (c) Check for overlap. For each separation axis, check whether the projections of the two OBBs overlap. If there exists a separation axis such that the absolute value of the projection difference is greater than the sum of the extended ranges of the two separation axes, then the projections on that separation axis do not overlap, and the two OBBs do not collide in 3D space.

[0080] (d) Determine if they intersect. If the projections on all the separation axes overlap, then the two OBBs intersect.

[0081] (e) If the length of the separating axis vector obtained by calculating the cross product is close to zero (e.g., the length is less than a certain threshold), then ignore this separating axis, because the separating axis may be approximately parallel in this case. To check if the length of the separating axis vector is close to zero, compare it with the square of the threshold length to avoid performing a square root calculation.

[0082] S2.2.3. If there is occlusion, automatically adjust the chart position until there is no occlusion. Use the simulated annealing algorithm to perform occlusion adjustment. Simulated annealing is a probabilistic search algorithm for solving global optimization problems. Its main idea comes from the simulation of the solid annealing process - after the solid is heated, the particles move randomly, and then slowly cool down, and the particles gradually tend to an ordered state. The specific process of step S2.2.3 can be divided into the following sub-steps:

[0083] (a) Initialization: Set the initial temperature T_initial, the cooling factor cooling_factor (usually less than 1 and close to 1), and the termination temperature T_final. Randomly select a reasonable initial chart position current_solution and calculate its occlusion degree as the initial cost current_cost;

[0084] (b) Iteration process: Repeat the following steps during the process of the temperature decreasing from T_initial to T_final:

[0085] (ba) Perturbation: Randomly select a new position new_solution, which can be a random perturbation of the current position current_solution; <0*********1> (bb) Calculate the cost change, judge whether it is a new solution and update the solution and cost: Calculate the occlusion degree new_cost of the new position new_solution. Calculate the cost change delta_cost = new_cost - current_cost. If delta_cost <= 0 (that is, the new solution is better than or equal to the current solution), accept the new solution; otherwise, accept the new solution with a certain probability. The acceptance probability is acceptance_prob = exp(-delta_cost / T), where exp represents the exponential function. A random number random_number (uniformly distributed between 0 and 1) can be used to judge whether to accept the new solution: If random_number < acceptance_prob, then accept the new solution, otherwise keep the current solution. If the new solution is accepted, update current_solution = new_solution and current_cost = new_cost;

[0087] [[ID=**15]](bc) Cooling: Update the temperature T = T * cooling_factor to simulate the cooling process; [[ID=1*********5> (c) End iteration: When the temperature drops below T_final, end the iteration. Return the current solution as the final solution.

[0089] S2.2.4, Automatically change the orientation of the chart target placement based on the perspective change in 3D space; A method integrating camera chart perspective and chart occlusion relationship is used, through quaternions and camera control, to realize perspective change and target placement orientation change. Quaternions are a mathematical concept that extends complex numbers and can be used to represent rotation in 3D space. Compared to Euler angles, quaternions can avoid gimbal lock problems, provide a more stable rotation expression, and quaternions can smoothly transition between two rotations through interpolation; The specific process of step S2.2.4 can be divided into the following sub-steps:

[0090] (a) Initialize the scene and objects: First, it is necessary to obtain the current world space position of the camera and the target icon in the 3D scene, as well as the camera's viewpoint and the target icon's orientation;

[0091] (b) Calculate the rotation quaternion: Based on the camera's viewpoint and the target chart's orientation, calculate the rotation quaternion of the target chart under the new viewpoint. Quaternion multiplication can be used to combine multiple rotation operations;

[0092] (c) Interpolation Transition: This implementation employs multiple methods to achieve smooth perspective transitions between quaternions. One method uses spherical linear interpolation to interpolate between the current quaternion and the target quaternion. The interpolation process allows for adjustable transition time and speed. This method calculates the quaternion dot product, determines the interpolation direction, calculates the interpolation angle, and applies a quaternion scheme that adjusts the interpolation effect based on time, achieving a smooth time-based perspective transition.

[0093] (d) Update object orientation: Apply the interpolated quaternion to the rotation transformation of the object to update the object's orientation in real time.

[0094] S3. Based on the acquired multimodal data chart content, call the rendering engine to draw the dynamic multimodal data chart onto the GPU texture data, and update the pixel data in the GPU texture in real time according to the data changes; the specific process of step S3 can be divided into the following sub-steps:

[0095] S3.1 Mapping Data to Chart Elements: For the selected chart type (such as line chart, bar chart, pie chart, etc.), map the multimodal data to the various elements of the chart. This includes determining the correspondence and numerical range of visual elements such as axes, colors, and labels;

[0096] S3.2, Create GPU Texture: Create a texture space in the GPU to store the graph's pixel data. Set the texture size and color format to match the graph's size, precision, and color depth.

[0097] S3.3, Rendering the graph to a texture: This step calls the rendering engine to render the graph's pixel data to the GPU texture. This includes creating a vertex buffer based on the graph's vertex data and uploading it to the GPU, setting the rendering pipeline state, setting appropriate blending modes for the rendering process, associating the created GPU texture with the graph, and then calling the rendering engine to render the graph's pixel data to the GPU texture.

[0098] S3.4 Based on data changes, the pixel data in the GPU texture is updated in real time to reflect the dynamic changes in the data. Multimodal data changes are monitored in real time through event listeners. When data changes, the chart elements are remapped and rendered locally, and the changed chart pixel data is uploaded to the GPU texture in real time to reflect the dynamic changes in the data.

[0099] S4. Based on the starting position of the chart and the target chart placement position obtained in step S3, calculate the translation and rotation dynamic interpolation path from the current position of the chart to the final target placement position of the chart. Use a variety of effective interpolation methods to calculate the path to achieve a smooth and natural switching process.

[0100] In this embodiment, step S4 uses various spline curve interpolation methods to calculate the path. One method is the Catmull-Rom spline interpolation method, a smooth interpolation method that can calculate a smooth curve using a set of control points. In this invention, the control points can be the starting position, target position, and intermediate position of the chart. Using spline interpolation, the translation and rotation paths of the chart during the switching process can be calculated to achieve a smooth and natural switching process. The specific process can be divided into the following sub-steps:

[0101] S4.1, Define the start and end positions: Based on the start position of the chart obtained in step S2 and the target chart placement position, define the start position startPos (e.g., in two-dimensional screen space) and end position endPos (e.g., in three-dimensional space). These two positions can be represented as two-dimensional or three-dimensional vectors;

[0102] S4.2, Define control points: Before calculating the spline shape, it is necessary to define a control point group, controlPoints. The shape of the spline is determined by a group of control points. At least four control points (including the start point and the end point) should be determined, but more control points are usually generated by random sampling to achieve more complex curve shapes. When determining control points, it is necessary not only to select the total number of control points, but also to determine the specific location of each control point.

[0103] S4.3, Calculate the spline curve equation: Given a set of control points and a parameter t (ranging from 0 to 1), use some spline calculation schemes to calculate the spline curve equation. For example, in this embodiment, a Catmull-Rom spline interpolation method is used. The formula for calculating any point on the spline is as follows: C(t) = 0.5 * [(2 * P1) + (-P0 + P2) * t + (2 * P0 - 5 * P1 + 4 * P2 - P3) * t^2 + (-P0 + 3 * P1 - 3 * P2 + P3) * t^3]. Where P0, P1, P2, and P3 are four adjacent control points, t is the parameter, and C(t) is the position of a point on the Catmull-Rom spline curve calculated based on the parameter t.

[0104] S4.4, Calculate translation and rotation: Use spline curves to calculate the translation and rotation of the chart. Translation is determined directly by the points on the curve, while rotation is determined by calculating the tangent to the curve. The direction of the tangent is the direction of rotation, and the tangent is calculated using the derivative of the spline curve.

[0105] S4.5, Dynamic Interpolation: Finally, the position and rotation of the chart are dynamically changed by changing the value of parameter t. When t changes from 0 to 1, the chart will move from the starting position to the ending position along the calculated spline curve, and rotate accordingly.

[0106] S5. During the switching process of multimodal data charts, dynamic multimodal data charts are drawn in three-dimensional space based on real-time updated GPU texture data and dynamic path interpolation data.

[0107] In this embodiment, step S5 utilizes the computing performance of the Nvidia RTX 2080 GPU and the UE5 graphics rendering engine to achieve the correct multimodal data chart coloring and drawing effect. The specific process can be divided into the following sub-steps:

[0108] S5.1, Prepare rendering parameters: Based on the dynamic path interpolation data (including translation vectors and rotation vectors) calculated in the previous steps, in order to realize the translation and rotation animation effects of the chart in three-dimensional space.

[0109] S5.2, Set up the camera and view: Configure camera parameters (such as camera position, viewpoint, etc.) and view settings (such as clipping plane, view frustum, etc.) in the 3D rendering environment to observe dynamic multimodal data charts at appropriate angles and viewing distances.

[0110] S5.3, Create a 3D chart model: Apply the GPU texture data of the dynamic multimodal data chart rendered in step S5 to a 3D chart model, apply the texture to a 3D planar model, and associate the texture coordinates with the vertex information of the model.

[0111] S5.4, Perform dynamic transformation: Apply dynamic path interpolation data to the 3D chart model based on the previously calculated translation and rotation vectors, and generate a new transformation matrix for the 3D chart model.

[0112] S5.5, Rendering Dynamic Multimodal Data Charts: Based on camera, view, and dynamic transformation settings, dynamic multimodal data charts are drawn into 3D space. During rendering, the rendering order is followed to prevent occlusion issues, and parameters such as transparency and color are appropriately adjusted for better visual effects.

[0113] S6. After reaching the target position of the multimodal data chart, the chart position remains stationary, but the chart content continues to be dynamically refreshed.

[0114] S6.1, Target Position Monitoring: Detects when the dynamic multimodal data chart reaches the target position, i.e., when the translation and rotation vectors reach the preset endpoint parameters. This is based on whether the interpolation parameter `t` is close to or equal to 1 (indicating the end position).

[0115] S6.2, Fix Chart Position and Rotation: Once the multimodal data chart has reached the target position, set its position and rotation to a static state. This means stopping interpolation calculations along the path and maintaining stable position and rotation values.

[0116] S6.3, Continuous Rendering and Chart Content Updates: Although the chart position and rotation remain static, the GPU texture data still needs to be updated in real time to dynamically refresh the chart content. The rendering engine is invoked to continuously redraw the chart and draw it onto the GPU texture with the new real-time data.

[0117] S6.4, Optimize Performance: To ensure superior performance when chart content is dynamically refreshed, consider appropriately optimizing the data and images. For example, use efficient algorithms to process frequently changing parts; for image rendering, mipmapping or LOD techniques can be used for optimization.

[0118] S6.5, Interaction and Feedback: Provides user interaction options, allowing users to continue exploring and analyzing even when the chart is static. For example, it allows users to zoom, scroll, or display additional data within the chart. Simultaneously, it responds to user actions in real time and updates the chart content, ensuring accurate chart feedback.

[0119] Figure 2 This is a visualization of the multimodal data chart in three-dimensional space in an embodiment of the present invention;

[0120] Figure 3 This is a visualization of the seamless and smooth switching of multimodal data charts in two-dimensional and three-dimensional space in an embodiment of the present invention;

[0121] Figure 4 This is a visualization of the multimodal data chart in two-dimensional space in an embodiment of the present invention.

[0122] This embodiment has been tested and can achieve a visualization frame rate of over 60 frames per second, providing seamless and smooth switching between two- and three-dimensional spaces.

[0123] In a second aspect of the present invention, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements steps S1-S6 of the above-described multimodal data chart visualization method based on real-time interactive seamless switching in two- and three-dimensional space.

[0124] Computer-readable storage media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0125] According to a second aspect of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement steps S1-S6 of the above-described multimodal data chart visualization method based on real-time interactive seamless switching in two- and three-dimensional space.

[0126] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. Memory can be the internal storage unit of a computer device, such as the hard drive or RAM. Memory can also be external storage devices, such as plug-in hard drives, Smart Media Cards (SMCs), Secure Digital (SD) cards, Flash Cards, etc.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for visualizing multimodal data charts, characterized in that, Including the following steps: S1. During real-time interaction, obtain multimodal data charts, select the chart display format through real-time interaction with the user, and select the corresponding seamless switching mode for real-time interaction in two-dimensional and three-dimensional space according to the specified display format. S2. Obtain the current position of the chart and the placement position of the target chart. First, obtain and record the starting position of the currently displayed chart in the three-dimensional space. Then, obtain and record the placement position of the target chart. The method for obtaining the placement position of the target chart includes: calculating the possible placement positions of the target chart in the corresponding space based on the two-dimensional or three-dimensional space characteristics of the target chart; determining whether the bounding box of the chart position intersects with the bounding boxes of other charts and obstructs each other; and automatically adjusting the position and chart orientation to avoid obstruction. S3. Collect and integrate multimodal data. Based on the characteristics of the multimodal data and visualization requirements, select appropriate chart types to display the data, call the rendering engine, draw dynamic multimodal data charts onto the GPU texture data, and update the pixel data in the GPU texture in real time according to the changes in the data. S4. Based on the starting position of the chart and the target chart placement position obtained in step S2, calculate the translation and rotation dynamic interpolation path from the current position of the chart to the final target placement position of the chart. Step S4 uses various spline curve interpolation methods to calculate the path, specifically including: S4.1, Define the start and end positions of the chart: Define the start and end positions of the chart based on the start position of the chart obtained in step S2 and the target chart placement position; S4.2, Define control points: Before calculating the spline shape, it is necessary to define a set of control points. The shape of the spline is determined by a set of control points, and at least four control points must be determined. S4.3, Calculate the equation of the spline curve: Given a set of control points and a parameter t, use the Catmull-Rom spline interpolation method to calculate the equation of the spline curve; S4.4, Calculate translation and rotation: Use spline curves to calculate the translation and rotation of the chart. Translation is determined directly by the points on the curve, while rotation is determined by calculating the tangent to the curve. The direction of the tangent is the direction of rotation, and the tangent is calculated using the derivative of the spline curve. S4.5, Dynamic Interpolation: The position and rotation of the chart are dynamically changed by changing the value of parameter t. When t changes from 0 to 1, the chart will move from the starting position to the ending position along the spline curve we have calculated, and rotate accordingly. S5. During the switching process of multimodal data charts, dynamic multimodal data charts are drawn in three-dimensional space based on real-time updated GPU texture data and dynamic path interpolation data. The specific steps of step S5 include: S5.1, Prepare rendering parameters: Based on the dynamic path interpolation data calculated in the previous steps, including translation vectors and rotation vectors, in order to realize the translation and rotation animation effects of the chart in three-dimensional space; S5.2 Configure camera parameters and view settings in the 3D rendering environment to observe dynamic multimodal data graphs at appropriate angles and viewing distances; S5.3 applies the GPU texture data of the rendered dynamic multimodal data chart to a 3D chart model, maps it to a 3D planar model, and associates the texture coordinates with the model vertex information; S5.4, Apply dynamic path interpolation data to the 3D chart model based on the previously calculated translation and rotation vectors, and generate a new transformation matrix for the 3D chart model; S5.5 plots dynamic multimodal data charts in three-dimensional space based on camera, view, and dynamic transformation settings; S6. After reaching the target position of the multimodal data chart, the chart position remains stationary, but the chart content continues to be dynamically refreshed.

2. The multimodal data chart visualization method according to claim 1, characterized in that, In step S2, the current position of the chart is obtained. Through real-time interaction, the spatial characteristics of the chart selected by the user and the position of the chart in the current space are used to generate and record the current three-dimensional spatial position of the chart.

3. The multimodal data chart visualization method according to claim 2, characterized in that, Step S2 includes the following specific steps: S2.2.1 Calculate the possible locations of the target chart in the corresponding space based on the two-dimensional or three-dimensional spatial characteristics of the target chart; S2.2.

2. Using the separating axis theorem to detect whether collisions occur between convex polygons, determine whether the bounding box of the chart position intersects with the bounding boxes of other charts and occludes each other; S2.2.3 If there is occlusion, the chart position will be automatically adjusted until there is no occlusion. Simulated annealing algorithm will be used to adjust for occlusion. S2.2.4 Automatically change the orientation of the chart target placement based on the change of viewpoint in three-dimensional space. Specifically, it uses a method that integrates the viewpoint of the chart and the chart occlusion relationship, and uses quaternions and camera control to realize the viewpoint change and target placement orientation change.

4. The multimodal data chart visualization method according to claim 3, characterized in that, Step S2.2.3, which uses the simulated annealing algorithm to adjust occlusion, includes the following steps: (a) Set the initial temperature, cooling coefficient and termination temperature, randomly select a reasonable initial chart position and calculate its occlusion degree as the initial cost; (b) Repeat the following steps as the temperature decreases from the initial temperature to the final temperature: (ba) Randomly select a new location, which can be a random perturbation of the current location; (bb) If the new location is unobstructed or the obstruction is reduced, then accept the new location; otherwise, accept the new location according to probability, where the probability is determined by the current temperature and the degree of obstruction between the old and new locations. (bc) Update the temperature by multiplying it by a coefficient less than 1 to simulate the cooling process; (c) The iteration ends when the temperature drops below the termination temperature.

5. The multimodal data chart visualization method according to claim 3, characterized in that, Step S2.2.4 automatically changes the orientation of the chart target placement based on the change of perspective in three-dimensional space, including the following steps: (a) Initialize the scene and objects: It is necessary to obtain the current world space position of the camera and the target icon in the 3D scene, as well as the camera's viewpoint and the target icon's orientation; (b) Calculate the rotation quaternion: Based on the camera's viewpoint and the target chart's orientation, calculate the rotation quaternion of the target chart under the new viewpoint; (c) Interpolation transition: Use spherical linear interpolation to interpolate between the current quaternion and the target quaternion; (d) Update object orientation: Apply the interpolated quaternion to the rotation transformation of the object to update the object's orientation in real time.

6. The multimodal data chart visualization method according to claim 1, characterized in that, Step S3 specifically includes: S3.1 Mapping data to chart elements: For the selected chart type, map the multimodal data to the individual elements of the chart; S3.2, Create GPU texture: Create a texture space in the GPU to store the pixel data of the chart; set the texture size and color format to match the chart based on the chart size, precision and color depth; S3.3, Rendering the chart to a texture: Call the rendering engine to render the chart's pixel data to the GPU texture; this includes creating a vertex buffer based on the chart's vertex data and uploading it to the GPU, setting the rendering pipeline state, setting an appropriate blending mode for the rendering process, associating the created GPU texture with the chart, and calling the rendering engine to render the chart's pixel data to the GPU texture. S3.4 Updates pixel data in GPU textures in real time based on data changes to reflect dynamic data changes.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the multimodal data chart visualization method as described in any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal data chart visualization method as described in any one of claims 1 to 6.

Images