Interactive selection and exploration methods for understanding tabular data in immersive environments

Abstract

This invention discloses an interactive selection and exploration method for understanding tabular data in immersive environments, comprising: S1, the user uploads tabular data; S2, an interactive 3D spatial data table is rendered in a virtual reality space; S3, the user uses a controller or bare-hand gestures to perform six table selection methods on the 3D spatial table data, including point selection, open path selection, and closed path selection; S4, feedback is provided on the user's selection operation and the selection area expansion function is activated; S5, the user uses a controller or bare-hand gestures to adjust the boundary of the selection area and finally confirm it. This invention greatly improves the efficiency of user table operations in virtual environments, especially in data analysis and visualization processes, allowing users to interact with complex tabular data more naturally and flexibly, promoting the depth and breadth of information exploration and data insight.

Description

Technical Field

[0001] This invention relates to the field of immersive visual analytics and human-computer interaction, and specifically proposes an interactive selection and exploration method for understanding tabular data in immersive environments. Background Technology

[0002] Tabular data is a common form of data analysis, widely used by data scientists, financial professionals, and policymakers. With advancements in virtual reality (VR) and human-computer interaction (HCI) technologies, users are leveraging immersive visual data analysis tools to achieve precise selection and embodied exploration of tabular data. Users are no longer limited to traditional two-dimensional desktop selection and exploration of tabular data, but are extending this to three-dimensional space. Existing immersive selection interaction technologies mainly fall into two categories: the first uses VR device controllers and virtual rays to select and explore areas of interest; the second recognizes user gestures to interact with targets in the virtual space. These two interaction methods ensure precise selection within an immersive environment. However, existing immersive data selection technologies focus only on broad-based data exploration and have not yet considered the selection and exploration of high-density data like tables. In the field of immersive data visualization, virtual reality (VR) provides an intuitive, multi-dimensional interactive experience, enabling users to understand and analyze complex data more deeply and comprehensively. Existing immersive interactive methods often face difficulties when performing tasks requiring high precision, especially in data selection. Accurately selecting subsets of data is crucial for users to progressively delve deeper into specific questions that are most valuable. Selecting a specific row, column, or data cell in a table requires a certain degree of precision, and current interactive technologies are insufficient in this regard, making it impossible to achieve precise interactive selection of table data. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the purpose of this invention is to enable users to efficiently and accurately select tabular data in immersive environments. This invention proposes a selection technology framework tailored for tabular data in immersive environments. For virtual reality environments, this framework introduces three interactive selection technologies: point selection, open path selection, and closed path selection. To adapt to different interaction methods, this invention develops multiple methods for controller-based and bare-handed interaction, aiming to provide a precise and intuitive interactive experience for selecting tabular data. The design of this invention includes various gestures for selecting data in various scenarios, such as single-block or multi-block selection, covering selection from single rows or columns to cells and continuous areas.

[0004] The interactive selection and exploration method for understanding tabular data in immersive environments provided by this invention includes the following five steps:

[0005] S1 user uploads table data;

[0006] S2 renders interactive 3D spatial data tables in virtual reality space;

[0007] S3 users can use the controller or bare-hand gestures to select three-dimensional spatial table data in six ways: point selection, open path selection, and closed path selection.

[0008] S4 provides feedback on the user's selection and activates the selection area expansion function;

[0009] S5 users can adjust the selection area boundaries using the controller or bare-hand gestures and then finalize the selection.

[0010] The specific steps are as follows:

[0011] S1 user uploads table data;

[0012] Users first upload tabular data to the virtual reality device. This method supports both common, plain tabular data and complex hierarchical tabular data. Users import files through the interface on the virtual reality device and then select the files to upload using controllers or bare-hand gestures. This method supports two tabular file formats: .xlsx and .csv.

[0013] S2 renders interactive 3D spatial data tables in virtual reality space;

[0014] This method automatically parses imported table files, identifies and constructs the table's row and column structure, and performs targeted processing for different types of table data. For ordinary table data, it first parses and splits the data, mapping each data item to a specific cell. Using a table layout algorithm, these cells are organized according to their original positions to construct the table. After construction, the method renders each cell to ensure correct data formatting and adds interactive functionality, allowing users to interact with each individual cell.

[0015] For hierarchical table data, the system first splits the data, marking empty values, header cells, and merged cells. Next, it iterates through rows and columns to identify cells that need merging, forming the headers of the multi-level table. During this process, the system intelligently determines the type of each cell, such as a regular data cell, a header cell, or a merged cell. Based on this information, a hierarchical table structure is built, and the data is rendered according to the hierarchical logic to ensure that the data display reflects its hierarchical relationship. In addition, the header cells of the hierarchical table are given specific functions; for example, selecting a header allows you to select an entire row or column of cells, and selecting both row and column headers simultaneously defines an overlapping selection area. When users interact with complex tables in the immersive space, the hierarchical structure of the table data is clearly visible, allowing for quick selection and exploration of the data.

[0016] The system renders the imported tabular data, using efficient rendering methods to ensure smooth table presentation in the virtual reality environment. The table data is presented to the user in three dimensions, preserving the structured information of traditional table operations. The rows and columns of the table maintain the same hierarchical structure as the original data, ensuring data integrity and readability. Furthermore, this method supports table zooming in and out, allowing users to efficiently view and accurately select data.

[0017] S3 users can use the controller or bare-hand gestures to select three-dimensional spatial table data in six ways: point selection, open path selection, and closed path selection.

[0018] The method constructs two interaction modes: one based on virtual reality device controllers and the other based on table data selection using bare-hand gestures. Users can choose different interaction modes according to their personal preferences.

[0019] 1. Virtual Reality Device Controller Interaction Method: Users select cells in a table using a handheld device (virtual reality device controller). Users use the controller to select cells or areas of any size within the table; the buttons on the controller are defined with different functions.

[0020] 2. Bare-hand Gesture Recognition Interaction: Users interact directly with the table using bare-hand gestures. This method requires no external devices, allowing users to select and explore virtual data tables through natural hand movements, providing a more immersive experience. Different bare-hand gestures correspond to different selection methods and table interaction functions.

[0021] For each interaction method, the system supports the following three operation modes:

[0022] (a) Point selection operation: Users can make selections by clicking on one or more cells.

[0023] In the virtual reality device controller interaction mode, users use the controller's Trigger button to perform click selection operations.

[0024] In the bare-hand gesture recognition interaction method, users can perform click selection operations by pinching their thumb and forefinger together.

[0025] The point-and-click operation is suitable for quickly selecting a single cell or a series of adjacent cells. Combining point-and-click with the cell header functionality of a hierarchical table enables complex table selection operations. For example, selecting the row header of a hierarchical table selects an entire row of cells, or the column header selects an entire column of cells. Holding down the X button on the VR headset controller or making a five-finger gesture with your left hand while simultaneously selecting the row header and column header allows you to select an intersecting area. Besides intersecting areas, point-and-click can also be used to select any data table area. In this case, you need to hold down the X button on the VR headset controller or make a five-finger gesture with your left hand, and then select the starting and ending cells to specify the boundary cells of the area, thus defining the region.

[0026] (b) Open path selection operation: The user draws an open path in the constructed data table, and uses the start and end points of the line as boundaries to determine the range of selected cells.

[0027] In the virtual reality device controller interaction mode, the user holds down the Grip button on the controller and draws an open path by moving the controller to complete the area selection.

[0028] In the bare-hand gesture recognition interaction mode, the user makes a pointing gesture with their index finger and draws an open path to select the desired cell range.

[0029] Open paths are similar to rectangular selection, suitable for selecting contiguous table regions. To standardize user-drawn lines, this method uses a coordinate transition matrix to standardize the coordinates of each point in the line, and then calculates whether the slope of each point relative to the starting point is within a threshold range to determine whether the user-drawn line conforms to the standard.

[0030] (c) Closed path selection operation: The user draws a closed shape (such as a circle, rectangle or custom polygon) in the constructed data table, and uses the upper and lower boundary points and the left and right boundary points of the shape to determine the range and realize the selection of cells.

[0031] In the virtual reality device controller interaction mode, the user presses and holds the Grip button on the controller to draw a closed path.

[0032] In the bare-hand gesture recognition interaction mode, the user makes a pointing gesture with their index finger to draw a closed shape, forming a selection area.

[0033] This operation method is suitable for scenarios requiring more complex area selection, allowing users to flexibly select multiple cells within irregular areas. The method determines whether a shape is closed by calculating whether the Euclidean distance between the start and end points of the drawn closed shape is less than a specified threshold.

[0034] When selecting multiple cells or regions, users can press and hold the Y key on the controller or make a fist gesture with their left hand to combine (a) point selection, (b) open path selection, or (c) closed path selection to achieve complex selection of multiple cells or regions.

[0035] For different selection area sizes, the three data table selection methods provide users with different granular selection options, meeting the needs of different usage scenarios and improving the efficiency and user experience of table selection operations.

[0036] S4 provides feedback on the user's selection, thereby activating the selection area expansion function;

[0037] Users can select specific cells or regions in a table using point selection, open path selection, and closed path selection methods as needed. The current selection results are reflected in real time in the virtual reality space. This method highlights the selected cell region in the virtual data table and displays the size of the current region in an M×N format in the center of the highlighted area (M represents the number of rows, and N represents the number of columns).

[0038] S5 users can use the controller or bare-hand gestures to adjust the boundaries of the selected area and finally confirm it;

[0039] When an initial selection is completed but discrepancies arise, users need to further refine the selection to ensure it meets their needs. Users can dynamically modify the boundaries of the selected area using virtual reality device controllers or bare-hand gestures, quickly expanding or shrinking the selected region.

[0040] (a) Expanded selection region function: By default, it is in expanded selection region mode.

[0041] In the virtual reality device controller interaction mode, the user pushes the left joystick to expand the border, and the edge of the selected area will extend in the specified direction.

[0042] In the bare-hand gesture recognition interaction mode, when the user points in the direction of the target with their left thumb, the edge of the selected area will expand in the specified direction.

[0043] (b) Shrink selection area function: If the user needs to shrink the selection area, the shrink function can be enabled.

[0044] In the virtual reality device controller interaction mode, the user presses and holds the B button on the controller and pushes the left joystick to shrink the boundary of the selected area, and the edge of the selected area will shrink in the specified direction.

[0045] In the bare-hand gesture recognition interaction mode, the user triggers the contraction operation by making a fist with their right hand, and the user uses their left thumb to indicate the direction of contraction. The edge of the selected area will contract in the specified direction.

[0046] (c) Confirm selection: After the adjustment is completed, the user can actively confirm the final selection area or wait 5 seconds for the selection to be automatically confirmed.

[0047] In the virtual reality device controller interaction mode, press the A key to confirm the selected area.

[0048] In the bare-hand gesture recognition interaction mode, make a gesture with your right thumb to confirm the selection area.

[0049] Once the virtual reality device receives the confirmation signal, the selected area is fixed and can be used for subsequent data processing, analysis, or visualization. This process allows users to select data areas more precisely, improving efficiency and ensuring accuracy.

[0050] The method described in this invention supports the construction of immersive spaces for both ordinary and hierarchical tabular data, and this visualization can serve as the foundation for immersive data analysis and exploration. Based on this, six table selection techniques are proposed for the precise selection of tabular data in immersive spaces. These techniques are combinations of selection methods (click, open path, and closed path) and interaction methods (virtual reality device controllers and bare-hand gestures). Specifically, the effects of this invention include:

[0051] It has the ability to construct three-dimensional spatial visualization of tabular data: Steps S1 and S2 above analyze ordinary tables and complex hierarchical tables to different degrees, reconstruct their structural hierarchical relationship in an immersive space, thereby giving the header of complex hierarchical tables richer functions, making the selection of table data more efficient, and thus achieving the invention effect.

[0052] Enhancing the flexibility of tabular data in immersive space: Step S2 above intuitively constructs tabular data in immersive space, allowing users to zoom and move the tabular data in three-dimensional space. This enables users to flexibly adjust the table view to adapt to different analysis tasks, thereby achieving the desired effect.

[0053] The efficiency and accuracy of immersive data table selection have been improved: Step S3 above allows users to explore immersive tables using six different table selection methods, and proposes different precision selection methods for different sizes of the selection area, thus improving selection efficiency. Steps S4 and S5 support precise adjustment of the selection area boundaries, reducing the number of times users need to reselect and enabling precise adjustment of the area range, thereby achieving the invention effect. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the interactive selection of tabular data in an immersive space according to the present invention.

[0055] Figure 2 This is a point selection method in a specific embodiment of the present invention;

[0056] Figure 3 This is a method for selecting open paths in a specific embodiment of the present invention;

[0057] Figure 4 This is a method for selecting a closed path in a specific embodiment of the present invention;

[0058] Figure 5 This refers to the expansion and contraction of the selected table area in a specific embodiment of the present invention;

[0059] Figure 6 This is a method for determining the compliance of open paths in a specific embodiment of the present invention;

[0060] Figure 7 This is a method for determining whether a closed path is closed in a specific embodiment of the present invention. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0062] Figure 1 This invention illustrates different interaction methods (virtual reality device controller interaction and bare-hand gesture interaction) for different types of tables (ordinary tables and hierarchical tables) in an immersive space. First, a table area is selected in the immersive space (a), the boundaries of the selected table area are adjusted (b), and finally, the selected area is confirmed by pressing the controller A button or using a right thumb gesture (c).

[0063] Figure 2The specific implementation effect of the point selection method of the present invention is shown. Virtual reality controller interaction and bare-hand gesture interaction methods were designed for selection areas of different sizes and numbers. Single-click selection is based on the controller trigger or a two-hand pinch gesture (right hand). Multi-selection is achieved using the controller's Y button or a left-hand fist gesture. Selection of arbitrary areas and intersecting areas can be achieved using the controller's X button or a gesture with the left hand's five fingers spread.

[0064] Figure 3 The specific implementation effect of the open path selection method of the present invention is illustrated. The size of the selected area determines the open path selection method. Selecting areas of different sizes is achieved by drawing open paths within the specified area. The open path selection operation is based on the Trigger button of the controller (right controller) or an extended index finger gesture (right hand). The Y button of the controller or a left-hand fist gesture indicates multi-selection functionality.

[0065] Figure 4 The specific implementation effect of the open path selection of the present invention is shown. The closed path selection operation is based on the Trigger button of the controller (right controller) or an extended index finger gesture (right hand). The Y button of the controller or a left-hand fist gesture indicates multi-selection function.

[0066] Figure 5 This invention demonstrates the function of expanding and shrinking the selected area. The user uses the joystick of the virtual reality device controller or their left thumb to indicate direction and change the selected area. The default is to expand the selected area; pressing and holding the B button on the controller or making a right fist gesture switches to shrinking the selected area.

[0067] Figure 6 This invention demonstrates how it determines the compliance of open paths. (a) shows the determination process, which involves normalizing the open path using a rotation matrix and calculating the slope to determine whether the open path is compliant. (b) shows the explicit selection of regions based on the start and end points of compliant open paths.

[0068] Figure 7 The present invention demonstrates how to determine whether a closed path is closed. (a) indicates that the closure of the path is determined by calculating whether the Euclidean distance between the start and end points of the closed path is less than a threshold. (b) indicates that the table area is defined based on the four boundary points of the closed path.

Claims

1. An interactive selection and exploration method for understanding tabular data in immersive environments, characterized in that, Includes the following steps: S1 user uploads table data; S2 renders interactive 3D spatial data tables in virtual reality space; S3 users can use the controller or bare-hand gestures to select three-dimensional spatial table data in six ways: point selection, open path selection, and closed path selection. S4 provides feedback on the user's selection and activates the selection area expansion function; S5 users can use the controller or bare-hand gestures to adjust the boundaries of the selected area and finally confirm it; The specific method is as follows: When the initial selection area is completed but deviations occur, users need to further refine the selection area to ensure that it meets their needs. Users can dynamically modify the boundaries of the selection area through virtual reality device controllers or bare-hand gestures to quickly expand or shrink the selected area. (a) Extended selection region function: By default, it is in extended selection region mode; In the virtual reality device controller interaction mode, the user pushes the left joystick to expand the border, and the edge of the selected area will extend in the specified direction; In the bare-hand gesture recognition interaction mode, when the user points in the target direction with their left thumb, the edge of the selected area will expand in the specified direction; (b) Shrink selection area function: Enable the shrink function if the user needs to shrink the selection area; In the virtual reality device controller interaction mode, the user presses and holds the B button on the controller and pushes the left joystick to shrink the boundary of the selected area, and the edge of the selected area will shrink in the specified direction; In the bare-hand gesture recognition interaction mode, the user triggers the contraction operation by making a fist with the right hand, and the user uses the left thumb to indicate the direction of contraction. The edge of the selected area will contract in the specified direction. (c) Confirm selection area: After the adjustment is completed, the user can actively confirm the final selection area or wait 5 seconds for the selection area to be automatically confirmed; In virtual reality device controller interaction mode, press the A key to confirm the selected area; In the bare-hand gesture recognition interaction mode, make a gesture with your right thumb to confirm the selection area; Once the virtual reality device receives the confirmation signal, the selected area will be fixed for subsequent data processing, analysis, or visualization.

2. The interactive selection and exploration method for understanding tabular data in immersive environments according to claim 1, characterized in that, The specific method for step S1 is as follows: Users first upload the table data to the virtual reality device; then, they import the file through the interface on the virtual reality device and select the file to upload using the controller or bare-hand gestures; two table file formats are supported, including .xlsx and .csv.

3. The interactive selection and exploration method for understanding tabular data in immersive environments according to claim 1, characterized in that, The specific method for step S2 is as follows: It automatically parses imported table files, identifies and constructs the row and column structure of the tables, and performs targeted processing on different types of table data; For ordinary tabular data, the data is first parsed and split, and each data item is mapped to a specific cell. Then, the cells are organized according to their original positions using a table layout algorithm to construct the table. After the table is constructed, each cell is rendered to ensure that the data is formatted correctly and interactive functions are added so that users can interact with each individual cell. For hierarchical table data, firstly, the table data is split, and special data items that represent empty values, table headers, or merged cells are marked; next, by traversing rows and columns, the cells that need to be merged are identified to form the table header of the multi-level table. In this process, the type of each cell is intelligently determined; Based on this information, a hierarchical table structure is built, and the data is rendered according to the logic of the hierarchy to ensure that the display of the data reflects its hierarchical relationship; the header cells of the hierarchical table are given specific functions. The imported table data is rendered using efficient rendering methods to ensure smooth table presentation in the virtual reality environment; the table data will be presented to the user in a three-dimensional form, while retaining the structured information of traditional table operations; the rows and columns of the table will maintain the same hierarchical structure as the original data, ensuring data integrity and readability; and table zooming is also supported.

4. The interactive selection and exploration method for understanding tabular data in immersive environments according to claim 1, characterized in that, The specific method for step S3 is as follows: Two interaction methods are constructed: one based on virtual reality device controllers and the other based on table data selection using bare hand gestures. Users can choose different interaction methods according to their personal preferences. (1) Virtual Reality Device Controller Interaction Method: Users select tables through the virtual reality device controller; users use the controller to select cells or areas of any size in the table, and the buttons in the controller are defined with different functions; (2) Bare-hand gesture recognition interaction method: Users interact directly with the table through bare-hand gestures.

5. The interactive selection and exploration method for understanding tabular data in immersive environments according to claim 4, characterized in that, In step S3, for each interaction method, the following three operation methods are supported: (a) Point selection: Users select directly by clicking on one or more cells; In the virtual reality device controller interaction mode, users use the controller's Trigger button to perform click selection operations; In the bare-hand gesture recognition interaction method, users can perform click selection operations by pinching their thumb and forefinger together; (b) Open path selection operation: The user draws an open path in the constructed data table, and uses the start and end points of the path as boundaries to determine the range of selected cells; (c) Closed path selection operation: The user draws a closed shape in the constructed data table, and uses the upper and lower boundary points and the left and right boundary points of the shape to determine the range and realize the selection of cells.

6. The interactive selection and exploration method for understanding tabular data in immersive environments according to claim 1, characterized in that, The specific method for step S4 is as follows: Users can select specific cells or areas in a table using point selection, open path selection, and closed path selection methods as needed, and the current selection results are displayed in real time in the virtual reality space.

Images