Interactive selection and exploration method oriented to table data understanding in immersive environment

Abstract

The invention discloses an interactive selection and exploration method oriented to table data understanding in an immersive environment. The method comprises the steps that S1, a user uploads table data; s2, rendering an interactive three-dimensional space data table in the virtual reality space; s3, a user uses a controller or a naked hand gesture to carry out six table selection modes including click selection, open path selection and closed path selection on the three-dimensional space table data; s4, feeding back a selection operation of the user and activating a selection area expansion function; and S5, the user uses a controller or a naked hand gesture to adjust the boundary of the selection area and finally determines the boundary. According to the method, the table operation efficiency of the user in the virtual environment is greatly improved, particularly in the data analysis and visualization process, the user can interact with complex table data more naturally and flexibly, and the depth and breadth of information exploration and data insight are promoted.

Description

Technical Field

[0001] The present invention relates to the field of immersive visual analysis and human-computer interaction, and specifically proposes an interactive selection and exploration method for understanding tabular data in an immersive environment for tabular data. Background Art

[0002] Tabular data is a common form of data analysis and is widely used by data scientists, financial practitioners, and policymakers. With the advancement of virtual reality technology and human-computer interaction technology, users can use immersive visual data analysis tools to accurately select and explore tabular data. Users are no longer limited to the traditional two-dimensional desktop selection and exploration of tabular data, but extend it to three-dimensional space. Existing immersive selection interaction technologies are mainly divided into two categories. The first type of technology uses virtual reality device controllers and virtual rays to select and explore areas of interest, and the second type recognizes user gestures to interact with targets in virtual space. These two interaction methods ensure that users can accurately select in an immersive environment. However, existing immersive data selection technologies only focus on large-scale data exploration, and have not yet considered the selection and exploration of high-density data such as tables. In the field of immersive data visualization, virtual reality (VR) provides an intuitive, multi-dimensional interactive experience that enables users to understand more deeply and analyze complex data more comprehensively. Existing immersive ring interaction methods often face difficulties in performing tasks that require high precision, especially in data selection. Accurately selecting data subsets is crucial for users to gradually dig deeper into specific problems that are most valuable. Selecting specific rows, columns, or data cells in a table requires a certain degree of precision, but current interactive technology is still insufficient in this regard and cannot achieve accurate interactive selection of table data. Summary of the invention

[0003] In view of the defects of the prior art, the purpose of the present invention is to support users to select tabular data efficiently and accurately in an immersive environment. The present invention proposes a selection technology framework tailored for tabular data in an immersive environment. For virtual reality environments, the framework introduces three interactive selection technologies: point selection, open path selection, and closed path selection. In order to adapt to different interaction methods, the present invention has developed a variety of methods for controller-based and bare-hand interactions, aiming to provide an accurate, intuitive and natural interactive experience for tabular data selection. The design of the present invention includes a variety of gestures for selecting data in various scenarios, such as single-block or multi-block selection, covering selections from single rows or columns to cells and continuous areas.

[0004] The interactive selection and exploration method for understanding table data in an immersive environment provided by the present invention includes the following five steps:

[0005] S1 user uploads table data;

[0006] S2 renders interactive three-dimensional spatial data tables in virtual reality space;

[0007] S3 users use the controller or bare-hand gestures to select three-dimensional space table data in six table selection modes: point selection, open path selection, and closed path selection;

[0008] S4 feedbacks the user's selection operation and activates the selection area expansion function;

[0009] S5 users use the controller or bare-hand gestures to adjust the selection area boundaries and finalize.

[0010] The specific contents of each step are as follows:

[0011] S1 user uploads table data;

[0012] The user first uploads the table data to the virtual reality device. This method supports common ordinary table data and complex hierarchical table data. The user imports the file through the interface in the virtual reality device, and then uses the controller or bare hand gestures to select the file to be uploaded. This method supports tables in two file formats, including .xlsx and .csv.

[0013] S2 renders interactive three-dimensional spatial data tables in virtual reality space;

[0014] This method automatically parses the imported table file, identifies and builds the row and column structure of the table, and makes targeted processing for different types of table data. For ordinary table data, the table data is first parsed and split, and each data item corresponds to a specific cell. Through the table layout algorithm, these cells are organized according to their original positions to realize the construction of the table. After the construction is completed, the method will render each cell to ensure that the data is in the correct format, and add interactive functions so that users can interact with each independent cell.

[0015] For hierarchical table data, first, split the table data, and mark the empty values ​​in the table data, special data items representing headers or merged cells. Next, identify the cells that need to be merged through row traversal and column traversal to form the header of the multi-level table. In this process, the system will intelligently determine the type of each cell, such as ordinary data cell, header cell, or merged cell. Based on this information, a hierarchical table structure is built and rendered according to the logic of the hierarchy to ensure that the display of data reflects its hierarchical relationship. In addition, the header cells of the hierarchical table are given specific functions, such as selecting the header to select the entire row or column of cells, and selecting the row header and column header at the same time to determine a cross selection area. When users interact with complex tables in immersive space, the hierarchical structure of the table data is clearly visible, and data can be quickly selected and explored.

[0016] The system renders the imported table data and uses an efficient rendering method to ensure smooth presentation of the table in the VR environment. The table data will be presented to the user in three dimensions and retain the structured information of traditional table operations. The rows and columns of the table will maintain a hierarchical structure consistent with the original data to ensure the integrity and readability of the data. In addition, this method also supports zooming in and out of the table so that users can efficiently view data and accurately select data.

[0017] S3 users use the controller or bare-hand gestures to select three-dimensional space table data in six table selection modes: point selection, open path selection, and closed path selection;

[0018] Two interaction modes were constructed: a table data selection interaction mode based on a virtual reality device controller and a bare-hand gesture-based interaction mode. Users can choose different interaction modes according to their personal preferences.

[0019] 1. Virtual reality device controller interaction mode: Users use handheld devices (virtual reality device controllers) to perform table selection operations. Users use controllers to select cells or areas of any size in the table, and the buttons in the controller are defined as different functions.

[0020] 2. Bare-hand gesture recognition interaction mode: Users interact directly with the table through bare-hand gestures. This method does not require external devices and allows users to select and explore virtual data tables through natural hand movements, providing a more immersive experience. Different bare-hand gestures correspond to different selection forms and table interaction functions.

[0021] In each interaction mode, the system supports the following three operation modes:

[0022] (a) Click operation: The user directly selects a cell or multiple cells by clicking on them.

[0023] In the virtual reality device controller interaction mode, the user uses the controller's Trigger key to implement click selection operations.

[0024] In the bare-hand gesture recognition interaction mode, users can click and select by pinching their thumb and index finger.

[0025] The click operation is suitable for quickly selecting a single or a series of adjacent cells. The click operation is combined with the cell header function of the constructed hierarchical table to realize complex table selection operations. For example, select the row header of the hierarchical table to select the entire row of cells or the column header to select the entire column of cells; press and hold the X key of the virtual reality device controller or make a five-finger gesture with your left hand, and select the row header and the column header at the same time to select the intersection area. In addition to the intersection area, click can also be used to select any data table area. At this time, you need to press and hold the X key of the virtual reality device controller or make a five-finger gesture with your left hand, select the starting cell and the ending cell respectively to specify the boundary cells of the area, and then determine this area.

[0026] (b) Open path selection operation: The user draws an open path in the constructed data table, and uses the starting and ending points of the line as boundaries to determine the range of the selected cells.

[0027] In the virtual reality device controller interaction mode, the user holds down the Grip button of the controller and draws an open path by moving the controller to complete the area selection.

[0028] In the bare-hand gesture recognition interaction mode, the user makes an index finger pointing gesture and draws an open path to select the desired cell range.

[0029] Open path is similar to rectangular selection and is suitable for selecting continuous table areas. In order to normalize the straight line drawn by the user, this method uses the coordinate transfer matrix to normalize the coordinates of each point in the straight line, and then calculates whether the slope of each point with the starting point is within the threshold range to determine whether the straight line drawn by the user meets the specifications.

[0030] (c) Closed path selection operation: The user draws a closed shape (such as a circle, rectangle, or custom polygon) in the constructed data table, and uses the upper and lower boundary points and the left and right boundary points of the shape to determine the range to select cells.

[0031] In the virtual reality device controller interaction mode, the user holds down the Grip button of the controller to draw a closed path.

[0032] In the bare-hand gesture recognition interaction mode, the user makes an index finger pointing gesture to draw a closed shape to form a selection area.

[0033] This operation mode is suitable for more complex area selection scenarios, allowing users to flexibly select multiple cells in irregular areas. This method calculates whether the Euclidean distance between the start and end points of the drawn closed figure is less than the specified threshold to determine whether the figure is a closed figure.

[0034] When multiple cells or regions are involved, the user can select multiple cells or regions by holding down the Y button on the controller or making a fist with the left hand, and performing any combination of (a) point selection, (b) open path selection, or (c) closed path selection.

[0035] For different selection area sizes, the three data table selection methods provide users with selection methods of different granularities to meet the needs of different usage scenarios and improve the efficiency and user experience of table selection operations.

[0036] S4 feeds back the user's selection operation and activates the selection area expansion function;

[0037] Users can select specific cells or areas in the table using point selection, open path selection, and closed path selection methods according to their needs, and the current selection results will be fed back in real time in the virtual reality space. This method highlights the selected cell area in the virtual data table and displays the size of the current area in the form of M×N in the center of the highlighted area (M represents the number of rows and N represents the number of columns).

[0038] S5 users use the controller or bare-hand gestures to adjust the selection area boundaries and finalize;

[0039] When the initial selection is completed and the selected area deviates, the user needs to further fine-tune the selection area to ensure that the selection meets the user's needs. The user can dynamically modify the boundaries of the selection area through the VR device controller or bare hand gestures to quickly expand or shrink the selected area.

[0040] (a) Extended selection area function: By default, it is in extended selection area mode.

[0041] In the VR device controller interaction mode, the user pushes the left joystick to expand the border, and the edge of the selection area will extend in the specified direction.

[0042] In the bare-hand gesture recognition interaction mode, the user points to the target direction with the left thumb, and the edge of the selection area will expand in the specified direction.

[0043] (b) Shrink selection area function: If the user needs to reduce the selection area, the shrink function can be enabled.

[0044] In the virtual reality device controller interaction mode, the user holds down the B button of the controller and pushes the left joystick to shrink the boundary of the selection area, and the edge of the selection area will shrink in the specified direction.

[0045] In the bare-hand gesture recognition interaction mode, the user triggers the contraction operation by making a fist with his right hand, and uses his left thumb to indicate the contraction direction, and the edge of the selection area will contract in the specified direction.

[0046] (c) Confirm the selection: After the adjustment is completed, the user can actively confirm the final selection area or wait for 5 seconds for the selection to be automatically confirmed.

[0047] In VR device controller interaction mode, press the A button to confirm the selection area.

[0048] In the bare-hand gesture recognition interaction mode, make a right thumb gesture to confirm the selection area.

[0049] Once the VR device receives the confirmation signal, the selection result will be fixed for subsequent data processing, analysis or visualization operations. Through this process, users can select data areas more accurately, improve work efficiency and ensure the accuracy of results.

[0050] The method of the present invention supports the construction of immersive space for ordinary table data and hierarchical table data, and the visualization can serve as the basis for immersive data analysis and exploration. On this basis, six table selection techniques are proposed for the precise selection of immersive space table data. These techniques are a combination of selection methods (click, open path and closed path) and interaction methods (virtual reality device controller and bare hand gestures). Specifically, the effects of the present invention include:

[0051] Possessing the ability to construct three-dimensional spatial visualization of table data: The above steps S1 and S2 perform different degrees of analysis on ordinary tables and complex hierarchical tables, and reconstruct their structural hierarchical relationships in the immersive space, thereby giving the complex hierarchical table headers richer functions, making the selection of table data more efficient, and thus achieving the effect of the invention.

[0052] Improving the flexibility of tabular data in the immersive space: The above step S2 intuitively constructs tabular data in the immersive space, allowing users to scale and move tabular data in three-dimensional space, so that users can flexibly adjust the table view to adapt to different analysis tasks, thereby achieving the effect of the invention.

[0053] Improved efficiency and accuracy of immersive data table selection: The above step S3 supports users to use six different table selection methods to explore immersive tables, and proposes selection methods of different precisions according to the size of the selection area, thereby improving the efficiency of selection. Steps S4 and S5 support precise adjustment of the boundaries of the selection area, reducing the number of times the user reselects and enabling precise area range adjustment, thereby achieving the effect of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of selecting and interacting with table data in an immersive space according to the present invention;

[0055] Figure 2 It is a point selection method in a specific embodiment of the present invention;

[0056] Figure 3 It is a method for opening path selection in a specific embodiment of the present invention;

[0057] Figure 4 A method for closed path selection in a specific embodiment of the present invention;

[0058] Figure 5 The expansion and contraction of the selected table area in the specific implementation mode of the present invention;

[0059] Figure 6 A method for determining compliance of an open path in a specific embodiment of the present invention;

[0060] Figure 7 This is a method for determining whether a closed path is closed in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0062] Figure 1 The present invention shows different interaction methods (virtual reality device controller interaction and bare hand gesture interaction) in immersive space for different types of tables (ordinary tables and hierarchical tables). First, a table area is selected in the immersive space (a), the boundary of the selected table area is adjusted (b), and finally the selected area is confirmed by the controller A key or the right thumb gesture (c).

[0063] Figure 2The specific implementation effect of the point selection of the present invention is shown. For selection areas of different sizes and numbers, virtual reality controller interaction and bare hand gesture interaction methods are designed. The single-click selection operation is based on the controller trigger or two light pinch gestures (right hand). The multiple selection function is realized by using the controller's Y key or the left hand fist gesture. The selection of any area and intersection area can be realized by using the controller's X key or the left hand five fingers outstretched gesture.

[0064] Figure 3 The specific implementation effect of the open path selection of the present invention is shown. The size of the selection area determines the open path selection method. The selection of areas of different sizes is achieved by drawing an open path in the specified area. The open path selection operation is based on the controller's Trigger key (right controller) or the index finger gesture (right hand). The controller's Y key or the left hand fist gesture indicates the multi-select function.

[0065] Figure 4 The specific implementation effect of the open path selection of the present invention is shown. The closed path selection operation is based on the controller's Trigger key (right controller) or the index finger gesture (right hand). The controller's Y key or the left hand fist gesture represents the multiple selection function.

[0066] Figure 5 The invention shows the function of expanding and contracting the selected area. The user uses the joystick of the virtual reality device controller or the left thumb to indicate the direction to change the selected area. The default is to expand the selected area. When the B button of the controller is pressed or the right hand fist gesture is made, the area is reduced.

[0067] Figure 6 The present invention shows the judgment of the compliance of the open path. (a) shows the judgment process, using the rotation matrix to normalize the open path, and calculating the slope to determine whether the open path is compliant. (b) shows the explicit selection of the area based on the start and end points of the compliant open path.

[0068] Figure 7 The present invention shows the judgment of whether a closed path is closed. (a) indicates whether the closed path is closed by calculating whether the Euclidean distance between the start point and the end point of the closed path is less than a threshold. (b) indicates that the table area is clarified according to the four boundary points of the closed path.

Claims

1. An interactive selection and exploration method for understanding tabular data in an immersive environment, characterized in that: The following steps are involved: S1 user uploads table data; S2 renders interactive three-dimensional spatial data tables in virtual reality space; S3 users use the controller or bare-hand gestures to select three-dimensional space table data in six table selection modes: point selection, open path selection, and closed path selection; S4 feedbacks the user's selection operation and activates the selection area expansion function; S5 users use the controller or bare-hand gestures to adjust the selection area boundaries and finalize.

2. The interactive selection and exploration method for understanding tabular data in an immersive environment according to claim 1, characterized in that: The specific method of step S1 is: The user first uploads the table data to the virtual reality device; the user imports the file through the interface in the virtual reality device, and then uses the controller or bare hand gestures to select the file to be uploaded; two file formats are supported, including .xlsx and .csv.

3. The interactive selection and exploration method for understanding tabular data in an immersive environment according to claim 1, characterized in that: The specific method of step S2 is: Automatically parse the imported table files, identify and construct the row and column structure of the table, and make targeted processing for different types of table data; For common table data, the table data is first parsed and split, and each data item is mapped to a specific cell. Through the table layout algorithm, these cells are organized according to their original positions to realize the construction of the table. After the construction is completed, each cell is rendered to ensure that the data format is correct, and interactive functions are added so that users can interact with each independent cell. For hierarchical table data, first, split the table data, mark the empty values ​​in the table data, and the special data items representing the table header or merged cells; then, identify the cells that need to be merged through row traversal and column traversal to form the table header of the multi-level table; In this process, the type of each cell is determined intelligently; A hierarchical table structure is built based on this information and rendered according to the logic of the hierarchy to ensure that the display of data reflects its hierarchical relationship; the header cells of the hierarchical table are given specific functions; Render the imported table data, using efficient rendering methods to ensure smooth presentation of the table in the VR environment; the table data will be presented to the user in three-dimensional form, while retaining the structured information of traditional table operations; the rows and columns of the table will maintain a hierarchical structure consistent with the original data, ensuring the integrity and readability of the data; it also supports zooming in and out of the table.

4. The interactive selection and exploration method for understanding tabular data in an immersive environment according to claim 1, characterized in that: The specific method of step S3 is: Build two interaction modes: one based on the VR device controller and the other based on bare hand gestures to select table data. Users can choose different interaction modes based on their personal preferences. (1) Virtual reality device controller interaction mode: Users use a handheld device, i.e., a virtual reality device controller, to perform table selection operations. Users use the controller to select cells or areas of any size in the table, and the buttons in the controller are defined as different functions. (2) Bare-hand gesture recognition interaction method: Users directly interact with the table through bare-hand gestures.

5. The interactive selection and exploration method for understanding tabular data in an immersive environment according to claim 4, characterized in that: In step S3, under each interaction mode, the following three operation modes are supported: (a) Click operation: the user directly selects a cell or multiple cells by clicking on them; In the virtual reality device controller interaction mode, the user uses the controller's Trigger key to implement click selection operations; In the bare-hand gesture recognition interaction mode, users can click and select by pinching their thumb and index finger together; (b) Open path selection operation: the user draws an open path in the constructed data table, and the start and end points of the line are used as boundaries to determine the range of the selected cells; (c) Closed path selection operation: The user draws a closed shape in the constructed data table, and uses the upper and lower boundary points and the left and right boundary points of the shape to determine the range to select cells.

6. The interactive selection and exploration method for understanding tabular data in an immersive environment according to claim 1, characterized in that: The specific method of step S4 is: Users can select specific cells or areas in the table using point selection, open path selection, and closed path selection methods according to their needs, and the current selection results are fed back in real time in the virtual reality space.

7. The interactive selection and exploration method for understanding tabular data in an immersive environment according to claim 1, characterized in that: The specific method of step S5 is: When the initial selection is completed and the selected area deviates, the user needs to further fine-tune the selected area to ensure that the selection meets the user's needs; the user dynamically modifies the boundaries of the selected area through the virtual reality device controller or bare hand gestures to quickly expand or shrink the selected area; (a) Extended selection area function: By default, it is in extended selection area mode; In the VR device controller interaction mode, the user pushes the left joystick to expand the border, and the edge of the selection area will extend in the specified direction; In the bare-hand gesture recognition interaction mode, the user points to the target direction with the left thumb, and the edge of the selection area will expand in the specified direction; (b) Shrinking the selected area: If the user needs to reduce the selected area, enable the shrinking function; In the VR device controller interaction mode, the user holds down the B button of the controller and pushes the left joystick to shrink the boundary of the selection area. The edge of the selection area will shrink in the specified direction. In the bare-hand gesture recognition interaction mode, the user triggers the contraction operation by making a fist with the right hand, and the user uses the left thumb to indicate the contraction direction, and the edge of the selection area will contract in the specified direction; (c) Confirm the selection: After the adjustment is completed, the user actively confirms the final selection area or the selection is automatically confirmed after waiting for 5 seconds; In the VR device controller interaction mode, press the A button to confirm the selection area; In the bare-hand gesture recognition interaction mode, make a right thumb gesture to confirm the selection area; Once the virtual reality device receives the confirmation signal, the selection result will be fixed for subsequent data processing, analysis or visualization operations.

Images