A multi-sensory reconfigurable tangible tabletop interaction system

Abstract

The present application relates to a multi-sensory reconfigurable tangible desktop interaction system. The present application comprises a tangible desktop physical interaction module, a scanning software end and an application program end. The tangible desktop physical interaction module serves as a signal input end, comprising an intelligent bottom plate and intelligent building block modules. The intelligent bottom plate is used to identify the intelligent building block modules placed on its surface and transmit information to the scanning software through a serial port. The scanning software end is used for signal processing, receiving the signals transmitted by the tangible desktop physical interaction module, processing the information and sending it to the application program end, and then transmitting the signals of the application program end back to the tangible desktop physical interaction module. The application program end serves as an information output end, and the multi-sensory interaction application can be customized according to requirements. The present application can reconfigure the intelligent building block modules, and the combination of different building block top sheets and bottom sheets can splice a variety of functional intelligent building blocks, and the stacking layer is not limited. The use of magnetic attraction principle makes the stacking of building blocks more simple and easy to use, while ensuring the stable connection of power and communication.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology and relates to a multi-sensory reconfigurable tangible desktop interaction system. Background Technology

[0002] During a child's early brain development, many cognitive, social, and emotional developmental stages are fostered through constructive play and exploration of the environment. For example, building blocks help improve children's spatial imagination and fine motor skills. While traditional building block toys offer these benefits, if these blocks could be imbued with intelligent perception and multi-sensory feedback, it would better help children explore the world around them, enabling them to learn, explore, and discover the world naturally, while simultaneously gaining physical, social, intellectual, and creative benefits. Currently, there are some electronic building block products on the market, but existing electronic building blocks offer limited feedback and lack interactivity. Furthermore, the appearance and functional structure of the building block modules are fixed, lacking flexible configuration options.

[0003] Tangible User Interfaces (TUIs) offer a novel way of interacting, combining physical and digital objects from an augmented reality perspective to create a seamless interactive space between the physical and virtual worlds. The core concept of TUIs is to make digital information tangible; TUIs "attempt to make computer interaction more like interaction with the real, non-digital world." One of the advantages of TUIs is their simple and intuitive operation, allowing users to manipulate digital information through natural, tactile interactions with objects. Tangible desktop systems utilize tangible modules embedded with interactive elements and multi-sensory feedback, creating an interface that allows interaction with real objects and can be used to promote children's fine motor coordination and cognitive development. These intelligent modules are capable of sensing children's hand operations such as grasping and inserting, providing context-aware feedback during learning through multiple senses, including visual, auditory, and tactile cues.

[0004] Digital augmentation can convey information that is impossible to represent in the physical world, providing opportunities to enhance children's further exploration, discovery, reflection, and collaboration. However, to encourage children to learn and understand through novel physical-digital coupling, open-ended and inspiring systems are needed. Currently, there are almost no relevant tangible interactive products and systems available for designing diverse educational games and providing tailored multi-sensory learning. Therefore, this invention provides a multi-sensory, reconfigurable tangible desktop interactive system that enables free 3D stacking, interactive control, and context-aware feedback through tactile, physical perception, and physical object interaction. Creative educators and application designers can use this tangible desktop interactive system to create unique interactive experiences. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-sensory reconfigurable tangible desktop interaction system for customizing multi-sensory (visual, auditory, tactile) interactive games. It is simple to operate, flexible in configuration, and highly playable.

[0006] This invention includes a tangible desktop entity interaction module, scanning software, and an application program. The tangible desktop entity interaction module, serving as a signal input terminal, includes a smart base plate and smart building block modules. The smart base plate identifies the smart building block modules placed on its surface and transmits the information to the scanning software via a serial port. The scanning software is used for signal processing. It receives signals transmitted from the tangible desktop entity interaction module, processes the information, sends it to the application program, and then transmits the signals back to the tangible desktop entity interaction module. The application program, serving as an information output terminal, allows for the creation of different scenarios and functional applications based on user needs.

[0007] The intelligent base plate includes a sensing module and a communication module. The sensing module includes multiple conductive contact surfaces for locating and identifying intelligent building blocks placed on its surface. The communication module is used to realize information communication with the scanning software.

[0008] The intelligent building block module includes a base plate, a stacking base plate, a base top plate, and a stacking top plate, which are assembled according to user needs. The base plate and stacking base plate each contain a microcontroller for storing information. The microcontroller stores the ID information of the building blocks and connects to the intelligent base plate communication module via spring-loaded pins to achieve signal communication. The base top plate has a touch switch and a vibration motor for sensing finger touches. The stacking top plate has conductive contact surfaces for sensing the building block modules stacked on top. The base plate and base top plate, as well as the stacking base plate and stacking top plate, are assembled by plugging and unplugging.

[0009] The intelligent building block module and the intelligent base plate are connected by spring pins to achieve power supply and signal transmission.

[0010] The sensing module of the intelligent base plate includes multiple conductive contact surfaces. Each conductive contact surface consists of a perforated conductive contact and an annular conductive contact. The perforated conductive contact is used to obtain power and determine direction. The two annular conductive contacts and the middle perforated conductive contact are data connection points for communication with the intelligent building block module.

[0011] The communication module of the intelligent baseboard includes a USB HUB module and two USB-to-serial modules designed using CH348Q chips. The USB HUB module is connected to the PC via a USB cable and sends data frames through the intelligent block scanning software running on the PC, and reads and writes the information and control information stored in the block module.

[0012] The transmitting signal lines of each row of conductive contact surfaces on the intelligent base plate are connected together, and the receiving signal lines are connected together. Data frames containing row number information are periodically transmitted through the transmitting signal lines. After receiving data with row number information, the intelligent building block reads the direction data, the layer number obtained from the data frame, and the ID information pre-stored in the internal microcontroller, and then sends the data frame to the intelligent building block scanning software through the intelligent base plate using the transmitting signal lines.

[0013] Based on the received data frames, the scanning software can obtain the row, column, direction, and layer information of the horizontal position of a certain ID information cell on the smart base plate.

[0014] The base top plate and the stacked top plate are respectively spliced ​​to the splicing base on the base plate and the stacked bottom plate by splicing pins.

[0015] The building block module has magnets at both ends and spring pins on the bottom. The smart base plate has corresponding hole-shaped and ring-shaped contacts on its surface and a magnetic attraction component on its bottom. The spring pins have five fixing interface holes and two contact ring-shaped contacts.

[0016] The application platform can be any of a projector, AR glasses, monitor, laptop, or tablet.

[0017] The base plate and base top plate are spliced ​​together to form a base block, which can be placed arbitrarily in the horizontal direction. It is connected to the intelligent base plate via magnetic spring pins to determine position and orientation in real time. The stacking base plate and stacking top plate are spliced ​​together to form a stacking block, which can be stacked arbitrarily in the vertical direction. Multiple stacking blocks are stacked vertically and connected to the intelligent base plate via a base-stacking block connection. The base plate and stacking top plate, spliced ​​together to form a base-stacking block connection, are used to connect the stacking blocks to the intelligent base plate.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] This invention offers novel solutions and optimizations for similar technologies in terms of reconfigurability, multi-sensory interaction, reusability, and spatial flexibility. The intelligent building block module provided by this invention is highly reconfigurable and modular, allowing for reconstruction based on user needs by replacing the base or top piece. The base top piece can sense hand touch and provide various modes of vibration feedback, while the application provides visual and auditory feedback, thus achieving multi-sensory interaction. This intelligent building block module can be spatially and temporally multiplexed, allowing for repeated use in different functional scenarios and settings. The power connection method of this invention can meet the power supply and data communication requirements of building blocks in any 3D stacked structure. Utilizing the magnetic principle makes building block stacking simpler and easier to use while ensuring stable power and communication connections. Furthermore, the technical solution provided by this invention overcomes the limitations of the number of stacking layers. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall functional architecture of the present invention;

[0021] Figure 2 for Figure 1 A schematic diagram of the structure of the intelligent base plate;

[0022] Figure 3 This is a schematic diagram of the data frame format of the present invention;

[0023] Figure 4 This is a structural diagram of the base film (without an outer shell);

[0024] Figure 5 This is a structural diagram of the base plate (without an outer shell);

[0025] Figure 6 A structural diagram of stacked films (without outer casing);

[0026] Figure 7 This is a structural diagram of the stacked top plates (without an outer shell);

[0027] Figure 8 This is a structural diagram of the basic building block module (without an outer shell);

[0028] Figure 9 A structural diagram of a basic stacked building block module (without an outer shell);

[0029] Figure 10 This is a structural diagram of a stacked building block module (without an outer shell).

[0030] Figure 11 for Figure 1 Diagram showing the connection between the intelligent base plate and the intelligent building block module;

[0031] Figure 12 A flowchart illustrating the process of the intelligent base plate sensing module identifying the building block module;

[0032] Figure 13 This is the principle for determining the number of stacking layers of the intelligent building blocks in this invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the specific embodiments described herein are only used to explain the present invention and do not limit it in any way.

[0034] like Figure 1 As shown, a multi-sensory reconfigurable tangible desktop interaction system includes a tangible desktop entity interaction module, a scanning software terminal, and an application terminal.

[0035] The tangible desktop interaction module includes a smart base and smart block modules. The smart base identifies the smart block modules placed on its surface and transmits the information to the scanning software via a serial port. The scanning software handles signal processing; it receives signals from the tangible desktop interaction module via the serial port, processes the information, sends it to the application program, and then sends the signals back to the tangible desktop interaction module. The application program provides various interaction scenarios and functions, and the application program and scanning software communicate bidirectionally using sockets.

[0036] like Figure 2 As shown, the smart base plate includes a contact surface 1, an insulating layer 2, and a magnetic attraction assembly. The magnetic attraction assembly includes an acrylic perforated plate 3 and a magnet 4.

[0037] Contact surface 1 is a printed circuit board (PCB). The PCB includes a sensing module and a communication module. The sensing module is used to receive information from the building block modules above the base plate, and the communication module is used for signal transmission.

[0038] In this embodiment, the sensing module has a total of 192 conductive contacts, arranged in 12 rows with 16 columns per row. Each conductive contact consists of a perforated conductive contact and annular conductive contacts. The perforated conductive contacts are used to obtain power and confirm direction, while the two annular conductive contacts and the middle perforated conductive contact are the data connection points for communication with the smart block module. On the diagonal of the four perforated conductive contacts, there is always one Vcc and one Gnd. Therefore, only one positive power supply will be in contact with the two pins 7 and 9 on the diagonal. By connecting a Schottky (0.3V dropout) diode in series with the two pins 7 and 9 on the diagonal of the smart block, the block module can obtain power.

[0039] The communication module includes a USB hub module and two USB-to-serial modules designed using the CH348Q chip, providing a total of 16 serial ports. The Tx signal lines of these 16 serial ports serve as data transmission lines for columns 1 to 16, and the Rx lines serve as data reception lines for rows 1 to 12. The USB hub module connects to the PC via a USB cable, and data frames are sent and information stored in the block modules is read and written via the intelligent block scanning software running on the PC.

[0040] The transmitting signal lines of each row of conductive contact surfaces on the smart baseboard are connected together, and the receiving signal lines are also connected together. Data frames containing row number information are periodically transmitted through the transmitting signal lines. After receiving data with row number information, the smart block module reads the direction data, the layer number obtained from the data frame, and the ID information pre-stored in the internal microcontroller, and then sends the data frame to the smart block scanning software through the smart baseboard using the transmitting signal lines.

[0041] Data frame format as follows Figure 3 As shown, the scanning software can obtain the row, column, direction, and layer information of the horizontal position of the smart base cell containing a certain ID based on the received data frame.

[0042] The intelligent building block module includes a base plate, stacking base plates, a base top plate, and stacking top plates. Users can reconstruct different functional intelligent building block modules by replacing the base plates or top plates. The base plate and stacking base plates contain microcontrollers for storing information. These microcontrollers store the ID information of the building blocks and connect to the intelligent base plate communication module via spring-loaded pins to achieve signal communication.

[0043] In this embodiment, the microcontroller is an STM8S207C8T6 main control chip.

[0044] like Figure 4 As shown, the base plate includes a magnet 4, a splicing base 5, and multiple spring pins. The magnetic force of the magnet 4 makes the connection between the spring pins and the contact surface of the smart base plate more stable. The base plate body 6 is a PCB board integrating an STM8S207C8T6 main control chip. It obtains power from two pins, the first spring pin 7 and the third spring pin 9, and confirms the direction from two pins, the first spring pin 7 and the second spring pin 8. It communicates with the smart base plate using the fourth spring pin 10 and the fifth spring pin 11.

[0045] like Figure 5 As shown, the base plate body 12 is a PCB board with a capacitive touch switch 23 on its surface and a vibration motor embedded therein. This component can sense finger touch and provide ten vibration amplitudes and ten vibration frequencies. There are two splicing pins 13 on the bottom of the body for splicing with the base plate splicing component 5 of the building block base.

[0046] like Figure 6 As shown, the stacked substrate includes a stacked substrate body 14, a magnet 4, a splicing base 5, and multiple spring pins. The stacked substrate body is a PCB board integrating an STM8S207C8T6 main control chip. It obtains power from two pins, the first spring pin 7 and the third spring pin 9, and confirms the direction from two pins, the first spring pin 7 and the second spring pin 8. It communicates with the stacked building block (basic substrate or stacked substrate) module stacked below it using the fourth spring pin 10 and the fifth spring pin 11.

[0047] like Figure 7 As shown, the stacking top plate body 15 is a PCB board with two annular contacts and five perforated contacts on its surface for contact with the stacking blocks stacked on top of it. A magnet 4 and two splicing pins 13 are located below the body for contact with the stacking bottom plate.

[0048] like Figure 8 As shown, the basic building blocks consist of a base plate 16 and a base plate 17, which can be placed arbitrarily in the horizontal direction (up / down / left / right). The direction is determined by the polarity (+ / -) of the first spring pin 7 and the second spring pin 8 on the basic building blocks. When both the first spring pin 7 and the second spring pin 8 are Gnd, the direction is up. When the first spring pin 7 is Vcc and the second spring pin 8 is Gnd, the direction is right. When both the first spring pin 7 and the second spring pin 8 are Vcc, the direction is down. When the first spring pin 7 is Gnd and the second spring pin 8 is Vcc, the direction is left. The basic building block module is connected to the intelligent base plate via the magnetic spring pins of the base plate, allowing for real-time determination of position and direction.

[0049] like Figure 9 As shown, the basic-stacked building blocks consist of a base plate 16 and a stacked top plate 18, which are spliced ​​together vertically. The basic-stacked building blocks are used to connect the smart base plate and the stacked building block module.

[0050] like Figure 10 As shown, this is a stacking block system including a stacking base plate 19 and a stacking top plate 18. The stacking blocks can be stacked vertically in any direction (up / down / left / right), with no limit on the number of stacking layers, and the system maintains good stability as the number of stacks increases. The stacking block modules are stacked one on top of the other, and are connected to the base-stacking block module by magnetic spring pins on the stacking base plate.

[0051] like Figure 11 As shown, the basic building block 20 is connected to the intelligent base plate via a magnetic spring pin on the base plate. The stacking building blocks 22 are stacked one on top of the other and connected to the intelligent base plate via the basic-stacked building blocks 21 for signal transmission.

[0052] like Figure 12 As shown, the specific process by which the intelligent base plate sensing module identifies the building block module is as follows:

[0053] The intelligent baseboard uses a column-by-column scanning method to query whether there is an intelligent block at the position in the nth column and mth row. The specific scanning steps are as follows:

[0054] (1) The first column initiates a query to see if there are any blocks in rows 1 to 12. If there is no response, it means that there are no smart blocks in the first column.

[0055] (2) The second column initiates a query to check if there are any blocks in rows 1 to 12. If there is a response in a row, the block information is added to the block module information list. This process continues until the sixteenth column completes its query, at which point one scan is finished.

[0056] (3) If a block at coordinate (m-th row, n-th column) is moved, and the query initiated by column n in the next scan does not receive a response from row m, it can be considered that the block at this position has been moved. In this case, the block information is removed from the list.

[0057] like Figure 13 As shown, the Rx_B of the first-layer smart block is connected to the base plate. When it receives control information from the base plate, the block module can assume it is at position Z0. The first-layer (Z0 position) block sends a request frame carrying its own Z0 information to the connected block module via Tx_T to read the information of the Z1 layer block. If there is no response, the smart block module can assume that there are no other blocks on it.

[0058] If the stacking number of the building blocks is greater than 1, the first-layer smart building block sends a request frame carrying its own Z0 information to the connected building blocks via Tx_T to read the information request frame of the Z1 layer building blocks. The second-layer building block receives the information request frame carrying its Z0 information from the connected building blocks via Rx_B. The second-layer building block determines whether it is the top Zj layer building block. If it is, it returns the request information to the next building block building block via Tx_B. If not, it continues to send the request Z0+n upwards until Z0+n = Zj before returning the information downwards. After j+1 queries, the first-layer smart building block can obtain the layer number Z and ID information of all the building blocks above it.

[0059] A multi-sensory, reconfigurable tangible desktop interaction system that can be used with multiple applications. Users can customize and develop solutions to create unique multi-sensory interactive experiences. Applications can be developed on any of the following devices: projectors, AR glasses, monitors, laptops, or tablets.

[0060] Let's take tangible programming games as an example. Tangible interfaces provide equal opportunities for participation and active learning. Compared to the traditional method of starting programming by inputting text code, tangible programming tools can help children learn programming in a simple and intuitive way, something previously considered difficult for them. By expressing procedural programming concepts such as sequences, repetition, conditions, functions, and parameters through hands-on activities, the learning curve for beginners in programming is significantly lowered.

[0061] In this embodiment, the scenario for the tangible programming application is a maze map. The game requires giving instructions to a character using tangible programming to guide the character from the starting point to the finish line. A tangible desktop entity interaction module is used as the console, and smart blocks are used to represent various programming elements and commands. Basic-stacked blocks and stacked block modules are used to represent concepts such as sequence, direction, loop, parameter, and function, respectively. The basic blocks are used as the program's run button.

[0062] The horizontal placement of the building blocks represents the logical sequence of commands, while the placement direction (up / down / left / right) indicates the direction in which the character moves. Command parameters are set by the stacking layer. After the command encoding is complete, clicking the run button (accompanied by haptic feedback) will run the program. The character in the maze game will then move according to the commands (accompanied by game sound effects). Once the character reaches the finish line, the game is considered successful, and a success screen will be displayed.

[0063] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and should not be considered as limitations on the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-sensory reconfigurable tangible desktop interaction system, comprising a tangible desktop entity interaction module, a scanning software terminal, and an application terminal, characterized in that: The tangible desktop entity interaction module serves as a signal input terminal, comprising a smart base plate and smart building block modules. The smart base plate identifies the smart building block modules placed on its surface and transmits the information to the scanning software via a serial port. The scanning software is used for signal processing; it receives signals transmitted from the tangible desktop entity interaction module, processes the information, sends it to the application terminal, and then transmits the signals from the application terminal back to the tangible desktop entity interaction module. The application terminal serves as an information output terminal, and different scenarios and functional applications are set according to user needs. The intelligent base plate includes a sensing module and a communication module. The sensing module includes multiple conductive contact surfaces for locating and identifying intelligent building blocks placed on its surface. The communication module is used to realize information communication with the scanning software. The intelligent building block module includes a base plate, a stacking base plate, a base top plate, and a stacking top plate, which can be assembled according to user needs. The base plate and the stacking base plate are equipped with microcontrollers for storing information. The microcontrollers are used to store the ID information of the building blocks and are connected to the intelligent base plate communication module through spring pins to realize signal communication. The base top plate is equipped with a touch switch and a vibration motor for sensing finger touch. The top stack plate is provided with conductive contact surfaces for sensing the building block modules stacked on top; the base plate and the base top plate, as well as the stack bottom plate and the stack top plate, are spliced ​​together by plugging and unplugging. The base plate and the base plate are spliced ​​together to form a basic building block, which can be placed arbitrarily in the horizontal direction; it is connected to the intelligent base plate through magnetic spring pins to determine the position information and orientation in real time. Stacking base pieces and stacking top pieces are spliced ​​together to form stacking blocks. Stacking blocks can be stacked arbitrarily in the vertical direction. Multiple stacking blocks are stacked one on top of the other and connected to the smart base plate through the base-stacking blocks. The base piece and stacking top pieces are spliced ​​together to form the base-stacking blocks, which are used to connect the stacking blocks to the smart base plate. The intelligent building block module and the intelligent base plate are connected by spring pins to achieve power supply and signal transmission.

2. The multi-sensory reconfigurable tangible desktop interaction system as described in claim 1, characterized in that: The sensing module of the intelligent base plate includes multiple conductive contact surfaces. Each conductive contact surface consists of a perforated conductive contact and an annular conductive contact. The perforated conductive contact is used to obtain power and determine direction. The two annular conductive contacts and the middle perforated conductive contact are data connection points for communication with the intelligent building block module.

3. The multi-sensory reconfigurable tangible desktop interaction system as described in claim 1, characterized in that: The communication module includes a USB hub module and two USB-to-serial modules designed using CH348Q chips. The USB hub module is connected to the PC via a USB cable and sends data frames through the intelligent block scanning software running on the PC, and reads and writes the information and control information stored in the block module.

4. The multi-sensory reconfigurable tangible desktop interaction system as described in claim 1, characterized in that: The transmitting signal lines of each row of conductive contact surfaces of the intelligent base plate are connected together, and the receiving signal lines are connected together. Data frames with row number information are periodically transmitted through the transmitting signal lines. After receiving the data with row number information, the intelligent building block reads the direction data, the layer number obtained from the data frame and the ID information pre-stored in the internal microcontroller, and sends the data frame to the intelligent building block scanning software through the intelligent base plate using the transmitting signal lines. Based on the received data frames, the scanning software can obtain the row, column, direction, and layer information of the horizontal position of a certain ID information cell on the smart base plate.

5. The multi-sensory reconfigurable tangible desktop interaction system as described in claim 1, characterized in that: The base top plate and the stacked top plate are respectively spliced ​​to the splicing base on the base plate and the stacked bottom plate by splicing pins.

6. The multi-sensory reconfigurable tangible desktop interaction system as described in claim 1, characterized in that: The building block module has magnets at both ends and spring pins on the bottom; the smart base plate has corresponding hole-shaped and ring-shaped contacts on its surface and a magnetic suction component on its bottom; the spring pin has five fixing interface holes and two contact ring-shaped contacts.

7. The multi-sensory reconfigurable tangible desktop interaction system as described in claim 1, characterized in that: The application platform can be any of a projector, AR glasses, monitor, laptop, or tablet.

Images