Biological specimen interactive keyboard system of modular hardware and working method thereof

Abstract

The invention discloses a biological specimen interactive keyboard system of modular hardware and a working method thereof, the system is composed of three modules: a specimen integrated keyboard module adopts a special keycap to package an entity biological specimen, and the specimen is used as a physical interface capable of triggering interaction; the multi-modal image acquisition module is fixed above the keyboard through a mechanical bracket, integrates visible light and infrared imaging units, and accurately acquires the characteristics of a specimen; the local processing and interaction control module serves as an off-line intelligent center, a local resource library and a special processing circuit are arranged in the local processing and interaction control module, feature matching and information output are achieved, all the modules cooperatively work through physical connection (such as a USB interface and an HDMI interface), and an independent hardware assembly line is formed. The system has the advantages of high integration, off-line operation, natural and smooth interaction and high reliability, and is suitable for teaching, scientific research, exhibition and other scenes.

Description

A modular hardware interactive keyboard system for biological specimens and its working method Technical Field

[0001] This invention belongs to the field of information processing technology, specifically relating to a modular hardware biological specimen interactive keyboard system and its working method. Background Technology

[0002] In teaching, research, and exhibition settings in fields such as biology and natural history, the core object of participants' cognition is physical specimens (such as plant leaves, seeds, and fossils). Traditional interactive processes require users to repeatedly switch between observing physical specimens and operating computers to retrieve digital information: first observe the specimen, then enter its name or number on the keyboard to search, and finally obtain information from the screen. This workflow is fragmented and discontinuous, severely disrupting focused observation and thinking, and reducing the efficiency and immersion of learning and research.

[0003] To improve this experience, the industry has made some technological attempts, but all have significant limitations: Currently, a common solution is to attach QR codes or RFID tags to specimens and then use dedicated scanning equipment to identify them and retrieve digital data. However, this solution relies on additional scanning equipment, a stable network connection, and pre-generated digital tags, making the equipment bulky and the process cumbersome. Its usability is greatly reduced, or even completely ineffective, in environments with poor network signals, such as field investigations, mobile displays in exhibition halls, or areas with poor network signals. Furthermore, as the core hub of human-computer interaction, the keyboard's function has long been limited to character and command input. Although "smart keyboards" with integrated small displays have emerged, their displayed content is usually unrelated to the physical object; they are merely another output window for computer information and cannot perceive or understand the physical object they carry. This means that even if a specimen is placed on the keyboard, the keyboard only treats it as an ordinary physical load, not a recognizable interactive object.

[0004] The two technical approaches described above reflect the fundamental problem of the current solutions: the observation space of physical specimens and the interaction space of digital information are physically and on equipment separate. Specimen display devices (such as display stands and scanners) are independent of the computer input system, forcing users' attention and actions to frequently switch between the two, making it impossible to achieve a natural and unified interactive experience of "what you see is what you get, what you touch is what you find".

[0005] In summary, existing technologies have failed to address the core issue of how to make physical specimens themselves a direct entry point for human-computer interaction in offline, mobile, and highly integrated scenarios. Therefore, there is an urgent need for an innovative keyboard system that can not only perform traditional input but also directly and proactively sense and identify the physical specimens placed on it, and present the corresponding digital information instantly and offline. This would transform the keyboard from a passive input tool into an intelligent interactive platform connecting the physical and digital worlds. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to provide a modular hardware biological specimen interactive keyboard system and its operating method.

[0007] The specific technical solution is as follows: A modular hardware biological specimen interactive keyboard system, which consists of the following hardware modules connected by physical connections: a specimen integrated keyboard module, including a standard keyboard base and multiple keycaps encapsulating physical biological specimens; a multimodal image acquisition module, fixed above the keyboard by a multi-directional adjustable mechanical bracket, used to acquire multimodal images of the specimens, with its optical axis forming a preset optimal angle of 45°±2° with the keyboard plane, which can minimize reflection and reduce image distortion; and a local processing and interactive control module, connected to the specimen integrated keyboard module and the multimodal image acquisition module through an internal bus, with a built-in local resource library and hardware processing unit, and implementing the following functional channels through its fixed hardware logic: signal reception and synchronization channel: adopting a hardware interrupt mechanism, receiving key trigger signals (including position encoding) from the keyboard module and image data streams from the image acquisition module to achieve microsecond-level time synchronization, ensuring that the acquired target image corresponds precisely to the trigger keycap.

[0008] Local Repository: An offline database is built using physical storage media (such as eMMC chips) to store multi-dimensional feature data, structured text, and multimedia information of biological specimens. The repository employs a hierarchical indexed storage architecture to ensure millisecond-level data retrieval even offline.

[0009] Feature matching and decision circuit: The features of the acquired multimodal images are extracted by hardened logic circuits or dedicated processors (such as NPU), and compared and analyzed quickly and in parallel with the pre-stored features in the local resource library. The matching algorithm is based on hardware-accelerated convolutional neural network (CNN) kernels, supports multi-feature fusion decision-making, and improves the accuracy to over 99%.

[0010] Output control interface: The matching results are directly converted into display instructions and output to the display device through a video interface (such as HDMI), forming an end-to-end offline information display path independent of the host computer operating system.

[0011] The specimen integrated keyboard module and the multimodal image acquisition module are connected to the local processing and interactive control module, which in turn is connected to the display device. The modules work together to achieve specimen recognition and information interaction without a network.

[0012] Furthermore, the keycaps in the specimen-integrated keyboard module have a three-layer physical structure: the bottom layer is a keycap base with circuit contacts (mechanically connected to the keyboard switch), the middle layer is a physical biological specimen, and the top layer is a high-transmittance encapsulation body (formed by casting and curing). The keycap base integrates a pressure sensor and supports multi-level triggering. This structure not only enables long-term preservation and intuitive display of the specimen, but more importantly, it transforms each specimen into a physical trigger that can be precisely located.

[0013] Furthermore, the multimodal image acquisition module includes at least one visible light camera and one infrared imaging unit, and integrates a supplementary lighting system. The infrared imaging unit has a wavelength range of 800-900nm, which can penetrate the surface of the specimen and collect non-visible light features such as its internal structure or thickness distribution. This provides an additional dimension of physical information to solve the problem of identifying incomplete or wrinkled specimens. The visible light camera is equipped with a 5-megapixel CMOS sensor and an f / 2.0 large aperture macro lens with a minimum focusing distance of 3cm. It is used for high-fidelity acquisition of macroscopic and microscopic appearance features such as color and texture of the specimen. The supplementary lighting system, such as a ring LED supplementary light, supports automatic white balance and 256 levels of brightness adjustment to ensure stable and uniform imaging effects under any lighting conditions. The module also integrates a polarizer component, which can selectively filter specular reflection light and enhance texture feature extraction. The bracket is driven by a stepper motor and supports software-defined programmable viewing angle adjustment to adapt to different keycap sizes.

[0014] Furthermore, the local processing and interactive control module is an embedded hardware unit, which includes a physical storage medium for storing specimen information and a dedicated processing circuit for feature comparison. The dedicated processing circuit is an NPU that supports INT8 quantization inference.

[0015] A biological specimen interaction method based on the above system includes the following steps: S1: The interaction signal is triggered by pressing the keycap of the specimen integrated keyboard module. One of the signals is sent to the computer as a regular keyboard input signal, and the other is sent to the local processing and interaction control module through the internal circuit as a dedicated trigger signal. The trigger signal precisely contains the physical position code of the pressed keycap, providing target positioning for subsequent accurate image acquisition. The trigger signal uses differential encoding, which has strong anti-interference capabilities. The position code and keycap coordinate mapping table are fixed in ROM to ensure zero-error positioning. S2: The local processing and interactive control module controls the multimodal image acquisition module to synchronously acquire images of the target specimen based on the trigger signal. The multimodal image acquisition module automatically adjusts the focus or quickly focuses on the specimen on the target keycap based on the received position code information, and simultaneously acquires visible light and infrared images. The acquisition process uses multi-frame synthesis technology to improve the image signal-to-noise ratio. The infrared image and visible light image are pixel-level aligned to provide redundant data for feature extraction. S3: The local processing and interactive control module performs hardware-level feature extraction based on the acquired images and matches them with the local resource library. This module uses its built-in hardware processing unit to preprocess the image (such as denoising and enhancement) and extract key physical features (such as shape, texture, and contour). Subsequently, the feature matching and decision circuit performs parallel comparison and retrieval of these features against pre-stored feature data in the local resource library. This process is entirely completed on local hardware, with no data uploaded to the network or relying on host computation. Feature extraction uses a hardware-based SIFT algorithm, optimizing the computation path. A confidence threshold is introduced in the matching stage; if the value falls below the threshold, a secondary acquisition process is initiated to reduce the false recognition rate. S4: After successful matching, the local processing and interactive control module retrieves relevant information and outputs it to the display device, controlling the displayed information. Specifically, the local processing and interactive control module retrieves the corresponding image, text, and video information from the local resource library. Through its output control interface, the information is generated and directly output to the display device for display. The output supports 4K resolution and integrates a GPU-accelerated rendering engine to achieve smooth animation transitions. The information content can be dynamically updated via an offline SD card slot.

[0016] Furthermore, the triggering interaction signal in step S1 includes the physical position information of the pressed keycap, which is encoded using Gray code.

[0017] Furthermore, in step S2, the multimodal image acquisition module is started via a hardware interrupt signal, and the interrupt priority can be configured programmably.

[0018] Furthermore, steps S3 and S4 are completed entirely within the local processing and interactive control module through hardware logic, which is customized based on the RISC-V instruction set.

[0019] The advantages of this invention are: high integration and offline operation: by physically integrating the three major hardware modules of specimen encapsulation, image acquisition, and local processing into the keyboard system, a compact, integrated device is achieved. All recognition and interaction logic is completed collaboratively through hardware modules, completely eliminating the dependence on network connectivity, making it suitable for any environment. The modular design supports customized expansion, such as adding an odor release module or a haptic feedback unit to enhance immersion.

[0020] Natural and smooth interaction: It realizes a direct interaction mode of "pressing the specimen keycap to trigger information display", using physical specimens as the interaction entry point to seamlessly connect the physical world and digital information. The user experience is intuitive and natural. Multi-level pressure sensing supports gesture interaction, such as light pressure for preview and heavy pressure for details, which enriches the operation level.

[0021] Clear functional separation: This invention strictly distinguishes the functional boundaries of hardware modules. The keyboard module is responsible for input and specimen carrying, the image acquisition module is responsible for physical information capture, and the local processing module is responsible for data matching and output control. Each module performs its own function, with clear logic, avoiding mixed descriptions of software and hardware functions. The solidified hardware logic reduces system complexity and facilitates large-scale production and maintenance.

[0022] High reliability: Based on hardware circuits and fixed logic, the implementation method offers more timely response and higher system stability compared to pure software solutions, and is less affected by computer operating systems or software environments. Attached Figure Description

[0023] Figure 1 is a schematic diagram of the physical connection of the hardware modules of the system of the present invention; Figure 2 is a schematic diagram of the cross-sectional structure of the specimen keycap of the present invention; Figure 3 is a schematic diagram of the installation and structure of the multimodal image acquisition module of the present invention; Figure 4 is a flowchart of the main process of the method of the present invention.

[0024] In the diagram: 1. Specimen keyboard integrated module; 2. Local processing and interactive control module; 3. Multimodal image acquisition module; 4. Display device; 5. Host computer; 6. Transparent encapsulation body; 7. Biological specimen; 8. Keycap substrate; 9. Visible light camera; 10. Infrared imaging unit; 11. Ring lighting system; 12. Adjustable robotic arm; 13. Keyboard substrate. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited thereto.

[0026] As shown in Figure 1, a modular hardware biological specimen interactive keyboard system is constructed by physically connecting the following hardware modules: a specimen integrated keyboard module 1, including a standard keyboard base 13 and multiple keycaps encapsulating physical biological specimens 7; a multimodal image acquisition module 2, fixed above the keyboard by a mechanical bracket (adjustable robotic arm 12), used to acquire multimodal images of the specimens; and a local processing and interactive control module 3, connected to the specimen integrated keyboard module 1 and the multimodal image acquisition module 2 via an internal bus, with a built-in local resource library and hardware processing unit. The specimen integrated keyboard module 1 and the multimodal image acquisition module 2 are respectively connected to the local processing and interactive control module 3, which is connected to a display device 4. The modules work together to achieve specimen identification and information interaction without a network.

[0027] As shown in Figure 2, the keycaps in the specimen-integrated keyboard module 1 have a three-layer physical structure: the bottom layer is the keycap base 8 with circuit contacts, the middle layer is the physical biological specimen 7, and the top layer is a highly transparent encapsulation body 6. The keycap base 8 integrates a pressure sensor and supports multi-level triggering.

[0028] As shown in Figure 3, the multimodal image acquisition module 2 includes at least one visible light camera 9 and one infrared imaging unit 10, and integrates a ring-shaped supplementary lighting system 11. It is adjusted by an adjustable robotic arm 12. The wavelength range of the infrared imaging unit is 800-900nm.

[0029] As shown in Figure 4, a biological specimen interaction method based on the above system includes the following steps: S1: The interaction signal is triggered by pressing the keycap of the specimen integrated keyboard module. One of the signals is sent to the computer as a regular keyboard input signal, and the other is sent to the local processing and interaction control module through the internal circuit as a dedicated trigger signal. The trigger signal precisely contains the physical position code of the pressed keycap, providing target positioning for subsequent accurate image acquisition. The trigger signal uses differential encoding, which has strong anti-interference capabilities. The position code and keycap coordinate mapping table are fixed in ROM to ensure zero-error positioning. S2: The local processing and interactive control module controls the multimodal image acquisition module to synchronously acquire images of the target specimen based on the trigger signal. The multimodal image acquisition module automatically adjusts the focus or quickly focuses on the specimen on the target keycap based on the received position code information, and simultaneously acquires visible light and infrared images. The acquisition process uses multi-frame synthesis technology to improve the image signal-to-noise ratio. The infrared image and visible light image are pixel-level aligned to provide redundant data for feature extraction. S3: The local processing and interactive control module performs hardware-level feature extraction based on the acquired images and matches them with the local resource library. This module uses its built-in hardware processing unit to preprocess the image (such as denoising and enhancement) and extract key physical features (such as shape, texture, and contour). Subsequently, the feature matching and decision circuit performs parallel comparison and retrieval of these features against pre-stored feature data in the local resource library. This process is entirely completed on local hardware, with no data uploaded to the network or relying on host computation. Feature extraction uses a hardware-based SIFT algorithm, optimizing the computation path. A confidence threshold is introduced in the matching stage; if the value falls below the threshold, a secondary acquisition process is initiated to reduce the false recognition rate. S4: After successful matching, the local processing and interactive control module retrieves relevant information and outputs it to the display device, controlling the displayed information. Specifically, the local processing and interactive control module retrieves the corresponding image, text, and video information from the local resource library. Through its output control interface, the information is generated and directly output to the display device for display. The output supports 4K resolution and integrates a GPU-accelerated rendering engine to achieve smooth animation transitions. The information content can be dynamically updated via an offline SD card slot.

[0030] In a specific implementation case, a prototype system containing 16 biological specimen keycaps was built. Test results show that the system performs excellently in the following key performance indicators: Response time: From the user pressing the specimen keycap (starting S1) to the display device fully displaying the corresponding specimen's detailed information (ending S4), the average response time for the entire offline recognition and display process is consistently within 1.5 seconds. Specifically, hardware interrupt triggering and image acquisition synchronization (S1-S2) takes less than 200 milliseconds, local feature matching (S3) is the core time-consuming step, averaging approximately 800 milliseconds, and information rendering output (S4) takes approximately 400 milliseconds. This speed is significantly better than solutions requiring network requests (typically exceeding 3-5 seconds), achieving a smooth "instant display" experience. By optimizing the interrupt handling circuitry, the system employs a time-slice rotation mechanism when multiple keys are triggered concurrently, avoiding resource contention and ensuring a response time standard deviation of less than 50 milliseconds.

[0031] Recognition accuracy: In a test involving 5000 random press triggers across 100 common plant species, the system achieved a 98.5% accuracy rate for intact specimens. Notably, for specimens with partial damage or wrinkles, the accuracy rate remained above 92% due to the auxiliary features of the infrared imaging unit, fully demonstrating the superiority of multimodal hardware acquisition over a single vision solution. The test environment covered strong light, low light, and complex backgrounds, and the system's robustness was verified. Accuracy can be further improved with expansion of the local resource library.

[0032] Offline reliability: During a 72-hour continuous offline operation test, the system did not experience any failures due to local processing module overload or resource library retrieval errors, demonstrating stable and reliable system functionality. The power management module effectively controlled energy consumption, meeting the needs of long-term teaching or exhibitions. Extended testing included high and low temperature environments (-10°C to 50°C) and humidity variations (30%-80%RH), during which the system operated normally, demonstrating strong adaptability.

Claims

1. A modular hardware biological specimen interactive keyboard system, characterized in that, The system consists of the following hardware modules physically connected together: a specimen-integrated keyboard module, including a standard keyboard base and multiple keycaps encapsulating actual biological specimens; a multimodal image acquisition module, fixed above the keyboard by a mechanical bracket, used to acquire multimodal images of the specimens; and a local processing and interactive control module, connected to the specimen-integrated keyboard module and the multimodal image acquisition module via an internal bus, with a built-in local resource library and hardware processing unit. The specimen-integrated keyboard module and the multimodal image acquisition module are respectively connected to the local processing and interactive control module, which is connected to a display device. All modules work together to achieve specimen identification and information interaction without a network.

2. The modular hardware biological specimen interactive keyboard system as described in claim 1, characterized in that, The keycaps in the specimen-integrated keyboard module have a three-layer physical structure: the bottom layer is the keycap base with circuit contacts, the middle layer is the actual biological specimen, and the top layer is a high-transmittance encapsulation. The keycap base integrates a pressure sensor and supports multi-level triggering.

3. The modular hardware biological specimen interactive keyboard system as described in claim 1, characterized in that, The multimodal image acquisition module includes at least one visible light camera and one infrared imaging unit, and integrates a supplementary lighting system. The wavelength range of the infrared imaging unit is 800-900nm.

4. The modular hardware biological specimen interactive keyboard system as described in claim 1, characterized in that, The local processing and interactive control module is an embedded hardware unit, which includes a physical storage medium for storing specimen information and a dedicated processing circuit for feature comparison. The dedicated processing circuit is an NPU that supports INT8 quantization inference.

5. A method for interacting with biological specimens based on any one of the systems described in claims 1-4, characterized in that, The process includes the following steps: S1: Triggering an interaction signal by pressing the keycap of the specimen-integrated keyboard module; S2: The local processing and interaction control module controls the multimodal image acquisition module to synchronously acquire images of the target specimen based on the trigger signal; S3: The local processing and interaction control module performs hardware-level feature extraction based on the acquired images and matches them with the local resource library; S4: After successful matching, the local processing and interaction control module retrieves relevant information and outputs it to the display device, and controls the displayed information.

6. The biological specimen interaction method as described in claim 5, characterized in that, The interaction signal triggered in step S1 contains the physical position information of the pressed keycap, which is encoded using Gray code.

7. The biological specimen interaction method as described in claim 5, characterized in that, In step S2, the multimodal image acquisition module is started via a hardware interrupt signal, and the interrupt priority can be configured programmably.

8. The biological specimen interaction method as described in claim 5, characterized in that, Steps S3 and S4 are completed entirely within the local processing and interactive control module through hardware logic, which is customized based on the RISC-V instruction set.

Images