A multi-sensory information conversion full perception system and its tactical application method

Abstract

The application discloses a multi-sensory information conversion full perception system and a tactical application method thereof, and belongs to the technical field of perception assistance and tactical communication. The system realizes multi-channel perception by converting environmental information into non-visual sensory signals. The system comprises an information acquisition module (ultrasonic sensor, infrared sensor, camera, etc.), an information processing module (AI algorithm, signal conversion unit) and a multi-sensory output module (audio feedback device, tactile feedback device). The spatial obstacle information is converted into different tones through ultrasonic ranging and tone mapping technology. The target position information is converted into tactile signals through a hydraulic tactile feedback device and wireless transmission technology. The application can be used for blind navigation assistance, night combat, special operation and the like, solves the limitations of traditional visual dependence, realizes zero-light and zero-sound concealed information transmission, and has important social value and military application prospect.

Description

Technical Field

[0001] This invention relates to the field of perception assistance and tactical communication technology, and in particular to a multi-sensory information conversion full-sensory system and its tactical application method. Background Technology

[0002] Humans primarily perceive the external environment through sight and hearing, with vision playing a dominant role. However, in certain specific scenarios, visual perception exhibits significant limitations: 1. Visually impaired groups: Blind or visually impaired people cannot obtain environmental information through vision. Traditional guide methods (such as guide dogs and white canes) have problems such as limited feedback information and inability to provide accurate distance and location.

[0003] 2. Nighttime or low-light environments: In scenarios such as nighttime combat, mining operations, and rescue, insufficient light or complete darkness limits visual perception. Traditional night vision equipment suffers from the following problems: Night vision devices require weak light sources or active infrared illumination, which easily reveals their location; The equipment is bulky, which hinders operational flexibility; Limited field of vision and poor peripheral perception.

[0004] 3. Covert communication requirements: In special operations, reconnaissance and other missions, information needs to be transmitted under conditions of zero light and zero sound. Traditional voice communication, gesture signals and light signals all pose a risk of exposure.

[0005] In the existing technology, there are some sensory substitution technologies, such as: Ultrasonic navigation devices for the visually impaired: These devices use ultrasonic ranging to emit a beeping sound to alert the user of obstacles, but the feedback information is limited and cannot distinguish between the types of obstacles or their precise locations. Haptic feedback systems, such as vibration motors and haptic gloves, are mainly used for virtual reality interaction, but they have low resolution and cannot transmit complex spatial information. Bone conduction headphones: solve the communication problem in noisy environments, but are essentially still auditory channels and do not explore other sensory potentials.

[0006] These technologies have failed to fully utilize the human body's multi-sensory channels, nor have they achieved efficient and accurate conversion of environmental information into non-visual signals. Summary of the Invention

[0007] Purpose of the Invention: To address the shortcomings of existing technologies, the purpose of this invention is to provide a multi-sensory information conversion system and method that converts environmental information into non-visual signals such as auditory and tactile senses, achieving the following objectives: 1. To provide blind or visually impaired individuals with environmental perception abilities similar to "bat vision," enabling them to locate objects by sound; 2. To provide combat personnel with a covert, zero-light and zero-sound method of information transmission at night or in low-light environments; 3. Fully develop the tactile perception potential of human skin and establish a high-bandwidth, high-precision tactile information channel; 4. By combining technologies such as IoT, AI, and edge computing, we can achieve real-time, intelligent multi-sensory information fusion and transformation.

[0008] Technical solution: To solve the above-mentioned technical problems, according to one aspect of the present invention, more specifically, a multi-sensory information conversion full-perception system, comprising: 1. Information Collection Module Ultrasonic sensor array: Multiple ultrasonic sensors are distributed in wearable devices such as helmets, glasses, and vests, providing 360-degree omnidirectional ranging with a detection range of 0.1m-10m and an accuracy of ±2cm; Infrared sensors: used for nighttime target detection, capable of identifying human heat sources, vehicles, etc. Optical sensors: including cameras and lidar, used for environmental modeling and target recognition; IMU (Inertial Measurement Unit): Acquires user attitude and motion state for coordinate system transformation.

[0009] 2. Information Processing Module Edge computing unit: Employs embedded AI chips (such as NVIDIA Jetson and Huawei Ascend) to achieve local real-time processing with a latency of <50ms; Distance calculation unit: fuses data from multiple sensors to calculate the distance, orientation, and height of obstacles; Target recognition unit: Based on deep learning models (such as YOLO, ResNet), it identifies target types (people, vehicles, buildings, animals, etc.); Signal mapping unit: Based on preset rules or AI learning models, it maps environmental information into sensory signal parameters.

[0010] 3. Multi-sensory output module (1) Audio feedback device Use bone conduction headphones or air conduction headphones; Supports two-channel stereo, indicating direction through the volume difference and phase difference between the left and right channels; Pitch mapping rules: Distance: Close distance (0-2m) → high frequency sound (2000-4000Hz), medium distance (2-5m) → mid frequency sound (1000-2000Hz), long distance (5-10m) → low frequency sound (500-1000Hz); Height: Head height → sharp tone, waist height → round tone, foot height → deep tone; Urgency level: Approaching obstacle at high speed → Continuous, rapid beeping; Type: Person → Voice prompt "person", Vehicle → Engine sound simulation.

[0011] (2) Haptic feedback device Tactile garment structure: includes an inner lining layer, a hydraulic layer, and an outer protective layer, with a total thickness of <5mm; Hydraulic unit array: 64 miniature hydraulic units are distributed throughout the torso (chest, abdomen, back), each unit including: Miniature hydraulic pump (8mm diameter, 5mm stroke); Pressure sensor, monitors applied pressure in real time; Flexible liquid capsule, with a skin contact area diameter of 20mm; Control circuit: Distributed control is adopted, with each of the eight hydraulic units managed by a microcontroller, interconnected via CAN bus or wireless Mesh network; Wireless communication module: Supports multiple protocols such as WiFi 6, Bluetooth 5.2, and LoRa, with encrypted transmission; Tactile encoding rules: Orientation: Activate the hydraulic unit in the corresponding direction. For example, if the target is in front, activate the chest unit; if the target is to the left rear, activate the left back unit. Distance: Pressure intensity, close distance → high pressure (50kPa), long distance → low pressure (10kPa); Urgency level: Vibration frequency, urgent → high frequency pulse (10Hz), normal → low frequency pulse (2Hz); Mode: Continuous pressure indicates a static target, and pulsed pressure indicates a moving target.

[0012] 4. System Integration and Communication It adopts a modular design, with each module connected via standard interfaces (USB-C, wireless); It supports standalone mode (sensors and feedback devices are directly connected) and network mode (information is provided through external devices such as drones and command centers); Power management: Lithium battery powered, with a battery life of >8 hours, supports fast charging and solar charging.

[0013] Application Scenario Examples Example 1: Navigation Assistance for the Blind Blind users wear smart glasses with integrated ultrasonic sensor arrays and bone conduction headphones: 1. When a waist-high obstacle (such as a table) is detected 3 meters ahead, the system emits a 1500Hz mid-frequency sound, with the volume slightly louder on the left, indicating that the obstacle is located to the left front. 2. When an obstacle (such as a tree branch) at head height is detected 1 meter ahead, the system emits a high-frequency sharp sound at 3000Hz and provides a voice prompt "danger ahead"; 3. A moving target (person) is detected 5 meters to the right. The system emits a rhythmic low-frequency sound and provides a voice prompt "person on right". 4. Through weeks of training, users develop a conditioned reflex to the "pitch-space" mapping, enabling them to locate sounds like bats.

[0014] Example 2: Individual Night Combat Soldiers wearing haptic suits establish encrypted communication links with aerial drones: 1. The drone, at an altitude of 300 meters, detected three enemy personnel 200 meters ahead using its infrared sensor; 2. The drone transmits the target's location (azimuth angle 30 degrees, distance 200 meters) to the haptic clothing via a LoRa encrypted channel; 3. The haptic garment activates the hydraulic unit on the right front chest, applying moderate pressure (30 kPa) for 3 seconds, followed by pulses at a frequency of 2 Hz, indicating "right front, medium distance, moving target"; 4. Soldiers adjust their direction of advance based on tactile signals, without needing to use night vision goggles or voice communication, maintaining a state of zero light and zero sound concealment throughout the process; 5. When approaching within 50 meters, the pressure increases to 50 kPa and the pulse frequency increases to 5 Hz, indicating that the distance has entered a dangerous zone.

[0015] Example 3: Multi-person cooperative tactics Three soldiers were on an infiltration mission, all wearing tactile vests and connected to the same command system. 1. The commander, through drone reconnaissance, discovered that the target building had two exits and three guards; 2. The commander uses tactile coding to send different instructions to three soldiers: Soldier A: Continuous chest pressure + left-side pulse, meaning "Keep moving forward, pay attention to the left side"; Soldier B: Continuous pressure on the right side, meaning "moving to the right"; Soldier C: Continuous pressure on the back, meaning "retreat and remain vigilant"; 3. The three soldiers completed the encirclement based on tactile commands, communicating silently throughout the process.

[0016] Example 4: Haptic AI Adaptive Learning The system has a built-in reinforcement learning model that optimizes the mapping strategy based on user feedback. 1. In the initial stage, a preset linear mapping rule (distance-pressure linear relationship) is used; 2. Record user reaction time, path selection, collision count, and other data in different scenarios; 3. Use the Deep Q-Network (DQN) algorithm to optimize the mapping parameters with "reducing collisions and improving movement efficiency" as the reward; 4. After 100 hours of training, the system learned that for this user, tactile sensitivity should be increased (pressure threshold lowered) in crowded environments, and information density should be reduced (pulse frequency decreased) in open environments. 5. The system is personalized based on individual user differences (skin sensitivity, reaction speed).

[0017] Beneficial effects: 1. Expanding sensory channels: Breaking through visual dependence, making full use of auditory, tactile and other senses to achieve multi-channel redundancy and improve the robustness of environmental perception; 2. High information bandwidth: Through 64 independent tactile units, it can simultaneously transmit multi-dimensional information such as orientation, distance, and type, with an information bandwidth far exceeding that of traditional vibration motors (approximately 10 times higher); 3. Zero-exposure covert communication: Tactile signals are light-free, sound-free, and free of electromagnetic radiation (at the receiving end), making them difficult for the enemy to detect in tactical environments; 4. Real-time performance: The edge computing architecture guarantees end-to-end latency of <50ms, meeting the needs for rapid response in dynamic environments; 5. Intelligent: AI-based target recognition and adaptive mapping improve the accuracy of information delivery and user-friendliness; 6. Modular and scalable: Supports flexible configuration of sensors and feedback devices according to application scenarios, such as audio feedback for blind navigation and tactile feedback for tactical applications; 7. Social Value: Providing more efficient environmental perception tools for approximately 250 million visually impaired people worldwide, significantly improving their quality of life; 8. Military value: Enhances the stealth and coordination efficiency of nighttime and special operations, and reduces casualties; 9. Technological Innovation: For the first time, a complete conversion framework for "environmental information → multi-sensory signals" was systematically proposed, providing new ideas for VR / AR, human-computer interface and other fields. Attached Figure Description

[0018] Figure 1 System overall architecture diagram; Figure 2 Layout diagram of ultrasonic sensor array (helmet type); Figure 3 Cross-sectional view of the tactile garment structure; Figure 4 Detailed structural diagram of the hydraulic unit; Figure 5 Audio feedback mapping rule flowchart; Figure 6 Flowchart of haptic feedback mapping rules; Figure 7 : Schematic diagram of individual soldier tactical application scenarios; Figure 8 : Illustration of navigation application scenarios for the blind; Figure 9 Information processing flowchart; Figure 10 : Diagram of multi-person collaborative tactical formation. Detailed Implementation

[0019] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] like Figure 1 , Figure 2 As shown, the system includes: The smart glasses frame integrates 8 ultrasonic sensors (HC-SR04 or similar models), located in 8 directions: front, back, left, right, left front, right front, left rear, and right rear. Each sensor has a field of view of 15 degrees and a detection distance of 0.2m-4m. Embedded processor (Raspberry Pi 4 or STM32) running a real-time operating system (such as FreeRTOS); Bone conduction headphones (such as AfterShokz) have a frequency response range of 20Hz-20kHz; The lithium battery (3.7V 2000mAh) provides approximately 10 hours of battery life.

[0021] Workflow (see) Figure 9 ): 1. Eight ultrasonic sensors cyclically emit ultrasonic pulses (40kHz) at a frequency of 50Hz and receive the echoes; 2. The processor calculates the distance to obstacles in each direction: Distance = (Echo time × Speed ​​of sound) / 2; 3. For detected obstacles, calculate the audio parameters based on the distance d (meters): Frequency f = 5000 / (d + 1) Hz (the closer the distance, the higher the frequency) Volume V = 60 × 10 d dB (The closer the distance, the louder the volume) Left and right channel ratio: Based on the obstacle azimuth angle θ, left channel = cos(θ), right channel = sin(θ). 4. Generate dual-channel audio signals and output them through bone conduction headphones; 5. When users hear prompts from different directions and frequencies, their brains integrate the information and form spatial cognition.

[0022] Optimization strategy: Priority mechanism: the nearest obstacle has the highest priority to avoid information overload; Tonal distinction: Use a triangle wave for obstacles at head height, a sine wave for waist height, and a square wave for feet height; Adaptive volume: Automatically adjusts the output volume based on ambient noise.

[0023] Implementation Method 2: Hydraulic-based haptic feedback clothing like Figure 3 , Figure 4 As shown, the haptic garment includes: Inner lining: Elastic and breathable fabric (such as Lycra) that conforms to the skin; Hydraulic layer: 64 hydraulic units are embedded in a flexible substrate with a unit spacing of 40mm, covering the main area of ​​the torso; Outer protective layer: abrasion-resistant and waterproof fabric.

[0024] Each hydraulic unit includes: Miniature hydraulic pump: driven by piezoelectric ceramic, operating voltage 12V, maximum pressure 80kPa, response time <20ms; Flexible fluid capsule: made of silicone, 20mm in diameter and 3mm thick, filled with saline or silicone oil; Pressure sensor: Thin-film pressure sensor, which monitors the pressure applied to the skin in real time and enables closed-loop control; Microcontroller: Each of the eight units shares one STM32 microcontroller, which controls the hydraulic pump via PWM.

[0025] Control architecture: The main control unit (ARM Cortex-A core) receives wireless signals and analyzes the target's location, distance, and type. Commands are sent to each distributed microcontroller via the CAN bus; The microcontroller drives the hydraulic pump, adjusts the pressure in real time, and closed-loop control ensures accuracy.

[0026] Example of tactile coding: Target azimuth 30 degrees (right front), distance 100 meters: Activate hydraulic units 15-18 (right anterior chest area), pressure 20 kPa, last for 2 seconds; The target rapidly approaches to within 50 meters: the pressure is increased to 40 kPa, pulsed at a frequency of 5 Hz; Simultaneously, the second target on the left appears: activate hydraulic units 40-43 (left waist area), independently coded.

[0027] Ergonomic design: Avoiding bony prominences, the hydraulic units are concentrated in areas rich in muscle (pectoralis major, rectus abdominis, latissimus dorsi). Adjustable tightness to fit different body types; Modular design allows for quick replacement of damaged units.

[0028] Implementation Method 3: Unmanned Aerial Vehicle Cooperative Tactical System System components: Reconnaissance drone: equipped with an infrared thermal imager (640×512 resolution, 2km detection range) and an AI edge computing module (Jetson Xavier NX); Haptic clothing: as described in implementation method two; Encrypted communication link: LoRa (frequency 433MHz, bandwidth 125kHz, encryption algorithm AES-256).

[0029] Workflow: 1. The drone cruises at an altitude of 300 meters, and the infrared camera captures thermal images at 30fps; 2. The AI ​​module runs the object detection model (YOLOv5 optimized version) to identify people and vehicles, and outputs bounding boxes and confidence scores; 3. Calculate the target's global coordinates based on the UAV's GPS and attitude data; 4. Calculate the target's azimuth and distance relative to the soldier based on the soldier's GPS position; 5. Encode into tactile control commands: ``` Data packet format: [Target ID (1 byte)] [Azimuth (2 bytes)] [Distance (2 bytes)] [Type (1 byte)] [CRC (2 bytes)] ``` 6. Transmitted to the soldier's haptic suit via LoRa; 7. The haptic clothing decodes the sensor and drives the corresponding hydraulic unit; 8. Soldiers adjust their tactical movements based on tactile feedback.

[0030] Multi-objective processing: A maximum of 8 targets can be tracked simultaneously; Sort by threat level, prioritizing the display of the most recent or fastest-moving targets; Through time-division multiplexing, each target tactile signal lasts for 1 second and is refreshed in turn.

[0031] Implementation Method 4: AI Adaptive Mapping Algorithm To optimize user experience, the system employs reinforcement learning algorithms to learn the optimal mapping strategy.

[0032] State space: Environmental conditions: Obstacle distribution (distance, location, type); User status: movement speed, historical trajectory, collision records; Physiological state: Skin sensitivity (adaptively measured via pressure sensor).

[0033] Action space: Audio parameters: frequency, volume, timbre, rhythm; Tactile parameters: pressure intensity, pulse frequency, and activation unit selection.

[0034] Reward function: Positive Rewards: Successfully avoiding obstacles (+10), efficiently reaching the target (+distance / time); Negative rewards: Collision (-50), Taking a longer route (-distance deviation).

[0035] algorithm: Use Deep Q-Network (DQN) or Policy Gradient (PPO); Offline training: Pre-training the base model in a simulated environment; Online fine-tuning: The model learns continuously during user use and updates its parameters weekly; Personalization: Maintain independent model parameters for each user.

[0036] Experimental results: Initial (rule mapping): Collision rate for blind navigation is 15%, average speed is 0.8 m / s; After 50 hours of training: the collision rate decreased to 3%, and the average speed increased to 1.2 m / s; Individual differences in tactile sensitivity: Users with sensitive skin require 30% less stimulation intensity.

[0037] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A multi-sensory information conversion full-perception system, characterized in that, include: An information acquisition module is used to collect environmental information, including at least one of ultrasonic sensors, infrared sensors, and optical sensors; An information processing module, connected to the information acquisition module, is used to analyze and process the acquired environmental information and convert it into non-visual sensory signals. The information processing module includes a distance calculation unit, a target recognition unit, and a signal mapping unit. A multi-sensory output module, connected to the information processing module, is used to output the converted signal to the user in an auditory or tactile manner, including an audio feedback device and / or a tactile feedback device. The signal mapping unit maps obstacles or targets of different distances, directions, and types to different tones, rhythms, or tactile intensities according to preset mapping rules.

2. The multi-sensory information conversion full-perception system according to claim 1, characterized in that, The ultrasonic sensors are multiple and distributed in different positions on the wearable device for 360-degree omnidirectional distance measurement; the audio feedback device outputs different tones of prompts based on the distance, height and direction of the obstacle detected by the ultrasonic sensors, realizing the function of sound location identification.

3. The multi-sensory information conversion full-perception system according to claim 2, characterized in that, The mapping rules include: Distance mapping: The closer the obstacle, the higher the pitch frequency or the faster the tactile vibration frequency; Orientation mapping: Indicates the location of obstacles by using the difference between the left and right audio channels or tactile points on different parts of the body; Height mapping: Obstacles of different heights correspond to different timbre or tactile modes; Type mapping: Based on the target recognition results, different signal features are used for different targets such as people, vehicles, and buildings.

4. The multi-sensory information conversion full-perception system according to claim 1, characterized in that, The tactile feedback device is a wearable tactile garment, comprising: Multiple hydraulic units are distributed in different parts of the garment, and each hydraulic unit includes a miniature hydraulic pump and a pressure sensor; The control circuit is used to receive wireless signals and control the working status of each hydraulic unit. A wireless communication module is used for data transmission with external devices. The hydraulic unit transmits information such as location, distance, and urgency to the user by applying pressure stimuli of different intensities and durations.

5. The multi-sensory information conversion full-perception system according to claim 4, characterized in that, This system is used in nighttime combat scenarios and specifically includes: Unmanned aerial vehicles (UAVs) or reconnaissance equipment serve as information gathering modules, using infrared sensors to detect the location of enemy targets. The information processing module converts the target's orientation and distance information into tactile control signals; Control signals are transmitted wirelessly to the tactile clothing worn by soldiers. The haptic garment uses pressure changes in hydraulic units at different locations to alert soldiers to the location and distance of enemy targets, achieving covert information transmission with zero light and sound.

6. The multi-sensory information conversion full-perception system according to claim 5, characterized in that, It also includes an environmental perception method based on multi-sensory information conversion, with the following steps: S1: Collect environmental information through a sensor array, including at least one of distance information, target information, and temperature information; S2: Perform AI analysis and processing on the collected information to identify target types, calculate target distance and orientation; S3: Based on the preset sensory mapping model, the processed information is converted into non-visual sensory signals, including audio signals or tactile signals; S4: Transmits the converted signal to the user through an audio output device or a haptic output device; The sensory mapping model is adaptively adjusted according to the user's usage scenario and needs.

7. The multi-sensory information conversion full-perception system according to claim 6, characterized in that, The sensory mapping model is trained using machine learning methods, and the training process includes: Collect sensor data and corresponding optimal feedback signal combinations under different environmental scenarios; Using reinforcement learning algorithms, the mapping parameters are optimized with the user's task completion efficiency and comfort as the reward function. Personalized model fine-tuning is performed for different users.

8. The multi-sensory information conversion full-perception system according to claim 6, characterized in that, The method supports simultaneous output of multimodal signals and, where the environment permits, can use both audio and haptic feedback to improve the accuracy and redundancy of information transmission.

9. The multi-sensory information conversion full-perception system according to claim 1, characterized in that, The system can also be used in the following scenarios: Daily navigation assistance for blind or visually impaired people; Operating instructions in high-noise environments; Haptic enhancement in virtual reality and augmented reality systems; Sensory substitution therapy in rehabilitation training.

10. A tactical application method for a multi-sensory information conversion full-perception system, using a multi-sensory information conversion full-perception system as described in any one of claims 1-9, characterized in that, In tactical application scenarios, the following steps are included: S1. Transmit tactile control commands via an encrypted wireless channel; S2. Set the priority of tactile signals according to tactical needs, and use high-intensity, high-frequency stimulation for emergency information; S3 supports multi-person collaboration, allowing multiple soldiers' haptic suits to receive information from the same command system.

Images