Perception guide interaction control system and method for visually impaired people

Abstract

The invention belongs to the field of visual impairment assistance, and particularly relates to a visual impairment person perception guide interaction control system and method, and the system comprises a multi-source sensor module, an information processing and analysis unit, an interaction feedback module, an intelligent Internet of Things module, and a user preference and scene self-adaption module. According to the visual impaired person perception guide interaction control system and method, through a multi-source sensor fusion technology, the limitation of a single sensor is effectively overcome. The high-definition imaging and low-light enhancement capability of the visual sensor, the close-range precise detection of the ultrasonic sensor, the stable motion monitoring of the inertial measurement unit and the high-precision positioning of the global navigation satellite system cooperate with one another, so that the comprehensive and precise sensing of the environment is realized, and the system is convenient to adapt to different environments. The accuracy of obstacle detection, target recognition and positioning is greatly improved, a solid and reliable environment cognition basis is provided for visually impaired people, and the visually impaired people can more accurately know the surrounding environment condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual impairment assistance, in particular to a visual impairment perception guidance interaction control system and method. BACKGROUND

[0002] The visual impairment group faces many severe challenges in daily life and travel. Due to the lack of visual function, they cannot accurately obtain information about the surrounding environment, which limits their actions and seriously affects their quality of life and social participation.

[0003] At present, the traditional scheme in the prior art relies on a single type of sensor (such as using only ultrasonic waves or a camera), which leads to a significant decrease in perception ability in complex scenes. For example, a pure ultrasonic wave scheme cannot identify visual information such as traffic signs and signal lights, and it is difficult to provide complete environmental cognition in outdoor street scenes; and a single camera scheme has a large decrease in detection accuracy in low light, shielding and other environments, such as high obstacle omission rate in night or indoor shadow area, and cannot build a reliable environment model for the visually impaired. In view of this, we propose a visual impairment perception guidance interaction control system and method. SUMMARY

[0004] The main purpose of the present application is to provide a visual impairment perception guidance interaction control system and method, which can solve the problems proposed in the background art.

[0005] To achieve the above purpose, the visual impairment perception guidance interaction control system proposed by the present application comprises:

[0006] Multi-source sensor module: containing ultrasonic sensor, high-definition camera, Beidou / GPS positioning module and inertial measurement unit (IMU), used for collecting obstacle distance, image feature, geographic position and motion state data;

[0007] Information processing and analysis unit: connected with the multi-source sensor module, constructs an environment model through a data fusion algorithm, integrates an edge computing module to realize local data preprocessing, and cooperates with a cloud background through a 4G / 5G network, edge computing module: using an ARM processor to complete 80% data preprocessing (such as noise filtering, image feature extraction) locally, with a response delay ≤1 second, only key events (such as SOS help alarm) are uploaded to the cloud, data fusion algorithm: based on Kalman filtering, dynamically allocates weights (vision 40%~60%, ultrasonic wave 20%~30%) according to sensor time stamps (accuracy ≤1 ms), constructs a dynamic environment model, with an update frequency ≥10 Hz, end-cloud cooperation: synchronizing data (such as longitude and latitude coordinates, device ID) with the cloud background through a 4G / 5G network, the cloud background supports historical trajectory storage (≥30 days) and abnormal behavior analysis (such as triggering a warning if stationary for more than 45 minutes);

[0008] Interactive feedback module: including bone conduction earphone and wearable tactile device, for outputting voice navigation instructions and different mode of vibration feedback, voice interaction: bone conduction earphone outputs natural voice (MOS score ≥4.0), supports voice instruction recognition such as "find the surrounding toilet", and the volume is automatically adjusted in the range of 0-40dB;

[0009] Intelligent Internet of Things module: based on Bluetooth / BLE protocol connection to sensing devices such as guide stick, supporting synchronization of sensor data and device status to cloud background, short-distance communication: connecting sensing devices such as guide stick through Bluetooth / BLE protocol (2.4GHz frequency band), real-time transmission of ultrasonic data, button state (such as SOS alarm) and power information, packet loss rate ≤5%, long-distance communication: using Internet of Things protocol to synchronize sensor data to cloud background, supporting Web platform remote monitoring of device status and issuing control instructions;

[0010] User preference and scene adaptive module: dynamically adjusting voice volume and tactile intensity according to environmental light, traffic flow, and automatically optimizing interaction strategy, personalized learning: recording user habits (such as voice speed, vibration intensity), supporting manual setting or automatic optimization, scene strategy: adjusting the voice volume of the device according to the volume of the external environment, and enhancing the feedback intensity; indoor shopping mall scene preferentially uses Bluetooth beacon positioning to avoid escalators.

[0011] Preferably, the high-definition camera supports automatic enhancement of imaging in low-light environments, and the ultrasonic sensor detection accuracy reaches meter level, high-definition camera: integrating low-light enhancement algorithm (such as infrared light supplement), still able to recognize obstacles, traffic signs and other targets under 0.05 lux illumination, accuracy ≥90%, ultrasonic sensor: using high-precision ranging module (detection accuracy ≤2cm), real-time scanning of near-range obstacles (0-5 meters), especially suitable for indoor narrow passage detection, inertial measurement unit (IMU): monitoring motion parameters such as walking speed and turning angle at a frequency of 100Hz, assisting trajectory calculation.

[0012] Preferably, the wearable tactile device provides at least three feedback modes, corresponding to direction guidance, danger warning and special area prompt respectively, tactile feedback: wearable device (such as bracelet) provides three vibration modes: periodic vibration (intensity 3 levels): direction guidance (such as "turn left in front"); pulse vibration (intensity 7 levels): emergency danger (such as obstacle 2 meters in front); continuous vibration (intensity 5 levels): special area prompt (such as elevator entrance).

[0013] Preferably, the intelligent Internet of Things module realizes data encryption transmission through the Internet of Things protocol, and supports simultaneous processing of four types of sensor data: ultrasonic, image, positioning and inertial measurement unit.

[0014] Preferably, the Beidou / GPS positioning module has a positioning accuracy of meter level in outdoor scenes, and the Beidou / GPS positioning module has an outdoor positioning accuracy of less than or equal to 3 meters and supports an electronic fence function; indoor positioning is achieved through a Bluetooth beacon, with an accuracy of less than or equal to 2 meters, and precise navigation is achieved in combination with a shopping mall electronic map.

[0015] The control method of the visual impairment person perception guiding interaction system provided by the application comprises the following steps:

[0016] S1, a data acquisition stage: a multi-source sensor acquires environmental data in real time, and transmits the data to a smart terminal through a Bluetooth / BLE protocol; an ultrasonic sensor scans obstacles at a frequency of 50 Hz; a high-definition camera acquires 30 frames of images per second; an inertial measurement unit outputs motion data in real time, and transmits the data to the smart terminal through Bluetooth / BLE; an indoor scene is automatically switched to Bluetooth beacon positioning, and electronic map data such as shopping mall store distribution and passage width is acquired;

[0017] S2, a data processing and analysis stage: an edge computing module pre-processes data, and completes dynamic environment modeling and target trajectory prediction in combination with a cloud background; the edge computing module pre-processes data (such as removing camera distortion, with a distortion rate of less than or equal to 0.5 %); multi-source information is fused through Kalman filtering, and a dynamic target trajectory is predicted (with an error of less than or equal to 0.5 meters); the cloud background performs secondary analysis on abnormal data (such as no motion for 5 seconds in succession), and generates an optimized suggestion which is fed back to the terminal;

[0018] S3, an interaction feedback stage: voice and vibration feedback are generated according to the processing result; a high-intensity warning is triggered in priority according to the danger level; a hierarchical feedback is triggered according to the danger level: first-level warning (such as an obstacle in front of less than 2 meters): pulse vibration + high-frequency voice "emergency avoidance", with a response time of less than or equal to 1 second; second-level guidance (such as a turning at an intersection): periodic vibration + voice "turn left in 50 meters";

[0019] S4, a system self-optimization stage: interaction parameters are adjusted through user feedback, with an optimization period of less than or equal to 24 hours; user voice feedback (such as "vibration is too weak") is received, and the haptic intensity parameter is adjusted through a reinforcement learning algorithm, with an optimization period of less than or equal to 24 hours; the cloud background automatically optimizes the navigation strategy of a frequently traveled route (such as avoiding a congested road section during peak hours) according to the user historical trajectory;

[0020] S5, a system self-checking stage: sensor accuracy (such as an ultrasonic error of less than or equal to 1 %), network connectivity (a time delay of less than or equal to 100 ms), and battery power (a remaining power of less than 15 % for early warning) are detected; sensor calibration: an ultrasonic sensor is regularly calibrated (with an error of less than or equal to 1 %), a camera automatically focuses, communication detection: a Bluetooth / 4G network time delay is less than or equal to 100 ms, and a voice early warning "charging is recommended" is given when the power is less than 15 %.

[0021] Preferably, in step S2, the data fusion algorithm adopts Kalman filtering, dynamically allocates weights combined with sensor timestamps (accuracy ≤1ms), wherein the weight of visual data is 40%-60%, and the weight of ultrasonic data is 20%-30%.

[0022] Preferably, in step S3, the naturalness MOS score of the voice navigation instruction is ≥4.0, and the response time of the danger warning is ≤1s.

[0023] Preferably, in step S3, the user feedback includes voice instructions "volume up" or "vibration enhancement", and the system automatically adjusts the parameters through the reinforcement learning algorithm.

[0024] Preferably, in step S2, the cloud background supports historical trajectory query, the storage period is ≥30 days, and the artificial customer service intervention mechanism is triggered for abnormal behavior (such as static for more than 45 minutes), and the response time is ≤5 minutes.

[0025] The application provides a visual impaired person perception guiding interactive control system and method.

[0026] (1) The visual impaired person perception guiding interactive control system and method effectively overcome the limitations of single sensor through multi-source sensor fusion technology. The high-definition imaging and low-light enhancement capability of the visual sensor, the near-distance accurate detection of the ultrasonic sensor, the stable motion monitoring of the inertial measurement unit, and the high-precision positioning of the global navigation satellite system cooperate with each other to realize the all-around and accurate perception of the environment. This greatly improves the accuracy of obstacle detection, target recognition and positioning, provides a solid and reliable environmental cognition basis for the visually impaired, and enables them to more accurately understand the surrounding environment.

[0027] (2) The visual impaired person perception guiding interactive control system and method has a voice and tactile multi-modal interactive feedback mode, which is highly consistent with human natural perception habits. The voice generated by the voice synthesis technology is clear and natural, the use of bone conduction earphones does not block the ear canal, and the user can simultaneously perceive the surrounding environmental sound. The various vibration modes of the vibration vest and bracelet can intuitively convey rich information, and feedback according to the priority of the danger level, so that the user can quickly and intuitively understand, and the interaction efficiency is significantly improved. This effectively reduces the confusion and incorrect operation of the visually impaired in travel and daily activities, and enhances the autonomy and safety of their actions.

[0028] (3) The visually impaired person’s perception guidance interactive control system and method automatically optimizes the interaction strategy by learning user preferences (such as speech speed and tactile intensity) and provides customized services. For example, in indoor scenarios such as shopping malls, Bluetooth beacon positioning is preferred to enhance voice and tactile feedback in noisy environments. At the same time, the cloud backend supports historical trajectory query and abnormal behavior warning, combined with emergency assistance, system self-checking and other functions, to build a full-process protection system covering perception, interaction and safety, and enhance the safety of visually impaired people’s independent travel and their confidence in social integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0030] Figure 1 This is a system block diagram of the present invention;

[0031] Figure 2 Schematic diagram of the steps of the method of the present invention;

[0032] Figure 3 Schematic diagram of some steps of the present invention Figure 1 ;

[0033] Figure 4 Schematic diagram of some steps of the present invention Figure 2 ;

[0034] Figure 5 Schematic diagram of some steps of the present invention Figure 3 ;

[0035] Figure 6 The processing flow of the present invention is shown as follows Figure 1 ;

[0036] Figure 7 The processing flow diagram of the present invention is shown as follows: Figure 2 .

[0037] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] Referring to Figures 1-7 The present application provides a visual impaired person perception guiding interaction control system, comprising: a multi-source sensor module, an information processing and analysis unit, an interaction feedback module, an intelligent internet module, a user preference and scene adaptive module, the system hardware architecture is described as follows: the system adopts modular design, and each component is connected through a standardized interface. In the multi-source sensor module, the ultrasonic sensor and the high-definition camera are integrated on the walking stick, and the inertial measurement unit and the Beidou / GPS module are built-in in the intelligent terminal (such as a waist-mounted controller); the bone conduction earphone and the vibrating bracelet of the interaction feedback module are connected with the intelligent terminal through Bluetooth; the intelligent internet module realizes data intercommunication with the cloud background through the 4G / 5G module; and the device configuration and initialization: before the visual impaired person goes out, the wearable device (such as intelligent guide glasses + vibrating bracelet) integrated with the multi-source sensor module and the interaction feedback module is worn, and the bone conduction earphone is turned on. After the device is started: sensor calibration: the imaging parameters of the visual sensor, the distance measuring offset of the ultrasonic sensor and the zero offset of the inertial measurement unit are calibrated through the built-in program; user preference loading: the user's historical settings (such as voice speed of 1.2 times and tactile intensity of 5 levels) are obtained from the cloud; system self-checking: the Bluetooth connection stability (packet loss rate <2%) and the 4G network signal strength (≥-90dBm) are detected to ensure the normal operation of the device.

[0040] Multi-source sensor module: containing ultrasonic sensor, high-definition camera, Beidou / GPS positioning module and inertial measurement unit (IMU), used for collecting obstacle distance, image feature, geographic position and motion state data; information processing and analysis unit: connected with the multi-source sensor module, constructing an environment model through a data fusion algorithm, integrating an edge computing module to realize local data preprocessing, and cooperating with the cloud background through 4G / 5G network for processing, edge computing module: adopting an ARM processor to complete 80% of local data preprocessing (such as noise filtering and image feature extraction) with a response delay ≤1 second, and only uploading the cloud for key events (such as SOS help alarm), data fusion algorithm: based on Kalman filtering, dynamically allocating weights (visual 40%-60%, ultrasonic 20%-30%) according to sensor time stamps (accuracy ≤1ms) to construct a dynamic environment model with an update frequency ≥10Hz, end-cloud cooperation: synchronizing data (such as longitude and latitude coordinates, device ID) with the cloud background through 4G / 5G network, and the cloud background supports historical trajectory storage (≥30 days) and abnormal behavior analysis (such as triggering a warning when being static for more than 45 minutes);

[0041] Interaction feedback module: including a bone conduction earphone and a wearable tactile device, used for outputting voice navigation instructions and vibration feedback in different modes, voice interaction: the bone conduction earphone outputs natural voice (MOS score ≥4.0), supports voice instruction recognition such as "finding a nearby toilet", and the volume is automatically adjusted in the range of 0-40dB;

[0042] Smart Internet of Things module: connect sensing devices such as guide canes based on Bluetooth / BLE protocol, support synchronization of sensor data and device status to cloud background, short-distance communication: connect sensing devices such as guide canes through Bluetooth / BLE protocol (2.4 GHz frequency band), real-time transmission of ultrasonic data, button state (such as SOS alarm) and power information, packet loss rate ≤5%, long-distance communication: use Internet of Things protocol to synchronize sensor data to the cloud background, support Web platform remote monitoring of device status, issue control instructions;

[0043] User preference and scene adaptation module: dynamically adjust voice volume and tactile intensity to automatically optimize interaction strategy according to environmental light, traffic flow, etc., personalized learning: record user habits (such as voice speed, vibration intensity), support manual setting or automatic optimization, scene strategy: adjust the voice volume of the device according to the volume of the external environment, enhance the feedback intensity; prefer to use Bluetooth beacon positioning to avoid escalators in indoor shopping mall scenarios.

[0044] Further, the high-definition camera supports automatic image enhancement in low-light environments, and the ultrasonic sensor detection accuracy reaches meter level, high-definition camera: integrates low-light enhancement algorithm (such as infrared fill light), can still recognize obstacles, traffic signs and other targets under 0.05 lux illumination, accuracy ≥90%, ultrasonic sensor: uses high-precision ranging module to scan near-range obstacles (0-5 meters) in real time, especially suitable for indoor narrow passage detection, inertial measurement unit (IMU): monitors motion parameters such as walking speed and turning angle at a frequency of 100Hz to assist in trajectory calculation.

[0045] Further, the wearable tactile device provides at least 3 vibration modes, corresponding to direction guidance, danger warning and special area prompt respectively, tactile feedback: wearable devices (such as wristbands) provide 3 vibration modes: periodic vibration (intensity 3 levels): direction guidance (such as "turn left ahead"); pulse vibration (intensity 7 levels): emergency danger (such as 2-meter obstacle ahead); continuous vibration (intensity 5 levels): special area prompt (such as elevator entrance).

[0046] Further, the smart Internet of Things module realizes data transmission through the Internet of Things protocol, supporting simultaneous processing of 4 types of sensor data: ultrasonic, image, positioning and inertial measurement unit.

[0047] Further, the Beidou / GPS positioning module has a positioning accuracy of meter level in outdoor scenarios, Beidou / GPS positioning module: outdoor positioning accuracy ≤3 meters, supports electronic fence function; indoor positioning through Bluetooth beacon (Beacon), accuracy ≤2 meters, combined with shopping mall electronic map to realize accurate navigation.

[0048] The application provides a control method of a visual impairment person perception guiding interaction system, and comprises the following steps.

[0049] S1, a data acquisition stage: multi-source sensors collect environment data in real time, and transmit the environment data to a smart terminal through a Bluetooth / BLE protocol; an ultrasonic sensor scans obstacles at a frequency of 50 Hz; a high-definition camera collects 30 frames of images per second; an inertial measurement unit outputs motion data in real time, and transmits the motion data to the smart terminal through the Bluetooth / BLE; and an indoor scene is automatically switched to a Bluetooth beacon positioning mode to obtain electronic map data such as a distribution of stores in a shopping mall and a width of a passageway.

[0050] S2, a data processing and analysis stage: an edge computing module pre-processes data, and a cloud background completes dynamic environment modeling and target trajectory prediction in combination with the pre-processing; the edge computing module pre-processes data (for example, removes camera distortion, and a distortion rate is less than or equal to 0.5 %); the edge computing module fuses multi-source information through Kalman filtering, and predicts a dynamic target trajectory (an error is less than or equal to 0.5 m); the cloud background performs secondary analysis on abnormal data (for example, no motion for 5 seconds in succession), and generates an optimized suggestion to feed back to the terminal.

[0051] S3, an interaction feedback stage: voice and vibration feedback are generated according to a processing result; a high-intensity warning is triggered preferentially according to a danger level; and a hierarchical feedback is triggered according to the danger level: first-level warning (for example, an obstacle in front of a user is less than 2 m): pulse vibration + high-frequency voice "urgent avoidance", and a response time is less than or equal to 1 s; second-level guidance (for example, a turning at an intersection): periodic vibration + voice: "turn left at a distance of 50 m in front of the user".

[0052] S4, a system self-optimization stage: interaction parameters are adjusted through user feedback, an optimization period is less than or equal to 24 hours, user voice feedback (for example, "vibration is too weak") is received, a haptic intensity parameter is adjusted through a reinforcement learning algorithm, an optimization period is less than or equal to 24 hours, and a navigation strategy of a frequently traveled route is automatically optimized (for example, a congested road section is avoided during a peak period) according to a user historical trajectory.

[0053] S5, a system self-checking stage: sensor accuracy (for example, an ultrasonic error is less than or equal to 1 %), network connectivity (a time delay is less than or equal to 100 ms) and a battery capacity (a remaining capacity is less than 15 % and a warning is given) are detected, sensor calibration is performed (an error is less than or equal to 1 %) and an ultrasonic sensor is regularly calibrated, a camera is automatically focused, a communication detection is performed (a Bluetooth / 4G network time delay is less than or equal to 100 ms), and a voice warning "charging is suggested" is given when the battery capacity is less than 15 %.

[0054] Further, in step S2, a Kalman filtering algorithm is used in a data fusion algorithm, and a weight is dynamically assigned in combination with a sensor time stamp (an accuracy is less than or equal to 1 ms), wherein a vision data weight is 40 % to 60 %, and an ultrasonic data weight is 20 % to 30 %.

[0055] Further, in step S3, the voice navigation instruction naturalness MOS score is ≥4.0, and the danger warning response time is ≤1s.

[0056] Further, in step S3, the user feedback includes voice instructions "volume up" or "vibration enhancement", and the system automatically adjusts the parameters through reinforcement learning algorithm.

[0057] Further, in step S2, the cloud background supports historical trajectory query, the storage period is ≥30 days, and the artificial customer service intervention mechanism is triggered for abnormal behavior (such as static for more than 45 minutes), and the response time is ≤5 minutes.

[0058] Embodiment 1: outdoor travel scenario

[0059] Device configuration and initialization: before going out, the visually impaired person wears the wearable device integrated with multi-source sensor module, interactive feedback module, etc. in good order, and wears the bone conduction earphone. After the device is turned on, a series of initialization processes are automatically triggered. First, sensor calibration is performed. Through the built-in calibration program, the imaging parameters of the visual sensor, the detection range of the ultrasonic sensor, and the zero offset of the inertial measurement unit are accurately calibrated to ensure the accuracy of the sensor data. Subsequently, the system is connected to the pre-bound user account, and the personalized preference settings accumulated by the user in the past use process are loaded from the cloud server, including the speech speed and volume of voice broadcast, the intensity and mode of tactile feedback, etc. Next, the system enters the self-checking stage, and the functions of each component part such as multi-source sensor module, information processing and analysis unit, interactive feedback module, etc. are detected. For example, a specific detection pattern is sent to the visual sensor to check whether its image acquisition and recognition function is normal; the ultrasonic sensor is driven to conduct a short-range distance measurement test to verify its distance measurement accuracy; the motion state change of the inertial measurement unit is simulated to test the stability of data output. If the system detects that a sensor data is abnormal during the self-checking process, such as image blur, excessive noise, etc. of the visual sensor, the user will be prompted immediately through voice, and the system will try to automatically recalibrate the sensor. If the automatic calibration fails, the system will further prompt the user about the possible problems and guide the user to perform simple troubleshooting operations, such as checking whether the sensor is blocked, whether the device connection is stable, etc. If the problem still cannot be solved, the system will automatically switch to the backup sensor (if equipped) to ensure that data acquisition work can continue, and to ensure that the device is in the best operating state before going out, providing reliable protection for the outdoor travel of the visually impaired person.

[0060] Data acquisition and processing: During outdoor walking, the multi-source sensor module is in full operation, collecting a variety of data in real time. The vision sensor continuously collects images and depth data at a high frequency of 30 frames per second. With its high-definition imaging capability and low-light automatic enhanced imaging function, it can clearly capture various objects in the surrounding environment, including pedestrians, vehicles, traffic lights, road signs, etc., and accurately obtain their distance, direction, and other spatial information, whether in the bright daylight or dim evening. The ultrasonic sensor rapidly detects the distance of the near-distance obstacle every 10 milliseconds, playing a key supplementary detection role in the near-distance scene where the vision sensor may be blocked or the detection accuracy is insufficient, ensuring that the visually impaired person will not collide with the near-distance obstacle. The inertial measurement unit outputs motion parameters uninterruptedly at a frequency of 100 Hz, monitoring the walking speed, direction change, and step count of the visually impaired person in real time, providing important motion state data support for subsequent positioning and navigation calculation. The global navigation satellite system updates the position information once every second according to the preset positioning accuracy requirement, building an accurate global position framework for the system in open areas. The edge computing module of the information processing and analysis unit performs preliminary noise reduction and feature extraction on the raw data locally, removing noise interference mixed in the sensor acquisition process and highlighting key target features. Then, using the Kalman filtering algorithm, the data of vision, ultrasonic, inertial measurement unit, and global navigation satellite system are dynamically weighted and fused according to their time stamps. Through advanced target recognition models, key objects in the environment are accurately classified and recognized, such as distinguishing different types of traffic lights (red, green, yellow), and identifying the driving direction and speed of the vehicle in front. At the same time, using high-precision positioning algorithms, considering the absolute position information of the global navigation satellite system, the relative motion information of the inertial measurement unit, and the environmental feature information provided by other sensors, the user's current position and walking direction are accurately calculated. For example, at a complex intersection, vehicles and pedestrians are passing by, the vision sensor captures image information of multiple traffic lights and a large number of pedestrians, the ultrasonic sensor detects the possible near-distance obstacles, the inertial measurement unit feedbacks the user's walking direction and speed change in real time, and the global navigation satellite system provides the approximate position coordinates. After receiving these multi-source data, the information processing and analysis unit accurately judges the specific position of the user, such as the southeast corner of the intersection, and the detailed surrounding environment, providing accurate and comprehensive basis for subsequent interactive feedback generation.

[0061] Interaction feedback and user experience: When the system detects that the intersection ahead has a green light and the user needs to go straight through according to the user's travel route planning through the processing and analysis of the collected data, the voice synthesis module quickly acts to clearly broadcast "the intersection ahead has a green light, please go straight through the zebra crossing" to the user through the bone conduction earphone. At the same time, the vibration bracelet in the interaction feedback module produces a continuous and smooth periodic vibration at the user's wrist, giving the user a clear straight-line guide from the tactile level, so that the user can accurately understand the travel instructions conveyed by the system through auditory and tactile perception. If the system detects that there are pedestrians suddenly crossing the user's walking path ahead, it will immediately start the emergency warning mechanism, issue an urgent and strong voice warning "there are pedestrians ahead, please be careful to avoid" through the voice synthesis module, and at the same time the vibration bracelet switches to a strong pulse vibration mode, quickly conveying dangerous information to the user from multiple sensory dimensions, reminding the user to take timely avoidance action. During the walking process of visually impaired people, they can at any time feedback their feelings and needs to the system through voice input devices such as high-sensitivity microphones built into the device. For example, if the user feels that the current environment is relatively noisy and the voice prompt is not very clear, they can directly say "make the sound louder"; if they feel that the tactile feedback is not obvious enough, they can feedback "make the vibration stronger" and other instructions. The voice recognition module equipped by the system quickly recognizes the user's voice instructions and transmits the related feedback information to the user preference and scene adaptation module in real time. This module uses reinforcement learning algorithm to deeply analyze and learn the user feedback data, and optimizes the subsequent information processing and feedback strategy according to the user's feedback in different scenes. For example, if the user repeatedly feedbacks that the voice prompt is not clear in noisy street environment, the system will automatically increase the voice broadcast volume in similar noisy environment, and adjust the timbre and tone of voice synthesis, so that it is clearer and more prominent, easy for the user to identify and understand in complex environment.

[0062] Scene adaptation and optimization: When a visually impaired user enters a high-rise building from an open street, the system can intelligently perceive this change and automatically adjust the data processing strategy due to the blocking of Global Navigation Satellite System signals by high-rise buildings, resulting in weakened or interrupted signal strength. Specifically, the system automatically enhances the weight of visual sensor and inertial measurement unit data in positioning calculation, uses techniques such as MEMSIMU-assisted search COA attitude solution method, calculates the pitch angle and yaw angle through MEMSIMU output data, determines the search boundary of the optimization algorithm, introduces GNSS carrier phase observation model and attitude observation model to define the fitness function to be optimized, and uses adaptive dynamic parameter adjustment mechanism and step adaptive adjustment mechanism to search and optimize the attitude within the spatial range determined by the inertial navigation system, and then uses the improved COA to search and optimize the attitude, so as to more accurately calculate and estimate the user's position. At the same time, combined with the more frequent capture of feature points in the surrounding environment by the visual sensor and the continuous monitoring of the user's motion state by the inertial measurement unit, the system can still provide high-precision positioning services for users in the case of poor Global Navigation Satellite System signals. At the same time, the system intelligently adjusts the interaction feedback strategy according to the characteristics of the scene, such as the increase in the number of pedestrians and vehicles and the complexity of the environment. For example, appropriately increase the frequency of voice prompts, and give the user advance warning of possible complex road conditions ahead; increase the volume of voice prompts so that they are more easily heard by the user in a noisy environment; and enhance the strength of tactile feedback so that the user can more obviously feel the information conveyed by the system in a tactile manner, to better cope with complex and changing environments. Throughout the outdoor journey, the system acts as an intelligent companion, constantly learning the user's behavior patterns and feedback information in different scenarios, continuously optimizing information processing and interaction strategies, and improving user experience. After a period of use, many visually impaired users feedback that they can better perceive the environment when using the system outdoors, and can walk more confidently and safely in various outdoor scenarios, greatly improving their travel autonomy and convenience. For example, after passing through the same complex road section several times, the system can predict the situation the user may face in that section in advance based on the user's previous feedback and behavior habits, and adjust the interaction strategy in advance, such as increasing the volume of voice prompts and the strength of tactile feedback when approaching the intersection, to remind the user to pay attention to traffic conditions, and to make the user more comfortable in dealing with various complex situations during the journey.

[0063] Example 2: Indoor shopping mall scenario

[0064] Environment adaptation and data preparation: Before entering the mall, the system first interacts efficiently with the internal positioning system of the mall (such as a Bluetooth beacon or Wi-Fi-based indoor positioning system) to quickly obtain detailed maps and related information of the mall, including the precise distribution of each store, the direction and width of the passageway, the specific location of service facilities (such as restrooms, rest areas, customer service centers), etc. At the same time, the system uses visual sensors to conduct a comprehensive scan of the entrance and surrounding environment of the mall, establishing an initial local environment model and identifying key elements such as landmark objects, steps, and automatic doors at the entrance. During this process, the system simultaneously performs self-checking on each module to ensure the accuracy of data collection and processing. For example, it checks whether the connection with the internal positioning system of the mall is stable and whether data transmission is smooth; it tests whether the visual sensor can clearly collect images and accurately identify key objects under the lighting conditions at the entrance of the mall. If the internal positioning system of the mall is unstable, with signal loss or large positioning deviation, the system will quickly activate the backup positioning scheme, combining data from the visual sensor and the inertial measurement unit, using methods such as indoor visual-inertial navigation algorithms and similar adaptive combination navigation methods, such as Hua Hang's adaptive combination navigation, to automatically adjust the measurement noise covariance matrix based on real-time monitoring of the user's motion state, continuously track and match the environmental features collected by the visual sensor, and accurately record the user's motion trajectory using the inertial measurement unit to assist in precise positioning, ensuring that the user's navigation needs within the mall can be stably met.

[0065] Navigation and interaction process: When walking in the mall, multiple sensors work closely together to continuously collect data. Since the global navigation satellite system signal is basically unable to cover the indoor environment, positioning and environment perception mainly rely on visual sensors, ultrasonic sensors and inertial measurement units at this time. According to the destination (such as a brand store) set by the user before entering the mall, the system uses an optimized path planning algorithm, combined with the mall map information and the real-time collected environmental data, to plan the optimal walking path, and guides the user forward in real time through voice and tactile feedback. For example, when the user approaches the area where the target store is located, the soft voice prompt "You are approaching the target store, in front of you on the left" is heard, and at the same time the vibration bracelet produces a slight periodic vibration at the left wrist, accurately indicating the direction from two dimensions of hearing and touch, helping the user to accurately find the store location. When passing through special areas such as elevators and escalators, the system issues detailed and clear voice and continuous vibration warnings in advance, informing the user of relevant matters needing attention, such as "The elevator entrance will be reached soon, please be careful on the ground and pay attention to the direction of the elevator operation", and at the same time, through the change of vibration mode, the user is reminded to pay attention to the existence of special areas. When the system simultaneously detects that the front elevator port is crowded and there are obstacles on the ground, according to the pre-set danger level priority judgment mechanism, it first reminds the user to pay attention to the ground obstacles through strong pulse vibration and emergency voice, to ensure the safety of the user's walking. In addition, the system also has intelligent environment perception and adaptive adjustment capability, which will dynamically adjust the speed and volume of voice prompts, as well as the frequency and intensity of tactile feedback, according to real-time factors such as the number of people and environmental noise in different areas of the mall. For example, in the mall atrium area with large number of people and noisy environment, the volume and speed of voice prompts are appropriately increased, and the frequency of tactile feedback is accelerated, so that the user can quickly obtain key information in a complex environment; while in the relatively quiet and less crowded internal passageway of the store, the voice prompt volume is reduced, the speed is slowed down, and the tactile feedback intensity is reduced, to avoid unnecessary interference to the user, and to improve the user's navigation experience in the mall.

[0066] User feedback and system optimization: During the use of the system in the mall by visually impaired people, they can conveniently feedback the problems encountered and the use experience to the system through voice input at any time. For example, if the navigation prompt for a certain area is not satisfactory, it is not clear or accurate enough, you can directly say "the last turn prompt is not clear"; or feel that some voice instructions are not clear, feedback "I didn't understand the voice instruction" and so on. The efficient voice recognition and feedback receiving module equipped by the system can quickly capture the feedback information of the user and accurately record it, and transmit it to the optimization module of the system. After receiving the feedback, the system immediately analyzes the relevant information in depth, combines the real-time environmental data in the mall and the historical use records of the user, uses big data analysis and machine learning algorithms to optimize the subsequent operation strategy. For example, if many users feedback that the navigation path near a certain store is unreasonable, the system will comprehensively analyze the environmental data of this area, including store layout, passage traffic conditions, crowd density distribution, etc., combined with the walking habit data of the user in this area, to optimize the navigation algorithm, and provide more reasonable and efficient path planning for subsequent users. At the same time, the system also regularly conducts systematic statistical analysis on the use data of the user in the mall, summarizes the user demand characteristics of different areas and different time periods, such as during the weekend shopping peak period, the user requires higher timeliness and accuracy of voice prompts; during the non-busy period of weekdays, the user pays more attention to the comfort of tactile feedback. According to these analysis results, further optimize the interaction strategy and information presentation method, continuously improve the applicability and user satisfaction of the system in the indoor mall scene, so that the visually impaired people can travel more smoothly and conveniently in the mall, and enjoy the same shopping experience as ordinary people.

[0067] It should be noted that the above-mentioned electrical components are all products of the prior art, which are selected, installed and debugged by those skilled in the art according to the needs of use to ensure that each electrical appliance can work normally, and the components are all general standard components or components known to those skilled in the art, the structure and principle of which can be known by technical personnel through technical manual or through conventional experimental method, which is not limited here.

[0068] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made according to the inventive concept of the present application, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A visually impaired person's perception guidance interactive control system, characterized by: include: Multi-source sensor module: includes ultrasonic sensors, high-definition cameras, Beidou / GPS positioning modules, and inertial measurement units (IMUs), used to collect obstacle distance, image features, geographic location, and motion status data; Information processing and analysis unit: Connects to multi-source sensor modules, builds environmental models through data fusion algorithms, integrates edge computing modules to implement local data preprocessing, and collaborates with the cloud backend through 4G / 5G networks; Interactive feedback module: including bone conduction headphones and wearable tactile devices, used to output voice navigation instructions and different modes of vibration feedback; Smart IoT module: connects to sensor devices such as guide sticks based on Bluetooth / BLE protocols, and supports synchronization of sensor data and device status to the cloud background; User preference and scene adaptation module: Dynamically adjusts voice volume and tactile intensity based on ambient light, pedestrian flow, etc., and automatically optimizes interaction strategies.

2. The visually impaired person perception guidance interactive control system according to claim 1, characterized in that: The high-definition camera supports automatic enhanced imaging in low-light environments, and the ultrasonic sensor's detection accuracy reaches the meter level.

3. The visually impaired person perception guidance interactive control system according to claim 1, characterized in that: The wearable tactile device provides at least three feedback modes, corresponding to direction guidance, danger warning and special area prompts respectively.

4. The visually impaired person perception guidance interactive control system according to claim 1, characterized in that: The intelligent IoT module realizes data transmission through the IoT protocol and supports simultaneous processing of four types of sensor data: ultrasonic, image, positioning and inertial measurement unit.

5. The visually impaired person perception guidance interactive control system according to claim 1, characterized in that: The Beidou / GPS positioning module has a positioning accuracy of meters in outdoor scenarios.

6. A control method based on the system according to any one of claims 1 to 5, characterized in that: include: S1, data collection stage: Multi-source sensors collect environmental data in real time and transmit it to smart terminals via Bluetooth / BLE protocol; S2, data processing and analysis stage: The edge computing module pre-processes the data and completes dynamic environment modeling and target trajectory prediction in conjunction with the cloud backend; S3, interactive feedback stage: Generate voice and vibration feedback based on the processing results, and the danger level will trigger high-intensity warnings first; S4, system self-optimization stage: adjust interaction parameters based on user feedback, with an optimization period of ≤ 24 hours; S5, system self-test phase: test sensor accuracy, network connectivity and battery power.

7. The method according to claim 6, characterized in that In step S2, the data fusion algorithm uses Kalman filtering and dynamically allocates weights in combination with sensor timestamps (accuracy ≤ 1ms), where the visual data has a weight of 40% to 60% and the ultrasonic data has a weight of 20% to 30%.

8. The method according to claim 6, wherein: In step S3, the naturalness MOS score of the voice navigation instruction reaches a high-quality level, and the hazard warning response time reaches seconds.

9. The method according to claim 6, wherein: In step S3, the user feedback includes the voice command "volume up" or "vibration enhancement", and the system automatically adjusts the parameters through the reinforcement learning algorithm.

10. The method according to claim 6, wherein: In step S2, the cloud backend supports historical trajectory query and triggers the manual customer service intervention mechanism for abnormal behavior.

Images