A multi-source signal cooperative blind guiding method and system

Abstract

The application discloses a kind of multi-source signal coordination's guide blind method and system, belong to blind person auxiliary navigation technical field;The method is carried out time synchronization and space registration to each sensor output;According to the working condition of each sensor under current environment quality factor is calculated, and according to environmental quality factor and inherent measurement standard deviation the observation of the same obstacle is adaptively weighted fusion according to confidence, and the fusion direction and fusion distance are obtained;Again, in combination with obstacle distance, approach speed and direction, the risk degree is calculated, and is mapped to passable grid map;Further search recommended direction of travel;Through voice, bone conduction tone and multi-modal prompting device such as tactile vibration, with the risk degree continuous modulation output early warning information and avoidance suggestion.The application can realize stable and accurate obstacle perception under strong light, rain and fog and other complex environments, and provide intuitive and timely obstacle avoidance guidance to blind people through multi-modal early warning linked with risk degree.

Description

Technical Field

[0001] This invention relates to the field of assistive navigation technology for the blind, and in particular to a multi-source signal coordinated navigation method and system for the blind. Background Technology

[0002] Visually impaired individuals face numerous safety challenges in their daily lives, struggling to perceive various obstacles such as steps, pedestrians, and vehicles. Traditional guidance methods primarily include guide canes and guide dogs; however, these methods have significant limitations. Guide canes have a limited sensing range, typically detecting only ground obstacles in a small area in front of the user, and are ineffective at perceiving obstacles above the ground or distant dynamic obstacles. While guide dogs can perceive a wider range of environmental information, their training costs are high, their numbers are limited, and they also struggle to anticipate dynamic obstacles in complex traffic scenarios. With the rapid development of wearable devices and sensor technology, electronic sensing-based assistive guidance solutions are gradually becoming a research hotspot.

[0003] Existing electronic navigation technologies for the visually impaired mainly fall into the following categories. The first is a single-vision approach, which uses a camera to capture images of the area ahead and employs image processing algorithms for obstacle detection. This is currently a widely used approach. However, the single-vision approach suffers from serious reliability issues in complex environments. In strong light, backlight, at night, or in adverse weather conditions such as rain or fog, the visual image quality deteriorates drastically, causing the obstacle detection algorithm to fail. The second approach adds a distance sensor to the single-vision approach, using ultrasonic, laser, or millimeter-wave radar devices to measure the distance to obstacles. However, the reliability of these distance sensors varies significantly under different media and weather conditions, and they only acquire distance information, failing to determine the type and movement of obstacles. The third approach is a simple multi-sensor parallel approach, where multiple sensors are simultaneously installed on the navigation device. However, each sensor operates independently, and the outputs are fragmented, failing to effectively fuse the signals from different sources. At the signal processing level, this approach typically uses a fixed sensor selection strategy or simple averaging, failing to consider the dynamic changes in the reliability of each sensor under different environments. This results in a significant decrease in overall perception accuracy when a sensor fails due to environmental factors.

[0004] The aforementioned existing technical solutions generally suffer from several common problems. First, existing technologies fail to dynamically assess the reliability of each heterogeneous sensor based on environmental conditions and perform adaptive confidence fusion accordingly, resulting in difficulties in maintaining stable and reliable obstacle perception capabilities in complex and ever-changing travel environments. Second, existing technologies lack a unified and comprehensive hazard assessment mechanism that integrates obstacle distance, movement trends, and spatial orientation, making it impossible to accurately determine the threat level of each obstacle to the wearer. Third, existing technologies lack a multimodal warning mechanism linked to hazard levels, making it difficult to intuitively and promptly convey the degree of danger of obstacles and safe avoidance directions to blind users.

[0005] To address the numerous problems existing in the prior art, this invention proposes a multi-source signal collaborative guidance method and system for the blind. It aims to achieve stable and reliable obstacle perception and intuitive and timely obstacle avoidance guidance in complex environments through confidence-adaptive fusion of multi-source heterogeneous sensors, construction of passable grid maps based on hazard levels, and a multimodal early warning mechanism linked to hazard levels. Summary of the Invention

[0006] The purpose of this invention is to address the problems of existing electronic guidance schemes, such as the easy failure of single sensors, the lack of effective fusion of multiple sensors, and the lack of hazard assessment and multimodal early warning, by providing a guidance method and system with multi-source signal coordination.

[0007] To achieve the above-mentioned objectives, the specific technical solution adopted by this invention is as follows:

[0008] A multi-source signal coordinated guidance method for the blind, the method being implemented based on guidance glasses worn by the blind, the guidance glasses integrating multiple heterogeneous sensors and a multimodal prompting device that outputs warnings to the wearer, the multiple heterogeneous sensors including a visual sensor, a ranging sensor, an inertial measurement sensor, and a sound sensor, the method comprising the following steps performed sequentially:

[0009] Step S1: Simultaneously collect multi-source signals of the environment in front of the wearer through multiple heterogeneous sensors of the guide glasses, and perform time synchronization and spatial registration of the output of each heterogeneous sensor.

[0010] The multi-source signals include images output by the visual sensor, distance signals output by the range sensor, head posture signals output by the inertial measurement sensor, and sound field signals output by the sound sensor; the observations of each sensor in the same spatial direction are unified to the same coordinate system and the same time with the wearer's head as the reference.

[0011] Step S2: Based on the working conditions of each heterogeneous sensor in the current environment, calculate the environmental quality factor of each sensor, and perform confidence-adaptive weighted fusion of the observations of each sensor about the same obstacle to obtain the fused orientation and fused distance of the obstacle.

[0012] The environmental quality factor characterizes the reliability of the sensor's output under the current environment; and based on the environmental quality factor of each sensor and its inherent measurement accuracy, the observations of each sensor regarding the same obstacle are adaptively weighted and fused with confidence.

[0013] Step S3: For each obstacle obtained by fusion, calculate the danger level of the obstacle by combining its fusion distance, approach speed relative to the wearer and relative orientation, and map the danger level of each obstacle to the passable grid map in front of the wearer, marking the passability of each grid.

[0014] The degree of danger increases as the distance to the obstacle decreases, the approach speed increases, and the deviation from the direction of travel decreases.

[0015] Step S4: Based on the passable grid map, search for a passable direction in front of the wearer that leads from the current position to the target direction while avoiding high-risk obstacles, and use it as a recommended direction of travel.

[0016] Step S5: Based on the recommended travel direction and the danger level of each obstacle, the multimodal prompting device is encoded with a warning, and warning information and avoidance suggestions are output to the wearer.

[0017] The multimodal prompting device is encoded with a warning code so that at least two of the voice prompts, bone conduction tones and tactile vibrations are continuously modulated according to the obstacle hazard level and output to the wearer. The device then periodically returns to step S1 and updates the fusion result, hazard level and warning output in real time according to changes in the environment and obstacles.

[0018] Furthermore, in step S2, the guide glasses are recorded as having a total of [number missing]. A heterogeneous sensor capable of observing obstacles, the first The environmental quality factor of each sensor is ( , Its inherent measurement standard deviation is Then the first The weights of each sensor in the adaptive weighted fusion of confidence levels for the same obstacle Determined by the following formula: Among them, the weights of each sensor satisfy the following: The higher the environmental quality factor and the smaller the inherent measurement standard deviation, the greater the weight of the sensor; when the first When a sensor fails in the current environment, take This prevents it from participating in the integration.

[0019] Furthermore, in step S2, the first Environmental quality factors of individual sensors The amount of environmental interference corresponding to this sensor is determined by the following formula: ;in, For the first The amount of environmental interference currently experienced by each sensor For the first The environmental interference reference value for each sensor; for a vision sensor, the environmental interference is the percentage of overexposed or underexposed pixels in the acquired image; for a ranging sensor, the environmental interference is the echo noise power of the distance signal; for a sound sensor, the environmental interference is the ratio of the background noise sound pressure level to the effective sound source sound pressure level.

[0020] Furthermore, in step S2, the same obstacle is... Distance observed by each sensor By performing fusion, the fusion distance of the obstacle can be obtained. : The fusion standard deviation of the fusion distance. Determined by the following formula: ; Location of obstacle fusion The azimuth was observed by each sensor using the same weighting method. Weighted average yields: The fusion orientation is the horizontal angle between the obstacle and the wearer's direction of travel.

[0021] Furthermore, in step S3, the first The risk level of each obstacle The fusion distance of the obstacle Relative approach speed With integration direction Based on a comprehensive assessment of all three factors: ;in, Distance-sensitive scale, unit is ; A speed-sensitive scale, unit is ; This is a location-sensitive scale, with units of 1. Relative approach speed The rate at which the distance between the obstacle and the wearer decreases is taken as the obstacle moves away from the wearer. At this point, the speed term is taken .

[0022] Furthermore, in step S3, the area in front of the wearer is discretized into several grids along the distance and orientation directions to form a passable grid map; for those falling into the first... The maximum hazard level of an obstacle in a given grid cell is taken as the grid hazard level for that grid cell. Unobstructed grid entry When the grid hazard level Greater than the passability threshold If the grid is impassable, mark it as passable; otherwise, mark it as passable. The passability threshold is... satisfy .

[0023] Furthermore, in step S4, each direction in the passable grid map is selected as a candidate travel direction, and a comprehensive cost is calculated for each candidate travel direction. ;in, The azimuth angle of the target direction relative to the wearer's direction of travel; This is the sum of the hazard levels of each grid cell along the candidate's travel direction; For the direction deviation weighting coefficient, For obstacle cost weighting coefficients, and , This will increase the overall cost. The recommended travel direction is the smallest candidate travel direction where all grid cells along the way are passable. .

[0024] Furthermore, in step S5, the vibration intensity of the tactile vibration The frequency of the bone conduction tone prompt. and the broadcast interval of the voice prompts According to the current highest risk level Continuous modulation as follows: ; ; ;in, , These are the lower and upper limits of the vibration intensity, respectively. , These are the lower and upper limits of the prompt audio rate, respectively, in units of... ; , These are the lower and upper limits of the voice broadcast interval, respectively, in units of... The higher the level of danger, the stronger the vibration, the higher the pitch, and the shorter the broadcast interval.

[0025] Furthermore, in step S5, the voice prompt includes the obstacle's location relative to the wearer and the recommended direction of travel, wherein the location is determined by a fused location. The tactile vibrations are mapped to directional terms according to the set orientation zones; the tactile vibrations are output through vibrators located on both temples of the guide glasses, and are fused according to the obstacle orientation. The positive or negative selection of the left or right vibrator output ensures that the vibrating side corresponds to the side where the obstacle is located; the periodic return to the update cycle of step S1. satisfy And when the current highest level of danger Greater than the emergency threshold At that time, the update cycle will be shortened to It immediately triggers tactile vibrations and voice prompts.

[0026] The present invention also provides a multi-source signal coordinated guide system for implementing the above method. The system includes a guide glasses body and multiple heterogeneous sensors, a signal processing module, and a multimodal prompting device disposed on the guide glasses body. The signal processing module includes a registration unit, a fusion unit, a risk assessment unit, and a decision coding unit. The multimodal prompting device includes a voice broadcasting unit, a bone conduction unit, and tactile vibrators disposed on both temples.

[0027] The multiple heterogeneous sensors include a vision sensor, a ranging sensor, an inertial measurement sensor, and a sound sensor, which are used to collect images of the environment in front of the wearer, distance signals, head posture signals, and sound field signals, respectively.

[0028] The signal processing module is connected to the multiple heterogeneous sensors and the multimodal warning device. The signal processing module includes a registration unit, a fusion unit, a hazard assessment unit, and a decision coding unit. The registration unit performs time synchronization and spatial registration of the outputs of each heterogeneous sensor. The fusion unit performs adaptive weighted fusion of observations of the same obstacle based on the environmental quality factor and inherent measurement standard deviation of each sensor to obtain the fused distance and fused orientation. The hazard assessment unit calculates the hazard level of each obstacle and generates a passable grid map. The decision coding unit searches for recommended travel directions and performs warning coding for the multimodal warning device.

[0029] The multimodal prompting device includes a voice broadcasting unit, a bone conduction unit, and tactile vibrators located on both sides of the temples of the guide glasses. It is used to output voice prompts, bone conduction tones, and tactile vibrations to the wearer after continuous modulation according to the warning code of the decision coding unit.

[0030] The beneficial effects of this invention are:

[0031] Firstly, by working together with multiple heterogeneous sensors, it covers multiple perception dimensions such as vision, distance, motion, and hearing, breaking through the limitations of single sensors in specific environments and significantly improving the coverage and environmental adaptability of obstacle perception.

[0032] Secondly, by dynamically evaluating the reliability of each heterogeneous sensor under the current environment through environmental quality factors, and combining the inherent measurement accuracy to perform confidence-adaptive weighted fusion, a fusion strategy of intelligently reducing the weight of failed sensors and automatically enhancing reliable sensors is realized. Even in complex environments such as strong light, rain and fog, it can still maintain stable and reliable obstacle perception accuracy, and the fusion ranging standard deviation is better than any single sensor.

[0033] Third, by constructing a hazard assessment model that integrates distance, approach speed, and orientation, the degree of obstacle threat is quantitatively represented and mapped into a passable grid map, realizing the transformation from point obstacle perception to regional path planning, and providing structured input for subsequent obstacle avoidance decisions.

[0034] Fourth, by using a comprehensive cost function to search for recommended travel directions in the passable grid map, taking into account both directional deviation and path safety, optimal avoidance can be achieved while ensuring the direction towards the target.

[0035] Fifth, by using multimodal prompting devices such as voice, bone conduction tone and tactile vibration to continuously modulate the output according to the degree of danger, a multi-channel, multi-sensory early warning mechanism driven by the degree of danger is realized, enabling blind people to intuitively and timely perceive the degree of threat of obstacles and the direction of avoidance, and automatically triggering the emergency early warning mode according to the increase of danger, which significantly improves the practicality and user experience of guide assist. Attached Figure Description

[0036] Figure 1 This is a flowchart of a multi-source signal coordinated guidance method for the blind according to the present invention;

[0037] Figure 2 This is a schematic diagram of a multi-source signal coordinated guidance system for the blind according to the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0039] The method and system of this invention are applicable to the auxiliary navigation of visually impaired people wearing guide glasses for daily travel, and are especially suitable for obstacle perception and obstacle avoidance guidance in complex environments where single sensors are prone to failure, such as strong light, backlight, night, rain and fog. In addition, this invention can also be used to assist obstacle avoidance perception for workers in low visibility environments such as fire fighting, rescue, and mining, as well as for multi-source obstacle avoidance navigation of mobile robots in changing environments, which have similar multi-source perception and fusion requirements.

[0040] For scenarios where environmental conditions are simple and a single sensor can work stably, or where only a single modal prompt is required, the configuration of heterogeneous sensors and the multimodal prompting method can be simplified or adjusted according to actual needs.

[0041] Example 1

[0042] like Figure 1 The diagram shows a flowchart of a multi-source signal coordinated guidance method for the blind, based on guide glasses worn by blind people. The guide glasses integrate multiple heterogeneous sensors and a multimodal prompting device that outputs warnings to the wearer. The multiple heterogeneous sensors include a visual sensor, a ranging sensor, an inertial measurement sensor, and a sound sensor. The method includes the following steps performed sequentially:

[0043] Step S1: Simultaneously collect multi-source signals of the environment in front of the wearer through multiple heterogeneous sensors of the guide glasses, and perform time synchronization and spatial registration of the output of each heterogeneous sensor.

[0044] The multi-source signals include images output by the visual sensor, distance signals output by the range sensor, head posture signals output by the inertial measurement sensor, and sound field signals output by the sound sensor; the observations of each sensor in the same spatial direction are unified to the same coordinate system and the same time with the wearer's head as the reference.

[0045] This embodiment selects a working condition where a blind person wearing guide glasses walks on a city road in a strong midday sun environment accompanied by light rain as the subject.

[0046] The guide glasses are integrated The three heterogeneous sensors capable of observing obstacles are a binocular vision sensor, a millimeter-wave radar, a ToF laser rangefinder, and an ultrasonic rangefinder. The inherent measurement standard deviations of each sensor are as follows: (Binocular vision) (Millimeter-wave radar) (ToF laser) (Ultrasound).

[0047] The parameters of the hazard model are taken at a distance-sensitive scale. Speed-sensitive scale Orientation-sensitive scale Passable determination threshold The multimodal coding parameters are taken as the lower limit of vibration intensity. Upper limit The audio rate lower limit is indicated. Upper limit Lower limit of voice broadcast interval Upper limit Update cycle Emergency threshold .

[0048] The guide glasses use four different sensors to simultaneously acquire multi-source signals from the environment in front of the wearer. The registration unit performs time synchronization and spatial registration of the outputs of each sensor, unifying the observations of each sensor in the same spatial direction in front of the wearer into the same coordinate system and at the same time, with the wearer's head as the reference.

[0049] At this moment, approximately in front of the wearer There is a pedestrian slowly approaching laterally, which is considered an obstacle to be evaluated. All four sensors have observed this obstacle.

[0050] Step S2: Based on the working conditions of each heterogeneous sensor in the current environment, calculate the environmental quality factor of each sensor, and perform confidence-adaptive weighted fusion of the observations of each sensor about the same obstacle to obtain the fused orientation and fused distance of the obstacle.

[0051] The environmental quality factor characterizes the reliability of the sensor's output under the current environment; and based on the environmental quality factor of each sensor and its inherent measurement accuracy, the observations of each sensor regarding the same obstacle are adaptively weighted and fused with confidence.

[0052] The guide glasses mentioned above are set up in total. A heterogeneous sensor capable of observing obstacles, the first The environmental quality factor of each sensor is ( , Its inherent measurement standard deviation is Then the first The weights of each sensor in the adaptive weighted fusion of confidence levels for the same obstacle Determined by the following formula: Among them, the weights of each sensor satisfy the following: The higher the environmental quality factor and the smaller the inherent measurement standard deviation, the greater the weight of the sensor; when the first When a sensor fails in the current environment, take This prevents it from participating in the integration.

[0053] No. Environmental quality factors of individual sensors The amount of environmental interference corresponding to this sensor is determined by the following formula: ;in, For the first The amount of environmental interference currently experienced by each sensor For the first The environmental interference reference value for each sensor; for a vision sensor, the environmental interference is the percentage of overexposed or underexposed pixels in the acquired image; for a ranging sensor, the environmental interference is the echo noise power of the distance signal; for a sound sensor, the environmental interference is the ratio of the background noise sound pressure level to the effective sound source sound pressure level.

[0054] Furthermore, in step S2, the same obstacle is... Distance observed by each sensor By performing fusion, the fusion distance of the obstacle can be obtained. : The fusion standard deviation of the fusion distance. Determined by the following formula: ; Location of obstacle fusion The azimuth was observed by each sensor using the same weighting method. Weighted average yields: The fusion orientation is the horizontal angle between the obstacle and the wearer's direction of travel.

[0055] The environmental quality factor for each sensor is calculated using the environmental quality factor formula. Under this operating condition: the binocular vision sensor experiences image overexposure due to strong midday sunlight; the percentage of overexposed pixels is... Reference values ,have to Millimeter-wave radar is less affected by rainfall, and its echo noise power is low. Reference values ,have to ToF lasers experience interference due to raindrop scattering. Reference values ,have to Ultrasonic sensor interference Reference values ,have to It is evident that the environmental quality factor of visual sensors is lowest under strong light.

[0056] The fusion weights of each sensor are calculated using the weighting formula, starting with the weighted inverse variance of each sensor. Binocular vision Millimeter-wave radar ToF laser Ultrasound Their sum is Therefore, the weights of each sensor are: , , , The sum of the weights is .

[0057] It is evident that in the current environment of strong light and rainfall, millimeter-wave radar and ToF lasers occupy the main weight due to their high reliability and high accuracy, while the weight of visual sensors that fail in strong light is automatically reduced.

[0058] The distances observed by the four sensors to the obstacle are as follows: , , , The observation azimuths are respectively , , , From the fusion distance formula fusion distance ; From the fusion orientation formula Find the fusion direction (Right now The obstacle is located to the right front of the direction of travel.

[0059] From the fusion standard deviation formula Less than the smallest single-sensor standard deviation among the four sensors. This indicates that multi-source fusion improves ranging accuracy.

[0060] Step S3: For each obstacle obtained by fusion, calculate the danger level of the obstacle by combining its fusion distance, approach speed relative to the wearer and relative orientation, and map the danger level of each obstacle to the passable grid map in front of the wearer, marking the passability of each grid.

[0061] The degree of danger increases as the obstacle distance decreases, the approach speed increases, and the deviation from the direction of travel decreases. The risk level of each obstacle The fusion distance of the obstacle Relative approach speed With integration direction Determined by a combination of three factors: ;in, Distance-sensitive scale, unit is ; A speed-sensitive scale, unit is ; This is a location-sensitive scale, with units of 1. Relative approach speed The rate at which the distance between the obstacle and the wearer decreases is taken as the obstacle moves away from the wearer. At this point, the speed term is taken .

[0062] The area in front of the wearer is discretized into several grids along the distance and orientation directions to form a passable grid map; for those falling into the first grid... The maximum hazard level of an obstacle in a given grid cell is taken as the grid hazard level for that grid cell. Unobstructed grid entry When the grid hazard level Greater than the passability threshold If the grid is impassable, mark it as passable; otherwise, mark it as passable. The passability threshold is... satisfy .

[0063] The approach velocity of the pedestrian barrier relative to the wearer is estimated by the rate of decrease of the fusion distance between adjacent cycles. Substituting the values ​​from the hazard formula... , , , , , Calculate each factor separately: distance term Speed ​​item Orientation item Therefore, the degree of danger .

[0064] Discretize the area in front of the wearer into a grid along the distance and orientation directions. The pedestrian obstacle falls approximately in front of the wearer. Distance approximately The grid, the grid's danger level .because The grid is marked as impassable; the remaining unobstructed grids in front of the wearer have a hazard level of [missing information]. Marked as passable.

[0065] Step S4: Based on the passable grid map, search for a passable direction in front of the wearer that leads from the current position to the target direction while avoiding high-risk obstacles, and use it as a recommended direction of travel.

[0066] In the accessible grid map, each direction is selected as a candidate travel direction, and a comprehensive cost is calculated for each candidate travel direction. ;in, The azimuth angle of the target direction relative to the wearer's direction of travel; This is the sum of the hazard levels of each grid cell along the candidate's travel direction; For the direction deviation weighting coefficient, For obstacle cost weighting coefficients, and , This will increase the overall cost. The recommended travel direction is the smallest candidate travel direction where all grid cells along the way are passable. .

[0067] Assuming the target direction is directly forward, that is... Each direction is selected as a candidate direction of travel, and a weighting coefficient for the deviation from the direction is applied. Obstacle cost weighting coefficient According to the comprehensive cost formula Calculate the cost for each candidate direction: forward direction ( The route passes through the aforementioned impassable fences. )cost However, it was excluded because the grid along the way made it impassable; left-leaning direction ( The sum of the danger levels of the grid along the way )cost Passable along the way; deviating to the right ( The sum of the danger levels of the grid along the way )cost It is passable along the way.

[0068] Among the candidate directions that are all passable along the way, the left-leaning direction has the lowest overall cost. Therefore, the recommended direction of travel is adopted. This means guiding the wearer to move to the left and front to avoid the pedestrian.

[0069] Step S5: Based on the recommended travel direction and the danger level of each obstacle, the multimodal prompting device is encoded with a warning, and warning information and avoidance suggestions are output to the wearer.

[0070] The multimodal prompting device is encoded with a warning code so that at least two of the voice prompts, bone conduction tones and tactile vibrations are continuously modulated according to the obstacle hazard level and output to the wearer. The device then periodically returns to step S1 and updates the fusion result, hazard level and warning output in real time according to changes in the environment and obstacles.

[0071] The vibration intensity of the tactile vibration The frequency of the bone conduction tone prompt. and the broadcast interval of the voice prompts According to the current highest risk level Continuous modulation as follows: ; ; ;in, , These are the lower and upper limits of the vibration intensity, respectively. , These are the lower and upper limits of the prompt audio rate, respectively, in units of... ; , These are the lower and upper limits of the voice broadcast interval, respectively, in units of... The higher the level of danger, the stronger the vibration, the higher the pitch, and the shorter the broadcast interval.

[0072] The voice prompts include the obstacle's location relative to the wearer and the recommended direction of travel, the location being determined by a fused orientation. The tactile vibrations are mapped to directional terms according to the set orientation zones; the tactile vibrations are output through vibrators located on both temples of the guide glasses, and are fused according to the obstacle orientation. The positive or negative selection of the left or right vibrator output ensures that the vibrating side corresponds to the side where the obstacle is located; the periodic return to the update cycle of step S1. satisfy And when the current highest level of danger Greater than the emergency threshold At that time, the update cycle will be shortened to It immediately triggers tactile vibrations and voice prompts.

[0073] Current highest level of danger The tactile vibration intensity is calculated using the multimodal modulation formula. Bone conduction indicates audio frequency. Voice broadcast interval Due to the fusion orientation The system is mapped by direction, and the voice announcement reads, "There is a pedestrian to your right; please move to your left." The vibrator on the right temple of the guide glasses is then adjusted to increase its intensity. Output vibration so that the vibrating side corresponds to the right side of the obstacle.

[0074] Due to the current highest level of danger If the emergency threshold is not reached, the system will update according to the normal update cycle. Return to step S1 periodically.

[0075] As the wearer walks and pedestrians move, the system repeats the above steps in real time:

[0076] When a pedestrian gets closer, reducing the merging distance and increasing the danger level to exceed the emergency threshold... At that time, the system will shorten the update cycle to It immediately triggers tactile vibration and voice prompts, with increased vibration intensity, higher prompt frequency, and shorter voice broadcast intervals, conveying a stronger and more urgent avoidance warning to the wearer;

[0077] When pedestrians move away from the wearer and the level of danger decreases, the intensity of each modal cue weakens accordingly until it is deactivated.

[0078] The above results demonstrate that the method described in this embodiment maintains stable obstacle perception (fusion ranging standard deviation) in complex environments with strong light and rainfall by adaptively fusing confidence levels of multi-source signals. (Superior to any single sensor), and through hazard assessment and multimodal early warning, it achieves intuitive and timely obstacle avoidance guidance, verifying the effectiveness of the method of the present invention.

[0079] Example 2

[0080] like Figure 2 As shown, this embodiment provides a multi-source signal coordinated guide system for the blind. The system includes a guide glasses body and multiple heterogeneous sensors, a signal processing module, and a multimodal prompting device disposed on the guide glasses body. The signal processing module includes a registration unit, a fusion unit, a risk assessment unit, and a decision coding unit. The multimodal prompting device includes a voice broadcasting unit, a bone conduction unit, and tactile vibrators disposed on both temples.

[0081] The guide glasses body is a glasses structure for blind people to wear, including a frame and temples on both sides, providing a mounting carrier and power supply for various heterogeneous sensors, signal processing modules and multimodal prompting devices.

[0082] The plurality of heterogeneous sensors include a visual sensor, a ranging sensor, an inertial measurement sensor, and a sound sensor. The visual sensor is a binocular camera mounted on the front of the frame to capture images of the environment in front of the wearer. The ranging sensor includes at least two of a millimeter-wave radar, a ToF laser ranging sensor, and an ultrasonic ranging sensor mounted on the front of the frame to acquire distance signals to obstacles in front. The inertial measurement sensor is located inside the frame to acquire head posture signals of the wearer. The sound sensor is a microphone array mounted on the frame or temples to acquire sound field signals of the environment in front. In this embodiment, the heterogeneous sensors consist of four sensors—a binocular visual sensor, a millimeter-wave radar, a ToF laser ranging sensor, and an ultrasonic ranging sensor—capable of observing obstacles.

[0083] The signal processing module is located within the guide glasses and is connected to the multiple heterogeneous sensors and multimodal prompting device. It serves as the core of the system's computation and control, comprising a registration unit, a fusion unit, a hazard assessment unit, and a decision coding unit. The registration unit performs time synchronization and spatial registration of the outputs from each heterogeneous sensor, unifying the observations of each sensor in the same spatial direction to the same coordinate system and time frame based on the wearer's head. The fusion unit is used to adjust the environmental quality factors of each sensor... vs. inherent measurement standard deviation By weight Observations of the same obstacle are subjected to confidence-adaptive weighted fusion to obtain the fused distance. With integration direction The hazard assessment unit is used to assess the risk level according to... Calculate the hazard level of each obstacle and map each obstacle hazard level to a passable grid map, then sort them by grid hazard level. With the passability determination threshold The accessibility of each grid cell is marked. The decision coding unit is used to calculate the overall cost. The minimum criterion is used to search for recommended travel directions, and the multimodal warning device is coded with warnings based on the recommended travel directions and the danger level of each obstacle.

[0084] The multimodal prompting device includes a voice broadcasting unit, a bone conduction unit, and tactile vibrators located on both temples of the guide glasses. The voice broadcasting unit announces the location of obstacles and recommends a direction of travel; the bone conduction unit outputs a prompting tone modulated according to the level of danger; the tactile vibrators output vibrations modulated according to the level of danger, and select the left or right vibrator output based on the positive or negative direction of the obstacle's location. The multimodal prompting device adjusts the tactile vibration intensity according to the warning code of the decision coding unit. Bone conduction cues audio rate Interval with voice broadcast According to the current highest level of danger It is continuously modulated and then output to the wearer.

[0085] In this embodiment, during system operation, various heterogeneous sensors acquire images, distance signals, head posture signals, and sound field signals in real time and report them to the signal processing module. The registration unit of the signal processing module completes spatiotemporal registration, the fusion unit completes confidence-adaptive weighted fusion, the hazard assessment unit completes hazard calculation and generates a passable grid map, and the decision coding unit completes recommended travel direction search and warning coding. The multimodal prompting device outputs voice, bone conduction, and tactile vibrations to the wearer based on the warning code and the hazard level. All modules work together to achieve multi-source signal coordinated guidance for the blind.

[0086] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A multi-source signal coordinated guidance method for the blind, wherein the multi-source signal coordinated guidance method is used in guidance glasses worn by blind people, characterized in that, The guide glasses integrate multiple heterogeneous sensors and a multimodal prompting device that outputs warnings to the wearer. The multiple heterogeneous sensors include a visual sensor, a distance sensor, an inertial measurement sensor, and a sound sensor. The multi-source signal coordinated guide method includes the following steps: S1, the multi-source signals of the environment in front of the wearer are simultaneously collected by multiple heterogeneous sensors of the guide glasses, and the outputs of each heterogeneous sensor are synchronized in time and space. The multi-source signals include the image output by the visual sensor, the distance signal output by the range sensor, the head posture signal output by the inertial measurement sensor, and the sound field signal output by the sound sensor; the observations of each sensor in the same spatial direction are unified to the same coordinate system and the same time with the wearer's head as the reference. S2, based on the working conditions of each heterogeneous sensor in the current environment, calculate the environmental quality factor of each sensor, and perform confidence-adaptive weighted fusion of the observations of each sensor about the same obstacle to obtain the fusion orientation and fusion distance of the obstacle; S3. For each obstacle obtained by fusion, the danger level of the obstacle is calculated by combining its fusion distance, approach speed relative to the wearer and relative orientation, and the danger level of each obstacle is mapped to the passable grid map in front of the wearer, and the passability of each grid is marked. S4. Based on the passable grid map, search for a passable direction in front of the wearer that leads from the current position to the target direction while avoiding high-risk obstacles, and use it as a recommended direction of travel. S5. Based on the recommended direction of travel and the danger level of each obstacle, the multimodal prompting device is encoded with a warning, and warning information and avoidance suggestions are output to the wearer.

2. The multi-source signal coordinated blind guidance method according to claim 1, characterized in that, In step S2, the guide glasses are set to a total of A heterogeneous sensor capable of observing obstacles, the first The environmental quality factor of each sensor is ( , Its inherent measurement standard deviation is Then the first The weights of each sensor in the adaptive weighted fusion of confidence levels for the same obstacle Determined by the following formula: Among them, the weights of each sensor satisfy the following: .

3. The multi-source signal coordinated blind guidance method according to claim 2, characterized in that, In step S2, the first Environmental quality factors of individual sensors The amount of environmental interference corresponding to this sensor is determined by the following formula: ;in, For the first The amount of environmental interference currently experienced by each sensor For the first Environmental interference reference values ​​for each sensor.

4. The multi-source signal coordinated blind guidance method according to claim 3, characterized in that, For visual sensors, the environmental interference is measured as the percentage of overexposed or underexposed pixels in the acquired image; for ranging sensors, the environmental interference is measured as the echo noise power of the distance signal; for sound sensors, the environmental interference is measured as the ratio of the background noise sound pressure level to the effective sound source sound pressure level.

5. The multi-source signal coordinated blind guidance method according to claim 3, characterized in that, In step S2, for the same obstacle, Distance observed by each sensor By performing fusion, the fusion distance of the obstacle can be obtained. : ; The fusion standard deviation of the fusion distance : ; Location of obstacle fusion The azimuth was observed by each sensor using the same weighting method. Weighted average yields: The fusion orientation is the horizontal angle between the obstacle and the wearer's direction of travel.

6. The multi-source signal coordinated blind guidance method according to claim 5, characterized in that, In step S3, the first The risk level of each obstacle The fusion distance of the obstacle Relative approach speed With integration direction Based on a comprehensive assessment of all three factors: ;in, Distance-sensitive scale, unit is ; A speed-sensitive scale, unit is ; This is a location-sensitive scale, with units of 1. Relative approach speed The rate at which the distance between the obstacle and the wearer decreases is taken as the obstacle moves away from the wearer. At this point, the speed term is taken ; The area in front of the wearer is discretized into several grids along the distance and orientation directions to form a passable grid map. For falling into the first The maximum hazard level of an obstacle in a given grid cell is taken as the grid hazard level for that grid cell. Unobstructed grid entry ; When the grid risk level Greater than the passability threshold If the grid is impassable, mark it as passable; otherwise, mark it as passable. The passability threshold is... satisfy .

7. The multi-source signal coordinated blind guidance method according to claim 6, characterized in that, In step S4, each direction is selected as a candidate travel direction in the passable grid map, and a comprehensive cost is calculated for each candidate travel direction. ;in, The azimuth angle of the target direction relative to the wearer's direction of travel; This is the sum of the hazard levels of each grid cell along the candidate's travel direction; For the direction deviation weighting coefficient, For obstacle cost weighting coefficients, and , This will increase the overall cost. The recommended travel direction is the smallest candidate travel direction where all grid cells along the way are passable. .

8. The multi-source signal coordinated blind guidance method according to claim 7, characterized in that, In step S5, the vibration intensity of the tactile vibration The frequency of the bone conduction tone prompt. and the broadcast interval of the voice prompts According to the current highest risk level Continuous modulation as follows: ; ; ;in, , These are the lower and upper limits of the vibration intensity, respectively. , These are the lower and upper limits of the prompt audio rate, respectively, in units of... ; , These are the lower and upper limits of the voice broadcast interval, respectively, in units of... The higher the level of danger, the stronger the vibration, the higher the pitch, and the shorter the broadcast interval.

9. The multi-source signal coordinated blind guidance method according to claim 8, characterized in that, In step S5, the voice prompt includes the obstacle's location relative to the wearer and the recommended direction of travel, wherein the location is determined by a fused location. Map the designated orientation zones to directional terms; The tactile vibration is output through vibrators located on both temples of the guide glasses, and is distributed according to the obstacle fusion orientation. The positive or negative selection allows you to choose the left or right vibrator output, so that the vibrating side corresponds to the side where the obstacle is located; The update cycle of the periodic return step S1 satisfy And currently the highest level of danger Greater than the emergency threshold At that time, the update cycle will be shortened to It immediately triggers tactile vibrations and voice prompts.

10. A multi-source signal coordinated guidance system for the blind, characterized in that, The system is used to implement the method of any one of claims 1 to 9, the system comprising a guide glasses body and a plurality of heterogeneous sensors, a signal processing module and a multimodal prompting device disposed on the guide glasses body: The multiple heterogeneous sensors include a vision sensor, a ranging sensor, an inertial measurement sensor, and a sound sensor, which are used to collect images of the environment in front of the wearer, distance signals, head posture signals, and sound field signals, respectively. The signal processing module is connected to the multiple heterogeneous sensors and the multimodal prompting device. The signal processing module includes a registration unit, a fusion unit, a hazard assessment unit, and a decision coding unit. The registration unit performs time synchronization and spatial registration of the outputs of each heterogeneous sensor. The fusion unit performs adaptive weighted fusion of observations of the same obstacle based on the environmental quality factor and inherent measurement standard deviation of each sensor to obtain the fused distance and fused orientation. The hazard assessment unit calculates the hazard level of each obstacle and generates a passable grid map. The decision coding unit is used to search for recommended travel directions and to perform early warning coding for the multimodal prompting device; The multimodal prompting device includes a voice broadcasting unit, a bone conduction unit, and tactile vibrators located on both sides of the temples of the guide glasses. It is used to output voice prompts, bone conduction tones, and tactile vibrations to the wearer after continuous modulation according to the warning code of the decision coding unit.

Images