Blind guiding method for blind people based on multi-modal perception

Abstract

This invention relates to a multimodal perception-based guidance method and wearable device for the visually impaired. The method includes the following steps: real-time acquisition of multimodal data; scene classification of environmental images to trigger tactile paving or intersection recognition; in an intersection environment, traffic light recognition and recording the color change patterns; and fusion and judgment of the traffic light recognition results and tactile paving recognition results: when the light is green and there is sufficient remaining time, and the tactile paving indicates a passable path, a passage instruction is generated; otherwise, a waiting instruction is generated, with the waiting instruction having higher priority than the tactile paving navigation instruction. This invention, by fusing multimodal information and introducing a safety priority coverage mechanism, achieves intelligent and safe guidance at complex intersections, significantly improving the safety of visually impaired individuals' travel.

Description

Technical Field

[0001] This invention relates to the field of intelligent guidance technology, and in particular to a method for guiding the blind based on multimodal perception. Background Technology

[0002] Currently, assistive travel devices for visually impaired individuals include smart guide canes. These canes typically integrate multiple sensors, such as cameras, ultrasonic sensors, and inertial measurement units, to perceive the surrounding environment. Existing technologies mainly focus on single-dimensional navigation assistance, such as using image recognition technology to detect obstacles on tactile paving, identify curbs or potholes, or providing voice prompts to indicate road conditions ahead. These technologies help visually impaired individuals avoid static obstacles during their journey to some extent, improving the safety of basic walking.

[0003] However, most existing technologies only focus on obstacle recognition or basic environmental perception, lacking the ability to handle complex traffic scenarios. When visually impaired individuals cross intersections with traffic lights, traditional obstacle detection functions alone cannot handle the complex intersection conditions, making it difficult for them to obtain safe and consistent guidance. Therefore, this paper proposes a method for guiding the blind that can solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a multimodal perception-based guidance method for the blind, which can effectively solve the aforementioned problems.

[0005] To achieve the above requirements, the technical solution adopted by the present invention to solve its technical problem is as follows: A multimodal perception-based guidance method for the blind, applied to wearable devices, is provided, characterized by the following steps: Real-time acquisition of environmental image data, environmental sensor data, and user walking data; Perform scene classification on environmental image data, including tactile paving recognition and intersection recognition; When the scene classification determines the current blind path area, blind path recognition is triggered; When the scene classification determines that the current environment is an intersection, traffic light recognition is triggered, including: The signal area of ​​traffic lights in environmental image data is located by identifying the model; Perform color segmentation on the signal area to identify the current light color; Record the color change pattern of the lights within the signal area and compare it with the preset value; The results of traffic light recognition and tactile paving recognition are combined for judgment: When the light color is green and the remaining green time is greater than the preset safe passage threshold, and the result of the tactile paving recognition indicates that there is a passable path, a passage instruction is generated. When the light color is identified as red, green, or the remaining time is insufficient, or when it cannot be confirmed, a waiting instruction is generated and given priority over the blind path navigation instruction.

[0006] This invention provides a multimodal perception-based guide method for the blind, wherein tactile paving identification includes: The blind path area in environmental image data is delineated by the identification model; Multiple tactile paving protrusions within the tactile paving area are located using a geometric shape detection algorithm, and the distance between them and adjacent tactile paving protrusions is measured. The height data of the corresponding location within the tactile paving area is obtained from environmental sensor data, and target areas that meet the preset height range are selected. The results of the spacing measurement and height data screening are cross-validated. When both meet the preset conditions, it is confirmed as a genuine tactile paving.

[0007] The present invention provides a method for guiding the blind based on multimodal perception, wherein traffic light recognition and tactile paving recognition are executed in parallel; tactile paving recognition is executed continuously at a first frequency, and traffic light recognition is triggered only when an intersection area is detected at a second frequency, the second frequency being higher than the first frequency; when leaving the intersection area, traffic light recognition automatically enters standby mode and only tactile paving recognition is executed.

[0008] The present invention provides a method for guiding the blind based on multimodal perception, wherein the step of scene classification to determine whether the current location is an intersection area includes: The model is used to classify environmental image data into scenes and identify intersection features. When multiple consecutive frames of environmental image data are identified as intersection scenes, the system confirms entry into the intersection area and activates traffic light recognition.

[0009] This invention provides a multimodal perception-based guide method for the blind, wherein changes in light color are recorded through time-series analysis; the time-series analysis includes: Record the color of the same traffic light in multiple consecutive frames of environmental image data; When a light color change is detected, determine whether the switching interval conforms to the preset traffic light cycle range; Exclude light and shadow effects with a duration shorter than a preset threshold.

[0010] The present invention provides a multimodal perception-based guidance method for the blind, which further includes an obstacle detection step, wherein obstacle detection is performed in parallel with tactile paving recognition and traffic light recognition. Obtain distance data of obstacles ahead from environmental sensor data; Generate obstacle avoidance prompts of different levels based on distance data; When the distance to the obstacle is greater than or equal to the first preset threshold, a first-level prompt is generated; when the distance to the obstacle is less than the first preset threshold but greater than or equal to the second preset threshold, a second-level prompt is generated; when the distance to the obstacle is less than the second preset threshold, a third-level prompt is generated.

[0011] The present invention provides a multimodal perception-based navigation method for the blind, wherein the output of navigation instructions adopts priority queue management; the priority queue management is in the following order from high to low: emergency safety risks, important obstacles or traffic signals, navigation guidance, environmental prompts or status feedback; High-level information will be broadcast in a limited manner, and information of the same level will be sorted according to its time urgency. Set a time window to suppress repeated broadcasts of the same information.

[0012] The present invention provides a multimodal perception-based guide glasses for the blind, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the multimodal perception-based guide method for the blind as described in any one of claims 1-7.

[0013] Furthermore, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-mentioned guide method for the blind.

[0014] The beneficial effects of this invention are as follows: The guidance method for the visually impaired provided by this invention overcomes the limitations of single-modal perception in complex intersection environments by deeply fusing the results of traffic light recognition and tactile paving recognition. Existing technologies often rely solely on tactile paving detection for navigation, leading to extremely high safety risks for blind individuals at intersections due to a lack of traffic signal perception. This method, after determining that the light is green and the remaining time is greater than a safety threshold, further integrates the path results from tactile paving recognition before generating the final passage instruction. This complementary mechanism avoids visually impaired individuals making biased decisions due to incomplete information, preventing them from rashly entering intersections in unknown conditions or neglecting the location of tactile paving due to excessive focus on traffic lights. This significantly improves the consistency and safety of passage through complex intersections, making guidance decisions more aligned with the actual physical environment.

[0015] Meanwhile, this method endows the system with proactive risk avoidance capabilities by setting clear conflict resolution rules. For the high-risk scenario of intersections, the method specifically stipulates that when the traffic light recognition result is red, the remaining green light time is insufficient, or the light color cannot be confirmed, the system will forcibly generate a waiting instruction, and this instruction has higher priority than regular tactile paving navigation instructions. This means that even when the tactile paving recognition module is working normally and continuously outputting navigation guidance, once the safety red line is crossed, the system will immediately interrupt the original navigation logic, with waiting as the highest priority instruction output. This priority override mechanism solves the problem of ambiguous system decision-making logic when there are multimodal information conflicts in existing technologies, fundamentally eliminating the possibility of blind people accidentally entering the lane due to misjudgment or information conflicts, elevating traffic safety to the top priority, and reflecting a design philosophy centered on user personal safety. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is an overall flowchart of a multimodal perception-based guide method for the blind according to the present invention.

[0017] Figure 2 This is a flowchart of tactile paving identification in a multimodal perception-based guide method for the blind according to the present invention.

[0018] Figure 3 This is a hardware structure block diagram of a guide glasses for the blind based on multimodal perception according to the present invention. Detailed Implementation

[0019] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] "Multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0022] Furthermore, the terms indicating orientation, such as "up," "down," "left," "right," "upper end," "lower end," and "longitudinal," are all based on the posture and position of the device or equipment described in this solution during normal use.

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0024] Example 1 This embodiment discloses, as follows: Figure 1 The illustrated method for guiding the blind based on multimodal perception, applied to wearable devices, includes the following steps: Step S10: Real-time acquisition of environmental image data, environmental sensor data, and user walking data; Step S20: Perform scene classification on the environmental image data. Scene classification includes tactile paving recognition and intersection recognition. When the scene classification determines the current blind path area, blind path recognition is triggered; When the scene classification determines that the current environment is an intersection, traffic light recognition is triggered, including: The signal area of ​​traffic lights in environmental image data is located by identifying the model; Perform color segmentation on the signal area to identify the current light color; Record the color change pattern of the lights within the signal area and compare it with the preset value; Step S30: Combine and judge the traffic light recognition results with the tactile paving recognition results: When the light color is green and the remaining green time is greater than the preset safe passage threshold, and the result of the tactile paving recognition indicates that there is a passable path, a passage instruction is generated. When the light color is identified as red, green, or the remaining time is insufficient, or when it cannot be confirmed, a waiting instruction is generated and given priority over the blind path navigation instruction.

[0025] Step S10 involves the simultaneous acquisition of multimodal data, providing a foundation for subsequent fusion and recognition. Environmental image data is acquired via a camera, while environmental sensing data is acquired via a VL53L0X TOF module, which can simultaneously obtain obstacle distances and tactile paving heights. Walking data is acquired via a BMI088 gyroscope and accelerometer to determine the user's walking direction and posture. The collaborative work of these multiple sensors provides comprehensive environmental perception input for real-time navigation.

[0026] Step S20 uses the YOLOv8-nano model for scene classification and traffic light detection. The model, after INT8 quantization, is deployed on a SOC processor. Traffic light recognition incorporates HSV color segmentation: red light H=0-10 / 160-179, green light H=35-77, and yellow light H=20-35, along with a timing verification mechanism, effectively eliminating interference from billboards, vehicle taillights, etc. Through a scene classification triggering mechanism, traffic light recognition is activated only in intersection areas, avoiding false triggering in non-intersection scenarios and reducing system power consumption.

[0027] Step S30 achieves multi-source decision fusion to avoid misjudgment caused by a single identification result.

[0028] In this embodiment, tactile paving identification includes: The blind path area in environmental image data is delineated by the identification model; Multiple tactile paving protrusions within the tactile paving area are located using a geometric shape detection algorithm, and the distance between them and adjacent tactile paving protrusions is measured. The height data of the corresponding location within the tactile paving area is obtained from environmental sensor data, and target areas that meet the preset height range are selected. The results of spacing measurement and height data screening are cross-validated. When both meet the preset conditions, it is confirmed as a real tactile paving. The tactile paving area is identified by YOLOv8-nano, and the protrusions on the tactile paving bricks are detected by combining Hough circle transform. Isolated protrusion interference is eliminated through periodic verification. For the depth part, the height data of the tactile paving area is obtained by VL53L0X. Protrusions of 3-5mm are screened and confirmed as real tactile paving after cross-validation, which effectively solves the problem of flat floor tile patterns being misjudged as tactile paving.

[0029] In this embodiment, traffic light recognition and tactile paving recognition are executed in parallel. Tactile paving recognition runs continuously at a first frequency, while traffic light recognition is triggered only when an intersection area is detected, running at a second frequency higher than the first. Once the intersection area is left, traffic light recognition automatically enters standby mode, performing only tactile paving recognition. A dynamic frequency scheduling strategy balances perception accuracy and system power consumption. Tactile paving recognition runs continuously at 15-20fps to meet real-time walking requirements; traffic light recognition is increased to 25-30fps in intersection areas to ensure rapid response during light color switching. Traffic light recognition is disabled in non-intersection areas to reduce processor load and significantly extend device battery life.

[0030] In this embodiment, the step of scene classification to determine whether the current area is an intersection includes: The model is used to classify environmental image data into scenes and identify intersection features. When multiple consecutive frames of environmental image data are identified as intersection scenes, the entry into the intersection area is confirmed and traffic light recognition is activated.

[0031] The visual model learns intersection features through training samples, and the combination of the two greatly enhances the accuracy of intersection recognition. A continuous multi-frame verification mechanism further eliminates momentary false detections, ensuring reliable triggering of traffic light recognition.

[0032] In this embodiment, the change in light color is recorded through time-series analysis; the time-series analysis includes: Record the color of the same traffic light in multiple consecutive frames of environmental image data; When a light color change is detected, determine whether the switching interval conforms to the preset traffic light cycle range; Exclude light and shadow effects with a duration shorter than a preset threshold.

[0033] The temporal analysis component utilizes a TCN temporal convolutional network. By analyzing the patterns of light color changes through the TCN, transient interference such as vehicle headlight flickering and billboard transitions can be effectively filtered out. The preset period range is based on local traffic signal statistics, and the duration threshold is set to 0.5 seconds. Any light color change shorter than this is considered interference.

[0034] In this embodiment, an obstacle detection step is also included, which is performed in parallel with tactile paving recognition and traffic light recognition. Obtain distance data of obstacles ahead from environmental sensor data; Generate obstacle avoidance prompts of different levels based on distance data; When the distance to the obstacle is greater than or equal to the first preset threshold, a first-level prompt is generated; when the distance to the obstacle is less than the first preset threshold but greater than or equal to the second preset threshold, a second-level prompt is generated; when the distance to the obstacle is less than the second preset threshold, a third-level prompt is generated.

[0035] The obstacle detection process implements tiered safety warnings, with distance thresholds adaptively adjusted based on the user's walking speed. The first level of warning is a standard voice announcement; the second level warning uses a faster speech rate to enhance user attention; and the third level warning simultaneously triggers a vibration motor within the wearable device to provide tactile feedback, ensuring risk perception even in noisy environments.

[0036] In this embodiment, the output of navigation instructions adopts priority queue management; the priority queue management is from high to low as follows: emergency safety risks, important obstacles or traffic signals, navigation guidance, environmental prompts or status feedback; High-level information will be broadcast in a limited manner, and information of the same level will be sorted according to its time urgency. Set a time window to suppress repeated broadcasts of the same information. Priority queue management ensures that critical information is not overwhelmed. Emergency safety risks are prioritized, followed by important obstacles / traffic signals such as "Green light has 5 seconds left"; navigation guidance, such as "Go straight along the tactile paving," and environmental prompts have the lowest priority. The repetition suppression window is set to 3 seconds to avoid frequent notifications of the same obstacle interfering with the user.

[0037] Example 2 This embodiment is basically the same as Embodiment 1, and the similarities will not be repeated. The difference is that this embodiment also provides a guide glasses for the blind based on multimodal perception, including a processor and a memory. The memory stores a computer program, and the processor executes the guide glasses for the blind based on multimodal perception as described in Embodiment 1.

[0038] The glasses include a main body, on which the following hardware units are integrated: Image acquisition module; used to acquire environmental image data in real time. This camera has an automatic gain control function in low-light conditions to ensure image clarity in scenes such as dusk and cloudy days.

[0039] Environmental sensing module; used to acquire obstacle distance data and tactile paving height data. This module can simultaneously measure the absolute distance to obstacles ahead and the relative height of ground targets.

[0040] Walking data acquisition module: integrates a three-axis gyroscope and a three-axis accelerometer to collect the user's walking direction and gait. Location module: Used to obtain the user's geographical coordinates to assist in the judgment of intersection scenarios.

[0041] Embedded processing module: Employs a SOC processor and is responsible for the synchronous acquisition, fusion processing, and navigation command generation of multi-source data.

[0042] Audio interaction module: used for voice wake-up and remote assistance calls; Tactile feedback module: Two sets of miniature vibration motors are embedded in the inner side of the left and right temples respectively, and the vibration can be controlled independently to provide directional tactile cues.

[0043] Physical trigger module: A large raised button is set on the outside of the right temple, which supports blind touch recognition. Press and hold for 2 seconds to trigger the remote assistance function.

[0044] When the blind guide glasses system of this embodiment is working: the image acquisition module collects environmental image data in real time, the environmental sensing module collects environmental sensing data in real time, and the walking data acquisition module collects walking data in real time. All three are transmitted synchronously to the embedded processing module. The embedded processing module fuses the received multi-source data and performs tasks such as tactile paving recognition, traffic light recognition, and obstacle detection as described in Example 1. Based on the recognition results, result guidance instructions are generated, which are then broadcast via voice through the audio interaction module and provide directional vibration prompts through the tactile feedback module. When a user triggers remote assistance via a physical trigger module or voice wake-up, the wireless communication module establishes a connection with the remote assistance end, transmits real-time video, audio, and environmental sensor data, and receives and executes instructions returned by the remote end.

[0045] Example 3 This embodiment is basically the same as Embodiment 1, and the similarities will not be repeated. The difference is that this embodiment also provides a storage medium on which a computer program is stored. When the program is executed by the processor, it implements the multimodal perception-based guide method for the blind as in Embodiment 1.

[0046] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal perception-based guidance method for the blind, applied to wearable devices, characterized in that, Includes the following steps: Real-time acquisition of environmental image data, environmental sensor data, and walking data; Scene classification is performed on the environmental image data, including tactile paving identification and intersection identification; When the scene classification determines the current blind path area, blind path recognition is triggered; When the scene classification determines that the current environment is an intersection, traffic light recognition is triggered, including: The signal area of ​​the traffic lights in the environmental image data is located by identifying the model; The signal region is segmented by color to identify the current light color; Record the color change pattern of the lights within the signal area and compare it with a preset value; The traffic light recognition result and the tactile paving recognition result are then fused for judgment. When the light color is identified as green and the remaining green time is greater than the preset safe passage threshold, and the result of the tactile paving identification indicates that there is a passable path, a passage instruction is generated; When the light color is identified as red, green, or the remaining time is insufficient, or when it cannot be confirmed, a waiting instruction is generated and output with priority over the blind path navigation instruction.

2. The multimodal perception-based guide method for the blind according to claim 1, characterized in that, The tactile paving identification includes: The blind path area in the environmental image data is delineated by the identification model; Multiple tactile paving protrusions within the tactile paving area are located using a geometric shape detection algorithm, and the distance between them and adjacent tactile paving protrusions is measured. The height data of the corresponding location within the blind path area is obtained from the environmental sensing data, and target areas that meet the preset height range are filtered out. The results of the spacing measurement and the height data screening are cross-validated. When both meet the preset conditions, it is confirmed as a genuine tactile paving path.

3. The multimodal perception-based guide method for the blind according to claim 2, characterized in that, The traffic light recognition and the tactile paving recognition are executed in parallel; the tactile paving recognition is executed continuously at a first frequency, and the traffic light recognition is triggered only when the intersection area is detected at a second frequency, which is higher than the first frequency; when leaving the intersection area, the traffic light recognition automatically enters standby mode and only executes the tactile paving recognition.

4. The multimodal perception-based guide method for the blind according to claim 1, characterized in that, The steps for classifying and determining whether the current location is an intersection area include: The environmental image data is classified into scenes using a recognition model to identify intersection features; When multiple consecutive frames of the environmental image data are identified as an intersection scene, the entry into the intersection area is confirmed and the traffic light recognition is activated.

5. The multimodal perception-based guide method for the blind according to claim 1, characterized in that, The changes in light color are recorded through time-series analysis; the time-series analysis includes: Record the color of the same traffic light in multiple consecutive frames of environmental image data; When a light color change is detected, determine whether the switching interval conforms to the preset traffic light cycle range; Exclude light and shadow effects with a duration shorter than a preset threshold.

6. The multimodal perception-based guide method for the blind according to claim 1, characterized in that, It also includes an obstacle detection step, which is performed in parallel with the tactile paving recognition and traffic light recognition: Obtain distance data of obstacles ahead from the environmental sensing data; Different levels of obstacle avoidance prompts are generated based on the distance data; When the distance to the obstacle is greater than or equal to the first preset threshold, a first-level prompt is generated; When the distance to the obstacle is less than the first preset threshold and greater than or equal to the second preset threshold, a second-level prompt is generated; When the distance to the obstacle is less than the second preset threshold, a third-level prompt is generated.

7. The multimodal perception-based guide method for the blind according to claim 1, characterized in that, The output of navigation commands is managed by a priority queue; the priority queue management is in the following order from high to low: emergency safety risks, important obstacles or traffic signals, navigation guidance, environmental prompts or status feedback; High-level information will be broadcast in a limited manner, and information of the same level will be sorted according to its time urgency. Set a time window to suppress repeated broadcasts of the same information.

8. A pair of guide glasses for the blind based on multimodal perception, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the multimodal perception-based guide method for the blind as described in any one of claims 1-8.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multimodal perception-based guide method for the blind as described in any one of claims 1 to 8.

Images