Computer-implemented method, wearable device, non-transitory computer-readable storage medium, computer program and system for assisting movement of a visually impaired user

Abstract

In a first aspect of the invention, a computer implemented method for assisting a visually impaired user in moving with the help of a wearable device is claimed, the method comprising the steps of: S1 - acquiring data from the environment of the visually impaired user, S2 - fusing the acquired data, creating, updating iteratively a live map, S3 - determining, updating iteratively and storing at least one navigation path and associated navigation guidance instructions for the visually impaired user to navigate from the current position of the visually impaired user to a point of interest, iteratively selecting a preferred navigation path from the at least one navigation path and iteratively sending the preferred navigation path and associated navigation guidance instructions to the visually impaired user. S4 - guiding the visually impaired user along the preferred navigation path by using a guidance mode for delivering each associated navigation guidance instruction comprising tactile and / or audible cues. The invention details how steps S2, S3 and S4 are performed and also details the content of the live map. The live map comprises data from sensors about objects and living beings, relationships between objects and living beings and different types of areas: walkable areas, non-walkable areas and conditionally walkable areas. In a second aspect of the invention, a wearable device for assisting a visually impaired user in moving is claimed, the device being configured to apply the steps of the method.

Description

[0001] Invention Field This invention relates to the field of mobility assistance for visually impaired individuals. Specifically, this invention relates to a method and wearable device for assisting visually impaired individuals in their movement, and throughout this invention, visually impaired individuals will be referred to as visually impaired users. Throughout this invention, the term "visual impairment" will encompass moderate to severe impairment (blindness). Background of the Invention Research indicates that 36 million people were affected by severe visual impairment in 2015, while 216.6 million suffered from moderate to severe visual impairment. As these numbers rise, people are becoming increasingly aware of their needs, and solutions designed to assist the visually impaired have emerged. For example, cities are becoming increasingly accessible to the blind using classic navigation methods such as canes or guide dogs. Technological solutions have also developed and are beginning to be accepted by the blind and visually impaired. Solutions proposed by companies such as OrCam Inc. and Microsoft Inc. have been adopted to varying degrees. However, despite technological advancements, the most commonly used solution for the visually impaired remains the cane.

[0003] Technological solutions for assisting visually impaired individuals are not new. Early efforts can be attributed to Paul Bach-Y-Ritta et al. in US3594823A [1]. In this study, visual signals were converted into tactile feedback on the back of the blind person.

[0004] Generally, technological solutions for assisting visually impaired individuals fall into one or more of the following categories: - Sensory substitution—using available senses to represent information that is not normally received through those senses. Several other efforts have been made in the following years to use other senses, such as touch or sound, to replace visual information. - By using sensory substitution, a path is generated for visually impaired individuals, and the path is communicated to them / him. - Methods for locating visually impaired individuals in their environment and / or locating various objects that can be considered obstacles and / or target destinations. - A method to communicate information about the environment, paths, and obstacles to visually impaired individuals and to receive feedback from them.

[0005] Different solutions involve one or more of the categories described above.

[0006] US9915545[2], published on July 26, 2015, discloses a method for providing direction to a blind person using a smart device. The method includes: detecting image data corresponding to the surrounding environment of the smart device and positioning data corresponding to the positioning of the smart device by means of at least two sensors and in response to the selection of a search mode of the smart device; receiving a desired object or a desired location by means of an input device; determining the initial position of the smart device by means of a processor based on the image data, positioning data and map data stored in the memory of the smart device; and providing direction to the desired object by means of an output device of the smart device based on the initial position of the smart device and the map data.

[0007] US9629774[3], published on July 16, 2015, discloses a smart necklace comprising a body defining at least one cavity and having a neck and first and second sides. The necklace includes a pair of stereo cameras configured to detect image data including depth information corresponding to the surrounding environment of the smart necklace. The necklace also includes a positioning sensor configured to detect positioning data corresponding to the positioning of the smart necklace. The necklace includes a non-transitory memory located in the at least one cavity and configured to store map data and object data. The smart necklace also includes a processor located in the at least one cavity and coupled to the pair of stereo cameras, the positioning sensor, and the non-transitory memory. The processor is configured to determine output data based on the image data, positioning data, map data, and object data.

[0008] In an academic article published on November 15, 2016 [4], “Wearable Indoor NavigationSystem with Context Based Decision Making for Visually Impaired” [4], the authors Xiaochen Zhang et al. proposed a wearable indoor navigation system for visually impaired individuals. The system uses Simultaneous Localisationn and Mapping (SLAM) and semantic path planning for localization and navigation, integrating multiple sensors and feedback devices, such as RGB-D cameras, inertial measurement units (IMUs), and network cameras. The system applies an RGB-D-based visual range algorithm to estimate the user’s position and orientation, and applies an inertial measurement unit (IMU) to improve orientation errors. Key landmarks such as room numbers and corridor corners are detected by network cameras and RGB-D cameras and matched with a digital floor map to locate the user. Paths and motion guidance are generated to guide the user to the desired destination. The article proposes a method to improve the fit between rigid commands and the best machine decisions for humans in the context-based decision-making mechanism for path planning to address user confusion caused by erroneous observations.

[0009] US2016 / 0033280A1[5] discloses a smart earphone worn on a user's ear, the smart earphone including a processor connected to an inertial measurement unit (IMU), a global positioning system (GPS) unit, and at least one camera coupled to the smart earphone. The IMU is configured to detect inertial measurement data corresponding to the positioning, speed, or acceleration of the smart earphone, the GPS unit is configured to detect location data corresponding to the position of the smart earphone, and the at least one camera is coupled to the smart earphone and configured to detect image data corresponding to the surrounding environment of the smart guidance device. The earphone also includes a processor connected to the IMU, the GPS unit, and the at least one camera. The processor is configured to identify objects in the surrounding environment. This is done by analyzing image data based on stored object data and at least one of inertial measurement data or location data. The processor may determine a destination based on a determined desired event or action. The processor may determine a navigation path for navigating the smart guidance device to the destination based on the determined destination, image data, inertial measurement data, or location data. The processor may determine output data based on the determined navigation path.

[0010] US2016 / 0033280A1 [5] discloses that the device does not have a specific adaptation for visually impaired individuals.

[0011] Those skilled in the art will understand that all devices providing mobility assistance to visually impaired individuals must primarily provide their users with information from the environment, which is inaccessible to them through their visual acuity, as visually impaired individuals cannot see or have very little vision. In other words, of all the sensors used in devices providing navigation assistance to visually impaired individuals, cameras are the most important.

[0012] Taking this important aspect into consideration, US2016 / 0033280A1 has already achieved [something related to ear protection], such as [something related to ear protection]. Figure 8B and Figure 9 The disclosure of a single earphone shown in B.

[0013] In cases where visually impaired individuals use the headphones described in US2016 / 0033280A1, because the headphones are positioned behind the ears, the camera's field of view is partially obstructed by the visually impaired individual's head, thus the headphones cannot provide an image that fully covers the environment. For example, if the headphones are placed behind the left ear, a portion of the right visual field of view will be obstructed by the user's head.

[0014] US2016 / 0033280A1 contains an incomplete disclosure of alternative locations for additional cameras, but does not detail the corresponding implementations that enable a person skilled in the art to reproduce the alternative locations of the additional cameras using common sense.

[0015] Disadvantages of existing technology For sensory substitution, known solutions essentially suggest an inappropriate substitution for the visual bandwidth, which is known to be far greater than that of hearing and / or touch, either providing too little information or too much unnecessary information, causing confusion or annoyance to the user.

[0016] Regarding methods for generating maps and for generating paths for visually impaired individuals and communicating those paths to them, existing navigation / GPS-based methods provide general paths but do nothing to avoid obstacles or creatures.

[0017] Regarding methods for locating visually impaired individuals in their environment and / or locating various objects that can be considered obstacles and / or target destinations, methods such as Microsoft Seeing AI or OrCam only identify objects present within their limited field of view without providing sufficient information about how to reach the target destination. Generally, known solutions have little or no chance of storing the positions of objects detected by sensors, and therefore little or no chance of recognizing previously detected objects. Known solutions lack information that could provide complete, useful information about objects, such as the size of the object and other physical and chemical characteristics, if the chair is occupied by someone else. This necessitates the visually impaired individual touching the object to perceive these physical and chemical characteristics, which is unhygienic, potentially dangerous, and could be too time-consuming.

[0018] Regarding methods for communicating information about the environment, paths, or obstacles to visually impaired individuals and receiving feedback from them, most current solutions are too slow to communicate information and / or receive feedback from them, making it difficult and delaying to modify the initial path.

[0019] For example, in the embodiment of US 9,629,774 [3], although the smart necklace can identify stairs, leaving a restroom or an empty seat, the necklace is not configured to provide more in-depth information, such as the characteristics of the object: orientation, where the latch or door handle is, or whether the empty seat is dirty.

[0020] US9,629 774 [3] teaches a smart necklace to transmit a limited number of types of commands to visually impaired users: different turning angles, such as 45-degree turns, 90-degree turns, left turns, and right turns. Using this method of communication, US9,629 774 [3] is rigid when compared with the natural path generation of non-visually impaired individuals.

[0021] US 9,629,774 [3] teaches a map that contains only the locations of various objects and no other features.

[0022] US 9,915,545 [2] teaches a method for providing orientation to blind users of electronic devices, the orientation being from the current location of the electronic device to the location of a desired object. In this method, there is no distinction between the types of regions on which paths are generated, no relationships are established between various objects, some important aspects of objects and organisms are omitted, such as the detection of the organism's emotional state or the cleanliness of the surface, and the map content is reduced to information from the sensor's field of view.

[0023] Problems solved by the invention The problem this invention aims to solve is to provide a method suitable for assisting visually impaired users in their movement, allowing them to navigate indoors and outdoors in a manner more similar to that used for non-visually impaired users. Specifically, the problem this invention aims to solve is: - Provides a more accurate and detailed representation of the user's environment, including providing a more accurate and detailed relationship between objects and organisms, with the aim of providing safer navigation for visually impaired users and more specific navigation targets; - Provides more accurate and detailed generation of navigation routes; - Provides better guidance for visually impaired users along navigation paths in terms of accuracy and safety; - Provides better guidance for visually impaired users along navigation paths in a more comfortable way, and gives them the same possibility of making decisions as non-visually impaired users. Summary of the Invention

[0024] In a first aspect of the invention, a method is provided that is suitable for assisting a visually impaired user in moving by means of a wearable device for assisting movement, the method comprising the following steps: S1 The sensor unit 2 of the wearable device acquires data from the environment of the visually impaired user, senses the field of view, and sends the acquired data to the sensory fusion subunit 30 of the processing and control unit of the wearable device. S2 The acquired data is fused by the sensory fusion subunit, and the fused data is sent to the real-time map subunit of the processing and control unit. The live map is created, repeatedly updated, and stored through its sub-units, and includes the following data: 1. A real-time map determination based on fused data received from the sensory fusion subunit, the real-time map determination including: - The position and orientation of the sensory unit, - Multiple objects, - Multiple organisms, 2. Based on the real-time map generated from multiple relationships between the multiple objects and / or the multiple organisms received from the relationship manager subunit of the processing and control unit, a determination is made. 3. Based on the real-time map generated from the free region, the free region is defined as the entire ensemble of the area on the ground not occupied by the multiple objects and multiple organisms Ln. Free region A includes: - A walkable area that meets a fixed set of predetermined walkable area requirements. - A conditionally walkable area that satisfies the set of fixed, predetermined walkable area requirements and at least one predictable conditionally walkable area requirement. S3 Automatically or in response to a request from a visually impaired user, the navigation manager subunit of the processing and control unit determines, iteratively updates, and stores at least one navigation path and associated navigation guidance instructions for the visually impaired user to navigate from the current location of the sensory unit to a point of interest selected from the plurality of objects and / or multiple organisms. Automatically or in response to a request from a visually impaired user, a preferred navigation path is repeatedly selected from the at least one navigation path, the preferred navigation path satisfying at least two navigation path requirements: i) Passing through walkable areas and / or through conditionally walkable areas, and ii) Meets a set of safety requirements, which includes a collision-free requirement and a non-aggressive requirement.

[0025] In this system, any request from a visually impaired user is made using a tactile or audio device via a user command interface. This request is received by the navigation manager subunit via the user command interface manager subunit of the processing and control unit. In addition, feedback is transmitted from the navigation manager subunit to the processing and control unit. i) Preferred navigation path, and ii) Associated navigation guidance instructions, Specifically, when the preferred navigation path SP passes through a conditionally traversable area, the navigation manager subunit sends a navigation guidance instruction to the feedback manager subunit that corresponds to the requirement of at least one predictable conditionally traversable area. S4 Along a preferred navigation path, guidance is provided to the visually impaired user by the feedback manager subunit using a guidance mode for transmitting each associated navigation guidance instruction. Each navigation guidance instruction includes tactile and / or auditory cues sent by the feedback manager subunit to a feedback unit in the processing and control unit, the feedback unit comprising: - Configured as a haptic feedback actuator placed on the head of a visually impaired user, and / or - Configured as an auditory feedback actuator to be placed in one or both ears of a visually impaired user. The guidance mode for each associated navigation guidance instruction is selected by the visually impaired user via a user command interface and through a user command received by the feedback manager subunit via the user command interface manager subunit.

[0026] In a second aspect of the invention, a wearable device 1 is provided for assisting a visually impaired user in movement, the wearable device comprising: - A sensory unit configured to be placed on the head of a visually impaired user, the sensory unit including basic sensors: - Camera, - Depth sensor, - Inertial Measurement Unit - Sound positioning sensor - A processing and control unit, the processing and control unit comprising: - Sensory fusion subunit, the sensory fusion subunit comprising: - Positioning module, - Walkable area detection module. - Directed calculation module, - Sound direction localization module, - Sound classification module, - Object 2D feature extraction module - Object 3D feature fusion module - Object sound feature fusion module - Live map sub-cell, - Relationship Manager subunit, - Navigation Manager subunit, - User command interface manager subunit - Feedback Manager subunit, - Sound representation subunit, - A feedback unit configured to be placed on the head of a visually impaired user, the feedback unit comprising: - Multiple haptic feedback actuators, said multiple haptic feedback actuators including: - Left haptic feedback actuator, - Right haptic feedback actuator, - Central haptic feedback actuator, - Multiple auditory feedback actuators, said multiple auditory feedback actuators including: - Left auditory feedback actuator, - Right auditory feedback actuator, - A user command interface configured to be placed on the head of a visually impaired user, the user command interface including: - Multiple user-commanded haptic devices, - Multiple user-commanded audio devices, - Power storage unit, - Memory, and - An electronic communication device that connects the sensing unit, processing and control unit, feedback unit, user command interface, power storage unit, and memory via a communication protocol. The wearable device is configured to apply the steps of a method according to any embodiment of the present invention (including combinations of the embodiments).

[0027] In a third aspect of the invention, a non-transitory computer-readable storage medium is provided, the computer program comprising instructions executable by one or more processors, which, upon execution, cause the one or more processors to perform operations of any preferred embodiment (including combinations thereof) of a computer-implemented method suitable for assisting a visually impaired user in moving.

[0028] In a fourth aspect of the invention, a computer program product is provided, the computer program product including instructions that, when the program is executed by a wearable device according to any embodiment of the invention, cause the wearable device to perform steps of a computer-implemented method suitable for assisting a visually impaired user to move in any preferred embodiment (including combinations thereof).

[0029] In a fifth aspect of the invention, a system is provided comprising one or more processors and one or more non-transitory computer-readable storage devices, the non-transitory computer-readable storage devices including a storage medium containing instructions executable by the one or more processors to perform operations of a computer-implemented method suitable for assisting a visually impaired user in any preferred embodiment (including combinations thereof).

[0030] According to one example implementation, a computer-implemented method includes: acquiring data from the environment of a visually impaired user, including sensing from a field of view by a sensory unit of a wearable device; sending the acquired data to a sensory fusion subunit of a processing and control unit of the wearable device; fusing the acquired data by the sensory fusion subunit; sending the fused data to a live map subunit of the processing and control unit; and creating, repeatedly updating, and storing a live map by the live map subunit. The real-time map includes: one or more real-time map decisions generated based on fused data received from the sensory fusion subunit at the processing and control unit, the one or more real-time map decisions including the position and orientation of the sensory unit, multiple objects, and multiple organisms; one or more real-time map decisions generated based on multiple relationships between multiple objects or multiple organisms or between multiple objects and multiple organisms received from the relationship manager subunit of the processing and control unit; and one or more real-time map decisions generated based on a free region, the free region being defined as the entire area on the ground not occupied by the multiple objects and the multiple organisms, the free region including a walkable region that satisfies a set of fixed, predetermined walkable region requirements, and a conditional walkable region that satisfies the set of fixed, predetermined walkable region requirements and at least one predictable conditional walkable region requirement.The method includes: automatically or in response to a first request from a visually impaired user, determining, iteratively updating, and storing, via a navigation manager subunit of a processing and control unit, at least one navigation path and associated navigation guidance instructions for the visually impaired user to navigate from the current location of a sensory unit to a point of interest selected from multiple objects or multiple organisms, or multiple objects and multiple organisms; automatically or in response to a second request from a visually impaired user, iteratively selecting from the at least one navigation path a preferred navigation path that (i) traverses a walkable area or passes through a conditionally walkable area or passes through both a walkable area and a conditionally walkable area and (ii) satisfies a set of safety requirements including collision-free requirements and non-aggression requirements, wherein any input from the visually impaired user is made using a tactile or audio device via a user command interface, the request being received by the navigation manager subunit via a user command interface manager subunit of the processing and control unit; and transmitting the preferred navigation path and associated navigation guidance instructions via the navigation manager subunit to a feedback manager subunit of the processing and control unit. When the preferred navigation path passes through a conditionally walkable area, the navigation manager subunit sends a navigation guidance instruction corresponding to the at least one predictable conditionally walkable area requirement to the feedback manager subunit. Along the preferred navigation path, guidance is provided to the visually impaired user via the feedback manager subunit using a guidance mode for transmitting each associated navigation guidance instruction. Each navigation instruction includes a tactile or auditory cue sent by the feedback manager subunit to a feedback unit of the processing and control unit. The feedback unit includes a tactile feedback actuator configured to be placed on the head of the visually impaired user, or an auditory feedback actuator configured to be placed in one or both ears of the visually impaired user, or a tactile feedback actuator configured to be placed on the head of the visually impaired user and an auditory feedback actuator configured to be placed in one or both ears of the visually impaired user. The guidance mode for each associated navigation guidance instruction is selected by the visually impaired user via a user command interface and through a user command received by the feedback manager subunit via the user command interface manager subunit.

[0031] Some implementations include one or more of the following features. For example, a method includes: creating and updating a real-time map, including repeatedly determining the location and orientation of a sensory unit, the location, orientation, and features of multiple objects and multiple organisms based on fused data received from a sensory fusion subunit, and repeatedly sending the created and updated real-time map to a localization module of the sensory fusion subunit; repeatedly generating and updating multiple relationships between multiple objects or between multiple organisms or between multiple objects and multiple organisms through a relationship manager subunit based on data obtained from the real-time map, including applying a set of predetermined relationship requirements, and repeatedly sending the updated multiple relationships to the real-time map; utilizing data received from the sensory unit and data from the real-time map... The localization algorithm applied to the image data repeatedly locates the position and orientation of the sensory unit relative to multiple objects and multiple organisms on the real-world map through the localization module, and repeatedly sends the localization data of the sensory unit's position and orientation to the walkable area detection module of the sensory fusion subunit. Based on the data received from the sensory unit, the data received from the localization module, a set of fixed, predetermined walkable area requirements calculated and stored in memory, and at least one predictable conditional walkable area requirement, the walkable area detection module repeatedly determines the free area, repeatedly sends the updated free area to the real-world map, and repeatedly stores the updated real-world map in memory. The real-world map is updated by the sensory fusion subunit using the Simultaneous Localization and Mapping (SLAM) algorithm. The method includes: sending information requested by a visually impaired user to a sound representation subunit of a processing and control unit regarding the selection of at least one object or at least one organism from a plurality of objects; extracting information about the selected at least one specific object or at least one specific organism from a live map via the sound representation subunit of the processing and control unit; representing the extracted information as corresponding spatialized sound; transmitting the spatialized sound to the visually impaired user via a feedback unit; and having the visually impaired user select a point of interest from the plurality of objects or from the plurality of organisms, and transmitting the corresponding selection request to a navigation manager subunit. The method also includes: determining a wandering path and associated navigation guidance instructions for the visually impaired user via a navigation manager, and sending the wandering path and associated navigation guidance instructions to a feedback processor subunit. Tactile cues vary in duration, periodicity, intensity, or frequency of vibration according to a predetermined preferred navigation path complexity criterion, and audio cues vary in frequency, duration, repetition intensity, or 3D spatial virtualization according to a predetermined preferred navigation path complexity criterion. A three-dimensional walkable tunnel is defined as a virtual tunnel with a predetermined cross-section, which uses the preferred navigation path as the horizontal longitudinal axis, and wherein the guidance mode also includes specific tactile cues sent to the visually impaired user as the user approaches the virtual wall of the walkable tunnel.The preferred navigation path is divided into predetermined segments defined by multiple mile markers, and the guidance mode includes tactile or auditory cues indicating the location of at least one subsequent mile marker, providing associated navigation guidance instructions to the visually impaired user from the current mile marker to the subsequent mile markers, and the length of the predetermined segments varies according to the complexity and length of the preferred navigation path. The guidance mode includes tactile or auditory cues, or tactile and auditory cues, indicating directions on the preferred navigation path. Directions on the preferred navigation path are defined by a line defined by the intersection of the origin of the sensory unit and a circle with a predetermined radius at the sensory unit's location, and the auditory cues indicating directions on the preferred navigation path originate from a spatialized sound source located at a predetermined first distance relative to the sensory unit. The auditory cue is a spatialized sound originating from a spatialized sound source that virtually travels from the position of the sensory unit along a predetermined second distance on a preferred navigation path until the spatialized sound source reaches the end of the predetermined second distance and returns to the position of the sensory unit.

[0032] In another conventional embodiment, a wearable device for assisting visually impaired users in movement includes a sensory unit configured to be placed on the head of the visually impaired user. The sensory unit includes a camera, a depth sensor, an inertial measurement unit, and a sound localization sensor. The device includes a processing and control unit, which includes a sensory fusion subunit. The sensory fusion subunit includes a localization module, a walkable area detection module, an orientation calculation module, a sound direction localization module, a sound classification module, an object 2D feature extraction module, an object 3D feature fusion module, and an object sound feature fusion module. The device includes a real-time map subunit, a relationship manager subunit, a navigation manager subunit, a user command interface manager subunit, a feedback manager subunit, and a sound representation subunit. The device includes a feedback unit configured to be placed on the head of the visually impaired user. The feedback unit includes: a plurality of tactile feedback actuators, including a left tactile feedback actuator, a right tactile feedback actuator, and a central tactile feedback actuator; and a plurality of auditory feedback actuators, including a left auditory feedback actuator and a right auditory feedback actuator. The device includes a user command interface configured to be placed on the head of a visually impaired user, the user command interface including multiple user command tactile devices and multiple user command audio devices. The device includes a power storage unit, a memory, a sensory unit, a processing and control unit, a feedback unit, the user command interface, and electronic communication components between the power storage unit and the memory.

[0033] Example implementations include one or more of the following features. The sensing unit includes at least one additional sensor, which is derived from a GPS sensor configured to determine the absolute position of the sensing unit or a temperature sensor configured to determine the temperature of objects and organisms. The sensing fusion subunit includes: a relative-to-absolute conversion module configured to fuse data from an object sound feature fusion module with data relating to the absolute position of the sensing unit; and an object temperature feature fusion module configured to fuse data from the object sound feature fusion module with data relating to the temperature of objects and organisms and send the fused data to a live map subunit.

[0034] Other example implementations include a system comprising: one or more processors; and one or more non-transitory machine-readable storage devices storing instructions executable by the one or more processors to perform operations corresponding to the disclosed methods; or a non-transitory computer storage medium encoded with a computer program including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations corresponding to the disclosed methods.

[0035] Any feature in one aspect of the invention may also be applied to other aspects of the invention in any suitable combination. In particular, method features may also be applied to device features, and vice versa.

[0036] Where applicable, means plus functional features can also be described in terms of their corresponding structural aspects, such as a properly programmed processor and associated memory.

[0037] Specific combinations of the various features of the present invention can be implemented and / or provided and / or used independently.

[0038] Advantages of the invention The main advantages of this invention are as follows: - It provides safer navigation for visually impaired users and more accurate touch on objects and creatures in everyday life for the following reasons: - To represent the user's environment more accurately and in more detail; - Generate navigation paths more accurately and in greater detail; - To guide users along the navigation path more accurately and safely; - It provides a navigation experience that is closer to the user experience of those without visual impairment in terms of comfort and decision-making possibilities. Attached Figure Description

[0039] Figure 1 Schematic representation of the method and wearable device 1 according to the present invention.

[0040] Figure 2 is an oblique view showing the schematic positioning of the components of the wearable device 1 in two preferred embodiments.

[0041] Figure 2A Two-component wearable device.

[0042] Figure 2B Wearable devices with a single component.

[0043] Figure 3 A schematic representation of the field of view 20 of the basic sensor.

[0044] Figure 4 A schematic representation of the content of the real and live map 310.

[0045] Figure 5 A schematic representation of the creation of a live map 310 when the wearable device 1 uses only basic sensors.

[0046] Figure 6 An oblique view of one of the preferred embodiments shows the headset component 11 as viewed from behind the head of a visually impaired user, and details the positioning of the components of the feedback unit 4 and the user command interface 5.

[0047] Figure 7 Figure 6 The front view of the preferred embodiment shows the head-mounted device component 11 as viewed from the forehead of a visually impaired user, and details the positioning of the four basic sensors, as well as the positioning of the components of the user command interface 5.

[0048] Figure 8A A schematic representation of the guidance mode using a walkable tunnel T.

[0049] Figure 8B The oblique view of the embodiment illustrates the guidance mode by using a walkable tunnel T.

[0050] Figure 9 The oblique view of the embodiment shows the guidance mode using mile marker 93.

[0051] Figure 10 The illustrative representation of the embodiment shows a guidance mode using a walkable tunnel T and a mile marker 93.

[0052] Figure 11 The illustrative representation of the embodiment illustrates the use of the following guidance mode.

[0053] Figure 11 A. Tactile cue.

[0054] Figure 11 B. Auditory cues.

[0055] Figure 12 The illustrative representation of the embodiment shows a guidance pattern using spatialized sound originating from a spatialized sound source S, which virtually travels along a preferred navigation path SP.

[0056] Figure 13 A schematic representation of an embodiment corresponding to a wearable device 1 including two additional sensors 25 and 26.

[0057] Figures 14 to 17 Method example – Example number 1.

[0058] Figure 14 A top-down view of the actual scene.

[0059] Figure 15 A slanted view of a real-world scene.

[0060] Figure 16 Details from auxiliary path 912.

[0061] Figure 17 Details – Doorbell 841 replaces door 84 as the focus PI.

[0062] Figures 18 to 28 Method example—Example number 2. Window 85 is represented symbolically.

[0063] Figure 18 Example No. 2-1 – Schematic representation of step S.3-0.2 of the method, wherein spatialized sound S86 is perceived as being at the same location as window 85.

[0064] Figure 19A Example number 2-2—a schematic representation of step S.3-0.2 of the method, wherein the spatialized sound S86 is an exaggerated or “scaled” representation of the sound relative to the window 85 in the elevation spatial dimension.

[0065] Figure 19B Example No. 2-3 – Schematic representation of step S.3-0.2 of the method, where spatialized sound S86 is an exaggerated or “scaled” representation of window 85 in the azimuth spatial dimension relative to the potential point of interest PPI.

[0066] Figure 20Example No. 2-4 – Schematic representation of step S.3-0.2 of the method, wherein two windows 85-1 and 85-2, which are disposed at unequal distances from the visually impaired user, are represented by virtualized sounds S86f-1 and correspondingly S86f-2 with different frequency characteristics according to the distance of windows 85-1 and 85-2 from the visually impaired user.

[0067] Figure 21 Example No. 2-5 – A schematic representation of step S.3-0.2 of the method, wherein the edges of window 85 are represented by two spatialized sounds S86P-E1 and S86P-E2 with different coding features, depending on the distance of each end relative to the visually impaired user.

[0068] Figure 22 Example number 2-6—a schematic representation of step S.3-0.2 of the method, shows the representation of the dimensions S86P and S86L of window 85.

[0069] Figure 23 Example No. 2-7 – A schematic representation of step S.3-0.2 of the method, wherein the shape of window 85 is represented by two spatialized point sound sources S86P1 and S86P2.

[0070] Figure 24 Example No. 2-8 – a schematic representation of step S.3-0.2 of the method, wherein the shape of window 85 is represented by two spatialized sound sources S86f1 and S86f2 with different frequencies in terms of azimuth.

[0071] Figure 25 Example No. 2-9 – A schematic representation of step S.3-0.2 of the method, wherein the shape of window 25 is represented by a single spatialized sound source S86, which virtually travels along the outline of window 85 from the starting point t0 until it returns to the starting point t0.

[0072] Figure 26 Example No. 2-10 – A schematic representation of step S.3-0.2 of the method, wherein the shape of window 25 is represented by a single spatialized sound source S86 that virtually travels according to an angled pattern.

[0073] Figure 27 Example No. 2-11 – A schematic representation of step S.3-0.2 of the method, showing representations of two windows 85-1 and 85-2 at different distances relative to the visually impaired user.

[0074] Figure 28, Example No. 2-12—a schematic representation of step S.3-0.2 of the method, shows the sound representation of window 85 by two spatialized sounds S861, S862, which start simultaneously and travel in an angular pattern within the inner frame of window 85.

[0075] List of reference numerals This list includes reference numerals for parts, parameters, or standards that appear in the specification and / or drawings. This list was created to facilitate reading of the invention.

[0076] 1. Wearable devices 11 Head-mounted device components 12 Wearable components 12 Wrist components – not shown graphically 13 Handheld components – not shown graphically 2 Sensory units Basic sensors: 21, 22, 23, 24 20 Field of view of basic sensors 21 cameras 22 Depth Sensors 21-22 Cameras and depth sensors – not shown graphically 23 Inertial Measurement Unit 24 Sound positioning sensors 21-22 Cameras and depth sensors – not shown graphically Additional sensors 25, 26 25 Global Positioning Sensors 26 Temperature Sensor 3. Processing and Control Unit 30 Sensory Fusion Subunit 301 Positioning Module 302 Walkable Area Detection Module 303 Directional Calculation Module 304 Sound Direction Positioning Module 305 Sound Classification Module 306 Object 2D Feature Extraction Module 307 Object 3D Feature Fusion Module 308 Object Sound Feature Fusion Module 309-1 Relative and Absolute Conversion Module 309-2 Object Temperature Feature Fusion Module 31 Real-time Map Sub-unit 310 Live Map 32 Relationship Manager Subunit 33 Navigation Manager Subunit 34 User Command Interface Manager Subunit 35 Feedback Manager Subunit 36 Sound representation subunits 4 Feedback Unit 41 Haptic Feedback Actuator 411 Left Feedback Actuator 412 Right Feedback Actuator 413 Center Feedback Actuator 42 Auditory feedback actuator 421 Left auditory feedback actuator 422 Right auditory feedback actuator 5 User command interface 51 User-commanded tactile devices 52 User-commanded audio devices 6. Power storage unit – not shown graphically M memory – not shown graphically 7. Communication Unit – Not shown graphically Contents of PES Map 310 Real-time map determination based on fused data received from sensory fusion subunit 30 - Position and orientation of sensory unit 2 - On multiple objects - Ln Multiple organisms - not represented graphically Based on the real-time map determination from the data received from the relation manager subunit 32: - Multiple relationships Rn = relationships between multiple objects On and / or multiple organisms Ln - not represented graphically Free Zone A: - Walkable area in WA - CWA Conditional Walkable Area - NA Unwalkable Area Minimum requirements - For regions: - A fixed set of predetermined walkable area requirements - At least one predictable conditionally walkable area requirement - Regarding relationships: - The categories of pre-defined relationship requirements include: - A pre-determined parent-child relationship, and - Predetermined conditional relationships - For navigation path Pn: - Two navigation path requirements - Safety requirements - No collision requirement - No offensive requirements - On WA or on CWA Selection criteria - A set of path selection criteria - Cost Standard Standard time taken to reach the destination - Comfort Standard - Predetermined preferred navigation path complexity criteria Standards for spatialized sound - Predetermined spatial sound standards Requests and choices made by visually impaired users - Startup request - Select Request - Information Request The path determined by the navigation manager subunit 33 - Pn indicates at least one navigation path—not shown graphically. - SP Preferred Navigation Path - WP wander path - not represented graphically - PI Focus - PI-OLD's old concerns - Potential concerns regarding PPI – not represented graphically Guidance mode used to transmit corresponding navigation guidance instructions T-shaped walkable tunnel 93 Mileage marker r is the radius of the circle whose origin is the position of sensory unit 2, and has a predetermined length. 94. The intersection of a circle with radius d1 and the preferred navigation path SP. S Spatialized sound source d1 Spatialized sound source S at a predetermined first distance relative to sensory unit 2 d2 is a predetermined second distance. Example No. 1 Figures 14 to 17 84 Initial focus = entrance gate, as a concrete example of an object On selected as the focus PI from multiple objects On. The two entrance doors 84-01 and correspondingly 84-02, not shown graphically, serve as examples of objects On from a plurality of objects On, from which the point of interest 84 is selected. 841 Further Focus = Doorbell 841, as another concrete example of a focus PI. 83 Dogs, as a concrete example of biological Ln from multiple biological Ln. 831 Traffic Lights 832 Pedestrian Crossing 942 Walkable region, as a specific example of a walkable region WA. 941 Unwalkable region, as a specific example of an unwalkable region NA. 943 Conditionally Walkable Areas (CWAs) serve as a specific example. 911 Initial navigation path, as a specific example of the preferred navigation path SP. 912 Auxiliary navigation path, as another specific example of the preferred navigation path SP. 922 Walkable tunnel, as a specific example of a walkable tunnel T. Example group number 2—— Figures 18 to 28 85 - Window, as a concrete example of object On from multiple objects On that are potential points of interest (PPI) and as an example of the category of object On. 85-1 First Window 85-2 Second Window 85-3 The Third Window – Not Represented Graphically 85-4 The Fourth Window – Not Represented Graphically The selected ends of any window 85: 85-E1 and 85-E2.

[0077] S86 Window's Corresponding Spatial Sound S86-1 corresponds to the first window 85-1—not shown graphically. S86-2 corresponds to the second window 85-2—not shown graphically. S86-3 corresponds to the third window 85-3—not shown graphically. S86-4 corresponds to the fourth window 85-4—not shown graphically. Specific examples of spatialized sound S86f spatialized sound with specific frequencies—not represented graphically. S86f -1 has spatialized sound at a specific frequency corresponding to window 85-1. S86f -2 has spatialized sound at a specific frequency corresponding to window 85-2. S86f1 and S86f2 are spatialized sound sources encoded at different frequencies, which virtually move along the outline of window 85. S86p features spatialized sound with specific pulses—not represented graphically. S86t spatialized sound with specific temporal characteristics—not represented graphically. S86t-1 and S86t-2 are spatialized sounds with different temporal characteristics used to represent the shapes of the frames of the two windows 85-1 and 85-2.

[0078] S86t11-1 and S86t12-1 contain spatialized sounds with different temporal characteristics. These spatialized sounds virtually move along the outline of window 85-1, representing the shape of the outer frame of window 85-1. S86t21-1 and S86t22-1 contain spatialized sounds with different temporal characteristics. These spatialized sounds virtually move along the outline of window 85-1, representing the shape of the inner frame of window 85-1. S86P dot-matrix sound, S86P1 and S86P2 virtually move along the outline of window 85, creating spatialized point-like sounds. The S86P-E1 and S86P-E2 correspond to the two spatialized point sounds at the ends of windows 85-E1 and 85-E2. S86L Linear Sound S861 and S862 are spatialized sounds that virtually move according to an angular pattern within the space between the outline of the inner frame and the outer outline of the window 85. The time starting point of the spatialized sound source that virtually moves along the outline of window 85, and t final The time endpoint of a spatialized sound source that moves virtually along the outline of window 85. Detailed description and implementation examples refer to Figure 1 The wearable device 1 includes the following: a sensing unit 2, a processing and control unit 3, a feedback unit 4, and a user command interface 5.

[0079] The wearable device 1 includes two hardware units not shown graphically: a power storage unit 6 and a memory M.

[0080] Throughout this invention, it should be understood that the visually impaired person wears wearable device 1, and wearable device 1 is powered on. Therefore, any reference to wearable device 1 or sensory unit 2 in the specification, claims, and drawings should be understood to include references to the location of the visually impaired person. For simplicity, throughout this invention, the visually impaired user will be referred to as "he / she," encompassing all genders.

[0081] Details regarding the configuration and location of the hardware units will be provided in the description section related to the configuration of wearable device 1.

[0082] To better understand the method, the basic components of the hardware unit are briefly described in parallel with the publication of the method.

[0083] Sensory unit 2 is placed on the head of a visually impaired user and includes basic sensors: - Camera 21, - Depth sensor 22, - Inertial Measurement Unit 23 - Sound positioning sensor 24 Processing and control unit 3, the processing and control unit comprising: - Feel the fusion subunit 30, - Live Map Sub-unit 31 - Relationship Manager Subunit 32, - Navigation Manager Subunit 33, - User command interface manager subunit 34 - Feedback Manager Subunit 35, - Sound representation subunit 36, Sensory fusion subunit 30, the sensory fusion subunit comprising: - Positioning module 301, - Walkable area detection module 302, - Directional calculation module 303, - Sound direction positioning module 304 - Sound classification module 305, - Object 2D feature extraction module 306, - Object 3D Feature Fusion Module 307 - Object sound feature fusion module 308 Figure 2 illustrates a preferred location of the sensory unit 2 on the forehead of a visually impaired user. Throughout the detailed description and accompanying figures, the preferred location on the user's forehead is illustrated. Those skilled in the art will understand that the invention is not limited to placing the sensory unit 2 on the forehead.

[0084] The method according to the invention comprises four steps. These four steps will first be briefly described in their order. Steps 2, 3, and 4 will then be described in detail.

[0085] S1 The sensory unit 2 of the wearable device 1 placed on the head of the visually impaired user acquires data from the environment of the visually impaired user.

[0086] Therefore, the sensing unit 2 senses from the field of view 20 with the position of the sensing unit 2 as the origin.

[0087] S2 sends the data sensed by the sensory unit 2 to the sensory fusion subunit 30.

[0088] The sensory fusion subunit 30 fuses the data acquired from the sensory unit 2 using data processing algorithms including filtering, smoothing, and artificial intelligence-based algorithms, and then sends the fused data to the real-time map subunit 31 of the processing and control unit 3.

[0089] Furthermore, the live map subunit 31 creates, repeatedly updates, and stores the live map 310. The live map 310 includes three types of data: 1. A real-time map determination based on the fused data received from the sensory fusion subunit 30. 2. Real-world map determination based on multiple relationships Rn between multiple objects On and / or multiple organisms Ln. 3. Determining the real-world map based on free region A.

[0090] The live map 310 is a database stored in the memory M. Throughout this invention, updating and storing data in the live map 310 should include updating and storing the live map 310 in the memory M. The storage method of the live map 310 is outside the scope of this invention.

[0091] Figure 3 The field of view 20, with the position of sensory unit 2 as the origin, is schematically shown. Figure 4 The contents of the live map 310 and the field of view 20 are schematically shown compared to the real environment.

[0092] In an embodiment of the invention, the live map 310 already exists in the memory M. In this case, the geographical extent of the live map 310 is determined by the previously stored content in the live map 310. For example, from... Figure 4 As can be seen, the live map 310 includes objects that are no longer in the field of view 20, but these objects were in the field of view 20 in the past. Figure 4 The live map 310 includes the old focus PI-OLD, which was a focus of attention in the past.

[0093] 1. The real-world map determination based on the fused data received from the sensory fusion subunit 30 includes the following determinations: - Position and orientation of sensory unit 2, - Multiple objects are on. For each object on, the determination refers to: - Its location, - Its physical, acoustic, and chemical characteristics, - Its orientation, - A prediction of its future position within a predetermined unit of time. - Multiple organisms Ln. For each organism Ln, the determination refers to... - Its location, - Its biological and acoustic characteristics, - Its orientation, current activity, and emotional state, - A prediction of its future position within a predetermined unit of time.

[0094] 2. Receive real-time map determinations from the relation manager subunit 32 of the processing and control unit 3 based on multiple relationships Rn between multiple objects On and / or multiple organisms Ln.

[0095] The Relationship Manager subunit 32 imports the latest update from the Live Map 310 by querying the Live Map subunit 31 for updates in the Live Map 310.

[0096] The calculation is based on predetermined relationship requirements, which at least include: - A pre-determined father-son relationship, and - Predetermined conditional relationships.

[0097] For the sake of simplicity, throughout the invention: - The term "relation" is used as an equivalent term for multiple relations Rn, and - Multiple relations Rn, also referred to as relations, refer to both static and dynamic relations.

[0098] After the calculation is performed, the relation manager subunit 32 sends the updated relation as the result of the calculation to the live map subunit 31 to store it in the live map 310.

[0099] Details regarding the generation of multiple relations Rn are given in the following section related to S.2.2.

[0100] 3. Determining the real-world map based on free region A Free region A is defined as the entire area on the ground that is not occupied by multiple objects On and multiple creatures Ln.

[0101] The free region A is divided into three categories: - A walkable area WA that meets a fixed set of predetermined walkable area requirements is defined as an area in which a visually impaired user can walk without injury. - A conditionally walkable area (CWA) that satisfies the set of fixed, predetermined walkable area requirements and at least one predictable conditionally walkable area requirement, and - An unwalkable area NA that neither meets the set of fixed, predetermined walkable area requirements nor at least one additional, predictable conditional walkable area requirement.

[0102] In S3, automatically or in response to a first request from a visually impaired user, the navigation manager subunit 33 of the processing and control unit 3 determines and repeatedly updates one or more navigation paths Pn for the visually impaired user to navigate from the current position of the sensory unit 2 to a point of interest PI selected from multiple objects On and / or multiple organisms Ln, and stores them in the memory M.

[0103] The term "navigation" in this invention should be understood to include: - Visually impaired users approach objects (On) or organisms (Ln). - Everyday gestures and movements made by visually impaired users using one or both hands or limbs to find and reach various objects such as toothbrushes, doorknobs, etc.

[0104] Non-restricted examples of navigation paths Pn include: - Navigation path Pn for outdoor walking - Navigation path Pn for walking indoors, - Navigation path Pn for retrieving many different objects for various purposes: from small objects such as combs to large objects such as airplanes.

[0105] The navigation manager subunit 33 automatically or in response to a second request from a visually impaired user repeatedly selects a preferred navigation path SP. If only one navigation path Pn is determined, then the preferred navigation path SP is that navigation path Pn. If two or more navigation paths Pn are determined, the navigation manager subunit 33 repeatedly selects one of them as the preferred navigation path SP. The preferred navigation path SP, along with associated navigation guidance instructions, is repeatedly sent by the navigation manager subunit 33 to the feedback manager subunit 35 of the processing and control unit 3.

[0106] To determine one or more navigation paths Pn, the navigation manager subunit 33 queries the live map 310 to check whether at least two navigation path requirements are met.

[0107] The first navigation path requirement is that all navigation paths Pn (and therefore the preferred navigation path SP) must pass through the walkable area WA and / or through the conditionally walkable area CWA.

[0108] The second navigation path requirement is to meet a set of safety requirements for multiple objects On and / or multiple organisms Ln that are located near or predicted to be near at least one navigation path Pn within a predetermined unit of time. The proximity is predetermined, for example, a distance of 0.3m from the location of the wearable device 1.

[0109] This set of safety requirements includes at least one non-collision requirement and at least one non-aggression requirement. Additional safety requirements may also be defined to address specific needs arising from the needs of visually impaired individuals (e.g., the elderly), the characteristics of the environments in which they typically reside (e.g., densely populated urban areas), or both.

[0110] The collision-free requirement means that the individual paths of multiple objects On and / or multiple organisms Ln must not collide with at least one navigation path Pn.

[0111] The non-aggressive requirement means that the emotions of multiple biological entities (Ln) should not be expected to result in aggressive actions against visually impaired users.

[0112] Other navigation route requirements can be defined by the user, such as, but not limited to, avoiding congested areas or areas with slopes higher than a predetermined value.

[0113] Navigation path requirements are predetermined and stored in memory M. These requirements are applied by the navigation manager subunit 33. Visually impaired users can also set other predetermined navigation path requirements using the user command interface manager subunit 34.

[0114] When the selected navigation path SP passes through the Conditionally Walkable Area (CWA), the navigation manager subunit 33 sends an associated navigation guidance instruction to the feedback manager subunit 35, which is associated with the at least one predictable Conditionally Walkable Area requirement.

[0115] The determination of at least one navigation path Pn is initiated automatically by the navigation manager subunit 33 or by receiving a start request from a visually impaired user. If the navigation manager subunit 33 determines two or more navigation paths Pn, the selection of the preferred navigation path SP is performed automatically by the navigation manager subunit 33 or by the navigation manager subunit 33 receiving a selection request from a visually impaired user.

[0116] The navigation manager subunit 33 can be configured such that, by default, the selection of the preferred navigation path SP is performed automatically by the navigation manager subunit 33 or based on a selection request from a visually impaired user.

[0117] When executed automatically by the navigation manager subunit 33, the selection of the preferred navigation route SP is based on a set of predetermined route selection criteria, including cost criteria, time taken to reach the destination criteria, and comfort criteria. The application of these route selection criteria is carried out in accordance with existing technology.

[0118] Requests made by visually impaired users are made using tactile devices 51 or audio devices 52 via user command interface 5. These requests are received by navigation manager subunit 33 via user command interface manager subunit 34 of processing and control unit 3.

[0119] In S4, using a guidance mode for transmitting each associated navigation guidance instruction received from the navigation manager subunit 33, the feedback manager subunit 35 guides the visually impaired user along the preferred navigation path SP.

[0120] The guidance mode is sent from the feedback manager subunit 35 to the feedback unit 4 of the processing and control unit 3. Each navigation guidance instruction includes haptic and / or auditory cues.

[0121] The boot mode is: - Tactile cues, which are achieved by using a tactile feedback actuator 41 configured to be placed on the forehead of a visually impaired user, or - Through auditory cues, this is achieved by using an auditory actuator 42 configured to be placed near one or both ears of a visually impaired user, or - By combining tactile cues with auditory cues.

[0122] The selection of the guidance mode for each associated navigation guidance instruction is performed by the visually impaired user through the user command interface 5 and by a user command received by the feedback manager subunit 35 via the user command interface manager subunit 34.

[0123] Details about S2 – Reference Figure 5 The Live Map 310 can be compared to a multi-layered cake, as several layers of information are added from the first sub-step to the last, as described below. Utilizing these layers, the Live Map 310 achieves a higher degree of detail and accuracy. Creation is continuous, with the Live Map 310 being continuously updated and stored as a result.

[0124] The advantage of creating multiple information layers in the live map 310 is related to the ease of understanding and accessing data. Because each individual layer contains specific information related to certain other components of the system, this facilitates faster information retrieval.

[0125] S 2.1. The real-time map subunit 31 creates and updates the real-time map 310 by repeatedly determining the position and orientation of the sensory unit 2, the position and orientation and features of multiple objects On, and the position and orientation and features of multiple organisms Ln based on the fused data received from the sensory fusion subunit 30, and repeatedly sends the created and updated real-time map 310 to the positioning module 301 of the sensory fusion subunit 30. S 2.2. The Relationship Manager subunit 32 repeatedly generates and updates multiple relationships Rn between multiple objects On and / or multiple organisms Ln based on data obtained from the live map 310, including applying a predefined set of relationship requirements. The repeatedly updated multiple relationships Rn are repeatedly sent to the live map 310, thereby using the layers of content involving the multiple relationships Rn to update the content of the live map 310 output from S 2.1. S 2.3. The positioning module 301 uses a positioning algorithm applied to data received from the sensory unit 2 and data from the live map 310 to repeatedly locate the position and orientation of the sensory unit 2 relative to multiple objects On and correspondingly relative to multiple organisms Ln in the live map 310. The location and orientation of the sensory unit 2 are repeatedly sent to the walkable area detection module 302 of the sensory fusion subunit 30, thereby updating the content of the live map 310 output from S 2.2 using the positioning data involving the position and orientation of the sensory unit 2 relative to multiple objects On and correspondingly relative to multiple organisms Ln.

[0126] S 2.4. The walkable area detection module 302 repeatedly determines the free area A, based on: i) Data received from sensory unit 2, ii) Data received from positioning module 301, iii) A fixed set of predetermined walkable area requirements, and iv) Calculate and store at least one predictable conditional walkable area requirement in the memory M of the walkable area detection module 302.

[0127] The components of free region A are repeatedly updated and sent to live map 310, thereby using the layers involving the components of free region A to update the content of live map 310 output from S 2.3.

[0128] S 2.5. The updated live map 310 is repeatedly stored in memory M.

[0129] S 2.1. Details regarding the determination of the real-world map based on the fused data The orientation calculation module 303 determines the current position and orientation of the sensing unit 2 of the wearable device 1, as well as the current position and orientation of multiple objects On and multiple biological entities Ln relative to the sensing unit 2, based on the inertial motion data provided by the inertial measurement unit 23. For this purpose, the orientation calculation module 303 applies an orientation calculation algorithm that calculates the orientation of the system on three axes (pitch, roll, and yaw) and applies it to the 3D positioning of objects, because the camera 21 and depth sensor 22 reveal the position of the detected objects relative to the camera 21, but do not reveal the orientation of the detected objects relative to the ground.

[0130] The object 2D feature extraction module 306 provides pixel-wise segmentation of the 2D image acquired from the camera 21, and detects each of the multiple objects On and the multiple organisms Ln placed in the field of view 20 in the pixel-wise segmented 2D image, and determines their corresponding 2D coordinate positions and their corresponding physical features.

[0131] The object 2D feature extraction module 306 uses an object 2D feature extraction algorithm, which combines several actions: - Object detection, used to determine the type of objects On or organisms Ln, their 2D positions and their relative 2D dimensions and their 2D centers, and their 2D motion vectors relative to camera 21 obtained by comparing data between several subsequent images; - Object pose detection, used to determine the 2D coordinate orientation of an object. - Skeletal pose detection is used to determine the posture of an organism's lungs through skeletal orientation. This is used to understand the activities (such as running, sitting, coughing, etc.) and states (such as sleeping, awake, etc.) of the organism's lungs.

[0132] - Facial feature detection is used to determine empathic states (such as smiling, laughing, crying, etc.) and activity states: sleeping, awake, tired, etc.

[0133] - Object feature determination, which includes algorithms for different aspects, such as: - The extent to which chairs, handlebars, refrigerators, or rooms are occupied – for example, by comparing how much of it is visible to how much should be visible according to conventional images; - The 2D fill level of the container, such as in the case of a transparent container; - The degree of dirtiness of the product, for example, by comparing an object On captured by camera 21 with a known image of a clean, similar object On, and calculating the difference.

[0134] In addition, the object 3D feature fusion module 307 receives data from the object 2D feature extraction module 306, the orientation calculation module 303, and the depth sensor 22, and determines further details about multiple objects On and multiple organisms Ln based on the 2D information received from the object 2D feature extraction module 306.

[0135] Therefore, the object 3D feature fusion module 307 determines the 3D coordinate position of each object On relative to the sensory unit 2, their physical characteristics such as size, composition, structure, color, shape, humidity, temperature, occupancy, cleanliness, usage, wear, stability, filling, and danger level, as well as their orientation relative to the sensory unit 2, and correspondingly, their future 3D coordinate position at a predetermined time based on the movement vector. The object 3D feature fusion module 307 also determines data related to the 3D coordinate position of each organism Ln, their physical characteristics (such as height), their skeletal pose orientation, and predictions of their future position at a predetermined unit of time. Based on skeletal pose orientation, facial expressions, and their physical characteristics, the object 3D feature fusion module 307 determines the current activity and emotional state of each organism Ln.

[0136] The sound direction localization module 304 determines the direction, expressed in 3D coordinates, of multiple sound streams emitted by each of the multiple objects On and multiple organisms Ln based on data received from the sound localization sensor 24.

[0137] In one embodiment of the method, the directions of multiple sound streams are determined by comparing the differences in sound streams between the microphones of the sound localization sensor 24, while knowing the positions of each microphone. The sound direction localization module 304 performs triangulation on the incoming sound streams to detect the direction from which the sound streams originate.

[0138] Each of the multiple sound streams whose direction has been determined by the sound direction positioning module 304 is classified into various sound types by the sound classification module 305.

[0139] The object sound feature fusion module 308 adds acoustic features to each of the multiple objects On and organisms Ln with determined 3D coordinates based on the sound type classified by the sound classification module 305.

[0140] Then, the object sound feature fusion module 308 sends all the fused data to the live map subunit 31 so that it can be stored in the live map 310.

[0141] S.2.2. Details on generating multiple relations Rn The real-time map determination based on data received from the relation manager subunit 32 will provide further detailed information defining the environment of the visually impaired user. In this way, the relation manager subunit 32 provides more accurate and detailed information about objects On and organisms Ln, achieving the objectives of the invention, namely, to navigate visually impaired users more safely and to more specific navigation targets, which are defined in the invention as points of concern PI.

[0142] The algorithms used by the processing and control unit 3 include, but are not limited to: object detection, object pose detection, and object feature determination. The algorithms used by the processing and control unit 3 are defined as follows (item): - From each of multiple objects On, - Each organism Ln from multiple organisms Ln, - The individual components of each object, such as the legs of a chair. - The various parts of each organism Ln.

[0143] For example, in the case where object On is a chair with four legs, the chair is defined as an item separate from each of its four legs.

[0144] The degree of projectization is predetermined and is outside the scope of this invention.

[0145] The processing and control unit 3 creates object clusters based on the physical relationships between objects.

[0146] Therefore, pre-defined parent-child relationships connect individual items so that they can form a whole consisting of an object On, an organism Ln, or two or more objects On, two or more organisms Ln, or objects On and organisms Ln. For example, a doorknob belongs to a door. Both the doorknob and the door are items. The main difference between items on one side and objects On and organisms Ln on the other side is that objects On and organisms Ln correspond to people's conventional expectations of what objects and organisms are, while for the algorithm, all objects On and organisms Ln, as well as their components, are treated as items. Pre-defined conditional relationships refer to connecting individual items only when a condition is met; for example, a pedestrian crossing is a conditionally walkable area, conditional on the color of a traffic light.

[0147] The Relationship Manager subunit 32 uses data from the live map 310 to calculate possible relationships using a specific algorithm.

[0148] For parent-child relationships, an unrestricted example of the algorithm is as follows: - Physical Proximity If an item is physically adjacent to an object On or a creature Ln, they form a parent-child relationship. - If the lid and the open kettle are close together, the kettle becomes the parent of the lid, and the lid becomes the child.

[0149] - If the keyboard or mouse is close to the computer, it becomes a child element facing the computer, and the computer becomes the parent element. - If a pet creature, such as a dog or cat, is detected near a human creature, the pet becomes the human's child and the human becomes its parent. - If a door handle is near a door, then the door handle becomes a child of the door. Similarly, a vehicle's door is a child of the vehicle.

[0150] - Physical proximity with inclusion If an object On or a creature Ln is physically adjacent to another object and one object or creature is contained within another object, then they form another parent-child relationship: - If a fish is detected approaching and included in the fish tank, the fish becomes a child of the fish tank, and the fish tank becomes the parent. - If liquid is detected in a transparent container, the container becomes the parent and the liquid becomes the child. - If a seat is detected to be adjacent to the bus and is included within the bus, then the seat becomes a subclass of the bus. - Physical proximity with intersection If objects are physically adjacent and one or more of them intersect in their plane, then they form another type of parent-child relationship: - If a door and a wall are detected to be close together and their planes match, then the door becomes a child of the wall. Create new projects based on detected relationships: - Physical proximity with intersection If an item corresponding to object On is adjacent to it and they intersect, then they create a new item type and are assigned as children of that item. If a floor, roof, and multiple walls are detected to be adjacent and intersecting—all of these are projects, they form a room, a room is another project, and they become children of the room. - If multiple rooms are generated in a neighboring and intersecting manner, they create floors and become children of those floors. - If one or more floors are generated, they create a building and become children of the building.

[0151] For conditional relationships, an unrestricted example of the algorithm is as follows: - Physical Proximity If an item corresponding to object On is adjacent to another item, then they form a conditional relationship: - If the doorbell is detected as being near the door, a conditional relationship is formed: the doorbell must be rung before entering. - If a keyhole is detected as being near a door handle, a conditional relationship is formed: the door must be unlocked before the door handle can be operated. - Physical proximity with orientation If items corresponding to object On are close together and are oriented in a certain way, they form another conditional relationship: - If a conditional walkable area (CA) such as a crosswalk is detected, and a pedestrian traffic light oriented toward the crosswalk is detected, then the conditional walkable area (CA) is conditional on the color of the detected traffic light.

[0152] Depending on the type of object On or creature Ln, certain properties can be passed from parent to child, for example: - If the door handle is a child of the door and the door is a child of the car, then when the car moves, the position of the door handle or the door will be updated even if the car's position has changed at the same time.

[0153] All parameters used in the algorithm for establishing multiple relations Rn are predetermined: for example, a predetermined distance range is used to determine physical proximity.

[0154] S.2.3. Details regarding the position and orientation of the localization sensory unit 2 Using data acquired from the camera 21, depth sensor 22, and inertial measurement unit 23 of the sensing unit 2, the positioning module 301 repeatedly determines the current position and orientation of the sensing unit 2 of the wearable device 1, as well as the current position and orientation of multiple objects On and organism Ln relative to the sensing unit 2 in 3D coordinates on the current real-time map 310.

[0155] The location results are sent to the walkable area detection module 302, which determines the components of the free area A.

[0156] S.2.4. Details regarding the determination of the components of free region A The set of fixed, predetermined walkable area requirements includes categories predetermined for each visually impaired user, taking into account a variety of general safety and comfort requirements.

[0157] The group includes at least two of the following categories: - The pre-determined walkable area requirement in terms of geometry refers to the geometry of a space virtually occupied by a visually impaired user. Therefore, imagine a virtual cuboid whose three dimensions are adjusted to the size of a visually impaired user. This virtual cuboid ensures that the visually impaired user is protected from injury when standing still or moving. Non-limiting examples of requirements from this category include: ground-level vibrations must not exceed a pre-determined height; beams placed at a certain distance from the ground are considered dangerous if that distance is below a pre-determined threshold, and so on.

[0158] - Predetermined walkable area requirements for surfaces: certain surface types are excluded, such as, but not limited to: water films exceeding a predetermined width (e.g., 5 cm); ice; mud; streets and roads.

[0159] The Conditionally Walkable Area (CWA) must satisfy the set of fixed, predetermined walkable area requirements and must also satisfy at least one additional predictable conditionally walkable area requirement.

[0160] The set of fixed walkable area requirements, along with at least one predictable conditional walkable area requirement, are pre-determined for each visually impaired user and stored in memory M. The walkable area determination module 302 applies these requirements to data received on one hand from camera 21 and depth sensor 22, and on the other hand from self-localization module 301, the data received from localization module 301 including updates to relationships received from relationship manager subunit 32 and stored in live map 310.

[0161] In another preferred embodiment, a portion of the live map 310 may be downloaded from the Internet from any geographic map website. This portion refers to the layers described in S2.1 to S2.4, and takes into account that, depending on the geographic map website from which the map is downloaded, the information in each layer may be partial or complete. The download from the Internet is performed using a communication unit 7 connected to the Internet, which is not graphically represented. In this case, the positioning module 301 locates the position and orientation of the sensing unit 2 on the downloaded live map 310.

[0162] If a previously stored live map 310 exists in the memory of the live map subunit 31, then the live map 310 is created based on the live map determination of the previously stored live map 310.

[0163] If a previously stored real-time map 310 exists in the memory M of the wearable device 1, either because it was previously determined by the real-time map subunit 31, or because it was downloaded from the Internet, or both, then based on the determination of the data received from the sensing unit 2, starting from step 2.3, the current position and orientation of the sensing unit 2 are identified within the previously stored real-time map 310 by means of the positioning module 301, and the free area A, including the walkable area WA and the conditionally walkable area CWA, is identified by the walkable area detection module 302 by means of the positioning algorithm applied to the data received from the sensing unit 2.

[0164] In addition, the live map 310 is repeatedly updated with the additional information described in S 2.1 and S 2.2.2, and the remainder of steps 2.3, 2.4 and 2.5 are performed as described above.

[0165] In a preferred embodiment, the real-time map (310) is updated by the sensory fusion subunit (30) using the Simultaneous Localization and Mapping SLAM algorithm.

[0166] SLAM algorithms are particularly advantageous because they use an iterative process to improve the estimated position with new location information. The more iterations, the higher the positioning accuracy. The trade-off is more time spent on computation and high-configuration hardware with parallel processing capabilities.

[0167] In another preferred embodiment, the SLAM algorithm used is a visual SLAM algorithm, which has the advantages of providing a large amount of information, being inexpensive and easy to implement, because it can be a used passive sensor and component with a very small size, weight and power SWaP footprint.

[0168] The currently disclosed invention relates to situations where the point of interest (PI) is known to the visually impaired user before the activation request is sent.

[0169] In other cases, visually impaired users do not have sufficient information about their point of interest (PI) before sending a mobilization request. Typical examples are when they arrive in a new environment or when something in a known environment (such as their usual seating location) changes.

[0170] One example is when a visually impaired user enters an unfamiliar room with four windows. They want to open a window. But which of the four windows should be chosen as the point of focus (PI)? Or, a visually impaired user enters a meeting room with, say, 30 occupied seats and 10 empty seats. Which of the 10 empty seats should be chosen as the PI? To cover cases where visually impaired users require additional information from their environment to select a point of interest (PI) before sending an initiation request, in another preferred embodiment, when the PI is not known to the visually impaired user, substep 3-0 is performed before all other substeps in step 3: In S.3-0.1, a visually impaired user sends an information request to the sound representation subunit 36 ​​of the processing and control unit 3 regarding at least one object On selected from a plurality of objects On or at least one biological Ln selected from a plurality of biological Ln, said at least one object On or at least one biological Ln serving as a potential point of interest (PPI) for the visually impaired user. An example of at least one object On selected from a plurality of objects On is a set of windows from a selected room, which may be named “windows”.

[0171] The term "potential" means that any object On from the group of objects On can be selected as the initial point of interest PI.

[0172] The sound representation subunit 36 ​​is: - Independent sub-units connected to the live map sub-unit 31, the feedback manager sub-unit 35, and the user command interface sub-unit 34, or - A sub-unit of navigation manager sub-unit 33, for simplicity... Figure 1 The Chinese text indicates that, or - Module of Feedback Manager subunit 35.

[0173] Taking a room with four windows as an example, a visually impaired user sends an information request named "windows" to the voice representation subunit 36 ​​via the user command interface 5. This information request indicates that the user wants to know how many windows are in the room, their locations within the room, the size or shape of the windows, and the location of the window handles. In this example, the windows are potential points of interest (PPIs). The information request involves a pre-determined area of ​​interest located near the location where the visually impaired user is standing when sending the information request; in this case, the room.

[0174] Information requests are transmitted from user command interface 5 to voice representation subunit 36 ​​via user command interface manager subunit 34, just like start requests and selection requests.

[0175] In S.3-0.2, the sound representation subunit 36 ​​extracts information about at least one selected specific object On or at least one specific creature Ln from the live map 310 and represents the at least one specific object On or at least one specific creature Ln as the corresponding spatialized sound. When the sound representation subunit 36 ​​is not part of the feedback manager subunit 35, the spatialized sound is transmitted to the feedback unit 4 via the feedback manager subunit 35.

[0176] The spatialized sound representation is generated by using the sound representation subunit 36 ​​to encode the classification sound type of the selected object On or the corresponding selected organism Ln based on a predetermined spatialized sound standard.

[0177] A non-restrictive and non-exhaustive example of a predefined spatialized sound standard is: - Binaural virtualization of sound based on specific features of object On from the specific category of object On or specific features of organism Ln from the specific category of organism Ln. - Based on the distance of the object On biological Ln relative to the visually impaired user, the changes in frequency, amplitude, period, frequency components, spatialized sound fill factor or duration, and repetitive spatialized sound.

[0178] The type of encoding for the classification sound type of a selected object On or correspondingly selected organism Ln, based on a pre-determined spatialized sound standard, is selected based on a testing process that determines the user's ability to distinguish different technical features of the sound.

[0179] Visually impaired users can leverage the natural human ability to process sounds emanating from a sound source, along with subsequent training using wearable devices to locate each spatialized sound.

[0180] The localization of spatialized sound is performed in three spatial dimensions: - Horizontal: The azimuth angle of wearable device 1, which basically corresponds to the azimuth angle of the forehead of the visually impaired user. - Vertical: The elevation measured from the ground up to the wearable device 1, which roughly corresponds to the elevation of the forehead of the visually impaired user. - The range of distances or near and far dimensions measured from the standing point of sensory unit 2.

[0181] In S.3-0.3, the visually impaired user selects a point of interest PI from the specific plurality of objects On or correspondingly from the plurality of organisms Ln, and transmits the corresponding selection to the navigation manager subunit 33.

[0182] Example group number 2 details the aspects of sound representation.

[0183] In some cases, the point of interest (PI) is not in the live map 310, such as when a visually impaired person arrives at a new destination.

[0184] In this case, the live map unit 31 sends a confirmation to the navigation manager subunit 33 and the user command interface manager subunit 34 that the point of interest PI is not in the live map 310. The method in S3 has an additional sub-step before determining, iteratively updating, and storing at least one navigation path (Pn): S3-1 Navigation Manager Subunit 33 determines the wandering path WP—not shown graphically—while repeating methods S1 and S2 until the point of interest PI is found and stored in the live map 310, the wandering path WP satisfying the at least two navigation path requirements.

[0185] Possibly, the navigation manager subunit 33 determines the wandering path WP while representing a specific category of object On or a specific category of organism Ln as the corresponding spatialized sound. After the decision regarding the selection of the focus PI is made, the remainder of step 3 and step 4 of the method are performed as disclosed.

[0186] Details about S4 All navigation modes aim to keep visually impaired users on the preferred navigation path SP during navigation. Each preferred navigation path SP has its own level of complexity, corresponding to various navigation situations encountered in real life. The inventors conceived of quantifying the complexity of the preferred navigation path SP by using a rating corresponding to a predetermined complexity criterion for the SP, which includes both objective and subjective criteria. The latter is the visually impaired user's own interpretation of the objective criteria: for example, what is perceived as distant by one visually impaired user may not be perceived as distant by other visually impaired users, as may be ambient noise or temperature.

[0187] The following are some non-restrictive and non-exhaustive examples of predetermined preferred navigation path complexity criteria: - Width of walkable area WA and width of conditional walkable area CWA: The navigation method is different if the walkable area WA is only 10 cm wide compared to a walkable area WA with a width of 3 m. - The remaining distance to the point of interest (PI) - The number of turns and the angle of each turn, e.g., 30°, 75°. - The slope and / or number of steps / stairs, - Ambient noise, as it may limit the use of audio cues.

[0188] Tactile cues vary in the duration, periodicity, intensity, or frequency of vibrations based on a pre-determined criterion for the complexity of the preferred navigation path.

[0189] Audio cues vary in frequency, duration, repetition, intensity, or 3D spatial virtualization based on pre-determined criteria for preferred navigation path complexity.

[0190] Changes in tactile cues and corresponding audio cues have the advantage of adapting guidance for visually impaired users to the complexity of each preferred navigation path, quantified by a pre-defined SP complexity standard.

[0191] The advantages of feature variations in tactile and auditory cues, as well as the advantages of the possibility of combining tactile and auditory cues, are as follows: - Provides better guidance for visually impaired users along navigation paths in terms of accuracy and safety. - Provides the possibility to personalize the guidance according to a predetermined preferred navigation path complexity standard; - Provides more comfortable navigation, offering visually impaired users the possibility of making more decisions and continuously adjusting prompts according to their needs.

[0192] Tactile cues Tactile cues are received via haptic feedback actuator 41. Visually impaired users are trained before using wearable device 1 to associate various types of haptic cues with specific guidance instructions.

[0193] refer to Figure 6 A minimum of three haptic feedback actuators 41 are required, all of which are positioned at the azimuth angle of the forehead—that is, on the same horizontal line—to ensure guidance in at least three directions: forward, left, and right.

[0194] - Left haptic feedback actuator 411 installed on the left side of the forehead. - Right haptic feedback actuator 412, mounted on the right side of the forehead. - A central haptic feedback actuator 413 installed in the center of the forehead.

[0195] The tactile feedback actuator 41 includes a vibration actuator and a near-field and long-field tactile device, such as an ultrasonic tactile feedback actuator.

[0196] The vibration actuator includes multiple resonant actuators that convert electrical signals received from the feedback manager 35 into forced vibrations felt on the forehead of a visually impaired user, the vibrations being associated with a specific guidance command.

[0197] A non-limiting example of the vibratory actuator used in this invention is a linear resonant actuator, and each of the left tactile feedback actuator 411, the right tactile feedback actuator 412, and the center tactile feedback actuator 413 may include one or more linear resonant actuators.

[0198] The use of linear resonant actuators is advantageous for this invention because of their known good tactile performance, their improved efficiency compared to other vibratory actuators at resonance, their ability to optimize power consumption, and their small size, which allows them to be configured, for example, in a matrix form, if guidance in more than three directions is desired.

[0199] There are two types of tactile cues: - Temporal tactile cues are cues received at equal or unequal time intervals using any tactile feedback actuator 41. - Spatiotemporal tactile cues, alternatively referred to as tactile pattern cues, have a temporal component combined with a spatial component, namely a pattern, that represents the direction in which the visually impaired user must be reoriented, such as from bottom to top or from top to bottom, or to the right or left, and so on. Tactile perception of direction is obtained, for example, by using linear resonant actuators, because the boost efficiency of the linear resonator at resonance enhances the variation in the duration, periodicity, intensity, or frequency of the tactile cue's vibrations. Multiple linear resonant actuators output vibrations in a predetermined, rapid sequence, vibrating one after another in the direction in which the visually impaired user must be reoriented, so as to give the visually impaired user the perceptual tactile sensation of being dragged along the direction in which they must be reoriented.

[0200] The following is a non-restrictive example of applying a pre-determined criterion for preferred navigation path complexity: - Tactile cues are proportionally stronger and / or more frequent and / or have a higher vibration frequency: - The degree of deviation from the preferred navigation path SP, - To distinguish between a 90° turn and a 30° turn, or to distinguish between climbing 10 steps and climbing only 2 steps, the amount of movement required for visually impaired users.

[0201] - If the estimated navigation time to the point of interest (PI) is higher than a predetermined time threshold, the haptic cue will have a shorter duration and / or a lower intensity and / or a lower vibration speed to avoid visual impairment user fatigue from receiving too many haptic cues.

[0202] The type of tactile cues is predetermined for each situation based on the needs of the visually impaired user. For the purpose of better understanding the teachings of the invention and not limiting it, examples of the predetermined tactile cues are presented below: - The first tactile cue is used to initiate navigation from the current position of sensory unit 2. - A second tactile cue is used to signal to visually impaired users that they have deviated to the left from the preferred navigation path SP. - A third tactile cue is used to signal to visually impaired users that they have deviated to the right from the preferred navigation path SP. - The fourth tactile cue, used to move forward. - The fifth tactile cue, used to indicate whether to turn left or right. - The sixth tactile cue, used to indicate going up or down. - The seventh haptic cue is used to temporarily stop navigation if the navigation manager subunit 33 detects that at least one navigation path requirement has not been met, or if it detects a conditionally walkable area (CA), it needs to send a navigation instruction to stop before at least one predictable conditionally walkable area requirement is met.

[0203] - The eighth tactile cue is used to signal that the navigation endpoint, which is the point of interest (PI), has been reached.

[0204] Additional haptic cues can also be defined to suit other navigation scenarios or the needs of visually impaired users.

[0205] To ensure more accurate guidance while avoiding unnecessarily overwhelming visually impaired users with tactile cues, different types of tactile cues can be combined. For example: - For simple navigation commands, such as start / stop—as shown in the first, seventh, and eighth prompts from the examples above—only time-based prompts will be used, while - For complex navigation instructions, spatiotemporal cues can be used because when vibration criteria are applied to haptic pattern cues, guidance for turning left / right or going up / down is more accurate than using only temporal haptic cues.

[0206] The allocation of various types of tactile cues to one or more of the feedback actuators 41 used is predetermined.

[0207] Auditory cues Auditory cues are human-perceptible sounds received through an auditory feedback actuator 42 in the ear of a visually impaired user.

[0208] The auditory feedback actuator 42 is a speaker, headset, or bone conduction speaker that converts the electrical signals received from the feedback manager subunit 35 into sound. The associated navigation guidance commands received through the auditory feedback actuator 42 are based on the principle of assigning a specific sound to each associated navigation guidance command.

[0209] refer to Figure 6 A minimum of two auditory feedback actuators 42 are used: - Left auditory feedback actuator 421 installed in or around the left ear. - Right auditory feedback actuator 422 installed in or around the right ear.

[0210] Each of the left auditory feedback actuator 421 and the right auditory feedback actuator 422 may include multiple speakers, headphones or bone conduction speakers positioned at the same azimuth angle.

[0211] The type of auditory cues is predetermined for each situation based on the needs of the visually impaired user. To better understand the teachings of the invention, and not to limit it, examples of predetermined auditory cues are presented below: - The first auditory cue is used to begin navigation from the current position of sensory unit 2. - A second auditory cue is used to signal to visually impaired users that they have deviated to the left from the preferred navigation path SP. - A third auditory cue is used to signal to visually impaired users that they have deviated to the right from the preferred navigation path SP. - The fourth auditory cue, used to move forward. - The fifth auditory cue, used to indicate whether to turn left or right. - The sixth auditory cue, used to indicate whether to go up or down. - The seventh auditory cue is used to temporarily stop navigation if it is not yet finished. - The eighth auditory cue is used to signal that the navigation endpoint, which is the point of interest (PI), has been reached. Other types of auditory cues can also be defined to suit navigation situations or the needs of visually impaired users.

[0212] The assignment of various types of auditory cues to one or more auditory feedback actuators from auditory feedback actuator 42 is predetermined.

[0213] Considering the source of the sound, two types of sound are provided: - Simple sounds originating from the auditory feedback actuator 42 are used for simple associated navigation guidance commands, such as start and stop. - Spatialized sounds originating from one or more spatialized sound sources S, used for all associated navigation guidance instructions except for start and stop.

[0214] exist Figure 8A and Figure 8B In a preferred embodiment illustrated, the guidance mode includes a three-dimensional walkable tunnel T, defined as a virtual tunnel with a predetermined cross-section, which uses the preferred navigation path SP as its horizontal longitudinal axis. The guidance mode S4 includes specific tactile cues sent to the visually impaired user as they approach the virtual wall of the walkable tunnel T.

[0215] The three-dimensional walkable tunnel T and the preferred navigation path SP are simultaneously determined by the navigation manager subunit 33 and then sent to the feedback manager subunit 35 along with haptic cues.

[0216] The advantage of the walkable tunnel T is that it utilizes the greater degree of freedom to the left and right defined by the virtual walls of the walkable tunnel T, allowing for more comfortable navigation for visually impaired users.

[0217] As a visually impaired user approaches the virtual wall of the walkable tunnel T, guidance prompts are transmitted to guide him back to the space defined by the virtual wall of the walkable tunnel T. In some embodiments, in addition to the guidance prompts that signal the virtual wall of the walkable tunnel T, other guidance prompts are transmitted to confirm that the visually impaired user is navigating safely within the virtual wall of the walkable tunnel T.

[0218] The cross-section of the walkable tunnel T is predetermined based on multiple feasible cross-sections along the preferred navigation path SP and the preferences of visually impaired users.

[0219] Example number 1 details the boot mode using a walkable tunnel T.

[0220] In another preferred embodiment, reference is made to... Figure 9 The preferred navigation path SP is divided into predetermined segments defined by multiple mile markers 93. Each segment has a current mile marker 93 at one end and a subsequent mile marker 93 at the other end. The terms "next" and "successor" have the same meaning.

[0221] The guidance mode of S4 includes tactile and / or auditory cues that signal the location of the next mile marker 93, providing visually impaired users with associated navigation guidance instructions from the current mile marker 93 to the next mile marker 93. When a visually impaired user has passed a next mile marker 93, the next mile marker 93 becomes the current mile marker 93, and so on.

[0222] The length of the predetermined segment varies depending on the complexity and length of the preferred navigation path SP.

[0223] The length of each segment between two successive mile markers 93 is inversely proportional to the predetermined criterion of preferred navigation path complexity: the more complex the preferred navigation path SP, the shorter the segments. Mile markers 93 are more frequent in sections that include changes in direction along horizontal or vertical planes compared to sections that are traveling in a straight line.

[0224] The length of each segment between two successive mile markers 93 is determined by applying a pre-determined preferred navigation path SP complexity criterion. This means that the lengths of segments along the preferred navigation path SP are not necessarily equal, as in... Figure 9 What I saw in the video.

[0225] The length of each segment can be calculated using a score corresponding to the predetermined preferred navigation path complexity criterion, or it can be dynamically adapted to use an artificial intelligence-based learning method. For example, if a visually impaired user has some repetitive preferred navigation path SP and chooses to use a navigation method guided by odometry as a preference, then using the learning method to dynamically adjust the length of the odometry would be convenient.

[0226] The prompt used in the guidance mode from the current mileage marker 93 to the next mileage marker 93 is: - Tactile cues - Auditory cues, or - Tactile and auditory cues.

[0227] Here are some non-restrictive examples of using haptic cues: - The first, seventh, and eighth tips are time-related.

[0228] The second and third prompts are spatiotemporal, signaling whether visually impaired users have deviated from the preferred navigation path SP. - The fourth prompt is temporal and spatial, indicating the next mile marker 93 as you move forward. - The fifth and sixth are spatiotemporal, signaling the next mile marker 93, which in this case is defined as a place where the visually impaired user must reorient his direction of movement in a horizontal plane or correspondingly vertical plane. The duration, periodicity, intensity, or frequency of the vibrations indicated by the tactile pattern are proportional to a predetermined standard of preferred navigation path complexity, and simultaneously inversely proportional to the remaining distance to the next mile marker 93.

[0229] Here are some non-restrictive examples of using auditory cues: The first, seventh, and eighth prompts are simple sounds. - All other cues are spatialized sounds. The auditory feedback actuator 42 repeatedly outputs the spatial location of the successor mile marker 93 (using the spatialized sound heard from the location of the successor mile marker 93) until the visually impaired user has arrived at the successor mile marker 93. After arriving at each mile marker 93, the spatialized sound of that mile marker is no longer heard, that mile marker becomes the current mile marker 93, and the spatialized sound corresponding to the successor mile marker 93 begins to be heard, and so on.

[0230] According to a predetermined criterion for the complexity of the preferred navigation path, the spatialized sound varies in a proportional manner in terms of frequency, duration, repetition, intensity, and 3D spatial virtualization, while simultaneously varying inversely with the remaining distance to the subsequent mileage marker 93.

[0231] In situations where there is only one successor landmark 93 that coincides with the point of interest (PI), using only auditory cues is advantageous: for example, if a visually impaired user needs to walk from the sofa to the kitchen, in this case, the kitchen is the only successor landmark 93. In this situation, the spatialized auditory cues correspond to the kitchen. Using auditory cues has the advantages of simplicity and predictability because it provides the visually impaired user with the possibility of associating the remaining distance to be navigated to the successor landmark 93 with the corresponding auditory cues heard from the location of the successor landmark 93, which improves their orientation and sense of security during navigation. When the point of interest (PI) is known to the visually impaired user and the distance to the point of interest (PI) is short (e.g., for navigation routes within a house), using only auditory cues is preferred.

[0232] When the guidance mode from the current mileage marker 93 to the subsequent mileage marker 93 is through tactile and auditory cues, one of the tactile and auditory cues can be defined as primary and the other as secondary. The secondary output cues are only used in specific, predetermined situations, such as the seventh prompt instruction to stop and continue.

[0233] In another preferred embodiment, reference Figure 10 The system can combine tactile cues using a walkable tunnel T guidance mode with auditory cues from the current mile marker 93 to the next mile marker 93. The associated navigation guidance instructions sent by the tactile cues are designed to keep the visually impaired user on the preferred navigation path SP and within the boundaries of the walkable tunnel T, while the auditory cues enable the visually impaired user to estimate the remaining distance to navigate to the next mile marker 93.

[0234] Combining these two guidance modes offers the advantage of combining the strengths of each: the comfort and safety of walkable tunnels, and the simplicity and predictability of guidance from the current mileage marker 93 to the next mileage marker 93.

[0235] In another preferred embodiment, the guidance mode of S4 includes tactile or auditory cues or tactile and auditory cues that signal directions on the preferred navigation path SP.

[0236] Here are some non-restrictive examples of using haptic cues: - The first, seventh, and eighth tips are time-related.

[0237] - The second and third prompts are spatial and temporal, signaling whether visually impaired users have strayed from the preferred navigation path SP. - The fourth prompt is temporal and spatial, indicating the direction of travel through signals. - The fifth and sixth prompts are spatiotemporal, signaling to visually impaired users that they must reorient their direction of movement in the horizontal plane or correspondingly vertical plane.

[0238] The tactile pattern cues are predetermined so that they give visually impaired users the impression that their foreheads are being constantly dragged in the direction they are moving by the person standing in front of them.

[0239] The following are non-restrictive examples of using auditory cues: The first, seventh, and eighth prompts are simple sounds. - All other cues are spatialized sounds.

[0240] Based on a predetermined criterion for optimal navigation path complexity, spatialized sound varies proportionally in frequency, duration, repetition, intensity, or 3D spatial virtualization. Visually impaired individuals follow the direction of the spatialized sound source S during navigation.

[0241] The main difference between the guidance mode based on indicating the direction on the preferred navigation path SP using signals and the guidance mode based on the distance from the current mileage marker 93 to the subsequent mileage marker 93 refers to the changes in the characteristics of tactile pattern cues and the corresponding spatialized sounds: - In both guidance modes, haptic pattern cues and corresponding spatialized sounds vary proportionally to a pre-determined preferred navigation path complexity criterion. - In the case of the guidance mode from the current mileage marker 93 to the subsequent mileage marker 93, there is an additional change related to the distance to the subsequent mileage marker 93, which does not exist in the guidance mode based on the direction indicated by the signal on the preferred navigation path SP, and depends on the distance to the subsequent mileage marker 93.

[0242] Using tactile or auditory cues that signal directions on the preferred navigation path SP is advantageous when the complexity of the preferred navigation path SP is lower than when using a guidance mode from the current mile marker 93 to the next mile marker 93 or a walking path T. One such example is when the same preferred navigation path SP is used frequently. The advantage of using tactile or auditory cues that signal directions on the preferred navigation path SP is that they cause less fatigue for visually impaired users.

[0243] A non-limiting example of using tactile cues to signal directions on the preferred navigation path SP. Figure 11 As given in section A. In this example, the path indicating the direction on the preferred navigation path SP is first determined by intersecting the preferred navigation path SP with a circle whose origin is the location of sensory unit 2 and whose radius r is a predetermined length, to obtain intersection point 94. An imaginary line connecting the origin of sensory unit 2 and intersection point 94 gives the direction to be followed by the visually impaired user, which is conveyed to him / her through the tactile or audio cues. The predetermined length of radius r is set according to the complexity of the preferred navigation path SP.

[0244] Another non-limiting example of using auditory cues to indicate directions on the preferred navigation path SP is... Figure 11 Given in B. In this example, the direction to be followed is... Figure 11 The same method is used to build it in example A.

[0245] The spatialized sound source S is placed at a predetermined first distance d1 relative to the sensory unit 2.

[0246] To achieve flexibility in the guidance mode and adapt it to the complexity of the preferred navigation path SP, the predetermined first distance d1 of the spatialized sound source S relative to the sensing unit 2 can be less than a predetermined length of radius r (e.g., in...). Figure 11 (as shown in B), or a predetermined length that is equal to or greater than the radius r.

[0247] Tactile cues can be combined with auditory cues, and this combination is not represented graphically.

[0248] In another preferred embodiment, reference Figure 12 The auditory cue is a spatialized sound originating from a spatialized sound source S, which virtually travels from the position of the sensory unit 2 along a predetermined second distance d2 on a preferred navigation path SP until the spatialized sound source S reaches the end of the predetermined second distance d2 and returns to the position of the sensory unit 2.

[0249] Here are some non-restrictive examples of using auditory cues: The first, seventh, and eighth prompts are simple sounds. - All other cues are spatialized sounds.

[0250] The auditory feedback actuator 42 repeatedly outputs a spatialized sound source S by means of variations in frequency, duration, repeatability, intensity, and 3D spatial virtualization that are proportional to a predetermined criterion for the complexity of the preferred navigation path.

[0251] The predetermined second distance d2 is inversely proportional to the predetermined complexity standard of the preferred navigation path SP. That is, the more complex the preferred navigation path SP is, the smaller the predetermined second distance d2 is.

[0252] The predetermined second distance d2 generally varies between 0.2m and 5m. If the preferred navigation path SP is very complex, the predetermined second distance d2 generally varies between 0.2m and 1m. Examples of values ​​for the predetermined second distance d2 are given for illustrative purposes only and should not be considered as limitations.

[0253] Example: The predetermined second distance d2 is 1.2m. This means that the spatialized sound source S virtually propagates at a distance of 0.2m from the location of sensory unit 2. The spatialized sound propagates back and forth from the location of sensory unit 2 until it reaches 1.2m along the navigation direction and then it returns to the location of sensory unit 2. Because the speed of sound is significantly higher than the speed of human walking, visually impaired users receive more detailed navigation guidance instructions than in any other guidance method disclosed in this invention, because in the guidance mode using the virtual propagation of spatialized sound, the sound propagates independently of the visually impaired user.

[0254] The characteristics of a sound—namely, frequency, duration, repetition, intensity, and spatial virtualization, or any combination thereof—varie inversely proportional to the distance remaining to a predetermined second distance d2. For example, when the spatial sound source S is at 0.1m, the auditory cue is more frequent and / or more intense and / or more spatially virtualized or longer in duration than when the spatial sound source S is at 0.2m.

[0255] The advantage of this guidance mode is that it allows for fine-tuning of navigation, which gives it an advantage in environments where the walkable area WA is very narrow, and therefore, the preferred navigation path SP looks like a slalom between object On and organism Ln.

[0256] In a second aspect of the invention, the wearable device 1 includes a sensing unit 2, a processing and control unit 3, a feedback unit 4, and a user command interface 5.

[0257] The wearable device 1 includes two hardware units not shown graphically: a power storage unit 6 and a memory M.

[0258] The term "memory M" should be understood to refer to multiple non-volatile memories that are either grouped together in a single specific hardware unit or distributed in each of other hardware units.

[0259] The memory M is configured to store at least the live map 310, all algorithms, all standards and requirements, and the preferences of visually impaired users, such as, but not limited to, their preferred type of prompts for receiving guidance instructions. Storage is implemented according to existing technology.

[0260] In a preferred embodiment, wearable device 1 is a single-component device, while in other preferred embodiments, wearable device is a multi-component device.

[0261] In the case of a single-component device 1, all hardware units are included in the wearable device 1, such as... Figure 2B As shown in the diagram, wearable device 1 is mounted on the head (not shown graphically). The positioning of the hardware units on the visually impaired user's head can be, for example, in the form of a headband, which has the advantage of providing good support and anchoring for all hardware units.

[0262] In the preferred embodiment of the multi-component device 1, refer to Figure 2A , Figure 6 and Figure 7 One of the components is a head-mounted device component 11, which includes a sensing unit 2 and a feedback unit 4. The head-mounted device component 11 can be, for example, in the form of a headband, such as... Figure 2A and Figure 2B As shown in the image.

[0263] Two non-limiting examples of a preferred embodiment of the multi-component device 1 illustrate two components: - Head-mounted device component 11, and - Wearable component 12, or corresponding wrist component 12. Wrist component 12 is not shown graphically.

[0264] In this case, the wearable component 12 or the corresponding wrist component 12 includes a processing and control unit 3, a user command interface 5, and a power storage unit 6.

[0265] The memory M can be included in either of the two components or distributed between the two components.

[0266] Figure 2A A preferred embodiment of a two-component device 1 having a head-mounted device component 11 and a wearable component 12 is illustrated.

[0267] The division of components between the head-mounted device component 11 and the wearable component 12 or corresponding wrist component 12 is primarily based on the size and weight of the unit.

[0268] The advantage of using a single-component device 1 is that its preferred position on the head provides visually impaired users of the wearable device 1 with a sensory experience very close to that of non-visually impaired individuals, close to the ear position where auditory cues can be heard.

[0269] However, in some cases, certain hardware units, such as the processing and control unit 3 and / or the power storage unit 6, may be heavy and bulky. In these cases, the multi-component device 1 has the advantage of placing the heavy and bulky hardware units in other parts of the body, such as, but not limited to, a belt or wrist.

[0270] As technology as a whole moves toward miniaturization of hardware units, this will lead to an increased possibility of using single-component devices¹ without placing too much strain on the head of visually impaired users.

[0271] In another preferred embodiment (not shown graphically), three components are provided: - A head-mounted device component 11 including a sensing unit 2 and a feedback unit 4. - Includes a wearable component 12 or a corresponding wrist assembly 12, comprising a processing and control unit 3 and a power storage unit 6. - Handheld component 13 (not shown graphically) including user command interface 5. The memory M may be included in or distributed in either the head-mounted device component 11, the wearable component 12, or the corresponding wrist component 12.

[0272] The configuration of the various units constituting the wearable device 1 for the present invention to function is not affected by the positioning of the hardware units relative to the parts of the human body in a single-component device or a corresponding multi-component device.

[0273] The hardware units communicate with each other via wired communication protocols, wireless communication protocols, or a combination of wired and wireless protocols, and the communication is carried out in accordance with existing technology.

[0274] Sensory Unit 2 Sensory unit 2 has an apparatus configured to collect data about the environment of a visually impaired user.

[0275] The data collected by sensory unit 2 involves a variety of features of objects On and organisms Ln that are typically recognized by people with good sensory abilities (including good vision). The data collected by sensory unit 2 reflects the complexity of the environment more accurately than existing technologies.

[0276] To achieve the goal of collecting more accurate data, sensing unit 2 requires a combination of various types of sensors, which will be described in detail. It should be understood that all examples of sensors are provided to better understand the teachings of the invention and are not intended to limit the invention.

[0277] The sensing unit 2 includes four basic sensors: camera 21, depth sensor 22, inertial measurement unit 23, and sound localization sensor 24.

[0278] The optimal position for the camera 21, depth sensor 22, and inertial measurement unit 23—regardless of whether the wearable device 1 is a single-component or multi-component device—is on the forehead, such as on... Figure 2A and Figure 2B As shown in the figure. There are three reasons why the sensory unit 2 is preferably located on the forehead: i) because humans, regardless of visual impairment, are accustomed to moving their heads when receiving cues (such as auditory or tactile cues) in the absence of any devices; ii) because the forehead is not currently being used for other devices or tasks; and iii) because the optimal field of view for both the camera 21 and the depth sensor 22 is on the forehead.

[0279] The configuration of sensory unit 2 on the forehead of a visually impaired user must ensure that the field of view 20 includes: - The feet of users with visual impairment - A component of the free area A immediately adjacent to the foot. - Real-time pace for visually impaired users The first type of sensor is a camera 21. The term "camera 21" throughout the invention refers to one or more digital cameras. This invention requires at least a digital camera.

[0280] Camera 21 is configured to acquire 2D images from the camera's field of view and send the acquired 2D images to the positioning module 301, the walkable area detection module 302, and the object 2D feature extraction module 306.

[0281] The term "image" includes still images and videos, depending on the image acquisition frame rate of camera 21.

[0282] The images acquired by camera 21 involve visual features of multiple objects On and multiple organisms Ln, such as appearance; category, such as tree, car; color, shape, size, and components of free region A.

[0283] Non-limiting examples of camera 21 include: an HD camera with a minimum video resolution of 1280 pixels × 720 pixels; and a VGA camera with a minimum video resolution of 320 pixels × 240 pixels. The minimum requirements for camera 21 are as follows: - A horizontal field of view between at least 50° and up to 180°, the larger the better, as it provides information from a larger area, and - A vertical field of view between at least 60° and up to 180°, the larger the better, as it provides information from a larger area.

[0284] Camera 21 can be an RGB camera or not. RGB features help provide more accurate information from the camera's field of view.

[0285] The more complex the camera, the more information the 2D images captured by the camera contain.

[0286] The second type of sensor is Depth sensor 22 The term "depth sensor 22" refers to one or more depth sensors throughout this invention. This invention requires at least one depth sensor.

[0287] The depth sensor 22 is configured to acquire 3D point cloud data corresponding to the 3D distance position and size of each object On and each organism Ln placed in the depth sensor's field of view, as if they were a continuous point cloud, and send them to the localization module 301, the walkable area detection module 302, and the object 3D feature fusion module 307.

[0288] The 3D point cloud data acquired by depth sensor 22 involves the 3D physical characteristics of object On and organism Ln, such as density and volume. Non-limiting examples of depth sensor 22 are stereo cameras, radar, LiDAR, ultrasonic sensors, and millimeter-wave radar sensors. Using millimeter-wave radar sensors is advantageous because it can sense the pulse or breathing of organism Ln, even when the organism Ln is moving, providing additional information to visually impaired users.

[0289] The camera 21 and depth sensor 22 can be combined into a single sensor, namely, camera and depth sensor 21-22. The advantage is that the size and weight of the two aforementioned sensors can be reduced by using only one sensor configured to perform the tasks of both sensors. A non-limiting example of a camera and depth sensor 21-22 is a time-of-flight (TOF) camera.

[0290] The third type of sensor is Inertial Measurement Unit 23The term "inertial measurement unit 23" throughout the invention refers to the entire assembly consisting of at least one accelerometer and at least one gyroscope, either as a single sensor or as a combined sensor. Preferably, at least one magnetometer is added to obtain better accuracy, either as a single sensor or in combination with at least one accelerometer and / or at least one gyroscope. A combined sensor is preferred because of the need to reduce the overall size and weight. The present invention requires at least one inertial measurement unit.

[0291] The inertial measurement unit 23 is configured to determine the orientation of the sensing unit 2 and send the determined orientation to the positioning module 301 and also to the feature fusion module 307 via the orientation calculation module 303.

[0292] Since the sensory unit 2 is placed on the forehead of the visually impaired user, the information obtained by the inertial measurement unit 23 implicitly relates to the orientation of the visually impaired user's head relative to the ground.

[0293] The fourth type of sensor is Sound positioning sensor 24 .

[0294] The term "sound localization sensor 24" throughout the present invention refers to one or more sensors that are used to determine the sources of various sounds in three-dimensional space substantially by means of the direction of the incoming sound waves and the distance between the source and the sensor.

[0295] The sound localization sensor 24 is configured to acquire multiple sound streams emitted by the object On and the organism Ln in three-dimensional space and send them to the sound direction localization module 304.

[0296] The information acquired by the sound localization sensor 24 pertains to the sounds emitted by the object On and the organism Ln, including the directionality of the sounds. The coverage of the environment by the sound localization sensor 24 is defined by its beam pattern.

[0297] A non-limiting example of a sound localization sensor is a microphone array. The minimum number of microphone arrays used for sound localization sensor 24 must ensure that the sum of their beam patterns equals the angle of the field of view 20. The maximum number of microphone arrays used for sound localization sensor 24 covers 360°. The microphone arrays are located within the head-mounted device such that the sum of their beam patterns is contained between the angle of the field of view 20 and 360°.

[0298] The base sensors receive a specific configuration from the sensor fusion subunit 30 of the processing and control unit 3, which includes the correlation between the respective fields of view of the camera 21, the depth sensor 22 and the measurement range of the inertial measurement unit 23, and the beam pattern of the sound localization sensor 24.

[0299] The interrelated fields of view 20 of the underlying sensors (in) are the result of this. Figure 3 (Illustrative illustration). The concept of the field of view 20 of the basic sensor does not mean that all basic sensors have exactly the same range. It should be understood as the area that all basic sensors will perceive, similar to the concept of a common denominator in mathematics. Typically, the field of view 20 of the basic sensor faces forward.

[0300] However, the sound localization sensor 24 can have a wider field of view than the basic sensor 20, for example, when the number of microphone arrays makes the sum of the beam patterns reach 360°. This is advantageous because it allows the collection of sound information originating from behind the visually impaired user.

[0301] exist Figure 13 In another preferred embodiment illustrated, more information about the environment of the visually impaired user can be provided by adding one or two additional sensors: a GPS sensor 25 and a temperature sensor 26.

[0302] Each additional sensor, in any combination with the basic sensor group, has the advantage of providing additional information to the processing and control unit 3, resulting in a more accurate and detailed real-time map 310.

[0303] Each of the two additional sensors has a corresponding module in the sensory fusion subunit 30, as follows: Global Positioning Sensor 25 The sensor is configured to determine the absolute position of sensory unit 2 and send the determination to relative to absolute conversion module 309-1, which converts the relative position of sensory unit 2 into an absolute position, so that the position of object On and the position of organism Ln are expressed as absolute positions.

[0304] In the case of a multi-component device, the optimal position of the GPS sensor 25 is on top of the head-mounted device component 11 of the wearable device 1, and correspondingly on top of the wearable device 1 in the case of a single-component device.

[0305] In the absence of the global positioning sensor 25, the sensory fusion subunit 30 determines the relative position of the wearable device 1 with respect to each object On and with respect to each organism Ln.

[0306] Temperature sensor 26 It is configured to determine the temperature of object On and organism Ln and send the determined temperature to object temperature feature fusion module 309-2.

[0307] When using any of the additional sensors, the data output by the object sound feature fusion module 308 is sent to the relative to absolute conversion module 309-1 or the object temperature feature fusion module 309-2, and fused with the data sent by the corresponding sensor, or the result is sent to the real-time map subunit 31.

[0308] In cases where two additional sensors are used, such as Figure 13 As shown in the diagram, the data output by the object sound feature fusion module 308 is sent to the relative to absolute conversion module 309-1, where it is fused with the data from the global positioning sensor 25. The result is then sent to the object temperature feature fusion module 309-2, where it is fused with the data sent by the temperature sensor 26. Finally, the result is sent to the real-time map subunit 31.

[0309] Processing and Control Unit 3 The processing and control unit 3 is a computing unit that includes at least one processor and at least one non-volatile memory, such as, but not limited to, a microcontroller, a computer, or a supercomputer. The term "computing unit" encompasses a single computing unit or multiple computing units located far apart from each other that communicate within a computer communication system.

[0310] The processing and control unit 3 includes: a sensory fusion subunit 30, a real-time map subunit 31, a relationship manager subunit 32, a navigation manager subunit 33, a user command interface manager subunit 34, a feedback manager subunit 35, and a sound representation subunit 36.

[0311] refer to Figure 5 The sensory fusion subunit 30 includes an apparatus configured to fuse and correlate decisions received from the four basic sensors of the sensory unit 2.

[0312] refer to Figure 13 The sensory fusion subunit 30 includes an apparatus configured to fuse and correlate decisions received from the four basic sensors of the sensory unit 2 and from one or two additional sensors.

[0313] Positioning Module 301 This includes a device configured to locate the current position and orientation of the wearable device 1's sensor unit 2 in 3D coordinates on a current real-world map 310, as well as the current position and orientation of multiple objects On and organisms Ln relative to the sensor unit 2, using a positioning algorithm applied to data acquired from the camera 21, depth sensor 22, and inertial measurement unit 23 of the sensor unit 2.

[0314] The positioning module 301 also includes a device configured to send the position and orientation of the sensing unit 2 to the walkable area detection module 302, thereby updating the content of the live map 310 output from S 2.2 using a layer of positioning data involving the position and orientation of the sensing unit 2 relative to multiple objects On and correspondingly relative to multiple organisms Ln.

[0315] Walkable area detection module 302 It includes an apparatus configured to receive data from camera 21 and depth sensor 22, an apparatus configured to receive data from positioning module 301, and an apparatus configured to define walkable area WA and conditional walkable area CWA by applying a set of fixed, predetermined walkable area requirements and predictable conditional walkable area requirements stored in memory M and sending them to the live map subunit 31 based on these two data sources.

[0316] Targeted Calculation Module 303 Includes a device configured to determine the orientation of the wearable device 1 based on inertial data provided by the inertial measurement unit 23 and to send the determination to the object 3D feature fusion module 307.

[0317] Sound direction positioning module 304 The device includes an apparatus configured to determine, in 3D coordinates, the directions of multiple sound streams emitted by each of a plurality of objects On and a plurality of organisms Ln based on data received from the sound localization sensor 24, and an apparatus configured to send the determined directions to the sound classification module 305.

[0318] Sound classification module 305 This includes an apparatus configured to classify multiple sound streams received from a sound direction localization module 304 into sound types and send the classified sound types to an object sound feature fusion module 308. The apparatus configured to classify multiple sound streams into sound types essentially uses artificial intelligence algorithms.

[0319] Object 2D Feature Extraction Module 306 The device includes an apparatus configured to: provide pixel-level segmentation of a 2D image acquired from a camera 21, detect each of a plurality of objects On and each of a plurality of organisms Ln placed in a field of view 20 in the pixel-level segmented 2D image, determine their corresponding 2D coordinate positions and their corresponding physical features, and send the determination to an object 3D feature fusion module 307.

[0320] Object 3D Feature Fusion Module 307 This includes an apparatus configured to receive and determine data from an object 2D feature extraction module 306, an orientation calculation module 303, and a depth sensor 22: - The 3D coordinate position of each object On relative to sensory unit 2, their orientation relative to sensory unit 2, and their future position at a predetermined time based on the movement vector.

[0321] - Physical characteristics of multiple objects, such as size, composition, structure, color, shape, humidity, temperature, occupancy level, cleanliness level, usage level, wear level, stability level, filling level, and hazard level. - The 3D coordinates of each organism Ln, their physical characteristics such as height and skeletal pose orientation, and predictions of their future positions in a predetermined unit of time based on movement vectors. - The current activity and emotional state of each organism Ln based on skeletal posture orientation, as well as their physical characteristics.

[0322] Object sound feature fusion module 308 This includes an apparatus configured to add acoustic features to each of a plurality of objects On and organisms Ln with determined 3D coordinates by associating the sound stream type, which is determined by the sound classification module 305, with the detected objects On and organisms Ln, and to send all data to the live map subunit 31.

[0323] In an embodiment of the present invention, the sensory fusion subunit 30 further includes Relative and Absolute Conversion Module 309-1 The module includes an apparatus configured to convert the relative position of the sensing unit 2 into an absolute position, fuse data from the object sound feature fusion module 308 with data relating to the absolute position of the sensing unit 2, and send the determination directly or via the object temperature feature fusion module 309-2 to the real-time map subunit 31.

[0324] In another embodiment of the invention, the sensory fusion subunit 30 further includes Object temperature feature fusion module 309-2 This module includes an apparatus configured to determine the temperature of a detected object On and a biological entity Ln, fuse data from the object sound feature fusion module 308 with data relating to the temperatures of the object On and the biological entity Ln, and send the fused data to the live map subunit 31. If the relative to absolute conversion module 309-1 is used, it sends data to the object temperature feature fusion module 309-2 and ultimately fuses that data with data relating to the temperatures of the object On and the biological entity Ln.

[0325] According to the present invention, Live Map Sub-unit 31This includes an apparatus configured to create, repeatedly update, and store a live map 310, and an apparatus for receiving data from the walkable area detection module 302 relating to the components of the free region A and a positioning layer relating to the position and orientation of the sensory unit 2, updated from the live map 310 output from S 2.2, data relating to each of the multiple objects On and the organism Ln in 3D coordinates, including acoustic features from the object sound feature fusion module 308, and sending all live map decisions to the positioning module 301.

[0326] The live map subunit 31 includes a device configured to receive: - The relationship manager sub-unit 32 queries the live map 310. - Queries from the navigation manager subunit 33 to the live map 310, including queries from the user command interface manager subunit 34 to the navigation manager subunit 33, if the point of interest PI is already in the live map 310. - Updated relation Rn via relation manager subunit 32 - Components of the updated free area A implemented through navigation manager subunit 33, - Sound representation module 36 queries for specific information related to object On.

[0327] The live map subunit 31 includes a device configured as follows: - In response to a query from the relation manager subunit 32 to the live map 310, multiple relations Rn are sent. - Components of free region A are sent in response to a query from navigation manager subunit 33. - In response to queries from the navigation manager subunit, all live map decisions are sent to the navigation manager subunit 33. Relationship Manager Subunit 32 This includes an apparatus configured to query the live map 310 and import recently updated data from the live map 310 as a result of the query. The recently updated data includes: - Multiple objects on, - Multiple organisms Ln, - Conditionally Walkable Area (CWA) because certain objects On and / or certain creatures Ln are associated with the CWA, such as traffic lights or dogs. - Existing relation Rn before the query.

[0328] Furthermore, the relationship manager subunit 32 includes an apparatus configured to perform calculations to determine and update relationships between multiple objects On and / or multiple organisms Ln, and to send the calculated updated relationships to the live map subunit 31 for storage in the live map 310.

[0329] Navigation Manager Subunit 33 This includes appliances configured such that: - Determine and repeatedly update at least one navigation path Pn and store it in memory M. - Repeatedly select the preferred navigation path SP from at least one navigation path Pn. - Repeatedly send the preferred navigation path SP along with the associated navigation guidance instructions to the feedback manager subunit 35.

[0330] - Receive start requests, selection requests, and information requests from the user command interface manager subunit 34.

[0331] - Query the real-world map 310 based on the real-world map determination and the components of the free region A: the walkable region WA, the conditionally walkable region CWA, and the inaccessible region NA, and receive the response corresponding to each query from the real-world map subunit 31.

[0332] - If the point of interest (PI) is already in the live map 310, then receive a query from the user command interface manager subunit 34. - Verify that at least two navigation path requirements are met.

[0333] - Send an associated navigation guidance instruction to the feedback manager subunit 35, which is related to the at least one predictable conditionally walkable area requirement.

[0334] User command interface manager subunit 34 This includes an apparatus configured to receive requests and selections made by a visually impaired user via the user command interface 5 and transmit them to the navigation manager subunit 33, and an apparatus configured to send the selected guidance mode to the feedback manager subunit 35.

[0335] User command interface manager subunit 34 also includes an apparatus for receiving requests from visually impaired users to make audio representations of a specific category of object On or a specific category of creature Ln from the live map 310.

[0336] Feedback Manager Subunit 35It includes an apparatus configured to guide a visually impaired person along a preferred navigation path SP by receiving guidance instructions from the navigation manager subunit 33 and a selected guidance mode from the user command interface manager subunit 34, and an apparatus configured to transmit corresponding associated guidance instructions to the feedback unit 4. The feedback manager subunit also includes an apparatus for transmitting sound representations of a specific category of object On or a specific category of organism Ln.

[0337] In embodiments where the sound representation subunit 36 ​​is a separate subunit and a subunit of the navigation manager subunit 33, the feedback manager subunit 35 further includes an apparatus for receiving sound representations from the sound representation subunit 36 ​​of a specific category of object On or a specific category of organism Ln.

[0338] Sound representation subunit 36 It includes an apparatus configured to receive requests from visually impaired individuals and extract corresponding information about a specific category of object On or a specific category of organism Ln from the live map 310, and an apparatus for representing the extracted information as corresponding spatialized sound and transmitting it to the feedback unit 4.

[0339] Feedback unit 4 (configured to be placed on the head of a visually impaired user) includes an instrument configured to guide the visually impaired user along a preferred navigation path SP by receiving associated guidance instructions from feedback manager subunit 35 and by sending tactile and / or auditory cues to the visually impaired person, as described in detail in the section on the details of step 4 of the method, and the feedback unit includes an instrument for sending sound representations of a specific type of object On or a specific category of biological Ln to the visually impaired user.

[0340] User command interface 5 (configured to be placed on the head of a visually impaired user) includes an appliance configured to receive requests (i.e., start requests, selection requests, and information requests) from the visually impaired user and to select a boot mode and send them to user command interface manager subunit 34.

[0341] The following is a non-restricted example of user command interface 5: - User commands the tactile device 51, such as buttons for simple requests corresponding to commonly used, pre-defined points of interest (PIs). For example, a first button might be named "Home," corresponding to the entrance door of a house inhabited by a visually impaired person; a second button might be named "Bathroom"; a third button might be named "Kitchen," and so on. The buttons can be analog or digital.

[0342] - User command audio device 52, such as a microphone, is used for less frequently used points of concern (PIs). The user command audio device includes a speech recognition device and a device that translates the speech of a visually impaired user into instructions sent to the user command interface manager subunit 34. For example, in a room with four windows, the visually impaired person says "windows" to microphone 52, and all four windows of the room are represented by sound.

[0343] The communication between the user command interface 5 and the visually impaired person, as well as with the user command interface manager subunit 34, is based on the prior art.

[0344] Power storage unit 6 The term "power storage unit 6" should be understood to refer to one or more batteries configured to power other hardware units of the wearable device 1. The power storage unit 6 powers the other hardware units of the wearable device 1 in a manner consistent with existing technology.

[0345] The communication unit 7 includes a device configured to download maps from the Internet, such as, but not limited to, downloadable maps.

[0346] In a third aspect of the invention, a non-transitory computer-readable storage medium is provided, the computer program comprising instructions executable by one or more processors, which, when executed, cause the one or more processors to perform operations of any preferred embodiment (including combinations thereof) of a computer-implemented method suitable for assisting a visually impaired user in moving.

[0347] In a fourth aspect of the invention, a computer program is provided, the computer program including instructions that, when executed by a wearable device according to any embodiment of the invention, cause the wearable device to perform steps of a computer-implemented method suitable for assisting a visually impaired user in any preferred embodiment (including combinations thereof).

[0348] In a fifth aspect of the invention, a system is provided comprising one or more processors and one or more non-transitory computer-readable storage devices, the non-transitory computer-readable storage devices including a storage medium containing instructions executable by the one or more processors to perform operations of a computer-implemented method suitable for assisting a visually impaired user in any preferred embodiment (including combinations thereof).

[0349] The term "computer" refers to a computing unit that includes at least one processor and at least one non-volatile memory, such as, but not limited to, a microcontroller, a computer, or a supercomputer. The term "computing unit" encompasses a single computing unit or multiple computing units located far apart from each other that communicate within a computer communication system.

[0350] Example No. 1 refer to Figure 14 , Figure 15 , Figure 16 and Figure 17 The following is a detailed description of the method, illustrated using real-life scenarios. Those skilled in the art should understand that the teachings of this invention are not limited to these examples.

[0351] In a real-life scenario, a visually impaired person 1 is standing on the sidewalk near a building entrance. Wanting to enter the building, they must navigate from their current position to the entrance door and find the doorbell.

[0352] This is a non-limiting example when a visually impaired user sends a start request to be directed to the building's entrance door.

[0353] exist Figure 14 and Figure 15 The image shows a portion of the live map 310, which includes: - Buildings and their components: each step, railing, each of the two handrails, entrance door 84 with components such as door lock, door handle and doorbell 841 considered as the initial point of interest, and doorbell 841 considered as the further point of interest.

[0354] - Dog 83 was identified as biological Ln.

[0355] - Walkable area WA 942, which includes a sidewalk and steps leading to the building entrance.

[0356] - No-walking area NA 941, which includes streets.

[0357] - Conditionally walkable area 943: two pedestrian crossings 832, each with a corresponding traffic light 831.

[0358] Examples of pre-defined requirements for walkable areas in terms of geometry include: the height of the sidewalk must not exceed 7cm, the distance from the guardrail must not exceed 0.5m, the distance from the edge of the sidewalk must not exceed 0.5m, and the height of the virtual cuboid is 2.20m, which is 40cm more than the height of a visually impaired person (1.80m).

[0359] An example of the static and dynamic physical relationship Rn is the relationship created in the real-time map 310 by the processing and control unit 3 for associating the color of traffic light 831 with the conditional status of conditionally walkable area 943: if the color is green, area 943 is walkable; and if the color is red, area 943 is not walkable.

[0360] A non-restrictive example of a conditionally walkable area (CWA) is represented by two pedestrian crossings 832 equipped with traffic lights 831. In a fixed, predetermined walkable area requirement, the street is defined as a non-walkable area NA. When approaching a pedestrian crossing 832, if there is no traffic light, it is pre-defined as a walkable area 942, and if there is a traffic light, it is pre-defined as a conditionally walkable area 943, meaning it is walkable only when traffic light 831 is green. This is an example of at least one predictable conditionally walkable area requirement, as the color of the traffic light can predictably change from red to green and from green to red.

[0361] When visually impaired user 1 sent the activation request, he was on the sidewalk of the building.

[0362] In several embodiments of the invention, the entrance gate 84 is already in the live map 310 because it was previously added to the live map in step 2 of the method.

[0363] In another embodiment of the invention, where the visually impaired user has just gotten out of a taxi to a completely new place and therefore the entrance door 84 has never been added to the live map 310 before, and thus the entrance door 84 is not yet in the live map 310 at the time the start request is sent, the navigation manager subunit 33 determines a wander path WP to repeatedly refocus the field of view 20, while repeating steps S1 and S2 until the entrance door 84 is found and stored in the live map 310.

[0364] In another embodiment of the invention, where a visually impaired user has just alighted from a taxi at a new location with two adjacent entrance doors 84-01 and 84-02, and entrance door 84 is unknown to the visually impaired user, the visually impaired user sends a request to the navigation manager subunit 33 to locate the "entrance door." The navigation manager subunit 33 then queries the real-time map 310 to locate entrance doors in the area of ​​interest near the visually impaired user and finds two entrance doors 84-01 and correspondingly 84-02.

[0365] If the two entrance doors 84-01 and 84-02 are not yet stored in the live map 310, the navigation manager subunit 33 determines the wander path WP until the entrance doors 84-01 and 84-02 are found and stored in the live map 310.

[0366] After locating the two entrance doors 84-01 and 84-02 and storing them in the live map 310, the navigation manager subunit 33 represents each of the two entrance doors as a corresponding spatialized sound and transmits them to the feedback unit 4 via the feedback manager subunit 35. The visually impaired user then selects one of the entrance doors 84-01 and 84-02 as the entrance door 84 that constitutes his initial point of focus.

[0367] In S3, the navigation manager subunit 33 determines a single navigation path Pn, which is the initial navigation path 911 for a visually impaired user to navigate from his standing point to the entrance door 84. The preferred navigation path SP is therefore the initial navigation path 911.

[0368] When the visually impaired user 1 navigates along the initial navigation path 911, the sensory unit 2 senses the dog 83.

[0369] The following are examples of how to detect a dog's aggression: Whether the dog is barking is sensed by the object sound feature fusion module 308. - Whether the dog has an aggressive expression on its face is sensed by the object 2D feature extraction module 306. - Whether the dog is moving or trembling because of anger is sensed by the object 3D feature fusion module 307.

[0370] Since the data sensed by the basic sensor and, where applicable, by additional sensors are fused and then sent to the live map subunit 31 for inclusion in the live map 310, the navigation manager subunit 33 checks at least two navigation path requirements when querying the live map 310 and detects that the non-aggression requirement is not met. For this reason, the navigation manager subunit 33 determines an auxiliary navigation path 912 toward the same initial point of interest PI 84. The preferred navigation path SP is now the auxiliary navigation path 912, which avoids a hostile reaction from the dog 83.

[0371] refer to Figure 14 , Figure 15 and Figure 16 The auxiliary path 912 must use two pedestrian crossings 832 (which are examples of conditional walking areas CWA 943) to cross the road (which are examples of non-walkable areas NA).

[0372] When a visually impaired user 1 approaches the first pedestrian crossing 832, the relationship manager subunit 32 determines the conditional region 943 based on the color of the first traffic light 831.

[0373] Therefore, the relationship manager subunit 32 establishes conditional relationships in the real-time map 310, associating the color of the first traffic light 831 with the conditional status of the first pedestrian crossing 832.

[0374] When traffic light 831 turns green, conditionally walkable area 943 is considered walkable, and visually impaired user 1 receives associated navigation guidance instructions to continue navigating on auxiliary path 912.

[0375] Repeat the same operation on the second pedestrian crossing 832.

[0376] Figure 16 The illustration depicts a visually impaired user 1 navigating on auxiliary path 912, waiting at the second pedestrian crossing 832 for the second traffic light 831 to turn green.

[0377] refer to Figure 17 A visually impaired user has crossed two crosswalks 832 and is approaching the initial point of interest, i.e., entrance door 84. The Relationship Manager subunit 32 determines a new parent-child relationship between the initial point of interest (door 84) and the further point of interest (doorbell 841). Therefore, a new item 841 corresponding to doorbell 841 is created in the live map 310, and the new parent-child relationship between door 84 and doorbell 841 is created by the Relationship Manager subunit 32 and updated in the live map 310. Doorbell 841 replaces door 84 as the point of interest PI.

[0378] exist Figure 15 and Figure 16 In the diagram, a walkable tunnel 922 is shown, having a cross-section of approximately 1 m, which is about 0.5 m to the left and about 0.5 m to the right from the auxiliary navigation path 912, which is represented as a line.

[0379] If the preferred navigation path SP passes through an indoor space such as an apartment, the cross-section is typically small, for example, about 0.5, which is about 0.25m to the left and about 0.25m to the right from the preferred navigation path SP.

[0380] The following combination Figure 15 and Figure 16 To illustrate the details of guiding a visually impaired user through a walkable tunnel 922. In this example, the wearable device 1 is provided with a left haptic feedback actuator 411, a right haptic feedback actuator 411, and a central haptic feedback actuator 412, each of which includes one or more linear resonant actuators.

[0381] In this example, a three-dimensional walkable tunnel T is selected to receive associated navigation guidance instructions.

[0382] The visually impaired user receives the start command via a first tactile cue (which is time-sensitive) and begins navigation.

[0383] The feedback manager subunit 35 will strive to keep the visually impaired user on the preferred navigation path SP and within the boundaries of the walkable tunnel 922 by providing directional tactile cues.

[0384] If a visually impaired user is too close to the left side of the walkable tunnel 922 while navigating, a second tactile cue (which is spatiotemporal) is received by the left feedback actuator 411. The linear resonant actuator of the left feedback actuator 411 outputs rapid, ordered vibrations, one after another, in the direction the visually impaired user must reorient, i.e., to the right, giving the visually impaired user the tactile sensation of someone pulling their forehead to the right. The duration, periodicity, intensity, or frequency of the second tactile cue's vibrations vary proportionally to the proximity to the left side of the walkable tunnel 922.

[0385] If a visually impaired user is too close to the right side of the walkable tunnel 922 while navigating, a third tactile cue is received by the right feedback actuator 412—it is spatiotemporal and has the same configuration as the second tactile cue, except that it indicates left rather than right as the direction for reorientation. The duration, periodicity, intensity, or frequency of the vibration of the third tactile cue varies proportionally to the proximity to the right side of the walkable tunnel 922.

[0386] The user is guided forward via a fourth tactile cue—which is spatiotemporal. This fourth tactile cue is received by a central feedback actuator 413. The duration, periodicity, intensity, or frequency of the vibrations of the fourth tactile cue are proportional to the speed a visually impaired user should have during navigation. If the visually impaired user must reorient their direction of movement on the horizontal plane during navigation, for example… Figure 16 As shown, when he reaches pedestrian crossing 943 and turns right, a fifth tactile pattern cue is received—it is spatiotemporal and is achieved through a central feedback actuator 413. The duration, periodicity, intensity, or frequency of the vibration of the fifth tactile cue varies proportionally to the angle of the turn.

[0387] If a visually impaired user needs to reorient their direction of movement in the vertical plane while navigating, for example, when the visually impaired user has crossed sidewalk 943 and is approaching the steps of a building and needs to climb some steps, a sixth tactile cue is received—it is spatiotemporal and is achieved through a central feedback actuator 413. The duration, periodicity, intensity, or frequency of the vibration of the sixth tactile cue varies proportionally to the amount of movement required by the visually impaired user.

[0388] When a visually impaired user reaches crosswalk 832 (at Figure 15 and Figure 16 When traffic light 831 is red (as shown in the diagram), a seventh tactile cue is received—it is time-sensitive, corresponding to a temporary pause in the associated navigation guidance instruction implemented via the central feedback actuator 413. Then, when traffic light 831 turns green, the seventh tactile cue is received again, this time corresponding to the resumption of navigation.

[0389] Upon reaching the point of interest PI, the eighth tactile pattern cue (which is time-dependent) signals the end of navigation, which is received from the central feedback actuator 413.

[0390] Other types of tactile pattern cues can also be defined to suit navigation situations or user requirements. For example, if a visually impaired user is centered within the walkable tunnel 922 of the assisted navigation path 912 while navigating, the right feedback tactile actuator 412 and the left feedback tactile actuator 411 may not present any type of tactile pattern cue, or they may present a ninth type of tactile pattern cue on both sides of the forehead to signal to the visually impaired user that they are navigating in a centered manner within the walkable tunnel 922.

[0391] Example group number 2 For example, consider the following description: A visually impaired user enters an unfamiliar room with four windows 85, namely window 85-1, window 85-2, window 85-3, and window 85-4, and he wants to open one of the four windows 85. The potential point of concern (PPI) is a group of four windows as at least one object On selected from a plurality of objects On.

[0392] The term "potential" means that any window in the room could potentially be selected as the initial point of interest (PI).

[0393] refer to Figures 18 to 28 In substep S.3-0.1, the visually impaired user sends a request for information named “window” as a potential point of concern (PPI) because he wants to find out more details about the window from the room in order to select the initial point of concern (PI).

[0394] Those skilled in the art will understand that the described examples are applicable to any kind of object On, and refer to the categories applicable to biological Ln.

[0395] In sub-step S.3.-0.2, the sound representation subunit 36 ​​represents each of the four windows 85 as a corresponding spatialized sound, namely the first spatialized sound S86-1, the second spatialized sound S86-2, the third spatialized sound S86-3, and the fourth spatialized sound S86-4, and transmits these four spatialized sounds to the feedback unit 4 via the feedback manager subunit 35 (when the sound representation subunit 36 ​​is not part of the feedback manager subunit 35).

[0396] In substep S.3-0.3, the visually impaired user selects one of the four windows 85-1, 85-2, 85-3, and 85-4 as the initial point of interest (PI) and sends the corresponding selection request, just like any other selection request.

[0397] The sound representation of sub-step S.3-0.2 is as follows (see reference). Figures 18 to 28 The examples are illustrated for a single window or for two windows in a group of four windows 85-1, 85-2, 85-3, and 85-4. Those skilled in the art will understand that the teachings of the invention are not limited to this group of examples and that reference to window 85 refers to any one of the four windows 85-1, 85-2, 85-3, and 85-4. Similarly, reference to spatialized sound S86 refers to each of the corresponding spatialized sounds S86-1, S86-2, S86-3, and S86-4.

[0398] In Example 2-1, refer to Figure 18 In a preferred embodiment, the sound representation module 36 extracts specific information about the window 85 as a potential point of interest (PPI) from the live map 310, encodes it into a spatialized sound S86 that is perceived by the visually impaired user as emanating from the location of the window 85, and sends the spatialized sound S86 to the visually impaired user via the auditory feedback actuator 42.

[0399] To illustrate additional features of the window 85 representing the potential point of interest (PPI), such as the window size or material density, the navigation manager 33 further encodes the spatialized sound S86 into a spatialized sound having a specific frequency S86f, a specific time period S86t, and / or a specific pulse S86p, which is not represented graphically, the frequency S86f, the specific time period S86t, and / or the pulse S86p corresponding to the additional features.

[0400] In Examples 2-2 and 2-3, refer to Figure 19A and accordingly Figure 19BIn another preferred embodiment, the sound representation subunit 36 ​​encodes specific information about the selected window 85 into a spatialized sound S86, which has an exaggerated or “scaled” representation of the location of the potential point of interest 85. For example, if the potential point of interest 85 is at a distance of 10 meters, it will be represented in the spatialized sound S861 as being at a distance of, for example, 1 meter. This scaling or exaggeration effect applies both to elevation (e.g., ...) and height (e.g., ...). Figure 19A As shown in the figure, and at the azimuth angle (such as Figure 19B (As shown in the image) is presented to visually impaired users.

[0401] In Example 2-4, refer to Figure 20 In another preferred embodiment, the visually impaired user sends a request for audio representation of two windows 85-1 and 85-2 located at unequal distances from the visually impaired user.

[0402] The sound representation subunit 36 ​​encodes specific information from the selected windows 85-1 and 85-2 from the live map 310 into spatialized sounds S86f-1 and S86f-2 with different frequency characteristics according to the distance between windows 85-1 and 85-2 and the visually impaired user, and sends the encoded spatialized sounds S86f-1 and S86f-2 to the visually impaired user.

[0403] Therefore, for example, the corresponding audio cues of the spatialized sounds S86f-1 and S86f-2, which correspond to the additional features of window 85 sent to the visually impaired user, differ in frequency: for window 85-2, the cues last longer and / or are more repetitive compared to the window that is closer to the visually impaired user (correspondingly window 85-1).

[0404] In Example 2-5, refer to Figure 21 In another preferred embodiment, the visually impaired user sends a request to indicate the edges of window 85 aloud. For simplicity, it is chosen that only two ends are indicated, corresponding to ends 85-E1 and 85-E2. Those skilled in the art will understand that the same indication applies to all ends of the window, depending on the shape of the window.

[0405] The sound representation subunit 36 ​​extracts specific information about the selected window 85 from the real-world map 310 and encodes it into spatialized sounds S86P-E1 and S86P-E2 corresponding to the window ends 85-1E and 85-E2. The spatialized sounds S86P-E1 and S86P-E2 have different encoding characteristics, depending specifically on the distance of each of the two selected ends relative to the visually impaired user. The distance can be measured with respect to azimuth, elevation, or the extent of window 85, or any combination thereof.

[0406] See Example 2-6. Figure 22 In another preferred embodiment, the visually impaired user sends a request made via the user command interface 5, through the user command interface subunit 34, to the sound representation subunit 36 ​​for representation of the dimensions of window 85, which will be a potential point of interest (PPI). The sound representation subunit 36 ​​encodes specific information about the dimensions of the selected window 85 extracted from the live map 310 into sounds and sends them to the visually impaired user via the auditory feedback actuator 42: a temporal spatialized sound S86P, representing a point sound along one of the three spatial dimensions between the selected ends of window 85; or a linear sound S86L moving along a straight path from end 85-E1 to end 85-E2. The same operation is performed on the other ends of window 85, particularly on 85-E3 and 85-E4 when window 85 is rectangular (not graphically represented).

[0407] The dimensions of window 85 are measured by means of an instrument known from the prior art along the corresponding spatial dimensions between the ends 85-E1 and 85-E2, 85-E3 and 85-E4 of window 85.

[0408] In the examples from 2-7 to 2-10, refer to Figure 23-26 In another preferred embodiment, the visually impaired user sends a request for a representation of the shape of window 85 or a portion of a window. In the examples from 2-7 to 2-10, window 85 may be a non-rectangular decorative window, or a mirror, or a decorative part of a door. For simplicity, it will be referred to as window 85.

[0409] The sound representation subunit 36 ​​extracts specific information from the real-time map 310 and encodes it into a temporal spatialized sound S86 representing the shape of the window 85.

[0410] In Example 2-7, refer to Figure 23 The shape of window 85 is represented by two spatialized point sounds S86P1 and S86P2, each of which virtually moves simultaneously from a starting point t0 placed on the vertical axis of symmetry of window 85 to an ending point t0 on half of the outline of window 85. final .

[0411] In Example 2-8, refer to Figure 24 The shape of window 85 is represented by two spatialized sounds S86f1 and S86f2, which virtually move along the outline of the window from the starting point t0 to the ending point t. final Meanwhile, the spatialized sounds S86f1 and S86f2 are encoded to have different frequencies, which depend on the distance of the window portion 85 traveled by the spatialized sound relative to the user.

[0412] In Example 2-9, refer to Figure 25 The shape of window 85 is represented by a single spatialized sound S86, which virtually moves from a starting point t0 on the outline of window portion 85 until it returns to the starting point t0.

[0413] In Example 2-10, refer to Figure 26 The shape of window 85 is represented by a single spatialized sound S86, which virtually moves in an angular pattern within the space between the inner and outer contours of window 85.

[0414] In Examples 2.11 and 2.12, refer to Figure 27 and Figure 28 In another preferred embodiment, the selected window may represent a window or a door. In both examples, the window has at least one inner frame for decorative purposes or for a technical function. The outline of the inner frame is important in these preferred embodiments, for example because of the risk of injury, or because the handle of the door or window is on the inner frame.

[0415] In Example 2.11, refer to Figure 27 There are two windows, 85-1 and 85-2, located at different distances relative to the visually impaired user. These two windows together represent a potential point of concern. They could be windows or doors. They represent a potential point of concern together because the actions the visually impaired user needs to take will involve both. For example, there are two separate windows that must both be open, or a corridor with two doors that the visually impaired user must pass through.

[0416] The two windows, 85-1 and 85-2, are separated by open spaces of different sizes (e.g., 5-10cm in the case of a window, or 1-2m in the case of a door). Visually impaired users are positioned closer to window 851.

[0417] A visually impaired user sends a request for an audio representation of the distance between the open space between the two windows 85-1 and 85-2, as well as for the shape of the inner frames of the two windows 85-1 and 85-2.

[0418] The sound representation subunit 36 ​​extracts specific information about the two windows 85-1 and 85-2 from the real-world map 310 and encodes it into spatialized sounds S86t-1 and S86t-2 with different temporal characteristics for representing the shape of the inner frames of the two windows 85-1 and 85-2.

[0419] like Figure 27The window 85-1, positioned closer to the visually impaired user, is represented by two spatialized sounds, S86t1-1 and S86t2-1. One spatialized sound, S86t1-1, represents the outer contour of window 85-1, and the other, S86t2-1, represents the inner contour of window 85-1. Each of the spatialized sounds S86t1-1 and S86t2-1 contains two other spatialized sounds that begin simultaneously: S86t11-1 and S86t12-1 represent the outer contour of window 85-1, and S86t21-1 and S86t21-1 represent the inner contour of window 85-1.

[0420] Because of the open space between the two windows 85-1 and 85-2, the window 85-1, which is closer to the visually impaired user, acts as a barrier for detecting detailed information about the second window 85-2. Therefore, the sound representation subunit 36 ​​can only output a simplified spatialized sound S86t-2 corresponding to the three-dimensional position and vertical dimensions of the window 85-2.

[0421] In Example 2-12, refer to Figure 28 In another preferred embodiment, the visually impaired user sends a request for an aural representation of the shape details of the inner frame of window 85.

[0422] For simplicity, Figure 28 The diagram illustrates a rectangular shape, but it can be any geometric structure. The sound representation subunit 36 ​​extracts specific information from the live map 310 and encodes it into two spatialized sounds S861 and S862 that start simultaneously. These two spatialized sounds virtually move in an angular pattern within the space between the outline of the inner frame and the outer outline of the window 85.

[0423] While the methods and systems have been described in detail with reference to preferred embodiments, those skilled in the art will understand that modifications may be made to suit specific circumstances without departing from the spirit and scope of the teachings of the present invention.

[0424] The reference numerals appearing in the claims are for illustrative purposes only and shall not limit the scope of the claims.

[0425] Reference List [1]-US 3594823 [2]-US 9915545 [3]-US 9 629 774 [4], Xiaochen Zhang et al. “Wearable Indoor Navigation System withContext Based Decision Making for Visually Impaired” [4]International Journalof Advanced Robotics and Automation [5] US2016 / 0033280 A1。

Claims

1. A computer-based method suitable for assisting visually impaired users in moving using a wearable device (1), the computer-based method comprising the following steps: S1 The sensory unit (2) of the wearable device (1) acquires data from the environment of the visually impaired user, senses from the field of view (20), and sends the acquired data to the sensory fusion subunit (30) of the processing and control unit (3) of the wearable device (1). S2 The acquired data is fused by the sensory fusion subunit (30), and the fused data is sent to the real-time map subunit (31) of the processing and control unit (3). A real-time map (310) is created, repeatedly updated, and stored through a real-time map subunit (31), the real-time map comprising: a) A real-time map determination based on the fused data received at the processing and control unit (3), the real-time map determination including: - The position and orientation of sensory unit (2), - Multiple objects (On) - Multiple organisms (Ln) The real-time map determination also includes using a positioning algorithm applied to the data received from the sensory unit (2) and the data from the real-time map (310) to repeatedly locate the position and orientation of the sensory unit (2) relative to the plurality of objects (On) and correspondingly relative to the plurality of organisms (Ln) on the real-time map (310) through the positioning module (301), and repeatedly sending the positioning data of the position and orientation of the sensory unit (2) to the walkable area detection module (302) of the sensory fusion subunit (30). b) Based on the real-time map generated from the free region (A), the free region is defined as the entire area on the ground not occupied by the plurality of objects (On) and the plurality of creatures (Ln), and the free region (A) includes: - A walkable area (WA) that meets a fixed set of predetermined walkable area requirements. The determination of the free region (A) is based on: - Data received from sensory unit (2), - Data received from the positioning module (301), - The set of fixed, predetermined walkable area requirements, In this process, the updated free area (A) is repeatedly sent to the live map (310), and the updated live map (310) is repeatedly stored in the memory M. S3 Automatically or in response to a first request from a visually impaired user, the navigation manager subunit (33) of the processing and control unit (3) determines, repeatedly updates, and stores at least one navigation path (Pn) and associated navigation guidance instructions for the visually impaired user to navigate from the current position of the sensory unit (2) to a point of interest (PI) selected from the plurality of objects (On) and / or the plurality of organisms (Ln). Any request from a visually impaired user is made using a tactile device (51) or an audio device (52) via a user command interface (5), and the request is received by the navigation manager subunit (33) via the user command interface manager subunit (34) of the processing and control unit (3). And the associated navigation guidance instructions are transmitted from the navigation manager subunit (33) to the feedback manager subunit (35) of the processing and control unit (3). S4 Using a guidance pattern for transmitting each associated navigation guidance instruction, guidance is provided to visually impaired users via a feedback manager subunit (35). Each navigation instruction includes tactile and / or auditory cues sent by the feedback manager subunit (35) to a feedback unit (4) of the processing and control unit (3), the feedback unit (4) comprising: - Configured as a tactile feedback actuator (41) placed on the head of a visually impaired user, and / or - Configured for placement in one or both ears of a visually impaired user (42). The guidance mode for each associated navigation guidance instruction is selected by the visually impaired user via the user command interface (5) and by a user command, which is received by the feedback manager subunit (35) via the user command interface manager subunit (34). Its features are: - The live map determination in S2 also includes: c) A reality map determination based on multiple relationships (Rn) between the multiple objects (On) and / or the multiple organisms (Ln) received from the relationship manager subunit (32) of the processing and control unit (3), wherein the multiple relationships (Rn) are repeatedly generated and updated by the relationship manager subunit (32) based on data obtained from the reality map (310), including applying a set of predetermined relationship requirements. Among them, the updated multiple relations (Rn) are repeatedly sent to the live map (310), and The localization algorithm is further applied to the multiple relations (Rn). And its characteristics - In S2 b), the determination of the real-world map generated based on the free region (A) also includes: - A conditionally walkable area (CWA) that satisfies the set of fixed, predetermined walkable area requirements and at least one predictable conditionally walkable area requirement. - and its characteristics S2 b) also includes repeatedly determining conditionally traversable regions (CWAs) by the traversable region detection module (302) simultaneously with determining traversable regions, including based on at least one predictable conditionally traversable region requirement calculated and stored in memory (M). And its characteristics - S3 also includes the following after determining the at least one navigation path (Pn): a) Automatically or in response to a second request from a visually impaired user, repeatedly select a preferred navigation path (SP) from the at least one navigation path (Pn): i) Passing through walkable areas (WA) and / or through conditionally walkable areas (CWA), and ii) It must meet a set of safety requirements, including no-collision and non-aggression requirements. The navigation manager subunit (33) further transmits the preferred navigation path (SP) to the feedback manager subunit (35) of the processing and control unit (3), and When the preferred navigation path (SP) passes through a conditionally traversable area (CWA), the navigation manager subunit (33) sends a navigation guidance instruction corresponding to the at least one predictable conditionally traversable area requirement to the feedback manager subunit. b) Provide guidance in S4 via the feedback manager subunit (35) along the preferred navigation path (SP).

2. The computer implementation method as described in claim 1, wherein, The real-time map (310) is updated by the sensory fusion subunit using the Simultaneous Localization and Mapping (SLAM) algorithm.

3. The computer implementation method as described in claim 1 or 2, include: In S3, when the point of interest (PI) is unknown to the visually impaired user, and before determining, iteratively updating, and storing at least one navigation path (Pn), sub-step 3-0 is performed: S. 3-0.

1. Regarding at least one object (On) selected from the plurality of objects (On) or at least one organism (Ln) selected from the plurality of organisms (Ln), the information request of the visually impaired user is sent to the sound representation subunit (36) of the processing and control unit (3). S. 3-0.

2. The sound representation subunit (36) of the processing and control unit 3 extracts information about at least one selected specific object (On) or at least one specific organism (Ln) from the live map (310), represents the extracted information as corresponding spatialized sound, and transmits the spatialized sound to the visually impaired user through the feedback unit (4). S. 3-0.

3. The visually impaired user selects a point of interest (PI) from the plurality of objects (On) or from the plurality of organisms (Ln) and sends the corresponding selection request to the navigation manager subunit (33).

4. The computer-implemented method as described in any one of claims 1 to 2, comprising: In S3, a preliminary sub-step is performed before determining, iteratively updating, and storing at least one navigation path (Pn): S. 3-1. The navigation manager (33) determines the wander path (WP) and the associated navigation guidance instructions for the visually impaired user, and sends the wander path (WP) and the associated navigation guidance instructions to the feedback manager subunit (35).

5. The computer-implemented method as described in any one of claims 1 to 2, in, The tactile cues vary in duration, periodicity, intensity, or frequency of vibration according to a predetermined preferred navigation path (SP) complexity criterion, and The audio prompts vary in frequency, duration, repetition intensity, or 3D spatial virtualization based on a pre-determined standard for the complexity of the preferred navigation path.

6. The computer-implemented method as described in any one of claims 1 to 2, wherein, A three-dimensional walkable tunnel (T) is defined as a virtual tunnel with a predetermined cross-section, the virtual tunnel using the preferred navigation path (SP) as its horizontal longitudinal axis, and wherein the guidance mode further includes specific tactile cues sent to the visually impaired user as the visually impaired user approaches a virtual wall of the walkable tunnel (T).

7. The computer-implemented method as described in any one of claims 1 to 2, in, The preferred navigation path (SP) is divided into predetermined segments defined by multiple mile markers (93), and The guidance mode includes tactile and / or auditory cues that signal the location of at least one next mile marker (93), providing associated navigation guidance instructions to visually impaired users from the current mile marker (93) to the subsequent mile marker (93), and wherein the length of the predetermined segment varies based on the complexity and length of the preferred navigation path (SP).

8. The computer-implemented method as described in any one of claims 1 to 2, in, The guidance modes of S4 include tactile and / or auditory cues that signal directions on the preferred navigation path (SP). The direction of the preferred navigation path (SP) is determined by a line defined by the intersection of the origin of the sensory unit (2) and the preferred navigation path (SP) with a circle whose origin is located at the sensory unit (2) and whose radius (r) is a predetermined length. The auditory cues indicating the direction on the preferred navigation path (SP) originate from a spatialized sound source (S) located at a predetermined first distance (d1) relative to the sensory unit (2).

9. The computer-implemented method as described in any one of claims 1 to 2, in, The auditory cues are spatialized sounds originating from a spatialized sound source (S) that virtually travels from the position of the sensory unit (2) along a predetermined second distance (d2) on a preferred navigation path (SP) until the spatialized sound source (S) reaches the end of the predetermined second distance (d2) and returns to the position of the sensory unit (2).

10. A wearable device (1) suitable for assisting visually impaired users in movement, the wearable device comprising: - A sensory unit (2) configured to be placed on the head of a visually impaired user, the sensory unit comprising: - Camera (21), - Depth sensor (22), - Inertial measurement unit (23). - Sound positioning sensor (24) - Processing and control unit (3), the processing and control unit comprising: - Sensory fusion subunit (30), the sensory fusion subunit comprising: - Positioning module (301). - Walkable area detection module (302). - Directional calculation module (303). - Sound direction positioning module (304). - Sound classification module (305). - Object 2D feature extraction module (306). - Object 3D Feature Fusion Module (307). - Object sound feature fusion module (308). - Live map sub-unit (31). - Relationship Manager Subunit (32) - Navigation Manager Subunit (33). - User command interface manager subunit (34). - Feedback Manager Subunit (35) - Sound representation subunit (36). - A feedback unit (4) configured to be placed on the head of a visually impaired user, the feedback unit comprising: - Multiple haptic feedback actuators (41), said multiple haptic feedback actuators including: - Left tactile feedback actuator (411). - Right tactile feedback actuator (412). - Central haptic feedback actuator (413). - Multiple auditory feedback actuators (42), said multiple auditory feedback actuators including: - Left auditory feedback actuator (421). - Right auditory feedback actuator (422). - A user command interface (5) configured to be placed on the head of a visually impaired user, the user command interface including: - Multiple user-commanded haptic devices (51). - Multiple user-commanded audio appliances (52). - Power storage unit (6) - Memory (M), and - An electronic communication device that connects the sensing unit (2), processing and control unit (3), feedback unit (4), user command interface (5), power storage unit (6), and memory (M) via a communication protocol. The wearable device (1) is configured to apply the steps of the method according to any one of claims 1 to 9.

11. The wearable device of claim 10, wherein, The sensing unit (2) also includes at least one additional sensor from: - A global positioning sensor (25) configured to determine the absolute position of the sensing unit (2), and - A temperature sensor (26) configured to determine the temperature of the object (On) and the organism (Ln), and The sensory fusion subunit (30) also includes: - A relative to absolute conversion module (309-1), configured to fuse data from the object sound feature fusion module (308) with data about the absolute position of the sensory unit (2), and - Object temperature feature fusion module (309-2), which is configured to fuse data from object sound feature fusion module (308) with data about the temperature of object (On) and organism (Ln) and send the fused data to real-time map subunit (31).

12. A non-transitory computer-readable storage medium encoded with a computer program, said computer program comprising instructions executable by one or more processors, said instructions, upon such execution, causing said one or more processors to perform operations of the computer-implemented method according to any one of claims 1 to 9.

13. A computer program product comprising instructions that, when executed by a wearable device (1) according to any one of claims 10 or 11, cause the wearable device (1) to perform the steps of the computer-implemented method according to any one of claims 1 to 9.

14. A system for assisting a visually impaired user in mobility, the system comprising one or more processors and one or more non-transitory computer-readable storage devices, the non-transitory computer-readable storage devices comprising a storage medium as described in claim 12 containing instructions executable by the one or more processors to perform operations of a computer-implemented method according to any one of claims 1 to 9.

Images