Hearing-based assistive perception headset for the blind

Abstract

An auditory-based blind person auxiliary perception head-mounted device, comprising a head-mounted fixing structure, a depth camera, multi-channel earphones, a temperature sensor, a processor and a satellite navigation module, wherein the depth camera is located above the head-mounted fixing structure, the earphones are located on both sides of the head-mounted fixing structure, the temperature sensor and the satellite navigation module are located between the head-mounted fixing structure and the depth camera, and the processor is located at the back of the head-mounted fixing structure and connected with the depth camera, the earphones, the satellite navigation module and the temperature sensor respectively. The present application uses sound simulation to visually help the blind person to avoid obstacles in the method of augmented reality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of navigation equipment, and particularly relates to a hearing-based auxiliary perception head-mounted device for the blind. BACKGROUND

[0002] The existing blind obstacle avoidance technology mostly adopts the method of voice prompt to remind the blind, which needs the brain to further process the text, is not intuitive, and too much information conveyed by text is easy to cause confusion. SUMMARY

[0003] The present application aims at the existing problems in the prior art, and provides a hearing-based auxiliary perception head-mounted device for the blind, which uses sound simulation to intuitively help the blind avoid obstacles in an augmented reality method.

[0004] The present application is implemented through the following technical scheme:

[0005] The present application relates to a hearing-based auxiliary perception head-mounted device for the blind, which comprises a head-mounted fixing structure, a depth camera, multi-channel earphones, a temperature sensor, a processor and a satellite navigation module, wherein the depth camera is located above the head-mounted fixing structure, the earphones are located on both sides of the head-mounted fixing structure, the temperature sensor and the satellite navigation module are located between the head-mounted fixing structure and the depth camera, and the processor is located at the rear side of the head-mounted fixing structure and connected with the depth camera, the earphones, the satellite navigation module and the temperature sensor respectively.

[0006] The processor comprises an image processing module, a temperature acquisition module, a voice recognition module, a visual-auditory conversion module and a sound output module connected in sequence, wherein the image processing module receives the depth information collected by the depth camera and identifies the type, position, distance and size of the object from the depth information; the temperature acquisition module collects the temperature information of the object through the temperature sensor and transmits the temperature information to the visual-auditory conversion module; the voice recognition module acquires and recognizes the instructions issued by the blind; the visual-auditory conversion module converts the type, position, distance and size of the object into corresponding sound information of different timbre, frequency, phase difference and intensity and transmits the sound information to the sound output module; and the sound output module is correspondingly distributed to different sound channels according to the position, distance and size of the obstacle and plays the sound through the corresponding sound channels of the earphones.

[0007] The image processing module adopts, but is not limited to, a low-latency high-precision neural network to realize the identification of the type, position, distance and size of the object.

[0008] The voice recognition module identifies the instructions for switching the normal obstacle avoidance mode or the navigation mode and the destination instructions in the navigation mode.

[0009] The sound information includes: simulated obstacle sound, high and low temperature warning, road identification and guidance, and biological and non-biological information identification.

[0010] The transmission in the processor can be programmed in a high-level language, such as python and matlab, wherein the audio transmission adopts a data stream non-blocking manner.

[0011] The conversion includes: obstacle type identification and tone matching, obstacle distance and blind person and sound intensity matching, obstacle size and sound source number matching, and dynamic obstacle and sound change rule matching.

[0012] The distribution refers to generating a final output audio matrix Y by the sound output module, wherein the audio matrix Y includes a matrix Y for the object o and a matrix Y for the enhanced edge. e That is, Y=Y o +Y e Wherein, the matrix for the object approximates the sound of the surface sound source in a pixel-precision manner to more fully utilize the depth information, and the generated audio matrix has three-dimensional sound information of the object shape dimension for each object, so as to provide the user with richer experience through the multi-channel earphone channel dimension three-dimensional sound information; the matrix for the enhanced edge is obtained by enhancing the edge sound information.

[0013] The matrix for the object Wherein, N is the number of channels, and the audio vector of each channel M is the number of objects detected in the current frame, β(θ) is a direction angle function, and the elements of the attenuation matrix a is a normalization factor, and the elements of the distance matrix D ijk =||XYZ[i,k,:]-H[j,k,:]||2 wherein i=1,2...N,j=1,2,...M,k=1,2,...K,||·|| is the l2 norm of the matrix, the matrix D is accurate to a pixel, the length of the audio sequence y is t last , f s are the audio duration and the audio sampling rate respectively, represents the floor function, and the point column delay matrix for the audio sequence is The relative distance matrix D rel =D-min(D,1), min(D,1) is the minimum value of the D matrix in each row direction, and u is the sound velocity in the current environment.

[0014] When the audio matrix to be generated is Y∈R L×N Wherein, N is the number of channels, and the object position matrix returned by the depth camera and the color camera K is the maximum number of pixels occupied by an object in the image, that is, the number of pixels contained in the contour, and the precision of the XYZ matrix is the coordinates of each pixel point contained in each object. The modeled sound channel position matrix The distance matrix D is generated N×M×K , R is a real number, and the distance matrix is obtained from the sound pressure attenuation empirical formula The attenuation matrix caused by attenuation is obtained, wherein Q is a directivity factor considering the position of the point sound source in the room; r is the distance of the measuring point from the sound source, the environmental constant is the average sound absorption coefficient in the room; s is the total surface area in the room; since the bearing information is provided by the phase difference between the sound channels in this problem, only the difference in distance is concerned when processing the phase, so the distance matrix D is converted into a relative distance matrix, and it is obvious that the processed D rel Each column has an element of 0, corresponding to the reference sound channel in positioning. Based on the relative distance matrix, the point column delay matrix for the audio sequence is obtained, from the point column delay matrix Delay and the corresponding sound source sequence y.

[0015] The reinforced edge refers to regarding the edge of the object as a linear sound source, and the integrated sound field when the sound reaches the two ears is obtained by calculation. The algorithm can adopt the following steps: the pixel coordinates of the detected edge endpoints are mapped to the camera coordinates by the camera intrinsic matrix, and the actual coordinates are obtained by combining the depth information, that is, the edge line in the real world, and then the sound pressure function of the edge line sound source is obtained by integrating the sound pressure attenuation formula in the free sound field, so as to obtain the distance matrix, thereby generating the reinforced edge matrix Y e . Specifically, edge detection can be performed using Hough transform, that is, finding imperfect instances of candidate objects within a certain type of shape by performing a voting procedure in a parameter space, in which the candidate objects are obtained by calculating the local maximum value in the accumulator space constructed by the Hough transform algorithm.

[0016] The Hough transform is to detect straight lines (line segments) from black and white images, can tolerate gaps in feature boundary description, and is relatively unaffected by image noise, specifically:

[0017] ①Let the pixel coordinates of the detected edge endpoints be x A and x B , respectively, and the camera intrinsic matrix is where f x = f / dx, f y = f / dy, f is the focal length, (u0, v0) is the coordinate of the image coordinate system origin in the pixel coordinate system, and dx and dy are the physical sizes of each pixel in the x and y directions of the image plane, respectively; map the pixel coordinates to the camera coordinates XA , X B , multiply the depth Z obtained by the depth camera to obtain the coordinates in the real world, that is, an edge line in the real world;

[0018] 2. For the edge line, the sound pressure function of the edge line sound source is obtained by integrating the sound pressure attenuation formula in the free sound field: Wherein: d is the distance between the observation point and the center line of the edge line sound source, theta A , theta B The angle between the observation point and A, B, thereby generating the matrix Y of the reinforced edge e .

[0019] Technical effects

[0020] The present application solves the problem that the prior art can measure the position of the obstacle but cannot distinguish the specific type of the obstacle, and cannot measure the size and distance of the object. At the same time, the prior art can only measure static objects and cannot judge the motion state of dynamic objects. Compared with the prior art, the present application measures the size and distance of the obstacle in front of the blind person, and transmits the frequency and intensity of the corresponding sound to the blind person, so that the blind person can intuitively feel the physical state of the obstacle. The present application has a measurement module for dynamic obstacles, which can transmit the motion state of the dynamic obstacle to the blind person through the change of the sound, helping the blind person to avoid the dynamic obstacle. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 It is a perspective view of the present embodiment;

[0022] Figure 2 It is a front view of the present embodiment;

[0023] Figure 3 It is a rear view of the present embodiment

[0024] In the figure: head-mounted fixing structure 1, depth camera 2, earphone 3, temperature sensor 4, processor 5, power supply 6, GPS 7;

[0025] Figure 4 It is a device workflow diagram in normal obstacle avoidance mode;

[0026] Figure 5 It is an earphone sound channel distribution diagram;

[0027] Figure 6 It is an image processing result diagram of a specific embodiment;

[0028] Figure 7 It is an audio visualization result diagram of a specific embodiment. DETAILED DESCRIPTION

[0029] As Figure 1 andFigure 2 As shown, this embodiment includes: a head-mounted fixed structure 1, a depth camera 2, headphones 3, a temperature sensor 4, a processor 5, a power supply 6 and a satellite navigation module 7, wherein: the depth camera 2 is located above the head-mounted fixed structure 1, the headphones 3 are located on both sides of the head-mounted fixed structure 1, the temperature sensor 4 and the satellite navigation module 7 are located between the depth camera 2 and the headphones 3, the processor 5 and the power supply 6 are connected and are located on the back side of the head-mounted fixed structure 1 and are respectively connected to the depth camera 2, the headphones 3, the temperature sensor 4 and the satellite navigation module 7.

[0030] The depth camera 2 is RealSense TM Depth Camera D435i camera.

[0031] The earphone 3 is a physical 7.1 channel earphone, such as RAZERTIAMAT 7.1 V2 .

[0032] The satellite navigation module is GPS.

[0033] The processor 5 includes: an image processing module, a temperature acquisition module, a speech recognition module, a visual and auditory conversion module and a sound output module connected in sequence, wherein: the image processing module receives depth information and identifies the contours in the depth image to obtain object information and the object temperature information collected by the temperature acquisition module is transmitted to the visual and auditory conversion module; the speech recognition module recognizes the instructions issued by the blind; the visual and auditory conversion module corresponds the object information to the corresponding sound timbre, frequency, and the phase difference and intensity of the sound in different channels to the sound output module; the sound output module transmits the sound information to the corresponding channel of the earphone 3.

[0034] The object information includes: type, position, distance and size.

[0035] The sound information includes: simulated obstacle sounds, high and low temperature warnings, road recognition and guidance, and biological and non-biological information recognition.

[0036] The visual-auditory conversion module matches the obstacle type identification with the tone color, matches the distance between the obstacle and the blind person with the sound intensity, matches the obstacle size with the number of sound sources, and matches the dynamic obstacle with the sound variation rule, that is, the synthetic simulation sound is used to represent the object type, for example, wooden objects such as tables and chairs, which use dull wooden tone color; metal objects, which use bright metal tone color; objects that have inherent impressions for people, such as people, which use footstep sound to represent; computers, which use the sound of tapping the keyboard to represent. The distance between the object and the blind person and the sound intensity are in a linear relationship, the closer to the blind person, the stronger the sound intensity. The number of sound sources corresponding to an object depends on the size distribution of the object, for small objects such as a water cup, a mobile phone, and a computer, a central sound source is used, and for large objects such as a table, a bed, and a door, multi-point sound sources at the boundary and the center are used. For dynamic obstacles, the Doppler effect is applied, the speed of the object is represented by the frequency change of the sound, the closer to the blind person, the higher the sound frequency and the faster the speed, the farther away from the blind person, the lower the sound frequency and the slower the speed.

[0037] The sound output module outputs different sound channels according to the obstacle orientation, distance, size, and / or temperature, and the sound output module includes 8 sound channels of a multi-channel earphone, wherein: one is a center channel, the left and right sides each have 6 channels of front, side surround, and rear surround, and one is a bass channel; when the simulated object is in different orientations, the phase difference and intensity difference of the sound received by different sound channels are applied to different orientation simulation sound sources. For example: when the temperature sensor measures an object higher than 65℃, a high temperature warning is given to the blind person; when an object lower than 0℃ is measured, a low temperature warning is given to the blind person; for dynamic objects with a temperature between 25-45℃, they are considered as living beings and special sound is transmitted to the blind person.

[0038] The voice recognition module recognizes the instructions for switching between the normal obstacle avoidance mode and the navigation mode in standby mode, and recognizes the instructions for the blind person to input the destination in the navigation mode.

[0039] The normal obstacle avoidance mode refers to: determining the relative position of the user and the depth camera 2 through the head-mounted fixed structure 1, based on the resolution capability of the depth camera 2 for object orientation information, the visual information collected by the depth camera 2, and the temperature information collected by the temperature sensor 4, the sound information is converted after being processed by the processor 5, and is transmitted to the physical 7.1 sound channel earphone, the sound is transmitted to the blind person by using the high restoration of the physical 7.1 sound channel earphone for three-dimensional space sound, the type, specific orientation, and distance of the obstacle are determined by distinguishing the tone color, intensity, and frequency information of the sound, so as to achieve the effect of obstacle avoidance.

[0040] The navigation mode refers to that the blind person issues a destination instruction, a voice recognition module in the processor 5 recognizes the destination and transmits the destination to the satellite navigation module 7, the satellite navigation module plans a route after receiving the instruction and transmits the route to the processor, a sound output module of the processor simulates road sounds of the target route through earphones and transmits the road sounds to the blind person, and the navigation effect is realized.

[0041] Through specific actual experiments: in the ordinary obstacle avoidance mode, for indoor environment, the blind person wearing the device can understand the structure of the room such as walls, doors and windows through prompt sounds, and also know the layout in the room such as tables, chairs, televisions, computers and the like, and the details in the room such as the positions of a water cup and a mobile phone; for outdoor environment, the device can prompt the blind person about the positions of lane lines and trees, and especially for dynamic obstacles such as cars, bicycles and pedestrians, the device can prompt the blind person about the speed and direction of the dynamic obstacles, helping the blind person to walk while avoiding obstacles.

[0042] Actual implementation scene examples are shown in the drawings. Figure 6 As shown in the actual scene picture taken by the depth camera, the types and sizes are recognized, and the distances between the obstacles and the blind person are obtained through the depth image. The obtained information of the obstacles is calculated through the above algorithm, the required sound is transmitted to the earphones, and the sound is visualized as shown in the drawing. Figure 7 As shown in the drawing, the vertical coordinate corresponds to the amplitude of the sound, and the horizontal coordinate corresponds to the number of sound points, wherein: ② and ④ are the output sounds corresponding to the two people in the drawing, ①, ③ and ⑤ are the output sounds corresponding to a television, a chair and a mouse respectively, and ⑥ is the mixed output sound of the above five objects.

[0043] Compared with the prior art, the size and distance of the obstacles in front of the blind person are measured, and the frequency and intensity of the corresponding sound are transmitted to the blind person, so that the blind person can intuitively feel the state of the obstacles. The measuring module for dynamic obstacles in the application can transmit the motion state of the dynamic obstacles to the blind person through the change of the sound, helping the blind person to avoid the dynamic obstacles. In addition, the temperature measuring module is added to the application, which can warn the blind person of low temperature or high temperature; the function of distinguishing living beings from non-living beings is added to help the blind person to better understand the environment in front of them and increase a layer of protection. The navigation mode is added to the application, and the blind person selects the above ordinary obstacle avoidance mode or the navigation mode. In the navigation mode, the blind person inputs the destination through voice instruction, the device simulates the road sound of the target route to guide the blind person, and the navigation function is realized. The modules are connected in series and multi-thread programming is used, two ways of pure python and python+matlab mixed programming are adopted, the data stream non-blocking programming mode is used when audio is generated, and the low-latency and high-precision neural network is combined for target recognition, so that the user can get low latency, high precision and rich experience of multiple information.

[0044] The above specific embodiments can be partially adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, each implementation within the scope is subject to the present application.

Claims

1. A hearing-based head-mounted device for assistive perception for the blind, characterized in that: include: A head-mounted fixed structure, a depth camera, a multi-channel headset, a temperature sensor, a processor, and a satellite navigation module, wherein: the depth camera is located above the head-mounted fixed structure, the headsets are located on both sides of the head-mounted fixed structure, the temperature sensor and the satellite navigation module are located on the front side of the head-mounted fixed structure, and the processor is located on the back side of the head-mounted fixed structure and is respectively connected to the depth camera, headset, satellite navigation module, and temperature sensor; The processor includes: an image processing module, a temperature acquisition module, a speech recognition module, a visual and auditory conversion module, and a sound output module, which are connected in sequence. Among them, the image processing module receives depth information collected by the depth camera and identifies the type, direction, distance, and size of the object from it; the temperature acquisition module collects the temperature information of the object through the temperature sensor and transmits it to the visual and auditory conversion module; the speech recognition module obtains and recognizes the instructions issued by the blind person; the visual and auditory conversion module converts the type, direction, distance, and size of the object into corresponding sound information with different timbre, frequency, phase difference, and intensity, and transmits it to the sound output module. The sound output module allocates it to different channels according to the direction, distance, and size of the obstacle and plays it through the corresponding channels of the headphones; The conversion includes: obstacle type identification and timbre matching, obstacle and blind person distance matching and sound intensity matching, obstacle size matching and sound source number matching, dynamic obstacle matching and sound change pattern matching; The sound output module outputs different sound channels according to the obstacle's orientation, distance, size, and / or temperature. The sound output module includes: a multi-channel headset with a total of 8 sound channels, including: a center channel, 6 sound channels for front, side surround, and rear surround on each of the left and right sides, and a bass channel; when the simulated object is in different orientations, the phase difference and intensity difference of the sound received by the different sound channels are calculated, and the obtained correspondence is applied to the simulated sound sources in different orientations; The image processing module uses a neural network to recognize the type, position, distance and size of the object; The speech recognition module recognizes instructions for switching between normal obstacle avoidance mode and navigation mode and for identifying destinations in navigation mode; The distribution generates the final output audio matrix through the sound output module , the audio matrix Y includes the matrix for the object and a matrix that strengthens the edges , which is , where: the matrix for objects uses a pixel-accurate approach to approximate the sound source of a surface to make full use of depth information. The generated audio matrix contains stereo information of the object's shape dimension for each object, and then provides users with a richer experience through the stereo information of the multi-channel headphone channel dimension; the matrix for enhancing edges is obtained by enhancing edge sound information; The speech recognition module recognizes the command to switch between the normal obstacle avoidance mode and the navigation mode in standby mode, and recognizes the command for the blind person to input the destination in the navigation mode; The normal obstacle avoidance mode is as follows: the relative position of the user and the depth camera is determined by the head-mounted fixed structure. Based on the depth camera's ability to resolve object orientation information, the visual information collected by the depth camera and the temperature information collected by the temperature sensor are converted into sound information after processing by the processor and transmitted to the physical 7.1-channel headphones. The physical 7.1-channel headphones are used to highly restore three-dimensional spatial sound to transmit the sound to the blind person. By distinguishing the timbre, intensity, and frequency information of the sound, the type, specific direction, and distance of the obstacle are determined, thereby achieving the effect of obstacle avoidance. The navigation mode is as follows: the blind person issues a destination command, the voice recognition module in the processor identifies the destination and transmits it to the satellite navigation module, the satellite navigation module plans a route after receiving the command and transmits it to the processor, and the sound output module of the processor simulates the road sound of the target route through headphones and transmits it to the blind person to achieve the navigation effect; The conversion of the sound output module includes: obstacle type identification and timbre matching, obstacle and blind person distance matching and sound intensity matching, obstacle size matching and sound source number matching, dynamic obstacle matching and sound change pattern matching.

Images