A method for guiding the blind by hearing instead of vision

Abstract

A kind of hearing instead of vision's guide blind method relates to electronic information field.The image of about 3 meters in front is collected by camera, and is sent into image to music module simultaneously to carry out image to music conversion, first, the image RGB space is converted to HSV space, and the value of each pixel point H, S, V is obtained.The value of H, S, V is mapped to the parameter of midi format file of music, there are six mapping modes, and the best mapping mode is that H value is given to the pitch parameter of midi file, S value is given to the volume parameter of midi file, and V value is given to the delta-time parameter of midi file.The midi parameter construction is carried out by the obtained parameter value of midi file, and then the midi music file is obtained.The H, S, V value is matched with volume, pitch and delta-time parameter after mapping, the converted music is obtained, the human ear listens to this sound, judges the existence condition of the obstacle in front, and the purpose of guide blind is achieved.The hearing instead of vision algorithm is realized by programming, and cheap and effective assistance is provided for blind people.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electronic information, and relates to a new type of guide for the blind, which is based on hearing instead of vision, and uses image-to-sound (music) technology to convert the scene in front into sound (music) signals, which are heard by the blind to make obstacle judgments, thus providing convenient and low-cost guide services for the blind. BACKGROUND

[0002] Blind people lose their vision due to congenital diseases or accidental incidents, and from then on they can only rely on touch and hearing to perceive the world. For the blind, a large amount of image information is useless to them. Especially when walking outdoors, the blind often have difficulty adapting and need to constantly rely on others for help or use assistive tools.

[0003] Therefore, some technology workers have begun to research in the field of electronic assistance walking. This research is divided into two parts: one part focuses on positioning and navigation, determining the position and destination through satellite signals and other technologies and providing navigation services. However, these devices cannot handle unexpected situations. The other part of the research focuses on handling emergency situations. Although these devices can handle some unexpected situations, they are often expensive and cannot be popularized to all blind people. It is worth noting that guide dogs also serve the blind. However, due to the limited number and the need for layer-by-layer screening and reservation, it is difficult to apply. In summary, the current electronic assistance guide for the blind method is high in cost and complex in operation, and is not suitable for the blind.

[0004] Although the blind have lost their visual perception ability, their hearing is relatively more acute. The present application proposes a new idea of information conversion through hearing instead of vision, i.e. converting the image in front into sound (music), which is heard by the blind to make judgments, to help the blind judge the situation in front. In this way, not only the hearing potential of the blind can be utilized, but also the implementation cost can be reduced and the method can be easily popularized.

[0005] The present application proposes a new method of image-to-sound (music) conversion, which converts the scene in front into sound (music) signals, and the blind can hear the converted sound through hearing to judge whether there is an obstacle in front. Experimental results show that this method is simple to implement and has wide application and popularization value.

[0006] In addition, we have also designed a new low-cost hearing instead of vision guide for the blind verification device. This guide for the blind device is simple and practical in design, and can provide basic navigation functions, so that the blind can better move and perceive the surrounding environment in daily life. The development of this low-cost device helps to meet the needs of low-income blind people and provides them with more opportunities for independent and autonomous life.

[0007] Through these innovative methods and designs, we hope to provide affordable and effective assistive technology for the blind to help them overcome visual obstacles in their daily lives and improve their quality of life. These achievements have important social significance and practical application value. SUMMARY

[0008] To solve the above technical problems, this research proposes an innovative image-to-sound (music) method, which converts the image in front of the blind person into a sound signal, allowing them to rely on their hearing to determine the situation in front of them, thereby addressing the potential dangers that may arise when obstacles are encountered while traveling outdoors. At the same time, a low-cost guide verification device specifically designed for low-income blind people is also designed. The method is as follows:

[0009] An image-to-sound method, where the sound signal is in midi format, which is the electronic musical instrument digital interface file format, widely used in electronic musical instruments. The specific idea is:

[0010] ① Convert the RGB color space of the image to HSV color space through a formula, as follows:

[0011] c max =max(R',G',B')

[0012] c min =min(R',G',B')

[0013] △=c max -c min

[0014]

[0015]

[0016]

[0017] V=c max

[0018] Where: H, S, and V correspond to the hue, saturation, and brightness values of the image; R', G', and B' are the normalized RGB values, Cmax and Cmin represent the maximum and minimum values of R, G, and B after normalization, H' and △ are intermediate quantities; H' represents the pre-computed quantity of the H component. Δ: represents the difference between the maximum and minimum values of the normalized R, G, and B color channels.

[0019] ② Conversion of image information to midi format parameters of music information

[0020] H, S, V three image features are mapped to three parameters of midi file of music: delta-time value, pitch and volume, and then pixel points are converted to notes in midi file; H value range is 0-360, S value range is 0-1, V value range is 0-1; pitch and volume value range is 0-127, and delta-time parameter can be any positive value, here we adopt one byte storage amount, i.e. 0-127, to represent delta-time parameter value range.

[0021] Image three component information is mapped to sound (music) three parameter information, and according to different mapping methods, finally six different mapped sounds are obtained, which are respectively:

[0022] 1) H value is mapped to midi file delta-time value, S value is mapped to midi file pitch, and V value is mapped to midi file volume;

[0023] 2) H value is mapped to midi file delta-time value, S value is mapped to midi file volume, and V value is mapped to midi file pitch;

[0024] 3) H value is mapped to midi file pitch, S value is mapped to midi file delta-time value, and V value is mapped to midi file volume;

[0025] 4) H value is mapped to midi file pitch, S value is mapped to midi file volume, and V value is mapped to midi file delta-time value;

[0026] 5) H value is mapped to midi file volume, S value is mapped to midi file pitch, and V value is mapped to midi file delta-time value;

[0027] 6) H value is mapped to midi file volume, S value is mapped to midi file delta-time value, and V value is mapped to midi file pitch;

[0028] Specific mapping method is that H, S, V value is matched and mapped with volume, pitch and delta-time parameter; wherein p, t, d respectively represent converted music pitch, volume and delta-time value.

[0029] 1) H, S, V are mapped with converted music delta-time, pitch and volume value:

[0030] d = (127 / 360) x H, p = 127 x S, t = 127 x V;

[0031] 2) H, S, V are mapped to delta-time, volume, pitch values of the converted music:

[0032] d = (127 / 360) x H, t = 127 x S, p = 127 x V;

[0033] 3) H, S, V are mapped to pitch, delta-time, volume values of the converted music:

[0034] p = (127 / 360) x H, d = 127 x S, t = 127 x V;

[0035] 4) H, S, V are mapped to pitch, volume, delta-time values of the converted music:

[0036] p = (127 / 360) x H, t = 127 x S, d = 127 x V;

[0037] 5) H, S, V are mapped to volume, pitch, delta-time values of the converted music:

[0038] t = (127 / 360) x H, p = 127 x S, d = 127 x V;

[0039] 6) H, S, V are mapped to volume, delta-time, pitch values of the converted music:

[0040] t = (127 / 360) x H, d = 127 x S, p = 127 x V;

[0041] ③ Image to sound conversion

[0042] The image pixel point values are converted into midi file notes, which are merged by merging the notes with the same pitch and volume, so as to reduce the length of the midi file and enhance its usability. When the pitch and volume of the notes are the same, the note with longer retention time is selected, and the note with shorter time is covered. In this way, the notes can be avoided too dense, and the redundant notes are reduced. The obtained note data is generated into midi format event code, and then the midi file header and track header are added to generate the midi file.

[0043] 2. The key parameters of the conversion are:

[0044] The H, S, V color components in the image information are delta-time values, pitch and volume values in the midi format music file.

[0045] 3. A low-cost blind guiding verification device for replacing vision with hearing is designed, and the specific working process is as follows:

[0046] First, power on the blind guiding verification device, and second, collect images of the scene 3 meters in front of the device through the camera, collect every 5 seconds and send to the image-to-music conversion software module of the blind guiding verification device for conversion.

[0047] The image-to-music conversion module converts the collected images into a midi format music file.

[0048] The generated music segment is played through the earphone, and the human ear judges whether there is an obstacle in front of the device according to the converted music.

[0049] When the blind guiding verification device completes the corresponding task, the power is turned off.

[0050] The verification device captures the scene about 3 meters in front of the device through the camera, and uses the collected images for image-to-music conversion. The converted sound segment is transmitted to the human ear through the earphone, and the presence of an obstacle in front of the device is determined by listening to the sound segment. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The flowchart of the image-to-music conversion method

[0052] Figure 2 The flowchart of the blind guiding verification device DETAILED DESCRIPTION

[0053] The hearing instead of vision blind guiding method will be further described below in combination with the drawings:

[0054] (1) The hardware part of the present application includes a head-mounted support, a camera, a Raspberry Pi circuit board, a mobile power supply and an earphone.

[0055] (2) First, power on the Raspberry Pi circuit board so that the Raspberry Pi circuit board and the camera can work normally. Second, use the camera to collect images of the scene 3 meters in front of the device through the image collection module every 5 seconds and send the images to the image-to-sound conversion module.

[0056] (3) The image-to-sound conversion module converts the images in front of the device into a midi file sound form output. The specific steps of this module are as follows:

[0057] ① Convert the RGB color space of the image to the HSV color space through the formula, and the conversion formula is as follows:

[0058] c max =max(R',G',B')

[0059] c min =min(R',G',B')

[0060] △=c max -c min

[0061]

[0062]

[0063]

[0064] V = c max

[0065] Wherein: H, S, V correspond to hue, saturation and brightness value; R', G', B' are normalized RGB values, Cmax and Cmin represent the maximum and minimum values of R, G, B after normalization respectively, H' and Δ are intermediate quantities; H' represents the pre-computed quantity of H component; Δ represents the difference between the maximum and minimum values of R, G, B after normalization;

[0066] 2) The conversion of image information to midi format parameters of music information

[0067] Map H, S, V three image features to three parameters of midi file of music: delta-time value, pitch and volume, and then convert the pixel points into notes in midi file; H value range is 0-360, S value range is 0-1, V value range is 0-1; The value range of pitch and volume is 0-127, and the delta-time parameter can be any positive value. Here we use one byte of storage, i.e. 0-127, to represent the value range of delta-time parameter.

[0068] Map three component information of image to three parameter information of sound (music), according to different mapping methods, finally get 6 different mapping sounds, which are:

[0069] 1) H value is mapped to the delta-time value of midi file, S value is mapped to the pitch of midi file, and V value is mapped to the volume of midi file;

[0070] 2) H value is mapped to the delta-time value of midi file, S value is mapped to the volume of midi file, and V value is mapped to the pitch of midi file;

[0071] 3) H value is mapped to the pitch of midi file, S value is mapped to the delta-time value of midi file, and V value is mapped to the volume of midi file;

[0072] 4) H value is mapped to the pitch of midi file, S value is mapped to the volume of midi file, and V value is mapped to the delta-time value of midi file;

[0073] 5) H value is mapped to the volume of the midi file, S value is mapped to the pitch of the midi file, V value is mapped to the delta-time value of the midi file;

[0074] 6) H value is mapped to the volume of the midi file, S value is mapped to the delta-time value of the midi file, V value is mapped to the pitch of the midi file;

[0075] The specific mapping method is: matching and mapping H, S, V values with volume, pitch, and delta-time parameters; wherein p, t, d respectively represent the pitch, volume, and delta-time values of the converted music.

[0076] 1) H, S, V are mapped to the delta-time, pitch, and volume values of the converted music:

[0077] d = (127 / 360) x H, p = 127 x S, t = 127 x V;

[0078] 2) H, S, V are mapped to the delta-time, volume, and pitch values of the converted music:

[0079] d = (127 / 360) x H, t = 127 x S, p = 127 x V;

[0080] 3) H, S, V are mapped to the pitch, delta-time, and volume values of the converted music:

[0081] p = (127 / 360) x H, d = 127 x S, t = 127 x V;

[0082] 4) H, S, V are mapped to the pitch, volume, and delta-time values of the converted music:

[0083] p = (127 / 360) x H, t = 127 x S, d = 127 x V;

[0084] 5) H, S, V are mapped to the volume, pitch, and delta-time values of the converted music:

[0085] t = (127 / 360) x H, p = 127 x S, d = 127 x V;

[0086] 6) H, S, V are mapped to the volume, delta-time, and pitch values of the converted music:

[0087] t = (127 / 360) x H, d = 127 x S, p = 127 x V;

[0088] ③ Image to sound conversion

[0089] The image pixel values are converted into midi file notes, and the same pitch and volume notes are merged to reduce the length of the midi file and enhance its usability. When the pitch and volume of the notes are the same, the notes with longer retention time are selected, and the notes with shorter coverage time are covered. This can avoid too dense notes and reduce redundant notes. The obtained note data is generated into midi format event code, and then the midi file header and track header are added to generate the midi file.

[0090] 2. The key parameters for conversion are:

[0091] The image information is H, S, V color components, and the midi format music file is delta-time value, pitch and volume value.

[0092] The best mapping among the above 6 mappings is determined by the human ear evaluation method.

[0093] Human ear evaluation:

[0094] I. Collection of test images

[0095] In order to test the judgment effect of the image-to-sound method and preliminarily understand the accuracy of each method, so as to select the best algorithm when building the blind guiding device, we need to collect some image data. For the sake of convenience, we choose to use a mobile phone as the image data collection tool.

[0096] The collected images can be classified according to the background. The specific classification is as follows:

[0097] 1) Background with obstacles.

[0098] 2) Background with obstacles.

[0099] 3) Background without obstacles.

[0100] 4) Background without obstacles.

[0101] In order to test the accuracy of the image-to-sound method, we will only consider the obstacles within 3 meters in front of the subject, and choose a mobile phone as the image collection tool. In order to maintain consistent shooting conditions, we take pictures at a height of 1.6 meters from the ground, and keep the angle between the shooting angle of the mobile phone and the vertical direction between 60° and 70°.

[0102] II. Test of 6 mapping relationships

[0103] The three components of the image are mapped to the three parameters of the midi file, and there are six mapping methods in total. Before conducting the human ear test for each mapping method, the testers need to undergo certain training. They will be provided with some data examples, including the sound examples mapped from the images with obstacles in front and the sound examples mapped from the images without obstacles in front. Such training aims to familiarize and understand the testers with the relationship between different sounds and the presence or absence of obstacles.

[0104] After completing the training, the testers will be required to conduct the test. They need to determine whether there is an obstacle in front by listening to the sound. The test result can only be 0 or 1, where 1 represents the presence of an obstacle, and 0 represents the absence of an obstacle. The judgment accuracy rate of each mapping is calculated according to the judgment results of all the sounds, and the best mapping is finally selected.

[0105] The present application randomly selects 50 daytime images for testing. They include 25 images with obstacles and 25 images without obstacles. Examples of images with obstacles include garbage cans piled up on the roadside, sand and gravel on the road, and construction signs, etc. Examples of images without obstacles include flat and static scenes, etc. The present application processes these images respectively and uses them for comparative experiments to test the most effective image-to-music mapping.

[0106] The H, S, and V values of each pixel point of the image are mapped to the parameters of the midi file, and there are six mappings in total, and finally 300 pieces of sound will be obtained. In order to evaluate the effect of the mapping, 10 pieces of audio samples are selected as training data for each mapping, and the remaining 40 pieces of audio samples are reserved as test data. We observe that the sound generated by the image with obstacles fluctuates greatly, while the sound generated by the image without obstacles is more peaceful. In the evaluation process, we adopt the evaluation method with the participation of the human ear.

[0107] Firstly, we let the artificial listen to the audio of the training samples, and judge whether the audio is converted from the image with obstacles or the image without obstacles through their perception. In this way, we can establish a mapping relationship according to the judgment results of the human ear. This mapping relationship corresponds the specific sound features (such as pitch, volume, timbre, etc.) to the image features (such as the presence or absence of obstacles).

[0108] Next, we use the established mapping relationship to evaluate the test data. Through the judgment of the human ear on the test data, we can determine whether the audio is converted from the image with obstacles or the image without obstacles, and calculate the judgment accuracy rate of the mapping. The mapping order of H, S, and V to the parameters of the midi file and the judgment accuracy rate are shown in Table 1 below, where p represents the pitch value, d represents the delta-time value, and t represents the volume value.

[0109] Table 1H, S, V to midi file parameter mapping order and obstacle judgment accuracy

[0110]

[0111] From the above table, it can be observed that when the chrominance (H) and luminance (V) of the image are mapped to the pitch and delta-time values of the midi file respectively, i.e. mapping 2 in the table, the best effect can be obtained. In addition, the accuracy of other mappings is also improved.

[0112] The present application utilizes more color information, not only using the chrominance (H) component of the image, but also using the saturation (S) and luminance (V) components. In this way, the color information of the image can be captured more comprehensively, and the generated audio can more accurately reflect the characteristics of the image. In addition, for notes with the same volume and pitch, the present application maps them together. In this way, the number of generated notes can be reduced, the length of the generated midi file can be shortened, and the practicability of the method is improved.

[0113] In summary, by utilizing more color information, merging similar notes, and optimizing the mapping relationship, the present application makes the conversion of images to music more accurate and accurate, and improves the accuracy of the judgment. In addition, the length of the midi file generated by the present application is greatly reduced, and the practicability is significantly improved.

[0114] 2. A low-cost blind guide verification device that replaces vision with hearing is designed, and the specific working process is as follows:

[0115] ① First, power on the Raspberry Pi circuit board, and then collect images 3 meters in front through the lens, collect every 1 time / 5 seconds and send to the image-to-music software module;

[0116] ② The image-to-music module converts the collected images into a midi format music file output;

[0117] ③ The generated music segment is played through the earphone, and the human ear judges whether there is an obstacle in front according to the listening music;

[0118] ④ When the blind guide verification device completes the corresponding task, turn off the power.

Claims

1. A method for guiding the blind using hearing instead of vision, characterized in that: ① Convert the image's RGB color space to HSV color space using the following formula: c max =max(R',G',B') c min =min(R',G',B') Δ=c max -c min V=c max Where: H, S, and V correspond to hue, saturation, and brightness values; R', G', and B' are normalized RGB values; Cmax and Cmin represent the maximum and minimum values ​​of R, G, and B after normalization, respectively; H' represents the pre-calculated amount of the H component; Δ represents the difference between the maximum and minimum values ​​of the three color channels R, G, and B after normalization. ② Conversion of MIDI format parameters from image information to music information The three image features H, S, and V are mapped to three parameters of the MIDI file: delta-time value, pitch, and volume, thereby converting pixels into musical notes in the MIDI file. The value range of H is 0 to 360, the value range of S is 0 to 1, and the value range of V is 0 to 1. The values ​​of pitch and volume are both 0 to 127. The delta-time parameter is an arbitrary positive value, using one byte of storage, i.e., 0 to 127, to represent the range of delta-time parameter values. The three components of the image are mapped to the three parameters of the music. Based on different mapping methods, six different mapped sounds are obtained, namely: 1) H value is mapped to the delta-time value of the MIDI file, S value is mapped to the pitch of the MIDI file, and V value is mapped to the volume of the MIDI file; 2) H value is mapped to the delta-time value of the MIDI file, S value is mapped to the volume of the MIDI file, and V value is mapped to the pitch of the MIDI file; 3) H values ​​are mapped to the pitch of the MIDI file, S values ​​are mapped to the delta-time value of the MIDI file, and V values ​​are mapped to the volume of the MIDI file; 4) H values ​​are mapped to the pitch of the MIDI file, S values ​​are mapped to the volume of the MIDI file, and V values ​​are mapped to the delta-time value of the MIDI file; 5) H value is mapped to the volume of the MIDI file, S value is mapped to the pitch of the MIDI file, and V value is mapped to the delta-time value of the MIDI file; 6) H value is mapped to the volume of the MIDI file, S value is mapped to the delta-time value of the MIDI file, and V value is mapped to the pitch of the MIDI file; The specific mapping method is as follows: the H, S, and V values ​​are matched and mapped with the volume, pitch, and delta-time parameters; where p, t, and d represent the pitch, volume, and delta-time values ​​of the converted music, respectively. 1) Map H, S, V to the delta-time, pitch, and volume values ​​of the converted music: d=(127 / 360)╳H, p=127╳S, t=127╳V; 2) Map H, S, V to the delta-time, volume, and pitch values ​​of the converted music: d=(127 / 360)╳H, t=127╳S, p=127╳V; 3) Map H, S, V to the pitch, delta-time, and volume of the converted music: p=(127 / 360)╳H, d=127╳S, t=127╳V; 4) Map H, S, V to the pitch, volume, and delta-time value of the converted music: p=(127 / 360)╳H, t=127╳S, d=127╳V; 5) Map H, S, V to the volume, pitch, and delta-time value of the converted music: t=(127 / 360)╳H, p=127╳S, d=127╳V; 6) Map H, S, V to the volume, delta-time, and pitch of the converted music: t=(127 / 360)╳H, d=127╳S, p=127╳V; ③ Image to sound conversion Merge the MIDI file notes converted from image pixel values ​​by merging notes with the same pitch and volume; When the pitch and volume of notes are the same, select the note with the longer retention time and overwrite the note with the shorter retention time; generate MIDI format event codes from the obtained note data, and add MIDI format file headers and track headers to generate a MIDI file.

2. A guide verification device employing the method described in claim 1, characterized in that, The work process is as follows: ① First, power on the guide verification device. Then, capture images from 3 meters in front of you through the lens. Capture images every 5 seconds and send them to the image-to-music software module of the guide verification device for conversion. ②The image-to-music module converts the captured image into a MIDI music file for output; ③ Play the generated music clip through headphones, and the human ear judges whether there are obstacles in front of us by listening to the converted music; ④ After the guide verification device has completed its corresponding task, turn off the power.

Images