An intelligent blind guiding integrated device and a blind guiding method

Abstract

The application discloses an intelligent guiding blind integrated device and method, the device comprises a millimeter wave and visual module information fusion guiding blind cap and a guiding blind stick capable of realizing environment perception and information interaction. The method in the application firstly designs a multi-sensor fusion perception algorithm, fuses millimeter wave and visual module information, perceives the surrounding environment, and gives a prompt to the blind person environment; subsequently, when the guiding blind stick touches an obstacle, continuous voice broadcast is carried out before the obstacle is touched, and after the guiding blind stick touches the obstacle, the guiding blind stick is started to vibrate, and the voice broadcast is stopped for several seconds and then restarted. On the basis of the traditional guiding blind stick, the guiding blind cap is added, the guiding blind reliability is ensured, and the multi-sensor fusion perception algorithm is designed on the intelligent guiding blind cap, so that the accuracy of sensor perception is improved, and safety redundancy is increased.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of blind guiding technology, in particular to an intelligent blind guiding integrated device and a blind guiding method. BACKGROUND

[0002] Blind people have great inconvenience in travel activities due to the lack of vision, and therefore need some auxiliary tools to help. Traditional blind guiding tools mainly include guide dogs and guide sticks, etc. However, the traditional guide stick often has poor effect due to functional limitations and cannot assist the blind people in comprehensively judging the road conditions. The guide dogs are scarce in resources and high in cost, and are difficult to use in a large range. SUMMARY

[0003] In order to solve the above problems, the present application provides an intelligent blind guiding integrated device and method. Specifically, a millimeter wave module is added to increase the perception sensor, a multi-sensor fusion perception algorithm is designed, millimeter wave and visual signals are fused, the surrounding environment distance is accurately detected, and the problem that the visual module is easily affected by the environment such as rain, fog, and insufficient light is solved. The data is processed by an expert system and output through a voice module to help the blind people effectively judge the nearby environment. The blind people use the guide stick with environment trigger perception and feedback function to perceive the environment, ensure the reliability of guiding, and increase the safety redundancy. At the same time, the guide stick and the guide cap can realize wireless interconnection.

[0004] An intelligent blind guiding integrated device includes a millimeter wave and visual module information fusion guide cap and a guide stick capable of realizing environment perception and information interaction. The guide cap includes a millimeter wave module, a visual module, a guide cap processor, and an audio playing module. The guide stick includes a vibration module and a guide stick processor.

[0005] The visual module is connected with the guide cap processor and is used for collecting obstacle video signals. The millimeter wave module is connected with the guide cap processor and is used for collecting obstacle position signals. The audio playing module is connected with the guide cap processor and is used for playing the audio synthesized by the processor. The vibration module is connected with the guide stick processor and is used for triggering a vibration alarm when the guide stick touches the obstacle. The guide cap processor and the guide stick processor realize information interaction through Bluetooth.

[0006] The guide cap processor is connected with the visual module, the millimeter wave module, and the audio playing module, respectively, and is used for extracting image information from the video signals, fusing the millimeter wave position information, designing a multi-sensor fusion perception algorithm to unify the obstacle position information, outputting accurate obstacle position information, and outputting the obstacle position through the audio playing module. The guide stick processor is connected with the vibration module. The vibration module will produce a vibration prompt for the blind people after colliding with the obstacle, and will transmit the signal to the guide stick processor. The guide stick processor will transmit the signal to the guide cap processor through a Bluetooth device, and will close the earphone of the audio playing module to prevent the blind people from being tired due to paying attention to the language information all the time.

[0007] An intelligent guide blind method, first, a multi-sensor fusion perception algorithm is designed, the millimeter wave and visual module information are fused, the surrounding environment is perceived, and the blind environment is prompted; subsequently, the blind person uses a guide stick for perception according to the guide hat prompt during the travel, the voice broadcast is continuously broadcasted when the guide stick touches the obstacle, and the guide stick opens the vibration after touching the obstacle, and the voice broadcast is restarted after 2s.

[0008] A multi-sensor fusion perception algorithm is designed, and an intelligent guide blind method is obtained, and the specific steps are as follows:

[0009] Step one: the millimeter wave module of the guide hat measures the distance of the obstacle from the blind person through formula (1), which is .

[0010] (1)

[0011] Wherein, is the speed of light, is the time difference of the millimeter wave module transmitter and receiver.

[0012] Step two: the visual module of the guide hat measures the distance of the obstacle from the blind person through formula (2), which is .

[0013] (2)

[0014] Wherein, is the height of the pinhole camera of the visual module, is the distance between the two pinhole cameras of the visual module, , respectively, the horizontal displacement of the two cameras at the height of the camera.

[0015] Step three: the physical model of the obstacle is established, and the physical model is shown in formula (3):

[0016] (3)

[0017] Wherein, , respectively, the distance from the position of the blind person at time and time ; , respectively, the speed of the obstacle at time and time ; is the acceleration of the obstacle at time ; is the time interval; For the time of noise.

[0018] The above formula can be simplified as formula (4) form, that is:

[0019] (4)

[0020] Wherein, , , , , .

[0021] The predicted value variance of the obstacle at time t+1 can be calculated as:

[0022] (5)

[0023] Wherein, , The predicted value variance of the obstacle at time t and time t+1 respectively, is the covariance matrix, is the hyperparameter at time t+1.

[0024] Step four: after establishing the physical model of the obstacle, update the Kalman gain of the obstacle physical information as :

[0025] (6)

[0026] Wherein, is the hyperparameter, which can be adjusted according to the actual situation.

[0027] Step five: after calculating the Kalman gain, the current state can be adjusted to obtain the filtered obstacle physical coordinate value :

[0028] (7)

[0029] Step six: through the above steps, the filtered distance of the obstacle position measured by the millimeter wave module and the visual module is obtained respectively. Taking the coordinate determined by the millimeter wave module as the spatial Cartesian coordinate, the distance difference of the obstacle trajectory measured by the millimeter wave module and the visual module is calculated.

[0030] (8)

[0031] Wherein, represents the distance difference of the obstacle trajectory measured by the millimeter wave module and the visual module at time The distance difference measured on the time obstacle; The obstacle physical coordinate value measured by the millimeter wave module after filtering at time The obstacle physical coordinate value measured by the vision module after filtering at time The obstacle physical coordinate value measured by the millimeter wave module after filtering at time The obstacle physical coordinate value measured by the vision module after filtering at time

[0032] Step seven: calculate the mean value of the obstacle trajectory distance difference at time And the variance At time .

[0033] Step eight: when Is in the interval , the distance at time Is retained; otherwise, it is deleted. The retained obstacle distance is calculated as follows:

[0034] (9)

[0035] Wherein, The obstacle distance value calculated after fusing the millimeter wave and vision modules.

[0036] Step nine: output the fused obstacle distance to the audio playback module to prompt the blind person about the distance and direction of the nearest obstacle.

[0037] Step ten: when the guide stick touches the obstacle, the two films inside the stick contact each other, the guide stick is electrified, and an electric signal is transmitted.

[0038] Step eleven: electrification causes the vibration sensor of the upper vibration module of the guide stick to trigger, and a vibration signal is fed back to the blind person.

[0039] Step twelve: the electric signal transmits a signal to the guide stick processor and transmits it to the guide cap processor through Bluetooth, so that the guide cap receives a power-off signal for 2s to turn off the language broadcast function of the guide cap, preventing the blind person from paying attention to the language content at the moment and being tired.

[0040] The present application has the following advantages:

[0041] 1. On the basis of the traditional guide stick, a guide cap is added to ensure reliable guide.

[0042] 2. A multi-sensor fusion perception algorithm is designed on the intelligent guide cap, and the blind person uses the guide stick with environment triggering perception and feedback function to perceive the environment, which improves the accuracy of sensor perception and increases safety redundancy. DETAILED DESCRIPTION

[0043] Figure 1 The flowchart of the present application;

[0044] Figure 2 This is a side view of a guide hat;

[0045] Figure 3 A schematic diagram of the principle of distance measurement for a vision module;

[0046] Figure 4 This is a schematic diagram of an uncertain region;

[0047] Figure 5 This is a diagram of a guide cane.

[0048] Figure 6 This is the internal circuit diagram of a guide cane. Detailed Implementation

[0049] A smart guide device for the visually impaired comprises a guide cane with environmental triggering perception and feedback functions, and a guide cap that fuses millimeter-wave and visual module information. The device includes a millimeter-wave module, a visual module, a guide cap processor, an audio playback module, a vibration module, and a guide cane processor. The visual module is connected to the guide cap processor to acquire video signals of obstacles; the millimeter-wave module is connected to the guide cap processor to acquire obstacle position signals; the audio playback module is connected to the guide cap processor to play audio synthesized by the processor; the vibration module is connected to the guide cane processor to trigger a vibration alarm when the guide cane touches an obstacle; and the guide cap processor and the guide cane processor interact via Bluetooth.

[0050] The guide cap processor connects to the vision module, millimeter-wave module, and audio playback module respectively. It extracts image information from the video signal, fuses millimeter-wave position information, designs a multi-sensor fusion perception algorithm to unify obstacle position information, outputs accurate obstacle position information, and outputs the obstacle position through the audio playback module. The guide cane processor connects to the vibration module. Upon collision with an obstacle, the vibration module generates a vibration to alert the blind person and transmits the signal to the guide cane processor. The guide cane processor then uses a Bluetooth device to turn off the headphones of the audio playback module, preventing fatigue caused by constantly focusing on verbal information. Figure 1 As shown, the smart guide hat first uses millimeter wave and visual module information fusion to perceive the surrounding environment and issue prompts to the blind person. Then, the blind person uses the guide cane to perceive the environment according to the prompts of the guide hat while walking. The guide cane continuously broadcasts voice prompts before touching an obstacle. After touching an obstacle, the guide cane starts vibrating, stops the voice broadcast for 2 seconds, and then restarts the voice broadcast.

[0051] The guide cap used in this invention is as follows: Figure 2 As shown, it includes an earphone 1, a USB interface 2, a processor module 3, a millimeter-wave module 4, and a vision module 5. The guide cane used in this invention is as follows... Figure 5As shown, containing telescopic bolt 6, USB interface 7, vibration sensor 8 and built-in processor module, wherein I represents the cross section of the cane surface enlargement, containing panel 9, optical glue 10, key piece 11, upper electrode film 12, lower electrode 13. The internal circuit diagram of the blind stick is as shown in Figure 6 As shown, containing vibration module 14, processing module 15 (including Bluetooth function), blind stick lower electrode 16, blind stick upper electrode film 17.

[0052] Design multi-sensor fusion perception algorithm, get a kind of intelligent guide blind method, the specific steps are as follows:

[0053] Step one: millimeter wave module is measured by formula (1) obstacle distance from the blind person, for .

[0054] (1)

[0055] Wherein, is the speed of light, is the time difference between the transmitter and receiver of millimeter wave module.

[0056] Step two: vision module is measured by formula (2) obstacle distance from the blind person, as Figure 3 As shown in the figure , Two places represent the two pinhole cameras of the vision module, represent the obstacle, then from the figure, the distance between the blind person and the obstacle is .

[0057] (2)

[0058] Wherein, is the height of the pinhole camera of the vision module, is the distance between the two pinhole cameras of the vision module, , respectively, the horizontal displacement of the two cameras at the height of the camera.

[0059] Step three: because there is a certain error in the detection distance of millimeter wave module and vision module, because the physical model of the obstacle needs to be established, the physical model is shown in formula (3):

[0060] (3)

[0061] Wherein, , respectively, the distance from the blind person at time and time ; , respectively represent the velocity of the obstacle at time and time . respectively represent the acceleration of the obstacle at time and time . represents the time interval. is the noise at time .

[0062] The above formula can be simplified as formula (4), that is:

[0063] (4)

[0064] wherein, , , , , .

[0065] The predicted value variance of the obstacle at time can be calculated as:

[0066] (5)

[0067] wherein, , respectively represent the predicted value variance of the obstacle at time and time , is the covariance matrix, is the hyperparameter at time .

[0068] After the physical model of the obstacle is established, the Kalman gain of the updated obstacle physical information is :

[0069] (6)

[0070] wherein, is the hyperparameter, which can be adjusted according to the actual situation.

[0071] After the Kalman gain is calculated, the current state can be adjusted to obtain the filtered obstacle physical state information :

[0072] (7)

[0073] Through the above method, the actual position of the obstacle position measured by the millimeter wave module and the visual module is obtained. The coordinate determined by the millimeter wave module is taken as the spatial Cartesian coordinate, such as Figure 4As shown, the space Cartesian coordinate is established with the millimeter wave module as the coordinate origin, and the two black dots represent the different positions of the obstacles detected by the vision module and the millimeter wave module in the same coordinate system. Figure 4 According to the uncertainty area diagram of the vision module, the distance difference of the obstacle trajectory measured by the two modules is calculated.

[0074] (8)

[0075] wherein, represents the distance difference measured by the two modules on the obstacle at time . represents the physical coordinate value of the obstacle measured by the filtered millimeter wave module at time . represents the physical coordinate value of the obstacle measured by the filtered millimeter wave module at time .

[0076] Step seven: calculate the mean and variance of the obstacle trajectory distance difference at time , as shown in formulas (9) and (10):

[0077] (9)

[0078] (10)

[0079] wherein, represents the mean of the trajectory difference at time . represents the standard deviation of the trajectory difference at time .

[0080] Step eight: when is in the interval , the distance at time is retained; otherwise, it is deleted. The retained obstacle distance is calculated as follows:

[0081] (11)

[0082] wherein, represents the obstacle distance value calculated after fusing the millimeter wave and vision modules.

[0083] Step nine: output the obtained obstacle physical position information to the audio playback module to prompt the blind person about the distance and direction of the obstacle.

[0084] Step ten: when the guide cane touches the obstacle, the panel and optical glue will be extruded due to the pressure, causing the key sheet to deform, so that the upper electrode plate film and the lower electrode on the surface of the guide cane are in contact with each other.

[0085] Step eleven: after the upper plate film and the lower electrode contact each other, as shown in Figure 2 the blind stick is powered on, the power-on causes the vibration sensor of the upper vibration module of the blind stick to trigger, and a vibration signal is fed back to the blind person.

[0086] Step twelve: at the same time, the electric signal transmits a signal to the blind stick processor, and transmits the signal to the blind cap processor through Bluetooth, so that the blind cap receives a power-off signal 2s to turn off the language broadcast function of the blind cap.

[0087] The content described in the embodiments of the specification is only a list of implementation forms of the inventive concept, and the protection scope of the application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. An intelligent guide blind method, the device used comprises a guide blind cap and a guide blind stick; The guide blind cap comprises a millimeter wave module, a vision module, a guide blind cap processor and an audio playing module; the guide blind stick comprises a vibration module and a guide blind stick processor; The visual module is connected with the blind guide cap processor and collects the obstacle video signal; the millimeter wave module is connected with the blind guide cap processor and collects the obstacle position signal; The audio playing module is connected with the guide blind cap processor, and plays the audio synthesized by the processor; The vibration module is connected with the guide blind stick processor, and triggers a vibration alarm when the guide blind stick touches an obstacle; the guide blind cap processor and the guide blind stick processor realize information interaction through Bluetooth; The method comprises the following steps: S1, the millimeter wave module and the vision module of the blind guiding cap respectively measure the distance between the obstacle and the blind person, for and ; S2, a physical model of the obstacle is established, as shown below: wherein, , respectively represent the position distance of the blind person at time and time ; , respectively represent the speed of the obstacle at time and time ; represents the acceleration of the obstacle at time ; represents the time interval; is the noise at time ; The above formula is simplified: wherein , , , , ; The predicted value variance of the obstacle at time is calculated as: wherein, , respectively denote the predicted value variance of the obstacle at time and time , is the covariance matrix, is the hyperparameter at time ; S3, after the physical model of the obstacle is established, the Kalman gain of the obstacle physical information is updated as , the current state is adjusted again, and the filtered obstacle physical coordinate value is obtained ; Computing a difference in obstacle trajectory distance measured by a millimeter wave module and a vision module ; S4, calculate the mean value of the obstacle trajectory distance difference at time t and the variance ; S5, when is in the interval, the distance at time is retained; otherwise, it is deleted; The remaining obstacle distance is calculated as follows: wherein, obstacle distance value calculated after fusion of the millimeter wave module and the vision module; obstacle physical coordinate value measured by the millimeter wave module after filtering at time obstacle physical coordinate value measured by the millimeter wave module after filtering at time obstacle physical coordinate value measured by the vision module after filtering at time obstacle physical coordinate value measured by the vision module after filtering at time S6, the fused obstacle distance is output to the audio playing module, prompting the blind person of the distance and direction of the nearest obstacle; S7, when the guide blind stick touches the obstacle, the two films inside the stick contact each other, the guide blind stick is electrified, and an electric signal is transmitted; The electrification causes the vibration sensor of the vibration module on the upper part of the guide blind stick to trigger, and a vibration signal is fed back to the blind person; S8, the electric signal transmits a signal to the guide blind stick processor, and transmits it to the guide blind cap processor through Bluetooth, so that the guide blind cap receives a power-off signal for a certain period of time to close the language broadcast function of the audio playing module of the guide blind cap.

2. The intelligent guiding method for the blind according to claim 1, wherein In S1, The millimeter wave module measures the distance of the obstacle from the blind person as , calculated as follows: wherein, is the speed of light, is the time difference of the mm-wave module transmitter and receiver; The visual module measures the distance of the obstacle from the blind person as , calculated as follows: wherein, is the pinhole camera height of the vision module, is the distance between the two pinhole cameras of the vision module, , are the horizontal displacements of the two cameras at the camera height, respectively.

3. The intelligent guiding method for the blind of claim 2, wherein, The specific process of S3 is as follows: S3.1, after the physical model of the obstacle is established, the Kalman gain of the obstacle physical information is updated as : wherein, is a hyperparameter, adjusted according to practical situations; S3.2, after the calculation of the Kalman gain, the current state is adjusted to obtain the filtered obstacle physical coordinate value : S3.3, through S3.1 and S3.2, the distances of the obstacle positions measured by the millimeter wave module and the vision module are obtained after filtering; taking the coordinates determined by the millimeter wave module as the spatial Cartesian coordinates, the distance difference of the obstacle track measured by the millimeter wave module and the vision module is calculated: wherein, represents the difference in distance measured on the obstacle at time t2.

Images