A portable device and method for assisting blind people to ride in a vehicle

Through the deep neural network model and ranging module, bus detection is detected, and combined with voice and display modules, portable bus arrival information and route display are provided, solving the problems of high transformation costs and inaccurate information in the existing technology, and improving blind people safety and convenience of rides.

CN110353958BInactive Publication Date: 2024-11-26朱原灏
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910733015.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-08-09
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art requires the transformation of bus facilities when assisting blind people on a ride, which is costly and inconvenient, and cannot provide accurate bus arrival information without modifying the existing system.

Method used

The deep neural network model is used in combination with the ranging module, and the bus target is detected through the camera and voice prompts are broadcast, bus arrival information is provided, and the voice input module and display module allow users to change and display bus routes.

Benefits of technology

It has achieved the provision of accurate bus arrival information without renovating bus facilities, improving the safety of blind people, and facilitating the bus ride process through voice interaction and route display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110353958B_ABST
    Figure CN110353958B_ABST
Patent Text Reader

Abstract

The present invention discloses a portable method for assisting the blind to take a bus, the method comprising: detecting multiple targets of pedestrians, buses and other motor vehicles by constructing a deep neural network model; obtaining the position, size and credibility of the bus target in the image; fusing the bus target information detected continuously in multiple frames, and combining the distance measured by the ranging module to evaluate the driving direction, moving speed and distance of the bus target; broadcasting voice to prompt the user of the bus arrival information. In addition, the blind can input the bus route to be taken by voice, which is displayed on the display screen, so that the bus driver and other passengers can provide assistance. The present invention also discloses a portable device, including a controller, an image capture module, a deep neural network reasoning module, a ranging module, a voice output module, a voice input module and a display module. The present invention can prompt the blind whether a bus has arrived at the station, and at the same time facilitate the blind to get help from others to take the bus.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the research field of navigation for the blind, and in particular relates to a portable device and a method for assisting the blind in riding a vehicle. Background Art

[0002] When blind people are waiting for buses at bus stops, they cannot see the arrival of motor vehicles due to their visual defects, nor can they determine whether the passing motor vehicles are buses, which makes travel very inconvenient. Currently, most of the methods used in this field are wireless communications. An electronic bus stop sign is set up at the bus stop, and equipment is added to each bus. When the bus arrives, the two devices communicate with each other to remind the blind that the bus has arrived; or each blind person carries a guide device by himself, and a sensing device is added to the bus. When the bus approaches, the two communicate with each other, and the guide device reminds the blind to get on the bus. The disadvantage of this type of device is that it requires a lot of renovation costs, such as renovating existing bus stops or adding additional equipment to each bus.

[0003] Therefore, it is necessary to provide a portable device and method for assisting blind people to take buses without relying on the modification of the existing public transportation system. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a portable device and method for assisting the blind to take a bus, which has the advantages of accurate recognition and easy carrying, does not require the modification of existing public transportation facilities, has strong scalability and low cost.

[0005] The object of the present invention is achieved through the following technical solution: A portable method for assisting blind people to ride a vehicle, comprising the steps of:

[0006] S1: Obtain training images, build a deep neural network model, and use the training images to train the model to achieve target detection of three categories: pedestrians, buses, and other motor vehicles;

[0007] S2: In actual application, the position, size and credibility of the bus target in the image are obtained through the above-trained deep neural network model;

[0008] S3: Fusion of bus target information detected continuously in multiple frames, combined with the distance measured by the ranging module, to evaluate the direction, moving speed and distance of the bus target;

[0009] S4: Broadcast voice to remind the user of the arrival of the bus.

[0010] Preferably, the constructed deep neural network model adopts the MobileNet SSD model.

[0011] Preferably, a deep neural network model is used to perform target detection to obtain a detected target set Ot ={o i,t}, target element o i,t =(class,x,y,w,h,c) i,t , where the target category class, the horizontal coordinate x and vertical coordinate y of the target in the image, the width w and height h of the target, and the credibility c of the target. The target category class is bus, other motor vehicles (car) or pedestrians (person).

[0012] Furthermore, after target detection, target elements whose confidence c is less than a threshold are deleted from the target set to improve efficiency and the accuracy of subsequent matching.

[0013] Preferably, in step S3, taking two frames of images taken at the previous and next moments as an example, for the image at the next moment, the target set O is obtained by the deep neural network model. t+1 Then, calculate the target set O at the previous moment t The bus target and the target set O at this moment t+1 The similarity of the bus targets in the previous and next moments is calculated, and the targets with similarity greater than the threshold are taken as the final bus targets; the bus targets identified at all moments are added to the bus target set. The accuracy of subsequent recognition can be improved by screening out the accurate bus target pairs in the previous and next moments through similarity.

[0014] Furthermore, to evaluate the direction and speed of the bus target, the steps are:

[0015] According to the coordinate position of the bus target in the image at each moment, the moving direction of the bus target is obtained, and whether it belongs to the bus target in the waiting direction is determined according to the moving direction. If not, the bus target is removed from the bus target set;

[0016] For the bus target in the waiting direction, the moving speed of the bus target is estimated according to the X-direction change of the bus target in the image at the previous and next moments.

[0017] Preferably, before measuring the distance through the ranging module, the distance between the bus target and the shooting position is roughly estimated based on the area of ​​the bus target in the image at the previous and next moments for the bus target in the waiting direction. When the rough estimated value is less than a certain value, the ranging module is automatically turned on to measure the accurate distance between the user and the bus target. The ranging module uses ultrasonic waves for measurement.

[0018] Preferably, the portable method for assisting the blind to take a bus further comprises a bus route number display step, which is:

[0019] Broadcast voice prompts to prompt the user to speak out the bus route to take;

[0020] Record the user's voice and save it as an audio file;

[0021] Perform speech recognition on the audio file to identify the bus route number;

[0022] The bus route number is stored and sent to a display module for external display.

[0023] By setting this display step, it is convenient for the driver and people around to provide help after the bus arrives at the station.

[0024] A portable device for assisting blind people to ride in a vehicle for implementing the above method comprises: a box body, and a controller arranged in the box body, a distance measurement module and an image capture module are arranged on one side of the box body, and the distance measurement module and the image capture module are both connected to the controller data; inside the box body, the controller is connected to a deep neural network reasoning module, and the controller is also connected to a voice output module.

[0025] Preferably, the controller is an embedded controller formed by a Raspberry Pi 3B board.

[0026] Preferably, the image capture module adopts a CSI camera, which is connected to the controller via a CSI bus, or adopts a USB camera, which is connected to the controller via a USB bus.

[0027] Preferably, the deep neural network inference module uses a Movidius neural computing stick, which includes a neural network acceleration chip and is existing hardware on which a fixed neural network model is loaded and can be directly applied.

[0028] Preferably, the distance measuring module adopts an ultrasonic distance measuring sensor. The ultrasonic distance measuring sensor has the advantage of being less affected by environmental factors.

[0029] Preferably, the voice output module includes a speaker or earphones. If earphones are used, the earphones are connected to the audio output interface on the controller via a data line.

[0030] Preferably, the portable device includes a display module, which uses a plurality of MAX7219 driven 8*8 LED dot matrixes and is connected to the controller via an SPI bus, so that bus drivers and passers-by can know the route that the blind person wants to take and actively provide help.

[0031] Preferably, the portable device includes a voice input module, which uses an I2S microphone and is connected to the controller via an I2S bus; or uses a USB microphone and is connected to the controller via a USB bus. The user can change the bus route to be detected by voice.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] 1. The present invention provides the blind with bus arrival information by visually detecting multiple targets such as buses and other motor vehicles without modifying public transportation facilities, effectively avoiding the blind getting forward as soon as they hear a bus coming, thus protecting the blind's safety to a certain extent.

[0034] 2. Compared with the prior art in which a blind person holds a cardboard with a fixed bus route, the present invention can conveniently change different bus routes through voice interaction. It is easier to get help from bus drivers and others through the display of routes, and it also makes up for the defect that machine vision is difficult to identify routes. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a structural block diagram of a device provided by an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of the structure of a device provided by an embodiment of the present invention;

[0037] Figure 3 is a flow chart of a method provided by an embodiment of the present invention;

[0038] Figure 4 is a structural block diagram of a device provided by another embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of the structure of a device provided by another embodiment of the present invention;

[0040] Figure 6 is a flow chart of a method provided by another embodiment of the present invention.

[0041] Figure 2 and Figure 5 In the figure: 1-embedded controller; 2-camera; 3-deep neural network inference module; 4-distance measurement module; 5-voice output module; 6-voice input module; 7-display module. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0043] Example 1

[0044] like Figure 1 and Figure 2 As shown, this embodiment discloses a portable device for assisting blind people to ride in a vehicle. The device includes a box body, an embedded controller and a deep neural network inference module are arranged inside the box body, an image capture module and a ranging module are arranged on the front of the box body, an opening is arranged at the upper end of the box body, and the controller is provided with an audio output interface. The voice output module is connected to the controller through the audio output interface.

[0045] The embedded controller is the control and decision center of the device, and is used to control other modules. The Raspberry Pi 3B board is preferably used as the embedded controller.

[0046] The image capture module is used to acquire images and input them to the embedded controller; the image capture module can use a CSI camera and connect to the Raspberry Pi via a CSI bus. The image capture module can also use a USB camera and connect to the Raspberry Pi via a USB bus.

[0047] The deep neural network inference module is used to perform deep neural network inference, input an image from the embedded controller, obtain the position, size and credibility of the target in the image, and then return the target detection result to the embedded controller. The deep neural network inference module uses Intel's Movidius neural computing stick; the deep neural network model uses MobileNet SSD.

[0048] The distance measuring module is used to measure the distance between the blind person and the target and input it into the embedded controller. The distance measuring module can use various distance measuring sensors such as infrared and ultrasonic, but considering the application scenario, this embodiment preferably uses one or more ultrasonic distance measuring sensors.

[0049] The voice output module is used for the embedded controller to play voice signals; the voice output module uses a headset to connect to the 3.5mm headset interface of the Raspberry Pi. The voice output module can use a speaker.

[0050] The method based on the above device includes a model training phase and an actual use phase, wherein the model training phase includes the following steps:

[0051] (1) Obtain sample image data and mark the image with at least the type and location information of the target; the marking information can be completed manually.

[0052] (2) The sample data is divided into training samples and test samples. The constructed deep neural network model is trained with the training samples. After the training, the test samples are used to test it. Through the above repeated training, the final deep neural network model is obtained.

[0053] In the actual use stage, such as Figure 3 As shown, the specific steps include:

[0054] (1) Capture an image. The user points the camera of the image capture module in the direction of the oncoming vehicle, obtains a frame of image at time t, and then sends the image to the embedded controller.

[0055] (2) Target detection. The embedded controller takes the image t Send it to the Intel Neural Compute Stick for target detection and return the target set O detected at time t t ={o i,t}. The target element o i,t =(class,x,y,w,h,c) i,t , including: target category class, pixel horizontal coordinate x and vertical coordinate y of the target in the image, pixel width w and height h of the target, and target credibility c. The target category class is bus, other motor vehicles (car) or person. Traverse the target set O t , delete the target o whose credibility c is less than the threshold i,t .

[0056] (3) Inter-frame matching. Calculate the current target set O t The bus target and the target set O at the previous moment t-1 The similarity of the bus targets in the target set B, targets with similarity greater than the threshold are added to the bus target set B t ={(o t-1,j ,o t,i )}.

[0057] (4) State estimation. According to the matching target pair (o t-1,j ,o t,i ) to estimate the direction and speed of the bus. Taking China’s traffic rules as an example, if the bus target moves to the right, it means that the bus is in the direction of the waiting vehicle. On the contrary, if the bus target moves to the left, it means that the bus is on the opposite road, and it should be removed from the bus target set B. t For the bus target waiting for the vehicle direction, according to the target o t,i The change in position x is used to estimate the speed of the bus. The larger the change, the faster the speed. Conversely, the smaller the change, the slower the speed.

[0058] In addition, according to the target t,i The size of the image can also be used to estimate the distance of the bus from the blind person. The size of the target area w*h means the distance of the bus. A small area means the bus is far away, and a large area means the bus is nearby. Thus, the location can be roughly estimated.

[0059] (5) Distance measurement: The ultrasonic distance sensor is used to measure the distance dist between the blind person and the target in front.

[0060] (6) Fusion decision. Make a decision by integrating the bus speed / distance information estimated by vision and the distance information of the ranging module. t,i If the estimated distance is short, the speed is slow, and the measured distance dist is less than the threshold, it is concluded that the bus is at the station.

[0061] (7) Voice prompts: Voice prompts are output through headphones or speakers to remind the blind person to get on the bus.

[0062] The device of this embodiment has a simple structure and can accurately determine whether a bus is approaching a station through the fusion of multiple methods such as visual detection and distance detection. When a bus is approaching a station, the blind person is reminded through headphones or speakers, and the blind person can ask people around him or the bus driver whether it is the bus he wants to take, thereby avoiding the situation where the blind person gets forward when he hears a bus is coming, thereby protecting the safety of the blind person.

[0063] Example 2

[0064] This embodiment has the same structure as Embodiment 1 except for the following features:

[0065] like Figure 4 and Figure 5 As shown, this embodiment discloses a portable device for assisting blind people to ride in a vehicle. Compared with the embodiment 1, the device is provided with a voice input module and a display module.

[0066] The voice input module is used to input the voice signal of the blind to the embedded controller. In this embodiment, the voice input module can use an I2S microphone to connect to the Raspberry Pi through the I2S bus, or a USB microphone to connect to the controller through the USB bus.

[0067] The display module is a display terminal of an embedded controller, which is arranged on the front of the box and is used to display the bus route to be taken, so that the bus driver and others can know the route to be taken by the blind. The display module uses 8*8 LED dot matrix driven by multiple MAX7219s and is connected to the controller via an SPI bus.

[0068] The working method of the portable device of this embodiment has no substantial difference from that of the first embodiment in terms of training the model and using the trained model for practical application. The difference mainly lies in the bus route input and display part, such as Figure 6 As shown, the specific steps include:

[0069] (1) Voice inquiry: The controller broadcasts voice through the voice output module, prompting the blind person to say the bus route they want to take.

[0070] (2) Voice recording: The voice input module records the voice of the blind person and saves it as an audio file;

[0071] (3) Voice recognition: The controller performs voice recognition and identifies the bus route;

[0072] (4) Display route: The controller outputs the bus route to the display module, and the display module displays the bus route to the outside.

[0073] By setting up a voice input module and a display module, blind people or other users can modify the bus route by voice, and after displaying it to the outside world, they can more conveniently get help from the driver and people around them.

[0074] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A portable method for assisting blind people to ride in a vehicle, It is characterized in that Includes steps: S1: Obtain training images, build a deep neural network model, and use the training images to train the model to achieve target detection of three categories: pedestrians, buses, and other motor vehicles; S2: In actual application, the position, size and credibility of the bus target in the image are obtained through the above-trained deep neural network model; S3: Fusion of bus target information detected continuously in multiple frames, combined with the distance measured by the ranging module, to evaluate the direction, moving speed and distance of the bus target; S4: broadcast a voice message to remind the user of the arrival of the bus; The constructed deep neural network model adopts the MobileNet SSD model; Use the deep neural network model to detect targets and obtain the target set O detected at time t t ={o i,t }, target element o i,t =(class,x,y,w,h,c) i,t , where the target category class, the horizontal coordinate x and vertical coordinate y of the target in the image, the width w and height h of the target, the credibility of the target c, and the target category class is bus, other motor vehicles or pedestrians; After target detection, the target set is traversed and target elements whose confidence c is less than the threshold are deleted from the target set; In step S3, taking the two frames of images taken at the previous and next moments as an example, for the image at the next moment, the target set O is obtained by the deep neural network model. t+1 Then, calculate the target set O at the previous moment t The bus target and the target set O at this moment t+1 The similarity of the bus targets in the dataset is calculated, and the targets with similarity greater than the threshold are taken as the final bus targets; the bus targets identified at all times are added to the bus target set; To evaluate the direction and speed of the bus target, the steps are: According to the coordinate position of the bus target in the image at each moment, the moving direction of the bus target is obtained, and whether it belongs to the bus target in the waiting direction is determined according to the moving direction. If not, the bus target is removed from the bus target set; For the bus target in the waiting direction, the moving speed of the bus target is estimated according to the X-direction change of the bus target in the image before and after the moment; According to the image size of the target, the distance between the bus and the blind person is estimated. The size of the target area w*h means the distance of the bus.

2. The portable method for assisting the blind to ride a vehicle according to claim 1, It is characterized in that Before measuring the distance through the ranging module, the distance between the bus target and the shooting position is roughly estimated based on the area of ​​the bus target in the image at the previous and next moments for the bus target in the waiting direction. When the rough estimate is less than a certain value, the ranging module is automatically turned on to measure the accurate distance between the user and the bus target.

3. The portable method for assisting the blind to ride a vehicle according to claim 1, It is characterized in that A portable method for assisting blind people to take a bus, further comprising a bus route number display step, wherein: Broadcast voice prompts to prompt the user to speak out the bus route to take; Record the user's voice and save it as an audio file; Perform speech recognition on the audio file to identify the bus route number; The bus route number is stored and sent to the display module for external display.

4. A portable device for assisting a blind person to ride a vehicle, for implementing the method according to any one of claims 1 to 3, It is characterized in that It includes: a box body, and a controller arranged in the box body, a distance measurement module and an image capture module are arranged on one side of the box body, and the distance measurement module and the image capture module are both connected to the controller data; inside the box body, the controller is connected to a deep neural network reasoning module, and the controller is also connected to a voice output module; The controller is the control and decision-making center of the device, which is used to control other modules; the image capture module is used to obtain images and input them to the controller; the deep neural network inference module is used to perform deep neural network inference, input images from the controller, obtain the position, size and credibility of the target in the image, and then return the target detection result to the controller; the distance measurement module is used to measure the distance between the blind person and the target and input it to the controller; the voice output module is used for the controller to play voice signals; The controller adopts an embedded controller composed of a Raspberry Pi 3B board; the image capture module adopts a CSI camera, which is connected to the controller through a CSI bus, or adopts a USB camera, which is connected to the controller through a USB bus; the deep neural network inference module uses a Movidius neural computing stick; the ranging module adopts an ultrasonic ranging sensor; the voice output module includes a speaker or a headset. If a headset is used, the headset is connected to the audio output interface on the controller through a data cable.

5. The portable device for assisting the blind to ride a vehicle according to claim 4, It is characterized in that The portable device comprises a display module which adopts an 8*8 LED dot matrix driven by multiple MAX7219s and is connected to a controller via an SPI bus.

6. The portable device for assisting the blind to ride a vehicle according to claim 4, It is characterized in that The portable device comprises a voice input module, which adopts an I2S microphone and is connected to a controller through an I2S bus; or adopts a USB microphone and is connected to a controller through a USB bus.