Blind guiding system, blind guiding method and blind guiding waistcoat

Abstract

The invention provides a blind guiding system, a blind guiding method and a blind guiding waistcoat, and relates to the technical field of blind guiding assistance, the system comprises an ultrasonic sensor for monitoring surrounding obstacles, a camera for acquiring traffic light image data, a microphone for acquiring human voice signals, and a positioning module for acquiring satellite signals; the obstacle avoidance module performs early warning on surrounding obstacles and outputs early warning information to the loudspeaker for playing; the traffic light recognition module inputs the traffic light image data into a pre-trained traffic light recognition model and outputs a traffic light recognition result to a loudspeaker for playing; the voice recognition and broadcast module performs voice recognition processing on the human voice signal and outputs a voice interaction result to the loudspeaker for playing; the navigation module carries out path planning processing on the satellite signals and outputs navigation information of path planning to the loudspeaker for playing. According to the invention, the environment identification precision and accuracy can be effectively improved, and the life quality and autonomous movement ability of visually impaired people are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of blind guide assistance, and in particular to a blind guide system, a blind guide method and a blind guide vest. Background Art

[0002] As a vulnerable group in society, visually impaired patients face many challenges in work and daily life due to the lack of vision, a key perception channel. Therefore, providing safe and efficient mobility solutions for the visually impaired has always been a focus of attention for scientific researchers and all sectors of society.

[0003] At present, the mainstream guide devices include guide glasses, guide canes, etc. Guide glasses are limited by their small size and are difficult to carry complex and efficient electronic equipment. At the same time, due to the close distance between the glasses and the user and the lack of sufficient interaction means, users have many inconveniences in obtaining environmental information. Although the guide cane can directly contact the ground and sense obstacles, its recognition range is limited and it cannot effectively identify obstacles at high or far places. In addition, the cane is far away from the user's head, making voice interaction difficult, and the cane itself is easy to lose, bringing additional risks to the user's travel. In order to solve the above problems, the prior art discloses some guide devices based on intelligent technology, which usually rely on the barrier-free function of the mobile phone to achieve navigation and identification. However, these guide devices still have many problems in practical applications. First, these guide devices basically do not interact with users, and only warn and remind traffic lights when encountering obstacles. Secondly, the operation on the mobile phone is still difficult for some visually impaired people because they cannot accurately touch the buttons or sliders on the screen. Then, these guide devices cannot be integrated with existing guide tools and lack system-level scheduling and management, which makes users feel cumbersome and confused when switching between different devices. Finally, cameras and lightweight AI models are used for environmental recognition, but due to the computing power limitations of the lightweight AI models themselves, the precision and accuracy of environmental recognition cannot meet expectations, which in turn affects the quality of life and autonomous mobility of the visually impaired. Summary of the invention

[0004] In order to solve the problem that the above-mentioned existing guide equipment has low environmental recognition accuracy and precision, resulting in the inability to improve the quality of life and autonomous mobility of visually impaired people, the present invention proposes a guide system, a guide method and a guide vest, which can effectively improve the environmental recognition accuracy and precision, and improve the quality of life and autonomous mobility of visually impaired people.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] A blind guide system, the system comprising: an external device and a controller, the external device is connected to the controller, the external device comprises an ultrasonic sensor, a camera, a microphone, a speaker and a positioning module, the controller is provided with an obstacle avoidance module, a traffic light recognition module, a voice recognition and broadcasting module and a navigation module;

[0007] The ultrasonic sensor is used to monitor surrounding obstacles, the camera is used to obtain traffic light image data, the microphone is used to obtain human voice signals, and the positioning module is used to obtain satellite signals;

[0008] The obstacle avoidance module is used to warn the surrounding obstacles and output warning information to the speaker for playback;

[0009] The traffic light recognition module is used to input the traffic light image data into a pre-trained traffic light recognition model and output the traffic light recognition result to a speaker for playback;

[0010] The speech recognition and broadcasting module is used to perform speech recognition processing on the human voice signal and output the speech interaction result to the speaker for playback;

[0011] The navigation module is used to perform path planning processing on the satellite signal and output navigation information of the path planning to the speaker for playing.

[0012] Preferably, the external device also includes a monitor for obtaining human life and health information, the monitor is electrically connected to the controller, and the controller transmits the human life and health information to the cloud server through a built-in wifi module. The user accesses the cloud server through a terminal device to view the human life and health information.

[0013] Preferably, the controller is provided with a GPIO interface, a CSI interface, a first USB interface, a second USB interface and a power interface, the GPIO interface is respectively connected to the ultrasonic sensor and the positioning module, the CSI interface is connected to the camera, the first USB interface is connected to the microphone, the second USB interface is connected to the speaker, and the power interface is connected to an external power supply.

[0014] Preferably, the cloud server includes a database for storing human life and health information, a back end and a front end, the front end is used to obtain user operations through the terminal device, generate user requests according to the user operations, and send the user requests to the back end; the back end receives the user request and accesses the human life and health information stored in the database through the add, delete, query and modify operation interface; the cloud server also includes a streaming media server and a docker container, the camera transmits the video stream to the streaming media server through the controller, the streaming media server receives the video stream, takes a screenshot of the video stream, obtains the screenshot data, and transmits the screenshot data to the docker container, a preset image recognition model runs in the docker container, the image recognition model is used to identify the screenshot data, and the image recognition result of the screenshot data is obtained, the image recognition result is written into the database for storage, and the controller periodically queries the database to obtain the latest image recognition result.

[0015] The present invention also proposes a blind guiding method, which is implemented based on the blind guiding system as described above, and the method comprises the following steps:

[0016] S1. Use ultrasonic sensors to monitor surrounding obstacles, cameras to obtain traffic light image data, microphones to obtain human voice signals, and positioning modules to obtain satellite signals;

[0017] S2. Use the obstacle avoidance module to warn of the surrounding obstacles and output the warning information to the speaker for playback; the traffic light recognition module inputs the traffic light image data into the pre-trained traffic light recognition model, and outputs the traffic light recognition result to the speaker for playback; the speech recognition and broadcasting module performs speech recognition processing on the human voice signal, and outputs the speech interaction result to the speaker for playback.

[0018] Preferably, the obstacle avoidance module warns of the surrounding obstacles, including:

[0019] S101. Using the ultrasonic sensor to transmit an ultrasonic signal to the surroundings, the ultrasonic signal is reflected after encountering an obstacle, and the reflected signal is returned to the ultrasonic sensor;

[0020] S102. Using the ultrasonic sensor to transmit the reflected signal to the obstacle avoidance module on the controller, the obstacle avoidance module calculates the distance between the ultrasonic sensor and the obstacle according to the transmission speed of the reflected signal;

[0021] S103. Determine whether the distance is lower than the obstacle avoidance distance threshold. If so, output a warning message to the speaker for playback; if not, do not issue a warning for the surrounding obstacles.

[0022] Preferably, the traffic light recognition module includes a model calling module and a control function module, the control function module is used to start the camera to obtain the traffic light image data, and control the speaker to play according to the traffic light recognition result; the model calling module is used to call a pre-trained traffic light recognition model from the cloud server, input the traffic light image data into the traffic light recognition model, and the traffic light recognition model outputs the traffic light recognition result to the control function module.

[0023] Preferably, the speech recognition and broadcasting module includes an audio processing module and a function calling module, and the audio processing module and the calling module are used to perform speech recognition processing on the human voice signal, including:

[0024] S201. Converting the human voice signal into a first audio file and inputting it into the audio processing module, the audio processing module calling the speech recognition interface and transferring the first audio file into the speech recognition interface;

[0025] S202. Using the speech recognition interface to convert the first audio file into a text recognition result, returning the text recognition result to the function calling module, the function calling module calling the text-to-speech interface, and passing the text recognition result to the text-to-speech interface;

[0026] S203. Use the text-to-speech interface to convert the text recognition result into a second audio file, and convert the second audio file into the voice interaction result to play it on the speaker.

[0027] Preferably, the navigation module performs path planning processing on the satellite signal, including:

[0028] S301. The positioning module transmits the satellite signal to the signal processing module in the navigation module, the signal processing module extracts and processes the satellite signal, extracts the detailed address of the starting point and the target point, and calls the coding interface using the control function;

[0029] S302. The encoding interface receives the detailed address of the starting point and the target point, and outputs the latitude and longitude information of the starting point and the target point to the route planning interface;

[0030] S303. The route planning interface receives the latitude and longitude information of the starting point and the target point, plans the path from the starting point to the target point, and outputs the optimal path as navigation information for the path planning.

[0031] The present invention also proposes a guide vest for the blind, comprising a guide vest body, a reflective warning coating is installed on the abdomen of the guide vest body, a camera is installed on one side of the chest of the guide vest body, and an ultrasonic sensor is installed on the other side, the guide vest body is equipped with a built-in positioning module, and a microphone is installed at the shoulder of the guide vest body, and the microphone is controlled by a sound volume regulator.

[0032] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0033] The present invention proposes a blind guide system, a blind guide method and a blind guide vest. Firstly, an external device is used to monitor surrounding obstacles and obtain traffic light image data, human voice signals and satellite signals, so as to realize monitoring and response to the surrounding environment, and improve the recognition accuracy of the surrounding environment for the visually impaired. Then, a traffic light recognition module uses a pre-trained traffic light recognition model to output a traffic light recognition result, so as to facilitate the visually impaired to accurately recognize the traffic light state, improve the recognition accuracy of the surrounding environment for the visually impaired, provide the visually impaired with real-time traffic signal information, and ensure travel safety. Furthermore, by integrating an obstacle avoidance module, a traffic light recognition module, a voice recognition and broadcasting module and a navigation module on a Raspberry Pi controller, the quality of life and autonomous mobility of the visually impaired are significantly enhanced, so as to assist the visually impaired to carry out daily activities more safely and confidently. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A structural block diagram of a blind guide system proposed in an embodiment of the present invention is shown;

[0035] Figure 2 A structural block diagram of a blind guide system proposed in an embodiment of the present invention is shown;

[0036] Figure 3 A cloud server structure connection diagram is shown in an embodiment of the present invention;

[0037] Figure 4 A block diagram showing a connection between a controller and an external device according to an embodiment of the present invention;

[0038] Figure 5 A block diagram showing the working principle of the cloud server proposed in an embodiment of the present invention;

[0039] Figure 6 A block diagram showing the principle of distributed real-time multi-model recognition proposed in an embodiment of the present invention;

[0040] Figure 7 A principle block diagram of the edge-cloud collaboration solution proposed in an embodiment of the present invention is shown;

[0041] Figure 8 A flowchart of a blind-guiding method proposed in an embodiment of the present invention is shown;

[0042] Fig. 9 A block diagram showing the working principle of the obstacle avoidance module proposed in an embodiment of the present invention;

[0043] Fig.10 A block diagram showing the working principle of the computer vision module proposed in an embodiment of the present invention;

[0044] Fig.11 A block diagram showing the working principle of the traffic light recognition module proposed in an embodiment of the present invention;

[0045] Fig.12 It represents the yolov5 training principle and model parameter diagram proposed in the embodiment of the present invention;

[0046] Fig.13 An example diagram of a traffic light data set proposed in an embodiment of the present invention is shown;

[0047] Fig.14 A block diagram showing the working principle of the speech recognition and broadcasting module proposed in an embodiment of the present invention;

[0048] Fig.15 A block diagram showing the working principle of the navigation module proposed in an embodiment of the present invention;

[0049] Fig.16 The figure shows a structural block diagram of a guide vest for the blind proposed in an embodiment of the present invention.

[0050] 1. Guide vest body; 11. Reflective warning coating; 12. Camera; 13. Ultrasonic sensor; 14. Positioning module; 15. Microphone; 16. Volume adjuster. DETAILED DESCRIPTION

[0051] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0052] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;

[0053] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0054] Example 1

[0055] like Figure 1 As shown, the present invention proposes a blind guide system, characterized in that the system comprises: an external device and a controller, the external device is connected to the controller, the external device comprises an ultrasonic sensor, a camera, a microphone, a speaker and a positioning module, and the controller is provided with an obstacle avoidance module, a traffic light recognition module, a voice recognition and broadcasting module and a navigation module;

[0056] The ultrasonic sensor is used to monitor surrounding obstacles, the camera is used to obtain traffic light image data, the microphone is used to obtain human voice signals, and the positioning module is used to obtain satellite signals;

[0057] The obstacle avoidance module is used to warn the surrounding obstacles and output warning information to the speaker for playback;

[0058] The traffic light recognition module is used to input the traffic light image data into a pre-trained traffic light recognition model and output the traffic light recognition result to a speaker for playback;

[0059] The speech recognition and broadcasting module is used to perform speech recognition processing on the human voice signal and output the speech interaction result to the speaker for playback;

[0060] The navigation module is used to perform path planning processing on the satellite signal and output navigation information of the path planning to the speaker for playing.

[0061] See Table 1, the Raspberry Pi controller is the Raspberry Pi 5 of the Raspberry Pi Foundation, the ultrasonic sensor is the Aircell HC-SR04 ultrasonic ranging module, and the camera is the RPi NolR Camera V2 official Raspberry Pi camera, see Figure 2 The positioning module is the Zhengdian Atom ATK-1618-BD Beidou GPS dual-mode positioning module, the microphone is a usbRaudio microphone, and the speaker is a usbRaudio speaker.

[0062] Table 1 Hardware information table

[0063]

[0064] Among them, see Figure 3 The external device also includes a monitor for obtaining human life and health information, the monitor is electrically connected to the controller, and the controller transmits the human life and health information to the cloud server through a built-in wifi module. The user accesses the cloud server through a terminal device to view the human life and health information. Among them, the built-in wifi module of the Raspberry Pi controller is used for networking, and the cloud server open platform interface is called to realize the functions of some modules.

[0065] See also Figure 4The Raspberry Pi controller is provided with a GPIO interface, a CSI interface, a first USB interface, a second USB interface and a power interface. The GPIO interface is respectively connected to the ultrasonic sensor and the positioning module. The CSI interface is connected to the camera and is a camera serial interface with high transmission efficiency. The first USB interface is connected to the microphone, the second USB interface is connected to the speaker, and the power interface is connected to an external power supply. The Raspberry Pi controller is also provided with a mircoHDMI interface and an Ethernet interface for connecting to a display.

[0066] The software environment running on the Raspberry Pi controller of this embodiment is as follows:

[0067] Raspberry Pi operating system version: Linux raspberrypi 6.6.20-v8-16k; Raspberry Pi host hardware architecture: aarch64; software running language environment: python3.10; Python virtual environment: Anaconda3-2024.02-1-Linux-aarch64; apt source: mirrors.tuna.tsinghua.edu.cn / debian; remote desktop software: RealVNCViewer; remote connection and programming software: Visual Studio Code+Remote-SSH;

[0068] The program code language and code specifications of the software are as follows:

[0069] Programming language: All module programs are programmed in Python; Code standards: PEP8; Use 4 spaces as indentation; No more than 79 characters per line; Use blank lines to separate function and class definitions; Function names are lowercase, with underscores separating words (snake_case); Class names use CamelCase; Comments should clearly express the intent of the code;

[0070] The main calling library and its functions of programming the software in Python are as follows:

[0071] Picamera2: used to call and start the camera connected to CSI to shoot and transmit video streams; Pynmea2: used to parse and process GPS data in NMEA format; Request: used to initiate HTTP requests and connect to the cloud interface; gpiozero: used to drive and call the Raspberry Pi GPIO interface; time: used to obtain the current time, calculate the time interval, etc.; json: used to process the file format transmitted in the network.

[0072] In this embodiment, firstly, an external device is used to monitor surrounding obstacles and obtain traffic light image data, human voice signals and satellite signals, so as to realize monitoring and response to the surrounding environment, thereby improving the recognition accuracy of the surrounding environment for the visually impaired; then the traffic light recognition module uses the pre-trained traffic light recognition model to output the traffic light recognition result, so as to facilitate the visually impaired to accurately identify the traffic light status, improve the recognition accuracy of the surrounding environment for the visually impaired, provide the visually impaired with real-time traffic signal information, and ensure travel safety; further, by integrating the obstacle avoidance module, the traffic light recognition module, the voice recognition and broadcasting module and the navigation module on the Raspberry Pi controller, the quality of life and autonomous mobility of the visually impaired are significantly enhanced, so as to assist the visually impaired to carry out daily activities more safely and confidently.

[0073] Example 2

[0074] The cloud server includes a database for storing human life and health information, a back end and a front end. The front end is used to obtain user operations through the terminal device, generate user requests based on the user operations, and send the user requests to the back end; the back end receives the user requests and accesses the human life and health information stored in the database through the add, delete, query and modify operation interface.

[0075] It should be noted that the database, backend and frontend are all deployed on Alibaba Cloud servers. The database is a MySQL database, the backend is equipped with a backend application based on the Node.js Express backend framework, and the frontend is equipped with a Vue frontend application. The Raspberry Pi controller obtains human information through the monitor by connecting to the network on the cloud server. The monitor is used to obtain human life and health information and store it in the MySQL database. The backend provides an operation interface for adding, deleting, checking and modifying database information, while the frontend is responsible for rendering database information. The port of the cloud server is opened to the outside world so that relatives and others can view data in real time through mobile devices. This architecture provides real-time data display and management functions, which is suitable for scenarios such as monitoring human information, so that guardians can monitor the wearer's body temperature, heart rate and other key health indicators in real time, discover abnormal situations in time and take corresponding measures, collect the wearer's health data in real time through the monitor, and use cloud computing technology to perform data analysis and abnormality detection to ensure the real-time and accuracy of health information; compared with traditional health monitoring equipment, this embodiment can discover and respond to health problems faster through the real-time connection between the monitor and the guardian, providing more timely and effective health protection for the visually impaired. At the same time, deploying the database, back-end and front-end applications on the same cloud server can simplify the deployment and maintenance process and reduce costs.

[0076] See also Figure 5 , the working principle of the database, backend and frontend is as follows:

[0077] The user logs in to the front-end through the terminal device and enters the account and password to the back-end. The back-end generates a key and queries the user key table in the database based on the key to determine whether the key is correct. If it is correct, it returns the correct information, otherwise it returns the error information;

[0078] When a user requests to view temperature data through the front-end interface, the front-end sends a request to the back-end. After receiving the request, the back-end obtains the latest temperature data from the temperature table by performing a database query operation. Then, the back-end organizes the data into a suitable format and returns it to the front-end. Finally, the front-end receives the data and renders it, displaying the temperature list for the user to view.

[0079] When a user requests to view heart rate data through the front-end interface, the front-end will send a request to the back-end; after receiving the request, the back-end will perform a database query operation to retrieve the latest heart rate data from the heart rate table; then, the back-end will organize the data into a format suitable for front-end display and return it to the front-end; finally, after receiving the data, the front-end will render and clearly display the heart rate list for the user to view;

[0080] When a user requests detailed information of a certain geographic location through the front-end interface, the front-end will first display a Base64-encoded map image of the location to the user and send a request to the back-end at the same time. After receiving the request, the back-end calls the Amap interface to obtain the precise geographic location information of the location. After obtaining the information, the back-end will save the data to the geographic information table of the database and return the necessary information (such as address description) to the front-end. Finally, after receiving the information, the front-end can mark the location on the map or display the geographic location information to the user in other ways.

[0081] When a user views a list of items or closes a list through the front-end interface, the front-end sends a request to the back-end to obtain relevant data; after receiving the request, the back-end queries the item table from the database, obtains the current item list or closed item list, and returns the data to the front-end for display. When a user needs to edit, add, or delete an item, the front-end collects the information entered by the user and sends a corresponding request to the back-end; after receiving the request, the back-end performs corresponding operations on the item table according to the request type (edit, add, delete), such as updating item information, adding new items, or deleting items, and returns the operation results to the front-end; after receiving the operation results, the front-end updates the item list or closed list on the interface to reflect the latest item status.

[0082] See also Figure 6This embodiment also proposes that the cloud server also includes a streaming server and a docker container. The camera transmits the video stream to the streaming server through the Raspberry Pi controller. The streaming server receives the video stream, takes a screenshot of the video stream, obtains screenshot data, and transmits the screenshot data to the docker container. A preset image recognition model runs in the docker container. The screenshot data is recognized by using the image recognition model to obtain an image recognition result of the screenshot data. The image recognition result is written into the database for storage. The Raspberry Pi controller periodically queries the database to obtain the latest image recognition result.

[0083] In order to address the problem that the computing power of the Raspberry Pi controller cannot support the use of a large number of model calculations and a large amount of heat is generated when the model is called, resulting in a decrease in user experience, this embodiment uses the streaming media server and docker container deployed on the cloud server to perform real-time video stream processing and multi-model recognition based on Raspberry Pi. The Raspberry Pi calls the camera to obtain the video stream, and uses the FFmpeg protocol to push the video stream to the streaming media server. The high-performance host is responsible for extracting the video stream from the streaming media server, and uses Docker containerization technology to implement multi-threaded processing, and performs multi-model recognition tasks (for example, pedestrian recognition, scene recognition, etc.). The image recognition results are synchronized to the cloud database in real time, and the Raspberry Pi controller obtains the database information in real time and responds accordingly.

[0084] It should be noted that the use of streaming media servers and Docker containers to achieve real-time video stream processing and multi-model recognition has the following advantages:

[0085] Strong real-time performance: Streaming media server and Docker container technology are used to achieve multi-model parallel recognition, ensuring the real-time performance of the system.

[0086] Scalability: Containerization technology makes it easier and more efficient to add new recognition models and tasks.

[0087] Data synchronization: Real-time data synchronization is achieved through the database, ensuring information consistency between components.

[0088] See also Figure 7 In actual application scenarios, network failures may cause the recognition system to crash, seriously affecting the system's functionality. To address this issue, the system also proposes an edge-cloud collaboration solution, which uses the guide system based on the guide vest described above for processing when the network connection is normal; when the network connection is abnormal, it switches to the lightweight model local to the Raspberry Pi controller for recognition to minimize the impact. The edge-cloud collaboration solution is as follows:

[0089] When the network connection is normal, the system uses the above-mentioned guide vest-based guide system to perform distributed real-time multi-model recognition.

[0090] When the network connection is abnormal, the system switches to local processing mode. At this time, the Raspberry Pi uses a local lightweight model for recognition. Although the model is simplified, it can maintain the basic functions of the system to a certain extent and ensure the continuity of the recognition task. The recognition results are stored locally and synchronized to the database after the network is restored.

[0091] The advantages of the edge-cloud collaboration solution are:

[0092] Through the edge-cloud collaboration solution, the system can continue to run even when the network fails, ensuring the continuity of the recognition task and the stability of the system. After the network is restored, the local recognition results are automatically synchronized to the cloud database to ensure data consistency and integrity.

[0093] The immediate family members of the visually impaired can log in to the user interface through terminal devices such as mobile phones through various precise data collection and powerful artificial intelligence model algorithm processing, browse exquisite data on the display screen of the terminal device, and intuitively view the life dynamics of their relatives. It is also equipped with a one-click alarm function, which makes the travel safety of the visually impaired even more guaranteed. The system promptly provides feedback to users through the intelligent voice broadcast system based on the content and time of the reminder, further improving the convenience of users' lives. Through the edge-cloud collaborative solution, the system can continue to run even in the event of a network failure. After the network is restored, the local recognition results are automatically synchronized to the cloud database to ensure data consistency and integrity.

[0094] This embodiment greatly improves the user's control experience and increases user satisfaction by 50%. When immediate family members are not available, they can remotely control and contact users through mobile phones in advance, set reminders, check the user's location and physical condition, and reduce the cost of family care. The edge-cloud collaboration solution ensures the continuity of the recognition task, the stability of the system, and the integrity of the data.

[0095] Example 3

[0096] like Figure 8 As shown, this embodiment proposes a blind guiding method, which is implemented based on the blind guiding system described in the above embodiment, and the method includes the following steps:

[0097] S1. Use ultrasonic sensors to monitor surrounding obstacles, cameras to obtain traffic light image data, microphones to obtain human voice signals, and positioning modules to obtain satellite signals;

[0098] S2. Use the obstacle avoidance module to warn of the surrounding obstacles and output the warning information to the speaker for playback; the traffic light recognition module inputs the traffic light image data into the pre-trained traffic light recognition model, and outputs the traffic light recognition result to the speaker for playback; the speech recognition and broadcasting module performs speech recognition processing on the human voice signal, and outputs the speech interaction result to the speaker for playback.

[0099] The obstacle avoidance module warns of surrounding obstacles, including:

[0100] S101. Using the ultrasonic sensor to transmit an ultrasonic signal to the surroundings, the ultrasonic signal is reflected after encountering an obstacle, and the reflected signal is returned to the ultrasonic sensor;

[0101] S102. Using the ultrasonic sensor to transmit the reflected signal to an obstacle avoidance module mounted on the Raspberry Pi controller, the obstacle avoidance module calculates the distance between the ultrasonic sensor and the obstacle according to the transmission speed of the reflected signal;

[0102] S103. Determine whether the distance is lower than the obstacle avoidance distance threshold. If so, output a warning message to the speaker for playback; if not, do not issue a warning for the surrounding obstacles.

[0103] In the obstacle avoidance module, see Fig. 9 The Raspberry Pi controller is used as the core controller and is equipped with an external ultrasonic sensor. The ultrasonic sensor emits ultrasonic waves. When the ultrasonic waves encounter obstacles, they are reflected. The Raspberry Pi calculates the distance between the ultrasonic sensor and the obstacle according to the speed of the reflected ultrasonic signal, and transmits the specific distance value back to the Raspberry Pi controller. The Raspberry Pi controller sets the obstacle avoidance distance threshold through the control function, and warns and announces according to the comparison result between the distance between the ultrasonic sensor and the obstacle and the obstacle avoidance distance threshold. When it is detected whether the distance is lower than the obstacle avoidance distance threshold, the corresponding warning and announcement functions of the speaker are triggered to realize the obstacle avoidance function.

[0104] In addition, the computer vision module is introduced into the Raspberry Pi controller, see Fig.10 , use the camera to obtain images of common objects, and output the common objects to the computer vision module of the Raspberry Pi controller. The computer vision module uses the main control function to call the preset image recognition model to recognize the common object images, and output the recognition results of the common object images to the speaker for playback. When the ultrasonic obstacle avoidance module is stopped in crowded places such as elevators and shopping malls, some common roadblocks such as steps and railings are broadcast.

[0105] See also Fig.11The traffic light recognition module includes a model calling module and a control function module. The control function module is used to start the camera to obtain the traffic light image data and control the speaker to play according to the traffic light recognition result; the model calling module is used to call the pre-trained traffic light recognition model from the cloud server, input the traffic light image data into the traffic light recognition model, and the traffic light recognition model outputs the traffic light recognition result to the control function module.

[0106] In the traffic light recognition module, the pre-trained traffic light recognition model is the YOLOv5 target detection model. The traffic light recognition module calls this efficient target detection model. The YOLOv5 target detection model is trained on a powerful host. After the training, it is transplanted to the Raspberry Pi controller as the called traffic light recognition model. The traffic light recognition module in the Raspberry Pi controller uses the camera to shoot traffic light image data, uses its CPU resources to call the trained model for image recognition, and obtains the status information of the traffic light in real time. According to the recognition results, the behavior of other modules is controlled to realize the traffic light recognition and reminder functions. At the same time, considering the multiple traffic lights that users may encounter in different directions at the intersection, a multi-traffic light recognition algorithm is developed, which uses the different coordinates of the traffic lights on the same plane and the navigation system direction to jointly calculate the traffic light direction of the user's forward direction, and broadcasts the recognition results at the same time.

[0107] The YOLOv5 target detection model uses CSPDarknet53 as the backbone network, which is a deep separable convolutional network with good feature extraction capabilities. At the same time, an additional convolutional layer is added after the CSPDarknet53 backbone network to further extract features. At the same time, a combined loss function is used to combine the target position loss, target confidence loss and category loss to train the YOLOv5 target detection model. The YOLOv5 target detection model provides multiple pre-trained models to adapt to different needs. The training principle and model parameters of the YOLOv5 target detection model can be found in Fig.12 The dataset used to train the YOLOv5 target detection model is the traffic light recognition target detection open source dataset, which contains 5786 images of 1080*1920 pixels and 720*1280 pixels. It also contains 14130 instances of 4 categories (including red, green, yellow, and closed). These scenes cover a considerable number of road scenes and types. See Fig.13 , such as busy inner city street scenes, dense stop-and-go traffic, drastic lighting / exposure changes, flashing / fluctuating traffic lights, multiple visible traffic lights, and image parts that could be confused with traffic lights (e.g., large round tail lights).

[0108] The speech recognition and broadcasting module includes an audio processing module and a function calling module, and the audio processing module and the calling module are used to perform speech recognition processing on the human voice signal. Fig.14 ,include:

[0109] S201. Converting the human voice signal into a first audio file and inputting it into the audio processing module, the audio processing module calls the speech recognition interface through the model call and control function, and transmits the first audio file into the speech recognition interface;

[0110] S202. Using the speech recognition interface to convert the first audio file into a text recognition result, returning the text recognition result to the function calling module, the function calling module calling the text-to-speech interface, and passing the text recognition result to the text-to-speech interface;

[0111] S203. Use the text-to-speech interface to convert the text recognition result into a second audio file, and convert the second audio file into the voice interaction result to play on the speaker.

[0112] See also Fig.15 , the navigation module performs path planning processing on the satellite signal, including:

[0113] S301. The positioning module transmits the satellite signal to the signal processing module in the navigation module, the signal processing module extracts and processes the satellite signal, extracts the detailed address of the starting point and the target point, and calls the coding interface using the control function;

[0114] S302. The encoding interface receives the detailed address of the starting point and the target point, and outputs the latitude and longitude information of the starting point and the target point to the route planning interface;

[0115] S303. The route planning interface receives the latitude and longitude information of the starting point and the target point, plans the path from the starting point to the target point, and outputs the optimal path as navigation information for the path planning.

[0116] The positioning module in the navigation module is the Beidou GPS positioning module, which obtains satellite signals. The satellite signals are NMEA protocol information streams. NMEA protocol information streams are a universal data format used for data exchange and communication between GPS receivers, radars, autopilot systems and other navigation equipment.

[0117] The coding interface and route planning interface are both API interfaces of the AutoNavi open platform. On the AutoNavi open platform, the coding interface is used to convert longitude and latitude into local location information, and the route planning function is realized by calling the route planning interface at the secondary level. The navigation function is realized by calling the route planning interface cyclically, providing users with accurate navigation guidance and helping to realize intelligent navigation and route planning.

[0118] It should be noted that the blind guide system based on the blind guide vest proposed in this embodiment has the following advantages:

[0119] (1) Improving autonomous mobility: By integrating navigation, obstacle avoidance, and traffic light recognition functions, the autonomous mobility of visually impaired people is significantly enhanced, enabling them to carry out daily activities more safely and confidently. The navigation module uses the Beidou GPS positioning module and the AutoNavi map API for precise path planning. The obstacle avoidance system uses ultrasonic sensors for real-time obstacle detection. The traffic light recognition system uses the YOLOv5 target monitoring model for efficient recognition. Compared with traditional guide canes and guide dogs, the guide vest-based guide system proposed in this embodiment provides more comprehensive auxiliary functions. It can not only detect obstacles, but also recognize traffic signals and provide navigation guidance, significantly improving the travel efficiency and safety of visually impaired people.

[0120] (2) High-precision environmental monitoring: By using ultrasonic sensors, high-definition cameras and Beidou GPS modules, real-time monitoring and response to the surrounding environment are achieved, which improves the visually impaired people's cognitive accuracy of the surrounding environment. The system achieves 360-degree environmental perception without blind spots through multi-sensor fusion technology; the real-time data processing algorithm can quickly respond to environmental changes and provide accurate navigation and obstacle avoidance information. Compared with a single guide cane, the multi-sensor fusion technology in the guide vest-based guide system proposed in this embodiment can provide more comprehensive and accurate environmental information, reducing the uncertainty and risk of visually impaired people in complex environments.

[0121] (3) High-accuracy traffic signal recognition: The system uses the YOLOv5 target monitoring model, which can accurately identify the status of traffic lights and provide real-time traffic signal information for the visually impaired to ensure travel safety. The YOLOv5 target monitoring model has efficient feature extraction capabilities and accurate target detection performance. Through training with a large amount of traffic light data, the model can accurately identify traffic lights in different states. Traditional traffic signal auxiliary equipment often relies on sound or simple light sensing technology, while this system uses the advanced YOLOv5 target monitoring model, which can provide more accurate and reliable traffic light recognition results, significantly improving the travel safety of the visually impaired.

[0122] Example 4

[0123] This embodiment proposes a guide vest, including a guide vest body 1, a reflective warning coating 11 is installed on the abdomen of the guide vest body 1, a camera 12 is installed on one side of the chest position of the guide vest body 1, and an ultrasonic sensor 13 is installed on the other side, the guide vest body 1 is built-in with a positioning module 14, and a microphone 15 is installed at the shoulder position of the guide vest body 1, and the microphone 15 is controlled by a sound volume regulator 16.

[0124] The guide vest uses the Raspberry Pi controller as the core controller, and integrates the ultrasonic sensor HC-SR04, the camera RPi NoIR Camera V2\Beidou GPS positioning module ATK-1618-BD, the usbRaudio microphone and the usbRaudio speaker in the guide vest. The guide vest is provided with the guide system of the above embodiment, the ultrasonic sensor is used to accurately measure the distance between the vest wearer and the obstacle; the camera is used to record the traffic light image in real time; the Beidou GPS positioning module is used to accurately locate the position of the vest wearer and the distance between the vest wearer and the destination; the microphone and the speaker are mainly used to realize the dynamic interaction between the user and the vest device. The Raspberry Pi controller obtains the dynamic information of obstacles and traffic lights through sensors and cameras, and uses the YOLOV5 target detection model and the traffic light data set to accurately capture the traffic light signal. In addition, the Raspberry Pi controller uses the positioning module to call the Amap API multiple times to realize navigation positioning. The above-mentioned information will be fed back to the Raspberry Pi controller and the user through the voice recognition interface and the text-to-speech interface to realize real-time human-computer interaction.

[0125] The guide vest of this embodiment is a wide style with multiple pockets and pure cotton, which is comfortable and ventilated, and has the following advantages:

[0126] All functions of the guide vest have been tested to be normal and complete, providing full protection for the user's travel safety; the user group of the guide vest is not limited to the visually impaired, but also suitable for middle-aged and elderly people who have difficulty in traveling, and has a wider application; all equipment is hidden in the guide vest, fully protecting the self-esteem of some users; the customizable service of the guide vest provides users with a comfortable and personalized experience; compared with traditional guide equipment, the guide vest has lower cost and appearance advantages, and is easy to wear, unlike canes and guide dogs that are easily detached in crowds or narrow roads.

[0127] Through intelligent voice recognition and broadcast modules, the guide vest has strong recognition capabilities in complex environments such as Cantonese, Mandarin, and Sichuanese, and can interact with users without barriers, completely freeing the user's hands. Raspberry Pi NoIRCamera V2 performed well after hardware testing. The camera showed excellent image quality and clarity in various tests, especially in low-light environments and infrared imaging applications. The navigation module performed well in various tests, with fast map loading speed, accurate route planning, and clear and accurate real-time traffic information updates and voice navigation. The system can adjust the route in time and provide a good navigation experience.

[0128] In this embodiment, the proposed guide vest is used as a carrier for integrating various sensors and actuators. The guide vest takes into account the comfort and convenience of the visually impaired, is easy to wear and easy to operate. The guide vest not only integrates a variety of sensors, but also pays special attention to barrier-free design, so that the visually impaired can use it easily, and at the same time provides a new form of auxiliary equipment. The guide vest integrates advanced voice recognition and broadcasting systems, can interact with users through natural language, and provide real-time navigation instructions and environmental information. The guide vest can recognize and respond to the user's voice instructions, and provide real-time voice feedback, so that the visually impaired can obtain navigation information through voice interaction, which enhances the naturalness and convenience of the interaction. The guide vest can also perform distributed real-time multi-model recognition with edge-cloud collaboration, can process real-time video streams, and use multiple deep learning models for parallel recognition tasks. The guide vest uses high-performance computing and containerization technology to achieve rapid and accurate recognition of complex environments, including traffic lights, obstacles, etc., improves the real-time and accuracy of the system, and ensures robustness in the case of network instability.

[0129] It is also necessary to specially state that the blind guide vest of the blind guide system proposed in the above embodiment is provided in this embodiment, which can not only effectively improve the precision and accuracy of environmental recognition, but also solve the deficiencies of existing assistive technologies for the visually impaired in terms of detection range, function, cost, real-time performance, adaptability and personalized services, and provide a comprehensive solution to improve the quality of life and autonomous mobility of the visually impaired. The deficiencies in detection range, function, cost, real-time performance, adaptability and personalized services are specifically as follows:

[0130] 1. Expand the detection range and improve functional integration: By integrating multiple sensors and intelligent algorithms, the guide vest can fully detect and respond to obstacles, traffic lights, etc. in the surrounding environment, and provide more comprehensive auxiliary information.

[0131] 2. Reduce costs and improve accessibility: Lower the threshold for visually impaired people to use assistive devices so that more visually impaired people can afford and benefit from this technology.

[0132] 3. Enhance real-time and adaptability: Utilize real-time data processing and intelligent learning algorithms to enable the auxiliary vest to provide personalized navigation and auxiliary services based on real-time environmental changes and users' walking habits.

[0133] 4. Provide personalized services: Through intelligent learning of user preferences and habits, provide customized navigation and auxiliary services, so that the use of auxiliary vests can be more in line with individual needs.

[0134] 5. Enhanced security and monitoring functions: Through remote monitoring technologies such as smart data screens, guardians can understand the status and location of visually impaired people in real time, improving safety.

[0135] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A blind guide system, characterized in that: The system comprises: an external device and a controller, wherein the external device is connected to the controller, the external device comprises an ultrasonic sensor, a camera, a microphone, a speaker and a positioning module, and the controller is provided with an obstacle avoidance module, a traffic light recognition module, a voice recognition and broadcasting module and a navigation module; The ultrasonic sensor is used to monitor surrounding obstacles, the camera is used to obtain traffic light image data, the microphone is used to obtain human voice signals, and the positioning module is used to obtain satellite signals; The obstacle avoidance module is used to warn the surrounding obstacles and output warning information to the speaker for playback; The traffic light recognition module is used to input the traffic light image data into a pre-trained traffic light recognition model and output the traffic light recognition result to a speaker for playback; The speech recognition and broadcasting module is used to perform speech recognition processing on the human voice signal and output the speech interaction result to the speaker for playback; The navigation module is used to perform path planning processing on the satellite signal and output navigation information of the path planning to the speaker for playing.

2. The blind guide system based on the blind guide vest according to claim 1, characterized in that: The external device also includes a monitor for obtaining human life and health information. The monitor is electrically connected to the controller. The controller transmits the human life and health information to the cloud server through a built-in wifi module. The user accesses the cloud server through a terminal device to view the human life and health information.

3. The blind guide system based on the blind guide vest according to claim 2, characterized in that: The controller is provided with a GPIO interface, a CSI interface, a first USB interface, a second USB interface and a power interface, the GPIO interface is respectively connected to the ultrasonic sensor and the positioning module, the CSI interface is connected to the camera, the first USB interface is connected to the microphone, the second USB interface is connected to the speaker, and the power interface is connected to an external power supply.

4. The blind guide system based on a blind guide vest according to claim 2, wherein the cloud server comprises a database for storing human life and health information, a back end and a front end, the front end is used to obtain user operations through the terminal device, generate user requests according to the user operations, and send the user requests to the back end; the back end receives the user request and accesses the human life and health information stored in the database through the add, delete, query and modify operation interface; the cloud server also includes a streaming media server and a docker container, the camera transmits the video stream to the streaming media server through the controller, the streaming media server receives the video stream, takes a screenshot of the video stream, obtains the screenshot data, and transmits the screenshot data to the docker container, a preset image recognition model runs in the docker container, the screenshot data is recognized by the image recognition model, and the image recognition result of the screenshot data is obtained, the image recognition result is written into the database for storage, and the controller periodically queries the database to obtain the latest image recognition result.

5. A blind guiding method, implemented based on the blind guiding system according to any one of claims 1 to 4, characterized in that: The method comprises the following steps: S1. Use ultrasonic sensors to monitor surrounding obstacles, cameras to obtain traffic light image data, microphones to obtain human voice signals, and positioning modules to obtain satellite signals; S2. Use the obstacle avoidance module to warn of the surrounding obstacles and output the warning information to the speaker for playback; the traffic light recognition module inputs the traffic light image data into the pre-trained traffic light recognition model, and outputs the traffic light recognition result to the speaker for playback; the speech recognition and broadcasting module performs speech recognition processing on the human voice signal, and outputs the speech interaction result to the speaker for playback.

6. The blind guiding method according to claim 5, characterized in that: The obstacle avoidance module warns of surrounding obstacles, including: S101. Using the ultrasonic sensor to transmit an ultrasonic signal to the surroundings, the ultrasonic signal is reflected after encountering an obstacle, and the reflected signal is returned to the ultrasonic sensor; S102. Using the ultrasonic sensor to transmit the reflected signal to the obstacle avoidance module on the controller, the obstacle avoidance module calculates the distance between the ultrasonic sensor and the obstacle according to the transmission speed of the reflected signal; S103. Determine whether the distance is lower than the obstacle avoidance distance threshold. If so, output a warning message to the speaker for playback; if not, do not issue a warning for the surrounding obstacles.

7. The blind guiding method according to claim 5, characterized in that: The traffic light recognition module includes a model calling module and a control function module, wherein the control function module is used to start the camera to obtain the traffic light image data and control the speaker to play according to the traffic light recognition result; The model calling module is used to call a pre-trained traffic light recognition model from a cloud server, input the traffic light image data into the traffic light recognition model, and the traffic light recognition model outputs the traffic light recognition result to the control function module.

8. The blind guiding method according to claim 5, characterized in that: The speech recognition and broadcasting module includes an audio processing module and a function calling module, and uses the audio processing module and the calling module to perform speech recognition processing on the human voice signal, including: S201. Converting the human voice signal into a first audio file and inputting it into the audio processing module, the audio processing module calling the speech recognition interface and transferring the first audio file into the speech recognition interface; S202. Using the speech recognition interface to convert the first audio file into a text recognition result, returning the text recognition result to the function calling module, the function calling module calling the text-to-speech interface, and passing the text recognition result to the text-to-speech interface; S203. Use the text-to-speech interface to convert the text recognition result into a second audio file, and convert the second audio file into the voice interaction result to play on the speaker.

9. The blind guiding method according to claim 5, characterized in that: The navigation module performs path planning processing on the satellite signal, including: S301. The positioning module transmits the satellite signal to the signal processing module in the navigation module, the signal processing module extracts and processes the satellite signal, extracts the detailed address of the starting point and the target point, and calls the coding interface using the control function; S302. The encoding interface receives the detailed address of the starting point and the target point, and outputs the latitude and longitude information of the starting point and the target point to the route planning interface; S303. The route planning interface receives the latitude and longitude information of the starting point and the target point, plans the path from the starting point to the target point, and outputs the optimal path as navigation information for the path planning.

10. A guide vest, characterized in that: The invention comprises a guide vest body (1), wherein a reflective warning coating (11) is installed on the abdomen of the guide vest body (1), a camera (12) is installed on one side of the chest of the guide vest body (1), and an ultrasonic sensor (13) is installed on the other side, the guide vest body (1) is built with a positioning module (14), and a microphone (15) is installed on the shoulder of the guide vest body (1), and the microphone (15) is controlled by a sound volume regulator (16).

Images