Blindway voice navigation method and system based on video target detection

Abstract

This invention provides a method and system for voice navigation of tactile paving based on video object detection. The method includes: Step 1: Constructing a tactile paving navigation dataset; Step 2: Constructing a recognition model for tactile paving and obstacles based on a YOLOv3 network, and training the recognition model using the tactile paving navigation dataset; Step 3: Acquiring real-time video, inputting the real-time video into the trained recognition model to obtain tactile paving and obstacle recognition results, the recognition results including the width, height, and center point coordinates of the recognized objects; Step 4: Providing voice navigation based on the tactile paving and obstacle recognition results.

Description

Technical Field

[0001] This invention relates to the field of tactile paving detection technology, and in particular to a tactile paving voice navigation method and system based on video target detection. Background Technology

[0002] The frequent obstruction of tactile paving in reality causes significant inconvenience for visually impaired individuals. Existing methods for tactile paving identification include those using video or images, and those using additional devices. Video or image-based methods primarily employ color segmentation and texture recognition. Methods using additional devices include IoT-based navigation systems for the blind, which include mobile terminals, canes, and RFID tags embedded in the tactile paving. The cane connects wirelessly to the handheld terminal, transmitting tactile paving information to the mobile terminal for processing and navigation instructions. Other methods utilize intelligent microcomputers with GPS coordinate positioning, ultrasonic obstacle detection, and voice prompts for obstacle detection. Existing tactile paving identification methods suffer from limitations due to variations in tactile paving colors, indistinct color differences, and varying texture types, as well as low accuracy and the inability to identify specific obstacle information. Canes and intelligent microcomputers, on the other hand, enhance the tools available to visually impaired individuals for navigation. Summary of the Invention

[0003] To address the problems of inaccurate identification of tactile paving categories, inability to identify the category of obstacles themselves, and increased burden on blind people due to the need for assistive tools for navigation, this invention provides a tactile paving voice navigation method and system based on video target detection.

[0004] On one hand, the present invention provides a method for tactile paving voice navigation based on video object detection, comprising:

[0005] Step 1: Construct a dataset for tactile paving navigation;

[0006] Step 2: Construct a recognition model for tactile paving and obstacles based on the YOLOv3 network, and train the recognition model using a tactile paving navigation dataset;

[0007] Step 3: Collect real-time video or images, and input the real-time video or images into the trained recognition model to obtain the recognition results of the blind path and obstacles. The recognition results include the width, height, and center point coordinates of the recognized object.

[0008] Step 4: Provide voice navigation based on the identification results of tactile paving and obstacles.

[0009] Furthermore, step 1 specifically includes:

[0010] Acquire images of tactile paving and label the tactile paving and obstacles in the images. All labeled images of tactile paving constitute a tactile paving dataset.

[0011] Obtain a dataset of common obstacles on tactile paving from an open-source dataset;

[0012] The tactile paving dataset and the tactile paving common obstacle dataset are combined to form a tactile paving navigation dataset.

[0013] Furthermore, in step 2, the recognition process of the recognition model specifically includes:

[0014] Feature extraction stage: Darknet53 multi-layer convolutional residual network is used to perform 8x, 16x, and 32x downsampling feature extraction on the input image; then convolution and upsampling operations are performed on the extraction results to obtain three feature maps of different sizes;

[0015] The blind path identification and labeling stage: The confidence thresholding method is used to classify image features, and the non-maximum suppression method is used to filter image features at different scales;

[0016] Image calibration and transformation stage: center point localization and image calibration are performed on continuous video frame images.

[0017] Furthermore, the convolution and upsampling operations performed on the extracted results to obtain three feature maps of different sizes specifically include: performing a convolution operation on the feature maps obtained after 32x downsampling to obtain feature maps. Figure 3 , for features Figure 3 After upsampling, the feature map is added to the feature map obtained by downsampling by 16 times, and then convolved with the result to obtain the feature map. Figure 2 , for features Figure 2 After upsampling, the feature map is added to the feature map obtained by downsampling by 8 times, and then convolved with the result to obtain the feature map. Figure 1 .

[0018] Furthermore, step 4 specifically includes:

[0019] When a tactile paving path is identified, a linear fit is performed on the coordinates of all the center points of the tactile paving path, and a straight line is drawn.

[0020] When an obstacle is detected, determine whether the obstacle's coordinates are on the straight line. If so, it is determined that there is an obstacle in front of the tactile paving, and at this time, a voice announcement of the obstacle in front is made.

[0021] If the difference between the maximum and minimum values ​​of the x-axis coordinates is greater than a certain range among all the center point coordinates, it is determined that there is a turning situation ahead of the tactile paving, and a voice broadcast for left / right turn is given.

[0022] On the other hand, the present invention provides a tactile paving voice navigation system based on video object detection, comprising:

[0023] The dataset building module is used to build a tactile paving navigation dataset;

[0024] The recognition model building and training module is used to build a recognition model for tactile paving and obstacles based on the YOLOv3 network, and train the recognition model using a tactile paving navigation dataset;

[0025] The video acquisition and recognition module is used to acquire real-time video or images, input the real-time video or images into a trained recognition model, and obtain the recognition results of blind paths and obstacles. The recognition results include the width, height, and center point coordinates of the recognized objects.

[0026] The voice navigation module is used to provide voice navigation based on the identification results of tactile paving and obstacles.

[0027] The beneficial effects of this invention are:

[0028] (1) By linking video target detection technology with tactile paving obstacle recognition, the accuracy of tactile paving recognition is improved, and the types of obstacles on tactile paving are identified quickly. At the same time, this invention does not require special equipment; tactile paving and obstacle recognition can be performed using only a smartphone, providing voice navigation for visually impaired people. It has a wide range of applications, low cost, and greatly improves the convenience of travel for visually impaired people.

[0029] (2) This invention uses the YOLOv3 object detection algorithm to extract tactile paving objects from videos, while simultaneously identifying obstacles on the tactile paving. It provides walking suggestions based on the direction of the tactile paving and the location of obstacles, and implements tactile paving voice navigation through voice prompts. Compared with other real-time object detection algorithms, the YOLO algorithm is compact and flexible, has high recognition accuracy, and a fast response speed, making it suitable for application in mobile devices and meeting the needs of tactile paving and obstacle recognition. Attached Figure Description

[0030] Figure 1 A flowchart illustrating a video target detection-based voice navigation method for tactile paving provided in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the annotation file provided in an embodiment of the present invention;

[0032] Figure 3 This is a network structure diagram of the recognition model provided in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of a tactile paving voice navigation system based on video target detection, provided as an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment of the invention provides a method for voice navigation of tactile paving based on video object detection, including the following steps:

[0037] S101: Construct a dataset for tactile paving navigation;

[0038] Specifically, this step includes: acquiring images of tactile paving and labeling the tactile paving and obstacles in the images, with all labeled images constituting a tactile paving dataset; acquiring a dataset of common obstacles in tactile paving from an open-source dataset; and combining the tactile paving dataset and the dataset of common obstacles in tactile paving to form a tactile paving navigation dataset.

[0039] Considering that the YOLO algorithm is used for target detection in subsequent steps of this invention, the annotation should be performed in YOLO format. Therefore, in this embodiment, the LabelImg tool, which supports YOLO format annotation, is used to annotate the tactile paving and obstacles in the tactile paving images. After annotation, each image will automatically generate a corresponding txt file (e.g., ...). Figure 2 As shown in the figure, the txt file contains target type and coordinate information; the coordinate information is represented as: the width w, height h of the marker box and the coordinates (x, y) of the center point of the marker box. Figure 2 In the diagram, the first column represents the category number of the identified target object, used to indicate the type of tactile paving and obstacle. The second and third columns represent the x and y coordinates of the center point of the normalized marker box; the fourth and fifth columns represent the width w and height h of the normalized marker box.

[0040] S102: Construct a recognition model for tactile paving and obstacles based on the YOLOv3 network, and train the recognition model using a tactile paving navigation dataset;

[0041] Specifically, the recognition model performs the target recognition process in the input video frame or image as follows: Figure 3 As shown, it specifically includes: feature extraction stage, tactile paving identification and annotation stage, and image calibration and transformation stage;

[0042] Feature extraction stage: Darknet53 multi-layer convolutional residual network is used to perform 8x, 16x, and 32x downsampling feature extraction on the input image; then convolution and upsampling operations are performed on the extraction results to obtain three feature maps of different sizes.

[0043] Among them, such as Figure 3 As shown, convolution is performed on the feature map obtained after 32x downsampling to obtain the feature. Figure 3 , for features Figure 3 After upsampling, the feature map is added to the feature map obtained by downsampling by 16 times, and then convolved with the result to obtain the feature map. Figure 2 , for features Figure 2 After upsampling, the feature map is added to the feature map obtained by downsampling by 8 times, and then convolved with the result to obtain the feature map. Figure 1 .

[0044] The blind path identification and labeling stage: The confidence thresholding method is used to classify image features, and the non-maximum suppression method is used to filter image features at different scales;

[0045] Image calibration and transformation stage: Center point positioning and image calibration are performed on continuous video frame images to solve the problems of image distortion and positional offset caused by tactile paving markings or video jitter, thereby ensuring the continuity and smoothness of continuous video frame content.

[0046] The purpose of confidence thresholding is to select the bounding box that is most likely to contain the target object; the purpose of non-maximum suppression is to exclude multiple bounding boxes of the same class first.

[0047] S103: Acquire real-time video or images, input the real-time video or images into the trained recognition model, and obtain the recognition results of blind paths and obstacles. The recognition results include the width, height, and center point coordinates of the recognized objects.

[0048] Specifically, real-time video or images can be captured using a user's personal terminal device. This personal terminal can be a smartphone or a smart wearable device with a camera. To ensure the quality of video or image capture, it is best to wear the personal terminal device around the user's neck.

[0049] S104: Provide voice navigation based on the results of tactile paving and obstacle identification.

[0050] Specifically, this step includes: when a tactile paving is identified, linear fitting is performed on the coordinates of all center points of the tactile paving to draw a straight line; when an obstacle is identified, it is determined whether the obstacle's coordinates are located on the straight line. If so, it is determined that there is an obstacle in front of the tactile paving, and a voice announcement of the obstacle is made; among all the center point coordinates, if the difference between the maximum and minimum x-axis coordinates is greater than a certain range, it is determined that there is a turning situation in front of the tactile paving, and a voice announcement of a left / right turn is made. In other cases (such as identifying that the tactile paving is for going straight, or that the obstacle is not located on the tactile paving), a voice announcement is not required.

[0051] Example 2

[0052] Corresponding to the above methods, embodiments of the present invention provide a tactile paving voice navigation system based on video target detection, such as... Figure 4 As shown, it includes: a dataset construction module, a recognition model construction and training module, a video acquisition and recognition module, and a voice navigation module.

[0053] The system includes the following modules: a dataset construction module for building a tactile paving navigation dataset; a recognition model construction and training module for building a recognition model for tactile paving and obstacles based on the YOLOv3 network, and training the recognition model using the tactile paving navigation dataset; a video acquisition and recognition module for acquiring real-time video, inputting the real-time video into the trained recognition model to obtain the recognition results for tactile paving and obstacles, including the width, height, and center point coordinates of the recognized objects; and a voice navigation module for providing voice navigation based on the recognition results for tactile paving and obstacles.

[0054] It should be noted that the video target detection-based blind path voice navigation system provided in this embodiment of the invention is to implement the above method embodiment. Its specific functions can be referred to the above method embodiment, and will not be repeated here.

[0055] This invention can be implemented using a smartphone or smart wearable device. It utilizes the built-in camera of the smart terminal (including smartphones or smart wearable devices) to monitor the tactile paving ahead and determine the direction of the tactile paving in real time. Therefore, without adding new hardware facilities, it can identify obstacles and determine the direction of the tactile paving, provide voice navigation for the tactile paving for visually impaired people.

[0056] Compared to traditional texture and color recognition, this invention can identify more types of tactile paving, with a wider recognition range and higher accuracy. It can also identify specific information about obstacles, enabling visually impaired patients to take appropriate actions based on the obstacle information.

[0057] Furthermore, the tactile paving obstacle recognition scheme proposed in this invention is scalable. That is, after obtaining new tactile paving obstacle data, the weight file obtained by training on these data can be added to the system, so that the system can continuously improve the recognition accuracy and increase the types of obstacles to be recognized.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for voice navigation of tactile paving based on video object detection, characterized in that, include: Step 1: Construct a dataset for tactile paving navigation; Step 2: Construct a recognition model for tactile paving and obstacles based on the YOLOv3 network, and train the recognition model using a tactile paving navigation dataset; Step 3: Collect real-time video or images, and input the real-time video or images into the trained recognition model to obtain the recognition results of the blind path and obstacles. The recognition results include the width, height, and center point coordinates of the recognized object. Step 4: Provide voice navigation based on the identification results of tactile paving and obstacles; specifically, when a tactile paving is identified, perform linear fitting on the coordinates of all center points of the tactile paving and draw a straight line; When an obstacle is detected, determine whether the obstacle's coordinates are on the straight line. If so, it is determined that there is an obstacle in front of the tactile paving, and at this time, a voice announcement of the obstacle in front is made. If the difference between the maximum and minimum values ​​of the x-axis coordinates is greater than a certain range among all the center point coordinates, it is determined that there is a turning situation ahead of the tactile paving, and a voice broadcast for left / right turn is given.

2. The method for tactile paving voice navigation based on video target detection according to claim 1, characterized in that, Step 1 specifically includes: Acquire images of tactile paving and label the tactile paving and obstacles in the images. All labeled images of tactile paving constitute a tactile paving dataset. Obtain a dataset of common obstacles on tactile paving from an open-source dataset; The tactile paving dataset and the tactile paving common obstacle dataset are combined to form a tactile paving navigation dataset.

3. The method for tactile paving voice navigation based on video target detection according to claim 1, characterized in that, In step 2, the recognition process of the recognition model specifically includes: Feature extraction stage: Darknet53 multi-layer convolutional residual network is used to perform 8x, 16x, and 32x downsampling feature extraction on the input image; then convolution and upsampling operations are performed on the extraction results to obtain three feature maps of different sizes; The blind path identification and labeling stage: The confidence thresholding method is used to classify image features, and the non-maximum suppression method is used to filter image features at different scales; Image calibration and transformation stage: center point localization and image calibration are performed on continuous video frame images.

4. The method for tactile paving voice navigation based on video target detection according to claim 3, characterized in that, The aforementioned convolution and upsampling operations on the extraction results yield three feature maps of different sizes. Specifically, this includes: performing a convolution operation on the feature map obtained after 32x downsampling to obtain feature map 3; performing an upsampling operation on feature map 3 and then adding it to the feature map obtained after 16x downsampling to obtain feature map 2; and performing an upsampling operation on feature map 2 and then adding it to the feature map obtained after 8x downsampling to obtain feature map 1.

5. A voice navigation system for tactile paving based on video object detection, characterized in that, include: The dataset building module is used to build a tactile paving navigation dataset; The recognition model building and training module is used to build a recognition model for tactile paving and obstacles based on the YOLOv3 network, and train the recognition model using a tactile paving navigation dataset; The video acquisition and recognition module is used to acquire real-time video or images, input the real-time video or images into a trained recognition model, and obtain the recognition results of blind paths and obstacles. The recognition results include the width, height, and center point coordinates of the recognized objects. The voice navigation module is used to provide voice navigation based on the identification results of tactile paving and obstacles; Specifically, when a tactile paving path is identified, it is used to perform linear fitting on the coordinates of all the center points of the tactile paving path and draw a straight line; When an obstacle is detected, determine whether the obstacle's coordinates are on the straight line. If so, it is determined that there is an obstacle in front of the tactile paving, and at this time, a voice announcement of the obstacle in front is made. If the difference between the maximum and minimum values ​​of the x-axis coordinates is greater than a certain range among all the center point coordinates, it is determined that there is a turning situation ahead of the tactile paving, and a voice broadcast for left / right turn is given.

Images