Blind person navigation method and equipment

Through the image depth prediction model based on hybrid network and the image detection model of depth-guided, combined with feature fusion and depth information, the problem of insufficient perception ability of blind navigation in complex environments is solved, and more accurate obstacle detection and navigation paths are achieved. Improves navigation accuracy and reliability.

CN118918542APending Publication Date: 2024-11-08罗卓然
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410947550.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing blind navigation technology has limited environmental perception capabilities in complex environments, making it difficult to accurately detect obstacles and paths, resulting in navigation offsets and error prompts.

Method used

Using a hybrid network-based image depth prediction model and a depth-guided image detection model, image data is captured in real time through the camera, combined with feature fusion and depth information, the accuracy of environmental perception and obstacle detection is improved, and converted into real three-dimensional coordinate information.

Benefits of technology

It significantly improves the environmental perception ability and obstacle detection accuracy of blind navigation, reduces path offsets and error prompts, and provides more accurate navigation information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918542A_ABST
    Figure CN118918542A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of navigation, in particular to a blind person navigation method and equipment. Firstly, an image depth prediction model based on a hybrid network is provided, image data captured by a camera in real time are input into the model, and the output quality of depth information is improved by adopting a feature fusion strategy; secondly, providing an image detection model based on depth guidance, embedding the output features of a depth prediction model into a traditional detection model by the model, and accurately obtaining the category and position information of an object by using a depth information guidance scheme; and finally, integrating the depth information, the category information and the position information, and converting the depth information, the category information and the position information into real three-dimensional coordinate information for judging a travel path and obstacles. According to the blind person navigation method and device, accurate obstacle avoidance information can be provided for the blind person, and the adaptability to complex scenes is high.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of navigation technology, and in particular to a navigation method and device for the blind. Background Art

[0002] Currently, deep learning algorithms can help the blind navigate more easily and enhance their quality of life in terms of navigation. They can be used for real-time image recognition, object detection, and scene understanding, so that the navigation system can help the blind identify road obstacles, traffic signs, sidewalks, etc., and provide real-time navigation guidance; they can analyze historical data, traffic conditions, and terrain information to help the blind plan the best path and avoid traffic jams or dangerous areas.

[0003] Simultaneous Localization and Mapping (SLAM) technology is a typical technology combined with deep learning algorithms. By collecting sensor data, SLAM algorithms can simultaneously locate the position of the device and build a map of the surrounding environment without knowing the map or location information in advance.

[0004] Traditional navigation methods for the blind usually involve using sensors (such as ultrasonic sensors or infrared sensors) to detect the surrounding environment and use non-visual information such as sound, touch or vibration to guide the blind to avoid obstacles and navigate to the destination. Due to the limitations of sensor types, the environmental perception capabilities of traditional methods are usually limited, and the ability to obtain complex environments or detailed information is relatively weak, making it difficult to cope with complex intersections or obstacle layouts.

[0005] At present, there are still some defects in the existing technical literature on blind navigation algorithms. For example, a Chinese patent with patent application number CN202210489717.1 proposes a blind obstacle avoidance navigation system based on visual SLAM; the patent uses image information and distance information collected by RGBD sensors to calculate direction calculation and target detection; at the same time, the patent uses the Yolo model for target detection to provide object type and location information.

[0006] Due to the accuracy limitations of traditional RGBD sensors, the patented method is relatively weak in acquiring complex environments or detailed information. There will be errors between the acquired images and distance information and the actual results, which will reduce the accuracy of target detection and cause path deviation and erroneous prompt information for blind navigation. At the same time, due to the limited performance of convolutional networks, the Yolo detection model cannot accurately detect objects in complex scenes.

[0007] In order to solve the above problems, the present invention provides a navigation method and device for the blind. Summary of the invention

[0008] The purpose of the present invention is to provide a navigation method and device for the blind, including: first, the present invention proposes an image depth prediction model based on a hybrid network, inputs image data captured by a camera in real time into the model, and improves the output quality of depth information by adopting a feature fusion strategy; secondly, a depth-guided image detection model is proposed, which embeds the output features of the depth prediction model into a traditional detection model, and accurately obtains the category and position information of the object by using a depth information-guided solution; finally, the depth information, category and position information are integrated and converted into real three-dimensional coordinate information for distinguishing the path and obstacles.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A navigation method for the blind, comprising:

[0011] Using a camera to obtain image data of the surrounding environment in real time; wherein the image data includes multiple consecutive frames of images;

[0012] Processing the image data using a positioning and map building algorithm to obtain camera position information;

[0013] Performing image depth processing on the image data through an image depth prediction model based on a hybrid network to obtain depth information of each pixel in the image data; wherein the image depth prediction model can fuse local and global features with the help of a feature fusion module;

[0014] Detecting the image data using a depth-guided image detection model to obtain category and position information of objects in the image data;

[0015] Integrate the pose information of the camera, the depth information of each pixel, and the category and position information of the object, and calculate the world coordinate position information of the obstacle according to the pinhole camera model formula;

[0016] The system determines whether the user has deviated from the walking route based on the predicted trajectory and the actual path, and sends a signal to the user whether there is an obstacle based on the location information of the obstacle detection.

[0017] The positioning and map construction algorithm is proposed by a Chinese patent with patent application number CN201711401325.0, and is a positioning and map construction algorithm based on ORB feature extraction.

[0018] Furthermore, the process of obtaining the camera pose information through the positioning and map construction algorithm based on ORB feature extraction includes a rotation matrix R and a translation vector t.

[0019] The hybrid network-based image depth prediction model is a hybrid network algorithm based on sensor feature embedding. The input image is pre-trained according to the hybrid network algorithm based on sensor feature embedding, and pixel depth information in the image data is output.

[0020] Furthermore, the hybrid network algorithm based on sensor feature embedding includes: convolutional layers, downsampling and upsampling, skip connections, residual blocks, Transformer blocks and feature fusion modules.

[0021] The specific implementation process of the hybrid network algorithm based on sensor feature embedding includes:

[0022] Performing feature extraction on the input image to the convolution layer to obtain an initial feature map;

[0023] The encoding end of the residual convolutional network first inputs the initial feature map into multiple serial residual blocks, and then performs the downsampling operation twice to obtain local features;

[0024] The encoding end of the Transformer network performs the downsampling operation twice on the initial feature map, and extracts global features using multiple serial Transformer blocks;

[0025] The decoding end of the residual convolutional network first uses a feature fusion module to combine the local features with the global features to obtain fused features; then the fused features are upsampled twice, and then the same number of residual blocks are used before each operation, and sensor features are embedded before the jump connection, so as to finally obtain the depth information of the image.

[0026] Furthermore, the specific implementation process of the feature fusion module in the residual convolutional network algorithm includes: the local feature map is fused through the channel dimension to obtain a fused feature map combined with the global feature; the fused feature map is subjected to a convolution kernel of size 3×3 to obtain a convolution intermediate feature map; the convolution intermediate feature map is subjected to channel fusion and a nonlinear activation function to obtain a weighted feature map mask; the weighted feature map mask is applied to the convolution intermediate feature map and summed with the local feature map to obtain a depth enhancement feature; the specific formula of the feature fusion module is:

[0027] M = Conv(Concat(X,Y));

[0028] F(X,M)=M*Sigmoid(Concat(X,M))+X;

[0029] Among them, F() represents the merging function of the feature fusion module; X represents the local feature map; Y represents the global feature map; M represents the fused feature map; Sigmoid() represents the nonlinear activation function; Conv() represents the convolution operation with a convolution kernel size of 3×3; Concat() represents the channel merging operation.

[0030] The image detection model based on depth guidance is a convolutional network algorithm based on embedding of depth information features, and the implementation process includes:

[0031] The convolutional network applies a convolutional layer to the input image to extract features and obtain an initial feature map;

[0032] The encoding end performs multiple downsampling operations on the initial feature map to reduce feature resolution and increase output receptive field. Before each operation, multiple feature extraction modules are used to improve the image feature capture capability, and high-level semantic features are obtained by encoding.

[0033] The decoding end performs multiple upsampling operations on the high-level semantic features to increase the feature resolution to be consistent with the resolution of the encoder at the same level. The same number of feature extraction modules are used before each operation to improve the accuracy of network detection. At the same time, the convolutional features before downsampling are added to the features after upsampling using jump connections to make up for the information lost due to downsampling and feature extraction. Then, the depth enhancement features are embedded before the jump connection, and the relative position relationship of the scene objects in the depth feature map is used to help locate the position of the detection target. The decoding end finally obtains the image category and position information;

[0034] The process of obtaining the image category and location information can be expressed by the following formula:

[0035] Inf = C(X,D);

[0036] Among them, C() represents the convolutional network that embeds the depth information feature; X represents the initial feature map; D represents the depth enhanced feature map; Inf represents the image category and position information.

[0037] A navigation device for the blind, comprising: a data acquisition module, an information processing module and a navigation module;

[0038] The data acquisition module is used to acquire image information in a blind person's visual field in real time through a camera, wherein the image information includes continuous multi-frame image data;

[0039] The information processing module can be divided into an algorithm unit and a conversion unit;

[0040] The algorithm unit is used to process the image information through a positioning and map building algorithm, an image depth prediction algorithm based on a hybrid network, and an image detection algorithm based on depth guidance to obtain the pose information of the camera, the depth information of each pixel, and the category and position information of the object in the image information;

[0041] The conversion unit is used to integrate the posture information of the camera, the depth information of each pixel, and the category and position information of the object in the image information and convert them into the spatial position of the scene object in the three-dimensional coordinate system, and obtain the predicted path trajectory after processing the posture information of the camera;

[0042] The navigation module is used to determine whether the user deviates from the walking route based on the predicted movement trajectory and the actual path, and to send a signal to the user whether there is an obstacle based on the position information of the obstacle detection;

[0043] Furthermore, the image detection model is a convolutional network algorithm based on deep information feature embedding, and the process includes:

[0044] The convolutional network applies a convolutional layer to the input image to extract features and obtain an initial feature map;

[0045] The encoding end performs multiple downsampling and feature extraction operations on the initial feature map, and encodes it to obtain high-level semantic features;

[0046] The decoding end performs multiple upsampling, feature extraction and skip connection operations on the high-level semantic features, and embeds depth enhancement features before the skip connection operation, and uses the relative position relationship of the scene objects in the depth feature map to help locate the position of the detection target. The decoding end finally obtains the image category and position information;

[0047] The process of obtaining the image category and location information can be expressed by the following formula:

[0048] Inf = C(X,D);

[0049] Among them, C() represents the convolutional network that embeds the depth information feature; X represents the initial feature map; D represents the depth enhanced feature map output by the image depth prediction model; Inf represents the image category and position information.

[0050] Furthermore, the image depth prediction model is a hybrid network algorithm based on sensor feature embedding, including: convolutional layers, downsampling and upsampling, skip connections, residual blocks and feature fusion modules;

[0051] The specific implementation process of the hybrid network algorithm based on sensor feature embedding can be expressed as:

[0052] Performing feature extraction on the input image to the convolution layer to obtain an initial feature map;

[0053] The encoding end of the residual convolutional network first inputs the initial feature map into multiple serial residual blocks, and then performs the downsampling operation twice to obtain local features;

[0054] The encoding end of the Transformer network performs the downsampling operation twice on the initial feature map, and extracts global features using multiple serial Transformer blocks;

[0055] The decoding end of the residual convolution network first uses a feature fusion module to combine the local features with the global features to obtain fused features; then the fused features are upsampled twice, and then the same number of residual blocks are used before each operation, and sensor features are embedded before the jump connection, so as to finally obtain the depth information of the image;

[0056] The specific implementation process of the feature fusion module can be expressed as follows:

[0057] The local feature map is fused with the global feature via the channel dimension to obtain a fused feature map;

[0058] The fused feature map is convolved through a convolution kernel of size 3×3 to obtain a convolution intermediate feature map;

[0059] The convolution intermediate feature map is subjected to channel fusion and nonlinear activation function to obtain a weighted feature map mask;

[0060] Applying the weighted feature map mask to the convolution intermediate feature map and summing it with the local feature map to obtain a depth enhancement feature;

[0061] The specific formula of the feature fusion module is:

[0062] M = Conv(Concat(X,Y));

[0063] F(X,M)=M*Sigmoid(Concat(X,M))+X;

[0064] Among them, F() represents the merging function of the feature fusion module; X represents the local feature map; Y represents the global feature map; M represents the fused feature map; Sigmoid() represents the nonlinear activation function; Conv() represents the convolution operation with a convolution kernel size of 3×3; Concat() represents the channel merging operation.

[0065] The blind navigation device based on depth prediction and guidance includes: a data acquisition module provided with a camera; an information processing module provided with an algorithm unit and a conversion unit, wherein the algorithm unit includes a positioning and map building algorithm, an image depth prediction algorithm based on a hybrid network, and an image detection algorithm based on depth guidance; and a navigation module provided with navigation information and obstacle avoidance information.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. The present invention proposes an image depth prediction model based on a hybrid network. By inputting the image data captured by the camera into the model and adopting the strategy of sensor feature embedding, the environmental perception ability of the model is improved. The model is a hybrid network algorithm based on sensor feature embedding. The input image is pre-trained according to the hybrid network algorithm based on sensor feature embedding, and the depth information in the image pixels is accurately output, thereby improving the accuracy of the detection model and three-dimensional coordinate conversion.

[0068] 2. The present invention proposes a feature fusion module, which increases the overall receptive field of the image depth prediction model based on the hybrid network by improving the traditional channel merging method, and improves the accuracy of depth information prediction. The module obtains a fused feature map combining local features and global features through channel dimension fusion operation, and then the fused feature map is used to filter the information in the feature map through convolution operation and weighted feature map mask operation, so as to avoid the interference of redundant information on depth prediction and improve the output quality of the image depth prediction model.

[0069] 3. The present invention proposes an image detection model based on depth guidance, which embeds the output features of the depth prediction model into the traditional detection model, and accurately obtains the category and position information of the object by using a depth information guided scheme; the model is a convolutional network algorithm based on depth information feature embedding, which embeds the depth enhancement features output by the depth prediction model before the decoder downsampling operation, and uses the relative position relationship of the scene objects in the depth feature map to help locate the position of the detection target, so that the decoding end finally obtains more accurate image category and position information. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flowchart of a navigation method for the blind provided by the present invention;

[0071] Figure 2 A schematic diagram of a specific implementation framework of a navigation method for the blind provided by the present invention;

[0072] Figure 3 A schematic diagram of the network structure of the image depth prediction model based on the hybrid network provided by the present invention;

[0073] Figure 4 A schematic diagram of a residual block and a feature fusion module provided by the present invention;

[0074] Figure 5 A schematic diagram of the network structure of the depth-guided image detection model provided by the present invention;

[0075] Figure 6 The present invention provides a structural schematic diagram of a navigation device for the blind.

[0076] In the figure: 10. A navigation device for the blind; 20. A data acquisition module; 30. An information processing module; 40. A navigation module. DETAILED DESCRIPTION

[0077] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Traditional navigation methods for the blind usually involve using sensors (such as ultrasonic sensors or infrared sensors) to detect the surrounding environment and use non-visual information such as sound, touch or vibration to guide the blind to avoid obstacles and navigate to the destination. Due to the limitations of sensor types, the environmental perception capabilities of traditional methods are usually limited, and the ability to obtain complex environments or detailed information is relatively weak, making it difficult to cope with complex intersections or obstacle layouts.

[0079] The existing technology has played a certain role in detecting simple obstacle avoidance in the blind navigation environment, but the use of traditional sensors to collect image information and distance information is relatively weak in the acquisition of complex environments or detailed information, which will cause incorrect prediction of the depth information of complex objects, which is not conducive to the subsequent conversion into accurate three-dimensional position information, resulting in path deviation and erroneous prompt information for blind navigation. At the same time, the depth information and target detection are not combined to improve the accuracy of user navigation.

[0080] A navigation method and device for the blind comprises: firstly, the present invention proposes an image depth prediction model based on a hybrid network, inputs image data captured by a camera in real time into the model, and improves the output quality of depth information by adopting a feature fusion strategy; secondly, a depth-guided image detection model is proposed, the model embeds the output features of the depth prediction model into a traditional detection model, and accurately obtains the category and position information of the object by using a depth information-guided solution; finally, the depth information, category and position information are integrated and converted into real three-dimensional coordinate information for distinguishing the path and obstacles.

[0081] Embodiment 1

[0082] The specific implementation process in the embodiment of the present application will be implemented by a blind navigation method of the present invention, see Figure 1 The process of the method proposed in the present invention is described in the following; the navigation method for the blind comprises:

[0083] S10. Using the camera to obtain image data of the surrounding environment in real time;

[0084] S20. Obtain the camera's posture information R and t, the depth information of each pixel, and the category and position information of the object through algorithm processing;

[0085] S30. Integrate the pose information of the camera, the depth information of each pixel, and the category and position information of the object and convert them into the spatial position of the scene object in the three-dimensional coordinate system;

[0086] S40. Determine whether the user deviates from the walking route based on the predicted movement trajectory and the actual path, and send a signal to the user whether there is an obstacle based on the location information of the obstacle detection.

[0087] Further, Figure 1 The steps in Figure 2 The algorithm model is implemented, and the specific working process can be summarized as follows:

[0088] Use the device camera to capture continuous image data to meet the input requirements of subsequent algorithm processing;

[0089] The camera posture, image depth information, and obstacle location and category information are obtained through algorithm processing.

[0090] The details are as follows: Use the positioning and mapping algorithm based on ORB feature extraction to obtain the camera pose information, and calculate the camera rotation matrix R and translation vector t; Use the trained image depth prediction model based on the hybrid network to obtain the depth map, which reflects the depth relationship between different pixels and objects in the scene. The redder the pixel, the deeper the depth of the pixel; Use the depth-guided image detection model to obtain the location and category information of the obstacle; Then, integrate different information and calculate the world coordinate position information of the obstacle according to the pinhole camera model formula to provide a basis for judging the subsequent navigation information;

[0091] The positioning information obtained by the algorithm is used to determine whether it is consistent with the user's preset path and provide navigation services. The user's surrounding environment information can be used to remind and avoid obstacles in time. For example, if there are trees blocking the user within 5 meters or a car passing by within 15 meters, the user will be reminded to avoid them.

[0092] The positioning and map building algorithm includes a positioning and map building algorithm based on ORB feature extraction.

[0093] Furthermore, the process of obtaining the camera posture information by the positioning and mapping algorithm based on ORB feature extraction includes:

[0094] First, ORB feature points are extracted for each frame of image.

[0095] Then, ORB feature point matching is performed on the previous and next two frames of images.

[0096] Next, it is determined whether the number of matching ORB feature points of two consecutive frames of images exceeds a preset matching threshold. If so, the basic matrix F and the homography matrix H are calculated using the matching ORB feature points.

[0097] According to the basic matrix F and the homography matrix H and the selection criteria, the matrix is ​​selected and calculated to obtain the position and posture information of the camera, including the rotation matrix R and the translation vector t.

[0098] In order to record the user's travel route, the negative transposed matrix of the rotation matrix R is multiplied by the translation vector t to obtain the three-dimensional coordinate points of the current user, and then each coordinate point is connected together to obtain the actual travel route.

[0099] The positioning and mapping algorithm in this example is based on ORB feature extraction. The ORB feature of this algorithm combines FAST key point detection and BRIEF descriptor, which enables the algorithm to quickly detect and describe feature points in the image; at the same time, the BRIEF descriptor used to describe key points is binary, has a small storage space requirement, and can resist the influence of noise and perspective changes to a certain extent. Therefore, this algorithm is suitable for real-time positioning and mapping applications.

[0100] The specific implementation process of the positioning and map construction algorithm described in this example includes: (i) extracting ORB feature points from each frame of the image; (ii) determining whether the number of ORB feature point matches between the two frames of the image exceeds a threshold; (iii) if the threshold is exceeded, the basic matrix F and the homography matrix H are calculated by matching feature points; (iv) the camera's position information is calculated. The algorithm uses ORB feature extraction and ORB feature matching mechanisms to obtain accurate camera position information, and obtains accurate user location information through calculation, thereby improving the accuracy of the navigation prompt route.

[0101] The hybrid network-based image depth prediction model is a hybrid network algorithm based on sensor feature embedding. The input image is pre-trained according to the hybrid network algorithm based on sensor feature embedding, and pixel depth information in the image data is output.

[0102] As an embodiment of the present invention, refer to Figure 3 , the image depth prediction model is a hybrid network algorithm based on sensor feature embedding, including: convolutional layer, downsampling and upsampling, skip connection, residual block and feature fusion module;

[0103] The specific implementation process of the hybrid network algorithm based on sensor feature embedding can be expressed as:

[0104] Performing feature extraction on the input image to the convolution layer to obtain an initial feature map;

[0105] The encoding end of the residual convolutional network first inputs the initial feature map into multiple serial residual blocks, and then performs the downsampling operation twice to obtain local features;

[0106] The encoding end of the Transformer network performs the downsampling operation twice on the initial feature map, and extracts global features using multiple serial Transformer blocks;

[0107] The decoding end of the residual convolutional network first uses a feature fusion module to combine the local features with the global features to obtain fused features; then the fused features are upsampled twice, and then the same number of residual blocks are used before each operation, and sensor features are embedded before the jump connection, so as to finally obtain the depth information of the image.

[0108] In addition, in order to further illustrate that the sensor feature embedding proposed in the present invention plays a certain effectiveness in image depth prediction, the model proposed in the present invention is compared with the model with the same function. The comparison results of various evaluation indicators are given in Table 1.

[0109] Table 1. Performance comparison of each model for image depth prediction

[0110]

[0111] The depth prediction performance comparison data of each model is given in Table 1, and in order to illustrate the necessity of deep feature embedding in the image depth prediction model. It can be concluded from the table that when the image depth prediction model does not embed sensor features, the root mean square error, relative error and accuracy are 0.592m, 14.26% and 88.41% respectively, and the model performance is significantly lower than the image depth prediction model with sensor feature embedding; therefore, it can be seen that sensor feature embedding can significantly improve the performance of the image depth prediction model.

[0112] In this example, CNN and Transformer networks are combined in the image depth prediction model based on sensor feature embedding. The input image is pre-trained according to the hybrid network algorithm based on sensor feature embedding, and the depth information of each pixel is output. The depth relationship and relative position relationship between each object in the image can be used to improve the detection performance of the target detection model on the one hand, and the accuracy of three-dimensional coordinate conversion on the other hand.

[0113] The hybrid network algorithm based on sensor feature embedding can extract CNN features containing spatial contextual relationships, and at the same time obtain global contextual features by utilizing the long-distance dependency characteristics of Transformer. By embedding sensor features, additional image features and distance features are added to the model, and depth enhancement features are output. In this way, the image depth prediction model can output more accurate image depth information and improve the accuracy of navigation prompt information.

[0114] As an embodiment of the present invention, refer to Figure 4 The specific implementation process of the residual block in the image depth prediction model includes: the input feature map is passed through a convolution kernel of size 3×3, a RELU activation function and a convolution kernel of size 3×3 to obtain the output feature; the specific formula of the residual block is:

[0115] O = Conv(RELU(Conv(I)));

[0116] Among them, I represents the input feature map; O represents the output feature map; RELU() represents the RELU activation function; Conv represents the convolution operation with a convolution kernel size of 3×3.

[0117] The specific implementation process of the feature fusion module includes: the local feature map is fused through the channel dimension to obtain a fused feature map combined with the global feature; the fused feature map is subjected to a convolution kernel of size 3×3 to obtain a convolution intermediate feature map; the convolution intermediate feature map is subjected to channel fusion and a nonlinear activation function to obtain a weighted feature map mask; the weighted feature map mask is applied to the convolution intermediate feature map and summed with the local feature map to obtain a depth enhancement feature; the specific formula of the feature fusion module is:

[0118] M = Conv(Concat(X,Y));

[0119] F(X,M)=M*Sigmoid(Concat(X,M))+X;

[0120] Among them, F() represents the merging function of the feature fusion module; X represents the local feature map; Y represents the global feature map; M represents the fused feature map; Sigmoid() represents the nonlinear activation function; Conv represents the convolution operation with a convolution kernel size of 3×3; Concat() represents the channel merging operation.

[0121] In addition, in order to further illustrate that the feature fusion module proposed in the present invention plays a certain effectiveness in image depth prediction, the model proposed in the present invention is compared with the model with the same function. The comparison results of various evaluation indicators are given in Table 2.

[0122] Table 2. Comparison of the performance of each model for image depth prediction

[0123]

[0124] The depth prediction performance comparison data of each model is given in Table 2, and in order to illustrate the necessity of the feature fusion module in the image depth prediction model. It can be concluded from the table that when the image depth prediction model has no feature fusion module, the root mean square error, relative error and accuracy are 0.624m, 15.01% and 87.96% respectively, and the model performance is significantly lower than the image depth prediction model with feature fusion module; therefore, it can be seen that the feature fusion module can significantly improve the performance of the image depth prediction model.

[0125] In this example, a feature fusion module is introduced into the image depth prediction model; after the channel dimension, 3×3 convolution kernel, nonlinear activation function and feature map mask weighting operation, the feature fusion module increases the overall receptive field of the image depth prediction model based on the hybrid network; at the same time, it can effectively fuse local features and global features, avoid redundant information in the output depth features, and improve the accuracy of the model depth information prediction.

[0126] As an embodiment of the present invention, refer to Figure 5 The image detection model based on depth guidance is a convolutional network algorithm based on embedding of depth information features, and the implementation process includes:

[0127] The convolutional network applies a convolutional layer to the input image to extract features and obtain an initial feature map;

[0128] The encoding end performs multiple downsampling operations on the initial feature map to reduce feature resolution and increase output receptive field. Before each operation, multiple feature extraction modules are used to improve the image feature capture capability, and high-level semantic features are obtained by encoding.

[0129] The decoding end performs multiple upsampling operations on the high-level semantic features to increase the feature resolution to be consistent with the resolution of the encoder at the same level. The same number of feature extraction modules are used before each operation to improve the accuracy of network detection. At the same time, the convolutional features before downsampling are added to the features after upsampling using skip connections to make up for the information lost due to downsampling and feature extraction. Then, the depth enhancement features are embedded before the skip connection operation, and the relative position relationship of the scene objects in the depth feature map is used to help locate the position of the detection target. The decoding end finally obtains the image category and position information;

[0130] The process of obtaining the image category and location information can be expressed by the following formula:

[0131] Inf = C(X,D);

[0132] Among them, C() represents the convolutional network that embeds the depth information feature; X represents the initial feature map; D represents the depth enhanced feature map; Inf represents the image category and position information.

[0133] In addition, in order to further illustrate the effectiveness of the deep feature embedding proposed in the present invention in image detection, the model proposed in the present invention is compared with the model with the same function. The comparison results of various evaluation indicators are given in Table 3.

[0134] Table 3. Comparison of image detection performance of each model

[0135]

[0136] The detection performance comparison data of each model is given in Table 3, and in order to illustrate the necessity of deep feature embedding in the image detection model. It can be concluded from the table that when the image detection model does not embed deep features, the accuracy, false alarm rate and detection rate are 81.65%, 13.28% and 82.04% respectively, and the model performance is significantly lower than the image detection model with deep feature embedding; therefore, it can be seen that deep feature embedding can significantly improve the detection performance of the image detection model.

[0137] In this example, the depth enhancement features with consistent scale are embedded in each level of the output end of the image detection model algorithm; the depth enhancement features contain the relative relationship between the depth relationship and position of each object in the scene. The image detection model can use the depth enhancement feature information to accurately determine the location information and category information of the obstacle, so that the system can output more accurate prompt information.

[0138] According to the actually obtained pixel depth information D, the camera intrinsic parameter K, the rotation matrix R, the translation vector t and the pixel coordinate Pr of the detected object, the real three-dimensional coordinate information Pw of the detected object can be obtained by the formula D*Pr=K*(R*Pw+t), so that corresponding obstacle avoidance prompt information can be provided according to the user's surrounding environment.

[0139] Embodiment 2

[0140] As an embodiment of the present invention, refer to Figure 6 , a blind navigation device 10 includes: a data acquisition module, an information processing module and a navigation module;

[0141] The data acquisition module 20 is used to acquire image information in a visual environment of a blind person in real time through a camera, wherein the image information includes continuous multi-frame image data;

[0142] The information processing module 30 integrates an algorithm unit and a conversion unit. The algorithm unit is used to process the image information through a positioning and map building algorithm, an image depth prediction algorithm based on a hybrid network, and an image detection algorithm based on depth guidance to obtain the camera pose information, the depth information of each pixel in the image information, and the category and position information of the object in the image information. The conversion unit is used to integrate the camera pose information, the depth information of each pixel in the image information, and the category and position information of the object in the image information and convert them into the spatial position of the scene object in the three-dimensional coordinate system, and obtain the predicted path trajectory after processing the camera pose information;

[0143] The navigation module 40 is used to determine whether the user deviates from the walking route according to the predicted movement trajectory and the actual path, and to send a signal to the user whether there is an obstacle according to the position information of the obstacle detection.

[0144] The process of obtaining the camera position information through the positioning and mapping algorithm based on ORB feature extraction includes:

[0145] First, ORB feature points are extracted for each frame of image;

[0146] Then, ORB feature point matching is performed on the two frames of images;

[0147] Next, determine whether the number of matching ORB feature points of two consecutive frames of images exceeds the preset matching threshold. If it exceeds, calculate the basic matrix F and the homography matrix H through the matching ORB feature points;

[0148] According to the basic matrix F and the homography matrix H and the selection criteria, the matrix is ​​selected and calculated to obtain the position and posture information of the camera, including the rotation matrix R and the translation vector t.

[0149] In order to record the user's travel route, the negative transposed matrix of the rotation matrix R is multiplied by the translation vector t to obtain the three-dimensional coordinate points of the current user, and then each coordinate point is connected together to obtain the actual travel route.

[0150] The image depth prediction model based on the hybrid network includes a hybrid network algorithm based on sensor feature embedding, the input image is pre-trained according to the hybrid network algorithm based on sensor feature embedding, and the pixel depth information in the image data is output. The specific implementation process includes:

[0151] Performing feature extraction on the input image to the convolution layer to obtain an initial feature map;

[0152] The encoding end of the residual convolutional network performs the downsampling operation twice on the initial feature map to reduce the feature resolution and increase the output receptive field. Before each operation, multiple serial residual blocks are used to improve the image feature capture capability and obtain local features.

[0153] The encoding end of the Transformer network performs the downsampling operation twice on the initial feature map, and extracts global features using multiple serial Transformer blocks;

[0154] Among them, the Transformer block is composed of residual connection, layer normalization, multi-head transposed attention, and gated convolutional forward network.

[0155] The decoder of the residual convolutional network first uses a feature fusion module to combine local features with global features extracted by Transformer to obtain fused features; then the fused features are upsampled twice to increase the feature resolution to keep it consistent with the resolution of the encoder at the same level; then the same number of residual blocks are used before each operation to improve the conversion quality of the depth features, and sensor features are embedded before the jump connection, and finally the image depth information is obtained;

[0156] Except for the last convolutional layer of the hybrid network, which has 3 output channels, the remaining channels are all 64.

[0157] The image detection model is a convolutional network algorithm based on deep information feature embedding. The specific implementation process includes:

[0158] The convolutional network applies a convolutional layer to the input image to extract features and obtain an initial feature map;

[0159] The encoding end performs multiple downsampling and feature extraction operations on the initial feature map, and encodes it to obtain high-level semantic features;

[0160] The decoding end performs multiple upsampling, feature extraction and skip connection operations on the high-level semantic features, and embeds depth enhancement features before the skip connection operation, and uses the relative position relationship of the scene objects in the depth feature map to help locate the position of the detection target. The decoding end finally obtains the image category and position information;

[0161] The process of obtaining the image category and location information can be expressed by the following formula:

[0162] Inf = C(X,D);

[0163] Among them, C() represents the convolutional network that embeds the depth information feature; X represents the initial feature map; D represents the depth enhanced feature map output by the image depth prediction model; Inf represents the image category and position information.

[0164] The image depth prediction model is a hybrid network algorithm based on residual convolution and Transformer, including: convolution layer, downsampling and upsampling, skip connection, residual block and feature fusion module;

[0165] The residual convolutional network extracts features from the input image to the convolutional layer to obtain an initial feature map;

[0166] The encoding end performs the downsampling operation twice on the initial feature map to reduce feature resolution and increase output receptive field, and uses multiple serial residual blocks before each operation to improve image feature capture capability and obtain low-dimensional features;

[0167] The decoding end performs the upsampling operation on the low-dimensional features twice to increase the feature resolution to be consistent with the resolution of the encoder at the same level. The same number of residual blocks are used before each operation to improve the conversion quality of the deep features. Then, the feature fusion module is used to combine the local features with the global features extracted by the Transformer. At the same time, the convolutional features before downsampling are added to the features after upsampling using jump connections to make up for the information lost due to downsampling and feature extraction. The decoding end finally obtains the image depth information. Among them, the specific implementation process of the feature fusion module can be expressed as:

[0168] The local feature map is fused with the global feature via the channel dimension to obtain a fused feature map;

[0169] The fused feature map is convolved through a convolution kernel of size 3×3 to obtain a convolution intermediate feature map;

[0170] The convolution intermediate feature map is subjected to channel fusion and nonlinear activation function to obtain a weighted feature map mask;

[0171] Applying the weighted feature map mask to the convolution intermediate feature map and summing it with the local feature map to obtain a depth enhancement feature;

[0172] The specific formula of the feature fusion module is:

[0173] M = Conv(Concat(X,Y));

[0174] F(X,M)=M*Sigmoid(Concat(X,M))+X;

[0175] Among them, F() represents the merging function of the feature fusion module; X represents the local feature map; Y represents the global feature map; M represents the fused feature map; Sigmoid() represents the nonlinear activation function; Conv represents the convolution operation with a convolution kernel size of 3×3; Concat() represents the channel merging operation.

[0176] The conversion unit can obtain the three-dimensional coordinate position information of the detected object through the pinhole camera model formula D*Pr=K*(R*Pw+t) according to the actual pixel depth information D, the camera intrinsic parameter K, the rotation matrix R, the translation vector t and the pixel coordinate Pr of the detected object. The navigation module 40 can provide the user with accurate navigation information and obstacle avoidance prompt information according to the user's surrounding environment, the actual travel route and the actual obstacle detection position.

[0177] While embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art that various changes and modifications may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A navigation method for the blind, characterized in that: include: Using a camera to obtain image data of the surrounding environment in real time; wherein the image data includes multiple consecutive frames of images; Processing the image data using a positioning and map building algorithm to obtain camera position information; Performing image depth processing on the image data through an image depth prediction model based on a hybrid network to obtain depth information of each pixel in the image data; wherein the image depth prediction model can fuse local and global features with the help of a feature fusion module; Detecting the image data using a depth-guided image detection model to obtain category and position information of objects in the image data; Integrate the position information of the camera, the depth information of each pixel, and the category and position information of the object, and calculate the world coordinate position information of the obstacle according to the pinhole camera model; The system determines whether the user has deviated from the walking route based on the predicted trajectory and the actual path, and sends a signal to the user whether there is an obstacle based on the location information of the obstacle detection.

2. A navigation method for the blind according to claim 1, characterized in that: The positioning and mapping algorithm is an algorithm based on ORB feature extraction.

3. A navigation method for the blind according to claim 2, characterized in that: The camera pose information obtained by the positioning and mapping algorithm includes a rotation matrix R and a translation vector t.

4. A navigation method for the blind according to claim 1, characterized in that: The hybrid network-based image depth prediction model is a hybrid network algorithm based on sensor feature embedding. The input image is pre-trained according to the hybrid network algorithm based on sensor feature embedding, and pixel depth information in the image data is output.

5. A navigation method for the blind according to claim 4, characterized in that: The hybrid network algorithm based on sensor feature embedding includes: convolutional layer, downsampling and upsampling, skip connection, residual block, Transformer block and feature fusion module; The specific implementation process of the hybrid network algorithm based on sensor feature embedding includes: Performing feature extraction on the input image to the convolution layer to obtain an initial feature map; The encoding end of the residual convolutional network first inputs the initial feature map into multiple serial residual blocks, and then performs the downsampling operation twice to obtain local features; The encoding end of the Transformer network performs the downsampling operation twice on the initial feature map, and extracts global features using multiple serial Transformer blocks; The decoding end of the residual convolutional network first uses a feature fusion module to combine the local features with the global features to obtain fused features; then the fused features are upsampled twice, and then the same number of residual blocks are used before each operation, and sensor features are embedded before the jump connection, so as to finally obtain the depth information of the image.

6. A navigation method for the blind according to claim 5, characterized in that: The specific implementation process of the feature fusion module in the hybrid network algorithm includes: the local feature map is fused through the channel dimension to obtain a fused feature map combined with the global feature; the fused feature map is subjected to a convolution kernel of size 3×3 to obtain a convolution intermediate feature map; the convolution intermediate feature map is subjected to channel fusion and a nonlinear activation function to obtain a weighted feature map mask; the weighted feature map mask is applied to the convolution intermediate feature map and summed with the local feature map to obtain a depth enhancement feature; the specific formula of the feature fusion module is: M = Conv(Concat(X,Y)); F(X,M)=M*Sigmoid(Concat(X,M))+X; Among them, F() represents the merging function of the feature fusion module; X represents the local feature map; Y represents the global feature map; M represents the fused feature map; Sigmoid() represents the nonlinear activation function; Conv() represents the convolution operation with a convolution kernel size of 3×3; Concat() represents the channel merging operation.

7. A navigation method for the blind according to claim 1, characterized in that: The image detection model based on depth guidance is a convolutional network algorithm based on embedding of depth information features. The specific implementation process includes: The convolutional network applies a convolutional layer to the input image to extract features and obtain an initial feature map; The encoding end performs multiple downsampling operations on the initial feature map, and obtains high-level semantic features by using multiple feature extraction modules before each downsampling operation; The decoding end performs multiple upsampling operations on the high-level semantic features to keep the feature resolution consistent with the resolution of the encoder at the same level. The same number of feature extraction modules are used before each operation. At the same time, the convolutional features before downsampling are added to the feature map of the upsampled features using skip connections. Then, the depth enhancement features are embedded before the skip connections. The relative position relationship of the scene objects in the depth feature map is used to help locate the position of the detection target, and the category and position information of the image are finally obtained. The process of obtaining the image category and location information can be expressed by the following formula: Inf = C(X,D); Among them, C() represents the convolutional network that embeds the depth information feature; X represents the initial feature map; D represents the depth enhanced feature map; Inf represents the image category and position information.

8. A navigation device for the blind, characterized in that: include: Data acquisition module, information processing module and navigation module; The data acquisition module is used to acquire image information in a blind person's visual field in real time through a camera, wherein the image information includes continuous multi-frame image data; The information processing module includes an algorithm unit and a conversion unit, wherein the algorithm unit is used to process the image information through a positioning and map building algorithm, an image depth prediction algorithm based on a hybrid network, and an image detection algorithm based on depth guidance to obtain the camera posture information, the depth information of each pixel in the image information, and the category and position information of the object in the image information; The conversion unit is used to integrate the camera posture information, the depth information of each pixel in the image information, and the category and position information of the object in the image information and convert them into position information in a three-dimensional coordinate system, and obtain a predicted path trajectory after processing the camera posture information; The navigation module is used to determine whether the user deviates from the walking route based on the predicted movement trajectory and the actual path, and to send a signal to the user whether there is an obstacle based on the location information of the obstacle detection.

9. A navigation device for the blind according to claim 8, characterized in that: The algorithm unit processes the multiple frames of image data using an image depth prediction model to obtain depth information of each pixel; The image depth prediction model is a hybrid network algorithm based on sensor feature embedding, including: convolutional layer, downsampling and upsampling, skip connection, residual block and feature fusion module; The specific implementation process of the hybrid network algorithm based on sensor feature embedding includes: Performing feature extraction on the input image to the convolution layer to obtain an initial feature map; The encoding end of the residual convolutional network first inputs the initial feature map into multiple serial residual blocks, and then performs the downsampling operation twice to obtain local features; The encoding end of the Transformer network performs the downsampling operation twice on the initial feature map, and extracts global features using multiple serial Transformer blocks; The decoding end of the residual convolution network first uses a feature fusion module to combine the local features with the global features to obtain fused features; then the fused features are upsampled twice, and then the same number of residual blocks are used before each operation, and sensor features are embedded before the jump connection, so as to finally obtain the depth information of the image; The specific implementation process of the feature fusion module can be expressed as follows: The local feature map is fused with the global feature via the channel dimension to obtain a fused feature map; The fused feature map is convolved through a convolution kernel of size 3×3 to obtain a convolution intermediate feature map; The convolution intermediate feature map is subjected to channel fusion and nonlinear activation function to obtain a weighted feature map mask; Applying the weighted feature map mask to the convolution intermediate feature map and summing it with the local feature map to obtain a depth enhancement feature; The specific formula of the feature fusion module is: M = Conv(Concat(X,Y)); F(X,M)=M*Sigmoid(Concat(X,M))+X; Among them, F() represents the merging function of the feature fusion module; X represents the local feature map; Y represents the global feature map; M represents the fused feature map; Sigmoid() represents the nonlinear activation function; Conv() represents the convolution operation with a convolution kernel size of 3×3; Concat() represents the channel merging operation.

10. A navigation device for the blind according to claim 8, characterized in that: The algorithm unit processes the multiple frames of image data using an image detection model to obtain the category and location information of the object; The image detection model is a convolutional network algorithm based on deep information feature embedding, and the specific implementation process includes: The convolutional network applies a convolutional layer to the input image to extract features and obtain an initial feature map; The encoding end performs multiple downsampling and feature extraction operations on the initial feature map to obtain high-level semantic features; The decoding end performs multiple upsampling, feature extraction and skip connection operations on the high-level semantic features, and embeds depth enhancement features before the skip connection operation, and uses the relative position relationship of the scene objects in the depth feature map to help locate the position of the detection target. The decoding end finally obtains the image category and position information; The process of obtaining the image category and location information can be expressed by the following formula: Inf = C(X,D); Among them, C() represents the convolutional network that embeds the depth information feature; X represents the initial feature map; D represents the depth enhanced feature map output by the image depth prediction model; Inf represents the image category and position information.