A dynamic obstacle avoidance method for assisting the blind

Abstract

This invention discloses a dynamic obstacle avoidance method for assistive visually impaired scenarios. First, a feature extraction module based on a hierarchical residual structure constructed within residual blocks is built, which expands the receptive field of deep features. Second, a spatial feature recovery module based on bilinear interpolation and transposed convolution upsampling is designed to make segmentation edges more accurate. Third, a discrete sampling strategy is used to extract obstacle category, distance, and contour information, and path planning is performed using a heuristic search algorithm considering safe distance constraints. Finally, priority rules and logical order rules for the obstacle avoidance warning system are assumed, and a decision is generated based on the planned path, effectively guiding blind users to avoid obstacles through auditory and tactile information. This invention solves the problem that single-stage instance segmentation algorithms struggle to simultaneously and accurately segment small target obstacles and background road surfaces, overcoming the difficulty of providing effective information for obstacle avoidance decisions in assistive visually impaired scenarios with instance segmentation results, enabling blind users to avoid obstacles more intelligently and autonomously.

Description

Technical Field

[0001] This invention relates to the fields of image processing and navigation for the blind, specifically to a dynamic obstacle avoidance method for blind scenes based on instance segmentation. Background Technology

[0002] Obstacle avoidance is a fundamental function of navigation. Obstacle avoidance methods in assistive devices for the visually impaired can be broadly categorized into three types based on the environmental perception sensors used: vision-based, LiDAR-based, and multi-sensor fusion-based. LiDAR-based obstacle avoidance is the most mature, offering high ranging accuracy and stability, and is widely used in robotics, autonomous driving, and assistive devices for the visually impaired. However, it suffers from drawbacks such as the inability to provide semantic information, high cost, and the large size and weight of LiDAR sensors. This patent proposes a dynamic obstacle avoidance method for assistive devices for the visually impaired, utilizing image instance segmentation, which is a vision-based method. It uses RGB cameras or depth cameras as sensors for environmental perception. Traditional vision-based obstacle avoidance methods can use depth cameras to obtain depth maps and other ranging capabilities, approaching the effect of LiDAR, but they still lack semantic information and fall far short of achieving human vision. This patent's dynamic obstacle avoidance method, based on instance segmentation technology, aims to leverage deep learning to enable the visually impaired to simultaneously obtain distance and semantic information about obstacles, achieving obstacle avoidance with a better understanding of the environment.

[0003] In recent years, in-depth research into deep learning algorithms has further improved environmental perception performance, even reaching or surpassing human recognition accuracy in tasks such as image classification, object detection, and image segmentation. Therefore, many deep learning-based image segmentation methods have been proposed. Compared to traditional methods, deep learning-based image segmentation methods have attracted widespread attention due to their stronger generalization and robustness. Image segmentation methods are mainly divided into two categories: semantic segmentation and instance segmentation. Semantic segmentation assigns a category to each pixel in an image, but does not distinguish between objects within the same category. Instance segmentation utilizes the results of both object detection and semantic segmentation, distinguishing different objects within the same category based on semantic segmentation. Deep learning-based image segmentation provides support for visual obstacle avoidance methods.

[0004] The dynamic obstacle avoidance method for assistive visually impaired scenarios proposed in this invention is based on instance segmentation technology. This method has not been proposed before, primarily for the following four reasons: First, instance segmentation algorithms applied to obstacle avoidance in assistive visually impaired scenarios need to accurately segment small obstacle targets and the global background road surface information. Current instance segmentation methods either use complex spatial information recovery strategies to pursue segmentation accuracy for small targets, or pursue extraction of global context information to improve the segmentation accuracy for background targets, resulting in the inability to simultaneously achieve high-precision segmentation for both small and background targets. Second, since the environment for dynamic obstacle avoidance is constantly changing, the obstacle avoidance system needs to capture and process environmental information in real time and generate decisions. Therefore, the real-time performance of the algorithm must be considered. The limitations of algorithm deployment methods in assistive visually impaired scenarios exacerbate this problem. When deployed on a remote computer, the remote transmission speed is affected by signal strength, and it may even be unusable in areas without signal. While deployment on edge devices such as small industrial control computers offers fast transmission, it is limited by their computing power and resources. Secondly, while instance segmentation results contain rich semantic information, existing sensing methods and representation strategies struggle to effectively convey this information to blind individuals. This results in a lack of a bridge between even perfect segmentation results and the ideal understanding of the blind, and even the use of segmentation results to guide blind individuals in obstacle avoidance is rarely explored in existing research. Finally, instance segmentation-based obstacle avoidance methods for the blind first require image processing, a process susceptible to interference from factors such as light intensity, special weather conditions, and motion blur, exhibiting poor robustness. Therefore, researchers in the field of obstacle avoidance for the blind tend to use traditional ranging and sensing methods such as LiDAR rather than visual sensing methods.

[0005] In recent years, with the rapid development of computer vision technology and deep learning methods, the speed and accuracy of image information processing have been greatly improved. Furthermore, due to the limitations of LiDAR in extracting environmental semantic information, vision-based perception technologies have gradually been widely applied. For example, Mancini et al. designed a monocular vision-based assistive system for the blind to help visually impaired individuals complete activities such as walking and jogging. This system includes a camera, a processor, and two gloves equipped with vibration motors. Images are captured by the color camera, the processor extracts line or track information from the images, and the gloves provide left or right movement commands. Experiments have shown that the system can accurately segment the track using deep learning methods, thereby guiding the blind person to avoid obstacles. However, it is only suitable for specific track scenarios and cannot detect multiple types of obstacles, resulting in very limited semantic information. Further research revealed that there is currently no instance segmentation-based dynamic obstacle avoidance method for assistive blind scenarios. Summary of the Invention

[0006] This invention proposes a dynamic obstacle avoidance method for assistive visually impaired scenarios. First, a feature extraction module based on a hierarchical residual structure constructed within residual blocks is built, which expands the receptive field of deep features. Second, a spatial feature recovery module based on bilinear interpolation and transposed convolution upsampling is designed to make the edges of the generated mask more accurate, meeting the accuracy and real-time requirements of assistive visually impaired scenarios. Then, based on the segmentation results, a raster semantic map is generated using a bilinear interpolation algorithm. Assuming a safe traversable area and a distant area for the blind, a discrete sampling strategy is used to determine the presence of obstacles in each area. A heuristic search algorithm is used to find the center point of the distant road surface, and an A* algorithm considering safe distance constraints is used for path planning. Finally, priority and sequence rules for the obstacle avoidance warning system are designed, combined with the planned path generation decision, to effectively guide the blind person to avoid obstacles.

[0007] This invention provides a dynamic obstacle avoidance method for assisting the blind, which consists of three stages: Stage 1 involves image instance segmentation, implemented by a feature extraction module based on deep residual structure, a spatial feature recovery module based on bilinear interpolation and transposed convolution upsampling, a prototype generation module based on multi-layer transposed convolution upsampling, and a coefficient matrix prediction module based on bounding box and mask position regression; Stage 2 extracts the type, location, and contour information of the road surface and obstacles based on the instance segmentation results, followed by path planning, including a raster semantic map generation algorithm, a discrete sampling strategy for determining obstacle type and location, assumptions about the blind person's fan-shaped safety zone and left-front-right-front regions, a heuristic search algorithm for the center point of the far road surface, an A* path planning algorithm considering safety distance constraints, and a path optimization strategy considering pedestrian movement; Stage 3 generates an obstacle avoidance strategy for the blind, including priority and sequence rules for the obstacle avoidance reminder system, which, combined with the results of Stage 2, generates a decision to guide the blind person through auditory and tactile senses.

[0008] Step 1: Prepare the training dataset by dividing the training set, validation set, and test set into a 7:2:1 ratio.

[0009] Step 2: For an image X∈R W*H*3 The first and second dimensions represent the length and width of the feature map, and the third dimension represents the number of channels in the feature map. To transform an image X∈R... W*H*3The input to the feature extraction module based on the deep residual structure first needs to convert the image information into one-dimensional sequence information that conforms to the input of the backbone network. Preprocessing includes image patch partitioning and merging, which solves the input problem. Feature extraction first passes through a 7×7 2D convolutional layer and a 3×3 pooling layer, then through multiple residual modules and cascaded pyramid pooling modules. Each residual module differs from the ResNet network's residual module only in that the 3×3 ordinary 2D convolutional layer is replaced with a novel 3×3 2D convolutional layer with an internally hierarchical residual structure; everything else remains the same. This novel convolution first evenly divides the input feature map into s feature map subsets, which are then divided by x... i This means that, where i∈{1,2,…,s}, each feature map subset has the same spatial size as the input feature map, but the number of channels becomes 1 / s, and each x except x1 has a different spatial size. i All of them undergo a 2D convolution operation with a 3×3 kernel, and this convolution operation is performed using K. i () indicates that y is used to represent y i K represents i The output feature map of (), the final output of this convolutional structure is The addition here means stitching the resulting feature maps together in the third channel, y i The calculation formula is as follows:

[0010]

[0011] The pyramid pooling module consists of three pooling channels with pooling sizes of 12×12, 6×6, and 3×3. Each pooling layer is followed by a 2D convolutional layer with a 1×1 kernel to transform the third dimension of the feature map to 256. Ultimately, this step can extract image feature maps of three different scales, i.e., X1∈R. 69*69*256 X2∈R 35*35*256 X3∈R 18*18*256 ; Set X3∈R 18 *18*256 X4∈R is obtained by a single 2D convolutional layer with a 3×3 kernel. 9*9*256 Then, through a 2D convolutional layer with a 3×3 kernel, X5∈R is obtained. 5*5*256 It should be noted that each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. During training, the parameters of the instance segmentation network are randomly initialized. A set of training data is input, and the network's forward propagation compares the results with the true label values ​​to obtain the error. Backpropagation updates the parameters, and the Adam gradient descent method is used to reduce the loss function. Cross-entropy loss is employed, defined as follows:

[0012]

[0013] Where yi p is the true value i The predicted value is L. t The definition is as follows:

[0014]

[0015] Where N is the number of samples, x is the input data for training, c is the class confidence score, and L... c (x,c) is the category confidence loss function, L b (x,l,g) is the bounding box loss function, where L... m (x,m,g m Let L be the mask position loss function, and α and β be hyperparameters, defined here as 1.25 and 6.125 respectively. c (x,c) is defined as follows:

[0016] L c (x,c)=f(c,1)

[0017] Where c is the category confidence, and L b (x,l,g) is defined as follows:

[0018] L v (x,l,g)=f(l,g)

[0019] Where l is the predicted bounding box value, g is the actual bounding box value, and L m (x,m,g m The definition is as follows:

[0020] L m (x,m,g m )=f(m,g m )

[0021] Where m is the predicted mask value and gt is the actual mask value.

[0022] Step 3: After training, the image is input into the feature extraction module based on the deep residual structure. The resulting feature maps are processed by spatial pooling pyramids to obtain multiple sets of fixed-size feature maps, among which the shallow feature map X1∈R 69*69*256 The feature map X7∈R is obtained by concatenating it with the feature map from the spatial feature recovery module. 69*69*280 The original image X∈R is input into a prototype generation module that generates a set of prototypes of the same size as the input image. The spatial feature recovery module consists of three cascaded 3×3 gated convolutional layers, 3×3 2D convolutional layers, and a bilinear interpolation layer, with the original image X∈R... W*H*3 As input, gated convolutional layers are used to extract edge features from the image. Transposed convolutional layers and bilinear interpolation layers are used to change the number of channels, ultimately generating a feature map X6∈R.69*69*24 splice it to X1∈R 69*69*256 From this, we obtain X7∈R 69*69*280 The prototype generation module based on multi-layer transposed convolution upsampling consists of four transposed convolutional layers with 3×3 kernels and one 2D convolutional layer with 1×1 kernel, yielding P∈R. W*H*32 Deep feature map X2∈R 35*35*256 X3∈R 18 *18*256 X3∈R 18*18*256 X4∈R 9*9*256 X5∈R 5*5*256 A set of coefficient matrices is obtained through the coefficient matrix prediction module based on mask position regression. This module first passes the input through two cascaded 3×3 2D convolutional layers, then through three parallel 3×3 2D convolutional layers, resulting in three sets of coefficient matrices, which are the class confidence prediction matrices C1∈R. W*H*(81*a) Bounding box prediction matrix C2∈R W*H*(4*a) With mask prediction matrix C3∈R W*H*(32*a) Where 'a' represents the number of preset anchor frames. Within certain limits, a larger 'a' results in more accurate position prediction, but the number of parameters increases exponentially, slowing down the algorithm's execution speed. Experiments have shown that a value of 3–8 is optimal. Let C1∈R W*H*(81*a) C2∈R W*H*(4*a) C3∈R W*H*(32*a) Each with P∈R W*H*32 After performing matrix multiplication, the matrix is ​​then processed by the Sigmoid activation function, the expression of which is as follows:

[0023]

[0024] Where x is the input value and e is the base of the natural logarithm. The prototype is then trimmed and concatenated to obtain the final instance segmentation result. This process can be described as follows:

[0025] M = Sigmoid(PC) T )

[0026] Where P is the prototype mask matrix of H×W×k, H and W are the height and width of the input image of the segmentation network, k is the number of prototype groups, C is an n×k coefficient matrix corresponding to n instances filtered by nonmaximum suppression and mask scoring, and M is the final generated mask.

[0027] Step 4: Extract a binary image from the segmentation result. The third channel of this binary image is either 0 or 255, and its size is H×W×1. H and W are the same as the height and width of the image input to the segmentation network. 0 represents a walkable region, and 255 represents an impassable region (containing various obstacles). Based on this, downsampling is used to rasterize the semantic map, resulting in a binary image of size H×W×1. Where S represents the downsampling factor, and its value range is... Max(W,H) represents taking the maximum value of W and H.

[0028] Step 5: Use a discrete sampling strategy to determine if there are obstacles, and define a fan-shaped safe zone two meters away from the blind person. Within this zone, sample four pixels at a time. For areas farther from the blind person, divide them into a left front area and a right front area, sampling six pixels at a time. Collect all pixels that detect obstacles and accumulate them to more than H pixels. o H indicates the presence of an obstacle; all other cases indicate a passable road surface. o Value Where S is the downsampling factor in step 4.

[0029] Step 6: Calculate the relative speeds of the pedestrian and the blind person. D(Δ x ,Δ y ) represents the distance between the pedestrian and the blind user in adjacent frames, where Δ x ,Δ y D(Δ) represents the difference between the x and y coordinates of the center point of the pedestrian instance mask bounding box in the rasterized semantic map and the x and y coordinates of the bottom center point of the binary image. x ,Δ y The distance is the Euclidean distance calculated from the difference between the horizontal and vertical coordinates, where T is the time difference between two adjacent frames. seconds, F r Set the frame rate; calculate the minimum safe distance d = vt, where v is the relative speed between the pedestrian and the blind person, and t is the reaction time, set to two seconds. Draw a circular area with the center point of the pedestrian instance mask bounding box as the center and d as the radius. This area is set as a non-passable area.

[0030] Step 7: To find the endpoint of the path planning, use the algorithm for finding the center point of the farthest road surface. From the top of the binary image (the farthest point from the blind person) to the bottom of the binary image (the current feet of the blind person), traverse line by line and count the pixels of the road surface type. Each time an obstacle type pixel is encountered, the count of the road surface type pixels must be greater than H. r Once a passable road surface is found, the coordinates of the center point of that segment are recorded as the endpoint coordinates of the path planning, and no further traversal is needed, provided the cumulative coordinates are less than H. r Then clear the statistical data to zero and continue iterating, H r Value Where S is the downsampling factor in step 4.

[0031] Step 8: Add the cost of the path distance to the obstacle to the cost function of the A* algorithm. The total cost formula is:

[0032] f(n) = g(n) + h(n) + I(n)

[0033] Where n is the coordinate value of the current node, f(n) is the total cost, g(n) is the actual path length between the current point and the initial point obtained by the recursive algorithm, h(n) is the Manhattan distance between the current node and the initial point, and I(n) is the distance cost between the current node and the obstacle, defined as:

[0034] I(n) = (Ld(n)) * k i

[0035] Where L is the expected pixel distance maintained between the path and the obstacle, set to 20, d(n) is the x-coordinate distance or y-coordinate distance between the current node and its nearest obstacle, and k i This is the proportionality coefficient, and its value range is... Where W and H are the width and height of the binary image, respectively, and k i Used to amplify (Ld(n)) so that Otherwise, as (g(n) + h(n)) increases, I(n) has almost no effect on f(n). Using the above algorithm to process the rasterized semantic map, we finally obtain the shortest path from the current foot point to the farthest road center while maintaining a certain distance from obstacles.

[0036] Step 9: A forward decision can only be made when the safe zone is free of obstacles. If an obstacle suddenly appears in the zone, both vibrating pads will vibrate simultaneously to trigger an alarm. After confirming the safe zone is free of obstacles, calculate the deviation angle between the planned path and the straight-line direction. If the deviation exceeds H... b When the time indicates that the blind person has deviated significantly from the suggested path, the vibration intensity of the left and right hand vibrating pads is used to inform the blind person; the greater the vibration intensity, the greater the deviation angle, so as to advise the blind person to walk in the direction of the suggested path. H b A 25° angle is used; combined with distance detection of distant obstacles, in open areas, the system can provide voice prompts to blind users to move quickly. During the voice prompts, vibration alerts can be provided simultaneously to deal with emergencies where pedestrians suddenly enter the area ahead. When a blind user activates the semantic perception function, the segmentation map is divided into nine regions, numbered 1 to 9. Within each region or a combination of multiple regions, the center point of the bounding box of each instance is measured using a Realsense D455 depth camera. If an object is present, its category, region location number (single number or a group of numbers), and the distance measurement result of the bounding box center point are communicated to the blind user via voice to assist in obstacle avoidance. Attached Figure Description

[0037] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0038] Figure 1 This is a flowchart of the dynamic obstacle avoidance method in this invention.

[0039] Figure 2 This is a schematic diagram of the instance segmentation network in this invention.

[0040] Figure 3 This is a schematic diagram comparing the ordinary residual structure with the residual structure within the feature extraction network of this invention.

[0041] Figure 4 This is a visualization of the safe area detection and obstacle detection effects of the discrete sampling strategy in this invention.

[0042] Figure 5 This is a diagram showing the effect of the heuristic search path planning method considering safety distance constraints in this invention. From left to right, it shows the path effect after successively increasing the distance factor weight.

[0043] Figure 6 This is a visualization of the path planning effect of the present invention in an experiment to assist the blind in obstacle avoidance.

[0044] Figure 7 This is a flowchart of the obstacle avoidance strategy generation process of the present invention. Detailed Implementation

[0045] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0046] This invention provides a dynamic obstacle avoidance method for assistive visually impaired scenarios. The method consists of three stages: road surface detection based on instance segmentation, heuristic path planning considering safety distance constraints, and decision generation based on priority and sequence rules, guiding the blind person's actions through auditory and tactile information. The details are as follows:

[0047] Step 1: Use a Logitech C920 camera to capture indoor and outdoor images. Keep the camera height between 160cm and 180cm and tilt it downwards at a 30° angle to the horizontal to ensure that the road surface can be detected. Collect 3000 images to create a training dataset for assistive scenarios, including categories such as road surface, pedestrians, tables and chairs, and potted plants. Use the Labelme tool for segmentation and labeling, separating different targets of the same category to create an instance segmentation dataset, dividing the training set, validation set, and test set into a 7:2:1 ratio.

[0048] Step 2: The server used for training is equipped with an Intel(R) Xeon(R) CPU E5-2620v4 @ 2.10GHz × 16, a 4-core NVIDIA GeForce GTX 2080 graphics card, 128GB of memory, and the operating system is Ubuntu 16.04. The training is implemented using Python 3.8 and the PyTorch deep learning framework. The training hyperparameters are: Batchsize = 32, Epochs = 100, Iterations = 18000, Optimizer = Adam. For an image X∈R... W*H*3 The first and second dimensions represent the length and width of the feature map, and the third dimension represents the number of channels in the feature map. To transform an image X∈R... W*H*3 The input to the feature extraction module based on the deep residual structure first needs to convert the image information into one-dimensional sequence information that conforms to the input of the backbone network. Preprocessing includes image patch partitioning and merging, which solves the input problem. Feature extraction first passes through a 7×7 2D convolutional layer and a 3×3 pooling layer, then through multiple residual modules and cascaded pyramid pooling modules. Each residual module differs from the ResNet network's residual module only in that the 3×3 ordinary 2D convolutional layer is replaced with a novel 3×3 2D convolutional layer with an internally hierarchical residual structure; everything else remains the same. This novel convolution first evenly divides the input feature map into s feature map subsets, which are then divided by x... i This means that, where i∈{1,2,…,s}, each feature map subset has the same spatial size as the input feature map, but the number of channels becomes 1 / s, and each x except x1 has a different spatial size. i All of them undergo a 2D convolution operation with a 3×3 kernel, and this convolution operation is performed using K. i () indicates that y is used to represent y i K represents i The output feature map of (), the final output of this convolutional structure is The addition here means stitching the resulting feature maps together in the third channel, y i The calculation formula is as follows:

[0049]

[0050] The pyramid pooling module consists of three pooling channels with pooling sizes of 12×12, 6×6, and 3×3. Each pooling layer is followed by a 2D convolutional layer with a 1×1 kernel to transform the third dimension of the feature map to 256. Ultimately, this step can extract image feature maps of three different scales, i.e., X1∈R. 69*69*256 X2∈R 35*35*256 X3∈R 18*18*256 ; Set X3∈R18 *18*256 X4∈R is obtained by a single 2D convolutional layer with a 3×3 kernel. 9*9*256 Then, through a 2D convolutional layer with a 3×3 kernel, X5∈R is obtained. 5*5*256 It should be noted that each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. During training, the parameters of the instance segmentation network are randomly initialized. A set of training data is input, and the network's forward propagation compares the results with the true label values ​​to obtain the error. Backpropagation updates the parameters, and the Adam gradient descent method is used to reduce the loss function. Cross-entropy loss is employed, defined as follows:

[0051]

[0052] Where y i p is the true value i The predicted value is L. t The definition is as follows:

[0053]

[0054] Where N is the number of samples, x is the input data for training, c is the class confidence score, and L... c (x,c) is the category confidence loss function, L b (x,l,g) is the bounding box loss function, where L... m (x,m,g m Let L be the mask position loss function, and α and β be hyperparameters, defined here as 1.25 and 6.125 respectively. c (x,c) is defined as follows:

[0055] L c (x,c)=f(c,1)

[0056] Where c is the category confidence, and L b (x,l,g) is defined as follows:

[0057] L b (x,l,g)=f(l,g)

[0058] Where l is the predicted bounding box value, g is the actual bounding box value, and L m (x,m,g m The definition is as follows:

[0059] L m (x,m,g m )=f(m,g m )

[0060] Where m is the predicted mask value and gt is the actual mask value.

[0061] Step 3: After training, the image is input into the feature extraction module based on the deep residual structure. The resulting feature maps are processed by spatial pooling pyramids to obtain multiple sets of fixed-size feature maps, among which the shallow feature map X1∈R 69*69*256 The feature map X7∈R is obtained by concatenating it with the feature map from the spatial feature recovery module. 69*69*280 The original image X∈R is input into a prototype generation module that generates a set of prototypes of the same size as the input image. The spatial feature recovery module consists of three cascaded 3×3 gated convolutional layers, 3×3 2D convolutional layers, and a bilinear interpolation layer, with the original image X∈R... W*H*3 As input, gated convolutional layers are used to extract edge features from the image. Transposed convolutional layers and bilinear interpolation layers are used to change the number of channels, ultimately generating a feature map X6∈R. 69*69*24 splice it to X1∈R 69*69*256 From this, we obtain X7∈R 69*69*280 The prototype generation module based on multi-layer transposed convolution upsampling consists of four transposed convolutional layers with 3×3 kernels and one 2D convolutional layer with 1×1 kernel, yielding P∈R. W*H*32 Deep feature map X2∈R 35*35*256 X3∈R 18 *18*256 X3∈R 18*18*256 X4∈R 9*9*256 X5∈R 5*5*256 A set of coefficient matrices is obtained through the coefficient matrix prediction module based on mask position regression. This module first passes the input through two cascaded 3×3 2D convolutional layers, then through three parallel 3×3 2D convolutional layers, resulting in three sets of coefficient matrices, which are the class confidence prediction matrices C1∈R. W*H*(81*a) Bounding box prediction matrix C2∈R W*H*(4*a) With mask prediction matrix C3∈R W*H*(32*a) Where 'a' represents the number of preset anchor frames. Within certain limits, a larger 'a' results in more accurate position prediction, but the number of parameters increases exponentially, slowing down the algorithm's execution speed. Experiments have shown that a value of 3–8 is optimal. Let C1∈R W*H*(81*a) C2∈R W*H*(4*a) C3∈R W*H*(32*a) Each with P∈R W*H*32 After performing matrix multiplication, the matrix is ​​then processed by the Sigmoid activation function, the expression of which is as follows:

[0062]

[0063] Where x is the input value and e is the base of the natural logarithm. The prototype is then trimmed and concatenated to obtain the final instance segmentation result. This process can be described as follows:

[0064] M = Sigmoid(PC) T )

[0065] Where P is the prototype mask matrix of H×W×k, H and W are the height and width of the input image of the segmentation network, k is the number of prototype groups, C is an n×k coefficient matrix corresponding to n instances filtered by nonmaximum suppression and mask scoring, and M is the final generated mask.

[0066] Step 4: Extract a binary image from the segmentation result. The third channel of this binary image is either 0 or 255, and its size is H×W×1. H and W are the same as the height and width of the image input to the segmentation network. 0 represents a walkable region, and 255 represents an impassable region (containing various obstacles). Based on this, downsampling is used to rasterize the semantic map, resulting in a binary image of size H×W×1. Where S represents the downsampling factor, and its value range is... Max(W,H) represents taking the maximum value of W and H.

[0067] Step 5: Use a discrete sampling strategy to determine if there are obstacles, and define a fan-shaped safe zone two meters away from the blind person. Within this zone, sample four pixels at a time. For areas farther from the blind person, divide them into a left front area and a right front area, sampling six pixels at a time. Collect all pixels that detect obstacles and accumulate them to more than H pixels. o H indicates the presence of an obstacle; all other cases indicate a passable road surface. o Value Where S is the downsampling factor in step 4.

[0068] Step 6: Calculate the relative speeds of the pedestrian and the blind person. d(Δ x ,Δ y ) represents the distance between the pedestrian and the blind user in adjacent frames, where Δ x ,Δ y D(Δ) represents the difference between the x and y coordinates of the center point of the pedestrian instance mask bounding box in the rasterized semantic map and the x and y coordinates of the bottom center point of the binary image. x ,Δ y The distance is the Euclidean distance calculated from the difference between the horizontal and vertical coordinates, where T is the time difference between two adjacent frames. seconds, F r Set the frame rate; calculate the minimum safe distance d = vt, where v is the relative speed between the pedestrian and the blind person, and t is the reaction time, set to two seconds. Draw a circular area with the center point of the pedestrian instance mask bounding box as the center and d as the radius. This area is set as a non-passable area.

[0069] Step 7: To find the endpoint of the path planning, use the algorithm for finding the center point of the farthest road surface. From the top of the binary image (the farthest point from the blind person) to the bottom of the binary image (the current feet of the blind person), traverse line by line and count the pixels of the road surface type. Each time an obstacle type pixel is encountered, the count of the road surface type pixels must be greater than H. r Once a passable road surface is found, the coordinates of the center point of that segment are recorded as the endpoint coordinates of the path planning, and no further traversal is needed, provided the cumulative coordinates are less than H. r Then clear the statistical data to zero and continue iterating, H r Value Where S is the downsampling factor in step 4.

[0070] Step 8: Add the cost of the path distance to the obstacle to the cost function of the A* algorithm. The total cost formula is:

[0071] f(n) = g(n) + h(n) + I(n)

[0072] Where n is the coordinate value of the current node, f(n) is the total cost, g(n) is the actual path length between the current point and the initial point obtained by the recursive algorithm, h(n) is the Manhattan distance between the current node and the initial point, and I(n) is the distance cost between the current node and the obstacle, defined as:

[0073] I(n) = (Ld(n)) * k i

[0074] Where L is the expected pixel distance maintained between the path and the obstacle, set to 20, d(n) is the x-coordinate distance or y-coordinate distance between the current node and its nearest obstacle, and k i This is the proportionality coefficient, and its value range is... Where W and H are the width and height of the binary image, respectively, and k i Used to amplify (Ld(n)) so that Otherwise, as (g(n) + h(n)) increases, I(n) has almost no effect on f(n). Using the above algorithm to process the rasterized semantic map, we finally obtain the shortest path from the current foot point to the farthest road center while maintaining a certain distance from obstacles.

[0075] Step 9: A forward decision can only be made when the safe zone is free of obstacles. If an obstacle suddenly appears in the zone, both vibrating pads will vibrate simultaneously to trigger an alarm. After confirming the safe zone is free of obstacles, calculate the deviation angle between the planned path and the straight-line direction. If the deviation exceeds H... b When the time indicates that the blind person has deviated significantly from the suggested path, the vibration intensity of the left and right hand vibrating pads is used to inform the blind person; the greater the vibration intensity, the greater the deviation angle, so as to advise the blind person to walk in the direction of the suggested path. H bA 25° angle is used; combined with distance detection of distant obstacles, in open areas, the system can provide voice prompts to blind users to move quickly. During the voice prompts, vibration alerts can be provided simultaneously to deal with emergencies where pedestrians suddenly enter the area ahead. When a blind user activates the semantic perception function, the segmentation map is divided into nine regions, numbered 1 to 9. Within each region or a combination of multiple regions, the center point of the bounding box of each instance is measured using a Realsense D455 depth camera. If an object is present, its category, region location number (single number or a group of numbers), and the distance measurement result of the bounding box center point are communicated to the blind user via voice to assist in obstacle avoidance.

[0076] The advantages of this method are as follows: First, the semantic segmentation network in the first stage can accurately segment small obstacle objects and road surface information spanning the entire globe. Tested on the constructed path dataset for assistive visually impaired scenarios, its mAP (maximum accuracy) for small obstacle objects is [not specified]. 0.5 >70, for background road surface AP 0.5 >95, AP 0.5 The intersection ratio (mAP) of the current predicted segmentation result and the actual segmentation result is greater than 50, and the correct category prediction indicates that the target was correctly predicted. This represents the proportion of such targets among the total number of targets. 0.5 It is AP for all categories 0.5 The results are summed and then averaged. Secondly, in the second and third stages of obstacle avoidance decision generation, a series of heuristic algorithms are used to effectively extract semantic, positional, and contour information of obstacles and road surfaces in the road image, enabling blind people to avoid obstacles more intelligently and autonomously.

[0077] To simplify the explanation of this method, the above figures and text are described as a series of steps. However, it should be understood and understood that these methods are not restricted by the order of operations, because some actions may occur in different orders when implemented according to one or more steps, but those skilled in the art can understand the principle behind the actions.

[0078] Although illustrative specific embodiments of the invention have been described step by step to enable those skilled in the art to understand them, the invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various modifications or variations within the scope of the claims, as long as such variations are within the spirit and scope of the invention as defined and determined by the appended claims.

Claims

1. A method for dynamic obstacle avoidance of a blind assistant scene, characterized in that, It is divided into three stages: The first stage carries out image instance segmentation, which is realized by a feature extraction module based on a deep residual structure, a spatial feature recovery module based on bilinear interpolation and transpose convolution upsampling, a prototype generation module based on multi-layer transpose convolution upsampling, and a coefficient matrix prediction module based on boundary box and mask position regression; The second stage extracts the category information, position information and contour information of the road surface and obstacles according to the instance segmentation result, and then carries out path planning, including a grid semantic map generation algorithm, a discrete sampling strategy for judging the obstacle category and position, a blind fan-shaped safety area and left front and right front area hypothesis, a far-end road surface center point heuristic search algorithm, an A* path planning algorithm considering safety distance constraints, and a path optimization strategy considering pedestrian movement; The third stage generates an assistive blind obstacle avoidance strategy, including priority rules and sequence rules of the obstacle avoidance reminding system, generates a decision based on the results of the second stage, and guides the blind person through hearing and touch; The method specifically comprises: Step 3: After the training is completed, the image is input into a feature extraction module based on a deep residual structure, and the obtained feature map is subjected to spatial pyramid pooling to obtain a plurality of groups of fixed-size feature maps, wherein a shallow layer feature map X1∈R 69*69*256 is spliced with a feature map from a spatial feature recovery module based on bilinear interpolation and transpose convolution upsampling to obtain a feature map X7∈R 69 *69*280 , which is input into a prototype generation module based on multi-layer transpose convolution upsampling to obtain a group of prototypes with the same size as the input image; wherein the spatial feature recovery module based on bilinear interpolation and transpose convolution upsampling is composed of three groups of convolution kernels of 3×3 gated convolution layers connected in series, a 2D convolution layer with a convolution kernel of 3×3, and a bilinear interpolation layer, with the original image X∈R W *H*3 As input, the edge features of the image are extracted using the gated convolution layer, the transpose convolution layer and the bilinear interpolation layer change the number of channels, and finally a feature map X6∈R 69*69*24 is obtained. 69*69*256 is spliced into X1∈R 69*69*280 to obtain X7∈R W*H*32 ; the prototype generation module based on multi-layer transpose convolution upsampling is composed of four transpose convolution layers with a convolution kernel of 3×3 and a 2D convolution layer with a convolution kernel of 1×1, and P∈R 35*35*256 is obtained; the deep layer feature maps X2∈R 18*18*256 , X3∈R 9*9*256 , X4∈R 5 , X5∈R *5*256 W*H*(81*a) A group of coefficient matrices obtained through a coefficient matrix prediction module based on bounding box and mask position regression; the coefficient matrix prediction module based on bounding box and mask position regression first passes the input through two groups of 2D convolution layers with a convolution kernel of 3×3 connected in series, and then passes the input through three 2D convolution layers with a convolution kernel of 3×3 connected in parallel to obtain three groups of coefficient matrices, which are respectively a category confidence prediction matrix C1∈R W*H*(4*a) , a bounding box prediction matrix C2∈R W*H*(32*a) and a mask prediction matrix C3∈R W*H*(81*a) , wherein a represents the number of preset anchor boxes, and a takes a value of 3-8; C1∈R W*H*(4*a) , C2∈R W*H*(32*a) , C3∈R W*H*32 are respectively subjected to matrix multiplication operation and then processed by a Sigmoid activation function, and the expression of the Sigmoid activation function is as follows: where x is the input value, e is the base of the natural logarithm; the prototype is cropped and spliced, and finally the instance segmentation result is obtained, which can be described as follows: M = Sigmoid(PC T ) Wherein P is an HxWxk prototype mask matrix, H and W are the height and width of the input image of the segmentation network, k is the number of prototypes, C is an n x k coefficient matrix corresponding to n instances after non-maximum suppression and mask score screening, and M is the finally generated mask; Step 4: The segmentation result is extracted into a binary image, the third channel of which is 0 or 255, and the size is HxWx1, where H, W and the height and width of the input image of the segmentation network are the same size, 0 represents a walkable area, and 255 represents an impassable area. On this basis, down-sampling is performed to rasterize the semantic map, and a binary image with a size of where S represents the down-sampling multiple, and the value range is Max(W, H) represents the maximum value of W and H. Step 5: Using the discrete sampling strategy to determine whether there is an obstacle, and to demarcate a two-meter distance from the blind person's fan-shaped safety zone, and in this area, four pixel points are sampled. The area far from the blind person is divided into left and right front areas, and six pixel points are sampled. When the obstacle pixel points are sampled and accumulated to more than H o represents the presence of an obstacle, and the rest represents the passable road surface, H o value where S is the down-sampling multiple in step 4; Step 6: Calculate the relative moving speed of the pedestrian and the blind person D(Δ x ,Δ y ) is the distance between the adjacent frame pedestrian and the blind user, wherein Δ x ,Δ y are the differences between the horizontal and vertical coordinates of the center point coordinates of the pedestrian instance mask bounding box and the bottom center point of the binary image in the grid semantic map, D(Δ x ,Δ y ) is the Euclidean distance calculated from the difference between the horizontal and vertical coordinates, T is the time difference between two adjacent frames, taken as 0.5 second, F r is the frame rate; calculate the minimum safety distance d = vt, wherein v is the relative moving speed of the pedestrian and the blind person, and t is the reaction time set as two seconds, draw a circular region with the center point coordinates of the pedestrian instance mask bounding box as the center and d as the radius, and the region is set as an impassable region; Step 7: To get the end point of path planning, use the remote road surface center point heuristic search algorithm, from the top of the binary image, which is the farthest from the blind person, to the bottom of the binary image, which is the current blind person's feet, traverse row by row and count the pixel points of the road surface type. Each time the obstacle type pixel point is traversed, the pixel point count of the road surface type is greater than H r It is considered that the passable road surface is found, the center point coordinates of the segment are recorded as the end point coordinates of path planning, and the traversal is stopped. The cumulative value is less than H r Then the statistical data is cleared, and the traversal is continued. H r The value is taken Where S is the down-sampling multiple in step 4; Step 8: Add the cost of the distance between the path and the obstacle to the cost function of the A* path planning algorithm considering safety distance constraints, and the total cost formula is: f(n)=g(n)+h(n)+I(n) Wherein n is the coordinate value of the current node, f(n) is the total cost, g(n) is the actual path length from the current point to the initial point obtained by the recursive algorithm, h(n) is the Manhattan distance between the current node and the initial point, and I(n) is the distance cost between the current node and the obstacle, defined as: I(n) = (L - d(n)) * k i where L is the expected pixel distance that the path keeps from the obstacles, set as 20, d(n) is the horizontal or vertical coordinate distance from the current node to the nearest obstacle, k i is the proportionality coefficient, whose value range is where W, H are the width and height of the binary image, respectively, k i to amplify (L - d(n)), so that Otherwise, with the increase of (g(n) + h(n)), I(n) can hardly affect f(n); using the above algorithm to process the rasterized semantic map, a shortest path from the current underfoot point to the farthest road center and keeping a certain distance from the obstacles is finally obtained.

2. The dynamic obstacle avoidance method for blind assistance scene according to claim 1, characterized in that: Before step 3, it further comprises: Step 1: Training data set preparation; Step 2: For an image X ∈ R W*H*3 , the first and second dimensions represent the length and width of the feature map, and the third dimension represents the channel number of the feature map. To input the image X ∈ R W*H*3 into the feature extraction module based on the deep residual structure, it is necessary to first convert the image information into one-dimensional sequence information conforming to the input of the backbone network. The preprocessing includes image block partitioning and image block merging operations, which can solve the input problem. The feature extraction first passes through a 2D convolution layer with a 7x7 convolution kernel and a pooling layer with a 3x3 pooling size, and then passes through multiple residual modules and cascades pyramid pooling modules. Each residual module is compared with the residual module of the ResNet network, and only one change is made to replace the 3x3 ordinary 2D convolution layer with a 3x3 new 2D convolution layer with internal hierarchical residual structure. The rest remains unchanged. The new 2D convolution layer first uniformly divides the input feature map into s feature map subsets, represented by x i , where i ∈ {1, 2, …, s}, each feature map subset has the same spatial size as the input feature map, but the channel number becomes 1 / s. Except x1, each i i goes through a 2D convolution operation with a 3x3 convolution kernel. The convolution operation process is represented by K i (), and the output feature map of K i () is represented by y i . The final output of the new 2D convolution layer structure is Here, the addition represents concatenating the resulting feature maps in the third channel, and the calculation formula of y i is as follows: The pyramid pooling module is composed of three pooling channels, and the pooling sizes are 12x12, 6x6, and 3x3, respectively. After each pooling layer, a 2D convolution layer with a convolution kernel of 1x1 is connected to change the third dimension of the feature map to 256. This step can extract three different scale image feature maps, i.e., X1eR 69*69*256 , X2eR 35 *35*256 , X3eR 18*18*256 ; X3eR 18*18*256 is obtained through a 2D convolution layer with a convolution kernel of 3x3, and X4eR 9*9*256 is obtained through a 2D convolution layer with a convolution kernel of 3x3 again, and X5eR 5*5*256 ; It should be pointed out that there is a batch normalization (BN) layer and a ReLU activation function after each convolution layer. During training, the parameters of the instance segmentation network are randomly initialized, a set of training data is input, the result is compared with the true label value through forward propagation of the network, the error is back propagated to update the parameters, the Adam gradient descent method is used to reduce the loss function, and the cross-entropy loss is adopted, which is defined as follows: where y i is the true value, p i is the predicted value, the overall loss L t is defined as follows: where N is the number of samples, x is the input data for training, c is the class confidence, L c (x,c) is the class confidence loss function, L b (x,l,g) is the bounding box loss function, L m (x,m,g m ) is the mask position loss function, a, b are hyperparameters, here 1.25, 6.125, respectively, L c (x,c) is defined as follows: L c (x,c) = f(c, 1) where c is the class confidence, L b (x, l, g) are defined as follows: L b (x, l, g) = f(l, g) where l is the bounding box prediction value, g is the bounding box ground truth value, L m (x, m, g m ) is defined as follows: L m (x,m,g m )=f(m,g m ) Wherein m is the mask prediction value, and gt is the mask actual value; After step 8, it further comprises: Step 9: The safety area is free of obstacles before the forward decision can be made, and when the area suddenly appears obstacles, the double shock pieces vibrate simultaneously to alarm. After determining that the safety area is free of obstacles, the deviation angle of the planned path and the straight direction is calculated. When the deviation angle exceeds H b , it indicates that the blind person deviates from the recommended path too much. The stronger the vibration intensity of the left and right hand shock pieces, the greater the deviation angle. The blind person is advised to walk along the recommended path direction. H b is taken as 25°; combined with the distance detection of distant obstacles, the blind person can be told to walk quickly in the open area. The vibration reminder can be performed simultaneously during the voice reminder process to deal with the emergency situation of pedestrians suddenly entering the front area. When the semantic perception function is turned on by the blind user, the segmentation graph is divided into nine areas numbered 1-9. The center point of the bounding box of each instance in each area or the combined area of multiple areas is measured by the depth camera Realsense D455. If there is an object, its category, area position number and bounding box center point measurement result are conveyed to the blind user through voice to assist in obstacle avoidance.

Images