A topology semantic map construction method for a blind assistance scene

Abstract

The application discloses a topological semantic map construction method for a blind-aiding scene. Firstly, the collected plan layout is preprocessed based on an image processing algorithm; secondly, a network structure based on clustering algorithm is adopted to preliminarily segment a mask of a passing area in the plan layout by fusing unsupervised image segmentation and OCR character recognition, and seed filling is used for screening; thirdly, intersection nodes in the passing area are fitted and generated, and room nodes and name information are detected and extracted; finally, connected relationships are generated by matching the nodes, a weighted distance matrix is established, an end-to-end shortest path in the plan layout is calculated by combining a Floyd algorithm, and a topological semantic map containing path guiding information is constructed. The application is applied to the blind-aiding scene, a fast analysis mechanism of a plan layout in a strange indoor environment is established, and the topological semantic map containing guiding information meeting the blind-aiding navigation requirement is constructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unsupervised learning and image processing, and particularly relates to a topological semantic map construction method for a blind assistance scene. BACKGROUND

[0002] China is the country with the largest number of blind people in the world. With the aggravation of aging, the number of blind people is further increasing, and people's demand for medical service level is increasing year by year. However, at present, China is short of professional nursing personnel, the auxiliary device is backward, and the blind assistance facilities are not standardized, which cannot meet the needs of blind people for normal life.

[0003] The main reason why the existing blind assistance device has not been widely applied is that the types of sensors are few, the dynamic environment understanding ability is weak, the positioning and navigation efficiency is low due to the complex indoor scene, the information dimension of hearing and touch is limited, and real-time interaction is difficult. In order to enable the robot to assist the blind people to safely, accurately and quickly reach the target position, a reliable topological semantic map construction technology for autonomous navigation is needed. The topological semantic map construction method disclosed by the present application can well solve the above problems.

[0004] An end-to-end topological semantic map construction method is proposed for indoor floor plan in an unfamiliar environment. The method performs semantic segmentation on the input floor plan by combining neural network with OCR text recognition, and generates a topological semantic map that can provide multi-dimensional output by combining image processing and path planning algorithm.

[0005] The commonly used deep learning segmentation algorithm has high segmentation accuracy, but a large number of labeled data sets are needed in the network training process, which is difficult to apply to indoor floor plans without public data sets, high annotation cost and variable styles. Although the pure unsupervised learning segmentation algorithm does not need many annotated data sets, it is difficult to directly segment the specified semantic region because its function is full image segmentation. How to realize semantic segmentation with a small amount of unlabeled data set has become a problem to be solved.

[0006] In addition, the image tilt and distortion caused by light and shooting angle in the process of processing floor plan photo collection, the detection and creation of passing nodes and connectivity, and the calculation of multi-dimensional navigation guidance information are all key problems in constructing topological semantic map to realize blind assistance navigation. SUMMARY

[0007] The following gives a brief summary of one or more aspects to address the basic understanding of these aspects. This summary does not detail all the conceived aspects, and the only purpose is to give some concepts of one or more aspects in a simplified form for the more detailed description later.

[0008] The application aims to solve the above problems and provides a topological semantic map construction method for a blind assistance scene.

[0009] The application provides a topological semantic map construction method for a blind assistance scene, which comprises the following steps:

[0010] Step 1: performing image processing algorithm-based preprocessing on the planar layout picture collected by the camera, and detecting, correcting and extracting the picture as the input of the segmentation network.

[0011] Step 2: combining a non-supervised image segmentation network based on a neural network clustering algorithm with an OCR character recognition network to perform semantic segmentation on the preprocessed image and extract the passable area in the layout picture.

[0012] Step 3: detecting the room nodes in the passable area, calculating the center nodes of the crossroads, generating a node connectivity matrix, and combining a Floyd path planning algorithm to calculate the shortest path from the starting node to the terminal node containing the directional guidance information.

[0013] According to the topological semantic map construction method for a blind assistance scene, in step 1, the following further steps are included:

[0014] Step 1: using an existing rectangle detection algorithm to detect and screen the maximum rectangular frame containing the map area boundary in the image, and obtaining the coordinates of the four corners of the rectangular frame.

[0015] Step 2: based on the four coordinates of the rectangular frame generated in step 1, using perspective transformation to convert the original perspective of the map area picture into an orthographic view plane and extract it.

[0016] According to the topological semantic map construction method for a blind assistance scene, in step 2, the following further steps are included:

[0017] Step 1: keeping the aspect ratio of the image extracted in step 1 unchanged, uniformly adjusting the long side to 4700 pixels, and using a CNN network to extract features, wherein the CNN network is composed of M convolution components, each component includes a two-dimensional 3*3 convolution with p channels, a ReLU activation function and a BatchNormalization function block, and M=2 and p=100.

[0018] Step 2: After feature extraction, a 1x1 convolution is used to convert to a q-dimensional clustering space, where q is the initial number of clusters, and q=100 is taken. Along the q axes of the space, the q-dimensional feature vector is normalized by a Batch Normalization function block;

[0019] Step 3: Use the argmax function to determine which dimension in the q-dimensional space the label of each pixel is, and calculate the feature similarity loss and spatial continuity loss according to the determined pseudo-label. The specific calculation formula is as follows:

[0020] Loss=L sim (r n ,c n )+μL con (r n )

[0021] Where μ represents the weight that balances the two losses, and μ=10.

[0022] L sim (r n ,c n ) is the feature similarity loss, and its expression is as follows:

[0023]

[0024]

[0025] Where i (i=1,…,q) is the clustering index, n (n=1,…,N) is the pixel, and N is the total number of pixels in the input image, r n is the normalized response map, c n is the clustering label of the normalized response r n after the argmax function, and r n,i is the i-th element in r n .

[0026] L con (r m ) is the spatial continuity loss, and its expression is as follows:

[0027]

[0028] Where W and H represent the width and length of the input image, and r ξ,η represents the value of the (ξ,η) pixel position in the response map r n .

[0029] Backpropagation is performed, and the parameters are updated using stochastic gradient descent. The output is the full image segmentation result, and different classes are labeled with different colors.

[0030] Step 4: The uniformly sized image in step 1 is simultaneously input into an OCR text recognition network to detect the coordinates of the corresponding key words of "passage" and "corridor";

[0031] Step 5: The coordinates are used to match the corresponding key words to the image segmentation result generated in step 3 after grayscale, and the grayscale value of the matched map area, i.e., the grayscale value of the color marked by the corresponding category, is stored;

[0032] Step 6: The segmentation result after grayscale in step 5 is input, and all image pixels are traversed. The pixels with a grayscale value consistent with the stored grayscale value in step 5 are filled with white, i.e., a grayscale value of 255, and the remaining pixels are filled with black, i.e., a grayscale value of 0. Finally, a mask of the passable area is obtained.

[0033] According to the topological semantic map construction method for the blind assistance scene, in step three, the following further steps are included:

[0034] Step 1: The mask of the passable area is first subjected to an erosion operation using a 5x5 rectangular kernel structure with an iteration number of 1, aiming to eliminate most noise points. Then, an inflation operation is performed using a 40x40 rectangular kernel structure with an iteration number of 1, aiming to eliminate the hollowed-out areas of the text part in the map;

[0035] Step 2: The first pixel point with a grayscale value of 255 in the top-left corner of the mask is taken as a seed point and independently pushed into a stack and marked. It is determined whether there is a pixel point with a grayscale value of 255 in the four neighborhoods of the current seed point. If there is, the point is stored in a list List_1 representing the first closed area, pushed into the stack as an updated seed point, and marked. If not, the top seed point in the stack is popped out and the determination continues. The cycle continues until all elements in the stack are popped out, and all pixel points of the closed area 1 are stored in List_1. The next pixel point with a grayscale value of 255 that has not been marked is taken as a new seed point, and the above operation continues until all pixel points with a grayscale value of 255 in the mask are marked. N lists of n closed areas in the mask are generated.

[0036] Step 3: The list with the largest number of elements in each list is calculated, and a new passable area mask is generated by traversing the list;

[0037] Step 4: The Harris algorithm is used to detect the corner points of the grayscale mask to obtain the corner region information and calculate the average coordinates of the pixel sets of each corner region to obtain the approximate center coordinates of each corner region;

[0038] Step 5: The center coordinates of the corner points are taken as the center of a circle, and the number of black and white pixels in a circular area with a radius of 20 pixels is counted. The corner points with a black and white pixel ratio greater than or equal to 1 are removed, and the inner corners with a ratio less than 1 are retained.

[0039] Step 6: Expand the black background with a 5×5 rectangular core. The purpose is to cover the remaining inner corners with the black background, so that all pixels between two corners that cannot be directly passed through are of the same color.

[0040] Step 7: Calculate the weighted distance between each pair of corner points, i.e., the weighted sum of the number of black and white pixels between two corner points:

[0041] L=μ1N b +μ2N w

[0042] Where L represents the weighted distance; μ1 represents the obstacle weight; μ2 represents the passage area weight; N b N represents the number of black pixels; w The number of white pixels is represented by μ1, which should be greater than μ2. In this invention, μ1 is set to 2 and μ2 to 1 respectively.

[0043] Step 8: Construct a weighted distance matrix between each corner point. Based on this matrix, match a maximum of 4 points for the corner point with the smallest weighted distance. For pairing two points, find the midpoint between the two points; for pairing three points, find the centroid of the three points; for pairing four points, find the intersection of the lines connecting the vertices. Finally, calculate the node symbolizing the intersection of the passage area.

[0044] Step 9: Using the grayscale segmentation result image from Step 5 in Step 2 as input, generate a grayscale histogram. Manually measure the grayscale value range of the image region symbolizing the doorway in the floor plan layout. Use the cv.Threshold function of the OpenCV computer vision library to process the grayscale segmentation result image. Set the threshold parameter to the lower bound of the grayscale value range, the maxval parameter to 255, and the type parameter to cv.THRESH_BINARY. Input the output of the function back into the cv.Threshold function, setting the threshold parameter to the upper bound of the grayscale value range, the maxval parameter to 255, and the type parameter to cv.THRESH_BINARY_INV. Finally, segment the doorway label region and calculate the mean pixel coordinates of each region to obtain the center node of the doorway.

[0045] Step 10: Flip the mask generated in Step 3 in black and white, and perform an AND operation with the preprocessed 1024×512 pixel front view plan layout. Input the generated image into the OCR text recognition network of the PaddlePaddle framework, output text recognition information, and match and store the text information with the nearest room node.

[0046] Step 11: Match the room nodes with the intersection nodes of the passage area with the closest weighted distance. This completes the basic topology map with intersection nodes as the main trunk and room nodes as branches.

[0047] Step 12: Construct a weighted graph based on the weighted distances between intersection nodes, and implement path planning between intersection nodes using the Floyd algorithm;

[0048] Step 13: When the current room name and the target room name are entered, the matching and stored room nodes are retrieved, and the node is further matched to the corresponding intersection node. The intersection node is used as the start and end node of the path planning part.

[0049] Step 14: Calculate the clockwise angle between the vectors of each pair of adjacent path nodes and the vertical coordinate direction, and use it as the direction guidance output in navigation. Attached Figure Description

[0050] The accompanying drawings, which constitute a part of this invention, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this invention.

[0051] Figure 1 This is the overall system flowchart of the present invention;

[0052] Figure 2 This is a schematic diagram of an example system of the present invention;

[0053] Figure 3 This invention relates to an image segmentation network framework based on unsupervised learning. Detailed Implementation

[0054] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0055] This invention provides a method for constructing topological semantic maps for assistive visually impaired scenarios. The flowchart of this method is as follows: Figure 1 As shown, its function is to convert the input image of the indoor layout of the environment captured by photography into a topological semantic map that can be used for autonomous navigation decision-making in indoor places such as hospitals and shopping malls, as detailed below:

[0056] An autofocus high-resolution monocular camera is used to capture floor plan images of indoor spaces (such as hospitals);

[0057] Preprocessing operations are performed on the image to detect, correct, and extract the front view image of the planar layout area;

[0058] The extracted images are resized to a uniform long side of 4700 pixels, while maintaining the same aspect ratio. A network structure framework is then created, such as... Figure 3 As shown, the first part describes the feature extraction network, which consists of M convolutional components. Each component includes a 3×3 convolution with p channels, a ReLU activation function, and a BatchNormalization block, where M = 2 and p = 100. After feature extraction, a 1×1 convolution transforms the feature vector into a q-dimensional clustering space. Along the q axes of this space, the BatchNormalization block normalizes the q-dimensional feature vector. The argmax function is then used for classification. Based on the pseudo-labels after classification, feature similarity loss and spatial continuity loss are calculated. The specific calculation formulas are as follows:

[0059] Loss = L sim (r n ,c n )+μL con (r n )

[0060] Where μ represents the weight that balances the two losses, and μ is usually taken as 10.

[0061] L sim (r n ,c n The feature similarity loss is expressed as follows:

[0062]

[0063]

[0064] Where i (i = 1, ..., q) is the cluster index, n (n = 1, ..., N) is the number of pixels, where N is the total number of pixels in the input image, and r n For the normalized response plot, c n The normalized response r after passing through the argmax function n Clustering labels, r n,i For r n The i-th element in.

[0065] L con (r n The spatial continuity loss is expressed as follows:

[0066]

[0067] Where W and H represent the width and length of the input image, r ξ,η Represents the response graph r n The value at the (ξ,η) pixel position.

[0068] Backpropagation is performed, and parameters are updated using stochastic gradient descent, ultimately allowing the calculation of the full image segmentation result. Each cluster segmentation region is drawn with a different color.

[0069] The front view image of the unified and adjusted plan layout area is input into the above segmentation network and OCR text recognition network respectively. The output image segmentation result is matched and shared with the OCR text recognition network. The gray value of the segmented area where the key texts "corridor" and "passage" are located is stored. Then, the pixels in the image with the same gray value as the stored gray value are extracted as the initial passage area mask.

[0070] The mask is eroded and expanded to reduce noise within the area. The n independent closed regions in the marked mask are calculated, and the largest closed region is retained as the updated passable region mask. The Harris corner detection algorithm is used to obtain corner regions, and the corner center coordinates are fitted. Outer corners are removed, and inner corners are retained.

[0071] Based on the weighted distance based on the number of black and white pixels, a weighted distance matrix is ​​constructed, and the nearest two corner points and four corner points are matched to calculate the intersection nodes respectively.

[0072] The room identifiers are segmented using the cv.Threshold threshold segmentation function of the OpenCV library. The mean pixel coordinates are calculated to obtain the room center node. The room name information is generated using a passage area mask and an OCR text recognition network. The room name information is then matched and stored with the nearest room center node. The room center node is matched and stored with the nearest intersection node. A topology map is constructed with intersection nodes as the main branches and room nodes as branches.

[0073] A weighted graph is constructed based on the weighted distance matrix. Path planning between intersection nodes is performed using the Floyd algorithm. The clockwise angle between the vectors of each pair of adjacent path nodes and the vertical axis is calculated to construct the shortest connectivity information matrix between points in the topology map.

[0074] To simplify the explanation of this method, the above figures and text are described as a series of steps. However, it should be understood and understood that these methods are not restricted by the order of operations, because some actions may occur in different orders when implemented according to one or more steps, but those skilled in the art can understand the principle behind the actions.

[0075] Although illustrative specific embodiments of the invention have been described step by step to enable those skilled in the art to understand them, the invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various modifications or variations within the scope of the claims, as long as such variations are within the spirit and scope of the invention as defined and determined by the appended claims.

Claims

1. A method for constructing topological semantic maps for assistive visually impaired scenarios, characterized in that, include: Step 1: Preprocessing operations including detection, correction, and extraction of the planar layout image captured by the camera; Step 2: Using a combination of an unsupervised image segmentation network based on neural network clustering algorithm and an OCR text recognition network, semantic segmentation is performed on the preprocessed image to extract the passable areas in the layout map; Step 3: Calculate and generate intersection nodes in the passable area, extract room nodes using threshold segmentation, and construct a connectivity matrix using weighted distance; after detecting room names using the OCR text recognition network of the PaddlePaddle platform, match room names, room nodes, and nearest intersection nodes using weighted distance as the criterion, and finally combine the Floyd algorithm to generate a planned path containing direction and angle guidance information between any two room nodes. Step 2 specifically includes the following steps: Step 2.1: Keep the aspect ratio of the image processed in Step 1 unchanged, adjust it uniformly to a long side of 4700 pixels, and input it into a CNN network. This network consists of two components, each of which contains a 2D 100-channel 3×3 convolution, a ReLU activation function, and a Batch Normalization function block to extract the output feature layer. Step 2.2: After feature extraction, the feature vector is transformed into a q-dimensional clustering space through a 1×1 convolution, where q is the initial number of clusters, and we take q = 100. Then, the q-dimensional feature vector of the clustering space is normalized through Batch Normalization. Step 2.3: Use the argmax function to classify each pixel, and calculate the feature similarity loss and spatial continuity loss based on this pseudo-label. The specific calculation formula is as follows: Loss=L sim (r n ,c n )+μL con (r n ) Where μ represents the weight that balances the two losses, and μ is 10; L sim (r n ,c n The feature similarity loss is expressed as follows: Where i is the clustering index, i = 1, ..., q, n is the number of pixels, n = 1, ..., N, where N is the total number of pixels in the input image, r n For the normalized response plot, c n The normalized response r after passing through the argmax function n Clustering labels, r n,i For r n The i-th element in; L con (r n The spatial continuity loss is expressed as follows: Where W and H represent the width and length of the input image, r ξ,η Represents the response graph r n The value at the (ξ,η) pixel position; Backpropagation is performed, parameters are updated using stochastic gradient descent, and the segmentation results of the entire image are output, with different categories labeled with different colors; Step 2.4: Input the uniformly sized images from Step 2.1 into the OCR text recognition network to detect the corresponding coordinates of the key texts "channel" and "corridor". Convert the segmentation result image output in Step 2.3 to grayscale and store the grayscale value of the key text coordinates in the grayscale image, which is the grayscale of the color marked by the corresponding category. Step 2.5: Using the grayscale segmentation result image from Step 2.4 as input, traverse all pixels in the image, assign a value of 255 to pixels whose grayscale value is equal to the grayscale value stored in Step 2.4 (i.e., fill with white), and assign a value of 0 to the remaining pixels (i.e., fill with black), and finally obtain the preliminary pass area mask.

2. The method for constructing a topological semantic map for assistive visually impaired scenarios according to claim 1, wherein: Step 1 specifically includes the following steps: Step 1.1: Use existing rectangle detection algorithms to detect the largest bounding box containing the map region boundary in the image, and obtain the coordinates of the four corners of the bounding box; Step 1.2: Using the perspective transformation based on the four corner coordinates in Step 1.1, the map area is corrected to the front view perspective plane and extracted.

3. The core of the topological semantic map construction method for assistive visually impaired scenarios according to claim 1 is: Step 3 specifically includes the following steps: Step 3.1: First, perform an erosion operation on the above mask using a 5×5 rectangular kernel, with one iteration, to eliminate most of the noise; then, perform an expansion operation using a 40×40 rectangular kernel, with one iteration, to eliminate the hollow areas of the text in the map. Step 3.2: Traverse the mask. Push the first pixel with a grayscale value of 255 onto the stack as a seed point and mark it. Check if there are any pixels with a grayscale value of 255 in the four neighborhoods of the current seed point. If there are, store the point in the list representing a closed region, push it onto the stack as the updated seed point, and mark it. If not, pop the top seed point from the stack and continue the check. Repeat until all elements pushed onto the stack have been popped. At this point, all pixels in the closed region have been stored in this list. Find the next unmarked pixel with a grayscale value of 255 as the new seed point and continue the above operation until all pixels with a grayscale value of 255 in the entire mask have been marked, generating n lists of n closed regions in the mask. Step 3.3: Calculate the number of elements in each list, keep the list with the largest number of elements, and traverse the elements of this list to generate a new passable field mask to replace it; Step 3.4: Based on the Harris algorithm, perform corner detection on the grayscale mask to obtain corner regions, and further fit the center coordinates; Step 3.5: Using the fitting center of the corner point as the center, count the number of black and white pixels in a circular area with a radius of 20 pixels, and only keep the interior corners where the ratio of black and white pixels is less than 1; Step 3.6: Expand the black background with a 5×5 rectangular kernel. The purpose is to cover the reserved inner corners with the black background, so that the area between two corners that cannot be directly passed is all black pixels, which will be used for subsequent weighted distance calculation. Step 3.7: Calculate the weighted distance L between the lines connecting each pair of corner points: L=μ1N b +μ2N w Where μ1 represents the obstacle weight; μ2 represents the passage area weight; N b N represents the number of black pixels; w This indicates the number of white pixels, where μ1 is 2 and μ2 is 1. Step 3.8: Construct a weighted distance matrix containing the weighted distance information between each corner point. For the corner point with the smallest weighted distance, perform matching with a maximum of 4 points. For matching two points, the midpoint between the two points is used; for matching three points, the centroid of the three points is used; for matching four points, the intersection of the lines connecting the opposite vertices is calculated and finally retained as the intersection node. Step 3.9: Using the grayscale segmentation result image from Step 2.4 as input, generate a grayscale histogram. Manually measure the grayscale value range of the image region symbolizing the doorway in the planar layout image. Use the cv.Threshold function of the OpenCV computer vision library to process the grayscale segmentation result image. Set the threshold parameter to the lower bound of the grayscale value range, the maxval parameter to 255, and the type parameter to cv.THRESH_BINARY. Input the output of the function back into the cv.Threshold function, setting the threshold parameter to the upper bound of the grayscale value range, the maxval parameter to 255, and the type parameter to cv.THRESH_BINARY_INV. Finally, segment the doorway label region and calculate the mean pixel coordinates of each region to obtain the center node of the doorway. Step 3.10: Flip the mask from Step 3.3 in black and white, and perform an AND operation with the preprocessed uniform-sized planar layout image. Input the generated image into the OCR text recognition network to generate text recognition information. Match the text information with the nearest room node and store it. Step 3.11: Match the room nodes with the intersection nodes of the passage area with the closest weighted distance. This completes the basic topology map with intersection nodes as the main trunk and room nodes as branches. Step 3.12: Construct a weighted graph containing weighted distance information between intersection nodes using intersection nodes and their weighted distances, and implement path planning between intersection nodes using the Floyd algorithm; Step 3.13: When the current room name and the target room name are entered, the matching and stored room nodes are retrieved, and the node is further matched to the corresponding intersection node. The intersection node is used as the start and end node of the path planning part. Step 3.14: Calculate the clockwise angle between the vectors of each pair of adjacent path nodes and the vertical coordinate direction, and use it as the direction guidance output in navigation.

Images