A room semantic segmentation method

Abstract

The application discloses a room semantic segmentation method, which comprises the following steps: S1, acquiring a topographic map of a house, detecting door features on the topographic map, and generating a setting area based on the detection positions corresponding to the detected door features; S2, segmenting the setting area to segment out doors from the setting area, and then verifying the segmented doors to determine the shapes and positions of the doors; and S3, based on the shapes and positions of the doors, performing semantic classification on the topographic map, and then generating a topographic map after the semantic classification. The room semantic information of a grid image is recognized and segmented in a deep learning manner, the understanding of a scene semantic by a robot is improved, and different cleaning strategies or other work can be used for different rooms.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent robots, and relates to a room semantic segmentation method. BACKGROUND

[0002] SLAM is the cornerstone of mobile robots, and mobile robots need to build a map and locate from the map. In the floor cleaning robot industry, the various parts of the room of the map are segmented and semantically recognized, so that the robot can better perform different cleaning strategies for different rooms. The existing robots have poor ability to segment and semantically recognize the various parts of the room of the map, cannot effectively segment and semantically classify the room of the map, and have poor cleaning ability of the robot to perform different cleaning strategies for different rooms, which affects the user experience. SUMMARY

[0003] In order to solve the above technical defects, the technical scheme of the present application discloses a room semantic segmentation method, which identifies and segments the room semantic information of a grid image through deep learning, improves the understanding of the robot for the scene semantics, and uses different cleaning strategies or performs other work for different rooms. The specific technical scheme is as follows:

[0004] A room semantic segmentation method, the method comprising the following steps: S1: obtaining a topographic map of a house, detecting door features on the topographic map, and generating a set region based on the detection position corresponding to the detected door features; S2: segmenting the set region to segment the door from the set region, and then verifying the segmented door to determine the shape and position of the door; S3: based on the shape and position of the door, performing semantic classification on the topographic map, and then generating the topographic map after semantic classification.

[0005] Further, in step S1, the door features on the topographic map are detected, and the set region is generated based on the detection position corresponding to the detected door features, specifically comprising the following steps: dividing the topographic map into S*S cells through a convolutional neural network; then traversing the cells to determine the cells containing door features; dividing a rectangular set region centered on the cells containing door features; wherein S is a positive integer.

[0006] Further, after traversing the cells to determine the cells containing door features, the cell located at the center of the door features is obtained, then the intersection-over-union of the cell located at the center of the door features and its adjacent cells is calculated respectively to determine the confidence of the adjacent cells of the cell located at the center of the door features, and then the adjacent cells of the cell located at the center of the door features with confidence not meeting the requirements are deleted.

[0007] Further, in step S2, the setting area is segmented to segment the door from the setting area, specifically including the following steps: obtaining an image of the setting area with door features through the Unet network, then processing the image through a plurality of different scale pooling layers, and performing pixel-level classification on the features of the image after the pooling layer processing; then performing a plurality of upsampling on the image after the pooling layer processing, and each time the upsampling is performed, the features of the same scale corresponding to the channel number in the pooling layer processing process are fused, and then the door in the terrain map is obtained from the fused image.

[0008] Further, in step S2, before the setting area with door features is segmented, the setting area generated on the terrain map is clustered and calculated, specifically including the following steps: calculating the intersection-over-union of all obtained setting areas with door features two by two, and dividing the setting areas with door features whose calculation results are less than a set value into setting areas with the same door position.

[0009] Further, in step S2, the segmented door is verified to determine the shape and position of the door, including the following steps: judging the segmented setting area with door features, judging whether N doors exist in the same setting area; if N doors exist in the same setting area, determining whether the N doors existing in the same setting area are the same door through the clustering calculation result; if the N doors existing in the same setting area are the same door, regularizing the N doors existing in the same setting area; wherein N is a positive integer greater than or equal to 2.

[0010] Further, the N doors existing in the same setting area are regularized together, including the following steps:

[0011] Taking the straight line between the endpoints of any two doors of the N doors existing in the same setting area as the side of the circumscribed rectangle, then calculating the area of each circumscribed rectangle to obtain the smallest circumscribed rectangle.

[0012] Further, after the N doors existing in the same setting area are regularized together through the smallest circumscribed rectangle, then the gray value in the smallest circumscribed rectangle is obtained through the watershed algorithm, then the regions on both sides of the door in the smallest circumscribed rectangle are identified according to the gray value, the door that divides the smallest circumscribed rectangle into the smallest region is taken as the only door in the setting area with N doors, and the shape and position of the only door in the setting area are obtained.

[0013] Further, based on the shape and position of the door, the topographic map is semantically classified in step S3, and then the topographic map after semantic classification is generated, including the following steps: obtaining straight lines on the topographic map through polygon detection, then identifying rectangular regions according to the straight lines, and then distinguishing rooms and corridors from the rectangular regions based on the shape and position of the door.

[0014] Further, the semantic classification of the topographic map includes the following steps: labeling the distinguished rooms and corridors, and generating corresponding segmentation maps or topological maps, and then exporting the map.

[0015] Compared with the prior art, the technical scheme of the present application divides the set region from the topographic map by recognizing the door feature on the topographic map, extracts the position of the door from the set region, identifies the rooms and corridors of the topographic map, and then performs semantic classification, which can improve the understanding of the scene semantics by the robot, use different cleaning strategies or perform other work for different rooms, and improve the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the room semantic segmentation method described in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The specific embodiments of the present application will be further described below with reference to the accompanying drawings. It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0018] As shown in Figure 1 A room semantic segmentation method, which identifies and segments the room semantic information of a grid image through deep learning, improves the understanding of the scene semantics by the robot, uses different cleaning strategies or performs other work for different rooms, and the method includes the following steps:

[0019] Step S1 involves obtaining a topographic map of the house. This map can be obtained using existing technologies or maps stored on the robot or in the cloud. Alternatively, a raster map generated by the robot using existing technologies can be used instead of a topographic map. Then, door features are detected on the topographic map, and a designated region is generated based on the detection locations corresponding to these door features. This is primarily achieved through a convolutional neural network (CNN) to detect door features and generate the designated region. Specifically, the CNN divides the topographic map into S*S grids; then iterates through these grids, where S is a positive integer. The CNN used is YOLOv3, an object detection network. YOLOv1 borrows the structure of the GoogLeNet classification network. However, YOLOv3 does not use the inception module; instead, it uses a simple 1x1 convolutional layer (used for cross-channel information integration) + 3x3 convolutional layer. The YOLOv1 network uses a fully connected layer for category output at the end, therefore the output dimension of the fully connected layer is S×S×(B×5+C). YOLOv1 is faster than VGG16 (less than 1 / 3 the floating-point operations of VGG), but its accuracy is slightly lower. To improve object localization accuracy and recall, the YOLO authors proposed "YOLO9000: Better, Faster, Stronger" (Joseph Redmon, Ali Farhadi, CVPR 2017, Best Paper Honorable Mention). Compared to v1, it improved the resolution of training images; introduced the anchor box concept from Faster R-CNN; improved the network structure design; replaced YOLO's fully connected layers with convolutional layers in the output layer; and jointly used COCO object detection labeled data and ImageNet object classification labeled data to train the object detection model. Compared to YOLO, YOLO9000 has significant improvements in recognition categories, accuracy, speed, and localization accuracy. YOLOv3 achieves 20 FPS on Pascal Titan X for processing 608x608 images and 57.9% mAP@0.5 on COCO test-dev, similar to RetinaNet (the single-stage network proposed in the FocalLoss paper) and 4 times faster. The YOLO v3 model is significantly more complex than previous models, and speed and accuracy can be balanced by changing the size of the model structure. (Original article: https: / / www.cnblogs.com / makefile / p / YOLOv3.html © Kangxing Tianxia). Door features are structures or markers on topographic maps that contain door information, including door shape, size, etc. Recognizing door features is a capability of the YOLOv3 neural network algorithm, primarily obtained through pre-training.In the training of the door feature, first, a batch of labeled picture data sets are obtained, and the labels are marked according to the rules, and then the neural network is used to fit the training data set. The label in the training data set is obtained by manual marking, when the neural network is fitted well enough, it is equivalent to that the neural network has a certain recognition ability as human, after the structure of the neural network is determined, the training effect is determined by the loss function and the optimizer. After the grid with the door feature is determined, the setting area of the rectangle is divided around the grid containing the door feature, the size of the setting area can be set by setting the parameters, after the door feature is recognized, the setting area with the setting size is directly divided, or the minimum rectangle can be made based on one side or one corner of the door. After the setting area is generated on the topographic map by the convolutional neural network, the grid is traversed, the grid containing the door feature is determined, the grid located at the center of the door feature is obtained, then the intersection over union (IOU, the calculation formula is: the area of the intersection of two boxes divided by the area of the union) of the grid located at the center of the door feature and its adjacent grid is calculated respectively to determine the confidence of the adjacent grid of the grid located at the center of the door feature, and then the adjacent grid of the grid located at the center of the door feature with the confidence not meeting the requirement is deleted.

[0020] In step S2, after the setting area is generated, the setting area is segmented to segment the door from the setting area. When segmenting, the setting area with door features is segmented at the pixel level by a classic Unet network (Unet network is an image semantic segmentation network. The image semantic segmentation network allows the computer to segment according to the semantics of the image, and can output a specified segmentation picture. The entire network is shaped like the letter U. In simple terms, the entire network is divided into two parts. The left part is responsible for feature extraction. As the network layer deepens, the channel of the network gradually increases, and the "picture" gradually becomes smaller. The right network is responsible for feature restoration. The entire network is actually an encoder-decoder. In the encoding process, some information is lost (Maxpooling and Conv2D). When decoding, the corresponding encoding layer information is added. The specific steps include the following: obtaining the image of the setting area with door features, then processing the image through a plurality of different scale pooling layers, and classifying the features of the image processed by the pooling layer at the pixel level; then the image processed by the pooling layer is up-sampled several times, and each time the up-sampling is performed, the features of the same scale corresponding to the channel number in the pooling layer processing process are fused to obtain a gray image with the outline of the object in the image, and then the specific shape of the door is obtained from the outline of the door. Before segmenting the setting area with door features, clustering calculation is performed on the setting area generated on the topographic map, which includes the following steps: calculating the intersection-over-union of all obtained setting areas with door features two by two, and dividing the setting areas with door features whose calculation result is less than a set value into setting areas with the same door location. One set value of the present application is 0.7. In step S2, the segmented door is verified to determine the shape and position of the door, which includes the following steps: judging whether N doors exist in the same setting area after segmentation; if N doors exist in the same setting area, determining whether the N doors existing in the same setting area are the same door through clustering calculation results; if the N doors existing in the same setting area are the same door, regularizing the N doors existing in the same setting area; wherein N is a positive integer greater than or equal to 2. If the N doors existing in the same setting area are not the same door, the setting areas can be re-clustered and then judged, or other sensors can be used to reduce detection errors. The N doors existing in the same setting area are regularized together by the minimum bounding rectangle, which includes the following steps: drawing a bounding rectangle with a straight line between the endpoints of any two doors in the same setting area as a side, then calculating the area of each bounding rectangle to obtain the smallest bounding rectangle.The N doors existing in the same setting area are regularized by the minimum circumscribed rectangle, then the gray value in the minimum circumscribed rectangle is obtained by the watershed algorithm, then the area on both sides of the door in the minimum circumscribed rectangle is identified according to the gray value, the door that divides the minimum circumscribed rectangle into the minimum area is taken as the only door in the setting area with N doors, the real position of the door is obtained, and the shape of the door has been obtained in the above steps, so the shape and position of each door in the topographic map are obtained at this time.

[0021] In step S3, the topographic map is semantically classified based on the shape and position of the door, and then a topographic map after semantic classification is generated. First, straight lines on the topographic map are obtained by polygon detection, and then rectangular areas are identified according to the straight lines. When the straight lines are obtained, the straight lines can be extended, and then the rooms and corridors are distinguished from the rectangular areas based on the shape and position of the door. Polygon detection or segmentation can be performed by arcinfo, and the rectangular areas on the topographic map can also be obtained by closure detection. When the rooms and corridors are distinguished from the rectangular areas, an embodiment of the present application uses long-short side judgment. As long as the longest side is greater than 3 times the second longest side, that is, it is a rectangle, it is judged as a corridor, otherwise it is a room. The distinguished rooms and corridors are labeled, and those rectangular areas are labeled as rooms, corridors, the first room, etc., and then a corresponding segmentation map or topology map is generated, and then the map is exported.

[0022] Compared with the prior art, the technical solution of the present application divides the setting area from the topographic map by identifying the door features on the topographic map, extracts the position of the door from the setting area, identifies the rooms and corridors of the topographic map, and then performs semantic classification, which can improve the understanding of the scene semantics by the robot, use different cleaning strategies or perform other work for different rooms, and improve the user experience.

[0023] The above is only a preferred embodiment of the present application, and is not intended to limit the present application in other forms. Any skilled person in the art can modify or change the above disclosed technical content to obtain equivalent embodiments. However, any simple modification, equivalent change and modification of the above embodiments without departing from the technical solution of the present application, according to the technical essence of the present application, still belongs to the protection scope of the technical solution of the present application.

Claims

1. A method of room semantic segmentation, the method comprising: The method comprises the following steps: S1: obtaining a topographic map of a house, detecting door features on the topographic map, and generating a setting area based on the detection positions corresponding to the detected door features; S2: segmenting the setting area to segment a door from the setting area, and verifying the segmented door to determine the shape and position of the door; S3: performing semantic classification on the topographic map based on the shape and position of the door, and generating a topographic map after semantic classification; In step S2, verifying the segmented door to determine the shape and position of the door comprises the following steps: Judging whether N doors exist in the same setting area based on the segmented setting area with door features; If N doors exist in the same setting area, determining whether the N doors existing in the same setting area are the same door through clustering calculation results; If the N doors existing in the same setting area are the same door, regularizing the N doors existing in the same setting area. Wherein, N is a positive integer greater than or equal to 2.

2. The room semantic segmentation method of claim 1, wherein, In step S1, detecting door features on the topographic map and generating a setting area based on the detection positions corresponding to the detected door features comprises the following steps: Dividing the topographic map into S*S grids through a convolutional neural network; Then traversing the grids to determine the grids containing door features; Dividing a rectangular setting area centered on the grids containing door features; Wherein, S is a positive integer.

3. The room semantic segmentation method of claim 2, wherein, After traversing the grids to determine the grids containing door features, obtaining the grid located at the center of the door features, then calculating the intersection-over-union of the grid located at the center of the door features and its adjacent grids to determine the confidence of the adjacent grids of the grid located at the center of the door features, and then deleting the adjacent grids of the grid located at the center of the door features with confidence not meeting the requirements.

4. The room semantic segmentation method of claim 1, wherein, In step S2, segmenting the setting area to segment a door from the setting area comprises the following steps: Obtaining an image of the setting area with door features through a Unet network, then processing the image through a plurality of different scale pooling layers, and performing pixel-level classification on the features of the image processed by the pooling layers; Then performing a plurality of upsampling on the image processed by the pooling layers, and fusing the features of the same scale corresponding to the channel number in the pooling layer processing process each time the upsampling is performed, and then obtaining the door in the topographic map from the fused image.

5. The room semantic segmentation method of claim 4, wherein, In step S2, before segmenting the setting area with door features, clustering calculation is performed on the setting areas generated on the topographic map, comprising the following steps: Calculating the intersection-over-union of all obtained setting areas with door features two by two, and dividing the setting areas with door features with a calculation result less than a setting value into setting areas with the same door position.

6. The room semantic segmentation method of claim 5, wherein, Regularizing the N doors existing in the same setting area together, comprising the following steps: Drawing an external rectangle with the straight line between the endpoints of any two doors existing in the same setting area as a side, then calculating the area of each external rectangle to obtain the external rectangle with the smallest area.

7. The room semantic segmentation method of claim 6, wherein, The N doors existing in the same setting area are regularized together by the minimum circumscribed rectangle, then the gray value in the minimum circumscribed rectangle is obtained by the watershed algorithm, then the area on both sides of the door in the minimum circumscribed rectangle is identified according to the gray value, the door that divides the minimum circumscribed rectangle into the minimum area is taken as the only door in the setting area with N doors, and the shape and position of the only door in the setting area are obtained.

8. The room semantic segmentation method of claim 1, wherein, In step S3, the topographic map is semantically classified based on the shape and position of the door, and then a topographic map after semantic classification is generated, including the following steps: obtaining straight lines on the topographic map by polygon detection, then identifying rectangular areas according to the straight lines, and then distinguishing rooms and corridors from the rectangular areas based on the shape and position of the door.

9. The room semantic segmentation method of claim 8, wherein, The semantic classification of the topographic map includes the following steps: labeling the distinguished rooms and corridors, and generating corresponding segmentation maps or topology maps, and then exporting the maps.

Images