A self-supervised method for real-time detection of indoor layout frames

Abstract

The application discloses a self-supervised real-time detection indoor layout framework method, and continuously optimizes the cleaning route of an indoor cleaning robot based on the method. Compared with the prior art, the application has the advantages that the application provides a cleaning direction for an indoor monocular vision intelligent mobile cleaning robot, improves the indoor cleaning efficiency of the cleaning robot, reduces the missed cleaning rate of corners, and continuously learns in practice to improve the indoor layout framework recognition rate without human intervention.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cleaning robots, in particular to a self-supervised method for real-time detection of indoor layout framework. BACKGROUND

[0002] When a monocular vision indoor intelligent mobile cleaning robot performs full-coverage cleaning in a room, if the initial cleaning direction is not perpendicular or parallel to the direction of the indoor layout (wall, wardrobe, screen) framework, the cleaning efficiency will be low and the corner area will be missed. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a self-supervised method for real-time detection of indoor layout framework.

[0004] To solve the above technical problems, the technical solution provided by the present application is as follows: a self-supervised method for real-time detection of indoor layout framework, comprising the following steps:

[0005] S1, constructing a real-time graphic line segment detection neural network recognition module;

[0006] S2, mounting the line segment recognition neural network recognition module on a monocular vision indoor intelligent mobile cleaning robot;

[0007] S3, the camera captures real-time pictures, and the real-time line segment neural network recognition module recognizes the line segment in the picture in the key frame selected by the monocular vision;

[0008] S4, according to the pre-prepared line segment features and line segment intersection features, the indoor layout framework is inferred, and the orientation of the indoor framework with the highest probability is taken as the conclusion;

[0009] S5, calculating the positional relationship between the indoor layout framework and the camera, and delivering the positional relationship to the robot cleaning module;

[0010] S6, after obtaining the positional relationship between the indoor layout framework and the camera, the robot cleaning module compares the current cleaning direction with the orientation of the indoor layout framework and judges the running track;

[0011] S7, the robot map module continuously generates a map according to the sensor data during the cleaning process;

[0012] S8, according to the map generated by the robot map module, the regularity degree of the current indoor layout framework is obtained;

[0013] S9, using the regularity degree to judge the rationality of the recognized indoor layout framework, and giving positive or negative feedback to the recognition;

[0014] S10, the indoor layout framework recognition system continuously learns according to the feedback to improve the recognition rate, and repeatedly steps 2-9 in the cleaning process to assist cleaning, while continuously learning according to the feedback.

[0015] As an improvement, S1 step, the line segment detection data set is used to train the line segment detection neural network module, so that it can achieve good line segment recognition effect on multiple real data sets.

[0016] As an improvement, S6 step, if the current cleaning direction is parallel or perpendicular to the framework direction, it is not necessary to adjust and continue the current direction; otherwise, adjust the robot cleaning direction to clean along or perpendicular to the indoor layout framework direction.

[0017] Compared with the prior art, the advantages of the present application are that it provides a cleaning direction for an indoor monocular vision intelligent mobile cleaning robot, improves the cleaning efficiency of the cleaning robot in the indoor cleaning, and reduces the missed cleaning rate of the corners. The method can also learn continuously in practice to improve the indoor layout framework recognition rate without human intervention. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of a self-supervised real-time detection indoor layout framework method. DETAILED DESCRIPTION

[0019] The present application will be further described in detail below in combination with the drawings.

[0020] In the specific implementation of the present application, a self-supervised real-time detection indoor layout framework method comprises the following steps:

[0021] S1, construct a real-time line segment detection neural network recognition module;

[0022] S2, load the line segment recognition neural network recognition module onto a monocular vision indoor intelligent mobile cleaning robot;

[0023] S3, the camera captures real-time pictures, and the real-time line segment neural network recognition module recognizes the line segments in the pictures in the key frames selected by the monocular vision;

[0024] S4, according to the pre-prepared line segment features and line segment intersection features, the indoor layout framework is inferred, and the indoor framework orientation with the highest probability is taken as the conclusion;

[0025] S5, calculate the positional relationship between the indoor layout framework and the camera, and deliver the positional relationship to the robot cleaning module;

[0026] S6, after obtaining the position relationship between the indoor layout framework and the camera, the robot cleaning module compares the current cleaning direction with the orientation of the indoor layout framework to determine the running track;

[0027] S7, the robot map module generates a map according to the sensor data during the cleaning process;

[0028] S8, according to the map generated by the robot map module, the regularity of the current indoor layout framework is obtained;

[0029] S9, the regularity is used to determine the rationality of the recognized indoor layout framework, and positive or negative feedback is given to the recognition;

[0030] S10, the indoor layout framework recognition system continuously learns to improve the recognition rate according to the feedback, and repeatedly steps 2-9 during the cleaning process to assist the cleaning, and continuously learns according to the feedback.

[0031] As an improvement, in step S1, a line segment detection data set is used to train the line segment detection neural network module, so that it can achieve good line segment recognition effect on multiple real data sets.

[0032] As an improvement, in step S6, if the current cleaning direction is parallel or perpendicular to the framework direction, it is not necessary to adjust and continue the current direction; otherwise, the robot cleaning direction is adjusted to clean along or perpendicular to the indoor layout framework direction.

[0033] The working principle of the application: in order to make the cleaning path of the cleaning robot more reasonable, maximize the cleaning efficiency and cleaning coverage, a self-supervised learning real-time detection indoor layout framework method is proposed, which can detect the indoor layout framework in real time, provide guidance for the cleaning route of the indoor cleaning robot, and learn through the layout map constructed after the cleaning of the cleaning robot as feedback, further improve the recognition rate of the indoor layout framework.

[0034] In order to improve the cleaning efficiency of monocular vision indoor intelligent mobile cleaning robot and reduce the missing cleaning rate, the application discloses a self-supervised learning method for real-time detection of indoor layout framework based on monocular vision and graphic line segment detection neural network. The working process of the method is as follows:

[0035] 1. Construct a real-time graphic line segment detection neural network recognition module, use a line segment detection data set to train the line segment detection neural network module, so that it can achieve good line segment recognition effect on multiple real data sets.

[0036] 2. The line segment recognition neural network recognition module is mounted on a monocular vision indoor intelligent mobile cleaning robot. During the movement of the robot, the camera captures real-time pictures at a speed of 15 frames per second, and the real-time line segment neural network recognition module recognizes the line segments in the pictures in the key frames selected by the monocular vision. The indoor layout framework is inferred according to the pre-prepared line segment features and line segment intersection features, and the orientation of the indoor framework with the highest probability is taken as the conclusion. The positional relationship between the indoor layout framework and the camera is calculated, and the positional relationship is transmitted to the robot cleaning module.

[0037] 3. After obtaining the positional relationship between the indoor layout framework and the camera in 2, the robot cleaning module compares the current cleaning direction with the orientation of the indoor layout framework. If the current cleaning direction is parallel or perpendicular to the framework direction, no adjustment is needed and the current direction is continued. Otherwise, the robot cleaning direction is adjusted to clean along or perpendicular to the indoor layout framework direction. The robot map module continuously generates a map according to the sensor data during cleaning, especially when the bump plate sensor collides with the indoor layout framework, the framework position and information are marked. The map generated by the map module can reflect the indoor layout framework in real time.

[0038] 4. According to the map generated by the robot map module in 3, the regularity of the current indoor layout framework is obtained. The regularity is used to judge the rationality of the indoor layout framework recognized in 2, and positive or negative feedback is given to the recognition in 2.

[0039] 5. The indoor layout framework recognition system in 2 continuously learns to improve the recognition rate according to the feedback in 4, and repeatedly steps 2-4 to assist cleaning during cleaning. At the same time, it continuously learns according to the feedback.

[0040] In addition, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more features, and in the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly specified and limited.

[0041] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection", "fixing" and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0042] In the present application, unless specifically defined and limited otherwise, the "on" or "under" of a first feature to a second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, the "on", "above" and "under" of a first feature to a second feature include that the first feature is directly above and obliquely above the second feature, or only means that the first feature is higher than the second feature in horizontal height. The "under", "below" and "under" of a first feature to a second feature include that the first feature is directly below and obliquely below the second feature, or only means that the first feature is lower than the second feature in horizontal height.

[0043] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0044] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments without departing from the principles and spirit of the present application within the scope of the present application.

Claims

1. A self-supervised real-time detection method for indoor layout frames, characterized in that, Includes the following steps: S1. Construct a real-time graphic line segment detection neural network recognition module; S2. The line segment detection neural network recognition module is mounted on a monocular vision indoor intelligent mobile cleaning robot; S3. The camera captures real-time images, and in the key frames selected by monocular vision, the line segment detection neural network recognition module identifies the line segments in the image. S4. Based on the characteristics of prefabricated line segments and the characteristics of line segment intersection, infer the indoor layout frame and take the indoor frame location with the highest probability as the conclusion. S5. Calculate the positional relationship between the indoor layout frame and the camera, and send the positional relationship to the robot cleaning module; S6. After obtaining the positional relationship between the indoor layout frame and the camera, the robot cleaning module will compare the current cleaning direction with the orientation of the indoor layout frame to determine the running trajectory. S7. The robot map module continuously generates maps based on data from various sensors during the cleaning process; S8. Based on the map generated by the robot map module, obtain the regularity of the current indoor layout framework; S9. Use the regularity level to judge the rationality of the identified indoor layout framework and give positive or negative feedback to the identification. The S10 indoor layout frame recognition system continuously learns from feedback to improve the recognition rate, and repeats steps 2-9 repeatedly during the cleaning process to assist in cleaning, while continuously learning from feedback.

2. The method for self-supervised real-time detection of indoor layout frames according to claim 1, characterized in that: In step S1, the line segment detection neural network recognition module is trained using a line segment detection dataset to achieve good line segment recognition results on multiple real datasets.

3. The method for self-supervised real-time detection of indoor layout frames according to claim 1, characterized in that: In step S6, if the current cleaning direction is already parallel or perpendicular to the frame direction, there is no need to adjust and the current direction can continue; otherwise, adjust the robot's cleaning direction to clean along or perpendicular to the indoor layout frame direction.

Images