Stair-related hazardous area identification method and system based on RGB-d camera

Abstract

The present application relates to a stair-related hazardous area identification method and system based on an RGB-D camera. The method is applied to a mobile robot. The method comprises: acquiring RGB images and depth images by means of an RGB-D camera; annotating a stair-related hazardous area in each RGB image, and then inputting the RGB image into a pre-established instance segmentation model for mask processing to obtain an RGB image undergoing mask processing; according to intrinsic and extrinsic parameters of the RGB-D camera, aligning pixel points of the RGB image undergoing mask processing with pixel points of a corresponding depth image to generate an RGB image undergoing alignment; constructing a color point cloud model by means of the RGB image undergoing alignment, the depth image, and three-dimensional space information obtained by means of conversion on the basis of the intrinsic and extrinsic parameters; and traversing all three-dimensional coordinate points of the color point cloud model, and solving position information of the stair-related hazardous area on the basis of the acquired number and coordinates of the three-dimensional coordinate points satisfying set conditions. The present application improves the sensing capability of the mobile robot on the stair-related hazardous area.

Description

Staircase dangerous area recognition method and system based on RGB-D camera

[0001] This application claims priority to a Chinese patent application filed with the Patent Office of China on December 26, 2023, with application number 202311807347.2 and application name “Method and system for identifying dangerous areas of stairs based on RGB-D camera”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to a method and system for identifying dangerous areas on stairs based on an RGB-D camera. Background Art

[0003] With the rapid advancement of science and technology, the development of mobile robots has also kept pace with the times. They have become a key area of ​​innovation and industry, and robotics technology has become a key indicator of a country's comprehensive scientific and technological development strength. Robots have now expanded beyond production and penetrated into everyday life, with their application scenarios expanding. Their intelligence, networking, and digitalization are becoming increasingly prominent, and their level of intelligent cognition is constantly deepening. Mobile robots used in specific scenarios have brought great convenience to people and liberated labor. Unlike industrial robots, which have a relatively fixed workspace, mobile robots often face more complex, unstructured environments with larger operating areas.

[0004] At present, most mobile robots control their posture based on SLAM technology. Research on SLAM systems that use visual sensors to obtain external information often assumes that the external environment has static characteristics. When people pass by a mobile robot in a shopping mall, the image collected by the visual sensor is blurred because it contains moving objects, which is different from the static environment assumption. The moving objects will destroy the static environment assumption. As the errors between image frames accumulate, the posture estimation of the mobile robot will be inaccurate, resulting in large posture estimation errors, resulting in poor positioning and mapping effects. These information errors will cause the robot to make inaccurate motion decisions, which will cause the mobile robot to enter some dangerous areas.

[0005] Therefore, in order to enhance the mobile robot's ability to perceive and understand dangers in complex application scenarios, it is necessary to identify specific dangerous areas such as stairs and feed them back to the robot's path decision process to adjust the path planning in a timely manner.

[0006] Summary of the Invention

[0007] (1) Technical issues to be resolved

[0008] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present application provides a method and system for identifying stair-type dangerous areas based on an RGB-D camera, which solves the technical problem that mobile robots have poor perception ability of stair-type dangerous areas.

[0009] (2) Technical solution

[0010] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0011] In a first aspect, an embodiment of the present application provides a method for identifying dangerous areas such as stairs based on an RGB-D camera. The method is applied to a mobile robot and includes:

[0012] Use an RGB-D camera to obtain RGB images and depth images of the mobile robot's operating environment;

[0013] The staircase-like dangerous areas in the RGB image are marked and input into the pre-established instance segmentation model for mask processing to obtain the masked RGB image;

[0014] Based on the obtained internal and external parameters of the RGB-D camera, the masked RGB image is aligned with the pixels of the corresponding depth image to generate an aligned RGB image;

[0015] A color point cloud model is constructed using the aligned RGB image, depth image, and 3D spatial information obtained by converting the internal and external parameters of the RGB-D camera.

[0016] All three-dimensional coordinate points of the color point cloud model are traversed, and the location information of the staircase-type dangerous area is obtained based on the number and coordinates of the obtained three-dimensional coordinate points that meet the set conditions.

[0017] Optionally, after marking the staircase-like dangerous area in the RGB image, inputting the label into a pre-established instance segmentation model for mask processing to obtain the masked RGB image, the method further includes:

[0018] Use an RGB-D camera to obtain several RGB images containing dangerous areas such as stairs;

[0019] The staircase-like dangerous areas in each RGB image are annotated in the form of polygonal polylines, and the corresponding image dataset in a data exchange format containing the annotation information is generated;

[0020] After the image dataset is divided into a training dataset and a validation dataset according to the set ratio, the dataset formats of the training dataset and the validation dataset are converted to COCO format datasets;

[0021] The pre-established initial instance segmentation model based on the fully convolutional neural network is trained using the COCO format training dataset and verification dataset to generate an instance segmentation model.

[0022] Optionally, the staircase-like dangerous areas in the RGB image are marked and then input into a pre-established instance segmentation model for mask processing, and the masked RGB image obtained includes:

[0023] Mark the staircase-related dangerous areas in the obtained RGB image;

[0024] Input the annotated RGB image into the trained instance segmentation model;

[0025] Generate the category confidence, position regression parameters and mask coefficients of each pixel in the RGB image through the first branch of the instance segmentation model;

[0026] The second branch of the instance segmentation model generates a prototype mask consistent with the RGB image, and the masked RGB image of the staircase dangerous area is obtained based on the prototype mask and the mask coefficient.

[0027] Optionally, aligning the masked RGB image with pixels of a corresponding depth image based on the acquired intrinsic and extrinsic parameters of the RGB-D camera to generate the aligned RGB image includes:

[0028] Establish a depth camera coordinate system with the depth camera as the origin and an RGB camera coordinate system with the RGB camera as the origin;

[0029] Get the intrinsic parameter matrix, rotation matrix, and translation vector of the RGB-D camera;

[0030] The pixel coordinates of the filtered depth map are transformed by the depth camera intrinsic parameter matrix and the depth value to obtain the three-dimensional coordinate points in the depth camera coordinate system;

[0031] According to the rotation matrix and translation vector of the RGB-D camera, the three-dimensional coordinate point in the RGB camera coordinate system is matched with the three-dimensional coordinate point in the depth camera coordinate system;

[0032] Project the three-dimensional coordinate points in the RGB camera coordinate system through the intrinsic parameter matrix of the RGB camera to obtain the pixel coordinates of the masked RGB image;

[0033] Traverse all pixels in the depth map and align them with the pixels of the masked RGB map to generate an RGB map that corresponds one-to-one to the pixels of the depth map.

[0034] Optionally, constructing a color point cloud model using the aligned RGB image, the depth image, and three-dimensional spatial information obtained by converting internal and external parameters of an RGB-D camera includes:

[0035] Establish a world coordinate system with the mobile robot's motion center as the origin;

[0036] The three-dimensional coordinates in the world coordinate system are mapped to the coordinates in the depth camera coordinate system through the mapping formula to obtain a point cloud model;

[0037] Assign the RGB information and mask color information of the second RGB image to the point cloud model to obtain a color point cloud model;

[0038] in,

[0039] The origin of the world coordinate system coincides with the origin of the depth camera coordinate system;

[0040] The mapping formula is:

[0041] Where Z c is the z-axis value of the depth camera coordinate, (u, v) is an arbitrary coordinate point in the depth camera coordinate system, (u0, v0) is the center coordinate of the depth map; [RT] is the external parameter matrix of the RGB-D camera, R is a 3*3 rotation matrix, and T is a 3*1 translation vector; (x w ,y w , z w ) is the three-dimensional coordinate point in the world coordinate system.

[0042] Optionally, traversing all three-dimensional coordinate points of the color point cloud model and obtaining the location information of the staircase-related dangerous area according to the number and coordinates of the obtained three-dimensional coordinate points that meet the set conditions includes:

[0043] Traverse all three-dimensional coordinate points in the color point cloud model and obtain the number and coordinates of three-dimensional coordinate points that meet the set conditions;

[0044] Determine whether the number of three-dimensional coordinate points that meet the set conditions exceeds a set number threshold;

[0045] If the number of three-dimensional coordinate points that meet the set conditions does not exceed the set number threshold, it is determined that there is no staircase-type dangerous area in the aligned RGB image;

[0046] If the number of three-dimensional coordinate points that meet the set conditions exceeds the set threshold, it is determined that there is a staircase-type dangerous area in the aligned RGB image, and the location information of the staircase-type dangerous area is obtained based on the number and coordinates of the three-dimensional coordinate points.

[0047] In a second aspect, an embodiment of the present application provides a staircase dangerous area identification system based on an RGB-D camera, which includes: a mobile robot and an RGB-D camera and a controller configured on the mobile robot;

[0048] The RGB-D camera is used to collect RGB images and depth images of the mobile robot's operating environment;

[0049] The controller is connected to the RGB-D camera and is used to execute the above-mentioned staircase dangerous area identification method based on the RGB-D camera.

[0050] Optionally, the controller includes:

[0051] Image information acquisition module, used to obtain RGB images and depth images of the operating environment of the mobile robot;

[0052] The mask processing module is used to mark the staircase-like dangerous areas in the RGB image and input them into the pre-established instance segmentation model for mask processing to obtain the masked RGB image;

[0053] An image alignment module is used to align the pixels of the masked RGB image with the corresponding depth image based on the acquired intrinsic and extrinsic parameters of the RGB-D camera to generate an aligned RGB image.

[0054] A color point cloud model construction module is used to construct a color point cloud model using the aligned RGB image, depth image, and three-dimensional spatial information obtained by converting the internal and external parameters of the RGB-D camera;

[0055] The staircase dangerous area location determination module is used to traverse all three-dimensional coordinate points of the color point cloud model and obtain the location information of the staircase dangerous area based on the number and coordinates of the obtained three-dimensional coordinate points that meet the set conditions.

[0056] (3) Beneficial effects

[0057] This application provides a method for identifying stairway hazardous areas based on an RGB-D camera. This method uses an instance segmentation model to mask an RGB image containing stairway hazardous areas, aligns the masked RGB image with the corresponding depth map, and then uses the internal and external parameters of the RGB-D camera to build a color point cloud model. Finally, the color point cloud model is specifically filtered to obtain the location information of stairway hazardous areas. Compared with existing technologies, this method has the following advantages:

[0058] First, the instance segmentation model used for masking dangerous areas such as stairs requires only a small amount of data to train the required model, reducing the training workload of the neural network model.

[0059] Secondly, when constructing the color point cloud model, the mask color information of the masked RGB image is used for assignment, which improves the accuracy of identifying dangerous areas such as stairs.

[0060] Finally, after specific screening of the color point cloud model, the position information of the mobile robot and the stair-like dangerous areas is obtained, which greatly improves the detection and early warning capabilities of the mobile robot for stair-like dangerous areas within a certain range. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] FIG1 is a flow chart of a method for identifying dangerous areas on stairs based on an RGB-D camera according to an embodiment of the present application;

[0062] FIG2 is a schematic diagram of a process for identifying staircase-like dangerous areas in an RGB image using an example segmentation model provided by an embodiment of the present application;

[0063] FIG3 is a flowchart illustrating a visualization of dangerous areas on stairs according to an embodiment of the present application;

[0064] FIG4 is a mapping diagram of world coordinate points and depth map pixel points provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to better explain the present application and facilitate understanding, the present application is described in detail below with reference to the accompanying drawings through specific implementation methods.

[0066] 1-4 , an embodiment of the present application proposes a method and system for identifying staircase-type dangerous areas based on an RGB-D camera, which is applied to a mobile robot, comprising: first, obtaining an RGB image and a depth map of the operating environment of the mobile robot through an RGB-D camera; second, marking the staircase-type dangerous areas in the RGB image and inputting them into a pre-established instance segmentation model for mask processing to obtain a masked RGB image; then, based on the obtained internal and external parameters of the RGB-D camera, aligning the pixel points of the masked RGB image with the corresponding depth map to generate an aligned RGB image; then, constructing a color point cloud model through the aligned RGB image, the depth map, and the three-dimensional spatial information obtained based on the internal and external parameters of the RGB-D camera; finally, traversing all three-dimensional coordinate points of the color point cloud model, and obtaining the position information of the staircase-type dangerous areas based on the number and coordinates of the obtained three-dimensional coordinate points that meet the set conditions.

[0067] This application adopts a technical solution that uses an instance segmentation model to mask an RGB image with stair-like dangerous areas, aligns the masked RGB image with the corresponding depth image, and then establishes a color point cloud model in combination with the internal and external parameters of the RGB-D camera. Finally, the location information of the stair-like dangerous areas is obtained by performing specific screening on the color point cloud model. Compared with the existing technology, it has the following beneficial effects: First, the instance segmentation model used for masking stair-like dangerous areas only requires less data to train the required model, reducing the training workload of the neural network model. Secondly, when constructing the color point cloud model, the mask color information of the masked RGB image is used for assignment, which improves the accuracy of identifying stair-like dangerous areas. Finally, after specific screening of the color point cloud model, the location information of the mobile robot and the stair-like dangerous areas is obtained, which greatly improves the detection and warning capabilities of the mobile robot for stair-like dangerous areas within a certain range.

[0068] To better understand the above technical solutions, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a clearer and more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0069] Specifically, the present application provides a method for identifying dangerous areas such as stairs based on an RGB-D camera. The method is applied to a mobile robot and includes:

[0070] S1. Use an RGB-D camera to obtain an RGB image and depth map of the mobile robot's operating environment. An RGB-D camera has both RGB and depth camera functions. It can acquire datasets such as RGB images, depth maps, and infrared ray maps. Calibrate the RGB-D camera using the RGB and infrared ray maps to obtain the internal and external parameters of the RGB and depth cameras.

[0071] S2. Label the staircase-like dangerous areas in the RGB image and input them into a pre-established instance segmentation model for mask processing to obtain a masked RGB image.

[0072] Optionally, before step S1, the method further includes:

[0073] F1. Use an RGB-D camera to obtain several RGB images containing dangerous areas such as stairs.

[0074] F2. Label the staircase-like dangerous areas in each RGB image in the form of polygonal polylines, and generate a corresponding image dataset in a data exchange format containing the labeling information.

[0075] F3. After the image dataset is divided into a training dataset and a validation dataset according to the set ratio, the dataset formats of the training dataset and the validation dataset are converted to COCO format datasets.

[0076] F4. Train the pre-established initial instance segmentation model based on the fully convolutional neural network using the COCO format training dataset and verification dataset to generate an instance segmentation model.

[0077] In a specific embodiment, first, 10,350 RGB images of actual staircase scene data are collected using an RGB-D camera; secondly, each RGB image is annotated with polygonal polylines for staircase-related dangerous areas using the image annotation software Labelme, where the coordinates of discrete points of the annotated bounding box are stored in each point, and the target and name of the corresponding RGB image annotation are generated and saved in an image dataset in a data exchange format, where the downstairs area is represented by a green border area and the upstairs area is represented by a red border area; then, GitHub (a hosting platform for open source and private software projects) is used to randomly divide the image dataset in the data exchange format into 9,100 training set images and 1,250 validation set images, and the dataset format is converted into the COCO dataset format suitable for instance segmentation model training; finally, the weight file generated by the training is used to evaluate and detect the test images to obtain the result image of staircase-related dangerous area recognition, where the red area mask is the identified upstairs area, and the blue area mask is the identified downstairs area.

[0078] Optionally, step S2 includes:

[0079] S21. Mark the staircase-related dangerous areas in the obtained RGB image.

[0080] S22. Input the annotated RGB image into the trained instance segmentation model.

[0081] S23. Generate the category confidence, position regression parameters, and mask coefficients of each pixel in the RGB image through the first branch in the instance segmentation model.

[0082] S24. Generate a prototype mask consistent with the RGB image through the second branch in the instance segmentation model, and obtain a masked RGB image of the staircase dangerous area based on the prototype mask and the mask coefficient.

[0083] The instance segmentation model achieves real-time instance segmentation task decomposition through two parallel sub-networks: one branch uses a fully convolutional neural network to generate a set of prototype masks that are the same size as the RGB image, and then multiplies the prototype masks and the mask coefficients of the mask to obtain the mask of each target object in the RGB image; the other branch generates the category confidence, position regression parameters and mask coefficients of each pixel in the RGB image.

[0084] S3. Based on the obtained internal and external parameters of the RGB-D camera, align the pixels of the masked RGB image with the corresponding depth image to generate an aligned RGB image.

[0085] Optionally, step S3 includes:

[0086] S31. Establish a depth camera coordinate system with the depth camera as the origin and an RGB camera coordinate system with the RGB camera as the origin.

[0087] S32. Obtain the intrinsic parameter matrix, rotation matrix, and translation vector of the RGB-D camera.

[0088] S33: Transform the pixel coordinates of the filtered depth map through the depth camera intrinsic parameter matrix and the depth value to obtain a three-dimensional coordinate point in the depth camera coordinate system.

[0089] S34. According to the rotation matrix and translation vector of the RGB-D camera, obtain the three-dimensional coordinate point in the RGB camera coordinate system that matches the three-dimensional coordinate point in the depth camera coordinate system.

[0090] S35. Project the three-dimensional coordinate points in the RGB camera coordinate system through the intrinsic parameter matrix of the RGB camera to obtain pixel coordinates of the masked RGB image.

[0091] S36: traverse all pixels in the depth map and align them with the pixels of the masked RGB map to generate an RGB map that corresponds one-to-one to the pixels of the depth map.

[0092] To further explain, the process of aligning the depth map with the RGB map is:

[0093] The first step is to obtain the depth map and RGB image by using an RGB-D camera.

[0094] The second step is to use Matlab software to obtain the intrinsic parameter matrix H of the depth camera d , the intrinsic parameter matrix H of the RGB camera r And the external parameter matrix [RT] between them, where R is the rotation matrix and T is the translation vector.

[0095] The third step is to take the depth map pixel points to construct a three-dimensional vector p d =[x d y d z], where x d ,y d is the pixel coordinate of the pixel in the depth image coordinates, and z is the depth value of the pixel.

[0096] The fourth step is to use the intrinsic parameter matrix H of the depth camera d The inverse of the pixel is multiplied by the three-dimensional vector p of the pixel point d , get the spatial coordinate P in the depth camera coordinate system d .

[0097] The fifth step is to convert the spatial coordinate P in the depth camera coordinate system d Multiply by a rotation matrix R and add a translation vector T to get the spatial coordinate P of the RGB camera coordinate system r .P r =RP d +T (2)

[0098] Step 6: Use the intrinsic parameter matrix H of the RGB camera r Multiply the spatial coordinate P of the RGB camera coordinate system r Find the RGB pixel coordinate p in the RGB image coordinate system r =(x r ,y r ), take the pixel coordinate p under the RGB image coordinate r The pixel value c. p r =H r P r (3)

[0099] In the seventh step, the pixels in the depth image are traversed to perform the process from the third step to the sixth step to generate a second RGB image that corresponds one-to-one to the pixels in the depth image.

[0100] Among them, the rotation matrix R of the depth camera d , the rotation matrix R of the RGB camera r , the translation vector T of the depth camera r And the translation vector T of the RGB camera r Together they form the extrinsic parameter matrix [RT] of the RGB camera. The extrinsic parameter matrix [RT] can transform a point P in one spatial coordinate system to another spatial coordinate system. The process of obtaining the extrinsic parameter matrix [RT] of the depth camera and the RGB camera is as follows:

[0101] The first step is to transform the spatial coordinate system of the RGB camera as shown in formula (4). r =Rr P+T r (4)

[0102] The second step is to transform the spatial coordinate system of the depth camera into P as shown in formula (5). d =R d P+T d (5)

[0103] The third step is to combine formula (4) and formula (5) to obtain formula (6).

[0104] The fourth step is to convert P d Transformed into P r , compared with the previous formula (4), the external parameter matrix [RT] of the depth camera and RGB camera can be obtained to obtain formulas (7) and (8).

[0105] S4. Construct a color point cloud model through the aligned RGB image, depth map, and three-dimensional spatial information obtained by converting the internal and external parameters of the RGB-D camera.

[0106] Optionally, step S4 includes:

[0107] S41. Establish a world coordinate system with the motion center of the mobile robot as the origin.

[0108] S42, map the three-dimensional coordinates in the world coordinate system to the coordinates in the depth camera coordinate system through the mapping formula (9) to obtain a point cloud model. w ,y w , z w ) to the coordinate point m(u, v) in the depth image coordinate system is shown in FIG4 .

[0109] Where Z c is the z-axis value of the depth camera coordinate, (u, v) is an arbitrary coordinate point in the depth camera coordinate system, (u0, v0) is the center coordinate of the depth map; [RT] is the external parameter matrix of the RGB-D camera, R is a 3*3 rotation matrix, and T is a 3*1 translation vector; (x w ,y w , z w ) is the three-dimensional coordinate point in the world coordinate system.

[0110] Among them, the origin of the world coordinate system coincides with the origin of the depth camera coordinate system, and the setting of [RT] can be shown in formula (10).

[0111] Formula (9) is further simplified according to formula (10) to obtain formula (11).

[0112] Through the above matrix transformation formula, we can get the three-dimensional coordinate point M(x,y) in the world coordinate system from any coordinate point m(u,v) in the depth camera coordinate system. w ,y w , z w ) is shown in formula (12):

[0113] S43. Assign the RGB information and mask color information of the second RGB image to the point cloud model to obtain a color point cloud model.

[0114] Point cloud reconstruction using depth maps alone lacks color texture information, resulting in a visual difference from the actual scene captured by an RGB-D camera. The masked and aligned RGB image data contains blue and red masks for dangerous areas such as stairs. By mapping the RGB information of a pixel in the masked and aligned RGB image to the corresponding depth map pixel, a point cloud model with color texture can be reconstructed. The 3D reconstruction results after color texture mapping reflect the actual scene, with the blue point cloud representing the area below the stairs and the red area representing the area above the stairs.

[0115] S5. Filter each three-dimensional coordinate point of the color point cloud model according to the set filtering conditions, and obtain the location information of the staircase-type dangerous area by calculating the average value of the filtering results.

[0116] Optionally, step S5 includes:

[0117] S51. Traverse all three-dimensional coordinate points in the color point cloud model to obtain the number and coordinates of three-dimensional coordinate points that meet the set conditions.

[0118] S52: Determine whether the number of three-dimensional coordinate points that meet the set conditions exceeds a set number threshold.

[0119] S53a: If the number of three-dimensional coordinate points that meet the set conditions does not exceed the set number threshold, it is determined that there is no stair-type dangerous area in the aligned RGB image.

[0120] S53b. If the number of three-dimensional coordinate points that meet the set conditions exceeds the set number threshold, it is determined that there is a staircase-type dangerous area in the aligned RGB image, and the location information of the staircase-type dangerous area is obtained through the number and coordinates of the three-dimensional coordinate points.

[0121] In a specific embodiment, all three-dimensional coordinate points in the color point cloud model are traversed, and the number and coordinates of the three-dimensional coordinate points whose RGB meets the set conditions (33≤R≤85, 127≤G≤150, 167≤B≤255) are recorded. By taking the average value, the position information of the stair-like dangerous area in the direction of movement of the mobile robot can be obtained. When the number of three-dimensional coordinate points exceeds the set lower limit of 100, it is determined that there is a stair-like dangerous area in front of the mobile robot, thereby assisting the mobile robot in making braking decisions.

[0122] On the other hand, the present application also provides a staircase dangerous area identification system based on an RGB-D camera, which includes: a mobile robot and an RGB-D camera and a controller configured on the mobile robot.

[0123] The RGB-D camera is used to collect RGB images and depth images of the mobile robot's operating environment;

[0124] The controller is connected to the RGB-D camera and is used to execute the above-mentioned staircase dangerous area identification method based on the RGB-D camera.

[0125] To further illustrate, the controller includes:

[0126] The image information acquisition module is used to obtain the RGB image and depth image of the operating environment of the mobile robot.

[0127] The mask processing module is used to mark the staircase-like dangerous areas in the RGB image and input them into a pre-established instance segmentation model for mask processing to obtain a masked RGB image.

[0128] The image alignment module is used to align the pixels of the masked RGB image with the corresponding depth image based on the internal and external parameters of the RGB-D camera to generate an aligned RGB image.

[0129] The color point cloud model construction module is used to construct a color point cloud model through the aligned RGB image, depth map, and three-dimensional spatial information obtained by converting the internal and external parameters of the RGB-D camera.

[0130] The staircase dangerous area location determination module is used to traverse all three-dimensional coordinate points of the color point cloud model and obtain the location information of the staircase dangerous area based on the number and coordinates of the obtained three-dimensional coordinate points that meet the set conditions.

[0131] In summary, the present application provides a method and system for identifying staircase-type dangerous areas based on an RGB-D camera. First, the present application uses an RGB-D camera to collect RGB images and depth images of a mobile robot in its operating environment, and calibrates the RGB-D camera using MATLAB software. Secondly, the staircase-type dangerous areas in the RGB image are marked and masked using a trained instance segmentation model to obtain a masked RGB image. Next, the masked RGB image and the depth image are aligned using the internal and external parameters of the RGB-D camera to obtain an aligned RGB image. Then, a color point cloud model is constructed using the aligned RGB image, the depth image, and the three-dimensional spatial information obtained by converting the internal and external parameters of the RGB-D camera. Finally, all three-dimensional coordinate points of the color point cloud model are traversed, and the location information of the staircase-type dangerous areas is obtained based on the number and coordinates of the obtained three-dimensional coordinate points that meet the set conditions. The present application provides a technical solution for a mobile robot to detect staircase-type dangerous areas during movement, thereby enhancing the mobile robot's perception of external staircase-type dangerous areas.

[0132] The systems / devices described in the above embodiments are systems / devices used to implement the methods of the above embodiments of this application. Therefore, based on the methods described in the above embodiments of this application, those skilled in the art will be able to understand the specific structure and variations of the systems / devices, and therefore will not be described in detail here. All systems / devices used in the methods of the above embodiments of this application fall within the scope of protection to be provided by this application.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0135] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present application may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third etc. is for convenience only and does not indicate any order. These words may be understood as part of the component name.

[0136] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0137] Although the embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments after learning the basic inventive concept. Therefore, the claims should be interpreted as including the embodiments of the present application and all changes and modifications that fall within the scope of the present application.

[0138] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application shall also include these modifications and variations.

Claims

1. A method for identifying dangerous areas of stairs based on an RGB-D camera, characterized in that, This method is applied to a mobile robot, and the method includes: Obtaining an RGB image and a depth image of the operating environment where the mobile robot is located through an RGB-D camera; Labeling the dangerous stair areas in the RGB image and then inputting it into a pre-established instance segmentation model for masking processing to obtain the masked RGB image; Aligning the pixel points of the masked RGB image with the corresponding depth image according to the internal and external parameters of the obtained RGB-D camera to generate an aligned RGB image; Constructing a color point cloud model through the aligned RGB image, depth image, and three-dimensional space information converted based on the internal and external parameters of the RGB-D camera; Traversing all three-dimensional coordinate points of the color point cloud model, and obtaining the position information of the dangerous stair areas according to the number and coordinates of the three-dimensional coordinate points that meet the set conditions.

2. The method for identifying dangerous areas of stairs based on an RGB-D camera according to claim 1, wherein Before labeling the dangerous stair areas in the RGB image and then inputting it into a pre-established instance segmentation model for masking processing to obtain the masked RGB image, it further includes: Obtaining several RGB images containing dangerous stair areas through an RGB-D camera; Labeling the dangerous stair areas in each RGB image in the form of a polygon polyline to generate a corresponding image data set in the data exchange format containing annotation information; After the image data set is allocated into a training data set and a validation data set according to a set ratio, converting the data set formats of the training data set and the validation data set into a COCO format data set; Training a pre-established initial instance segmentation model based on a fully convolutional neural network through the COCO format training data set and validation data set to generate an instance segmentation model.

3. The method for identifying dangerous areas of stairs based on an RGB-D camera according to claim 2, wherein Labeling the dangerous stair areas in the RGB image and then inputting it into a pre-established instance segmentation model for masking processing to obtain the masked RGB image includes: Labeling the dangerous stair areas in the obtained RGB image; Inputting the labeled RGB image into the trained instance segmentation model; Generating the class confidence, position regression parameters, and mask coefficients of each pixel point in the RGB image through the first branch in the instance segmentation model; Generating a prototype mask consistent with the RGB image through the second branch in the instance segmentation model, and obtaining the masked RGB image of the dangerous stair areas based on the prototype mask and the mask coefficients.

4. The method for identifying dangerous areas of stairs based on an RGB-D camera according to claim 1, wherein Aligning the pixel points of the masked RGB image with the corresponding depth image according to the internal and external parameters of the obtained RGB-D camera to generate an aligned RGB image includes: Establishing a depth camera coordinate system with the depth camera as the origin and an RGB camera coordinate system with the RGB camera as the origin; Obtaining the internal parameter matrix, rotation matrix, and translation vector of the RGB-D camera; Converting the pixel coordinates of the filtered depth image through the depth camera internal parameter matrix and the depth value to obtain three-dimensional coordinate points in the depth camera coordinate system; According to the rotation matrix and translation vector of the RGB-D camera, obtaining the three-dimensional coordinate points in the RGB camera coordinate system that match the three-dimensional coordinate points in the depth camera coordinate system; Project the three-dimensional coordinate points in the RGB camera coordinate system through the internal parameter matrix of the RGB camera to obtain the pixel coordinates of the masked RGB image; Traverse all the pixel points in the depth image and align them with the pixel points of the masked RGB image to generate an RGB image corresponding one-to-one with the depth image pixel points.

5. The method for identifying dangerous areas of stairs based on an RGB-D camera according to claim 4, characterized in that, Construct a colored point cloud model through the aligned RGB image, depth image, and three-dimensional space information obtained by converting the internal and external parameters of the RGB-D camera, including: Establish a world coordinate system with the movement center of the mobile robot as the origin; Map the three-dimensional coordinates in the world coordinate system to the coordinates in the depth camera coordinate system through the mapping formula to obtain a point cloud model; Assign the RGB information and mask color information of the second RGB image to the point cloud model to obtain a colored point cloud model; Wherein, The origin of the world coordinate system coincides with the origin of the depth camera coordinate system; The mapping formula is as follows: Where, Z c is the z-axis value of the depth camera coordinates, (u, v) is any coordinate point in the depth camera coordinate system, and (u0, v0) is the center coordinate of the depth map; [R T] is the external parameter matrix of the RGB-D camera, R is a 3*3 rotation matrix, and T is a 3*1 translation vector; (x w , y w , z w ) is the three-dimensional coordinate point in the world coordinate system.

6. The method for identifying dangerous areas of stairs based on an RGB-D camera according to claim 5, characterized in that, Traverse all the three-dimensional coordinate points of the colored point cloud model, and obtain the position information of the staircase-like dangerous area according to the number and coordinates of the three-dimensional coordinate points that meet the set conditions, including: Traverse all the three-dimensional coordinate points in the colored point cloud model, and obtain the number and coordinates of the three-dimensional coordinate points that meet the set conditions; Judge whether the number of three-dimensional coordinate points that meet the set conditions exceeds the set number threshold; If the number of three-dimensional coordinate points that meet the set conditions does not exceed the set number threshold, it is determined that there is no staircase-like dangerous area in the aligned RGB image; If the number of three-dimensional coordinate points that meet the set conditions exceeds the set number threshold, it is determined that there is a staircase-like dangerous area in the aligned RGB image, and the position information of the staircase-like dangerous area is obtained by calculating the number and coordinates of the three-dimensional coordinate points.

7. A dangerous area recognition system for staircases based on an RGB-D camera, characterized in that, Including: A mobile robot and an RGB-D camera and a controller configured on the mobile robot; The RGB-D camera is used to collect the RGB image and depth image of the running environment of the mobile robot; The controller is connected to the RGB-D camera and is used to execute the staircase-like dangerous area recognition method based on the RGB-D camera as described in claims 1-6.

8. The staircase-like dangerous area recognition system based on an RGB-D camera according to claim 7, characterized in that, The controller includes: An image information acquisition module for acquiring the RGB image and depth image of the running environment where the mobile robot is located; A mask processing module for marking the staircase-like dangerous area in the RGB image and then inputting it into a pre-established instance segmentation model for mask processing to obtain a masked RGB image; An image alignment module for aligning the pixel points of the masked RGB image with the corresponding depth image according to the internal and external parameters of the obtained RGB-D camera to generate an aligned RGB image; A colored point cloud model construction module for constructing a colored point cloud model through the aligned RGB image, depth image, and three-dimensional space information obtained by converting the internal and external parameters of the RGB-D camera; A staircase-like dangerous area position determination module for traversing all the three-dimensional coordinate points of the colored point cloud model and obtaining the position information of the staircase-like dangerous area according to the number and coordinates of the three-dimensional coordinate points that meet the set conditions.

Images