Robot passable area detection method, device, terminal and storage medium

Abstract

The application provides a robot passable area detection method and device, a terminal and a storage medium, and relates to the technical field of robots. The method comprises: acquiring depth image data and RGB image data; performing joint semantic segmentation on the depth image data and the RGB image data to generate a passable area mask image; performing coordinate conversion on the coordinates of each pixel in the passable area mask image based on depth information reflected by the depth image data to generate three-dimensional passable area information in a robot base coordinate system; and converting the three-dimensional passable area information to generate a two-dimensional passable area grid map. The application can accurately detect a passable area, and avoids the situation that a robot cannot accurately perform obstacle avoidance due to under-segmentation and over-segmentation in recognizing a passable area caused by robot sensor jitter in a point cloud geometry segmentation method.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, device, terminal, and storage medium for detecting robot-accessible areas. Background Technology

[0002] During movement, robots perceive traversable areas in their surroundings and plan their paths accordingly. Currently, traversable area detection typically relies on depth maps generated by depth cameras to construct point cloud data, followed by point cloud geometric segmentation methods for traversable area detection. However, this obstacle avoidance method, due to its dependence on planar geometric segmentation, suffers from low accuracy in detecting traversable areas when faced with vibrations caused by the robot's movement. Summary of the Invention

[0003] This application provides a method, apparatus, terminal, and storage medium for detecting passable areas for a robot, enabling accurate detection of passable areas even when the robot is shaking, thus facilitating efficient path planning. The technical solution is as follows: Firstly, a method for detecting traversable areas by a robot is provided, the method comprising: Acquire depth image data and RGB image data; Joint semantic segmentation is performed on depth image data and RGB image data to generate a passable region mask image. The passable region mask image is used to represent the location of passable and impassable regions in the image corresponding to the RGB image data. Based on the depth information reflected by the depth image data, the pixel coordinates of each pixel in the passable area mask image in the pixel coordinate system are converted into three-dimensional world coordinates in the world coordinate system. For each pixel's 3D world coordinates and pixel value, a 2D passable area raster map is generated. The passable area raster map includes rasters corresponding to passable areas and rasters corresponding to impassable areas.

[0004] Secondly, a robot-accessible area detection device is provided, the device comprising: The image acquisition module is used to acquire depth image data and RGB image data; The semantic segmentation module is used to perform joint semantic segmentation on depth image data and RGB image data to generate a passable region mask image. The passable region mask image is used to represent the location of passable and impassable regions in the image corresponding to the RGB image data. The coordinate transformation module is used to convert the pixel coordinates of each pixel in the passable area mask image in the pixel coordinate system into three-dimensional world coordinates in the world coordinate system based on the depth information reflected by the depth image data. The map generation module is used to generate a two-dimensional passable area raster map based on the three-dimensional world coordinates and pixel values ​​of each pixel. The passable area raster map includes rasters corresponding to passable areas and rasters corresponding to impassable areas.

[0005] Thirdly, a terminal is provided, the terminal including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described robot traversable area detection method.

[0006] Fourthly, a computer-readable storage medium is also provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described robot traversable area detection method.

[0007] The technical solution provided in this application brings at least the following beneficial effects: This application performs joint semantic segmentation on RGB image data and depth image data collected by a robot to obtain a passable area mask image. Based on the depth information reflected by the depth image data, the pixel coordinates of each pixel in the passable area mask image are converted into three-dimensional world coordinates in the world coordinate system. Based on the three-dimensional world coordinates and pixel values, a two-dimensional passable area grid map is generated. The passable area grid map includes grids corresponding to the passable areas and grids corresponding to the impassable areas. Since this application performs joint semantic segmentation on RGB image data and depth image data to generate a two-dimensional passable area grid map, the passable area can be comprehensively analyzed from color and geometric information. This also avoids the undersegmentation and oversegmentation of passable areas caused by robot sensor jitter in point cloud geometric segmentation methods, which prevent the robot from accurately avoiding obstacles. At the same time, since this application converts the three-dimensional passable area mask image into a two-dimensional passable area grid map, the efficiency of path planning by the robot based on the two-dimensional passable area grid map can be improved. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1This is a flowchart of a robot traversable area detection method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the semantic segmentation model provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of a robot traversable area detection device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The contents of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure.

[0011] Please see Figure 1 This application provides a method for detecting navigable areas for robots, particularly humanoid robots, and the method includes steps 101-104: Step 101: Obtain depth image data and RGB image data.

[0012] In this embodiment of the application, the robot is equipped with a depth camera. During the operation of the robot, the robot collects depth image data and RGB image data in real time through the equipped depth camera. The pixel value of the pixel in the depth image data is used to represent the distance information of the point in the space corresponding to the pixel to the depth camera, and the pixel value of the pixel in the RGB image data is used to represent the color information of the point in the space corresponding to the pixel.

[0013] The depth camera performs spatial and temporal alignment operations on the depth image data and RGB image data to ensure consistency between the two, enabling accurate determination of the robot's traversable area. Spatial alignment error is controlled within 2 pixels, and temporal alignment error is controlled within 3ms.

[0014] Step 102: Perform joint semantic segmentation on the depth image data and RGB image data to generate a passable region mask image. The passable region mask image is used to represent the location of passable and impassable regions in the image corresponding to the RGB image data.

[0015] The joint semantic segmentation model is a multi-task learning model that fuses RGB image data and depth map data to enhance the model's understanding of spatial geometric information during semantic segmentation.

[0016] In this embodiment, depth image data and RGB image data are input into a semantic segmentation model for joint semantic segmentation. This achieves accurate segmentation by combining the distance information represented by the depth image data and the color information represented by the RGB image data, generating a passable region mask image. Specifically, in this embodiment, the training set after enhancement is input into the initial semantic segmentation model to obtain the segmentation result output by the initial semantic segmentation model. A loss function is constructed based on the recognition result and a reference set. The initial semantic segmentation model is then trained using the loss function and the training set to obtain the semantic segmentation model.

[0017] In this embodiment, during the training of the semantic segmentation model, enhancement operations are performed on the training set, which includes depth image training data and RGB image training data. The enhancement operations include setting rotation parameters, adjusting contrast, and adjusting brightness for the depth image training data and RGB image training data. The rotation parameter values ​​are controlled within a target rotation range, preferably (-45°, 45°), meaning the rotation parameter values ​​are controlled within a 45° range. Since the left-right sway variation of a robot during normal walking is no greater than 45°, introducing rotation parameters can improve the segmentation capability of the trained semantic segmentation model. When adjusting contrast and brightness, the contrast and brightness values ​​can be randomly adjusted.

[0018] The enhancement operation also includes noise adjustment, which can add appropriate noise to the depth image training data and RGB image training data. Among them, the depth camera mainly uses Gaussian noise and salt-and-pepper noise to avoid excessive deviation in the depth data. The amount of noise data can be set as needed, and this application does not limit it.

[0019] The pixel values ​​in the passable area mask image represent the regions of interest and regions of non-interest obtained by partitioning the RGB image data. The regions of interest correspond to the passable areas in the RGB image, and the regions of non-interest correspond to the impassable areas in the RGB image.

[0020] In one possible implementation, joint semantic segmentation is performed on depth image data and RGB image data to generate a passable region mask image, including: inputting depth image data and RGB image data into a semantic segmentation model to generate an initial passable region mask image; and performing morphological processing on the initial passable region mask image to generate a passable region mask image.

[0021] In one possible implementation, before inputting depth image data and RGB image data into the semantic segmentation model to generate an initial access region mask image, the method further includes: The depth image data and RGB image data are preprocessed, and then the preprocessed depth image data and RGB image data are input into the semantic segmentation model.

[0022] Morphological processing is used to further extract information from an image.

[0023] In this embodiment, morphological processing of the initial access region mask image includes: first performing a dilation operation on the initial access region mask image output by the semantic segmentation model, and then performing an erosion operation. The dilation operation expands the boundaries of the content in the image, while the erosion operation eliminates a portion of the boundaries of the content in the image.

[0024] By first performing a dilation operation on the initial passable region mask image and then performing an erosion operation, local minor noise in the initial passable region mask image can be eliminated, thus avoiding region fragmentation caused by errors during semantic segmentation by the semantic segmentation model.

[0025] In one possible implementation, inputting the depth image data and the RGB image data into a semantic segmentation model to generate an initial access region mask image includes: a first sub-step and a second sub-step.

[0026] The first sub-step involves inputting depth image data and RGB image data into the encoder to generate global feature information.

[0027] In this embodiment of the application, the semantic segmentation model includes an encoder and a decoder.

[0028] In this embodiment of the application, the depth image data and RGB image data input to the semantic segmentation model are feature encoded by an encoder to obtain global feature information.

[0029] Figure 2 This is a schematic diagram of the semantic segmentation model in an embodiment of this application, as shown below. Figure 2 As shown, the encoder includes a first-channel attention module, a second-channel attention module, and a multi-level residual network.

[0030] The first channel attention module is used to process the depth image features at the current level and generate the first depth image features at the current level.

[0031] The second channel attention module is used for the RGB image features of the current level to generate the first RGB image features of the current level.

[0032] It should be noted that there can be multiple first-channel attention modules and second-channel attention modules. The first-channel attention modules corresponding to different levels of depth image features can be the same or different. That is, the first-channel attention module of the current level is used to process the depth image features of the current level and generate the first depth image features of the current level. Similarly, the second-channel attention modules corresponding to different levels of RGB image features can be the same or different. That is, the second-channel attention module of the current level is used to process the RGB image features of the current level and generate the first RGB image features of the current level.

[0033] A multi-level residual network comprises multi-level residual blocks. Each level of residual block includes a first residual block 1i, a second residual block 2i, and a third residual block 3i, where i is the level number of the residual block in the multi-level residual network. For example, ... Figure 2 As shown, the first-level residual blocks in the four-level residual network are first residual block 11, second residual block 21, and third residual block 31; the second-level residual blocks are first residual block 12, second residual block 22, and third residual block 32; the third-level residual blocks are first residual block 13, second residual block 23, and third residual block 33; and the fourth-level residual blocks are first residual block 14, second residual block 24, and third residual block 34.

[0034] The first residual block is used to process the depth image features of the current level and generate the depth image features of the next level. The second residual block is used to process the RGB image features of the current level and generate the RGB image features of the next level. The third residual block is used to process the fused image features of the current level and generate the RGB image features of the next level. The fused image features of the current level are obtained by fusing the fused image features of the previous level, the first depth image features of the current level, and the first RGB image features of the current level.

[0035] In one possible implementation, depth image data and RGB image data are input into an encoder to generate global feature information, including: convolving the depth image data and RGB image data to generate first-level depth image features and first-level RGB image features; starting from the first-level depth image features and first-level RGB image features, performing the following steps for each level of depth image features and RGB image features: Input the current level depth image features and the current level RGB image features into the current level residual block to generate the next level depth image features and the next level RGB image features; The current level depth image features are input into the first channel attention module to generate the current level first depth image features; the current level RGB image features are input into the second channel attention module to generate the current level first RGB image features; the current level fused image features, the current level first depth image features, and the current level first RGB image features are fused to generate the current level first fused image features; the current level first fused image features are input into the current level residual block to generate the next level fused image features; this process continues until global feature information is generated, which is obtained by fusing the last level first depth image features, the last level first RGB image features, and the last level residual block output fused image features.

[0036] In this embodiment of the application, the current level depth image features and the current level RGB image features are input into the current level residual block to generate the next level depth image features and the next level RGB image features. This includes: inputting the current level depth image features into the first residual block corresponding to the current level to generate the next level depth image features; and inputting the current level RGB image features into the second residual block corresponding to the current level to generate the next level RGB image features.

[0037] In this embodiment of the application, inputting the first fused image feature of the current level into the residual block of the current level to generate the fused image feature of the next level includes: inputting the first fused image feature of the current level into the third residual block of the current level to generate the fused image feature of the next level.

[0038] Specifically, for the first fused image feature at the first level, only the first depth image feature and the first RGB image feature at the first level need to be fused. That is, the first RGB image feature at the first level is obtained by fusing the first depth image feature and the first RGB image feature at the first level.

[0039] In this embodiment, an attention mechanism is introduced to learn the weights of RGB image features and depth image features during fusion. This allows for adaptive adjustment of the weights of each feature based on the degree of influence on the RGB and depth image data. When RGB image data is severely interfered with by reflections on the ground, but the distance information represented by the depth image data remains normal, adjusting the weights using the attention mechanism allows the semantic segmentation model to focus more on depth image features. Similarly, when depth image features are abnormal due to noise in the depth image data, but the RGB image data remains normal, adjusting the weights using the attention mechanism allows the semantic segmentation model to focus more on RGB image features. Compared to semantic segmentation of single-modal data, the semantic segmentation model in this embodiment performs joint semantic segmentation of depth and RGB image data. This significantly improves the detection of passable areas for low obstacles and demonstrates strong resistance to noise in the depth map under complex visible light conditions, resulting in higher robustness and better segmentation performance.

[0040] The second sub-step involves inputting the global feature information into the decoder for decoding, generating an initial passable area mask image.

[0041] In this embodiment, the global feature information is upsampled to the original image resolution using a decoder to generate an initial passable area mask image.

[0042] In this embodiment of the application, the second sub-step includes: continuously upsampling the global feature information to generate an initial access area mask image, wherein the resolution of the initial access area mask image is the same as the resolution of the RGB image data.

[0043] In one possible implementation, the global feature information is continuously upsampled, including: convolving the global feature information to generate a first feature; and upsampling the first feature five times consecutively.

[0044] In this embodiment, the semantic segmentation model includes an encoder and a decoder. The encoder is mainly used to encode the features of the depth image data to obtain depth image features, encode the features of the RGB image data to obtain RGB image features, and fuse the depth image features and RGB image features to obtain fused image features. It also uses the residual block structure of ResNet in the residual network to perform layer-by-layer downsampling to obtain local feature information and global feature information. The local feature information is output by the first N-1 levels of residual blocks, where N is the number of layers (levels) of the residual blocks in the residual network. The global feature information is generated based on the depth image features, RGB image features, and fused image features output by the Nth level residual block. Finally, the decoder continuously upsamples the global feature information to the resolution of the original image (RGB image data / depth image data) to generate an initial passable region mask image.

[0045] Step 103: Based on the depth information reflected by the depth image data, convert the pixel coordinates of each pixel in the passable area mask image in the pixel coordinate system into three-dimensional world coordinates in the world coordinate system.

[0046] Since the access area mask image generated in step 102 is based on depth image data and RGB image data acquired by a depth camera, and the depth image data and RGB image data use pixel coordinates, but robot navigation requires coordinates in the world coordinate system, that is, coordinates relative to the robot's base coordinate system, it is necessary to back-project the pixel coordinates corresponding to the access area mask image to the robot coordinate system.

[0047] For example, if the pixel coordinates of a certain pixel in the access area mask image are (u,v,1), the depth information corresponding to that pixel in the depth image data is d, and the transformation matrix of the depth camera pixel coordinate system relative to the robot base coordinate system is known to be T_camera_to_robot and the depth camera parameter matrix is ​​K, and the depth camera has completed distortion correction, then the coordinates P_robot(x,y,z) of the pixel in the access area mask image corresponding to the robot base coordinate system satisfy the following formula (1): d pixel = K T_camera_to_robot P_robot(1) Where T_camera_to_robot is the transformation matrix, P_robot is the 3D world coordinates in the world coordinate system, pixel is the pixel coordinates of the pixel, and d is the depth information of the pixel in the depth image data.

[0048] Based on formula (1), the three-dimensional world coordinates in the world coordinate system corresponding to each pixel in the passable area mask image can be obtained, thus completing the back projection of the passable area mask image. The depth camera parameter matrix K is determined based on the depth camera intrinsic and extrinsic parameter matrices.

[0049] Step 104: Generate a two-dimensional passable area raster map based on the three-dimensional world coordinates and pixel values ​​of each pixel. The passable area raster map includes rasters corresponding to passable areas and rasters corresponding to impassable areas.

[0050] Since robots typically do not pay attention to the height of obstacles when navigating, generating a two-dimensional grid map of the navigable area using the three-dimensional world coordinates and pixel values ​​of pixels can improve the efficiency of robots in detecting navigable areas in the three-dimensional world.

[0051] For example, assuming the starting point coordinates of the raster map are (x_start, y_start) and the raster map resolution is grid_size (which can be preset), then for a pixel in the passable area mask image with 3D world coordinates P_robot(x, y, z) in the world coordinate system, the corresponding coordinates of that pixel in the raster map are (grid_x, grid_y), where (grid_x, grid_y) are determined by the following formula: grid_x=(x-x_start) / grid_size, grid_y=(y-y_start) / grid_size.

[0052] After obtaining the coordinates of the pixels in the raster map, since the pixel values ​​of the pixels in the passable area mask image represent impassable and passable areas, a two-dimensional passable area raster map can be generated based on the pixel values ​​of the pixels and the coordinates of the pixels in the raster map. The passable area raster map includes rasters corresponding to passable areas and rasters corresponding to impassable areas.

[0053] In this embodiment of the application, after obtaining the grid map of the passable area, the robot performs path planning based on the grids corresponding to the passable areas and the grids corresponding to the impassable areas in the grid map, and completes the driving task by following the planned path.

[0054] Because the robot can move its limbs and torso, it can adopt different postures. If the robot can pass through obstacles in a specific posture, in another embodiment of this application, a two-dimensional passable area grid map is generated based on the three-dimensional world coordinates and pixel values ​​of each pixel, including: If the robot pose is obtained, and it is determined that the robot can pass through the target grid in the grid map under the target pose based on the robot pose, the target grid corresponds to the passable area, so that the robot can use the target grid as the grid corresponding to the passable area when planning the path.

[0055] This application performs joint semantic segmentation based on RGB image data and depth image data to generate a two-dimensional passable area grid map. Therefore, passable areas can be comprehensively analyzed from color and geometric information. It also avoids the situation where point cloud geometric segmentation methods are affected by robot sensor jitter, resulting in undersegmentation or oversegmentation of passable areas, and the robot cannot accurately avoid obstacles. In other words, compared with point cloud geometric segmentation methods, this application avoids the problem that point cloud geometric segmentation methods are greatly affected by jitter and are prone to causing the ground normal vector to shift. It can accurately detect passable areas even when the sensor is jittered.

[0056] See Figure 3 This application provides a robot traversable area detection device 300, which includes: Image acquisition module 301 is used to acquire depth image data and RGB image data; The semantic segmentation module 302 is used to perform joint semantic segmentation on depth image data and RGB image data to generate a passable region mask image. The passable region mask image is used to characterize the position of the passable region and the position of the impassable region in the image corresponding to the RGB image data. The coordinate transformation module 303 is used to convert the pixel coordinates of each pixel in the passable area mask image in the pixel coordinate system into three-dimensional world coordinates in the world coordinate system based on the depth information reflected by the depth image data. The map generation module 304 is used to generate a two-dimensional passable area raster map based on the three-dimensional world coordinates and pixel values ​​of each pixel. The passable area raster map includes rasters corresponding to passable areas and rasters corresponding to impassable areas.

[0057] In one possible implementation, the semantic segmentation module 302 includes: The first semantic segmentation submodule is used to input depth image data and RGB image data into the semantic segmentation model to generate an initial passable region mask image. The second semantic segmentation submodule is used to perform morphological processing on the initial access region mask image to generate an access region mask image.

[0058] In one possible implementation, the semantic segmentation model includes: an encoder and a decoder, and a first semantic segmentation submodule, which includes: The encoding module is used to input depth image data and RGB image data into the encoder to generate global feature information; The decoding module is used to input global feature information into the decoder for decoding and generate an initial passable area mask image.

[0059] In one possible implementation, the encoder includes a multi-level residual block, a first-channel attention module, and a second-channel attention module; The encoding module is specifically used to convolve the depth image data and RGB image data to generate first-level depth image features and first-level RGB image features. Starting from the first-level depth image features and first-level RGB image features, the following steps are performed for each level of depth image features and RGB image features: inputting the current level's depth image features and current level's RGB image features into the current level's residual block to generate the next level's depth image features and next level's RGB image features; inputting the current level's depth image features into the first channel attention module to generate the current level's first depth image features; inputting the current level's RGB image features into the second channel attention module to generate the current level's first RGB image features; fusing the current level's fused image features, the current level's first depth image features, and the current level's first RGB image features to generate the current level's first fused image features; inputting the current level's first fused image features into the current level's residual block to generate the next level's fused image features; until global feature information is generated, which is obtained by fusing the last level's first depth image features, the last level's first RGB image features, and the fused image features output from the last level's residual block.

[0060] In one possible implementation, the decoding module is specifically used to continuously upsample the global feature information to generate an initial pass area mask image, the resolution of which is the same as the resolution of the RGB image data.

[0061] This device performs joint semantic segmentation based on RGB image data and depth image data, and then generates a two-dimensional passable area grid map. Therefore, the passable area can be comprehensively analyzed from color information and geometric information. It also avoids the situation where point cloud geometric segmentation methods are under-segmented or over-segmented due to robot sensor jitter, which leads to the robot's inability to accurately avoid obstacles. In other words, even if the robot sensor jitters, the passable area can be accurately detected.

[0062] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0063] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program, which is loaded and executed by a processor of a computer device to enable the computer to implement any of the robot-accessible area detection methods described above.

[0064] In one possible implementation, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0065] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the robot-accessible area detection methods described above.

[0066] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0067] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0068] It should be noted that the terms "first," "second," etc. (if applicable) in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0069] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting robot-accessible areas, characterized in that, The method includes: Acquire depth image data and RGB image data; Joint semantic segmentation is performed on the depth image data and the RGB image data to generate a passable region mask image. The passable region mask image is used to characterize the positions of passable and impassable regions in the image corresponding to the RGB image data. Based on the depth information reflected by the depth image data, the pixel coordinates of each pixel in the passable area mask image in the pixel coordinate system are converted into three-dimensional world coordinates in the world coordinate system. A two-dimensional passable area raster map is generated based on the three-dimensional world coordinates of each pixel and the pixel value. The passable area raster map includes rasters corresponding to the passable areas and rasters corresponding to the impassable areas.

2. The method according to claim 1, characterized in that, The step of performing joint semantic segmentation on the depth image data and the RGB image data to generate a passage region mask image includes: The depth image data and the RGB image data are input into the semantic segmentation model to generate an initial passage area mask image; The initial access area mask image is subjected to morphological processing to generate the access area mask image.

3. The method according to claim 2, characterized in that, The semantic segmentation model includes an encoder and a decoder. The step of inputting the depth image data and the RGB image data into the semantic segmentation model to generate an initial passage region mask image includes: The depth image data and the RGB image data are input into the encoder to generate global feature information; The global feature information is input into the decoder for decoding to generate the initial passage area mask image.

4. The method according to claim 3, characterized in that, The encoder includes multi-level residual blocks, a first-channel attention module, and a second-channel attention module. The step of inputting the depth image features and the RGB image features into the encoder to generate global feature information includes: The depth image data and the RGB image data are convolved to generate first-level depth image features and first-level RGB image features; Starting with the first-level depth image features and the first-level RGB image features, the following steps are performed for each level of depth image features and RGB image features: The depth image features and RGB image features of the current level are input into the residual block of the current level to generate the depth image features and RGB image features of the next level; The depth image features of the current level are input into the first channel attention module to generate the first depth image features of the current level; the RGB image features of the current level are input into the second channel attention module to generate the first RGB image features of the current level; the fused image features of the current level, the first depth image features of the current level, and the first RGB image features of the current level are fused to generate the first fused image features of the current level; the first fused image features of the current level are input into the residual block of the current level to generate the fused image features of the next level; until global feature information is generated, which is obtained by fusing the first depth image features of the last level, the first RGB image features of the last level, and the fused image features output by the residual block of the last level.

5. The method according to claim 4, characterized in that, The step of inputting the global feature information into the decoder for decoding to generate the initial access area mask image includes: The global feature information is continuously upsampled to generate the initial access area mask image, and the resolution of the initial access area mask image is the same as the resolution of the RGB image data.

6. A robot-accessible area detection device, characterized in that, The device includes: The image acquisition module is used to acquire depth image data and RGB image data; The semantic segmentation module is used to perform joint semantic segmentation on the depth image data and the RGB image data to generate a passable area mask image. The passable area mask image is used to characterize the position of the passable area and the position of the impassable area in the image corresponding to the RGB image data. The coordinate transformation module is used to convert the pixel coordinates of each pixel in the passable area mask image in the pixel coordinate system into three-dimensional world coordinates in the world coordinate system based on the depth information reflected by the depth image data. The map generation module is used to generate a two-dimensional passable area raster map based on the three-dimensional world coordinates of each pixel and the value of the pixel. The passable area raster map includes rasters corresponding to the passable areas and rasters corresponding to the impassable areas.

7. The apparatus according to claim 6, characterized in that, The semantic segmentation module includes: The first semantic segmentation submodule is used to input the depth image data and the RGB image data into the semantic segmentation model to generate an initial passage area mask image; The second semantic segmentation submodule is used to perform morphological processing on the initial access region mask image to generate the access region mask image.

8. The apparatus according to claim 7, characterized in that, The semantic segmentation model includes an encoder and a decoder, and the first semantic segmentation submodule includes: The encoding module is used to input the depth image data and the RGB image data into the encoder to generate global feature information; The decoding module is used to input the global feature information into the decoder for decoding and generate the initial passage area mask image.

9. A terminal, characterized in that, The terminal includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the robot navigable area detection method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the robot traversable area detection method as described in any one of claims 1 to 5.

Images