Deep learning-based RGBD mobile device stair detection method, storage medium, and electronic device

Abstract

This invention discloses a deep learning-based RGBD mobile device stair detection method, storage medium, and electronic device, comprising the following steps: acquiring an RGB image and a depth image input from a camera; inputting the RGB image and depth image into a stair detection model for feature extraction and fusion, and performing image segmentation based on the fused features to obtain a result image; reading the pixel coordinates of each stair point in the result image, and calculating the relative position of each stair point based on the camera's intrinsic parameters; combining the relative position of the stair point based on the camera's position and the transformation matrix of the robot's bottom relative to the camera's center to calculate the position of each stair point relative to the robot; using the above technical solution, key areas such as stair points can be segmented more accurately, adapting to different environments, different lighting conditions, and different stair structures, and has stronger practicality and generalization ability in practical applications; thereby achieving precise spatial positioning of stair points.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology, and in particular to a deep learning-based RGBD mobile device stair detection method, storage medium, and electronic device. Background Technology

[0002] Mobile devices (such as robot navigation, automated handling equipment on production lines, and smart home devices) need to detect passable areas to obtain information about accessible zones. This is a fundamental function for mobile devices to autonomously explore unknown environments. Passable area information is a crucial prerequisite for global exploration planning and local path planning used for obstacle avoidance and tracking global paths. In some application scenarios with stairwells, it is necessary to include the stairwells in the scope of passable area detection.

[0003] Staircases have distinct geometric structural features. Current common staircase detection technologies primarily use multi-line LiDAR to illuminate the staircase area, selecting point cloud lines within a certain angle in front of the staircase. These point cloud lines are then fitted; if the slope of the fitted line falls within a certain threshold, it is considered a point cloud. By extracting the point cloud features from the LiDAR scan, the location of the staircase, the height, width, and depth of the steps can be detected. However, this technology heavily relies on multi-line LiDAR, making it expensive; furthermore, it can only detect upward-facing staircases, not downward-facing ones, increasing the risk of mobile devices falling during operation.

[0004] See CN118447181A, a method and apparatus for modeling multi-level stairs based on lidar, which discloses obtaining a point cloud map from stair point cloud data; extracting and classifying planes from the point cloud map and merging the same planes; classifying stair step surfaces according to the stair geometry to generate at least one stair model; and estimating the step dimensions of the stair model through region segmentation based on the stair principal coordinate system. This technology uses lidar for deep learning-based point cloud detection; however, most point cloud detection models require significant computation, and pure lidar has weak semantic perception information, making point cloud detection models computationally resource-intensive. Summary of the Invention

[0005] Therefore, a deep learning-based RGBD mobile device stair detection method is needed to solve the above problems.

[0006] To achieve the above objectives, this invention provides a deep learning-based RGBD mobile device stair detection method, which includes the following steps:

[0007] Acquire the RGB image and depth image input from the camera;

[0008] The RGB image and depth image are input into the stair detection model for feature extraction and fusion, and image segmentation is performed based on the fused features to obtain the result image;

[0009] Read the pixel coordinates of each stair point in the result image, and calculate the relative position of each stair point based on the camera's position according to the camera's intrinsic parameters;

[0010] The pose transformation matrix R of the camera relative to the center of the robot's bottom is obtained through calibration, and the transformation matrix R of the robot's bottom relative to the center of the camera is calculated. -1 ;

[0011] Combining the relative positions of the stair points based on the camera positions and the transformation matrix R of the robot's bottom relative to the camera center. -1 Calculate the position of each stair point relative to the robot.

[0012] Furthermore, before the steps of inputting the RGB image and depth image into the staircase detection model for feature extraction and fusion, and performing image segmentation based on the fused features to obtain the result image, the following steps are also included:

[0013] Obtain a staircase image dataset, which includes several staircase RGB images and corresponding depth images;

[0014] The stair detection model was trained using a stair image dataset.

[0015] Furthermore, the staircase detection model includes an input part, a feature fusion part, a feature enhancement and upsampling part, and an output part;

[0016] The input part receives the RGB image and the corresponding depth image respectively, and extracts the RGB image and the corresponding depth image for multi-scale feature extraction to obtain RGB image features R and depth image features D at multiple levels;

[0017] The feature fusion section fuses RGB image features R and depth image features D at the same level to obtain feature maps E at multiple levels.

[0018] The feature enhancement and upsampling part enhances the highest-level feature map in the feature map E of multiple levels, and gradually combines and upsamples it with the feature map of the next higher level to obtain a joint feature map F0;

[0019] The output part reduces the number of channels in the joint feature map F0 to 1, and then performs image segmentation using an activation function to obtain the result map.

[0020] Furthermore, the input portion includes an RGB input branch for receiving the RGB image and a depth input branch for receiving the depth image, wherein the RGB input branch has 3 channels and the depth input branch has 1 channel;

[0021] The RGB input branch and the depth input branch are respectively equipped with multiple levels of image feature extraction units, and the image feature extraction unit includes a CBR module; the CBR module has a Conv step size of 2 for downsampling.

[0022] Furthermore, the feature enhancement and upsampling part enhances the highest-level feature map and progressively combines and upsamples it with the feature map of the previous level to obtain a joint feature map, including the following steps:

[0023] Feature maps of multiple levels E{E1, E2, ..., E i}, where i is a positive integer;

[0024] Using the SPPCSPC module to analyze the highest-level feature map E i Feature enhancement is performed to obtain F i To keep the number and size of input and output channels the same;

[0025] After feature enhancement is completed, F i and E i Perform concat and upsampling to obtain F i-1 , will F i-1 and E i-1 Combined and upsampled to obtain F i-2 This process continues until the joint feature map F0 is obtained.

[0026] Furthermore, the step of reducing the number of channels in the joint feature map F0 to 1 in the output section and obtaining the result map by performing image segmentation through an activation function includes the following steps:

[0027] The joint feature map F0 is converted to 1 channel using the CBR module, and a sigmoid activation function is added to map the output value to between 0 and 1, resulting in the image where the value of the stair area is recorded as 1 and the value of non-stair areas is 0.

[0028] Furthermore, the step of reading the pixel coordinates of each stair point in the result image and calculating the relative position of each stair point based on the camera's intrinsic parameters includes the following steps:

[0029] Read the pixel coordinates (x, y) of each stair point in the result image;

[0030] According to camera intrinsic parameters, including focal length (f) x fy ), principal point (c x c y Then, the relative position (X, Y, Z) of each pixel based on the camera position is:

[0031] Z = depth(x, y);

[0032] X = xc x / f x *Z;

[0033] Y = yc y / f y * Z.

[0034] Furthermore, the combination of the stair point is based on the relative position of the camera position and the transformation matrix R of the robot's bottom relative to the camera center. -1 The step of calculating the position of each stair point relative to the robot is to use the relative position (X, Y, Z) of each pixel based on the camera position and the transformation matrix R of the robot's bottom relative to the camera center. -1 The product is obtained by multiplying.

[0035] A storage medium storing a computer program that, when executed by a processor, implements the steps of the deep learning-based RGBD mobile device stair detection method described above.

[0036] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the above-described deep learning-based RGBD mobile device stair detection method.

[0037] Unlike existing technologies, the above-mentioned technical solution uses semantic segmentation in deep learning to construct a neural network that extracts and fuses features from the RGB and depth maps transmitted by the camera before performing image segmentation. This fully utilizes the complementary features in the RGB and depth maps, thereby more accurately segmenting key areas such as stair points. Furthermore, it possesses powerful feature extraction and generalization capabilities, adapting to different environments, lighting conditions, and stair structures, making it more practical and versatile in real-world applications. Then, based on the camera's intrinsic parameters, the relative position of each stair point is calculated according to the camera's location. Since the relative positional relationship between the camera and the robot is known, the position of each stair point relative to the robot can be further calculated to achieve precise spatial positioning of the stair points. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the deep learning-based RGBD mobile device staircase detection method of the present invention. Detailed Implementation

[0039] To explain in detail the technical content, structural features, objectives, and effects of the technical solution, the following description is provided in conjunction with specific embodiments and accompanying drawings.

[0040] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0041] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0042] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0043] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0044] Unless otherwise specified, the use of terms such as “comprising,” “including,” “having,” or other similar expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0045] Similar to the interpretation in the Patent Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also interpreted in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0046] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0047] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0048] See Figure 1As shown, this invention provides a deep learning-based RGBD mobile device stair detection method. By employing semantic segmentation in deep learning, a neural network is constructed to extract and fuse features from the RGB and depth maps transmitted by the camera before image segmentation. This fully utilizes complementary features in the RGB and depth maps, resulting in more accurate segmentation of key areas such as stair points. Furthermore, it possesses strong feature extraction and generalization capabilities, adapting to different environments, lighting conditions, and stair structures, demonstrating greater practicality and generalization ability in real-world applications. The method then calculates the relative position of each stair point based on the camera's intrinsic parameters. Since the relative positional relationship between the camera and the robot is known, the position of each stair point relative to the robot can be further calculated, achieving precise spatial positioning of the stair points. Accurate identification of stair points and calculation of their positions allows for more accurate perception of the stair's location and shape, better adapting to different stair passage requirements. This is fundamental for enabling mobile devices to navigate stairs, avoid collisions, and prevent falls. This technology can be applied to scenarios requiring precise spatial positioning, such as robot navigation, automated handling equipment on production lines, and smart homes.

[0049] To further understand the deep learning-based RGBD mobile device stair detection method of the present invention, the following implementation scheme is provided, which includes the following steps:

[0050] Acquire the RGB image and depth image input from the camera;

[0051] The RGB image and depth image are input into the stair detection model for feature extraction and fusion, and image segmentation is performed based on the fused features to obtain the result image;

[0052] Read the pixel coordinates of each stair point in the result image, and calculate the relative position of each stair point based on the camera's position according to the camera's intrinsic parameters;

[0053] The pose transformation matrix R of the camera relative to the center of the robot's bottom is obtained through calibration, and the transformation matrix R of the robot's bottom relative to the center of the camera is calculated. -1 ;

[0054] Combining the relative positions of the stair points based on the camera positions and the transformation matrix R of the robot's bottom relative to the camera center. -1 Calculate the position of each stair point relative to the robot.

[0055] The aforementioned camera is mounted on the robot, either on the upper or middle part, and configured to shoot downwards. The angle of the downward shot is adjusted to determine the RGB and depth images of the downward-facing and / or upward-facing stairs, thus detecting the position, shape, and orientation of the stairs. Alternatively, it can be mounted on the lower part of the robot and configured to shoot upwards, capturing the RGB and depth images of the upward-facing stairs to detect their position and shape. The camera can be an RGB-D camera.

[0056] Before the steps of inputting the RGB image and depth image into the stair detection model for feature extraction and fusion, and then performing image segmentation based on the fused features to obtain the result image, the stair detection model is trained to accurately segment key regions such as stair points. Specifically, this includes the following steps:

[0057] Obtain a staircase image dataset, which includes several staircase RGB images and corresponding depth images;

[0058] The stair detection model was trained using a stair image dataset.

[0059] The aforementioned staircase image dataset includes several staircase RGB images and corresponding depth maps. Preferably, the staircase image dataset includes staircase RGB images and corresponding depth maps under different scenes, and / or staircase RGB images and corresponding depth maps under different lighting conditions, occlusion conditions, etc., in the same scene, to improve the robustness of the staircase detection model's image segmentation. The specific training process involves segmenting and labeling the staircase RGB images and corresponding depth maps in the staircase image dataset, using the results as labels for the training data. The staircase detection model is trained using the staircase RGB images and corresponding depth maps in the staircase image dataset as training data. The output of the staircase detection model is calculated through forward propagation, and then the difference between the output and the labels of the training data is calculated using a loss function. The staircase detection model parameters are then updated using a backpropagation algorithm to obtain the final staircase detection model. In some embodiments, before segmenting and labeling the staircase RGB images and corresponding depth maps in the staircase image dataset and using them as training data, preprocessing methods such as denoising and data augmentation can be performed on the staircase RGB images and corresponding depth maps in the staircase image dataset. For example, data augmentation can be performed on images in the image sample set by rotating, scaling, or flipping them to improve the generalization ability and adaptability of the staircase detection model.

[0060] The aforementioned staircase detection model may include an input part, a feature fusion part, a feature enhancement and upsampling part, and an output part;

[0061] The input part receives the RGB image and the corresponding depth image respectively, and extracts the RGB image and the corresponding depth image for multi-scale feature extraction to obtain RGB image features R and depth image features D at multiple levels;

[0062] The feature fusion section fuses RGB image features R and depth image features D at the same level to obtain feature maps E at multiple levels.

[0063] The feature enhancement and upsampling part enhances the highest-level feature map in the feature map E of multiple levels, and gradually combines and upsamples it with the feature map of the next higher level to obtain a joint feature map F0;

[0064] The output part reduces the number of channels in the joint feature map F0 to 1, and then performs image segmentation using an activation function to obtain the result map.

[0065] The staircase detection model is further illustrated below using i levels, where i is a positive integer. Specifically, the input portion extracts RGB images and corresponding depth maps for multi-scale feature extraction, obtaining RGB image features R{R1, R2, ..., R...} at four levels. i} and depth image features D{D1, D2, ..., D i}; The RGB image features R and depth image features D at the same level are fused to obtain feature maps E{E1, E2, ..., E} at four levels. i At this point, for the highest-level feature map E i After feature enhancement, it becomes F i After feature enhancement is completed, F will be... i and E i Perform concatenation and upsampling to obtain F i-1 , will F i-1 and E i-1 Combined and upsampled to obtain F i-2 This process is repeated until the joint feature map F0 is obtained. Note that an upsampling is required after each joint to ensure the same feature resolution. Finally, the number of channels in the joint feature map F0 is reduced to 1, and image segmentation is performed using an activation function to obtain the result image, which marks the staircase area and non-staircase areas.

[0066] In some specific embodiments, the input portion includes an RGB input branch for receiving the RGB image and a depth input branch for receiving the depth image, wherein the RGB input branch has 3 channels and the depth input branch has 1 channel;

[0067] The RGB input branch and the depth input branch are respectively equipped with multiple levels of image feature extraction units. The image feature extraction unit includes a CBR module (Conv+Bn+Rule); the Conv step size of the CBR module is set to 2 for downsampling.

[0068] In some specific embodiments, the feature fusion part performs a concat channel merging operation for each stack of the RGB input branch and the depth input branch, obtaining feature maps E{E1, E2, ..., E} at multiple levels. i}; that is, E1 = Concat(R1, D1), E2 = Concat(R2, D2), ..., E i =Concat(R) i D i ).

[0069] In some specific embodiments, the feature enhancement and upsampling part enhances the highest-level feature map and progressively combines and upsamples it with the feature map of the next higher level to obtain a joint feature map, including the following steps:

[0070] Feature maps of multiple levels E{E1, E2, ..., E i};

[0071] Using the SPPCSPC module to analyze the highest-level feature map E i Feature enhancement is performed to obtain F i To keep the number and size of input and output channels the same;

[0072] After feature enhancement is completed, F i and E i Perform concat and upsampling to obtain F i-1 , will F i-1 and E i-1 Combined and upsampled to obtain F i-2 This process continues until the joint feature map F0 is obtained.

[0073] In some specific embodiments, the output section reduces the number of channels of the joint feature map F0 to 1, and the step of obtaining the result map by image segmentation through an activation function includes the following steps:

[0074] The joint feature map F0 is converted to 1 channel using the CBR module, and a sigmoid activation function is added to map the output value to between 0 and 1, resulting in the image where the value of the stair area is recorded as 1 and the value of non-stair areas is 0.

[0075] The steps for obtaining the pixel coordinates of each stair point in the above-mentioned image and calculating the relative position of each stair point based on the camera's intrinsic parameters include the following:

[0076] Read the pixel coordinates (x, y) of each stair point in the result image;

[0077] According to camera intrinsic parameters, including focal length (f) x f y ), principal point (c x c y Then, the relative position (X, Y, Z) of each pixel based on the camera position is:

[0078] Z = depth(x, y);

[0079] X = xc x / f x *Z;

[0080] Y = yc y / f y * Z.

[0081] The above combines the relative position of the stair point based on the camera position and the transformation matrix R of the robot's bottom relative to the camera center. -1 The step of calculating the position of each stair point relative to the robot is to use the relative position (X, Y, Z) of each pixel based on the camera position and the transformation matrix R of the robot's bottom relative to the camera center. -1 The product is obtained by multiplying.

[0082] A storage medium stores a computer program that, when executed by a processor, implements the steps of the deep learning-based RGBD mobile device stair detection method described above. The computer program involved in the embodiments can be stored in a computer-readable storage medium, including but not limited to disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc., as well as other biological, physical, or chemical structures capable of performing similar or equivalent functions to the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium can be one of the above-mentioned media types or a combination of the above media types. In different embodiments, the computer program involved in the embodiments can be centrally stored in a single medium or distributed across multiple media. The memory containing the computer-readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device or connected to the device of the embodiments as an external device or part of an external device. In some embodiments, the memory having the computer device readable storage media is deployed locally; in other embodiments, the memory can be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network can be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as it enables computer device access to the memory. Furthermore, the computer programs involved in the embodiments can be stored in plaintext / ciphertext form or designed as training data, which can be integrated and recombined through model training and implicitly stored in the parameter states of deep neural networks or other machine learning models.

[0083] An electronic device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned deep learning-based RGBD mobile device stair detection method. The processor described in this application embodiment can be implemented through hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures capable of performing similar or equivalent functions to the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., thereby enabling the processor to execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0084] It should be noted that although the above embodiments have been described herein, this does not limit the scope of patent protection of the present invention. Therefore, any changes and modifications made to the embodiments described herein based on the innovative concept of the present invention, or equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields, are all included within the scope of patent protection of the present invention.

Claims

1. A method for detecting stairs based on deep learning of an RGBD mobile device, characterized in that, The method comprises the following steps: obtaining an RGB image and a depth image input by a camera; inputting the RGB image and the depth image into a staircase detection model to respectively extract and fuse features thereof, and performing image segmentation based on the fused features to obtain a result image; the staircase detection model comprises an input part, a feature fusion part, a feature enhancement and up-sampling part, and an output part; The input part respectively receives the RGB image and the corresponding depth image, extracts the RGB image and the corresponding depth image for multi-scale feature extraction, and obtains a plurality of levels of RGB image features and depth image features ​ The feature fusion part fuses the RGB image features and the depth image features of the same level to obtain feature maps of multiple levels ;​​ The feature enhancement is performed on the feature maps of multiple levels by the feature enhancement and up-sampling part The feature maps of the highest level are subjected to feature enhancement and are gradually combined and up-sampled with the feature maps of the previous level to obtain combined feature maps ; The output part will combine the feature maps. The number of channels is reduced to 1, and the image is segmented using an activation function to obtain the result image; reading pixel coordinates of each staircase point in the result image, and calculating a relative position of each staircase point based on a camera position according to an internal parameter of the camera; Obtain the pose transformation matrix of the camera relative to the center of the bottom of the robot through calibration ; combining the relative position of the stair point based on the camera position and the pose transformation matrix of the camera relative to the center of the robot's bottom , calculating the position of each stair point relative to the robot; The feature enhancement and up-sampling part performs feature enhancement on the feature map of the highest level, and gradually performs joint and up-sampling with the feature map of the previous level to obtain a joint feature map comprising the steps of: Multiple levels of feature maps wherein is a positive integer; Reuse SPPCSPC module to the highest level feature map Perform feature enhancement to obtain To keep the same number of channels and size of input and output; After the end of feature enhancement and concat joint and up-sampling to obtain , joint and and up-sampling to obtain , and so on until the joint feature map is obtained. 2.The deep learning-based RGBD mobile device stair detection method of claim 1, wherein, before the step of inputting the RGB image and the depth image into the staircase detection model to respectively extract and fuse features thereof, and performing image segmentation based on the fused features to obtain a result image, the method further comprises: obtaining a staircase image dataset, the staircase image dataset comprising a plurality of staircase RGB images and corresponding depth images; training the staircase detection model through the staircase image dataset. 3.The deep learning based RGBD mobile device stair detection method of claim 1, wherein, the input part comprises an RGB input branch receiving the RGB image and a depth input branch receiving the depth image, wherein the channel number of the RGB input branch is 3, and the channel number of the depth input branch is 1; the RGB input branch and the depth input branch are respectively provided with a plurality of picture feature extraction units of corresponding levels, and the picture feature extraction unit comprises a CBR module; the CBR module is configured to perform down-sampling with a Conv step length of 2. 4.The deep learning based RGBD mobile device stair detection method of claim 1, wherein, The output part changes the channel number of the joint feature map to 1, and obtains a result image through an activation function. The joint feature map The CBR module is used to change the channel number to 1, add a sigmoid activation function to map the output value to 0 to 1, and obtain a result map, in which the value of the stair area is recorded as 1 and the non-stair is 0. 5.The deep learning based RGBD mobile device stair detection method of claim 1, wherein, the step of reading pixel coordinates of each staircase point in the result image, and calculating a relative position of each staircase point based on a camera position according to an internal parameter of the camera, comprises the following steps: reading the pixel coordinates of each stair point in the result image ; According to the camera intrinsic parameters including focal length , principal point , the relative position of each pixel coordinate based on the camera position is : = depth ; = * ； = / * 。 6.The deep learning based RGBD mobile device stair detection method of claim 1, wherein, The binding stair point is based on the relative position of the camera position and the pose transformation matrix of the camera relative to the center of the robot bottom The step of calculating the position of each stair point relative to the robot is obtained by multiplying the relative position of the camera position based on each pixel coordinate and the pose transformation matrix of the camera relative to the center of the robot bottom .

7. A storage medium, characterized by the storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the RGBD mobile device staircase detection method based on deep learning according to any one of claims 1-6.

8. An electronic device, comprising: The device comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to realize the steps of the RGBD mobile device staircase detection method based on deep learning according to any one of claims 1-6.

Images