RGBD mobile device stair detection method based on deep learning, storage medium and electronic device

Abstract

The invention discloses an RGBD mobile device stair detection method based on deep learning, a storage medium and an electronic device. The RGBD mobile device stair detection method comprises the following steps: acquiring an RGB image and a depth image input by a camera; inputting the RGB image and the depth image into a stair detection model, performing feature extraction and fusion on the RGB image and the depth image, and performing image segmentation based on fused features to obtain a result image; reading the pixel coordinate of each stair point in the result map, and calculating the relative position of each stair point based on the position of the camera according to the internal reference of the camera; calculating the position of each stair point relative to the robot by combining the relative position of the stair point based on the camera position and the transformation matrix of the bottom of the robot relative to the camera center; by the adoption of the technical scheme, stair points and other key areas can be segmented more accurately, the method adapts to scenes of different environments, different illumination conditions and different stair structures, and higher practicability and generalization ability are achieved in practical application; therefore, accurate spatial positioning of stair points is realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous navigation technology, and in particular to a deep learning-based RGBD mobile device stair detection method, a storage medium and an electronic device. Background Art

[0002] Mobile devices (such as robot navigation, automatic handling equipment on production lines, smart homes, etc.) perform traversable area detection to obtain traversable area information, which is one of the basic functions required for mobile devices to autonomously explore in unknown environments. Traversable area information is an important prerequisite for global exploration planning and local path planning for obstacle avoidance and tracking global paths. In some application scenarios with stair spaces, the stair spaces need to be included in the scope of traversable area detection.

[0003] Stairs have obvious geometric structural features. The existing commonly used stair detection technology mainly uses multi-line laser radar to illuminate the stair area, select the point cloud line beam within a certain angle in front, and fit the point cloud line beam. If the slope of the fitted line is within a certain threshold, it is considered to be a point cloud. By extracting the point cloud features scanned by the laser radar, the location of the stairs, the height, width, depth and other geometric information of the steps are detected. This technical solution strongly relies on multi-line laser radar and is expensive; it can only detect upward stairs, but not downward stairs, and mobile devices are prone to the risk of falling during operation.

[0004] See the application publication number CN118447181A, which discloses a method and device for modeling multi-layer stairs based on laser radar, which obtains a point cloud map based on the stair point cloud data; extracts and classifies planes from the point cloud map and merges the same plane; classifies the stair step surfaces by the stair geometry to generate at least one stair model; and estimates the step size of the stair model by region segmentation based on the stair main coordinate system. This technology uses laser radar for deep learning point cloud detection, but most point cloud detection models require a large amount of calculation, and the semantic perception information of pure laser radar is weak, so the use of point cloud detection models requires high computing resources. Summary of the invention

[0005] Therefore, it is necessary to provide a RGBD mobile device staircase detection method based on deep learning to solve the above problems.

[0006] To achieve the above object, the present invention provides a RGBD mobile device staircase detection method based on deep learning, which comprises the following steps:

[0007] Get the RGB image and depth image input by the camera;

[0008] The RGB image and the depth image are input into the staircase detection model to extract and fuse their features respectively, and the image is segmented based on the fused features to obtain the result image;

[0009] Read the pixel coordinates of each stair point in the result image, and calculate the relative position of each stair point based on the camera position according to the camera's internal parameters;

[0010] The camera's position transformation matrix R relative to the robot's bottom center is obtained through calibration, and the transformation matrix R of the robot's bottom relative to the camera center is calculated -1 ;

[0011] Combine the relative position of the stair point based on the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 , calculate the position of each stair point relative to the robot.

[0012] Furthermore, before the step of inputting the RGB image and the depth image into the staircase detection model to extract and fuse the features thereof respectively, and performing image segmentation based on the fused features to obtain the result image, the step further includes:

[0013] Obtain a staircase image dataset, wherein the staircase image dataset includes a plurality of staircase RGB images and corresponding depth images;

[0014] The stair detection model is trained using a stair image dataset.

[0015] Furthermore, the staircase detection model includes an input part, a feature fusion part, a feature enhancement and upsampling part, and an output part;

[0016] The input part receives the RGB image and the corresponding depth image respectively, and extracts the RGB image and the corresponding depth image to perform multi-scale feature extraction to obtain multiple levels of RGB image features R and depth image features D;

[0017] The feature fusion part fuses the RGB image features R and the depth image features D at the same level to obtain feature maps E at multiple levels;

[0018] The feature enhancement and upsampling part performs feature enhancement on the highest level feature map among the multiple levels of feature maps E, and gradually combines and upsamples the feature map with the previous level to obtain a joint feature map F0;

[0019] The output part changes the number of channels of the joint feature map F0 to 1, and performs image segmentation through an activation function to obtain a result map.

[0020] Further, the input part includes an RGB input branch for receiving the RGB image and a depth input branch for receiving the depth image, wherein the number of channels of the RGB input branch is 3, and the number of channels of the depth input branch is 1;

[0021] The RGB input branch and the depth input branch are respectively provided with corresponding multiple-level image feature extraction units, and the image feature extraction units include a CBR module; the Conv step size of the CBR module is set to 2 for downsampling.

[0022] Furthermore, the feature enhancement and upsampling part performs feature enhancement on the feature map of the highest level, and gradually combines and upsamples the feature map of the previous level to obtain a joint feature map, including the following steps:

[0023] Multiple levels of feature maps E{E1, E2, ..., E i}, where i is a positive integer;

[0024] Using the SPPCSPC module to analyze the highest level feature map E i Perform feature enhancement to obtain F i , to keep the number and size of channels of input and output the same;

[0025] After feature enhancement, F i and E i Perform concat and upsample to obtain F i-1 , F i-1 and E i-1 Combine and upsample to obtain F i-2 , and so on until the joint feature map F0 is obtained.

[0026] Furthermore, the output part changes the number of channels of the joint feature map F0 to 1, and performs image segmentation through an activation function to obtain a result map, including the following steps:

[0027] The CBR module is used to change the number of channels of the joint feature map F0 to 1, and a sigmoid activation function is added to map the output value to between 0 and 1 to obtain the result map, in which the value of the staircase area is 1 and the value of the non-staircase area is 0.

[0028] Furthermore, the step of reading the pixel coordinates of each stair point in the result image and calculating the relative position of each stair point based on the camera position according to the intrinsic parameters of the camera includes the following steps:

[0029] Read the pixel coordinates (x, y) of each stair point in the result image;

[0030] According to the camera internal parameters including focal length (f x , fy ), main point (c x , c y ), then the relative position (X, Y, Z) of each pixel based on the camera position is:

[0031] Z = depth(x, y);

[0032] X = xc x / f x *Z;

[0033] Y=yc y / f y * Z.

[0034] Furthermore, the combined stair point is based on the relative position of the camera position and the transformation matrix R of the robot bottom relative to the camera center. -1 , the step of calculating the position of each stair point relative to the robot is to use the relative position (X, Y, Z) of each pixel based on the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 The multiplication of .

[0035] A storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned RGBD mobile device staircase detection method based on deep learning are implemented.

[0036] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the above-mentioned deep learning-based RGBD mobile device staircase detection method are implemented.

[0037] Different from the existing technology, the above technical solution uses the semantic segmentation method in deep learning to build a neural network to extract and fuse the features of the RGB image and depth image transmitted by the camera before performing image segmentation. It can make full use of the complementary features in the RGB image and the depth image, so as to more accurately segment key areas such as stair points. At the same time, it has powerful feature extraction and generalization capabilities, can adapt to scenes with different environments, different lighting conditions and different stair structures, and has stronger practicality and generalization capabilities in practical applications; then the relative position of each stair point based on the camera position is calculated according to the internal parameters of the camera. Since the relative position relationship between the camera and the robot is known, the position of each stair point relative to the robot can be further calculated to achieve accurate spatial positioning of the stair point. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of the RGBD mobile device staircase detection method based on deep learning of the present invention. DETAILED DESCRIPTION

[0039] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.

[0040] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0041] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0042] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0043] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0044] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0045] Similar to the understanding in the Patent Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0046] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0047] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0048] See also Figure 1As shown, the present invention provides a RGBD mobile device stair detection method based on deep learning. Through the semantic segmentation method in deep learning, a neural network is constructed to extract and fuse the RGB image and depth image transmitted by the camera before image segmentation. The complementary features in the RGB image and the depth image can be fully utilized to more accurately segment key areas such as stair points. At the same time, it has strong feature extraction and generalization capabilities, can adapt to different environments, different lighting conditions and scenes with different stair structures, and has stronger practicality and generalization capabilities in practical applications; then the relative position of each stair point based on the camera position is calculated according to the internal parameters of the camera. Since the relative position relationship between the camera and the robot is known, the position of each stair point relative to the robot can be further calculated to achieve accurate spatial positioning of the stair point; by accurately identifying the stair point and calculating its position, the position and shape of the stair can be more accurately perceived, so as to better adapt to the passage requirements of different stairs, which is the basis for realizing mobile devices going up and down stairs, avoiding collisions with stairs, and avoiding falling down stairs. The above technology can be applied to scenes requiring accurate spatial positioning, such as robot navigation, automatic handling equipment on production lines, smart homes and other fields.

[0049] To further understand the RGBD mobile device staircase detection method based on deep learning of the present invention, the following implementation scheme is provided, which includes the following steps:

[0050] Get the RGB image and depth image input by the camera;

[0051] The RGB image and the depth image are input into the staircase detection model to extract and fuse their features respectively, and the image is segmented based on the fused features to obtain the result image;

[0052] Read the pixel coordinates of each stair point in the result image, and calculate the relative position of each stair point based on the camera position according to the camera's internal parameters;

[0053] The camera's position transformation matrix R relative to the robot's bottom center is obtained through calibration, and the transformation matrix R of the robot's bottom relative to the camera center is calculated -1 ;

[0054] Combine the relative position of the stair point based on the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 , calculate the position of each stair point relative to the robot.

[0055] The camera is installed on the robot. It can be installed on the upper or middle part of the robot and configured to shoot downwards. By adjusting the angle of downward shooting, the RGB image and depth image used to shoot the downward and / or upward stairs are determined to detect the position, shape and direction of the stairs. It can also be installed on the lower part of the robot and configured to shoot upwards to shoot the RGB image and depth image of the upward stairs to detect the position and shape of the stairs. The camera can be an RGB-D camera.

[0056] Before inputting the RGB image and the depth image into the stair detection model to extract and fuse their features respectively, and performing image segmentation based on the fused features to obtain the result image, the stair detection model is also trained so that the stair detection model can accurately segment key areas such as stair points. Specifically, the following steps are included:

[0057] Obtain a staircase image dataset, wherein the staircase image dataset includes a plurality of staircase RGB images and corresponding depth images;

[0058] The stair detection model is trained using a stair image dataset.

[0059] The above-mentioned staircase image dataset includes several staircase RGB images and corresponding depth maps. Preferably, the staircase image dataset includes staircase RGB images and corresponding depth maps in different scenes, and / or includes staircase RGB images and corresponding depth maps in different lighting changes, occlusions, etc. in the same scene, so as to improve the robustness of image segmentation of the staircase detection model. The specific training process is to segment and annotate the staircase RGB image and the corresponding depth map of the staircase image dataset, and use the result as the label of the training data, use the staircase RGB image and the corresponding depth map in the staircase image dataset as training data to train the staircase detection model, calculate the output of the staircase detection model through forward propagation, and then use the loss function to calculate the difference between the output and the label of the training data, and update the parameters of the staircase detection model through the back propagation algorithm to obtain the final staircase detection model. In some embodiments, before the staircase RGB image and the corresponding depth map of the staircase image dataset are segmented and annotated and used as training data, the staircase RGB image and the corresponding depth map of the staircase image dataset can also be pre-processed, such as denoising, data enhancement, etc. Taking data enhancement as an example, operations such as rotating, scaling, and flipping the images in the image sample set can improve the generalization ability and adaptability of the stair detection model.

[0060] The above-mentioned staircase detection model may include an input part, a feature fusion part, a feature enhancement and upsampling part, and an output part;

[0061] The input part receives the RGB image and the corresponding depth image respectively, and extracts the RGB image and the corresponding depth image to perform multi-scale feature extraction to obtain multiple levels of RGB image features R and depth image features D;

[0062] The feature fusion part fuses the RGB image features R and the depth image features D at the same level to obtain feature maps E at multiple levels;

[0063] The feature enhancement and upsampling part performs feature enhancement on the highest level feature map among the multiple levels of feature maps E, and gradually combines and upsamples the feature map with the previous level to obtain a joint feature map F0;

[0064] The output part changes the number of channels of the joint feature map F0 to 1, and performs image segmentation through an activation function to obtain a result map.

[0065] The following is a further explanation of the staircase detection model with i levels, where i is a positive integer. That is, the above input part extracts the RGB image and the corresponding depth image for multi-scale feature extraction to obtain 4 levels of RGB image features R{R1, R2, …, R i} and deep image features D{D1, D2, …, D i}; The RGB image features R and the depth image features D at the same level are fused to obtain the feature maps E{E1, E2, …, E i}, at this time, the highest level feature map E i After feature enhancement, it is F i After the feature enhancement is completed, F i and E i Combine and upsample to obtain F i-1 , F i-1 and E i-1 Combine and upsample to obtain F i-2 , and so on until the joint feature map F0 is obtained. Note that upsampling is required after each combination to ensure the same feature resolution. Finally, the number of channels of the joint feature map F0 is changed to 1, and the image is segmented through the activation function to obtain the result map, in which the staircase area and non-staircase area are marked.

[0066] In some specific embodiments, the input part includes an RGB input branch for receiving the RGB image and a depth input branch for receiving the depth image, wherein the number of channels of the RGB input branch is 3, and the number of channels of the depth input branch is 1;

[0067] The RGB input branch and the depth input branch are respectively provided with corresponding multiple-level image feature extraction units, and the image feature extraction units include a CBR module (Conv+Bn+Rule); the CBR module Conv step size is set to 2 for downsampling.

[0068] In some specific embodiments, the feature fusion part performs a concat channel merging operation on each stacking of the RGB input branch and the depth input branch to obtain feature maps E{E1, E2, ..., E i}; that is, E1 = Concat (R1, D1), E2 = Concat (R2, D2), ..., E i =Concat(R i , D i ).

[0069] In some specific embodiments, the feature enhancement and upsampling part performs feature enhancement on the feature map of the highest level, and gradually combines and upsamples the feature map of the previous level to obtain a joint feature map, including the following steps:

[0070] Multiple levels of feature maps E{E1, E2, ..., E i};

[0071] Using the SPPCSPC module to analyze the highest level feature map E i Perform feature enhancement to obtain F i , to keep the number and size of channels of input and output the same;

[0072] After feature enhancement, F i and E i Perform concat and upsample to obtain F i-1 , F i-1 and E i-1 Combine and upsample to obtain F i-2 , and so on until the joint feature map F0 is obtained.

[0073] In some specific embodiments, the output part changes the number of channels of the joint feature map F0 to 1, and performs image segmentation through an activation function to obtain a result map, comprising the following steps:

[0074] The CBR module is used to change the number of channels of the joint feature map F0 to 1, and a sigmoid activation function is added to map the output value to between 0 and 1 to obtain the result map, in which the value of the staircase area is 1 and the value of the non-staircase area is 0.

[0075] The above step of reading the pixel coordinates of each stair point in the result image and calculating the relative position of each stair point based on the camera position according to the intrinsic parameters of the camera includes the following steps:

[0076] Read the pixel coordinates (x, y) of each stair point in the result image;

[0077] According to the camera internal parameters including focal length (f x , f y ), main point (c x , c y ), then the relative position (X, Y, Z) of each pixel based on the camera position is:

[0078] Z = depth(x, y);

[0079] X = xc x / f x *Z;

[0080] Y=yc y / f y * Z.

[0081] The above combines the relative position of the stair point based on the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 , the step of calculating the position of each stair point relative to the robot is to use the relative position (X, Y, Z) of each pixel based on the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 The multiplication of .

[0082] A storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned RGBD mobile device staircase detection method based on deep learning are implemented. The computer program involved in the embodiment can be stored in a computer device readable storage medium, and the computer device readable storage medium includes but is not limited to a disk, a tape, a magnetic card, a floppy disk, a flash memory, an optical disk, an optical card, a read-only memory (ROM), a random access memory (RAM), an erasable programmable ROM (EPROM) and an electrically erasable programmable ROM (EEPROM), etc., and also includes other biological, physical or chemical structures that can realize the same or equivalent functions as the storage media listed above, such as DNA, RNA, protein and other units with information storage capabilities. In a specific embodiment, the storage medium involved can be one of the above-mentioned media types, or a combination of the above-mentioned media types. In different embodiments, the computer program involved in the embodiment can be stored in a single medium in a centralized manner, or it can be stored in multiple media in a distributed manner. The memory containing a computer device readable storage medium can be a non-volatile memory or a random access memory. These computer device readable storage media may be built into the device, or may be connected to the device involved in the embodiment as an external device or part of an external device. In some embodiments, a memory with a computer device readable storage medium is deployed locally; in other embodiments, a solution of deploying the memory away from the processor may also be adopted, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), etc., or a suitable combination thereof, as long as the computer device can access the memory. In addition, the computer program involved in the embodiment may be stored in plaintext / ciphertext form, or may be designed as training data, which may be integrated and reorganized through model training and implicitly stored in the parameter state of a deep neural network or other machine learning model.

[0083] An electronic device, comprising a memory and a processor, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the above-mentioned RGBD mobile device staircase detection method based on deep learning are implemented. The processor described in the embodiment of the present application can be implemented by hardware, firmware, software or a combination thereof, and can use a circuit, a single or multiple application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), a field programmable gate array (Field Programmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, at least one of a microprocessor, and also includes other physical, biological or chemical structures that can realize functions similar to or equivalent to the above-listed processors, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in each embodiment of the present application or any combination of the steps mentioned therein.

[0084] It should be noted that, although the above embodiments have been described in this article, the patent protection scope of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present invention specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields, are all included in the patent protection scope of the present invention.

Claims

1. A RGBD mobile device staircase detection method based on deep learning, characterized in that: The following steps are involved: Get the RGB image and depth image input by the camera; The RGB image and the depth image are input into the staircase detection model to extract and fuse their features respectively, and the image is segmented based on the fused features to obtain the result image; Read the pixel coordinates of each stair point in the result image, and calculate the relative position of each stair point based on the camera position according to the camera's internal parameters; The camera’s position transformation matrix R relative to the robot’s bottom center is obtained through calibration, and the transformation matrix R of the robot’s bottom relative to the camera’s center is calculated. -1 ; Combine the relative position of the stair point based on the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 , calculate the position of each stair point relative to the robot.

2. The RGBD mobile device staircase detection method based on deep learning according to claim 1 is characterized in that: Before the steps of inputting the RGB image and the depth image into the staircase detection model to extract and fuse the features thereof respectively, and performing image segmentation based on the fused features to obtain the result image, the steps also include: Obtain a staircase image dataset, wherein the staircase image dataset includes a plurality of staircase RGB images and corresponding depth images; The stair detection model is trained using a stair image dataset.

3. The RGBD mobile device staircase detection method based on deep learning according to claim 1 is characterized in that: The staircase detection model includes an input part, a feature fusion part, a feature enhancement and upsampling part, and an output part; The input part receives the RGB image and the corresponding depth image respectively, and extracts the RGB image and the corresponding depth image for multi-scale feature extraction to obtain multiple levels of RGB image features R and depth image features D; The feature fusion part fuses the RGB image features R and the depth image features D at the same level to obtain feature maps E at multiple levels; The feature enhancement and upsampling part performs feature enhancement on the highest level feature map among the multiple levels of feature maps E, and gradually combines and upsamples the feature map with the previous level to obtain a joint feature map F0; The output part changes the number of channels of the joint feature map F0 to 1, and performs image segmentation through an activation function to obtain a result map.

4. The RGBD mobile device staircase detection method based on deep learning according to claim 3 is characterized in that: The input part includes an RGB input branch for receiving the RGB image and a depth input branch for receiving the depth image, wherein the number of channels of the RGB input branch is 3, and the number of channels of the depth input branch is 1; The RGB input branch and the depth input branch are respectively provided with corresponding multiple-level image feature extraction units, and the image feature extraction units include a CBR module; the Conv step size of the CBR module is set to 2 for downsampling.

5. The RGBD mobile device staircase detection method based on deep learning according to claim 3 is characterized in that: The feature enhancement and upsampling part performs feature enhancement on the feature map of the highest level, and gradually combines and upsamples the feature map of the previous level to obtain a joint feature map, including the following steps: Multiple levels of feature maps E{E1, E2, ..., E i }, where i is a positive integer; Using the SPPCSPC module to analyze the highest level feature map E i Perform feature enhancement to obtain F i , to keep the number and size of channels of input and output the same; After feature enhancement, F i and E i Perform concat and upsample to obtain F i-1 , F i-1 and E i-1 Combine and upsample to obtain F i-2 , and so on until the joint feature map F0 is obtained.

6. The RGBD mobile device staircase detection method based on deep learning according to claim 3 is characterized in that: The output part changes the number of channels of the joint feature map F0 to 1, and performs image segmentation through an activation function to obtain a result map, including the following steps: The CBR module is used to change the number of channels of the joint feature map F0 to 1, and a sigmoid activation function is added to map the output value to between 0 and 1 to obtain the result map, in which the value of the staircase area is 1 and the value of the non-staircase area is 0.

7. The RGBD mobile device staircase detection method based on deep learning according to claim 1 is characterized in that: The step of reading the pixel coordinates of each stair point in the result image and calculating the relative position of each stair point based on the camera position according to the intrinsic parameters of the camera includes the following steps: Read the pixel coordinates (x, y) of each stair point in the result image; According to the camera internal parameters including focal length (f x , f y ), main point (c x , c y ), then the relative position (X, Y, Z) of each pixel based on the camera position is: Z = depth(x, y); X=x-c x / f x *Z; Y=y-c y / f y *Z。 8. The RGBD mobile device staircase detection method based on deep learning according to claim 1 is characterized in that: The combined stair point is based on the relative position of the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 , the step of calculating the position of each stair point relative to the robot is to use the relative position (X, Y, Z) of each pixel based on the camera position and the transformation matrix R of the robot bottom relative to the camera center -1 The multiplication of .

9. A storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the deep learning-based RGBD mobile device staircase detection method as described in any one of claims 1 to 8 are implemented.

10. An electronic device, characterized in that It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the RGBD mobile device staircase detection method based on deep learning as described in any one of claims 1 to 8 are implemented.

Images