A Fast Binocular Method for Active Obstacle Avoidance of Mobile Robots

Abstract

A fast binocular method applicable to active obstacle avoidance of a mobile robot is provided. The method includes: S110) obtaining a first disparity map and a depth and point cloud map having the same first resolution as the first disparity map based on an initial left image and an initial right image in a binocular camera; S120) using a predetermined number of disparity grid gradient operators (e.g., 32) to generate a disparity grid gradient texture map of a predetermined number of bits (e.g., 32 bits) from the first disparity map with a super-grid composed of m rows * m columns of grids of the first disparity map as a unit; S130) obtaining the positions of obstacle foreground objects and the ground from the disparity grid gradient texture map; S140) determining, based on the obtained positions of the obstacle foreground objects and the ground, the obstacle foreground objects and the ground for which the mobile robot needs to actively avoid obstacles from the depth and point cloud map, where each grid is composed of n rows * n columns of pixels, and each of m and n is an integer greater than or equal to 1 and m is an odd number.

Description

Technical Field

[0001] This application relates to obstacle avoidance for robots, and more particularly, to a fast binocular method for active obstacle avoidance applicable to mobile robots with binocular cameras. Background Art

[0002] Currently, due to characteristics such as simple structure, high stability, low cost, adaptability to indoor and outdoor scenarios, and large parallax estimation range, binocular-based obstacle avoidance solutions have received increasing attention in the field of robot obstacle avoidance. However, currently, it is difficult to balance the accuracy and performance of its parallax estimation, the generalization ability of obstacle detection is insufficient, and the noise robustness is not high, which seriously restricts the application of binocular obstacle avoidance solutions in actual scenarios. Therefore, this application proposes a fast binocular method applicable to active obstacle avoidance of mobile robots. Summary of the Invention

[0003] The purpose of providing this Summary of the Invention section is to briefly introduce the selection of inventive concepts, and these inventive concepts will be further described in the following Detailed Description section. The purpose of this Summary of the Invention section is not to identify the key features or essential features of the claimed subject matter, nor to assist in determining the scope of the claimed subject matter.

[0004] According to one aspect of the present invention, there is provided a fast binocular method applicable to active obstacle avoidance of mobile robots, the method may include: S1) generating a first disparity map and a depth and point cloud map corresponding to the first disparity map with the same first resolution as the first disparity map based on an initial left image and an initial right image in a binocular camera; S2) using a predetermined number of disparity grid gradient operators to generate a disparity grid gradient texture map with a predetermined number of bits from the first disparity map in units of super grids composed of m rows * m columns of grids of the first disparity map; S3) obtaining the positions of obstacle foreground objects and the ground from the disparity grid gradient texture map; S4) determining, based on the obtained positions of the obstacle foreground objects and the ground, the obstacle foreground objects and the ground for which the mobile robot needs to actively avoid obstacles from the depth and point cloud map, where each grid is composed of n rows * n columns of pixels, and each of m and n is an integer greater than or equal to 1 and m is an odd number.

[0005] Step S1) may include: converting an initial left image and an initial right image into aligned first left and right images; successively downsampling the first left and right images as the bottom layer of a pyramid to obtain a first image pair composed of a first left image and a first right image with a first resolution, a second image pair composed of a second left image and a second right image with a second resolution lower than the first resolution, and a third image pair composed of a third left image and a third right image with a third resolution lower than the second resolution; respectively obtaining a second disparity map with the second resolution and a third disparity map with the third resolution from the second and third image pairs; establishing a disparity model based on the third and second disparity maps; generating a first disparity map of the first image pair using the disparity model; and generating a depth and point cloud map with the same first resolution as the first disparity map from the first disparity map.

[0006] Respectively obtaining a second disparity map with the second resolution and a third disparity map with the third resolution from the second and third image pairs may be performed using the BM binocular matching algorithm.

[0007] Establishing a disparity model based on the third and second disparity maps may include: dividing the third disparity map into multiple image blocks and obtaining the disparity range of each image block; and mapping the third disparity map to the second disparity map and establishing a disparity model in combination with the edge detection result of the second disparity map.

[0008] Mapping the third disparity map to the second disparity map may include mapping the disparity range, hole coordinates, and noise coordinates of the third disparity map to the second disparity map.

[0009] Generating a first disparity map of the first image pair using the disparity model may include: performing a Census transform on the first left and right images of the first image pair; determining the cost function of the first left and right images after the Census transform based on the disparity range in the disparity model; performing fast mean filtering using the noise coordinates, hole coordinates, and edge coordinates in the disparity model; and for each pixel, generating a first disparity map by outputting pixel data with the minimum disparity from the mean-filtered disparity model based on the cost function.

[0010] The disparity range in the disparity model may be dynamically updated with the low-speed movement of the mobile robot.

[0011] The disparity range in the disparity model being dynamically updated with the low-speed movement of the mobile robot may include: updating the disparity range in the disparity model by using the disparity range in the neighborhood of the image block corresponding to each position in the current frame of the previous frame.

[0012] Step S1) may include: generating an initial disparity map for an initial left image and an initial right image; converting the initial disparity map into an initial depth and point cloud map having the same first resolution as the initial disparity map; and removing data having a height greater than or equal to a predetermined height from the initial depth and point cloud map to form a depth and point cloud map, and generating a first disparity map from the depth and point cloud map from which the data has been removed.

[0013] The predetermined height may be 0.2 meters.

[0014] The above-mentioned predetermined quantity may be 32, and the above-mentioned predetermined number of bits may also be 32.

[0015] The 32 disparity grid gradient operators may include 8 predetermined point gradient operators, 14 predetermined line gradient operators, 6 predetermined surface gradient operators, and 4 random gradient operators.

[0016] The 8 predetermined point gradient operators may include: a first type of point gradient operator for representing the gray level change between a central grid at a central position in a super grid and the nearest neighbor grids; and a second type of point gradient operator for representing the gray level change between the central grid and the grids at the farthest distance.

[0017] The first type of point gradient operator may be respectively used to represent: the gray level change between the central grid and the nearest neighbor left grid; the gray level change between the central grid and the nearest neighbor upper grid; the gray level change between the central grid and the nearest neighbor right grid; and the gray level change between the central grid and the nearest neighbor lower grid.

[0018] The second type of point gradient operator may be respectively used to represent: the gray level change between the central grid and the grid at the first row and first column; the gray level change between the central grid and the grid at the first row and the m-th column; the gray level change between the central grid and the grid at the m-th row and m-th column; and the gray level change between the central grid and the grid at the m-th row and first column.

[0019] The 8 predetermined point gradient operators may include: a first type of point gradient operator for representing the gray level change in the up direction, down direction, left direction, and right direction with respect to a grid at a central position in a super grid; and a second type of point gradient operator for representing the gray level change in the upper left diagonal direction, upper right diagonal direction, lower right diagonal direction, and lower left diagonal direction with respect to the central grid.

[0020] The 14 predefined line gradient operators may include: a first type of line gradient operator for representing the gray-scale change between m grids on one row and m grids on another row in a super-grid; a second type of line gradient operator for the gray-scale change between m grids on one column and m grids on another column in the super-grid; a third type of line gradient operator for representing the gray-scale change between m grids on one row and m grids on one column in the super-grid; and a fourth type of line gradient operator for representing the gray-scale change between m grids on one diagonal of the super-grid and m grids on the other diagonal of the super-grid.

[0021] The first type of line gradient operator can be respectively used to represent: the gray-scale change between m grids on the first row and m grids on the second row in the super-grid; the gray-scale change between m grids on the (m - 1) / 2-th row and m grids on the (m + 1) / 2-th row in the super-grid; the gray-scale change between m grids on the (m + 1) / 2-th row and m grids on the (m + 3) / 2-th row in the super-grid; and the gray-scale change between m grids on the (m - 1)-th row and m grids on the m-th row in the super-grid.

[0022] The second type of line gradient operator can be respectively used to represent: the gray-scale change between m grids on the first column and m grids on the second column in the super-grid; the gray-scale change between m grids on the (m - 1) / 2-th column and m grids on the (m + 1) / 2-th column in the super-grid; the gray-scale change between m grids on the (m + 1) / 2-th column and m grids on the (m + 3) / 2-th column in the super-grid; and the gray-scale change between m grids on the (m - 1)-th column and m grids on the m-th column in the super-grid.

[0023] The third type of line gradient operator can be respectively used to represent: the gray-scale change between m grids on the first column and m grids on the first row in the super-grid; the gray-scale change between m grids on the first row and m grids on the m-th column in the super-grid; the gray-scale change between m grids on the m-th column and m grids on the m-th row in the super-grid; the gray-scale change between m grids on the (m + 1) / 2-th row and m grids on the (m + 1) / 2-th column in the super-grid; and the gray-scale change between m grids on the m-th row and m grids on the first column in the super-grid.

[0024] The six predefined face gradient operators may include: a first type of face gradient operator for representing the gray-scale change between at least two adjacent rows of grids and another at least two adjacent rows of grids in a super-grid; a second type of face gradient operator for representing the gray-scale change between at least two adjacent columns of grids and another at least two adjacent columns of grids in a super-grid; a third type of face gradient operator for representing the gray-scale change between all grids on a diagonal and on one side of the diagonal and all grids on the diagonal and on the other side of the diagonal in a super-grid; a fourth type of face gradient operator for representing the gray-scale change between all grids at the edge and q*q grids at the central position in a super-grid, where q is an odd number greater than or equal to 3; and a fifth type of face gradient operator for representing the gray-scale change between two sets of k*k grids symmetric with respect to the central grid along a first diagonal direction and two sets of k*k grids symmetric with respect to the central grid along a second diagonal direction in a super-grid, where k is an integer greater than or equal to 2.

[0025] The first type of face gradient operator can be used to represent the gray-scale change between a rectangular region composed of m grids in the first row and m grids in the second row in a super-grid and a rectangular region composed of m grids in the (m - 1)th row and m grids in the mth row.

[0026] The second type of face gradient operator can be used to represent respectively: the gray-scale change between a rectangular region composed of m grids in the first column and m grids in the second column in a super-grid and a rectangular region composed of m grids in the (m - 1)th column and m grids in the mth column.

[0027] The third type of face gradient operator can be used to represent respectively: the gray-scale change between a triangular region composed of grids on and between the first column, the first row and the first diagonal in a super-grid and a triangular region composed of grids on and between the first diagonal, the mth row and the mth column; and the gray-scale change between a triangular region composed of grids on and between the first row, the mth column and the second diagonal in a super-grid and a triangular region composed of grids on and between the second diagonal, the first column and the mth row.

[0028] The fourth type of face gradient operator can be used to represent: the gray-scale change between all grids on the first row, the first column, the mth row and the mth column in a super-grid and q*q grids at the central position.

[0029] The fifth type of surface gradient operator can be used to represent the gray-scale change between the region composed of the grids at the first row and first column, the grids at the first row and second column, the grids at the second row and first column, the grids at the second row and second column, the grids at the (m - 1)th row and (m - 1)th column, the grids at the (m - 1)th row and mth column, the grids at the mth row and (m - 1)th column, and the grids at the mth row and mth column in the super grid, and the region composed of the grids at the (m - 1)th row and first column, the grids at the (m - 1)th row and second column, the grids at the mth row and first column, the grids at the mth row and second column, the grids at the first row and (m - 1)th column, the grids at the first row and mth column, the grids at the second row and (m - 1)th column, and the grids at the second row and mth column.

[0030] m can be greater than or equal to 5.

[0031] n can be greater than or equal to 2.

[0032] Obtaining the positions of the obstacle foreground and the ground from the disparity grid gradient texture map can be achieved by using a predetermined convolutional neural network.

[0033] According to another aspect of the present invention, an electronic device is provided. The electronic device may include: a processor, a memory connected to the processor, and instructions stored in the memory. When the instructions are executed, the processor is caused to execute the above method.

[0034] According to still another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic flowchart showing a fast binocular method applicable to active obstacle avoidance of a mobile robot according to an embodiment of the present application.

[0036] Figure 2 is a schematic flowchart showing Figure 1 step S110 according to an embodiment of the present application.

[0037] Figure 3 is a schematic flowchart showing Figure 2 step S240 according to an embodiment of the present application.

[0038] Figure 4 is a schematic flowchart showing Figure 2 step S250 according to an embodiment of the present application.

[0039] Figure 5 is a schematic flowchart showing according to another embodiment Figure 1Schematic flowchart of step S110.

[0040] Figure 6A and Figure 6B is for illustrating Figure 4 schematic diagram of fast mean filtering in step S430.

[0041] Figure 7 is a schematic diagram for illustrating an example of a super grid (disparity grid) according to an embodiment of the present application.

[0042] Figures 8A to 8H is a schematic diagram for illustrating an example of an 8-point gradient operator according to the present application.

[0043] Figures 9A to 9N is a schematic diagram for illustrating an example of a 14-line gradient operator according to the present application.

[0044] Figures 10A to 10F is a schematic diagram for illustrating an example of a 6-face gradient operator according to the present application.

[0045] Figures 11A to 11D is a schematic diagram for illustrating an example of a random gradient operator according to the present application.

[0046] Figure 12 is a schematic diagram showing an example of a convolutional neural network model applied to Figure 1 step S130 according to an embodiment of the present application. Detailed implementation manners

[0047] The following description is provided to enable those skilled in the art to understand the present application more thoroughly and completely. The embodiments in the following description are for illustrative and descriptive purposes only and are not intended to be limiting. Without departing from the spirit and scope of the present application, other obvious variations can be conceived by those skilled in the art.

[0048] It should be understood that when a specific embodiment can be implemented differently, the specific process order can be executed differently from the described order. For example, two consecutively described processes can be executed substantially simultaneously or in an order opposite to the described order.

[0049] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this application, the singular forms "a" and "an" are also intended to include the plural forms unless the context clearly dictates otherwise. It will be further understood that when the terms "comprises," "comprising," "has," "having" are used in this specification, they specify the presence of the stated features, integers, steps, operations, operators, disparity maps, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, operators, disparity maps, elements, components, and / or combinations thereof. As used in this application, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms "first," "second," "third," etc. may be used in this application to describe various operators, disparity maps, elements, or components, these operators, disparity maps, elements, or components should not be limited by these terms. These terms are only used to distinguish one operator, disparity map, element, or component from another. Thus, without departing from the spirit and scope of this application, the first operator, first disparity map, first element, or first component discussed below may be referred to as the second operator, second disparity map, second element, or second component.

[0051] In this application, the description of the terms "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0052] The features of the examples described in this application can be combined in various ways that will be apparent after obtaining an understanding of the disclosure of this application. In addition, although the examples described in this application have various configurations, other configurations that will be apparent after understanding the disclosure of this application are also feasible.

[0053] The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0054] Figure 1 is a schematic flowchart showing a fast binocular method applicable to active obstacle avoidance of a mobile robot according to an embodiment of this application. Figure 2 is shown according to an embodiment of this applicationFigure 1 Schematic flowchart of step S110. Figure 3 It is a schematic flowchart showing Figure 2 of step S240 according to an embodiment of the present application. Figure 4 It is a schematic flowchart showing Figure 2 for generating a first disparity map using a disparity model according to an embodiment of the present application. Figure 5 It is a schematic flowchart showing Figure 1 of step S110 according to another embodiment.

[0055] As Figure 1 shown, the fast binocular method applicable to active obstacle avoidance of a mobile robot according to the present application includes the following steps.

[0056] Step S110: Generate a first disparity map and a depth and point cloud map corresponding to the first disparity map and having the same first resolution as the first disparity map based on an initial left image and an initial right image in a binocular camera. As is well known to those skilled in the art, the disparity map and the corresponding depth and point cloud map can be converted into each other through the following equation (1):

[0057]

[0058] where Z represents the depth of a pixel, d represents the disparity of a pixel, b represents the distance between the left camera and the right camera in the direction perpendicular to the optical axis, and f represents the focal lengths of the left camera and the right camera.

[0059] The initial left image may be the original image captured by the left camera in the binocular camera, and the initial right image may be the original right image captured by the right camera in the binocular camera at the same moment as the original image captured by the left camera. The initial left image and the initial right image may be obtained from the binocular camera in real time, or from a module or device communicatively connected to the binocular camera.

[0060] According to some embodiments of the present application, the first disparity map may be a disparity map directly converted from the aligned first left image and first right image after first converting the initial left image and the initial right image into the aligned first left image and first right image based on the internal parameters and external parameters of the binocular camera as in the prior art (the first method). According to other embodiments of the present application, the first disparity map may be a disparity map indirectly generated by first converting the initial left image and the initial right image into the aligned first left image and first right image based on the internal parameters and external parameters of the binocular camera, and then through pyramid layering and establishing a disparity model (the second method), which will be described below with reference to Figure 2A description will be given. In still other embodiments of the present application, the first disparity map may be a disparity map obtained by converting the disparity map obtained by the above first method or second method into a depth and point cloud map, removing some point clouds from the depth and point cloud map, and then generating a disparity map from the depth and point cloud map with some point clouds removed (the third method), which will be described below with reference to Figure 5 for description.

[0061] Step 120: Use a predetermined number of disparity grid gradient operators (e.g., 32) to generate a disparity grid gradient texture map (DGGMap) with a predetermined number of bits (e.g., 32 bits) from the first disparity map in units of a super grid composed of m rows * m columns of grids in the first disparity map, where each grid is composed of n rows * n columns of pixels, and each of m and n is an integer greater than or equal to 1 and m is an odd number.

[0062] In step 120, the first disparity map is divided into a plurality of super grids, each super grid is composed of m rows * m columns of grids, and each grid is composed of n rows * n columns of pixels, where each of m and n is an integer greater than or equal to 1 and m is an odd number. In other words, each grid can represent a pixel or a super pixel composed of multiple pixels. For example, m can be set to be greater than or equal to 5, and n can be set to be greater than or equal to 2. Preferably, m = 5 and n = 2 to improve the operation speed and reduce the cost while ensuring the determination accuracy.

[0063] For example, the disparity grid gradient texture map (DGGMap) according to the present application can be defined by the following equation (2):

[0064]

[0065] where D i = GM A - GM B , the subscripts A and B represent the A region and the B region in the super grid. Therefore, D i is the i-th operator used to represent the gray level change between the A region and the B region, and GM A and GM B are the average values of the gray levels of the pixels in two region sets in the gradient operator, and B i represents the weight of the i-th gradient operator.

[0066] In an exemplary embodiment of the present invention, the 32 disparity grid gradient operators adopted may include 8 predetermined point gradient operators (b4 to b11), 14 predetermined line gradient operators (b12 to b25), 6 predetermined surface gradient operators (b26 to b31), and 4 random gradient operators (b00 to b3). The bit positions corresponding to these operators are shown in Table 1 below, and will be referred to Figures 8A to 11DDescribe examples of these gradient operators.

[0067] Table 1

[0068] b31 … b26 b25 … b12 b11 … b4 b3 … b0

[0069] Step S130: Obtain the positions of the obstacle foreground and the ground from the disparity grid gradient texture map.

[0070] In step S130, a trained neural network model can be used to obtain the positions of the obstacle foreground and the ground from the disparity grid gradient texture map generated in step S120.

[0071] The trained neural network model can be obtained by training an existing neural network model with a large number of disparity grid gradient texture maps (for example, a large number of disparity grid texture maps obtained by performing the above steps S110 and S120 on a large number of initially acquired left and right images by a binocular camera stored previously).

[0072] Optionally, the trained neural network model can be obtained by training the Disparity Grid Ground Net (DGGNet) as shown in Figure 12 this application with a large number of disparity grid gradient texture maps (for example, a large number of disparity grid texture maps obtained by performing the above steps S110 and S120 on a large number of initially acquired left and right images by a binocular camera stored previously). Those skilled in the art can clearly understand the DGGNet adopted in this application from Figure 12 the schematic diagram of the DGGNet shown, so the specific description of each functional module in this network is omitted here.

[0073] Step S140: Based on the obtained positions of the obstacle foreground and the ground, determine the obstacle foreground and the ground that the mobile robot needs to actively avoid obstacles from the depth and point cloud map obtained in step S110.

[0074] In step S140, based on the obtained positions of the obstacle foreground and the ground, the obstacle foreground and the ground that the mobile robot needs to actively avoid obstacles can be determined from the depth and point cloud map obtained in step S110 based on the method of point cloud clustering.

[0075] Next, step S110 in Figure 2 will be exemplarily described with reference to Figure 1 .

[0076] Refer to Figure 2, in the present application, in order to optimize the speed while ensuring the accuracy, preferably, in Figure 1 step S110 of

[0077] , the initial left image and the initial right image are converted into an aligned first left image and a first right image (step S210); the first left image and the first right image with the original resolution are layered by using the pyramid image layering technique to obtain a first image pair (including the first left image and the first right image) with the original resolution (high resolution, hereinafter referred to as "the first resolution"), a second image pair (including a second left image and a second right image) with the medium resolution (hereinafter referred to as "the second resolution"), and a third image pair (including a third left image and a third right image) with the low resolution (hereinafter referred to as "the third resolution") (step S220); a second disparity map with the second resolution and a third disparity map with the third resolution are respectively obtained from the second image pair and the third image pair (step S230); a disparity model is established according to the second disparity map and the third disparity map (step S240); a first disparity map of the first image pair is generated by using the established disparity model (step S250); and the first disparity map is converted into a depth and point cloud map with the first resolution (step S260).

[0078] The geometric model will be described below.

[0079] For the left camera, the matrix transformation from the pixel coordinate system to the three-dimensional world coordinate system is as follows:

[0080]

[0081] where is the internal parameter matrix of the left camera, is the external parameter matrix of the left camera, and Z r is the scaling factor.

[0082] Similarly, for the right camera, the matrix transformation from the pixel coordinate system to the three-dimensional world coordinate system is as follows:

[0083]

[0084] Among them, is the internal parameter matrix of the right camera, is the external parameter matrix of the right camera, and Z′ r is the scaling factor.

[0085] According to an embodiment of the present application, based on this geometric model, distortion correction can be performed on the initial left image and the initial right image, and then the left camera coordinate system and the right camera coordinate system are multiplied by their respective rotation matrices R and R′, so that the principal optical axes of the left and right cameras are parallel, and the image planes are parallel to the baseline, thereby obtaining the aligned first left image and the first right image. In this way, only the matching points of the first left image and the first right image need to be searched on the same row to improve the search efficiency.

[0086] According to some embodiments of the present application, in step S230, based on the BM binocular matching algorithm, fast disparity calculation can be performed on the second image pair and the third image pair to accelerate the operation speed.

[0087] For acceleration, in some embodiments of the present application, a Census transform with a 5*5 neighborhood, for example, can be adopted for the third image pair with a low resolution (the third resolution):

[0088]

[0089]

[0090] Among them, (u, v) are the pixel coordinates of the central pixel in the considered 5*5 neighborhood (the image of the point (point of interest) in the three-dimensional world of interest on the imaging plane), and (u + i, v + j) are the pixel coordinates of the pixels adjacent to the central pixel.

[0091] Then, the matching cost of the pixel points of the third left image and the third right image in the third image pair is obtained through the hamming distance:

[0092] C(u, v, d): = Hamming(C sl (u, v), C sr (u - d, v))

[0093] Among them, d is the disparity between the left and right images of the point of interest. For a certain point of interest, the pixels in the left image and the pixels in the right image corresponding to the minimum matching cost are the pixels that match each other with respect to this feature point.

[0094] Therefore, in the present application, preferably, the Census transform and the Hamming distance can be utilized to obtain a matching cost to find matching pixels in the left and right images. For the mutually matching pixels, data with the minimum disparity is output in the disparity dimension direction to generate a disparity map (the third disparity map) of the third left image and the third right image. Similarly, a disparity map (the second disparity map) of the second left image and the second right image can be generated. It should be understood that this is only an exemplary illustration of obtaining the second disparity map with the second resolution and the third disparity map with the third resolution from the second image pair and the third image pair respectively (the above step S230), but it is not limited thereto.

[0095] The following will refer to Figure 3 to Figure 2 exemplarily illustrate step S240 in

[0096] According to some embodiments of the present application, since the second disparity map has a higher resolution than the third disparity map, the imaging of the edge information in the second disparity map will be clearer. Thus, step S240 can include Figure 3 steps S310 and S320 shown in Figure 3 to achieve higher precision while accelerating. Specifically, in step S310, the third disparity map is divided into multiple image patches, and the disparity range of each image patch is obtained; in step S320, the third disparity map is mapped to the second disparity map, and a disparity model is established in combination with the edge detection result of the second disparity map. According to some embodiments of the present application, mapping the third disparity map to the second disparity map includes mapping the disparity range, hole coordinates, and noise coordinates of the third disparity map to the second disparity map. It should be understood that

[0097] is only an exemplary illustration of establishing a disparity model based on the third disparity map and the second disparity map (the above step S240), but the specific process of step S240 is not limited thereto.

[0097] For ease of understanding, the following will refer to Figure 4 to Figure 2 exemplarily illustrate step S250 in

[0098] As Figure 4 shown in Figure 2 according to the exemplary embodiments of the present application,

[0099] step S250 in

[0100] In step S420, based on the disparity range in the established disparity model, the cost functions of the first left image and the first right image after the Census transform are determined. Specifically, the candidate disparity range of each image patch of the first left image and the first right image after the Census transform can be obtained based on the disparity range in the established disparity model, and the cost functions of the first left image and the first right image after the Census transform are determined in the manner of variable candidate disparity range per pixel for each image patch, thereby accelerating the operation speed.

[0101] In step S430, fast mean filtering is performed using the noise coordinates, hole coordinates, and edge coordinates in the established disparity model.

[0102] In step S440, for each pixel, pixel data with the minimum disparity is output from the disparity model after mean filtering based on the cost function to generate the first disparity map.

[0103] It should be understood that Figure 4 merely Figure 2 is a preferred process of step S250 in Figure 2 but the specific process of step S250 in

[0104] Figure 6A and Figure 6B are schematic diagrams for illustrating the fast mean filtering in step S430 of Figure 4

[0105] In the exemplary embodiment of the present application, fast mean filtering can be performed on H*W*D positions using positions such as noise, holes, and edges in the disparity model to ensure the balance between accuracy and performance, where H represents H rows on the pixel plane, W represents W columns on the pixel plane, and D represents the disparity in the disparity dimension.

[0106] Combined with Figure 6A and Figure 6B the initialization example of the fast mean filtering algorithm adopted in the present application is as follows:

[0107] V0_i = P00 + P10 + P20 + P30 + P40

[0108] V1_i = P01 + P11 + P21 + P31 + P41

[0109] V2_i = P02 + P12 + P22 + P32 + P42

[0110] V3_i = P03 + P13 + P23 + P33 + P43

[0111] V4_i = P04 + P14 + P24 + P34 + P44​

[0112] …

[0113] The Kernel example of the fast mean filtering algorithm adopted in this application is as follows:

[0114] V0 = V0_i + P40 - P00

[0115] V1 = V1_i + P41 - P01

[0116] V2 = V2_i + P42 - P02

[0117] V3 = V3_i + P43 - P03

[0118] V4 = V4_i + P44 - P04

[0119] …

[0120] SUM_1 = V0 + V1 + V2 + V3 + V4

[0121] SUM_2 = SUM_1 + V5 – V0

[0122] SUM_3 = SUM_2 + V6 – V1

[0123] …

[0124] Among them, P00 to P53… are pixel values, and V0_i to V4_i, etc. are the sums of pixel values in the vertical direction on the H*W plane.

[0125] Since the mean filtering is only related to the H*W plane and has no relation with the neighborhood in the disparity dimension D direction, in practical applications, SIMD parallel computing can be performed at every 8 positions in the disparity dimension direction under the Movidus platform to achieve the effect of accelerating with the hardware instruction set. The example of the SIMD instruction is shown in Table 2 below.

[0126] Table 2

[0127] Functional description C interface SIMD Assembly SIMD 8 short addition instruction __builtin_shave_vau_iadds_i16_rr VAU.IADDS.i16 8 short subtraction instruction __builtin_shave_vau_isubs_i16_rr VAU.ISUBS.i16 8 short shift instruction __builtin_shave_vau_shr_i16_ri VAU.SHR.i16

[0128] The following will refer to Figure 5 to describe the specific process of Figure 1 step S110 according to another embodiment of the present application.

[0129] As Figure 5 shown, generating the first disparity map and the depth and point cloud map corresponding to the first disparity map with the same first resolution as the first disparity map based on the initial left image and the initial right image in the binocular camera (the above step S110) may include step S510, step S520, and step S530.

[0130] In step S510, an initial disparity map of the initial left image and the initial right image is generated. This initial disparity map is a disparity map generated using the first left image and the first right image after aligning the initial left image and the initial right image. The disparity map generated in step S510 can be a disparity map generated from the initial left image and the initial right image using a method similar to that in steps S210 to S250. To avoid repetition, this process will not be described again.

[0131] In step S520, the initial disparity map is converted into an initial depth and point cloud map with a first resolution. Specifically, the initial disparity map can be converted into an initial depth and point cloud map using the above equation (1).

[0132] In step S530, data with a height greater than or equal to a predetermined height is removed from the initial depth and point cloud map to form Figure 1 the depth and point cloud map in, and a first disparity map is generated from the formed depth and point cloud map (for example, using the above equation (1)). For example, the predetermined height can be 0.2 meters.

[0133] It should be understood that step S110 is not limited to Figure 5 the specific process shown in, and those skilled in the art can also adopt Figure 2 the process shown in or combine Figure 2 and Figure 5 the process shown in to implement step S110.

[0134] The above describes an exemplary process for generating a first disparity map according to the disparity range in the disparity model. However, in practical applications, due to the low-speed movement of the mobile machine, the disparity range in the disparity model of this application is dynamically updated as the mobile robot moves at a low speed. Specifically, the disparity range in the disparity model is updated by using the disparity range within the neighborhood of the image patch corresponding to each position in the current frame and the previous frame.

[0135] The gradient operators used in the embodiments of this application will be described in detail below. To describe these gradient operators, the concept of a disparity grid will be introduced first.

[0136] Figure 7 is a schematic diagram for illustrating an example of a disparity grid according to an embodiment of this application.

[0137] As described in step S120, a disparity grid gradient texture map with a predetermined number of bits (for example, 32 bits) is generated from the first disparity map using a predetermined number of disparity grid gradient operators (for example, 32) with respect to a super grid composed of m rows × m columns of grids in the first disparity map. The super grid composed of m rows × m columns of grids in step S120 is also referred to as a disparity grid. An example of the disparity grid is Figure 7as shown in Figure 7 illustrates a super grid (parallax grid) composed of a 5-row * 5-column grid according to some embodiments of the present application. However, the size of the parallax grid of the present application is not limited thereto. In Figure 7 the example of, each of the grids G0 to G24 is composed of 2 rows * 2 columns of pixels (therefore, each of the grids G0 to G24 can be a super grid) to accelerate the operation speed. However, the size of the grid of the present application is not limited thereto.

[0138] The gradient operator adopted in the present application will be described based on the above super grid (parallax grid).

[0139] Figures 8A to 8H is a schematic diagram for illustrating an example of an 8-point gradient operator according to the present application.

[0140] According to some exemplary embodiments of the present application, the 8 predetermined point gradient operators may include a plurality of first-type point gradient operators and a plurality of second-type point gradient operators, wherein the first-type point gradient operator is used to represent the gray-scale change between the central grid at the central position in the super grid and the nearest neighbor grid, and the second-type point gradient operator is used to represent the gray-scale change between the central grid and the grid at the farthest distance.

[0141] For example, 4 first-type point gradient operators can be respectively used to represent: the gray-scale change between the central grid and the nearest neighbor left grid; the gray-scale change between the central grid and the nearest neighbor upper grid; the gray-scale change between the central grid and the nearest neighbor right grid; and the gray-scale change between the central grid and the nearest neighbor lower grid. 4 second-type point gradient operators can be respectively used to represent: the gray-scale change between the central grid and the grid at the first row and the first column; the gray-scale change between the central grid and the grid at the first row and the m-th column; the gray-scale change between the central grid and the grid at the m-th row and the m-th column; and the gray-scale change between the central grid and the grid at the m-th row and the first column.

[0142] According to still some exemplary embodiments of the present application, the 8 predetermined point gradient operators may include a plurality of first-type point gradient operators and a plurality of second-type point gradient operators, wherein the first-type point gradient operator can be used to represent the gray-scale change in the upward, downward, leftward, and rightward directions with respect to the grid at the central position in the super grid, and the second-type point gradient operator can be used to represent the gray-scale change in the upper left diagonal direction, upper right diagonal direction, lower right diagonal direction, and lower left diagonal direction with respect to the central grid.

[0143] In Figures 8A to 8DIn the example, the point gradient operators b4 to b7 of the first type are respectively used to represent the gray level change between the grid G12 (the grid in the 3rd row and 3rd column, the central grid) and the grid G11 (refer to Figure 8A ), the gray level change between the grid G12 and the grid G7 (refer to Figure 8B ), the gray level change between the grid G12 and the grid G13 (refer to Figure 8C ), and the gray level change between the grid G12 and the grid G17 (refer to Figure 8D ). It should be understood that this example of the point gradient operator of the first type is merely illustrative, and the point gradient operator of the first type is not limited thereto. For example, in some exemplary embodiments, the point gradient operator of the first type can also be respectively used to represent the gray level change between the grid G12 and the grid G2, the gray level change between the grid G12 and the grid G10, the gray level change between the grid G12 and the grid G22, and the gray level change between the grid G12 and the grid G14.

[0144] In Figures 8E to 8H 's example, the point gradient operators b8 to b11 of the second type are respectively used to represent the gray level change between the grid G12 and the grid G0 (refer to Figure 8E ), the gray level change between the grid G12 and the grid G4 (refer to Figure 8F ), the gray level change between the grid G12 and the grid G24 (refer to Figure 8G ), and the gray level change between the grid G12 and the grid G20 (refer to Figure 8H ). It should be understood that this example of the point gradient operator of the second type is merely illustrative, and the point gradient operator of the second type is not limited thereto. For example, in some exemplary embodiments, the point gradient operator of the second type can also be respectively used to represent the gray level change between the grid G12 and the grid G6, the gray level change between the grid G12 and the grid G8, the gray level change between the grid G12 and the grid G16, and the gray level change between the grid G12 and the grid G18.

[0145] Figures 9A to 9N is a schematic diagram for illustrating an example of 14 line gradient operators according to the present application.

[0146] According to some exemplary embodiments of the present application, the 14 predetermined line gradient operators include: a first type of line gradient operator for representing the gray-scale change between m grids on one row and m grids on another row in the super grid; a second type of line gradient operator for the gray-scale change between m grids on one column and m grids on another column in the super grid; a third type of line gradient operator for representing the gray-scale change between m grids on one row and m grids on one column in the super grid; and a fourth type of line gradient operator for representing the gray-scale change between m grids on one diagonal of the super grid and m grids on the other diagonal of the super grid.

[0147] According to some exemplary embodiments of the present application, the first type of line gradient operator can be respectively used to represent the gray-scale change between m grids on the first row and m grids on the second row in the super grid; the gray-scale change between m grids on the (m - 1) / 2-th row and m grids on the (m + 1) / 2-th row in the super grid; the gray-scale change between m grids on the (m + 1) / 2-th row and m grids on the (m + 3) / 2-th row in the super grid; and the gray-scale change between m grids on the (m - 1)-th row and m grids on the m-th row in the super grid. For example, in Figures 9A to 9D the example of, 4 first type of line gradient operators (b12 to b15) are respectively used to represent the gray-scale change between 5 grids on the first row and 5 grids on the second row (refer to Figure 9A ); the gray-scale change between 5 grids on the second row and 5 grids on the third row (refer to Figure 9B ); the gray-scale change between 5 grids on the third row and 5 grids on the fourth row (refer to Figure 9C ); the gray-scale change between 5 grids on the fourth row and 5 grids on the fifth row. It should be understood that this example of the first type of line gradient operator is merely illustrative, and the first type of line gradient operator is not limited thereto.

[0148] According to some exemplary embodiments of the present application, the second type of line gradient operator can be respectively used to represent: the gray-scale change between m grids on the first column and m grids on the second column in the super grid; the gray-scale change between m grids on the (m - 1) / 2-th column and m grids on the (m + 1) / 2-th column in the super grid; the gray-scale change between m grids on the (m + 1) / 2-th column and m grids on the (m + 3) / 2-th column in the super grid; and the gray-scale change between m grids on the (m - 1)-th column and m grids on the m-th column in the super grid. For example, in Figures 9E to 9HIn the example, four line gradient operators of the second type (b16 to b19) can be respectively used to represent the gray-scale change between the five grids on the first column and the five grids on the second column (refer to Figure 9E ), the gray-scale change between the five grids on the second column and the five grids on the third column (refer to Figure 9F ), the gray-scale change between the five grids on the third column and the five grids on the fourth column (refer to Figure 9G ), and the gray-scale change between the five grids on the fourth column and the five grids on the fifth column (refer to Figure 9H ). It should be understood that this example of the line gradient operator of the second type is merely illustrative, and the line gradient operator of the second type is not limited thereto.

[0149] According to some exemplary embodiments of the present application, the line gradient operators of the third type can be respectively used to represent: the gray-scale change between the m grids on the first column and the m grids on the first row in the super grid (refer to Figure 9F ); the gray-scale change between the m grids on the first row and the m grids on the m-th column in the super grid; the gray-scale change between the m grids on the m-th column and the m grids on the m-th row in the super grid; the gray-scale change between the m grids on the m-th row and the m grids on the first column in the super grid; and the gray-scale change between the m grids on the (m + 1) / 2-th row and the m grids on the (m + 1) / 2-th column in the super grid. For example, in the example of FIG. 9, five line gradient operators of the third type (b20 to b24) can be respectively used to represent the gray-scale change between the five grids on the first column and the five grids on the first row (refer to Figure 9I ), the gray-scale change between the five grids on the first row and the five grids on the fifth column (refer to Figure 9J ), the gray-scale change between the five grids on the fifth column and the five grids on the fifth row (refer to Figure 9K ), the gray-scale change between the five grids on the fifth row and the five grids on the first column (refer to Figure 9L ), and the gray-scale change between the five grids on the third row and the five grids on the third column (refer to Figure 9M ). It should be understood that this example of the line gradient operator of the third type is merely illustrative, and the line gradient operator of the third type is not limited thereto.

[0150] As described above, according to some exemplary embodiments of the present application, the line gradient operator of the fourth type can be used to represent the gray-scale change between the m grids on the diagonal in the super grid and the m grids on the other diagonal in the super grid. For example, in Figure 9NIn the example, the fourth type of line gradient operator (b25) can be used to represent the gray-scale change between the grids G0, G6, G12, G18, and G24 on the first diagonal and the grids G4, G8, G12, G16, and G20 on the second diagonal (refer to Figure 9N ). It should be understood that this example of the fourth type of line gradient operator is merely illustrative, and the fourth type of line gradient operator is not limited thereto.

[0151] Figures 10A to 10F FIG. is a schematic diagram for illustrating an example of six surface gradient operators according to the present application.

[0152] According to some exemplary embodiments of the present application, the six predetermined surface gradient operators may include: a first type of surface gradient operator for representing the gray-scale change between at least two adjacent rows of grids and at least two other adjacent rows of grids in a super-grid; a second type of surface gradient operator for representing the gray-scale change between at least two adjacent columns of grids and at least two other adjacent columns of grids in a super-grid; a third type of surface gradient operator for representing the gray-scale change between all grids on a diagonal and on one side of the diagonal and all grids on the diagonal and on the other side of the diagonal in a super-grid; a fourth type of surface gradient operator for representing the gray-scale change between all grids at the edge and q*q grids at the central position in a super-grid, where q is an odd number greater than or equal to 3; and a fifth type of surface gradient operator for representing the gray-scale change between two sets of k*k grids symmetric with respect to the central grid along the first diagonal direction and two sets of k*k grids symmetric with respect to the central grid along the second diagonal direction in a super-grid, where k is an integer greater than or equal to 2.

[0153] According to some exemplary embodiments of the present application, the first type of surface gradient operator can be used to represent the gray-scale change between a rectangular region composed of m grids on the first row and m grids on the second row in a super-grid and a rectangular region composed of m grids on the (m - 1)th row and m grids on the mth row. For example, in Figure 10A 's example, the first type of surface gradient operator (b26) can be used to represent the gray-scale change between 10 grids on the first row and the second row and 10 grids on the fourth row and the fourth column. It should be understood that this example of the first type of surface gradient operator is merely illustrative, and the first type of surface gradient operator is not limited thereto.

[0154] According to some exemplary embodiments of the present application, the second type of surface gradient operator can be respectively used to represent the gray-scale change between a rectangular region composed of m grids on the first column and m grids on the second column in the super-grid and a rectangular region composed of m grids on the (m - 1)th column and m grids on the mth column. For example, in Figure 10B 's example, the second type of surface gradient operator (b27) can be used to represent the gray-scale change between 10 grids on the first and second columns and 10 grids on the fourth and fifth columns. It should be understood that this example of the second type of surface gradient operator is merely illustrative, and the second type of surface gradient operator is not limited thereto.

[0155] According to some exemplary embodiments of the present application, the third type of surface gradient operator can be respectively used to represent: the gray-scale change between a triangular region composed of grids on the first column, the first row, and the first diagonal and between them in the super-grid and a triangular region composed of grids on the first diagonal, the mth row, and the mth column and between them; and the gray-scale change between a triangular region composed of grids on the first row, the mth column, and the second diagonal and between them in the super-grid and a triangular region composed of grids on the second diagonal, the first column, and the mth row and between them. For example, in Figure 10C and Figure 10D 's example, the third type of surface gradient operator (b28 and b29) can be respectively used to represent the gray-scale change between grids G0 to G8, G10 to G12, G15, G16, and G20 and grids G4, G8, G9, G12 to G14, and G16 to G24 (refer to Figure 10C ); and the gray-scale change between grids G0 to G4, G6 to G9, G12 to G14, G18, G19, and G24 and grids G0, G5, G6, G10 to G12, G15 to G18, and G20 to G24 (refer to Figure 10D ). It should be understood that this example of the third type of surface gradient operator is merely illustrative, and the third type of surface gradient operator is not limited thereto.

[0156] According to some exemplary embodiments of the present application, the fourth type of surface gradient operator can be used to represent: the gray-scale change between all grids on the first row, the first column, the mth row, and the mth column in the super-grid and q*q grids at the central position, where q is an odd number greater than or equal to 3. For example, in Figure 10EIn the example, the fourth type of face gradient operator (b30) can be used to represent the gray-scale change between the meshes G0 to G4, G5, G9, G10, G14, G15, G19 and G20 to G24 and the meshes G6 to G8, G11 to G13 and G16 to G18. It should be understood that this example of the fourth type of face gradient operator is merely illustrative, and the fourth type of face gradient operator is not limited thereto.

[0157] According to some exemplary embodiments of the present application, the fifth type of face gradient operator can be used to represent: the region composed of the meshes in the first row and first column, the meshes in the first row and second column, the meshes in the second row and first column, the meshes in the second row and second column, the meshes in the (m - 1)th row and (m - 1)th column, the meshes in the (m - 1)th row and mth column, the meshes in the mth row and (m - 1)th column, and the meshes in the mth row and mth column in the super mesh and the region composed of the meshes in the (m - 1)th row and first column, the meshes in the (m - 1)th row and second column, the meshes in the mth row and first column, the meshes in the mth row and second column, the meshes in the first row and (m - 1)th column, the meshes in the first row and mth column, the meshes in the second row and (m - 1)th column, and the meshes in the second row and mth column. For example, in Figure 10F the example, the fifth type of face gradient operator (b31) can be used to represent the gray-scale change between the meshes G0, G1, G5, G6, G18, G19, G23 and G24 and the meshes G3, G4, G8, G9, G15, G16, G20 and G21. It should be understood that this example of the fifth type of face gradient operator is merely illustrative, and the fifth type of face gradient operator is not limited thereto.

[0158] Figures 11A to 11D is a schematic diagram for illustrating an example of the random gradient operator according to the present application.

[0159] The random gradient operator can be any gradient operator based on the above superlattice (disparity grid), which can be used to represent the gray-scale change between any two meshes or any multiple meshes. For example, in Figures 11A to 11D the example, 4 random gradient operators (b0 to b3) can be respectively used to represent: the gray-scale change between the meshes G5, G6, G11 and G16 and the meshes G13, G14 and G17 (refer to Figure 11A ); the gray-scale change between the meshes G11, G12 and G18 and the meshes G20 to G22 (refer to Figure 11B ); the gray-scale change between the meshes G12, G16, G18 and G20 and the meshes G3, G6 to G8 (refer to Figure 11C);The gray-scale change between grids G6, G12, G13, G16, and G21 and grids G4, G9, G14, G19, G23, and G24 (reference Figure 11D ). It should be understood that this example of the stochastic gradient operator is merely illustrative and is not limited thereto.

[0160] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0161] The electronic device includes a processor and a memory. The memory is connected to the processor and stores instructions that, when executed, cause the processor to perform the methods described above.

[0162] The methods described above can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0164] In the context of the present disclosure, a computer-readable medium may be a tangible medium that can contain or store a program or instructions for executing the above-described method. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] The above specific embodiments do not constitute a limitation on the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A fast binocular method applicable to active obstacle avoidance of a mobile robot, the method comprising: S1) Generating a first disparity map and a depth and point cloud map corresponding to the first disparity map and having the same first resolution as the first disparity map based on an initial left image and an initial right image in a binocular camera, wherein the depth and point cloud map is converted based on the first disparity map, and the depth of the pixels of the depth and point cloud map is inversely proportional to the disparity of the pixels of the first disparity map; S2) Use a predetermined number of parallax grid gradient operators to generate a parallax grid gradient texture map with a predetermined number of bits from the first parallax map in units of super-grids composed of m rows and m columns of grids in the first parallax map, where the parallax grid gradient operator is the gray-scale change between multiple grids in the super-grid; ​ Wherein, the disparity grid gradient texture map is expressed as: ； Among them, , the subscripts A and B represent the A region and the B region in the super grid, so is the i-th operator for representing the gray-scale change between the A region and the B region, while and are the average values of the gray-scale sums of the pixels in the two region sets in the gradient operator, represents the weight of the i-th gradient operator, and N represents the said predetermined quantity; S3) Obtaining the positions of obstacle foreground objects and the ground from the disparity grid gradient texture map; S4) Based on the obtained positions of the obstacle foreground objects and the ground, determining the obstacle foreground objects and the ground for which the mobile robot needs to actively avoid obstacles from the depth and point cloud map, where each of the grids consists of n rows and n columns of pixels, where each of m and n is an integer greater than or equal to 1 and m is odd.

2. The method according to claim 1, wherein, The step S1) includes: Converting the initial left image and the initial right image into aligned first left image and first right image; Performing downsampling in sequence with the first left image and the first right image as the bottom layer of the pyramid, to obtain a first image pair composed of a first left image and a first right image having the first resolution, a second image pair composed of a second left image and a second right image having a second resolution lower than the first resolution, and a third image pair composed of a third left image and a third right image having a third resolution lower than the second resolution; Obtaining a second disparity map having the second resolution and a third disparity map having the third resolution from the second image pair and the third image pair respectively; Establishing a disparity model according to the third disparity map and the second disparity map; Generating the first disparity map of the first image pair by using the disparity model; and Generating the depth and point cloud map having the same first resolution as the first disparity map from the first disparity map.

3. The method according to claim 2, wherein, Obtaining the second disparity map having the second resolution and the third disparity map having the third resolution from the second image pair and the third image pair respectively is performed by using the BM binocular matching algorithm.

4. The method according to claim 2, wherein Establishing a disparity model according to the third disparity map and the second disparity map includes: Dividing the third disparity map into a plurality of image blocks, and obtaining the disparity range of each image block; and Mapping the third disparity map to the second disparity map, and establishing the disparity model in combination with the edge detection result of the second disparity map.

5. The method according to claim 4, wherein Mapping the third disparity map to the second disparity map includes: mapping the disparity range, hole coordinates and noise coordinates of the third disparity map to the second disparity map.

6. The method according to claim 2, wherein, Generating the first disparity map of the first image pair by using the disparity model includes: Performing Census transformation on the first left image and the first right image of the first image pair; Determining the cost function of the first left image and the first right image after Census transformation based on the disparity range in the disparity model; Performing fast mean filtering by using the noise coordinates, hole coordinates and edge coordinates in the disparity model; For each pixel, based on the cost function, pixel data with the minimum disparity is output from the mean-filtered disparity model to generate the first disparity map.

7. The method according to claim 2, wherein The disparity range in the disparity model is dynamically updated as the mobile robot moves at a low speed.

8. The method according to claim 7, wherein The dynamic update of the disparity range in the disparity model as the mobile robot moves at a low speed includes: updating the disparity range in the disparity model by using the disparity range within the image block neighborhood corresponding to each position in the current frame of the previous frame.

9. The method according to claim 1, wherein The step S1) includes: Generating an initial disparity map for the initial left image and the initial right image; Converting the initial disparity map into an initial depth and point cloud map with the same first resolution as the initial disparity map; and Removing data with a height greater than or equal to a predetermined height from the initial depth and point cloud map to form the depth and point cloud map, and generating the first disparity map from the depth and point cloud map from which the data has been removed.

10. The method according to claim 9, wherein, The predetermined height is 0.2 meters.

11. The method according to any one of claims 1 to 10, wherein The predetermined number is 32, and the predetermined number of bits is 32.

12. The method according to claim 11, wherein, The 32 disparity grid gradient operators include 8 predetermined point gradient operators, 14 predetermined line gradient operators, 6 predetermined surface gradient operators, and 4 random gradient operators.

13. The method according to claim 12, wherein, The 8 predetermined point gradient operators include: A first type of point gradient operator for representing the gray-level change between the central grid at the central position in the super grid and the nearest neighbor grid; and A second type of point gradient operator for representing the gray-level change between the central grid and the grid at the farthest distance.

14. The method according to claim 13, wherein, The first type of point gradient operator is respectively used to represent: The gray-level change between the central grid and the nearest neighbor left grid; The gray-level change between the central grid and the nearest neighbor upper grid; The gray-level change between the central grid and the nearest neighbor right grid; And The gray-level change between the central grid and the nearest neighbor lower grid.

15. The method according to claim 13, wherein, The second type of point gradient operator is respectively used to represent: The gray-level change between the central grid and the grid at the first row and first column; The gray-level change between the central grid and the grid at the first row and the m-th column; The gray-level change between the central grid and the grid at the m-th row and m-th column; And The gray-level change between the central grid and the grid at the m-th row and first column.

16. The method according to claim 12, wherein The 8 predetermined point gradient operators include: A first type of point gradient operator for representing the gray-level change in the up direction, down direction, left direction, and right direction with respect to the central grid at the central position in the super grid; and A second type of point gradient operator for representing the gray-level change in the upper left diagonal direction, upper right diagonal direction, lower right diagonal direction, and lower left diagonal direction with respect to the central grid.

17. The method according to claim 12, wherein, The 14 predetermined line gradient operators include: A first type of line gradient operator for representing the gray-level change between m grids in one row and m grids in another row in the super grid; A second type of line gradient operator for the gray-scale change between m grids in one column and m grids in another column in the super-grid; A third type of line gradient operator for representing the gray-scale change between m grids in one row and m grids in one column in the super-grid; and A fourth type of line gradient operator for representing the gray-scale change between m grids on one diagonal and m grids on the other diagonal in the super-grid.

18. The method according to claim 17, wherein, The first type of line gradient operator is respectively used to represent: The gray-scale change between m grids in the first row and m grids in the second row in the super-grid; The gray-scale change between m grids in the (m - 1) / 2-th row and m grids in the (m + 1) / 2-th row in the super-grid; The gray-scale change between m grids in the (m + 1) / 2-th row and m grids in the (m + 3) / 2-th row in the super-grid; And The gray-scale change between m grids in the (m - 1)-th row and m grids in the m-th row in the super-grid.

19. The method according to claim 17, wherein, The second type of line gradient operator is respectively used to represent: The gray-scale change between m grids in the first column and m grids in the second column in the super-grid; The gray-scale change between m grids in the (m - 1) / 2-th column and m grids in the (m + 1) / 2-th column in the super-grid; The gray-scale change between m grids in the (m + 1) / 2-th column and m grids in the (m + 3) / 2-th column in the super-grid; And The gray-scale change between m grids in the (m - 1)-th column and m grids in the m-th column in the super-grid.

20. The method according to claim 17, wherein, The third type of line gradient operator is respectively used to represent: The gray-scale change between m grids in the first column and m grids in the first row in the super-grid; The gray-scale change between m grids in the first row and m grids in the m-th column in the super-grid; The gray-scale change between m grids in the m-th column and m grids in the m-th row in the super-grid; The gray-scale change between m grids in the (m + 1) / 2-th row and m grids in the (m + 1) / 2-th column in the super-grid; And The gray-scale change between m grids in the m-th row and m grids in the first column in the super-grid.

21. The method according to claim 12, wherein The 6 predetermined surface gradient operators include: A first type of surface gradient operator for representing the gray-scale change between at least two adjacent rows of grids and at least two other adjacent rows of grids in the super-grid; A second type of surface gradient operator for representing the gray-scale change between at least two adjacent columns of grids and at least two other adjacent columns of grids in the super-grid; A third type of surface gradient operator for representing the gray-scale change between all grids on the diagonal and on one side of the diagonal and all grids on the diagonal and on the other side of the diagonal in the super-grid; The fourth type of surface gradient operator is used to represent the gray-scale change between all the grids located at the edges in the super-grid and the q grids located at the central position, where q is an odd number greater than or equal to 3; and The fifth type of surface gradient operator is used to represent two sets of k grids that are symmetric with respect to the central grid along the first diagonal direction in the supergrid and two sets of k grids that are symmetric with respect to the central grid along the second diagonal direction in the gray-level change between the k grids, where k is an integer greater than or equal to 2.

22. The method according to claim 21, wherein, The first type of surface gradient operator is used to represent the gray-scale change between a rectangular region composed of m grids on the first row and m grids on the second row in the super-grid and a rectangular region composed of m grids on the (m - 1)-th row and m grids on the m-th row.

23. The method according to claim 21, wherein The second type of surface gradient operator is used to represent: The gray-scale change between a rectangular region composed of m grids on the first column and m grids on the second column in the super-grid and a rectangular region composed of m grids on the (m - 1)-th column and m grids on the m-th column.

24. The method according to claim 21, wherein, The third type of surface gradient operator is respectively used to represent: The gray-scale change between a triangular region composed of grids on the first column, the first row, and the first diagonal line and between them in the super-grid and a triangular region composed of grids on the first diagonal line, the m-th row, and the m-th column and between them; And The gray-scale change between a triangular region composed of grids on the first row, the m-th column, and the second diagonal line and between them in the super-grid and a triangular region composed of grids on the second diagonal line, the first column, and the m-th row and between them.

25. The method according to claim 21, wherein The fourth type of surface gradient operator is used to represent: The gray-scale change between all the grids in the first row, the first column, the m-th row, and the m-th column of the super-grid and the q q grids at the central position.

26. The method according to claim 21, wherein, The fifth type of surface gradient operator is used to represent: The gray-scale change between a region composed of the grid on the first row and first column, the grid on the first row and second column, the grid on the second row and first column, the grid on the second row and second column, the grid on the (m - 1)-th row and (m - 1)-th column, the grid on the (m - 1)-th row and m-th column, the grid on the m-th row and (m - 1)-th column, and the grid on the m-th row and m-th column in the super-grid and a region composed of the grid on the (m - 1)-th row and first column, the grid on the (m - 1)-th row and second column, the grid on the m-th row and first column, the grid on the m-th row and second column, the grid on the first row and (m - 1)-th column, the grid on the first row and m-th column, the grid on the second row and (m - 1)-th column, and the grid on the second row and m-th column.

27. The method according to any one of claims 1 to 10, wherein m is greater than or equal to 5.

28. The method according to any one of claims 1 to 10, wherein n is greater than or equal to 2.

29. The method according to any one of claims 1 to 10, wherein Obtaining the positions of the obstacle foreground and the ground from the disparity grid gradient texture map is implemented by using a predetermined convolutional neural network model.

30. An electronic device, the electronic device includes: A processor, A memory connected to the processor, and instructions are stored in the memory, and when the instructions are executed, the processor is caused to execute the method according to any one of claims 1 - 29.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 - 29.

Images