Map construction method, device, equipment and storage medium

Abstract

The present invention provides a map construction method, device, equipment and storage medium, which relate to the field of artificial intelligence, and are used to improve the robustness of robot positioning and eliminate the influence of dynamic objects on robot positioning and mapping in dynamic scenes, including: a map construction device determines the dynamic feature points in the original binocular image collected by the robot, and determines the dynamic target object in the original binocular image according to the dynamic feature points of the original binocular image. Furthermore, the map construction device deletes the dynamic target object from the original binocular image to obtain a static binocular image, and determines the motion state of the robot based on the static binocular image, wherein the motion state includes position, posture and speed. In addition, the map construction device also performs depth estimation on the original binocular image to determine the depth information of each pixel in the original binocular image. Finally, the map construction device constructs a map according to the motion state and the depth information of each pixel in the original binocular image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a map construction method, device, equipment and storage medium. Background Art

[0002] In recent years, some robots for indoor and outdoor services have gradually appeared in daily life. For example, sweeping robots can automatically and accurately locate themselves and complete room cleaning through efficient path planning; security inspection robots can automatically patrol and conduct safety inspections in squares where humans and machines coexist, providing safety guarantees for people's travel; and unmanned vehicles can realize automatic driving functions. Behind the booming development of these related fields, there is an important technology: simultaneous localization and mapping (SLAM). The so-called SLAM is to integrate robot positioning with environmental map creation, that is, the robot builds an incremental environmental map based on its own posture estimation and sensor perception of the environment during movement, and uses the map to achieve its own positioning.

[0003] Most of the current SLAM technologies are based on an important assumption: the environment is static. However, the real environment is dynamic, such as pedestrians moving in the field of view, the open and closed state of doors, and the different positions of movable objects. In a dynamic environment, moving objects will bring great noise and errors to positioning and map construction. Summary of the invention

[0004] The present invention provides a map construction method, device, equipment and storage medium, which improve the robustness of robot positioning and eliminate the influence of dynamic objects on robot positioning and mapping in dynamic scenes.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, a map construction method is provided, the method comprising: a map construction device determines dynamic feature points in an original binocular image collected by a robot, and determines a dynamic target object in the original binocular image based on the dynamic feature points of the original binocular image. Furthermore, the map construction device deletes the dynamic target object from the original binocular image to obtain a static binocular image, and determines the motion state of the robot based on the static binocular image, wherein the motion state includes position, posture, and speed. Furthermore, the map construction device also performs depth estimation on the original binocular image to determine the depth information of each pixel in the original binocular image. Finally, the map construction device constructs a map based on the motion state and the depth information of each pixel in the original binocular image.

[0007] In the map construction method provided by the present invention, before determining the state of the robot, the map construction device first determines the dynamic target object in the current scene corresponding to the original binocular image, and after determining the dynamic target object, deletes the dynamic target object in the original binocular image to obtain a static binocular image. In this way, the robot can determine the motion state of the robot based on the static binocular image, which improves the robustness of the robot positioning and eliminates the influence of dynamic objects on the robot positioning and mapping in dynamic scenes.

[0008] In a possible design, the map construction device constructs a map according to the motion state and the depth information of each pixel in the original binocular image, including: the map construction device determines the position of the object in the original binocular image according to the motion state and the depth information of each pixel in the original binocular image. Further, the map construction device generates an occupied grid map corresponding to the original binocular image according to the position of the object in the original binocular image, and the state of the voxel corresponding to the position of the object in the original binocular image is the occupied state. The map construction device determines the dynamic object in the original binocular image, updates the state of the voxel corresponding to the position of the dynamic object to the idle state, and determines the point cloud data of the dynamic object to generate a three-dimensional occupied grid. Finally, the map construction device constructs a map according to the state of the voxel and the three-dimensional occupied grid. In this design, the map construction device distinguishes between dynamic objects and static objects to construct a map, which can better plan the robot's running route, so that the robot runs close to static objects and away from dynamic objects, and avoids being disturbed by dynamic objects during operation.

[0009] In a possible design, the map construction device determines the dynamic feature points in the original binocular image collected by the robot, including: the map construction device determines the first feature point in the original binocular image, and the second feature point in the adjacent binocular image, wherein the adjacent binocular image is a binocular image of an adjacent frame to the original binocular image, and the first feature point and the second feature point are used to represent the same content. Further, the map construction device determines the basic matrix according to the first feature point and the second feature point, and determines the polar line corresponding to the first feature point according to the basic matrix and the first feature point. Finally, the map construction device determines the target distance from the second feature point to the polar line, and when the target distance is greater than the first threshold, determines the first feature point as a dynamic feature point. In this design, the map construction device determines the dynamic feature points in the binocular image, so that the dynamic feature points determined in the subsequent process can determine the dynamic target object.

[0010] In a possible design, the map construction device determines the dynamic target object in the original binocular image based on the dynamic feature points of the original binocular image, including: the map construction device performs semantic segmentation on the original binocular image to determine the boundary points of the first object in the original binocular image, the first object being any object in the original binocular image; and when the number of dynamic feature points included in the boundary points of the first object is greater than a second threshold, the first object is determined to be the dynamic target object in the binocular image. This design implements that after determining the dynamic feature points, the map construction device determines the dynamic target object based on the dynamic feature points.

[0011] In a second aspect, a map construction device is provided, comprising a determination unit and a processing unit. The determination unit is used to determine the dynamic feature points in the original binocular image collected by the robot. The determination unit is also used to determine the dynamic target object in the original binocular image based on the dynamic feature points of the original binocular image. The processing unit is used to delete the dynamic target object from the original binocular image to obtain a static binocular image. The determination unit is also used to determine the motion state of the robot based on the static binocular image, and the motion state includes position, posture and speed. The determination unit is also used to perform depth estimation on the original binocular image and determine the depth information of each pixel in the original binocular image. The processing unit is also used to construct a map based on the motion state and the depth information of each pixel in the original binocular image.

[0012] In one possible design, the processing unit is specifically used to determine the position of the object in the original binocular image according to the motion state and the depth information of each pixel in the original binocular image. According to the position of the object in the original binocular image, an occupancy grid map corresponding to the original binocular image is generated, and the state of the voxel corresponding to the position of the object in the original binocular image is the occupied state. The dynamic object in the original binocular image is determined, and the state of the voxel corresponding to the position of the dynamic object is updated to the idle state. The point cloud data of the dynamic object is determined, and a three-dimensional occupancy grid is generated. The map is constructed according to the state of the voxel and the three-dimensional occupancy grid.

[0013] In a possible design, the determination unit is specifically used to determine a first feature point in an original binocular image and a second feature point in an adjacent binocular image; the adjacent binocular image is a binocular image of an adjacent frame to the original binocular image, and the first feature point and the second feature point are used to represent the same content. A basic matrix is ​​determined based on the first feature point and the second feature point. Based on the basic matrix and the first feature point, an epipolar line corresponding to the first feature point is determined. A target distance from the second feature point to the epipolar line is determined. When the target distance is greater than a first threshold, the first feature point is determined to be a dynamic feature point.

[0014] In one possible design, the determination unit is specifically used to perform semantic segmentation on the original binocular image, determine the boundary points of a first object in the original binocular image, and the first object is any object in the original binocular image. When the number of dynamic feature points included in the boundary points of the first object is greater than a second threshold, the first object is determined to be a dynamic target object in the binocular image.

[0015] In a third aspect, a map construction device is provided, which includes a memory and a processor; the memory and the processor are coupled, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the map construction device executes the map construction method in the first aspect.

[0016] In a fourth aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed on a map building device, the map building device executes the map building method in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A structural schematic diagram of a map construction system provided by an embodiment of the present invention;

[0018] Figure 2 A flowchart of a map construction method provided by an embodiment of the present invention Figure 1 ;

[0019] Figure 3 A flowchart of a map construction method provided by an embodiment of the present invention Figure 2 ;

[0020] Figure 4 A schematic diagram of a depth estimation process provided by an embodiment of the present invention;

[0021] Figure 5a A schematic diagram of a MADnet network structure provided by an embodiment of the present invention Figure 1 ;

[0022] Figure 5b A schematic diagram of a MADnet network structure provided by an embodiment of the present invention Figure 2 ;

[0023] Figure 5c A schematic diagram of a MADnet network structure provided by an embodiment of the present invention Figure 3 ;

[0024] Figure 6 A flowchart of a map construction method provided by an embodiment of the present invention Figure 3 ;

[0025] Figure 7 A schematic diagram of a system architecture for establishing a Euclidean signed distance functions (ESDF) map provided by an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of input and output of a Mobilenet-SSD model provided by an embodiment of the present invention;

[0027] Fig. 9 A schematic diagram of a map constructed by a map construction device provided in an embodiment of the present invention;

[0028] Fig.10 A flowchart of a map construction method provided by an embodiment of the present invention Figure 4 ;

[0029] Fig.11 A schematic diagram of an optical flow pyramid tracking feature point provided by an embodiment of the present invention Figure 1 ;

[0030] Fig.12 A schematic diagram of an optical flow pyramid tracking feature point provided by an embodiment of the present invention Figure 2 ;

[0031] Fig.13 A schematic diagram of determining polar lines by a map construction device provided by an embodiment of the present invention;

[0032] Fig.14 Schematic diagram 5 of a map construction method provided by an embodiment of the present invention;

[0033] Fig.15 A schematic diagram of the architecture of a SegNet deep network model provided by an embodiment of the present invention;

[0034] Fig.16 A schematic diagram of an image processed by a SegNet deep network model provided by an embodiment of the present invention;

[0035] Fig.17 A schematic diagram of the structure of a map construction device provided by an embodiment of the present invention;

[0036] Fig.18 A schematic diagram of a map construction device provided in an embodiment of the present invention Figure 1 ;

[0037] Fig.19 A schematic diagram of a map construction device provided in an embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0040] In the description of the present invention, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" and "plurality" refer to two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0041] At present, robots used for indoor and outdoor services are gradually appearing in daily life. The booming development of these related fields is inseparable from SLAM, which integrates robot positioning with environmental map creation. That is, the robot builds an incremental environmental map based on its own posture estimation and sensor perception of the environment during movement, and uses the map to achieve its own positioning.

[0042] In the existing technology, most SLAM technologies are based on an important assumption: the environment is static. However, the real environment is dynamic, such as pedestrians moving in the field of view, the opening and closing status of doors, and the different positions of movable objects. In a dynamic environment, moving objects will bring great noise and errors to positioning and map construction.

[0043] In order to solve the above problems, the present invention provides a map construction method, device, equipment and storage medium. The map construction device determines the dynamic feature points in the original binocular image collected by the robot, and determines the dynamic target object in the original binocular image according to the dynamic feature points of the original binocular image. Further, the map construction device deletes the dynamic target object from the original binocular image to obtain a static binocular image, and determines the motion state of the robot based on the static binocular image. Wherein, the motion state includes position, posture and speed. In addition, the map construction device also performs depth estimation on the original binocular image to determine the depth information of each pixel in the original binocular image. Finally, the map construction device constructs a map according to the motion state and the depth information of each pixel in the original binocular image. In the map construction method provided by the present invention, the map construction device first determines the dynamic target object in the current scene corresponding to the original binocular image before determining the state of the robot, and after determining the dynamic target object, deletes the dynamic target object in the original binocular image to obtain a static binocular image, so that the robot can determine the motion state of the robot based on the static binocular image, improves the robustness of robot positioning, and eliminates the influence of dynamic objects on robot positioning and mapping in dynamic scenes.

[0044] The map construction method provided in the embodiment of the present invention may be applicable to a map construction system, where the map construction system is used to obtain binocular images collected by a robot and to construct a map based on the collected binocular images. Figure 1 A structural diagram of the map construction system is shown in FIG. Figure 1 As shown, the map construction system 10 includes a map construction device 11 and a robot 12 .

[0045] The map construction device 11 is connected to the robot 12. The map construction device 11 and the robot 12 may be connected in a wired manner or in a wireless manner, which is not limited in the embodiment of the present invention.

[0046] The map building device 11 can be used to obtain the binocular images collected by the robot 12 and determine the dynamic feature points in the binocular images.

[0047] The map building device 11 may also be used to determine dynamic target objects in the binocular images based on dynamic feature points in the binocular images.

[0048] The map building device 11 can also be used to delete the dynamic target objects in the binocular image after determining the dynamic target objects in the binocular image to obtain a static binocular image including only static objects, thereby determining the motion state of the robot based on the static binocular image.

[0049] Among them, the robot's motion state includes position, posture and speed.

[0050] The map building device 11 may also be used to perform depth estimation on the binocular image to determine the depth information of each pixel in the binocular image.

[0051] The map construction device 11 can also be used to construct a map according to the determined motion state of the robot and the depth information of each pixel point in the binocular image.

[0052] The robot 12 includes a binocular camera, which is used to collect binocular images.

[0053] The robot 12 may be configured to send the binocular images to the map building device after the binocular images are captured by the binocular camera.

[0054] In some embodiments, the map construction device 11 and the robot 12 may be independent devices or integrated into the same device, which is not specifically limited in the embodiment of the present invention.

[0055] When the map construction device 11 and the robot 12 are integrated into the same device, the data transmission mode between the map construction device 11 and the robot 12 is the data transmission between the internal modules of the device. In this case, the data transmission process between the two is the same as the "data transmission process between the two when the map construction device 11 and the robot 12 are independent of each other".

[0056] In the following embodiments provided by the embodiments of the present invention, the embodiments of the present invention are described by taking the map construction device 11 and the robot 12 as being independently configured.

[0057] Figure 2 is a flowchart of a map construction method according to some exemplary embodiments. In some embodiments, the map construction method can be applied to Figure 1 The map construction device 11 shown can also be applied to a device in which the map construction device 11 is integrated with a robot 12, or other similar devices. In the following, the map construction method is described by taking the application of the map construction method to the map construction device as an example.

[0058] like Figure 2 As shown, the map construction method provided by the embodiment of the present invention is applied to the map construction device 11 in the above-mentioned map construction system 10, including S201-S206.

[0059] S201. A map building device determines dynamic feature points in an original binocular image collected by a robot.

[0060] As a possible implementation method, after the map construction device is connected to the robot, it obtains multiple frames of continuous original binocular images from the robot. Further, the map construction device determines the feature points in the original binocular image based on the optical flow method, and determines whether the feature points in the original binocular image are dynamic feature points based on whether the feature point pairs between two adjacent frames are on the epipolar line.

[0061] Among them, if the feature point pair between two adjacent frames is on the epipolar line, it is determined that the feature point in the original binocular image is not a dynamic feature point; if the feature point pair between two adjacent frames is not on the epipolar line, it is determined that the feature point in the original binocular image is a dynamic feature point.

[0062] It should be noted that how the map construction device determines the feature points in the original binocular image based on the optical flow method, and determines whether the feature points in the original binocular image are on the epipolar line, can be referred to the subsequent records of the embodiments of the present invention, and will not be repeated here.

[0063] S202: The map building device determines the dynamic target object in the original binocular image according to the dynamic feature points of the original binocular image.

[0064] As a possible implementation, the map construction device determines each object in the original binocular image. Further, the map construction device determines the number of dynamic feature points included in each object, and when the number of dynamic feature points included in the object is greater than or equal to a third threshold, the map construction device determines that the object is a dynamic target object.

[0065] It should be noted that the third threshold value may be set in advance in the map building device by the operation and maintenance personnel of the map building device, and the embodiment of the present invention does not limit this.

[0066] How the map building device determines each object in the original binocular image can be referred to in the subsequent description of the embodiment of the present invention, and will not be described in detail here.

[0067] S203: The map building device deletes the dynamic target object from the original binocular image to obtain a static binocular image.

[0068] As a possible implementation manner, after determining the dynamic object in the original binocular image, the map building device deletes the dynamic target object to obtain a static binocular image.

[0069] It is understandable that in the presence of dynamic objects such as people and cars, the robot's positioning accuracy will decrease or even fail. After the dynamic objects in the original binocular image are deleted in step S203, the influence of dynamic objects on the robot's positioning can be avoided, thereby improving the positioning accuracy in a dynamic environment.

[0070] S204: The map building device determines the motion state of the robot based on the static binocular image.

[0071] The motion state includes position, posture and speed.

[0072] As a possible implementation, the map building device determines the motion state of the robot based on a visual-inertialodometry (VIO) positioning solution.

[0073] It should be noted that the VIO positioning solution adopted in the embodiment of the present invention includes a fusion positioning system of a binocular camera and an inertial measurement unit (IMU). The fusion positioning system adopts the design concept of tight data coupling and nonlinear optimization. The process of determining the motion state of the robot is as follows: Figure 3 As shown, including S2041-S2044.

[0074] S2041. The map building device obtains input data.

[0075] The input data is the static binocular image acquired in step S203 and the IMU data measured by the IMU.

[0076] S2042: The map building device performs data preprocessing.

[0077] This includes feature point extraction and tracking of static binocular images and pre-integration processing of IMU data.

[0078] For static binocular images, feature points are extracted and optical flow tracking is performed to provide data support for the subsequent visual initialization to solve the binocular camera position and posture. For IMU data, the IMU data is pre-integrated to obtain the current posture, speed, rotation angle, and pre-integrated increments, pre-integrated covariances, and Jacobian matrices between adjacent frames required in the subsequent optimization process.

[0079] S2043: The map building device is initialized.

[0080] This includes the relative external parameters of the IMU and the binocular camera, as well as the initialization system status information.

[0081] The initialization process mainly provides a set of initial states for the positioning system, including the relative external parameters of the IMU sensor coordinate system and the binocular camera coordinate system, the position and speed of each image frame in the initial window, the measurement bias of the IMU sensor, the depth of the feature points and other state information.

[0082] S2044. The map construction device performs nonlinear optimization.

[0083] This includes nonlinear optimization of binocular vision and inertial navigation tightly coupled with a sliding window in a map building device.

[0084] The back-end nonlinear optimization module optimizes and adjusts all state quantities in the window using nonlinear optimization methods based on the measurements and predictions in the preprocessing stage and the state in the current sliding window. Finally, the system outputs the motion state (position, posture, and speed) of the robot after optimization.

[0085] S205. The map building device performs depth estimation on the original binocular image to determine the depth information of each pixel in the original binocular image.

[0086] As a possible implementation method, the map building device uses a deep network of a real-time domain adaptive stereo matching network (Modularly ADaptive Network, MADnet) to estimate the depth of the original binocular image and determine the depth information of each pixel in the original binocular image.

[0087] It should be noted that MADnet can be deployed on robots and perform real-time depth estimation. The benchmark process of MADnet for depth estimation of binocular images is as follows: Figure 4 As shown. It includes four basic steps, namely matching cost, cost aggregation, disparity calculation and disparity post-processing / optimization. Among them, matching cost refers to the cost of the left and right image pixels of the binocular image calculated by similarity; cost aggregation refers to optimizing the cost matrix through the connection between adjacent pixels, dealing with the problem that images cannot be accurately paired in weak texture or no texture areas, and reducing the impact of abnormal points; disparity calculation refers to taking the point with the smallest cost value within the disparity range as the corresponding matching point, and calculating the disparity of the point based on the triangulation principle; the purpose of disparity post-processing is to solve problems such as inaccurate occlusion disparity and mismatched points.

[0088] The MADnet network structure is as follows Figure 5a , Figure 5b as well as Figure 5c As shown in the figure, the main structure of the network is a lightweight encoder-decoder structure, and both the encoding and decoding parts are pyramid structures. The F1-F6 of the encoding part are resolutions of 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 respectively, and the D2-D6 of the decoding part are disparity estimation modules corresponding to the F2-F6 resolutions. Each decoding module is used to output a predicted disparity map of the resolution size. The circle between the encoding and decoding is shown in the figure. Figure 5c As shown, it represents the cost volume constructed after the warp operation.

[0089] Figure 5aIt represents a fully adaptive depth estimation method using unsupervised loss and backpropagating through the entire network.

[0090] Figure 5b It is the Modular Adaptation (MAD) algorithm proposed by the MADnet network structure. The encoding and decoding modules of the same resolution are regarded as a combined modulation module M. For example, F3 and D3 form the M3 modulation module. After each loss calculation, only one module is selected for back propagation. In this way, compared with Figure 5a The fully adaptive depth estimation method represented by can greatly improve the speed of online adaptation. The embodiment of the present application adopts Figure 5b The network structure shown performs depth estimation on the original stereo image.

[0091] After calculating the loss function, MADnet proposes a heuristic reward-penalty mechanism to dynamically select the updated module for backpropagation each time, and then estimates the depth of the stereo image by inputting the stereo image through end-to-end training.

[0092] S206: The map building device builds a map according to the motion state and the depth information of each pixel in the original binocular image.

[0093] As a possible implementation method, the map construction device determines the position of each object in the scene corresponding to the original binocular image based on the robot's position, posture, and depth information of pixel points, and then constructs a map based on the position of each object.

[0094] It should be noted that how the map construction device specifically determines the position of each object and constructs a map according to the position of each object can be referred to the subsequent description of the embodiment of the present invention, and will not be repeated here.

[0095] In one design, in order to better plan the robot's running route, so that the robot runs close to static objects and away from dynamic objects, and avoids being disturbed by dynamic objects during operation, the map construction method provided by the embodiment of the present invention enables the map construction device to distinguish between dynamic objects and static objects to construct maps. Figure 6 As shown, the map construction method provided by the embodiment of the present invention also includes S301-S305.

[0096] S301. A map building device determines the position of an object in an original binocular image according to a motion state and depth information of each pixel in the original binocular image.

[0097] As a possible implementation method, the map construction device determines the shooting direction of the original binocular image according to the position and posture of the robot. Furthermore, the map construction device determines the distance between the robot and the object corresponding to each pixel point according to the depth information of each pixel point in the original binocular image. In this way, the map construction device can calculate the position of the object corresponding to each pixel point in the scene corresponding to the original binocular image according to the shooting direction of the original binocular image taken by the robot and the distance between the robot and the object corresponding to each pixel point.

[0098] S302: The map building device generates an occupancy grid map corresponding to the original binocular image according to the positions of the objects in the original binocular image.

[0099] Among them, the state of the voxel corresponding to the position of the object in the original binocular image is the occupied state.

[0100] It should be noted that the occupancy grid map divides the continuous state space into small cubes of a set size. In the occupancy grid map, the small cubes are called voxels. In obstacle avoidance and motion planning applications, the typical size of a voxel is 5cm*5cm*5cm or 10cm*10cm*10cm, which is not limited in the embodiments of the present invention.

[0101] As a possible implementation manner, the map building device determines whether the object in the binocular image exists in the space corresponding to the voxel in the grid map according to the position of the object in the original binocular image.

[0102] When there is an object in the binocular image in the space corresponding to the voxel, the map construction device sets the state of the voxel to an occupied state; when there is no object in the binocular image in the space corresponding to the voxel, the state of the voxel is set to an idle state.

[0103] In some embodiments, in order to ensure that the robot's running route is a safe trajectory, it is necessary to keep the trajectory as far away from the object as possible. Based on this, the embodiment of the present invention establishes an ESDF map to represent the distance information of the point in the state space from the object. In order to calculate the ESDF value, it is necessary to calculate the nearest object to each grid. In order to reduce the amount of calculation and realize real-time operation, the FIESTA algorithm is used.

[0104] The system architecture for building ESDF maps is as follows Figure 7As shown in the figure, three information structures are used to construct the voxel information structure (VIS), index data structure and doubly linked list (DLL). Among them, VIS includes voxel coordinates, occupancy probability, voxel coordinates of the nearest object, Euclidean distance to the object, and all observed neighboring voxels of the voxel; the index data structure maps the voxel coordinates to the index of VIS and manages it through a hash table. DLL can quickly traverse the relevant voxels when the voxel occupancy state changes and update the map efficiently.

[0105] It should be noted that for determining the occupancy probability of a voxel, for each voxel in the occupancy grid map, the probability of the voxel being occupied is represented by the probability value P(s), P(s=1) represents the voxel is occupied, and P(s=0) represents the voxel is idle. In the construction algorithm of the occupancy grid map, the ratio of the voxel occupied state to the idle state is introduced, which is denoted as To ensure the real-time performance of the algorithm, the mapping algorithm adopts a progressive update method, using each measurement data to update the logarithm of the Odd of the voxel to be updated, and then determine the occupancy probability of the voxel.

[0106] It should be noted that in order to update the state of voxels in the FIESTA algorithm, two queues, insertQueue and deleteQueue, are established to store voxels whose occupancy status has changed. When the state of a voxel changes from occupied to idle, the voxel is added to the insertQueue queue; when the state of a voxel changes from idle to occupied, the voxel is added to the deleteQueue queue. Furthermore, the FIESTA algorithm adds both the InsertQueue and the deleteQueue to the updateQueue, after which the algorithm will update the elements in the updateQueue. The update of the elements in the updateQueue includes the following two situations:

[0107] Case 1: The voxel changes from an idle state to an occupied state.

[0108] The FIESTA algorithm marks the voxel as cur and calculates the neighboring voxel nbr in the binocular image whose distance is less than the fourth threshold. If the ESDF value of nbr before the update is greater than the distance from nbr to cur, the ESDF value of the voxel is updated to the distance from nbr to cur and updated in the DLL.

[0109] Case 2: The voxel changes from occupied state to idle state.

[0110] First, the FIESTA algorithm determines the voxel as the nearest neighbor voxel by querying the DLL, and sets the ESDF value of the voxel to infinity, and sets the nearest object voxel to null. Further, the FIESTA algorithm searches for the occupied voxels in the nearest binocular image around the voxel, and sets the ESDF value of the voxel to the distance from the occupied voxel.

[0111] S303: The map building device determines a dynamic object in the original binocular image, and updates the state of a voxel corresponding to the position of the dynamic object to an idle state.

[0112] As a possible implementation, the map construction device uses a global nearest neighbor search and a visible human detector to detect and track dynamic objects. Further, the map construction device determines the voxel corresponding to the position of the dynamic object according to the position of the dynamic object, and sets the state of the voxel corresponding to the position of the dynamic object to an idle state.

[0113] S304: The map building device determines the point cloud data of the dynamic object and generates a three-dimensional occupancy grid.

[0114] As a possible implementation, the map building device uses the Mobilenet-SSD model to perform point cloud detection of dynamic objects and determine point cloud data for representing the dynamic objects. Further, the map building device determines the minimum three-dimensional occupied grid including the point cloud data of the dynamic objects.

[0115] For example, Figure 8 As shown, Figure 8 a in the figure is an exemplary input of the Mobilenet-SSD model for the map construction device. Figure 8 b in the figure is the map construction device using the Mobilenet-SSD model, according to Figure 8 An exemplary output of detecting a dynamic object after the input of a in the figure. The Mobilenet-SSD model combines detection and tracking to improve the robustness of dynamic object tracking.

[0116] S305: The map construction device constructs a map according to the state of the voxels and the three-dimensional occupancy grid.

[0117] As a possible implementation method, the map construction device constructs a map based on the occupancy status of voxels corresponding to objects in the binocular image determined in the above step S302, the voxels corresponding to the positions of dynamic objects, and the point cloud data and three-dimensional occupancy grid of the dynamic objects.

[0118] For example, Fig. 9 As shown, Fig. 9 a in the figure is the original binocular image collected by the robot. Fig. 9b in the figure is a map constructed by the map construction device. In the map constructed by the map construction device, dynamic objects are represented by point cloud data and three-dimensional occupancy grids, and static objects are identified by voxels in an occupied state. In this way, the robot can determine whether an object in the binocular image is a dynamic object or a static object based on the representation of the object in the map, so that when planning the robot's running route, it can stay away from dynamic objects as much as possible to avoid being affected by dynamic objects and causing the robot to be unable to run safely.

[0119] In one design, in order to determine the dynamic feature points in the original binocular image, such as Fig.10 As shown, the map construction method provided by the embodiment of the present invention further includes S401-S405.

[0120] S401: A map constructing apparatus determines a first feature point in an original binocular image and a second feature point in an adjacent binocular image.

[0121] The adjacent binocular images are binocular images of frames adjacent to the original binocular image, and the first feature points and the second feature points are used to represent the same content.

[0122] As a possible implementation method, the map construction device uses an optical flow pyramid to track feature points, determine a first feature point in the original binocular image, and further determine a second feature point in an adjacent binocular image that represents the same content as the first feature point.

[0123] It should be noted that the adjacent binocular image may be a binocular image frame before the original binocular image, or may be a binocular image frame after the original binocular image, which is not limited in the embodiment of the present invention.

[0124] Exemplarily, the map building device uses an optical flow pyramid to track feature points. Fig.11 As shown, Fig.11 A in FIG. 1 shows the hierarchical processing of the original stereo image by the optical flow pyramid. Fig.11 b in FIG. 1 shows the hierarchical processing of adjacent binocular images by the optical flow pyramid, wherein the hierarchical processing of the binocular images by the optical flow pyramid is divided into four layers to meet the requirements of tracking feature points. The images of two adjacent frames are processed in layers, and the 0th layer is the original binocular image. The 1st layer image is obtained by performing Gaussian convolution (Gaussian filtering) and down sampling (deleting all even rows and columns) on the 0th layer image, and then performing Gaussian convolution and down sampling on the 1st layer image to obtain the 2nd layer image, and then performing Gaussian convolution and down sampling on the 2nd layer image to obtain the 3rd layer image.

[0125] After obtaining the optical flow pyramid of two adjacent frames of binocular images, the map construction device starts processing from the highest layer of the optical flow pyramid. The point u3 in the top layer of the first frame of binocular image corresponds to v31 in the top layer of the adjacent frame of binocular image, where v31 and u3 are equal. Then, after the algorithm meets the corresponding conditions, the minimum error point v3n of the current layer and the corresponding optical flow are obtained, where n represents n calculations. Further, the map construction device uses the calculated optical flow to find point v21 in the adjacent frame of binocular images as the initial point of the second layer, and so on, and the same iterative calculation as the third layer is performed to finally obtain the total optical flow containing the optical flow components of each layer, thereby obtaining the final corresponding point v0r, where u0 and v0r are a pair of feature points, u0 is the first feature point, and v0r is the second feature point.

[0126] For example, Fig.12 , the binocular images collected at t1, t2 and t3 are shown in FIG. If the map construction device uses the binocular image at t2 as the original binocular image, the binocular images collected at t1 and t2 are both adjacent binocular images of the original binocular image. Furthermore, the map construction device can determine the feature points of the binocular images collected at t1, t2 and t3 through the optical flow pyramid.

[0127] S402: The map constructing apparatus determines a basic matrix according to the first feature point and the second feature point.

[0128] It should be noted that the map construction device can use the eight-point estimation method to determine the basic matrix. The basic matrix is ​​defined by the equation x′ T Fx=0, x′ and x are any pair of the eight pairs of first feature points and second feature points. Since the matching of each pair of feature points provides a linear equation for calculating the F coefficient, when at least 7 points (a 3×3 homogeneous matrix minus a scale and a rank 2 constraint) are given, the equation can calculate the unknown F.

[0129] S403: The map constructing device determines an epipolar line corresponding to the first feature point according to the basic matrix and the first feature point.

[0130] As a possible implementation manner, the map constructing device determines the homogeneous coordinates of the first feature point, and then determines the epipolar line corresponding to the first feature point according to the homogeneous coordinates of the first feature point and the basic matrix.

[0131] If necessary, Fig.13 As shown, O 1 , O 2 is the position of the optical center of the binocular camera in three-dimensional space, P is a point in three-dimensional space; O 1 , O 2 The plane determined by the three points P and the imaging plane I of the two binocular cameras 1,I 2 intersect at four points; among them, p 1 、p 2 Point P is the imaging plane I of the two binocular cameras 1 ,I 2 The projection on 1 、e 2 For the straight line O 1 , O 2 Imaging plane I with two binocular cameras 1 ,I 2 There are two focal points. Fig.13 The baseline is O 1 , O 2 The line segment determined, the polar plane is O 1 , O 2 The plane determined by the three points P, with the pole being point e 1 、e 2 , the polar line is p 1 、e 1 The determined straight line l 1 and p 2 、e 2 The determined straight line l 2 .

[0132] Among them, if the homogeneous coordinates of the first feature point are p 1 =[u 1 , v 1 , 1], the homogeneous coordinates of the second feature point are P 2 =[u 2 , v 2 , 1], then the epipolar line corresponding to the first feature point determined by the map construction device u 1 , v 1 is the horizontal and vertical coordinates of the first feature point, and F is the basic matrix.

[0133] S404: The map constructing device determines a target distance from the second feature point to the epipolar line.

[0134] As a possible implementation manner, the map construction device determines the distance from the second feature point to the polar line corresponding to the first feature point as the target distance based on the homogeneous coordinates of the first feature point and the second feature point determined in step S403 and the polar line corresponding to the first feature point.

[0135] Exemplarily, the following formula may be used to determine the distance D from the second feature point to the epipolar line corresponding to the first feature point.

[0136]

[0137] S405: When the target distance is greater than a first threshold, the map constructing device determines that the first feature point is a dynamic feature point.

[0138] It should be noted that when the target distance is less than the first threshold, the second feature point is on the polar line corresponding to the first feature point, that is, the first feature point is determined to be a non-dynamic feature point. The first threshold can be determined in advance in the map construction device by the operator of the map construction device.

[0139] In some designs, in order to determine the dynamic target object in the original binocular image, the map construction method provided by the embodiment of the present invention is as follows: Fig.14 As shown, it also includes S501-S502.

[0140] S501. A map building device performs semantic segmentation on an original binocular image to determine boundary points of a first object in the original binocular image.

[0141] The first object is any object in the original binocular image.

[0142] It should be noted that the map construction device uses the SegNet deep network model to perform semantic segmentation on the objects in the original binocular image to determine the objects in the original two-sided image. Further, the map construction device determines the boundary points of the objects based on the determined objects in the element binocular image.

[0143] It should be noted that the architecture of the SegNet deep network model is as follows Fig.15 As shown in the figure, it includes encoding and decoding processes. In the encoding process, features are extracted through convolution. The convolution used by SegNet is the same convolution, that is, the original size of the image is maintained after convolution; in the Decoder process, the same convolution is also used, which is used to enrich the information of the upsampled image, so that the information lost in the pooling process can be obtained through learning in the Decoder process.

[0144] For example, the process of semantic segmentation of objects in the original binocular image by the SegNet deep network model can be as follows: Fig.16 As shown, after the map building device inputs the original binocular image, it is encoded and decoded by the SegNet deep network model to obtain a semantically segmented image.

[0145] S502: When the number of dynamic feature points included in the boundary points of the first object is greater than a second threshold, the map building apparatus determines that the first object is a dynamic target object in the binocular image.

[0146] As a possible implementation, the map construction device determines the number of dynamic feature points included in the boundary points of the first object based on the dynamic feature points determined in the above step S405 and the boundary points of the first object determined in step S501. Further, the map construction device determines whether the first object is a dynamic target object in the binocular image according to whether the number of dynamic feature points included in the boundary points of the first object is greater than a second threshold.

[0147] The present invention provides a map construction method, device, equipment and storage medium. In the map construction method provided by the present invention, before determining the state of a robot, the map construction device first determines the dynamic target object in the current scene corresponding to the original binocular image, and after determining the dynamic target object, deletes the dynamic target object in the original binocular image to obtain a static binocular image. In this way, the robot can determine the motion state of the robot based on the static binocular image, thereby improving the robustness of the robot positioning and eliminating the influence of dynamic objects on the robot positioning and mapping in dynamic scenes.

[0148] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed in this article, the embodiment of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0149] The embodiment of the present invention can divide the user equipment into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0150] Fig.17 Schematic diagram of the structure of a map construction device provided by an embodiment of the present invention. The map construction device is used to execute the above-mentioned map construction method. Fig.17 As shown, the map construction device 60 includes a determination unit 601 and a processing unit 602. For example, Figure 2 As shown, the determining unit 601 can be used to execute S201.

[0151] The determination unit 601 is used to determine the dynamic feature points in the original binocular image collected by the robot. Figure 2 As shown, the determining unit 601 can be used to execute S201.

[0152] The determination unit 601 is further configured to determine the dynamic target object in the original binocular image according to the dynamic feature points of the original binocular image. Figure 2 As shown, the determining unit 601 can be used to execute S202.

[0153] The processing unit 602 is used to delete the dynamic target object from the original binocular image to obtain a static binocular image. Figure 2 As shown, the processing unit 602 can be used to execute S203.

[0154] The determination unit 601 is further used to determine the motion state of the robot based on the static binocular image, where the motion state includes position, posture and speed. Figure 2 As shown, the determining unit 601 can be used to execute S204.

[0155] The determination unit 601 is further configured to perform depth estimation on the original binocular image and determine the depth information of each pixel in the original binocular image. Figure 2 As shown, the determining unit 601 can be used to execute S205.

[0156] The processing unit 602 is further configured to construct a map based on the motion state and the depth information of each pixel in the original binocular image. Figure 2 As shown, the processing unit 602 can be used to execute S206.

[0157] Optional, such as Fig.17 As shown, in the map construction device 60 provided in the embodiment of the present invention, the processing unit 602 is specifically used to determine the position of the object in the original binocular image according to the motion state and the depth information of each pixel in the original binocular image. Figure 6 As shown, the processing unit 602 can be used to execute S301.

[0158] The state of the voxel corresponding to the position of the object in the original binocular image is set as the occupied state, and an occupancy grid map corresponding to the original binocular image is generated. Figure 6 As shown, the processing unit 602 can be used to execute S302.

[0159] Determine the dynamic object in the original binocular image, and update the state of the voxel corresponding to the position of the dynamic object to the idle state. Figure 6 As shown, the processing unit 602 can be used to execute S303.

[0160] Determine the point cloud data of the dynamic object and generate a 3D occupancy grid. Figure 6 As shown, the processing unit 602 can be used to execute S304.

[0161] A map is constructed based on the states of voxels and a 3D occupancy grid. Figure 6 As shown, the processing unit 602 can be used to execute S305.

[0162] Optional, such as Fig.17 As shown, in the map construction device 60 provided by the embodiment of the present invention, the determination unit 601 is specifically used to determine the first feature point in the original binocular image and the second feature point in the adjacent binocular image; the adjacent binocular image is a binocular image of a frame adjacent to the original binocular image, and the first feature point and the second feature point are used to represent the same content. Fig.10 As shown, the determining unit 601 can be used to execute S401.

[0163] The basic matrix is ​​determined based on the first feature point and the second feature point. Fig.10 As shown, the determining unit 601 can be used to execute S402.

[0164] According to the basic matrix and the first feature point, the epipolar line corresponding to the first feature point is determined. Fig.10 As shown, the determining unit 601 can be used to execute S403.

[0165] Determine the target distance from the second feature point to the epipolar line. Fig.10 As shown, the determining unit 601 can be used to execute S404.

[0166] When the target distance is greater than the first threshold, the first feature point is determined to be a dynamic feature point. Fig.10 As shown, the determining unit 601 can be used to execute S405.

[0167] Optional, such as Fig.17 As shown, in the map construction device 60 provided by the embodiment of the present invention, the determination unit 601 is specifically used to perform semantic segmentation on the original binocular image and determine the boundary points of the first object in the original binocular image, where the first object is any object in the original binocular image. Fig.14 As shown, the determining unit 601 can be used to execute S501.

[0168] When the number of dynamic feature points included in the boundary points of the first object is greater than the second threshold, the first object is determined to be a dynamic target object in the binocular image. Fig.14 As shown, the determining unit 601 can be used to execute S502.

[0169] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, an embodiment of the present invention provides a possible structural diagram of a map construction device. The map construction device is used to execute the map construction method executed by the map construction device in the above-mentioned embodiment. Fig.18 As shown, the map construction device 70 includes a processor 701, a memory 702 and a bus 703. The processor 701 and the memory 702 may be connected via the bus 703.

[0170] The processor 701 is the control center of the map construction device, which can be a processor or a general term for multiple processing elements. For example, the processor 701 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0171] As an embodiment, the processor 701 may include one or more CPUs, such as Fig.18 CPU 0 and CPU 1 are shown in .

[0172] The memory 702 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0173] As a possible implementation, the memory 702 may exist independently of the processor 701, and the memory 702 may be connected to the processor 701 via a bus 703 to store instructions or program codes. When the processor 701 calls and executes the instructions or program codes stored in the memory 702, the map construction method provided in the embodiment of the present invention can be implemented.

[0174] In another possible implementation, the memory 702 may also be integrated with the processor 701 .

[0175] The bus 703 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.18 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0176] It should be pointed out that Fig.18 The structure shown does not constitute a limitation on the map building device 70. Fig.18 In addition to the components shown, the map construction device 70 may include Fig.18 More or fewer components, or combinations of certain components, or different arrangements of components may be shown.

[0177] As an example, combining Fig.17 The functions implemented by the determination unit 601 and the processing unit 602 in the map construction device 60 are similar to those Fig.18 The function of processor 701 in is the same.

[0178] Optional, such as Fig.18 As shown, the map building device provided by the embodiment of the present invention may further include a communication interface 704 .

[0179] The communication interface 704 is used to connect with other devices through a communication network. The communication network may be Ethernet, wireless access network, wireless local area network (WLAN), etc. The communication interface 704 may include an acquisition unit for receiving data and a sending unit for sending data.

[0180] In one design, in the map building device provided by an embodiment of the present invention, the communication interface may also be integrated into the processor.

[0181] Fig.19 FIG. 2 shows another hardware structure of a map construction device in an embodiment of the present invention. Fig.19 As shown, the map construction device 80 may include a processor 801 and a communication interface 802. The processor 801 is coupled to the communication interface 802.

[0182] The functions of the processor 801 may refer to the description of the processor 701. In addition, the processor 801 also has a storage function, which may refer to the function of the memory 702.

[0183] The communication interface 802 is used to provide data to the processor 801. The communication interface 802 may be an internal interface of the map construction device, or an external interface of the map construction device (equivalent to the communication interface 704).

[0184] It should be pointed out that Fig.19 The structure shown in the figure does not constitute a limitation on the map construction device, except Fig.19 In addition to the components shown, the map building device 80 may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional units is used as an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0186] An embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When a computer executes the instructions, the computer executes each step in the method flow shown in the above method embodiment.

[0187] An embodiment of the present invention provides a computer program product comprising instructions. When the instructions are executed on a computer, the computer is enabled to execute the map construction method in the above method embodiment.

[0188] Among them, the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, and a hard disk. Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), registers, hard disks, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any other form of computer-readable storage medium in a suitable combination of the above, or numerical values ​​in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an Application Specific Integrated Circuit (ASIC). In embodiments of the present invention, computer-readable storage media may be any tangible media that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0189] Since the apparatus, device computer-readable storage medium, and computer program product in the embodiments of the present invention can be applied to the above method, the technical effects that can be obtained can also refer to the above method embodiments, and the embodiments of the present invention will not be repeated here.

[0190] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A map construction method, characterized in that: The method comprises: Determine the dynamic feature points in the original binocular image collected by the robot, and determine the dynamic target object in the original binocular image according to the dynamic feature points of the original binocular image; Deleting the dynamic target object from the original binocular image to obtain a static binocular image, and determining the motion state of the robot based on the static binocular image, wherein the motion state includes position, posture and speed; Performing depth estimation on the original binocular image to determine depth information of each pixel in the original binocular image; Building a map according to the motion state and the depth information of each pixel in the original binocular image; The step of constructing a map according to the motion state and the depth information of each pixel in the original binocular image includes: Determining the shooting direction of the original binocular image according to the position and posture of the robot; Determine the distance between the robot and the object corresponding to each pixel point according to the depth information of each pixel point in the original binocular image; Detecting and tracking dynamic objects in the original binocular image; and determining a three-dimensional occupancy grid including the dynamic objects.

2. The map construction method according to claim 1, characterized in that: The step of constructing a map according to the motion state and the depth information of each pixel in the original binocular image includes: Determine the position of the object in the original binocular image according to the motion state and the depth information of each pixel in the original binocular image; According to the position of the object in the original binocular image, an occupancy grid map corresponding to the original binocular image is generated, and the state of the voxel corresponding to the position of the object in the original binocular image is the occupancy state; Determine a dynamic object in the original binocular image, and update the state of a voxel corresponding to the position of the dynamic object to an idle state; Determine the point cloud data of dynamic objects and generate a three-dimensional occupancy grid; A map is constructed based on the states of the voxels and the three-dimensional occupancy grid.

3. The map construction method according to claim 1 or 2, characterized in that: The step of determining the dynamic feature points in the original binocular image collected by the robot includes: Determine a first feature point in the original binocular image and a second feature point in an adjacent binocular image; the adjacent binocular image is a binocular image of a frame adjacent to the original binocular image, and the first feature point and the second feature point are used to represent the same content; Determine a basic matrix according to the first feature point and the second feature point; Determining, according to the basic matrix and the first feature point, an epipolar line corresponding to the first feature point; determining a target distance from the second feature point to the epipolar line; When the target distance is greater than a first threshold, the first feature point is determined to be a dynamic feature point.

4. The map construction method according to claim 1 or 2, characterized in that: The step of determining the dynamic target object in the original binocular image according to the dynamic feature points of the original binocular image comprises: Performing semantic segmentation on the original binocular image to determine boundary points of a first object in the original binocular image, where the first object is any object in the original binocular image; When the number of dynamic feature points included in the boundary points of the first object is greater than a second threshold, the first object is determined to be a dynamic target object in the binocular image.

5. A map construction device, characterized in that: comprising a determination unit and a processing unit; The determination unit is used to determine the dynamic feature points in the original binocular image collected by the robot; The determining unit is further used to determine the dynamic target object in the original binocular image according to the dynamic feature points of the original binocular image; The processing unit is used to delete the dynamic target object from the original binocular image to obtain a static binocular image; The determining unit is further used to determine the motion state of the robot based on the static binocular image, wherein the motion state includes position, posture and speed; The determining unit is further used to perform depth estimation on the original binocular image to determine the depth information of each pixel in the original binocular image; The processing unit is further used to construct a map according to the motion state and the depth information of each pixel in the original binocular image; The processing unit is further used to determine the shooting direction of the original binocular image according to the position and posture of the robot; determine the distance between the robot and the object corresponding to each pixel point according to the depth information of each pixel point in the original binocular image; detect and track dynamic objects in the original binocular image; and determine a three-dimensional occupancy grid including dynamic objects.

6. The map construction device according to claim 5, characterized in that: The processing unit is specifically used to determine the position of the object in the original binocular image according to the motion state and the depth information of each pixel in the original binocular image; According to the position of the object in the original binocular image, an occupancy grid map corresponding to the original binocular image is generated, and the state of the voxel corresponding to the position of the object in the original binocular image is the occupancy state; Determine a dynamic object in the original binocular image, and update the state of a voxel corresponding to the position of the dynamic object to an idle state; Determine the point cloud data of dynamic objects and generate a three-dimensional occupancy grid; A map is constructed based on the states of the voxels and the three-dimensional occupancy grid.

7. The map construction device according to claim 5 or 6, characterized in that: The determining unit is specifically used to determine a first feature point in the original binocular image and a second feature point in an adjacent binocular image; the adjacent binocular image is a binocular image of an adjacent frame to the original binocular image, and the first feature point and the second feature point are used to represent the same content; Determine a basic matrix according to the first feature point and the second feature point; Determining, according to the basic matrix and the first feature point, an epipolar line corresponding to the first feature point; determining a target distance from the second feature point to the epipolar line; When the target distance is greater than a first threshold, the first feature point is determined to be a dynamic feature point.

8. The map construction device according to claim 5 or 6, characterized in that: The determining unit is specifically configured to perform semantic segmentation on the original binocular image to determine a boundary point of a first object in the original binocular image, where the first object is any object in the original binocular image; When the number of dynamic feature points included in the boundary points of the first object is greater than a second threshold, the first object is determined to be a dynamic target object in the binocular image.

9. A map construction device, characterized in that: including memory and processor; The memory is coupled to the processor; The memory is used to store computer program code, wherein the computer program code includes computer instructions; When the processor executes the computer instructions, the map building device performs the map building method according to any one of claims 1-4.

10. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When the instruction is executed on a map building device, the map building device is caused to execute the map building method according to any one of claims 1 to 4.

Images