Dynamic scene localization and mapping method and related device

Abstract

The application provides a positioning and map construction method of a dynamic scene and related equipment, comprising: determining an RGBD image, performing target detection and feature point extraction on the RGBD image, and determining target objects and static feature points; determining a 3D bounding box according to the target objects, and determining a dynamic pose according to the 3D bounding box and the RGBD image; calculating a depth pyramid and an image pyramid according to the RGBD image; performing camera pose estimation by an optical flow tracking method according to the depth pyramid and the image pyramid, and determining a camera pose; and performing positioning and map construction according to the static feature points, the dynamic pose and the camera pose.

Description

Technical Field

[0001] This application relates to the field of positioning and mapping technology, and in particular to a method and related equipment for positioning and mapping in dynamic scenes. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, mobile robots are increasingly being used in the real world, from self-driving cars to industrial automation, from augmented reality to intelligent inspection. However, navigation and localization in dynamic environments remains a challenge for these robots.

[0003] In the field of navigation and positioning, robots need to be able to detect, track, and adapt to these changes in real time to ensure the accuracy of their positioning and mapping. However, current positioning and mapping technologies suffer from insufficient robustness, leading to problems such as matching errors and trajectory drift, and insufficient utilization of semantic information, resulting in a limited amount of usable static semantic information, which affects positioning and mapping. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a method and related equipment for localization and mapping of dynamic scenes.

[0005] To achieve the above objectives, this application provides a method for localization and map construction in dynamic scenes, including:

[0006] Determine the RGBD image, perform target detection and feature point extraction on the RGBD image, and determine the target object and static feature points;

[0007] A 3D bounding box is determined based on the target object, and a dynamic pose is determined based on the 3D bounding box and the RGBD image.

[0008] Calculate the depth pyramid and image pyramid based on the RGBD image;

[0009] Based on the depth pyramid and the image pyramid, camera pose is estimated using optical flow tracing to determine the camera pose.

[0010] Localization and map construction are performed based on the static feature points, the dynamic pose, and the camera pose.

[0011] Optionally, the step of performing target detection and feature point extraction on the RGBD image to determine the target object and static feature points includes:

[0012] The target object is determined by performing target detection on any of the RGBD images using a target detection network.

[0013] The dynamic feature points are determined by extracting feature points from any of the target images using a feature point extraction algorithm.

[0014] The static feature points are determined by filtering the dynamic feature points based on the target object.

[0015] Optionally, determining the 3D bounding box based on the target object includes:

[0016] A 2D bounding box is determined based on the target detection network; wherein the target detection network is the YOLOv7 target detection network.

[0017] Determine the vanishing point on the RGBD image, and based on the vanishing point and the 2D bounding box, determine at least two predicted 3D bounding boxes;

[0018] The at least two predicted 3D bounding boxes are filtered to determine the 3D bounding boxes.

[0019] Optionally, determining the static feature points by removing the dynamic feature points based on the target object includes:

[0020] Based on the target object, the dynamic feature points are filtered using a depth map to determine the static feature points.

[0021] Optionally, the step of filtering the at least two predicted 3D bounding boxes to determine the 3D bounding boxes includes:

[0022] The 3D bounding boxes are determined by filtering at least two predicted 3D bounding boxes using the loss function shown below; wherein the loss function includes distance error, angle alignment error, and shape error.

[0023] E(O|I)=φ dist (O,I)+w1φ angle (O,I)+w2φ shape (O)

[0024] Where I is the target image, O is the predicted 3D bounding box, and φ dist For distance error, φ angle For angular alignment error, φ shape For shape error, w1 is a hyperparameter representing the weight of angle alignment error, and w2 is a hyperparameter representing the weight of shape error.

[0025] Optionally, the step of localization and map construction based on the static feature points, the dynamic pose, and the camera pose includes:

[0026] Determine the factor map based on the dynamic pose and the camera pose;

[0027] Determine the descriptor based on the static feature points;

[0028] Solve the factor graph based on the descriptor to determine the current camera pose, the current dynamic object pose, and the current static object pose;

[0029] Based on the current camera pose, the current dynamic object pose, and the current static object pose, localization and map construction are performed.

[0030] Based on the same inventive concept, embodiments of this application also provide a dynamic scene positioning and map building device, including:

[0031] The detection and extraction module is configured to determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine the target object and static feature points;

[0032] The dynamic pose determination module is configured to determine a 3D bounding box based on the target object, and to determine a dynamic pose based on the 3D bounding box and the RGBD image.

[0033] The calculation module is configured to calculate a depth pyramid and an image pyramid based on the RGBD image;

[0034] The camera pose determination module is configured to perform camera pose estimation using optical flow tracing based on the depth pyramid and the image pyramid, and determine the camera pose.

[0035] The localization and map building module is configured to perform localization and map building based on the static feature points, the dynamic pose, and the camera pose.

[0036] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic scene localization and map construction method as described in any of the above.

[0037] Based on the same inventive concept, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute any of the above-described methods for locating and mapping dynamic scenes.

[0038] Based on the same inventive concept, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to execute any of the above-described methods for locating and mapping dynamic scenes.

[0039] As can be seen from the above, the localization and mapping method and related equipment for dynamic scenes provided in this application, by using the optical flow tracking method of depth map, combining RGB images and RGBD images to establish image pyramids and depth pyramids, and adaptively adjusting the tracking strategy according to the tracking key points, can efficiently track camera pose, thereby effectively improving the accuracy and robustness of SLAM system localization in dynamic environments. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the dynamic scene localization and map construction method according to an embodiment of this application;

[0042] Figure 2 This is a schematic diagram illustrating the execution flow of the dynamic scene localization and map construction method according to an embodiment of this application;

[0043] Figure 3 This is a schematic diagram illustrating the generation of a 3D bounding box according to an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of the pose graph structure of an embodiment of this application;

[0045] Figure 5 This is a schematic diagram of the pose graph structure between two consecutive frames in an embodiment of this application;

[0046] Figure 6 This is a schematic diagram of the structure of the dynamic scene localization and map building device according to an embodiment of this application;

[0047] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0049] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] To facilitate understanding of the technical solutions disclosed herein, some technical terms involved in this disclosure will be introduced below.

[0051] SLAM technology: Simultaneous Localization and Mapping.

[0052] RGBD images are composed of a series of RGBD images, each containing a standard RGB three-channel color image and a depth image. RGBD images are typically captured by specialized sensors such as Kinect and RealSense, which combine a traditional color camera with a depth sensor to generate the RGBD image. RGBD images are stored as multi-channel images, with each pixel containing a red, green, and blue component, as well as a depth component.

[0053] An RGB image is an image generated based on different intensities of the three primary colors: red, green, and blue. In an RGB image, each pixel typically consists of three components, representing the brightness values ​​of red, green, and blue. Traditional color cameras can capture RGB images without the need for an additional depth sensor.

[0054] World coordinate system: The absolute coordinate system of the system, used to describe the position of all other coordinate systems or objects. The world coordinate system is usually set as a reference coordinate system, such as Xw—Yw—Zw, where (xw,yw,zw) represents the coordinates of a point P in space in the world coordinate system.

[0055] Odometry estimation: a method and technique for recursively estimating the real-time position, attitude, and other physical quantities of a robot or other moving object using sensor information.

[0056] Vanishing Point (VP): In perspective projection, all parallel lines intersect at a single point in an image; this point is called the vanishing point. In 3D space, parallel lines in different directions will have different vanishing points. The vanishing point provides important clues about scene depth and camera perspective. (Determination of the vanishing point: (Image processing-based methods) ① Use edge detection algorithms (such as Canny edge detection) to identify edges in the image, analyze the edges, infer parallel lines, and find the vanishing point; ② Use line detection algorithms (such as Hough transform) to identify lines in the image, analyze the direction and intersection of the lines to determine the vanishing point; ③ Specialized vanishing point detection algorithms (such as NeurVPS neural vanishing point scanning technology) automatically detect vanishing points in images.)

[0057] 2D bounding box: This is a rectangular box around the edge of an object on the image plane. It gives the object's position and size on the two-dimensional image.

[0058] Canny edge detection: a method that uses a multi-level edge detection algorithm to detect image edges. The basic idea is to find the location with the strongest gray-level intensity change in the image (i.e., the gradient direction) and construct a distance transformation map (distance transformation is mainly a transformation performed on a binary image. The gray value of the distance map output after the binary image has only undergone transformation means: the distance from the pixel to the nearest background point (the background is generally black).

[0059] To make the technical solutions of this disclosure clearer and easier to understand, the method for locating and constructing dynamic scenes provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0060] As described in the background section, with the rapid development of artificial intelligence and robotics, mobile robots are increasingly widely used in the real world, from autonomous vehicles to industrial automation, from augmented reality to intelligent inspection. However, the navigation and localization of these robots in dynamic environments remains a challenge.

[0061] In the field of navigation and localization, robots need to be able to detect, track, and adapt to these changes in real time to ensure the accuracy of their localization and mapping. Current localization and mapping technologies suffer from insufficient robustness, leading to problems such as matching errors and trajectory drift, and insufficient utilization of semantic information, resulting in a limited amount of usable static semantic information, which affects localization and mapping.

[0062] In practical applications, dynamic environments such as urban traffic, indoor robot navigation, drone operation, and agricultural robots all require SLAM systems to effectively handle dynamic objects. Therefore, robots need to be able to detect, track, and adapt to these changes in real time to ensure the accuracy of their localization and mapping.

[0063] In view of this, embodiments of this application provide a method, apparatus, electronic device, storage medium, and program product for localization and mapping of dynamic scenes, including odometry estimation, local mapping, and loop closure detection. In odometry estimation, feature point extraction, object detection, and image pyramid construction are performed simultaneously on the RGBD image. In the dynamic object fitting module, the 3D bounding box of the object in the world coordinate system is calculated based on the 2D bounding box obtained from object detection. The extracted feature points can be initially classified by combining them with object detection. Further, in the local mapping part, for dynamic feature points in the target bounding box, the RANSAC method of joint depth map is used to classify dynamic and static feature points in the target bounding box, thereby removing dynamic feature points from the entire image, and further preliminarily estimating the camera pose based on the static feature points. Then, in the loop closure detection part, global pose optimization is achieved based on the camera pose, static feature points, and dynamic pose.

[0064] This application uses a depth map optical flow tracing method, combining RGB and RGBD images to establish an image pyramid and a depth pyramid. It adaptively adjusts the tracking strategy based on the tracking key points, which can efficiently track camera pose, thereby effectively improving the accuracy and robustness of SLAM system localization in dynamic environments.

[0065] like Figure 1 As shown, the method for localization and map construction of the dynamic scene includes:

[0066] Step S102: Determine the RGBD image, perform target detection and feature point extraction on the RGBD image, and determine the target object and static feature points;

[0067] Step S104: Determine a 3D bounding box based on the target object, and determine the dynamic pose based on the 3D bounding box and the RGBD image;

[0068] Step S106: Calculate the depth pyramid and image pyramid based on the RGBD image;

[0069] Step S108: Based on the depth pyramid and the image pyramid, perform camera pose estimation using optical flow tracing to determine the camera pose;

[0070] Step S110: Based on the static feature points, the dynamic pose, and the camera pose, perform localization and map construction.

[0071] Step S102: First, acquire an RGBD image or an RGBD image sequence, such as... Figure 2As shown, after acquiring an RGBD image or RGBD image sequence, feature point extraction and object detection are performed simultaneously on the acquired RGBD image. Then, based on the object detection results, the depth-based RANSAC method is used to remove the bounding boxes (i.e., 2D bounding boxes) of the prior dynamic object categories. Specifically, feature point extraction uses the FAST feature point extraction method to determine static feature points; object detection uses the YOLOv7 network to identify target objects.

[0072] Specifically, step S102 includes the following steps:

[0073] Step (1) Feature point classification: Denote all feature points in a bounding box as set S. Based on the different depth information, these feature points are divided into two categories: feature points on the surface of the target object, i.e., dynamic feature points, and feature points of the static background, i.e., static feature points.

[0074] Step (2) Loop processing: Perform k loops, each loop including the following steps:

[0075] a. Select depth difference calculation points: Select any two feature points and calculate their depth difference; for example, if the depth values ​​of the two feature points are z1 and z2, then the depth difference is Δz = z1 - z2.

[0076] b. Traverse and filter feature points: Traverse all feature points in set S and find feature points whose depth difference is within an acceptable range.

[0077] c. Classify interior and exterior points: Feature points that meet the depth difference requirement are designated as interior points, and other feature points are designated as exterior points. Interior points are dynamic feature points.

[0078] Step (3) Determine the dynamic point set: After k iterations, select the inner point set containing the most feature points – the dynamic point set – as the dynamic point set. Then, distinguish the feature points on the surface of the target object from the feature points in the static background, and remove or filter the static feature points to determine the dynamic feature points.

[0079] The above method combines depth maps and target detection results to use the RANSAC method for fine-grained removal of feature points from dynamic objects.

[0080] In step S104, a 3D bounding box is determined based on the target object. This determination involves two steps: 3D bounding box generation and 3D bounding box filtering. Specifically: First, a 2D bounding box is determined using an object detection network, and objects within these 2D bounding boxes can be considered as the target object. Then, the vanishing point on the RGBD image is determined, and at least two predicted 3D bounding boxes are determined based on the vanishing point and the 2D bounding boxes. Finally, the at least two predicted 3D bounding boxes are filtered to determine the final 3D bounding box. The object detection network used is the YOLOv7 network.

[0081] In some optional implementations, for 3D bounding box generation, the vanishing point VP and the 2D bounding box are first determined. Based on the vanishing point VP and the 2D bounding box, the eight corner points of the 3D bounding box can be obtained, and then the 3D bounding box of the dynamic object is calculated. The vanishing point VP is the intersection of parallel lines projected onto the perspective view of the 3D bounding box. The 3D cube O can be represented by nine degrees of freedom parameters, and the position matrix t = [t...]. x , t y , t z ], dimension matrix d = [t x , t y , t z And the rotation matrix R. A 3D cuboid has three orthogonal axes. Based on the camera parameters K and the object's rotation matrix R, three VPs can be formed after projection, as shown in the following formula.

[0082] VP i =KR col (i), i∈{1,2,3}

[0083] Where K represents the camera parameters, serving as the calibration matrix, and R... col(i) Let R be the i-th column, where i∈{1,2,3}.

[0084] like Figure 3 As shown, observing a cube in a 2D image can be divided into three categories. A maximum of three faces of a cube can be observed simultaneously, such as... Figure 3 As shown in (1). If the three VP points and point 1 are known, then The remaining points can be obtained by analogy.

[0085] Furthermore, since the pitch and roll angles of dynamic objects on the ground are 0, the vertices located on the ground can be directly back-projected onto the 3D ground plane, and then the positions of other vertical vertices can be calculated using this method. For example, vertex 5 on the 3D ground plane mentioned above can be expressed as [n, m] (vectors and distances in the camera frame), and the corresponding 3D vertex P5 is the back-projected ray K. -1The intersection of p5 and the ground plane is shown in the following formula, where n and m represent the vector and distance of vertex 5 on the 3D ground plane in the camera frame, respectively, K represents the camera parameters, p5 represents the position of vertex 5 in the camera frame, and P5 represents the position of vertex 5 on the 3D ground plane.

[0086]

[0087] For the 3D bounding box filtering, a loss function is defined to filter the obtained 3D bounding box proposals. The loss function consists of three parts: distance error, angle alignment error, and shape error, as shown in the following formula:

[0088] E(O|I)=φ dist (O,I)+w1φ angle (O,I)+w2φ shape (O)

[0089] Where I represents the image, O represents the 3D bounding box, and φ dist φ represents the distance error. angle Indicates angular alignment error, φ angle The shape error is represented by w1 and w2, which are hyperparameters representing the weights of the angular alignment error and the shape error.

[0090] In some embodiments, distance error, angular alignment error, and shape error are considered. It should be noted that:

[0091] Regarding distance error: Since the edges of the cuboid projected onto the 2D image by the 3D bounding box should match the edges of objects in the image, the 3D bounding box is constrained based on the distance between them. First, Canny edge detection is performed on the 3D bounding box to construct a distance transformation map. Second, 10 points are uniformly sampled for each visible cuboid edge, and the distance map values ​​of all points are calculated, summed, and normalized by dividing by the diagonal length of the 2D bounding box.

[0092] For angular alignment error: Since distance error is highly sensitive to false positive edge noise (e.g., object surface texture), it is also necessary to detect long line segments and measure whether they are aligned with the vanishing point VP calculated during 3D bounding box generation. The angular error is defined as the error between the angle of the edge forming the vanishing point (composed of 2D points) and the angle of the corresponding edge obtained from line detection, as shown in the following formula for each VP. i We can find two outermost line segments with minimum and maximum slopes, denoted as ... <l i_ms , l i_mt >and <l i_ns , l i_nt >,<a,b> This represents the slope angle of a line segment with endpoints a and b.

[0093]

[0094] Where I represents the image, O represents the 3D bounding box, || is the Euclidean norm of the vector, and l i_ms l is the starting point of the line segment with the minimum slope. i_mt The endpoint of the line segment with the minimum slope, l i_ns As the starting point of a line segment with a large slope, l i_nt It is the endpoint of a line segment with a large slope. <VP i , l i_nt > represents the slope of the line connecting the vanishing point i to the endpoint of the segment with the maximum slope. <VP i , l i_mt > is the slope of the line connecting the vanishing point i to the endpoint of the line segment with the minimum slope.

[0095] Regarding shape error: Since similar 2D cuboid corners can generate completely different 3D cuboids, a shape error function is added to filter cubes with abnormal skew ratios (s = length / width), as shown in the following formula:

[0096] φ shape (O) = max(s - σ, 0)

[0097] Where σ represents the preset abnormal skew ratio threshold, and s represents the skew ratio.

[0098] In step S106, based on the RGBD image, the depth pyramid and image pyramid are calculated as follows: optical flow tracing adds depth information to the LK pyramid optical flow algorithm to assist in decision-making.

[0099] Image pyramids are a type of multi-scale image representation. They are an effective yet conceptually simple structure for interpreting images at multiple resolutions, likening the layers of an image to a pyramid; the higher the layer, the smaller the image and the lower the resolution. Depth pyramids are deep learning models based on a pyramid structure that extract information from different scales or resolutions of an image. In a depth pyramid, the bottom is a high-resolution representation of the image to be processed, and the top is a low-resolution representation.

[0100] It should be noted that an RGBD image consists of four layers. When determining the image pyramid, the first three layers (RGB) of the RGBD image are used for calculation. When determining the depth pyramid, the four layers of the RGBD image are used for calculation.

[0101] In some implementations, the pyramid size decreases to half the size of the previous layer each time from top to bottom. Optical flow is calculated from top to bottom, with the calculation result of the previous layer serving as the initial estimate for the current layer. Only the residual needs to be calculated each time, and the residuals are accumulated to obtain the final optical flow. For camera pose estimation based on the depth pyramid and the image pyramid, the camera pose is determined using optical flow tracking, such as... Figure 2 As shown, 3D key points are first tracked in the two layers of the image pyramid with high resolution. If the proportion of 3D key points that fail to be tracked is greater than 2 / 3, then all 2D and 3D key points are tracked in all layers of the image pyramid. The initial position of the 3D key points is the position in the previous frame.

[0102] Otherwise, track the 3D keypoints that failed to be tracked in the previous step in the first two layers of the depth pyramid, and determine whether the proportion of the number of 3D keypoints that failed to be tracked is still greater than 2 / 3. If so, track all 2D and 3D keypoints in all layers of the image pyramid; otherwise, only track the 3D keypoints that failed to be tracked in the previous step. Finally, remove the points that failed to be tracked.

[0103] In some embodiments, to obtain a unified static global reference frame, further pose optimization is performed for each keyframe based on the pose graph. Therefore, localization and map construction are performed based on the static feature points, the dynamic pose, and the camera pose.

[0104] Furthermore, such as Figure 4 The diagram shown is a simplified pose graph structure, where nodes represent global static pose estimation. An edge represents the relative pose between two nodes (a special Euclidean group). Therefore, the diagram contains the formula shown. The constraints can be further used to obtain the following formula: The equation.

[0105] Changing the transformation order does not affect the consistency of the loops in the pose graph; therefore, the formula... It can be further written as:

[0106]

[0107] In some embodiments, such as Figure 5 The image shows the pose graphs of two consecutive frames. Based on the odometry estimation, the camera poses in the global static coordinate system W at times t-1 and t can be obtained. and Based on this, the dynamic object fitting module can obtain the pose of the dynamic object V in the global static coordinate system W at time t-1 and time t. and Camera - Camera Side This can be obtained through an odometer (a device and technology used to measure the distance traveled by objects, especially vehicles), and the camera-object edge. The object-to-object edge mapping can be obtained through the dynamic object fitting module, combined with depth map and odometry estimation. According to the formula The loss function γ for the four edges can be obtained. cc γ cv (t-1), γ cv (t) and γ vv The specific calculation method is shown in the following formula:

[0108]

[0109] according to Figure 5 The consistency relationship of the rings in the loss function is as follows:

[0110] Υ=Υ cc ×Υ cv(t) ×(Υ vv ) -1 ×(Υ cv(t-1) ) -1

[0111] The loss function Υ combining four edges cc , Υ cv(t-1) , Υ cb(t) , Υ vv From the formula for calculating the loss function Υ, we can further obtain the following constraint relationship:

[0112]

[0113] Solving the pose graph optimization problem constructed above yields the optimized poses of the camera and moving / static objects, enabling further localization and map construction. By fitting semantic information from moving / static objects, the pose solution is incorporated into the graph optimization problem, achieving the extraction and full utilization of semantic information from moving / static objects in the scene.

[0114] As can be seen from the above, the localization and mapping method for dynamic scenes provided in this application uses the optical flow tracing method of depth map, combines RGB images and RGBD images to establish image pyramids and depth pyramids, and adaptively adjusts the tracking strategy according to the tracking key points. This can efficiently track camera pose, thereby effectively improving the accuracy and robustness of SLAM system localization in dynamic environments.

[0115] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0116] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a dynamic scene positioning and map building device.

[0118] refer to Figure 6 The dynamic scene localization and mapping device includes:

[0119] The detection and extraction module 602 is configured to determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine the target object and static feature points;

[0120] The dynamic pose determination module 604 is configured to determine a 3D bounding box based on the target object, and to determine a dynamic pose based on the 3D bounding box and the RGBD image.

[0121] The calculation module 606 is configured to calculate a depth pyramid and an image pyramid based on the RGBD image;

[0122] The camera pose determination module 608 is configured to perform camera pose estimation using optical flow tracing based on the depth pyramid and the image pyramid, and determine the camera pose.

[0123] The localization and map building module 610 is configured to perform localization and map building based on the static feature points, the dynamic pose, and the camera pose.

[0124] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0125] The apparatus described above is used to implement the localization and map construction method for the corresponding dynamic scene in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0126] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic scene localization and map construction method described in any of the above embodiments.

[0127] Figure 7 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0128] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0129] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0130] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0131] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0132] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0133] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0134] The electronic devices described above are used to implement the localization and map construction methods for the corresponding dynamic scenes in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0135] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the dynamic scene localization and map construction method as described in any of the above embodiments.

[0136] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0137] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the dynamic scene localization and map construction method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0138] Based on the same inventive concept, corresponding to the dynamic scene localization and mapping method described in any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the dynamic scene localization and mapping method. Corresponding to the execution entity for each step in each embodiment of the dynamic scene localization and mapping method, the processor executing the corresponding step can belong to the corresponding execution entity.

[0139] The computer program product of the above embodiments is used to enable the computer and / or the processor to execute the dynamic scene localization and map construction method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0140] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0141] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0142] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0143] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for localization and map construction in a dynamic scene, characterized in that, include: Determine the RGBD image, perform target detection and feature point extraction on the RGBD image, and determine the target object and static feature points; A 3D bounding box is determined based on the target object, and a dynamic pose is determined based on the 3D bounding box and the RGBD image; wherein, determining the 3D bounding box based on the target object includes: A 2D bounding box is determined based on the target detection network; wherein the target detection network is the YOLOv7 target detection network. Determine the vanishing point on the RGBD image, and based on the vanishing point and the 2D bounding box, determine at least two predicted 3D bounding boxes; Filtering the at least two predicted 3D bounding boxes to determine the 3D bounding boxes includes: The 3D bounding boxes are determined by filtering at least two predicted 3D bounding boxes using the loss function shown below; wherein the loss function includes distance error, angle alignment error, and shape error. Where I represents the target image and O represents the predicted 3D bounding box. For distance error, This is due to angular alignment error. For shape error, w1 is a hyperparameter representing the weight of angle alignment error, and w2 is a hyperparameter representing the weight of shape error; Calculate the depth pyramid and image pyramid based on the RGBD image; Based on the depth pyramid and the image pyramid, camera pose is estimated using optical flow tracing to determine the camera pose. Based on the static feature points, the dynamic pose, and the camera pose, localization and map construction are performed, including: Determine the factor map based on the dynamic pose and the camera pose; Determine the descriptor based on the static feature points; Solve the factor graph based on the descriptor to determine the current camera pose, the current dynamic object pose, and the current static object pose; Based on the current camera pose, the current dynamic object pose, and the current static object pose, localization and map construction are performed.

2. The method according to claim 1, characterized in that, The step of performing target detection and feature point extraction on the RGBD image to determine the target object and static feature points includes: The target object is determined by performing target detection on any of the RGBD images using a target detection network. Feature points are extracted from any of the target images using a feature point extraction algorithm to determine dynamic feature points; The static feature points are determined by filtering the dynamic feature points based on the target object.

3. The method according to claim 2, characterized in that, The step of determining the static feature points by eliminating the dynamic feature points based on the target object includes: Based on the target object, the dynamic feature points are filtered using a depth map to determine the static feature points.

4. A positioning and mapping device, characterized in that, include: The detection and extraction module is configured to determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine the target object and static feature points; A dynamic pose determination module is configured to determine a 3D bounding box based on the target object, and to determine a dynamic pose based on the 3D bounding box and the RGBD image; wherein, determining the 3D bounding box based on the target object includes: A 2D bounding box is determined based on the target detection network; wherein the target detection network is the YOLOv7 target detection network. Determine the vanishing point on the RGBD image, and based on the vanishing point and the 2D bounding box, determine at least two predicted 3D bounding boxes; Filtering the at least two predicted 3D bounding boxes to determine the 3D bounding boxes includes: The 3D bounding boxes are determined by filtering at least two predicted 3D bounding boxes using the loss function shown below; wherein the loss function includes distance error, angle alignment error, and shape error. Where I represents the target image and O represents the predicted 3D bounding box. For distance error, This is due to angular alignment error. For shape error, w1 is a hyperparameter representing the weight of angle alignment error, and w2 is a hyperparameter representing the weight of shape error; The calculation module is configured to calculate a depth pyramid and an image pyramid based on the RGBD image; The camera pose determination module is configured to perform camera pose estimation using optical flow tracing based on the depth pyramid and the image pyramid, and determine the camera pose. The localization and map building module is configured to perform localization and map building based on the static feature points, the dynamic pose, and the camera pose, including: Determine the factor map based on the dynamic pose and the camera pose; Determine the descriptor based on the static feature points; Solve the factor graph based on the descriptor to determine the current camera pose, the current dynamic object pose, and the current static object pose; Based on the current camera pose, the current dynamic object pose, and the current static object pose, localization and map construction are performed.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the computer program, implements the method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method according to any one of claims 1 to 3.

7. A computer program product comprising computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-3.

Images