Dynamic scene positioning and mapping method and related equipment

Abstract

The invention provides a dynamic scene positioning and mapping method and related equipment, and the method comprises the steps: determining an RGBD image, carrying out the target detection and feature point extraction of the RGBD image, and determining a target object and a static feature point; determining a 3D bounding box according to the target object, and determining a dynamic pose according to the 3D bounding box and the RGBD image; calculating a depth pyramid and an image pyramid according to the RGBD image; according to the depth pyramid and the image pyramid, performing camera pose estimation through an optical flow tracking method, and determining a camera pose; and according to the static feature points, the dynamic pose and the camera pose, performing positioning and map construction.

Description

Technical Field

[0001] The present application relates to the field of positioning and mapping technology, and in particular to a positioning and mapping method for dynamic scenes and related equipment. Background Art

[0002] With the rapid development of artificial intelligence and robotics, mobile robots are increasingly used in the real world, from self-driving cars to industrial automation, from augmented reality to smart inspection. However, the navigation and localization problem of these robots in dynamic environments remains a challenge.

[0003] In the field of navigation and positioning, robots need to be able to detect, track and adapt to these changes in real time to ensure the accuracy of their positioning and mapping. However, the current positioning and mapping technologies are not robust enough, which makes the system prone to matching errors and trajectory drift, and the lack of semantic information leads to little available static semantic information, which affects positioning and mapping. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a positioning and map construction method and related equipment for dynamic scenes.

[0005] Based on the above objectives, the present application provides a method for positioning and mapping a dynamic scene, including:

[0006] Determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine a target object and static feature points;

[0007] Determine a 3D bounding box according to the target object, and determine a dynamic pose according to the 3D bounding box and the RGBD image;

[0008] According to the RGBD image, a depth pyramid and an image pyramid are calculated;

[0009] According to the depth pyramid and the image pyramid, the camera pose is estimated by an optical flow tracking method to determine the camera pose;

[0010] Positioning and map construction are performed according to the static feature points, the dynamic poses and the camera poses.

[0011] Optionally, performing target detection and feature point extraction on the RGBD image to determine the target object and static feature points includes:

[0012] Performing target detection on any of the RGBD images through a target detection network to determine the target object;

[0013] Extracting feature points from any of the target images using a feature point extraction algorithm to determine the dynamic feature points;

[0014] The dynamic feature points are filtered according to the target object to determine the static feature points.

[0015] Optionally, determining a 3D bounding box according to the target object includes:

[0016] Determine a 2D bounding box according to the target detection network; wherein the target detection network is a YOLOv7 target detection network;

[0017] Determine a vanishing point on the RGBD image, and determine at least two predicted 3D bounding boxes based on the vanishing point and the 2D bounding box;

[0018] The at least two predicted 3D bounding boxes are screened to determine the 3D bounding box.

[0019] Optionally, the determining the static feature points by eliminating the dynamic feature points according to the target object includes:

[0020] According to the target object, the dynamic feature points are filtered through a depth map to determine the static feature points.

[0021] Optionally, screening the at least two predicted 3D bounding boxes to determine the 3D bounding box includes:

[0022] Screening at least two predicted 3D bounding boxes by using a loss function as shown below to determine the 3D bounding box; wherein the loss function includes a distance error, an angle alignment error, and a shape error;

[0023] E(O|I)=φ dist (O, I)+w 1 φ angle (O, I)+w 2 φ shape (O)

[0024] Where I is the target image, O is the predicted 3D bounding box, φ dist is the distance error, φ angle is the angular alignment error, φ shape is the shape error, w 1 is the hyperparameter representing the weight of the angle alignment error, w 2 is the hyperparameter representing the weight of the shape error.

[0025] Optionally, the performing positioning and map construction according to the static feature points, the dynamic pose and the camera pose includes:

[0026] Determining a factor graph according to the dynamic pose and the camera pose;

[0027] Determine a descriptor according to the static feature points;

[0028] Solving the factor graph according to the descriptor to determine the current camera pose, the current dynamic object pose, and the current static object pose;

[0029] Positioning and map construction are performed according to the current camera pose, the current dynamic object pose and the current static object pose.

[0030] Based on the same inventive concept, the embodiment of the present application also provides a positioning and map building device for a dynamic scene, including:

[0031] A detection and extraction module is configured to determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine a target object and static feature points;

[0032] a dynamic pose determination module, configured to determine a 3D bounding box according to the target object, and determine a dynamic pose according to the 3D bounding box and the RGBD image;

[0033] A calculation module, configured to calculate a depth pyramid and an image pyramid according to the RGBD image;

[0034] A camera pose determination module is configured to estimate the camera pose by an optical flow tracking method according to the depth pyramid and the image pyramid to determine the camera pose;

[0035] The positioning and map building module is configured to perform positioning and map building according to the static feature points, the dynamic posture and the camera posture.

[0036] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for positioning and mapping a dynamic scene as described in any one of the above is implemented.

[0037] Based on the same inventive concept, an embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned dynamic scene positioning and map construction methods.

[0038] Based on the same inventive concept, an embodiment of the present application also provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes any of the above-mentioned methods for positioning and mapping dynamic scenes.

[0039] From the above, it can be seen that the positioning and mapping method and related equipment for dynamic scenes provided by the present application, the present application uses the optical flow tracking method of the depth map, combines RGB images and RGBD images, establishes image pyramids and depth pyramids, and adaptively adjusts the tracking strategy according to the key points of tracking. It can efficiently track the camera pose, thereby effectively improving the positioning accuracy and robustness of the SLAM system in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of the flow of a method for positioning and mapping a dynamic scene according to an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the execution flow of the method for positioning and mapping a dynamic scene according to an embodiment of the present application;

[0043] Figure 3 A schematic diagram of generating a 3D bounding box according to an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the posture graph structure of an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a pose graph structure between two consecutive frames according to an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of a dynamic scene positioning and map building device according to an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0049] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0050] In order to facilitate understanding of the technical solutions of the present disclosure, some technical terms involved in the present disclosure are introduced below.

[0051] SLAM technology: simultaneous positioning and mapping technology.

[0052] RGBD image: It is composed of a series of RGBD images, each of which contains a normal RGB three-channel color image and a depth image (depth map). RGBD images are usually captured by special sensors such as Kinect, RealSense and other devices, which combine traditional color cameras with depth sensors to generate RGBD images. RGBD images are stored in the form of multi-channel images, and each pixel contains a red, green, blue component and a depth component.

[0053] RGB images are images of various colors generated based on different intensities of the three basic colors red, green, and blue. In RGB images, each pixel is usually composed of three components, representing the brightness values ​​of the three colors red, green, and blue. Among them, traditional color cameras can capture RGB images without the need for additional depth sensors.

[0054] World coordinate system: The absolute coordinate system of a system, used to describe the positions of all other coordinate systems or objects. The world coordinate system is usually set as a reference coordinate system, such as Xw-Yw-Zw, where (xw, yw, zw) represents the coordinates of a spatial point P in the world coordinate system.

[0055] Odometry estimation: methods and techniques for recursively estimating the real-time position, posture and other physical quantities of a robot or other moving object through sensor information.

[0056] Vanishing Point (VP): In perspective projection, all parallel lines intersect at one point on the image, which is called the vanishing point. In three-dimensional space, parallel lines in different directions will have different vanishing points. Vanishing points provide important clues about scene depth information and camera perspective. (Determination of vanishing points: (Image processing-based methods) ① Use edge detection algorithms (such as Canny edge detection) to identify edges in the image, analyze edges, infer parallel lines and find vanishing points, ② Use straight line detection algorithms (such as Hough transform) to identify straight lines in the image, analyze the direction and intersection of the straight lines to determine the vanishing point, ③ Specialized vanishing point detection algorithms (such as NeurVPS neural vanishing point scanning technology) automatically detect vanishing points in the image.

[0057] 2D Bounding Box: This is a rectangular box around the edges of an object on the image plane. It gives the location and size of the object on the 2D image.

[0058] Canny edge detection: a method that uses a multi-level edge detection algorithm to detect image edges. The basic idea is to find the position in the image where the grayscale intensity changes most strongly, that is, the gradient direction) to construct a distance transformation map (the distance transformation is mainly a transformation of a binary image. The grayscale value of the distance image output after the binary image is transformed means: the distance from the pixel to the nearest background point (the background is generally black).

[0059] In order to make the technical solution of the present disclosure clearer and easier to understand, the positioning and map construction method of the dynamic scene provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0060] As mentioned in the background technology section, with the rapid development of artificial intelligence and robotics, mobile robots are increasingly used in the real world, from self-driving cars to industrial automation, from augmented reality to smart inspection. However, the navigation and localization problem of these robots in dynamic environments remains a challenge.

[0061] In the field of navigation and positioning, robots need to be able to detect, track and adapt to these changes in real time to ensure the accuracy of their positioning and mapping. The current positioning and mapping technologies have problems such as insufficient robustness, which makes the system prone to matching errors and trajectory drift, and insufficient use of semantic information, which results in little available static semantic information, affecting positioning and mapping.

[0062] In practical applications, dynamic environments such as urban traffic, indoor robot navigation, drone operations, and agricultural robots require SLAM systems to effectively handle dynamic objects. Therefore, the robot needs to be able to detect, track, and adapt to these changes in real time to ensure the accuracy of its positioning and mapping.

[0063] In view of this, the embodiments of the present application provide a method, device, electronic device, storage medium and program product for positioning and mapping of dynamic scenes, including odometer estimation, local mapping and loop detection. In the odometer estimation, three operations of extracting feature points, detecting targets and building image pyramids are performed on the RGBD image at the same time. In the dynamic object fitting module, the 3D bounding box of the object in the world coordinate system is calculated based on the 2D bounding box obtained by target detection. The extracted feature points can be preliminarily classified in combination with target detection. Furthermore, in the local mapping part, for the dynamic feature points in the target frame, the RANSAC method of the joint depth map is used to classify the dynamic and static feature points in the target bounding box, thereby eliminating the dynamic feature points in the entire image, and further preliminarily estimating the camera pose based on the static feature points. Furthermore, in the loop detection part, the global pose is optimized based on the camera pose, static feature points and dynamic pose.

[0064] This application uses the optical flow tracking method of the depth map, combines RGB images and RGBD images, establishes image pyramids and depth pyramids, and adaptively adjusts the tracking strategy according to the key points of tracking. It can efficiently track the camera pose, thereby effectively improving the accuracy and robustness of SLAM system positioning in dynamic environments.

[0065] like Figure 1 As shown, the positioning and map construction method of the dynamic scene includes:

[0066] Step S102: determining an RGBD image, performing target detection and feature point extraction on the RGBD image, and determining a target object and static feature points;

[0067] Step S104: determining a 3D bounding box according to the target object, and determining a dynamic pose according to the 3D bounding box and the RGBD image;

[0068] Step S106, calculating a depth pyramid and an image pyramid according to the RGBD image;

[0069] Step S108, estimating the camera pose by optical flow tracking method according to the depth pyramid and the image pyramid, and determining the camera pose;

[0070] Step S110: performing positioning and map construction according to the static feature points, the dynamic poses and the camera poses.

[0071] Step S102: First, obtain an RGBD image or an RGBD image sequence, such as Figure 2As shown in the figure, after obtaining an RGBD image or an RGBD image sequence, feature point extraction and target detection are performed on the obtained RGBD image at the same time, and then the bounding box (i.e., 2D bounding box) of the prior dynamic target category is eliminated using the depth-based RANSAC method according to the target detection result. Among them, feature point extraction uses the FAST feature point extraction method to determine static feature points; target detection uses the YOLOv7 network to determine the target object.

[0072] Specifically, step S102 includes the following steps:

[0073] Step (1) Feature point classification: All feature points in a certain bounding box are recorded as a set S. According to the depth information, these feature points are divided into two categories: feature points on the surface of the target object, i.e., dynamic feature points, and feature points on the static background, i.e., static feature points.

[0074] Step (2) loop processing: perform k cycles, each cycle comprising the following steps:

[0075] a. Select depth difference calculation points: arbitrarily select two feature points and calculate their depth difference; for example, the depth values ​​of the two feature points are z1 and z2, then the depth difference is Δz = z1-z2.

[0076] b. Traverse and filter feature points: Traverse all feature points in the set S and find feature points whose depth difference is within an acceptable range.

[0077] c. Classify inliers and outliers: The feature points that meet the depth difference requirements are regarded as inliers, and the other feature points are regarded as outliers. Inliers are dynamic feature points.

[0078] Step (3) Determine the dynamic point set: After k cycles, select the internal point-dynamic point set containing the largest number of feature points as the dynamic point set. Then distinguish the feature points on the surface of the target object from the feature points of the static background, and remove or filter the static feature points to determine the dynamic feature points.

[0079] By combining the depth map with the target detection results, the RANSAC method is used to finely remove the feature points of dynamic objects.

[0080] In step S104, a 3D bounding box is determined according to the target object, and the 3D bounding box is determined by two steps: 3D bounding box generation and 3D bounding box screening. Specifically, first, a 2D bounding box is determined according to the target detection network, and the object in the 2D bounding box can be used as the target object; then, a vanishing point on the RGBD image is determined, and at least two predicted 3D bounding boxes are determined according to the vanishing point and the 2D bounding box; finally, the at least two predicted 3D bounding boxes are screened to determine the 3D bounding box. The target detection network is a YOLOv7 network.

[0081] In some optional implementations, for 3D bounding box generation, the vanishing point VP and the 2D bounding box are first determined, and the 8 corner points of the 3D bounding box can be obtained based on the vanishing point VP and the 2D bounding box, and then the 3D bounding box of the dynamic object is calculated. The vanishing point VP is the intersection of the parallel lines projected from the 3D bounding box to the perspective view. Among them, the 3D cube box O can be expressed as 9 degrees of freedom parameters, and the position matrix t = [t x , t y , t z ], dimension matrix d = [t x , t y , t z ] and the rotation matrix R. A 3D cuboid has three orthogonal axes. According to the camera parameter K and the object's rotation matrix R, three VPs can be formed after projection, as shown in the following formula:

[0082] VP i =KR col (i),i∈{1,2,3}

[0083] Among them, K is the camera parameter, as the calibration matrix, R col(i) is the i-th column of R, i∈{1,2,3}.

[0084] like Figure 3 As shown in , the situations of observing a cube in a 2D image can be divided into three categories. At most, three faces of a cube can be observed at the same time, such as Figure 3 As shown in (1). If three VP points and point 1 are known, then And so on to get the remaining points.

[0085] Furthermore, since the pitch and roll angles of dynamic objects on the ground are 0, we can directly reverse-project the vertices on the ground onto the 3D ground plane, and then calculate the positions of other vertical vertices based on this method. For example, the vertex 5 on the 3D ground plane can be expressed as [n,m] (vector and distance in the camera frame), and the corresponding 3D vertex P5 is the reverse-projected ray K -1 p 5The intersection point with the ground plane is shown in the following formula, where n and m represent the vector and distance of vertex 5 on the three-dimensional ground plane in the camera frame, K represents the camera parameters, and p 5 represents the position of vertex 5 in the camera frame, and P5 represents the position of vertex 5 on the three-dimensional ground plane.

[0086]

[0087] For the 3D bounding box screening part, the obtained 3D bounding box proposals are screened by defining a loss function. The loss function consists of three parts: distance error, angle alignment error, and shape error. The loss function is shown in the following formula:

[0088] E(O|I)=φ dist (O, I)+w 1 φ angle (O, I)+w 2 φ shape (O)

[0089] Where I represents the image, O represents the 3D bounding box, and φ dist Represents the distance error, φ angle represents the angular alignment error, φ angle represents the shape error, w 1 and w 2 As a hyperparameter, it represents the weight of the angle alignment error and the shape error.

[0090] In some embodiments, for distance error, angle alignment error and shape error, it should be noted that:

[0091] For distance error: Since the cuboid edge of the 3D bounding box projected on the 2D image should match the edge of the object in the image, the 3D bounding box is constrained according to the distance between the two. First, Canny edge detection is performed on the 3D bounding box to construct a distance transformation map. Then, 10 points are uniformly sampled for each visible cuboid edge, and the distance map values ​​of all points are calculated and normalized by dividing by the diagonal length of the 2D bounding box.

[0092] For angular alignment error: Since the distance error is very sensitive to false positive edge noise (such as surface texture of objects), it is also necessary to detect long line segments and measure whether they are aligned with the vanishing points VP calculated during 3D bounding box generation. The angular error is defined as the error between the angle of the edge (composed of 2D points) forming the vanishing point and the angle of the corresponding edge obtained by line detection, as shown in the following formula. For each VP i We can find the two outermost line segments with the minimum and maximum slopes, denoted as <l i_ms , l i_mt > and <l i_ns , l i_nt>,<a,b> It represents the slope angle of the line segment with endpoints a and b.

[0093]

[0094] Where I represents the image, O represents the 3D bounding box, || is the Euclidean norm of the vector, and l i_ms is the starting point of the line segment with the minimum slope, l i_mt is the end point of the line segment with the minimum slope, l i_ns is the starting point of the line segment with large slope, l i_nt is the end point of the line segment with a large slope, <VP i , l i_nt > is the slope of the line connecting the vanishing point i and the end point of the maximum slope segment, <VP i , l i_mt > is the slope of the line connecting the vanishing point i and the end point of the minimum slope segment.

[0095] For shape error: Since similar 2D cuboid corners may generate completely different 3D cuboids, a shape error function is added to filter cubes with abnormal skew ratios (s = length / width), as shown in the following formula:

[0096] φ shape (O) = max(s - σ, 0)

[0097] Wherein, σ represents a preset abnormal skew ratio threshold, and s represents the skew ratio.

[0098] In step S106, according to the RGBD image, the depth pyramid and the image pyramid are calculated as follows: optical flow tracking adds depth information to assist decision making based on the LK pyramid optical flow algorithm.

[0099] Among them, the image pyramid is a kind of multi-scale expression of images. It is an effective but simple structure to explain images at multiple resolutions. The layers of images are likened to a pyramid. The higher the level, the smaller the image and the lower the resolution. The deep pyramid is a deep learning model based on the pyramid structure, which extracts information from different scales or resolutions of the image. In the deep pyramid, the bottom is the high-resolution representation of the image to be processed, and the top is the low-resolution representation.

[0100] It should be noted that the RGBD image consists of four levels. When determining the image pyramid, the image pyramid is determined based on the first three levels, namely RGB, in the RGBD image; when determining the depth pyramid, the depth pyramid is determined based on the four levels in the RGBD image.

[0101] In some embodiments, the size of the pyramid is reduced from high to low each time to 1 / 2 of the previous layer, and the optical flow is calculated from high to low. The calculation result of the previous layer is used as the initial estimation of the current layer. Only the residual is calculated each time, and the residual is accumulated to obtain the final optical flow. For the camera pose estimation based on the depth pyramid and the image pyramid, the optical flow tracking method is used to determine the camera pose, such as Figure 2 As shown, 3D key points are first tracked in the two layers with high resolution of the image pyramid. If the proportion of the number of 3D key points that fail to be tracked is greater than 2 / 3, all 2D and 3D key points are tracked in all layers of the image pyramid, and the initial position of the 3D key point is the position in the previous frame.

[0102] Otherwise, track the 3D key points that failed to be tracked in the previous step in the first two layers of the depth pyramid, and determine whether the proportion of the number of 3D key points that failed to be tracked is still greater than 2 / 3. If so, track all 2D and 3D key points in all layers of the image pyramid, otherwise only track the 3D key points that failed to be tracked in the previous step. Finally, remove the points that failed to be tracked.

[0103] In some embodiments, in order to obtain a unified static global reference system, for each key frame, further pose optimization is performed according to the pose graph. Therefore, positioning and map construction are performed according to the static feature points, the dynamic poses and the camera poses.

[0104] Further, such as Figure 4 The following is a schematic diagram of a simple pose graph structure, where the nodes represent global static pose estimates. Edges represent the relative position between two nodes (special Euclidean group) Therefore, the figure has the formula The restriction relationship can be further obtained as follows The equation.

[0105] Among them, changing the transformation order does not affect the consistency of the ring in the pose graph, so the formula It can be further written as:

[0106]

[0107] In some embodiments, Figure 5 The figure shows the pose graph of two consecutive frames. According to the front-end odometer estimation, the pose of the camera at time t-1 and time t in the global static coordinate system W can be obtained. and Based on this, the dynamic object fitting module can obtain the position and posture of the dynamic object V at time t-1 and time t in the global static coordinate system W. and Camera-Camera side It can be obtained by odometer (device and technology used to measure the distance traveled by objects, especially vehicles), camera-object edge The object-object edge can be obtained by combining the depth map and the odometer estimation. According to the formula The loss function γ of the four edges can be obtained cc , γ cv (t-1), γ cv (t) and γ vv , the specific calculation method is shown in the following formula:

[0108]

[0109] according to Figure 5 The consistency relationship of the ring in , the loss function has the following relationship:

[0110] Υ=Υ cc ×Υ cv(t) ×(Υ vv ) -1 ×(Υ cv(t-1) ) -1

[0111] Combining the loss function of the four edges cc , Υ cv(t-1) , Υ cb(t) , Υ vv And the calculation formula of the loss function Υ, we can further obtain the following constraint relationship:

[0112]

[0113] By solving the pose graph optimization problem constructed above, the optimized poses of the camera and dynamic and static objects can be obtained, and further positioning and map construction can be performed. By fitting dynamic and static objects to extract semantic information, the pose solution is incorporated into the graph optimization problem, and the semantics of dynamic and static objects in the scene can be extracted and fully utilized.

[0114] From the above, it can be seen that the positioning and mapping method of dynamic scenes provided by the present application, by using the optical flow tracking method of the depth map, combines RGB images and RGBD images, establishes image pyramids and depth pyramids, and adaptively adjusts the tracking strategy according to the key points of tracking. It can efficiently track the camera pose, thereby effectively improving the positioning accuracy and robustness of the SLAM system in dynamic environments.

[0115] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0116] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a positioning and map building device for dynamic scenes.

[0118] refer to Figure 6 , the dynamic scene positioning and map building device comprises:

[0119] The detection and extraction module 602 is configured to determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine a target object and static feature points;

[0120] A dynamic pose determination module 604 is configured to determine a 3D bounding box according to the target object, and determine a dynamic pose according to the 3D bounding box and the RGBD image;

[0121] A calculation module 606 is configured to calculate a depth pyramid and an image pyramid according to the RGBD image;

[0122] A camera pose determination module 608 is configured to estimate the camera pose by an optical flow tracking method according to the depth pyramid and the image pyramid to determine the camera pose;

[0123] The positioning and map building module 610 is configured to perform positioning and map building according to the static feature points, the dynamic poses and the camera poses.

[0124] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0125] The device of the above embodiment is used to implement the positioning and map construction method of the corresponding dynamic scene in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0126] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for positioning and mapping the dynamic scene described in any of the above embodiments is implemented.

[0127] Figure 7 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0128] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0129] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0130] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0131] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0132] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0133] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0134] The electronic device of the above-mentioned embodiment is used to implement the positioning and map construction method of the corresponding dynamic scene in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the positioning and mapping method of the dynamic scene as described in any of the above embodiments.

[0136] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0137] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the positioning and map construction method of the dynamic scene as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0138] Based on the same inventive concept, corresponding to the method for positioning and mapping a dynamic scene described in any of the above embodiments, the present disclosure further provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the method for positioning and mapping a dynamic scene. Corresponding to the execution subject corresponding to each step in each embodiment of the method for positioning and mapping a dynamic scene, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0139] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the positioning and map construction method of the dynamic scene as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0140] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0141] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0142] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0143] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for positioning and mapping a dynamic scene, characterized in that: include: Determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine a target object and static feature points; Determine a 3D bounding box according to the target object, and determine a dynamic pose according to the 3D bounding box and the RGBD image; Calculate a depth pyramid and an image pyramid according to the RGBD image; According to the depth pyramid and the image pyramid, the camera pose is estimated by an optical flow tracking method to determine the camera pose; Positioning and map construction are performed according to the static feature points, the dynamic poses and the camera poses.

2. The method according to claim 1, characterized in that The performing target detection and feature point extraction on the RGBD image to determine the target object and static feature points includes: Performing target detection on any of the RGBD images through a target detection network to determine the target object; Extracting feature points from any of the target images using a feature point extraction algorithm to determine the dynamic feature points; The dynamic feature points are filtered according to the target object to determine the static feature points.

3. The method according to claim 2, characterized in that Determining a 3D bounding box according to the target object includes: Determine a 2D bounding box according to the target detection network; wherein the target detection network is a YOLOv7 target detection network; Determine a vanishing point on the RGBD image, and determine at least two predicted 3D bounding boxes based on the vanishing point and the 2D bounding box; The at least two predicted 3D bounding boxes are screened to determine the 3D bounding box.

4. The method according to claim 2, characterized in that: The step of eliminating the dynamic feature points according to the target object to determine the static feature points includes: According to the target object, the dynamic feature points are filtered through a depth map to determine the static feature points.

5. The method according to claim 3, characterized in that: The screening of the at least two predicted 3D bounding boxes to determine the 3D bounding box includes: Screening at least two predicted 3D bounding boxes by using a loss function as shown below to determine the 3D bounding box; wherein the loss function includes a distance error, an angle alignment error, and a shape error; E(O|I)=φ dist (O,I)+w1ϕ angle (O,I)+w2ϕ shape (THE) Where I is the target image, O is the predicted 3D bounding box, φ dist is the distance error, φ angle is the angular alignment error, φ shape is the shape error, w1 is the hyperparameter representing the weight of the angle alignment error, and w2 is the hyperparameter representing the weight of the shape error.

6. The method according to claim 1, characterized in that The performing positioning and map construction according to the static feature points, the dynamic pose and the camera pose comprises: Determining a factor graph according to the dynamic pose and the camera pose; Determine a descriptor according to the static feature points; Solving the factor graph according to the descriptor to determine the current camera pose, the current dynamic object pose, and the current static object pose; Positioning and map construction are performed according to the current camera pose, the current dynamic object pose and the current static object pose.

7. A positioning and map building device, characterized in that: include: A detection and extraction module is configured to determine an RGBD image, perform target detection and feature point extraction on the RGBD image, and determine a target object and static feature points; a dynamic pose determination module, configured to determine a 3D bounding box according to the target object, and determine a dynamic pose according to the 3D bounding box and the RGBD image; A calculation module, configured to calculate a depth pyramid and an image pyramid according to the RGBD image; A camera pose determination module is configured to estimate the camera pose by an optical flow tracking method according to the depth pyramid and the image pyramid to determine the camera pose; The positioning and map building module is configured to perform positioning and map building according to the static feature points, the dynamic posture and the camera posture.

8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.

Images