Method, device and electronic device for simultaneous positioning and mapping in dynamic scenes

Abstract

The present invention provides a method, device and electronic device for simultaneous positioning and mapping in a dynamic scene. In the method, a lightweight LSS target detection network is used for target detection, which has good real-time performance. When determining the relationship between a target detection box and a tracking bounding box, only the association cost of each tracking bounding box of the target detection box and its k nearest neighbor tracking bounding boxes is calculated, which greatly reduces the time complexity and improves the real-time performance of simultaneous positioning and mapping. In addition, after obtaining a static map, a statistical method is also used to optimize the static map, thereby improving the accuracy of the target static map. That is, the method for simultaneous positioning and mapping in a dynamic scene of the present invention has good real-time performance and high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device and electronic device for simultaneous positioning and mapping in a dynamic scene. Background Art

[0002] In challenging dynamic environments, simultaneous localization and mapping (SLAM) is crucial for vehicles to achieve safe navigation. To achieve this goal, it is necessary to separate dynamic and static objects with high efficiency and accuracy, and use the obtained robust static object feature points for real-time localization and mapping. To achieve this goal, existing methods usually use the method of detection-tracking-elimination of dynamic objects-static object localization and mapping, but they often lack real-time performance and accuracy, which makes it difficult to perform real-time navigation and accurate static map construction.

[0003] In summary, in dynamic scenes, how to efficiently and accurately separate dynamic and static objects to achieve simultaneous positioning and mapping has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a method, device and electronic device for simultaneous positioning and mapping in dynamic scenes, so as to alleviate the technical problem that the existing methods for simultaneous positioning and mapping in dynamic scenes lack real-time performance and accuracy.

[0005] In a first aspect, an embodiment of the present invention provides a method for simultaneous positioning and mapping in a dynamic scene, comprising:

[0006] Use the LSS target detection network to detect the target in the current frame image and obtain the target detection frame;

[0007] Determine k tracking bounding boxes that are the nearest neighbors to each of the target detection boxes, and calculate the association cost between each of the target detection boxes and each of the tracking bounding boxes corresponding to it, and then determine the target tracking bounding box associated with each of the target detection boxes in the tracking bounding box according to the association cost, wherein k is a preset value, and the tracking bounding box is a target detection box obtained after performing target detection on the previous frame image and then obtained by Kalman filtering;

[0008] Determine the static or dynamic state of the target in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith, and select the target detection frame corresponding to the static target and the target tracking bounding frame of the static target associated therewith according to the static or dynamic state of each target;

[0009] Performing positioning and mapping based on the target detection box corresponding to the static target and the target tracking bounding box of the static target associated with the static target, to obtain a static map constructed according to the static target;

[0010] A statistical method is used to determine the dynamic and static targets in the static map, and the static map is optimized according to the dynamic and static targets in the static map to obtain a target static map.

[0011] Further, the association cost includes: a Euclidean distance cost between bounding box centers, an intersection-over-union cost of bounding boxes, and a volume cost of bounding boxes, and determining a target tracking bounding box associated with each of the target detection boxes in the tracking bounding box according to the association cost includes:

[0012] The tracking bounding box corresponding to the minimum association cost is used as the target tracking bounding box associated with each of the target detection boxes.

[0013] Further, determining the dynamic and static states of the targets in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith includes:

[0014] The dynamic and static states of the targets in each target detection frame are determined based on the distance between the center of each target detection frame and the center of the target tracking bounding box associated therewith.

[0015] Further, determining the dynamic and static state of the target in each target detection frame based on the distance between the center of each target detection frame and the center of the target tracking bounding frame associated therewith includes:

[0016] If the distance is greater than a preset threshold, determining that the target in the corresponding target detection frame is a dynamic target;

[0017] If the distance is not greater than the preset threshold, it is determined that the target in the corresponding target detection frame is a static target.

[0018] Furthermore, a statistical method is used to determine the dynamic and static targets in the static map, including:

[0019] Statistically calculating the sum of proximity values ​​of the target detection frame corresponding to each of the static targets and the target tracking boundary frames of each of the static targets within a first preset time range to obtain a first proximity value sum;

[0020] Statistically calculating the sum of proximity values ​​between the target detection frame corresponding to each of the static targets and the target tracking bounding frame of each of the static targets within a second preset time range, to obtain a second proximity value sum, wherein if the Euclidean distance between the center of the target detection frame corresponding to one of the static targets and the center of the target tracking bounding frame of the static target is less than a preset distance threshold, the proximity value between the center of the target detection frame corresponding to the static target and the center of the target tracking bounding frame of the static target is 1, otherwise, the proximity value is 0;

[0021] If the first approximation value sum corresponding to the target detection frame corresponding to the static target is less than the second approximation value sum, determining that the target in the corresponding target detection frame is a dynamic target;

[0022] If the sum of the first approximation values ​​corresponding to the target detection frame corresponding to a static target is not less than the sum of the second approximation values, the target in the corresponding target detection frame is determined to be a static target, and then the dynamic and static states of each target in the static map are obtained.

[0023] Further, the static map is optimized according to the dynamic and static targets in the static map, including:

[0024] The dynamic target in the static map is deleted to obtain the target static map.

[0025] Further, statistically calculating the sum of proximity values ​​between the target detection frame corresponding to each of the static targets and the target tracking bounding frames of each of the static targets within the first preset time range includes:

[0026] According to the first calculation formula ∈∈[-τ d , τ d ] Statistically calculate the sum of the proximity values ​​of the target detection box corresponding to each of the static targets and the target tracking bounding box of each of the static targets within the first preset time range, where N dyn represents the first approximation value and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a first preset time range, represents the proximity between the center of the target detection box corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the first preset time range, ∈ represents the time difference, τ d Indicates the first preset duration, d represents the preset distance threshold;

[0027] Statistically calculating the sum of proximity values ​​of the target detection frame corresponding to each of the static targets and the target tracking bounding boxes of each of the static targets within a second preset time range, including:

[0028] According to the second calculation formula ∈∈[0,-τ s ],[τ s ,τ the ] Statistically calculate the sum of the proximity values ​​of the target detection box corresponding to each of the static targets and the target tracking bounding boxes of each of the static targets within the second preset time range, where Nstat represents the second approximation value and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a second preset time range, represents the proximity between the center of the target detection frame corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the second preset time range, ∈ represents the time difference, τ s represents the second preset duration, τ the Indicates the third preset duration, d represents the preset distance threshold.

[0029] In a second aspect, an embodiment of the present invention further provides a device for simultaneous positioning and mapping in a dynamic scene, including:

[0030] The target detection unit is used to perform target detection on the current frame image using the LSS target detection network to obtain a target detection frame;

[0031] A determination and calculation unit, configured to determine k tracking bounding boxes that are the nearest neighbors of each of the target detection boxes, and calculate the association cost between each of the target detection boxes and each of the tracking bounding boxes corresponding to the target detection boxes, and then determine a target tracking bounding box associated with each of the target detection boxes in the tracking bounding boxes according to the association cost, wherein k is a preset value, and the tracking bounding box is a target detection box obtained after performing target detection on the previous frame image and then obtained by Kalman filtering;

[0032] A determination and screening unit, configured to determine the static or dynamic state of the target in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith, and to screen the target detection frame corresponding to the static target and the target tracking bounding frame of the static target associated therewith according to the static or dynamic state of each target;

[0033] A mapping unit, configured to perform positioning and mapping based on a target detection frame corresponding to the static target and a target tracking bounding frame of the static target associated therewith, to obtain a static map constructed according to the static target;

[0034] The optimization unit is used to determine the dynamic and static targets in the static map by using a statistical method, and optimize the static map according to the dynamic and static targets in the static map to obtain a target static map.

[0035] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute any method described in the first aspect above.

[0037] In an embodiment of the present invention, a method for simultaneous positioning and mapping in a dynamic scene is provided, comprising: performing target detection on a current frame image using an LSS target detection network to obtain a target detection frame; determining k tracking boundary frames that are the nearest neighbors of each target detection frame, and calculating the association cost between each target detection frame and each corresponding tracking boundary frame, and then determining a target tracking boundary frame associated with each target detection frame in the tracking boundary frame according to the association cost, wherein k is a preset value, and the tracking boundary frame is a target detection frame obtained after performing target detection on a previous frame image and then obtained by Kalman filtering; determining the dynamic and static state of the target in each target detection frame according to each target detection frame and the target tracking boundary frame associated with it, and screening the target detection frame corresponding to the static target and the target tracking boundary frame of the static target associated with it according to the dynamic and static state of each target; performing positioning and mapping based on the target detection frame corresponding to the static target and the target tracking boundary frame of the static target associated with it, and obtaining a static map constructed according to the static target; determining the dynamic and static targets in the static map using a statistical method, and optimizing the static map according to the dynamic and static targets in the static map, and obtaining a target static map. From the above description, it can be seen that in the method for simultaneous positioning and mapping in dynamic scenes of the present invention, a lightweight LSS target detection network is used for target detection, which has good real-time performance. When determining the relationship between the target detection box and the tracking bounding box, only the association cost of each tracking bounding box of the target detection box and its k nearest neighbor tracking bounding boxes is calculated, which greatly reduces the time complexity and improves the real-time performance of simultaneous positioning and mapping. In addition, after obtaining the static map, a statistical method is also used to optimize the static map, thereby improving the accuracy of the target static map. That is, the method for simultaneous positioning and mapping in dynamic scenes of the present invention has good real-time performance and high accuracy, which alleviates the technical problem that the existing methods for simultaneous positioning and mapping in dynamic scenes lack real-time performance and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a method for simultaneous positioning and mapping in a dynamic scene provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a device for simultaneous positioning and mapping in a dynamic scene provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Traditional methods of simultaneous positioning and mapping in dynamic scenes lack real-time performance and accuracy.

[0044] Based on this, in the method for simultaneous positioning and mapping in dynamic scenes of the present invention, a lightweight LSS target detection network is used for target detection, which has good real-time performance. When determining the relationship between the target detection box and the tracking bounding box, only the association cost of each tracking bounding box of the target detection box and its k nearest neighbor tracking bounding boxes is calculated, which greatly reduces the time complexity and improves the real-time performance of simultaneous positioning and mapping. In addition, after obtaining the static map, a statistical method is also used to optimize the static map, thereby improving the accuracy of the target static map. That is, the method for simultaneous positioning and mapping in dynamic scenes of the present invention has good real-time performance and high accuracy.

[0045] To facilitate understanding of this embodiment, a method for simultaneous positioning and mapping in a dynamic scene disclosed in an embodiment of the present invention is first introduced in detail.

[0046] Embodiment 1:

[0047] According to an embodiment of the present invention, an embodiment of a method for simultaneous positioning and mapping in a dynamic scene is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Figure 1 is a flow chart of a method for simultaneous positioning and mapping in a dynamic scene according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0049] Step S102, using the LSS target detection network to perform target detection on the current frame image to obtain a target detection frame;

[0050] Specifically, in order to achieve simultaneous positioning and mapping in real-time dynamic scenes, a lightweight 3D target detection is first required. Specifically, the present invention uses an LSS target detection network to complete this function.

[0051] Step S104, determining k nearest neighboring tracking bounding boxes to each target detection box, and calculating the association cost between each target detection box and each corresponding tracking bounding box, and then determining the target tracking bounding box associated with each target detection box in the tracking bounding box according to the association cost, wherein k is a preset value, and the tracking bounding box is the target detection box obtained after target detection on the previous frame image and then obtained by Kalman filtering;

[0052] Specifically, the LSS target detection network is used to perform target detection on the previous frame image, and the obtained target detection frame is then passed through the Kalman filter to obtain the tracking boundary frame.

[0053] In the traditional solution (traditional tracking algorithm), when associating the target detection box with the tracking bounding box, the cost matrix C = N(t)×N(t-1), specifically: the cost matrix C=N(t)×N(t-1), is calculated based on all N(t) target detection boxes (t represents time t) and N(t-1) tracking bounding boxes, that is, the cost of associating each target detection box with all tracking bounding boxes. However, calculating the cost matrix of size N(t)×N(t-1) each time is a very time-consuming task, requiring at least O(N(t)×N(t-1)) time complexity.

[0054] To solve the above problems, the present invention proposes an efficient hierarchical association cost matrix, which reduces the time complexity of tracking associations from O(N 2 ) is greatly reduced to O(N).

[0055] The inventor assumes that in adjacent time frames, the center points of the target detection frames of the same target (i.e., the target detection frame and tracking bounding box of the same target) will be very close in distance. Therefore, only the k tracking bounding boxes that are the nearest neighbors to each target detection frame are selected as association candidates, and then the association cost of each target detection frame and each corresponding tracking bounding box is calculated, and the target tracking bounding box associated with each target detection frame is determined in the tracking bounding box according to the association cost. This hierarchical calculation only needs to compare N(t)×min(N(t-1), k) bounding boxes, which greatly reduces the time complexity of the calculation to O(kN(t)), and at the same time realizes the matching association of all target detection boxes and target tracking bounding boxes. The following table compares the performance of the traditional association algorithm and the association algorithm of the present invention:

[0056] method Frame rate sec / frame <![CDATA[O(N 2 ) matches]]> 360 0.307 O(N) matching 360 0.029

[0057] When determining the k nearest neighbor tracking bounding boxes, the k nearest neighbor tracking bounding boxes to each target detection box may be determined according to the distance between the center of the target detection box and the center of each tracking bounding box.

[0058] Step S106, determining the static or dynamic state of the target in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith, and filtering out the target detection frame corresponding to the static target and the target tracking bounding frame of the static target associated therewith according to the static or dynamic state of each target;

[0059] Step S108, positioning and mapping are performed based on the target detection box corresponding to the static target and the target tracking bounding box of the static target associated therewith, to obtain a static map constructed according to the static target;

[0060] Specifically, after obtaining the target detection box corresponding to the static target and the target tracking bounding box of the static target associated with it, the association relationship between the static target in the current frame image and the static target in the target tracking bounding box is obtained, and then the feature points of the static target in the current frame image can be extracted, and real-time positioning and mapping can be performed according to the ORB SLAM2 algorithm, thereby obtaining a static map constructed according to the static target.

[0061] Step S110, using a statistical method to determine dynamic and static targets in the static map, and optimizing the static map according to the dynamic and static targets in the static map to obtain a target static map.

[0062] In traditional tracking methods, judging whether a target is dynamic or static only based on the length of the tracked target trajectory may lead to many misjudgments. When tracking fails, many false negative points may appear, that is, the tracking may miss a moving target and think it is stationary; when detection fails, many false positive points may appear, that is, detection errors, causing the system to mistakenly think that a stationary target is moving. In order to address the above problems, the present invention proposes a statistical-based method, which can focus on the temporal and spatial differences between the tracked dynamic targets and static targets to accurately distinguish between dynamic targets and static targets, and then optimize the static map to obtain a target static map.

[0063] In an embodiment of the present invention, a method for simultaneous positioning and mapping in a dynamic scene is provided, comprising: performing target detection on a current frame image using an LSS target detection network to obtain a target detection frame; determining k tracking boundary frames that are the nearest neighbors of each target detection frame, and calculating the association cost between each target detection frame and each corresponding tracking boundary frame, and then determining a target tracking boundary frame associated with each target detection frame in the tracking boundary frame according to the association cost, wherein k is a preset value, and the tracking boundary frame is a target detection frame obtained after performing target detection on a previous frame image and then obtained by Kalman filtering; determining the dynamic and static state of the target in each target detection frame according to each target detection frame and the target tracking boundary frame associated with it, and screening the target detection frame corresponding to the static target and the target tracking boundary frame of the static target associated with it according to the dynamic and static state of each target; performing positioning and mapping based on the target detection frame corresponding to the static target and the target tracking boundary frame of the static target associated with it, and obtaining a static map constructed according to the static target; determining the dynamic and static targets in the static map using a statistical method, and optimizing the static map according to the dynamic and static targets in the static map, and obtaining a target static map. From the above description, it can be seen that in the method for simultaneous positioning and mapping in dynamic scenes of the present invention, a lightweight LSS target detection network is used for target detection, which has good real-time performance. When determining the relationship between the target detection box and the tracking bounding box, only the association cost of each tracking bounding box of the target detection box and its k nearest neighbor tracking bounding boxes is calculated, which greatly reduces the time complexity and improves the real-time performance of simultaneous positioning and mapping. In addition, after obtaining the static map, a statistical method is also used to optimize the static map, thereby improving the accuracy of the target static map. That is, the method for simultaneous positioning and mapping in dynamic scenes of the present invention has good real-time performance and high accuracy, which alleviates the technical problem that the existing methods for simultaneous positioning and mapping in dynamic scenes lack real-time performance and accuracy.

[0064] The above content briefly introduces the method for simultaneous positioning and mapping in dynamic scenes of the present invention. The specific contents involved are described in detail below.

[0065] In an optional embodiment of the present invention, the association cost includes: the Euclidean distance cost between the centers of the bounding boxes, the intersection-over-union cost of the bounding boxes, and the volume cost of the bounding boxes. According to the association cost, the target tracking bounding box associated with each target detection box is determined in the tracking bounding box, specifically including the following steps:

[0066] The tracking bounding box corresponding to the minimum association cost is used as the target tracking bounding box associated with each target detection box.

[0067] Specifically, the calculation formula for associated costs is: C ij =c d +c o +c v , where C ij represents the associated cost, c d represents the Euclidean distance cost between the bounding box centers, c o represents the intersection-over-union cost of the bounding box, c v represents the volume cost of the bounding box, c d =||c i -c j ||2, c i represents the center of the target detection box, c j represents the center of the tracking bounding box, c o =1-IoU(b i , b j ), b i represents the target detection box, b j represents the tracking bounding box, v i Represents the volume of the target detection box, v j Represents the volume of the tracking bounding box.

[0068] An association cost is calculated between a target detection box and each of the k tracking bounding boxes. In this way, k association costs can be calculated between a target detection box and the k tracking bounding boxes. The tracking bounding box corresponding to the minimum association cost is used as the target tracking bounding box associated with the target detection box. In this way, the target tracking bounding boxes associated with each target detection box can be obtained.

[0069] In an optional embodiment of the present invention, determining the dynamic and static state of the target in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith specifically includes the following steps:

[0070] The dynamic and static states of the targets in each target detection frame are determined based on the distance between the center of each target detection frame and the center of the target tracking bounding box associated therewith.

[0071] Specifically, if the distance is greater than a preset threshold, the target in the corresponding target detection frame is determined to be a dynamic target; if the distance is not greater than the preset threshold, the target in the corresponding target detection frame is determined to be a static target.

[0072] Dynamic objects will move between adjacent time frames, but their trajectories (i.e., their positions) are usually continuous and related within these adjacent time frames. For example, a car will have a certain displacement between the previous second and the next second, and these displacements are connected in space, so in adjacent time frames, its trajectory can be seen to have obvious movement. That is, the trajectory of dynamic objects mainly shows strong correlation in the adjacent areas of adjacent time frames. When implemented, if the distance is greater than the preset threshold, the target in the corresponding target detection box is determined to be a dynamic target.

[0073] The position of static objects does not change significantly over time. They remain in the same position in multiple time frames. That is, the trajectory of static objects maintains a consistent association over time. When implemented, if the distance is not greater than the preset threshold, the target in the corresponding target detection frame is determined to be a static target.

[0074] In an optional embodiment of the present invention, a statistical method is used to determine dynamic and static targets in a static map, which specifically includes the following steps:

[0075] (1) Statistically calculating the sum of proximity values ​​of the target detection box corresponding to each static target and the target tracking bounding box of each static target within a first preset time range to obtain a first proximity value sum;

[0076] Specifically, the target detection frame corresponding to the static target may be obtained by performing target detection on an image acquired at a certain time after a static target is just detected.

[0077] When implemented, according to the first general calculation formula ∈∈[-τ d ,τ d ] Statistically calculate the sum of the proximity values ​​of the target detection box corresponding to each static target and the target tracking bounding box of each static target within the first preset time range, where N dyn represents the first approximation and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a first preset time range, represents the proximity between the center of the target detection box corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the first preset time range, ∈ represents the time difference, τ d Indicates the first preset duration, d represents the preset distance threshold;

[0078] The first preset time range is specifically t-τ d to t+τ d time range, d represents the preset distance threshold at which two bounding boxes (a target detection box corresponding to a static target and a target tracking bounding box of a static target) are considered to be the same static target. dyn represents the first approximation value and, that is, the dynamic count. In layman's terms, it represents the change in the center position of the object (i.e., the target) at multiple time points. The above formula indicates that the movement of the target will be checked within a small time range, indicating that the position change of the object will be checked at multiple time points. If the position of the object changes frequently within this time range, then N dyn The value of will be relatively large, indicating that the object is dynamic.

[0079] (2) Statistically calculating the sum of proximity values ​​between the target detection frame corresponding to each static target and the target tracking bounding frame of each static target within a second preset time range, to obtain a second proximity value sum, wherein if the Euclidean distance between the center of the target detection frame corresponding to a static target and the center of the target tracking bounding frame of the static target is less than a preset distance threshold, then the proximity value between the center of the target detection frame corresponding to the static target and the center of the target tracking bounding frame of the static target is 1, otherwise, the proximity value is 0;

[0080] When implemented, according to the second general formula ∈∈[0,-τ s ],[τ s ,τ the ] Statistically calculate the sum of the proximity values ​​of the target detection box corresponding to each static target and the target tracking bounding box of each static target within the second preset time range, where N stat represents the second closest value and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a second preset time range, represents the proximity between the center of the target detection frame corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the second preset time range, ∈ represents the time difference, τ s represents the second preset duration, τ the Indicates the third preset duration, d represents the preset distance threshold.

[0081] The second preset time range is specifically from 0 to t-τ s , t+τs to t+τ the time range, d represents the preset distance threshold at which two bounding boxes (a target detection box corresponding to a static target and a target tracking bounding box of a static target) are considered to be the same static target, τ the According to the frame number N frames and the frame rate of the sensor. stat represents the second closest value and, that is, the static count. In layman's terms, it means checking the stability of the object (i.e., the target) at multiple time points. The above formula indicates that the object will be checked for stability over a larger time range. This formula indicates that the change in the position of the object is checked over a longer period of time. If the position of the object does not change much over this longer period of time, then N stat The value of will be relatively large, indicating that the object is stationary.

[0082] (3) if the sum of the first approximation values ​​corresponding to the target detection frame corresponding to a static target is less than the sum of the second approximation values, then the target in the corresponding target detection frame is determined to be a dynamic target;

[0083] (4) If the sum of the first approximation values ​​corresponding to the target detection frame corresponding to a static target is not less than the sum of the second approximation values, the target in the corresponding target detection frame is determined to be a static target, and then the dynamic and static states of each target in the static map are obtained.

[0084] Specifically, Among them, F(c (t) ) represents the final voting result, which is used to determine whether the object is stationary or dynamic at time t. represents a static target, Indicates dynamic targets. Through the above method, false negative and false positive points can be reduced, and the quality of static maps can be improved.

[0085] In an optional embodiment of the present invention, the static map is optimized according to the dynamic and static targets in the static map, specifically comprising the following steps:

[0086] Delete the dynamic target in the static map and get the target static map.

[0087] This paper proposes an integrated real-time framework, using a lightweight 3D object detection network LSS for object detection. At the same time, in order to achieve real-time positioning and mapping, this paper introduces an efficient hierarchical association cost matrix. Finally, in order to build an accurate static map, this paper proposes a statistical-based method that achieves accurate dynamic object removal and static object restoration by emphasizing the spatiotemporal differences between dynamic objects and static objects.

[0088] A new SLAM framework for dynamic scenes is proposed to overcome the limitations of previous studies and operate robustly in unstructured environments. The main points are as follows:

[0089] 1) An online SLAM framework for dynamic scenes is proposed, which greatly improves the real-time performance while ensuring accuracy;

[0090] 2) An efficient hierarchical association cost matrix is ​​proposed, which reduces the time complexity of tracking associations from O(N 2 ) is greatly reduced to O(N);

[0091] 3) A statistics-based approach is proposed that focuses on the spatiotemporal differences between tracked static and dynamic objects, which greatly reduces erroneous static and dynamic object recognition.

[0092] In order to evaluate the performance of the present invention in challenging dynamic scenes, the present invention uses the Nuscenes dataset as the evaluation standard and compares it with the traditional ORB SLAM2 algorithm. The evaluation standard is the root mean square error of the absolute trajectory error of the ego vehicle. The absolute trajectory error of the i-th frame is calculated as P esti,i =T esti,i *P;P gt,i =T gt,i *P;

[0093] The above formula describes the calculation method of absolute trajectory error (ATE), which is often used to evaluate the accuracy of SLAM or positioning systems. Below is an explanation of the meaning of each symbol:

[0094] P esti，i : represents the estimated pose at the i-th moment. It is obtained by estimating the pose transformation matrix T esti，i The result obtained by acting on a fixed reference point P;

[0095] T esti，i : represents the estimated pose transformation matrix at the i-th moment, which is used to describe the estimated pose of the system at that moment;

[0096] T gt，i : represents the real pose at the i-th moment. It is obtained by the real transformation matrix T gt，i The result obtained by acting on the same reference point P;

[0097] T gt，i : Represents the real transformation matrix at the i-th moment, describing the real posture of the system at that moment.

[0098] ATE (Absolute Trajectory Error): ATE calculates the error between the estimated trajectory and the true trajectory. The formula calculates the ATE at the i-th moment and is defined as the estimated pose P esti，i and the true pose T gt，i The difference between.

[0099] This representation is the transformation relationship used to calculate ATE, by estimating the pose T esti，i The inverse matrix of the true pose Multiply them together to get the difference between the two.

[0100] The core idea of ​​the above formula is to calculate the error between the estimated pose and the actual pose by comparing them.

[0101] Specifically, the pose of the system at a certain time i can be expressed as a transformation matrix, which transforms the reference point P to the estimated pose P esti，i and the true pose P gt，i .

[0102] By calculation The error of the estimated pose relative to the true pose, namely the absolute trajectory error (ATE), can be obtained to measure the accuracy of the estimated trajectory.

[0103] The specific evaluation results are shown in the following table:

[0104] algorithm Input RMSE(m) / APE ORB-SLAM2 Front camera 0.282 Ours Front camera 0.140

[0105] Embodiment 2:

[0106] An embodiment of the present invention also provides a device for simultaneous positioning and mapping in a dynamic scene. The device for simultaneous positioning and mapping in a dynamic scene is mainly used to execute the method for simultaneous positioning and mapping in a dynamic scene provided in Embodiment 1 of the present invention. The following is a specific introduction to the device for simultaneous positioning and mapping in a dynamic scene provided in an embodiment of the present invention.

[0107] Figure 2 is a schematic diagram of a device for simultaneous positioning and mapping in a dynamic scene according to an embodiment of the present invention. Figure 2 As shown, the device mainly includes: a target detection unit 10, a determination and calculation unit 20, a determination and screening unit 30, a mapping unit 40, and an optimization unit 50, wherein:

[0108] The target detection unit is used to perform target detection on the current frame image using the LSS target detection network to obtain a target detection frame;

[0109] A determination and calculation unit, used to determine k tracking bounding boxes that are the nearest neighbors of each target detection box, and calculate the association cost between each target detection box and each corresponding tracking bounding box, and then determine the target tracking bounding box associated with each target detection box in the tracking bounding box according to the association cost, wherein k is a preset value, and the tracking bounding box is a target detection box obtained after target detection on the previous frame image and then obtained by Kalman filtering;

[0110] A determination and screening unit, configured to determine the static or dynamic state of the target in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith, and to screen the target detection frame corresponding to the static target and the target tracking bounding frame of the static target associated therewith according to the static or dynamic state of each target;

[0111] A mapping unit, used for positioning and mapping based on a target detection box corresponding to the static target and a target tracking bounding box of the static target associated therewith, to obtain a static map constructed according to the static target;

[0112] The optimization unit is used to determine the dynamic and static targets in the static map by using a statistical method, and optimize the static map according to the dynamic and static targets in the static map to obtain a target static map.

[0113] In an embodiment of the present invention, a device for simultaneous positioning and mapping in a dynamic scene is provided, comprising: using an LSS target detection network to perform target detection on a current frame image to obtain a target detection frame; determining k tracking boundary frames that are the nearest neighbors of each target detection frame, and calculating the association cost between each target detection frame and each corresponding tracking boundary frame, and then determining a target tracking boundary frame associated with each target detection frame in the tracking boundary frame according to the association cost, wherein k is a preset value, and the tracking boundary frame is a target detection frame obtained after performing target detection on a previous frame image and then obtained by Kalman filtering; determining the dynamic and static state of the target in each target detection frame according to each target detection frame and the target tracking boundary frame associated with it, and screening the target detection frame corresponding to the static target and the target tracking boundary frame of the static target associated with it according to the dynamic and static state of each target; performing positioning and mapping based on the target detection frame corresponding to the static target and the target tracking boundary frame of the static target associated with it, and obtaining a static map constructed according to the static target; using a statistical method to determine the dynamic and static targets in the static map, and optimizing the static map according to the dynamic and static targets in the static map to obtain a target static map. From the above description, it can be seen that in the device for simultaneous positioning and mapping in dynamic scenes of the present invention, a lightweight LSS target detection network is used for target detection, which has good real-time performance. When determining the relationship between the target detection box and the tracking bounding box, only the association cost of each tracking bounding box of the target detection box and its k nearest neighbor tracking bounding boxes is calculated, which greatly reduces the time complexity and improves the real-time performance of simultaneous positioning and mapping. In addition, after obtaining the static map, a statistical method is also used to optimize the static map, thereby improving the accuracy of the target static map. That is, the method for simultaneous positioning and mapping in dynamic scenes of the present invention has good real-time performance and high accuracy, which alleviates the technical problem that the existing methods for simultaneous positioning and mapping in dynamic scenes lack real-time performance and accuracy.

[0114] Optionally, the association cost includes: the Euclidean distance cost between the centers of the bounding boxes, the intersection-and-union cost of the bounding boxes, and the volume cost of the bounding boxes. The determination and calculation unit is further used to: use the tracking bounding box corresponding to the minimum association cost as the target tracking bounding box associated with each target detection box.

[0115] Optionally, the determination and screening unit is further used to determine the dynamic and static state of the target in each target detection frame based on the distance between the center of each target detection frame and the center of the target tracking boundary frame associated therewith.

[0116] Optionally, the determination and screening unit is also used to: if the distance is greater than a preset threshold, determine that the target in the corresponding target detection frame is a dynamic target; if the distance is not greater than the preset threshold, determine that the target in the corresponding target detection frame is a static target.

[0117] Optionally, the optimization unit is further used to: statistically calculate the sum of proximity values ​​between the target detection frame corresponding to each static target and the target tracking boundary frame of each static target within a first preset time range to obtain a first proximity value sum; statistically calculate the sum of proximity values ​​between the target detection frame corresponding to each static target and the target tracking boundary frame of each static target within a second preset time range to obtain a second proximity value sum, wherein, if the Euclidean distance between the center of a target detection frame corresponding to a static target and the center of a target tracking boundary frame of a static target is less than a preset distance threshold, then the proximity value between the center of the target detection frame corresponding to the static target and the center of the target tracking boundary frame of the static target is 1, otherwise, the proximity value is 0; if the first proximity value sum corresponding to the target detection frame corresponding to a static target is less than the second proximity value sum, then it is determined that the target in the corresponding target detection frame is a dynamic target; if the first proximity value sum corresponding to the target detection frame corresponding to a static target is not less than the second proximity value sum, then it is determined that the target in the corresponding target detection frame is a static target, thereby obtaining the dynamic and static states of each target in the static map.

[0118] Optionally, the optimization unit is further used to: delete the dynamic target in the static map to obtain a target static map.

[0119] Optionally, the optimization unit is further used to: according to the first general calculation formula ∈∈[-τ d , τ d ] Statistically calculate the sum of the proximity values ​​of the target detection box corresponding to each static target and the target tracking bounding box of each static target within the first preset time range, where N dyn represents the first approximation and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a first preset time range, represents the proximity between the center of the target detection frame corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the first preset time range, ∈ represents the time difference, τ d Indicates the first preset duration, d represents the preset distance threshold; according to the second general calculation formula ∈∈[0,-τ s ],[τ s ,τ the ] Statistically calculate the sum of the proximity values ​​of the target detection box corresponding to each static target and the target tracking bounding box of each static target within the second preset time range, where N stat represents the second closest value and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a second preset time range, represents the proximity between the center of the target detection frame corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the second preset time range, ∈ represents the time difference, τ s represents the second preset duration, τ the Indicates the third preset duration, d represents the preset distance threshold.

[0120] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0121] like Figure 3 As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the method for simultaneous positioning and mapping in dynamic scenarios as described above.

[0122] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memories and processors, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned method of simultaneous positioning and mapping in dynamic scenes.

[0123] The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0124] Corresponding to the method for simultaneous positioning and mapping in the above-mentioned dynamic scene, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to execute the steps of the method for simultaneous positioning and mapping in the above-mentioned dynamic scene.

[0125] The device for simultaneous positioning and mapping in dynamic scenes provided in the embodiments of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiments of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiments. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0126] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0127] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method for simultaneous positioning and mapping in dynamic scenes described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0131] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0132] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for simultaneous positioning and mapping in a dynamic scene, characterized in that: include: Use the LSS target detection network to detect the target in the current frame image and obtain the target detection frame; Determine k tracking bounding boxes that are the nearest neighbors to each of the target detection boxes, and calculate the association cost between each of the target detection boxes and each of the tracking bounding boxes corresponding to it, and then determine the target tracking bounding box associated with each of the target detection boxes in the tracking bounding box according to the association cost, wherein k is a preset value, and the tracking bounding box is a target detection box obtained after performing target detection on the previous frame image and then obtained by Kalman filtering; Determine the static or dynamic state of the target in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith, and select the target detection frame corresponding to the static target and the target tracking bounding frame of the static target associated therewith according to the static or dynamic state of each target; Performing positioning and mapping based on the target detection box corresponding to the static target and the target tracking bounding box of the static target associated with the static target, to obtain a static map constructed according to the static target; A statistical method is used to determine the dynamic and static targets in the static map, and the static map is optimized according to the dynamic and static targets in the static map to obtain a target static map.

2. The method according to claim 1, characterized in that The association cost includes: the Euclidean distance cost between the centers of the bounding boxes, the intersection-over-union cost of the bounding boxes, and the volume cost of the bounding boxes. Determining the target tracking bounding box associated with each of the target detection boxes in the tracking bounding box according to the association cost includes: The tracking bounding box corresponding to the minimum association cost is used as the target tracking bounding box associated with each of the target detection boxes.

3. The method according to claim 1, characterized in that: Determining the dynamic and static states of the targets in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith includes: The dynamic and static states of the targets in each target detection frame are determined based on the distance between the center of each target detection frame and the center of the target tracking bounding box associated therewith.

4. The method according to claim 3, characterized in that Determining the dynamic and static state of the target in each target detection frame based on the distance between the center of each target detection frame and the center of the target tracking bounding frame associated therewith includes: If the distance is greater than a preset threshold, determining that the target in the corresponding target detection frame is a dynamic target; If the distance is not greater than the preset threshold, it is determined that the target in the corresponding target detection frame is a static target.

5. The method according to claim 1, characterized in that The dynamic and static targets in the static map are determined using a statistical method, including: Statistically calculating the sum of proximity values ​​of the target detection frame corresponding to each of the static targets and the target tracking boundary frames of each of the static targets within a first preset time range to obtain a first proximity value sum; Statistically calculating the sum of proximity values ​​between the target detection frame corresponding to each of the static targets and the target tracking bounding frame of each of the static targets within a second preset time range, to obtain a second proximity value sum, wherein if the Euclidean distance between the center of the target detection frame corresponding to one of the static targets and the center of the target tracking bounding frame of the static target is less than a preset distance threshold, then the proximity value between the center of the target detection frame corresponding to the static target and the center of the target tracking bounding frame of the static target is 1, otherwise, the proximity value is 0; If the first approximation value sum corresponding to the target detection frame corresponding to the static target is less than the second approximation value sum, determining that the target in the corresponding target detection frame is a dynamic target; If the sum of the first approximation values ​​corresponding to the target detection frame corresponding to a static target is not less than the sum of the second approximation values, the target in the corresponding target detection frame is determined to be a static target, and then the dynamic and static states of each target in the static map are obtained.

6. The method according to claim 1, characterized in that Optimizing the static map according to the dynamic and static targets in the static map includes: The dynamic target in the static map is deleted to obtain the target static map.

7. The method according to claim 5, characterized in that Statistically calculating the sum of proximity values ​​of the target detection frame corresponding to each of the static targets and the target tracking bounding boxes of each of the static targets within a first preset time range, including: According to the first calculation formula The sum of the proximity values ​​of the target detection frame corresponding to each static target and the target tracking bounding frame of each static target within the first preset time range is statistically calculated, where N dyn represents the first approximation value and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a first preset time range, represents the proximity between the center of the target detection box corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the first preset time range, ∈ represents the time difference, τ d Indicates the first preset duration, , d represents the preset distance threshold; Statistically calculating the sum of proximity values ​​of the target detection frame corresponding to each of the static targets and the target tracking bounding boxes of each of the static targets within a second preset time range, including: According to the second calculation formula The sum of the proximity values ​​of the target detection frame corresponding to each of the static targets and the target tracking bounding boxes of each of the static targets within the second preset time range is statistically calculated, where N stat represents the second approximation value and, c (t) Represents the center of the target detection box corresponding to a static target at time t, represents the center of a target tracking bounding box of a static target within a second preset time range, represents the proximity between the center of the target detection frame corresponding to a static target at time t and the center of the target tracking bounding box of a static target within the second preset time range, ∈ represents the time difference, τ S represents the second preset duration, τ the Indicates the third preset duration, , d represents the preset distance threshold.

8. A device for simultaneous positioning and mapping in a dynamic scene, characterized in that: include: The target detection unit is used to perform target detection on the current frame image using the LSS target detection network to obtain a target detection frame; A determination and calculation unit, configured to determine k tracking bounding boxes that are the nearest neighbors of each of the target detection boxes, and calculate the association cost between each of the target detection boxes and each of the tracking bounding boxes corresponding to the target detection boxes, and then determine a target tracking bounding box associated with each of the target detection boxes in the tracking bounding boxes according to the association cost, wherein k is a preset value, and the tracking bounding box is a target detection box obtained after performing target detection on the previous frame image and then obtained by Kalman filtering; A determination and screening unit, configured to determine the static or dynamic state of the target in each target detection frame according to each target detection frame and the target tracking bounding frame associated therewith, and to screen the target detection frame corresponding to the static target and the target tracking bounding frame of the static target associated therewith according to the static or dynamic state of each target; A mapping unit, configured to perform positioning and mapping based on a target detection frame corresponding to the static target and a target tracking bounding frame of the static target associated therewith, to obtain a static map constructed according to the static target; The optimization unit is used to determine the dynamic and static targets in the static map by using a statistical method, and optimize the static map according to the dynamic and static targets in the static map to obtain a target static map.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to execute the method according to any one of claims 1 to 7.

Images