Image processing method, pose solving method and related device
By screening and processing feature points in the dynamic detection box, the problem of low self-positioning accuracy caused by multiple moving objects in the image in visual SLAM technology is solved, and the efficient and accurate self-positioning ability of the device in the environment is improved.
Patent Information
- Application Number
- CN202411124116.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-26
AI Technical Summary
In visual SLAM technology, when most objects in the image are in a moving state, it is difficult for traditional methods to achieve high efficiency and high accuracy self-positioning of the device because the number of static feature points is small.
By obtaining the dynamic detection box, it is determined whether the image meets the preset filtering conditions. If it is satisfied, delete the dynamic feature points in the dynamic detection box. If it is not satisfied, delete all feature points to obtain the target image, and then realize the self-positioning of the device.
Improve the number of static feature points in the target image and enhance the accuracy of the device's self-positioning. Especially when there are many dynamic objects in the image, static feature points are retained to improve positioning accuracy; when there are many static objects, dynamic feature points are directly deleted to improve efficiency. .
Smart Images

Figure CN119027452A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to an image processing method, a posture solving method and related devices. Background Art
[0002] Visual Simultaneous Localization and Mapping (VSLAM) is a technology that uses images captured by a camera to enable a device (camera or a vehicle equipped with a camera) to self-locate in an unknown environment and build an environmental map.
[0003] Visual SLAM relies on static feature points in the image (i.e., feature points of objects at rest in the image, such as trees or houses) to self-locate the device in the environment. In traditional solutions, after identifying the dynamic detection frame of the image (the detection frame of objects in motion, such as pedestrians or vehicles), the feature points in the dynamic detection frame are deleted to obtain the target image; based on the target image, the device can self-locate in the environment.
[0004] When there are many stationary objects in the image, this solution can efficiently and accurately achieve self-positioning of the device; however, when there are many moving objects in the image, such as pedestrians or vehicles, the above method obtains fewer static feature points in the target image, making it difficult to achieve high-precision self-positioning of the device.
[0005] How to rely on captured images and use visual SLAM technology to achieve efficient and accurate self-positioning of devices in the environment has become a technical problem that needs to be urgently solved in the field of computer vision technology. Summary of the invention
[0006] Based on the above problems, the present application provides an image processing method that can rely on the captured images and use visual SLAM technology to achieve high-efficiency and high-accuracy self-positioning of the device in the environment.
[0007] The embodiments of the present application disclose the following technical solutions:
[0008] The first aspect of the present application discloses an image processing method, comprising:
[0009] Acquire a dynamic detection frame in the first image; the dynamic detection frame includes an object in motion;
[0010] Determining whether the first image satisfies at least one of preset screening conditions; the preset screening conditions include: the number of the dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to a second preset value;
[0011] If at least one of the preset screening conditions is met, the dynamic feature points in the dynamic detection frame are deleted to obtain a target image; the dynamic feature points are feature points that characterize the object in the motion state;
[0012] If the preset screening condition is not met, all feature points in the dynamic detection frame are deleted to obtain the target image.
[0013] In an optional implementation manner, before determining whether the first image satisfies at least one of the preset screening conditions, the method further includes:
[0014] Acquire a target feature point pair in the first image; the target feature point pair is composed of a first feature point in the transformed image and a second feature point in the first image whose similarity to the first feature point is less than or equal to a preset similarity threshold; the transformed image is an image obtained by projecting a key frame corresponding to the first image onto a coordinate system where the first image is located;
[0015] The preset screening condition also includes: the number of the target feature point pairs is less than or equal to a third preset value.
[0016] In an optional implementation, the step of acquiring the dynamic detection frame includes:
[0017] Determine a plurality of detection frames in the first image; the first image is an image captured at a first moment; each of the detection frames includes an object in motion or an object in a stationary state;
[0018] For each detection frame in the multiple detection frames, determining a prediction frame corresponding to the detection frame in a second image based on a uniform Kalman filter model; the second image is an image captured at a second moment; the second moment is later than the first moment;
[0019] After acquiring multiple detection frames in the second image, calculating the distance between each prediction frame in the first image and each detection frame in the second image;
[0020] The dynamic detection frame is determined from multiple detection frames in the first image according to the calculated distance between each prediction frame in the first image and each detection frame in the second image through the Hungarian algorithm.
[0021] In an optional implementation manner, calculating the distance between each prediction box in the first image and each detection box in the second image includes:
[0022] For each prediction box in the first image and each detection box in the second image, obtaining a first distance between a center point of the prediction box and a center point of the detection box;
[0023] Obtaining a union of the prediction box and the detection box as a merged box, and obtaining a diagonal length of the merged box as a second distance;
[0024] Get the intersection area and union area between the prediction box and the detection box;
[0025] The distance between the prediction box and the detection box is obtained according to the first distance, the second distance, the intersection area and the union area.
[0026] In an optional implementation, deleting the dynamic feature points in the dynamic detection frame to obtain the target image includes:
[0027] Acquire a plurality of ORB feature points of the first image and a plurality of ORB feature points of the key frame;
[0028] Projecting the map point corresponding to each ORB feature point in the key frame to the coordinate system where the first image is located to obtain the projection point corresponding to the map point;
[0029] Acquire an ORB feature point matching the projection point from the first image, and determine a pose transformation matrix based on the ORB feature point and the projection point;
[0030] For each ORB feature point in the first image, judging whether the ORB feature point is the dynamic feature point according to the pose transformation matrix, the coordinates of the ORB feature point, and the coordinates of the ORB feature point corresponding to the ORB feature point in the key frame;
[0031] After determining the multiple dynamic feature points, the multiple dynamic feature points are deleted from the dynamic detection frame to obtain the target image.
[0032] The second aspect of the present application discloses a posture solving method, comprising:
[0033] Acquire a target image; the target image is an image acquired based on the method described in any one of the first aspects;
[0034] Based on the target image, a position and a posture of a camera that captures the target image are determined.
[0035] The third aspect of the present application discloses an image processing device, comprising:
[0036] A dynamic detection frame determination module, used to obtain a dynamic detection frame in the first image; the dynamic detection frame includes an object in motion;
[0037] a parameter determination module, configured to determine whether the first image satisfies at least one of preset screening conditions; the preset screening conditions include: the number of the dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frames to the area of the first image is greater than or equal to a second preset value;
[0038] A first target image acquisition module, configured to delete the dynamic feature points in the dynamic detection frame to obtain a target image if at least one of the preset screening conditions is met; the dynamic feature points are feature points that characterize the object in the motion state;
[0039] The second target image acquisition module is used to delete all feature points in the dynamic detection frame to obtain the target image if the preset screening condition is not met.
[0040] The fourth aspect of the present application discloses a posture solving device, comprising:
[0041] An image acquisition module, used to acquire a target image; the target image is an image acquired based on the method described in any one of the first aspects;
[0042] The position and posture solving module is used to determine the position and posture of the camera that shoots the target image based on the target image.
[0043] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any implementation of the first aspect and the second aspect.
[0044] A sixth aspect of the present application provides an electronic device, including:
[0045] a memory having a computer program stored thereon;
[0046] A processor is used to execute the computer program in the memory to implement the steps of the method introduced in any implementation of the first aspect and the second aspect.
[0047] Compared with the prior art, this application has the following beneficial effects:
[0048] The present application discloses an image processing method, including: obtaining a dynamic detection frame in a first image; determining whether the first image satisfies at least one of the preset screening conditions; if at least one of the preset screening conditions is satisfied, deleting the dynamic feature points in the dynamic detection frame to obtain a target image; if the preset screening conditions are not satisfied, deleting all the feature points in the dynamic detection frame to obtain the target image. The preset screening conditions in the present application include: the number of dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to a second preset value.
[0049] When the first image meets at least one of the preset screening conditions, it means that there are many dynamic detection frames in the first image or the area of the dynamic detection frame is large. If all the feature points in the dynamic detection frame are directly deleted, there will be very few static feature points in the target image, and it is difficult to obtain a high-accuracy self-positioning of the device based on such a target image; and deleting the dynamic feature points in the detection frame can increase the number of static feature points in the target image, and more accurate self-positioning of the device can be obtained. For the first image that does not meet the preset screening conditions, it means that there are many static objects in the first image. In order to improve efficiency, all the feature points in the dynamic detection frame are directly deleted. There are still many static feature points in the target image, and the self-positioning of the device can still be obtained more accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0051] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;
[0052] Figure 2 A schematic diagram of calculating the DIOU distance provided in an embodiment of the present application;
[0053] Figure 3 A schematic diagram of the effect of an image processing method provided in an embodiment of the present application;
[0054] Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;
[0055] Figure 5 A schematic diagram of the structure of a posture solving device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0056] Visual Simultaneous Localization and Mapping (VSLAM) is a technology that uses images captured by a camera to enable a device (camera or a vehicle equipped with a camera) to self-locate in an unknown environment and build an environmental map.
[0057] Visual SLAM relies on static feature points in the image (i.e., feature points of objects at rest in the image, such as trees or houses) to self-locate the device in the environment. In traditional solutions, after identifying the dynamic detection frame of the image (the detection frame of objects in motion, such as pedestrians or vehicles), the feature points in the dynamic detection frame are deleted to obtain the target image; based on the target image, the device can self-locate in the environment.
[0058] When there are many stationary objects in the image, this solution can efficiently and accurately achieve self-positioning of the device; however, when there are many moving objects in the image, such as pedestrians or vehicles, the above method obtains fewer static feature points in the target image, making it difficult to achieve high-precision self-positioning of the device.
[0059] How to rely on captured images and use visual SLAM technology to achieve efficient and accurate self-positioning of devices in the environment has become a technical problem that needs to be urgently solved in the field of computer vision technology.
[0060] The present application discloses an image processing method, including: obtaining a dynamic detection frame in a first image; determining whether the first image satisfies at least one of the preset screening conditions; if at least one of the preset screening conditions is satisfied, deleting the dynamic feature points in the dynamic detection frame to obtain a target image; if the preset screening conditions are not satisfied, deleting all the feature points in the dynamic detection frame to obtain the target image. The preset screening conditions in the present application include: the number of dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to a second preset value.
[0061] When the first image meets at least one of the preset screening conditions, it means that there are many dynamic detection frames in the first image or the area of the dynamic detection frame is large. If all the feature points in the dynamic detection frame are directly deleted, there will be very few static feature points in the target image, and it is difficult to obtain a high-accuracy self-positioning of the device based on such a target image; while only deleting the dynamic feature points in the detection frame, there will be more static feature points remaining in the first image, which can increase the number of static feature points in the target image and obtain a more accurate self-positioning of the device. For the first image that does not meet the preset screening conditions, it means that there are many static objects in the first image. In order to improve efficiency, all the feature points in the dynamic detection frame are directly deleted. There are still many static feature points in the target image, and the self-positioning of the device can still be obtained more accurately.
[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0063] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application. Figure 1 As shown, the image processing method disclosed in this application includes:
[0064] S101: Acquire a dynamic detection frame in a first image.
[0065] The dynamic detection frame in the present application includes objects in motion. For example, the first image includes trees, houses, sky, roads, walking pedestrians A, and moving vehicles B. Trees, sky, roads, and houses are objects in a stationary state in the first image, and pedestrians A and vehicles B are objects in motion in the first image. The dynamic detection frame of the first image is the detection frame including pedestrian A and the detection frame including vehicle B.
[0066] It is understandable that pedestrian A and vehicle B are moving on the road, and the dynamic detection frame will inevitably include objects in motion such as the road and the sky. That is, the dynamic detection frame of the first image includes objects in motion and some objects in a stationary state.
[0067] In an optional implementation, the following steps may be used to obtain a dynamic detection frame in the first image. Specifically:
[0068] In the first step, after acquiring the first image captured at the first moment, the first image is input into the detector Yolox, and multiple detection frames in the first image are obtained through the high-performance detector Yolox; the multiple detection frames include both dynamic detection frames and static detection frames. The dynamic detection frame is the detection frame where the object in motion is located; the static detection frame is the detection frame where the object in a stationary state is located.
[0069] It should be noted that Yolox is an object detector that uses an anchor-free mechanism, a decoupled detection head, an advanced label allocation strategy, and powerful data enhancement, aiming to provide an efficient, accurate, and flexible object detection solution. Yolox is well known to those skilled in the art and is not limited in this application.
[0070] In the second step, for each detection frame in the first image, based on the uniform Kalman filter model, a prediction frame corresponding to the detection frame in the second image is determined, wherein the second image is an image taken at a second moment; the second moment is later than the first moment.
[0071] For ease of understanding, the second step is described in detail by taking the target detection frame in the first image as an example. The target detection frame in the first image is any one of the multiple detection frames in the first image.
[0072] The uniform Kalman filter model is a system state estimation method that combines the uniform motion model and the Kalman filter algorithm. The process of calculating the prediction box corresponding to the target detection box in the second image through the uniform Kalman filter model is:
[0073] Assuming that the target object in the target detection frame is in a uniform motion state in a short period of time, the state vector x of the target object in the target detection frame is t It can be expressed as formula (1):
[0074]
[0075] In formula (1), x and y are the center coordinates of the target detection frame in the image coordinate system, w is the width of the target detection frame, and h is the height of the target detection frame. are the velocity values of the center coordinates, They are the change values of width and height respectively.
[0076] The position of the target object in the target detection frame in the second image is predicted based on the uniform Kalman filter model.
[0077] First, according to the state equation and the state estimate at the previous moment Predict the current state estimate
[0078]
[0079] F=H (3)
[0080] in, is the state estimate at the previous moment, i.e., at time t-1, is the state estimate at the current moment, that is, at time t; F is the state transfer matrix, and H is the unit matrix.
[0081] Secondly, predict the covariance matrix of the current state estimate to represent the uncertainty of the state estimate. This process is shown in formula (4):
[0082]
[0083] Among them, F in formula (4) T is the transpose of the state transfer matrix, is the covariance estimate of the forecast error at the current moment, i.e. at time t, Q t is the covariance matrix of process noise, P t-1 is the covariance estimate at the previous moment, that is, at time t-1.
[0084] In the initial process, the state transfer matrix is a unit matrix. As the number of image frames changes, the detected target object moves in the pixels and is affected by noise. Under the influence of the moving speed and noise, the position of the target object in the target detection frame in the second image can be predicted, that is, the prediction frame where the target object in the target detection frame is located in the second image, that is, the corresponding prediction frame.
[0085] The third step is to obtain multiple detection frames in the second image and then calculate the distance between each prediction frame in the first image and each detection frame in the second image.
[0086] For the process of obtaining multiple detection frames in the second image, refer to the process of obtaining multiple detection frames in the first image disclosed in the aforementioned embodiment, which will not be repeated here.
[0087] Figure 2 A schematic diagram of calculating the DIOU distance provided in an embodiment of the present application. Taking the calculation of the distance between any prediction box in the first image (referred to as the first prediction box in the present application) and any detection box in the second image (referred to as the first detection box in the present application) as an example, the process of calculating the distance between each prediction box in the first image and each detection box in the second image is described in detail.
[0088] Combination Figure 2 As shown, box 1 is the first predicted box, box 2 is the first detected box, and box 3 is the merged box. The process of calculating the distance between the first predicted box and the second detected box, that is, the DIOU distance, is:
[0089] First, obtain the first distance between the center point of the first prediction frame and the center point of the first detection frame, that is, Figure 2 In d.
[0090] Secondly, get the diagonal length of the merged box as the second distance, that is Figure 2 c in.
[0091] Then, the intersection area and the union area between the first prediction box and the first detection box are obtained.
[0092] Finally, the first distance, the second distance, the intersection area and the union area are substituted into formula (5) to obtain the distance between the first prediction box and the first detection box. Formula (5) is as follows:
[0093]
[0094] Among them, DIOU in formula (5) distance is the distance between the first prediction box and the first detection box, IOU is the ratio of the intersection area to the union area; d is the first distance, and c is the second distance.
[0095] In target detection, DIOU solves the problem of slow convergence of IOU loss. In multi-target tracking, adding the center point distance as one of the considerations of DIOU distance, because it takes more factors into consideration, better measures the distance between the target prediction box and the detection box, which is beneficial for the subsequent Hungarian algorithm to solve the matching problem.
[0096] In the fourth step, a dynamic detection frame is determined from multiple detection frames in the first image according to the calculated distance between each prediction frame in the first image and each detection frame in the second image through the Hungarian algorithm.
[0097] After determining the distance between each prediction frame in the first image and each detection frame in the second image, that is, the similarity between each prediction frame in the first image and each detection frame in the second image, the Hungarian algorithm can be used to determine the dynamic detection frame of the first image from multiple detection frames in the first image.
[0098] It should be noted that, since the Hungarian algorithm is used to calculate the distance between each prediction box in the first image and each detection box in the second image, determining the dynamic detection box from multiple detection boxes in the first image is a content well known to those skilled in the art, and no further elaboration is made on this content in the present application.
[0099] S102: Determine whether the first image satisfies at least one of preset screening conditions.
[0100] The preset screening conditions in the present application include: the number of dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to a second preset value.
[0101] The present application does not limit the specific values of the first preset value and the second preset value, and those skilled in the art can set the values of the first preset value and the second preset value according to their needs.
[0102] S103: If at least one of the preset screening conditions is satisfied, the dynamic feature points in the dynamic detection frame are deleted to obtain a target image.
[0103] It is understandable that if the number of target frames in the first image is greater than the first preset threshold, it means that there are more dynamic objects in the image, and more target frames of dynamic objects occupy more areas, and the occlusion of static areas will result in fewer static feature points. If the detected dynamic object is closer to the camera, a larger target frame will appear on the image and occupy a larger area. If all feature points in the target frame are removed, more static points will be discarded.
[0104] That is, if the number of dynamic detection frames in the first image is greater than or equal to the first preset value, it means that there are many objects in motion in the first image. If the ratio of the area after the union of the dynamic detection frames in the first image to the area of the first image is greater than or equal to the second preset value, it means that the area occupied by the dynamic objects in the first image is very large. In either case, if all the feature points in the dynamic detection frames are directly deleted, the number of static feature points remaining in the first image will be small, that is, the number of static feature points in the target image will be small, and it is difficult to determine the self-positioning of the device in the environment through visual SLAM based on such a target image.
[0105] For such a first image containing a large number of dynamic objects (objects in motion), in order to determine the self-positioning of the device in the environment based on the target image corresponding to the first image, after determining that the image is a high-dynamic image, the present application does not delete all the feature points in the dynamic detection frame, but deletes the dynamic feature points in the dynamic detection frame, that is, deletes the feature points in the dynamic detection frame that represent the objects in motion. The advantage of doing so is that more static feature points in the first image can be retained, which can improve the ability to determine the self-positioning of the device in the environment through visual SLAM, that is, determine the position and posture of the device in the environment with higher accuracy.
[0106] In an optional implementation, after determining that the first image satisfies at least one of the preset screening conditions, the following steps may be performed to delete the dynamic feature points in the dynamic detection frame to obtain the target image. Specifically:
[0107] First, a plurality of ORB feature points of the first image and a plurality of ORB feature points of the key frame are obtained.
[0108] In the field of image processing, key frame is a common concept. The key frame in this application can be understood as an image that is highly correlated with the first image in time and space and selected by those skilled in the art according to a pre-set screening criterion. Since key frames are well known to those skilled in the art, the content of key frames will not be further elaborated in this application.
[0109] ORB (Oriented FAST and Rotated BRIEF) feature point is an algorithm for key point detection and description in an image. It combines the advantages of FAST (Features from Accelerated Segment Test) key point detection algorithm and BRIEF (Binary Robust Independent Elementary Features) descriptor algorithm. The following describes how to obtain multiple ORB feature points of the first image, including the following steps:
[0110] (1) After grayscale processing, the first image is input into the constructed image pyramid network, and the first image is scaled to different scales through the image pyramid network.
[0111] The image pyramid network in this application has eight layers, from L0 to L7. Each layer in the image pyramid network outputs a first image of a different scale. Since the image pyramid network is well known to those skilled in the art, this application does not limit this content.
[0112] (2) After obtaining eight first images of different scales, for each first image, the FAST corner points of the image can be extracted.
[0113] Specifically, for each pixel point P in the first image after scaling, its grayscale value is I p , set a brightness threshold T, T is I p Draw a circle with p as the center and radius r (the value of r is selected according to actual needs), select M pixels on the circle, and if there are N (N≤M) consecutive pixels with brightness within I P If it is outside the range of ±T, the area where the circle is located is determined to be a FAST corner point.
[0114] In the above manner, all pixel points in each of the eight first images scaled at different scales are screened to obtain FAST corner points at all positions of the first image.
[0115] (3) Select the grayscale centroid of the area where the circle is located, connect the centroid and the center of the circle to form a vector. The angle and direction of the vector are the angle and direction of the FAST corner point.
[0116] (4) Calculate the BRIEF descriptor. Specifically, a 31×31 block is selected with the circle as the center, and N′ point pairs (N′ is generally 256) are selected in this block. By comparing the grayscale values of two point pairs {A, B}, a binary result is determined, as shown in formula (6):
[0117]
[0118] In formula (6), p′ represents the grayscale, τ() represents the comparison operation; p′(A) represents the grayscale of point A, and p′(B) represents the grayscale of point B.
[0119] An ORB feature point is composed of a FAST corner point and a BRIEF descriptor. Similarly, ORB feature points are extracted for the entire image. The ORB feature points extracted from each first image after scaling at eight different scales are superimposed as the ORB feature points in the unscaled first image.
[0120] It is understandable that those skilled in the art can refer to this method to obtain multiple ORB feature points of a key frame.
[0121] After determining that the first image satisfies at least one of the preset screening conditions, the dynamic feature points in the dynamic detection frame in the first image need to be deleted to obtain the target image. In this application, each ORB feature point in the first feature map is analyzed by limit constraints to determine whether the feature point moves, that is, to determine whether the feature point is a dynamic feature point.
[0122] Specifically:
[0123] In the tracking thread, the set of ORB feature points obtained in the first image is expressed as formula (7):
[0124] A K ={p1,p2,p3…,p i} (7)
[0125] Among them, A in formula (7) K is the set of all feature points on the first image, p i is the i-th feature point in the first image; i is an integer greater than or equal to 3.
[0126] In the target tracking thread, multiple detection boxes are obtained at the same time. There will be multiple ORB feature points in each detection box, which is recorded as formula (8):
[0127] R K = {R1, R2, R3…, R i} (8)
[0128] R i ={p i,1 ,p i,2 ,p i,3 …,p i,i}
[0129] Among them, R K is the set of all detection box areas in the target tracking thread, R i For R K The i-th detection box in R i p i,i Represents the i-th feature point in the i-th detection frame.
[0130] Then, the limit constraint condition is used to determine whether the feature point in the detection frame is moving, that is, whether the feature point in the detection frame is a dynamic feature point.
[0131] The first image and the key frame corresponding to the first image are matched with feature points to obtain a pair of feature points matched between the first image and the corresponding key frame.
[0132] Specifically, the map point corresponding to each ORB feature point on the key frame image is projected to the coordinate system of the first image, and the ORB feature point matching the projection point is obtained from the first image. The ORB feature points obtained from the first image that match the projection point do not overlap, and at this time, BA optimization can be used to continuously reduce the error value between the two to obtain the pose transformation matrix P.
[0133] For each ORB feature point in the first image, it is determined whether the ORB feature point is a dynamic feature point according to the pose transformation matrix P, the coordinates of the ORB feature point, and the coordinates of the ORB feature point corresponding to the ORB feature point in the key frame. The specific calculation process is:
[0134] Assume that K1 is the key frame image corresponding to the first image, that is, K1 is the key frame; K2 is the first image, p1 = [x1, y1, 1] T and p2 = [x2, y2, 1] T is a matching feature point pair; x1 and y1 are the horizontal and vertical coordinates of the feature point p1 in the key frame; x2 and y2 are the horizontal and vertical coordinates of the feature point p2 in the first image. Whether the feature point p2 is a dynamic feature point can be determined according to the equation group (9). The expression of the equation group (9) is:
[0135]
[0136] p2e2 in equation group (9) is the epipolar line corresponding to the feature point p2 of the first image, where A, B, and C are line vectors; d is the distance between the feature point p2 and the corresponding epipolar line p2e2. The meanings of other letters in equation group (9) refer to the introduction of the above embodiment and will not be repeated here.
[0137] After the distance d is calculated, it is compared with the preset distance threshold d K If the distance d is greater than the distance threshold d K , then the feature point p2 is determined to be a dynamic feature point; if the distance d is less than or equal to the distance threshold d K , then the feature point p2 is judged to be a static feature point. Find the target frame with dynamic feature points, remove all feature points in the target frame area, and obtain the target image.
[0138] It should be noted that, since for each ORB feature point in the first image, judging whether the ORB feature point is the dynamic feature point based on the pose transformation matrix P, the coordinates of the ORB feature point, and the coordinates of the ORB feature point corresponding to the ORB feature point in the key frame is a numerical content of technicians in this field, this content will not be elaborated in this application.
[0139] S104: If the preset screening condition is not satisfied, all feature points in the dynamic detection frame are deleted to obtain the target image.
[0140] If the first image does not meet at least one of the preset screening conditions, it is considered that there are not many objects in motion in the first image. To improve efficiency, all feature points in the dynamic detection frame in the first image are directly removed to obtain the target image. There are still many static feature points in the target image, and the self-positioning of the device can still be obtained relatively accurately.
[0141] In an optional implementation, before determining whether the first image satisfies at least one of the preset screening conditions, the image processing method in the present application further includes:
[0142] Obtain target feature point pairs in the first image.
[0143] The target feature point pair is composed of a first feature point in the transformed image and a second feature point in the first image whose similarity to the first feature point is less than or equal to a preset similarity threshold. The transformed image is an image obtained by projecting the key frame corresponding to the first image to the coordinate system where the first image is located.
[0144] It should be noted that, since the key frame corresponding to the first image is projected and projected to the coordinate system where the first image is located to obtain a transformed image; and obtaining the first feature point from the transformed image, as well as the second feature point in the first image whose similarity to the first feature point is less than or equal to a preset similarity, that is, obtaining the target feature point pair is a content well known to those skilled in the art, and no further elaboration is made in this application.
[0145] After obtaining the target feature point pairs in the first image, the preset filtering conditions in S102 can be updated to: the number of dynamic detection frames is greater than or equal to the first preset value, the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to the second preset value, and the number of target feature point pairs is less than or equal to the third preset value.
[0146] It can be understood that the benefit of obtaining the target feature point pairs in the first image and adding preset screening conditions is that this scheme adds a way to determine whether the first image includes a large number of feature points in motion, which can better determine how the first image should be processed, that is, more accurately determine whether to directly delete all feature points in the dynamic detection frame in the first image, or only delete the dynamic feature points in the dynamic detection frame in the first image.
[0147] Figure 3 A schematic diagram of the effect of an image processing method provided in an embodiment of the present application. Figure 3 (a) shows the result of target detection on the image; Figure 3 (b) shows the effect before removing dynamic features in the image; Figure 3 (c) shows the effect after removing dynamic features in the image. Figure 3 (a) shows three detection frames in an image, in which the object in one detection frame is a car, the object in another detection frame is a bench, and the object in the third detection frame is also a car. Through the method in the present application, it is determined that the dynamic feature points in the detection frame including the car need to be deleted, so the dynamic feature points in the detection frame including the car are deleted, while the feature points in the detection frame including the bench are not affected and are not deleted.
[0148] In summary, the present application discloses an image processing method, including: obtaining a dynamic detection frame in a first image; determining whether the first image satisfies at least one of the preset filtering conditions; if at least one of the preset filtering conditions is satisfied, deleting the dynamic feature points in the dynamic detection frame to obtain a target image; if the preset filtering conditions are not satisfied, deleting all the feature points in the dynamic detection frame to obtain the target image. The preset filtering conditions in the present application include: the number of dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to a second preset value.
[0149] When the first image meets at least one of the preset screening conditions, it means that there are many dynamic detection frames in the first image or the area of the dynamic detection frame is large. If all the feature points in the dynamic detection frame are directly deleted, there will be very few static feature points in the target image, and it is difficult to obtain a high-accuracy self-positioning of the device based on such a target image; while deleting the dynamic feature points in the detection frame and retaining the static feature points can increase the number of static feature points in the target image, and more accurate self-positioning of the device can be obtained. For the first image that does not meet the preset screening conditions, it means that there are many static objects in the first image. In order to improve efficiency, all the feature points in the dynamic detection frame are directly deleted. There are still many static feature points in the target image, and the self-positioning of the device can still be obtained more accurately.
[0150] Furthermore, the present application also discloses a posture solving method, comprising:
[0151] The first step is to obtain a target image. For the specific implementation of obtaining the target image, please refer to the introduction of the relevant contents of the image processing method disclosed in the above embodiment, which will not be repeated here.
[0152] The second step is to determine the position and posture of the camera that captured the target image based on the target image. In this application, the process of determining the position and posture of the camera is referred to as posture solution.
[0153] Exemplarily, the pose solving module of the present application includes a tracking module, and the tracking module includes a pose tracking and solving unit, a local map tracking unit and a key frame selection unit. There are three methods for tracking and solving poses, namely uniform motion model tracking, reference key frame tracking and repositioning. At different times, select a suitable tracking method to solve the motion pose of the camera. However, this pose result is not accurate enough. In the next step, BA optimization is performed through the map points in the local map and some key frames to further optimize the positioning data. After the optimization is completed, the key frame screening strategy is used to find the key frames that meet the requirements and insert them into the map, and finally the tracking module is completed.
[0154] It should be noted that the process of posture solving is not limited in the present application, and those skilled in the art may adopt any posture solving method to determine the position and posture of the camera of the target image based on the target image.
[0155] Based on an image processing method disclosed in the aforementioned embodiment, the present application also discloses an image processing device. Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 4 As shown, the image processing device 400 in the present application includes:
[0156] A dynamic detection frame determination module 401 is used to obtain a dynamic detection frame in the first image; the dynamic detection frame includes an object in motion;
[0157] The parameter determination module 402 is used to determine whether the first image satisfies at least one of the preset screening conditions; the preset screening conditions include: the number of the dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to a second preset value;
[0158] The first target image acquisition module 403 is used to delete the dynamic feature points in the dynamic detection frame to obtain the target image if at least one of the preset screening conditions is met; the dynamic feature points are feature points that characterize the object in the motion state;
[0159] The second target image acquisition module 404 is used to delete all feature points in the dynamic detection frame to obtain the target image if the preset screening condition is not met.
[0160] In an optional implementation, the image processing apparatus 400 further includes:
[0161] a target feature point pair acquisition module, used to acquire a target feature point pair in the first image; the target feature point pair is composed of a first feature point in a transformed image and a second feature point in the first image whose similarity to the first feature point is less than or equal to a preset similarity threshold; the transformed image is an image obtained by projecting a key frame corresponding to the first image onto a coordinate system where the first image is located;
[0162] The parameter adding module is used for the preset screening condition and further includes: the number of the target feature point pairs is less than or equal to a third preset value.
[0163] In an optional implementation, the image processing apparatus 400 further includes:
[0164] a detection frame determination module, configured to determine a plurality of detection frames in the first image; the first image is an image captured at a first moment; each of the detection frames includes an object in motion or an object in a stationary state;
[0165] A prediction frame determination module is used to determine, for each detection frame in the multiple detection frames, a prediction frame corresponding to the detection frame in a second image based on a uniform Kalman filter model; the second image is an image taken at a second moment; the second moment is later than the first moment;
[0166] a distance calculation module, configured to calculate the distance between each prediction frame in the first image and each detection frame in the second image after acquiring the plurality of detection frames in the second image;
[0167] The dynamic block screening module is used to determine the dynamic detection frame from multiple detection frames of the first image according to the calculated distance between each prediction frame in the first image and each detection frame in the second image through the Hungarian algorithm.
[0168] In an optional implementation, the distance calculation module includes:
[0169] A first distance determining unit, configured to obtain, for each prediction box in the first image and each detection box in the second image, a first distance between a center point of the prediction box and a center point of the detection box;
[0170] a second distance determining unit, configured to obtain a union of the prediction frame and the detection frame as a merged frame, and obtain a diagonal length of the merged frame as a second distance;
[0171] An area acquisition unit, used to obtain the intersection area and the union area between the prediction box and the detection box;
[0172] A distance calculation unit is used to obtain the distance between the prediction box and the detection box according to the first distance, the second distance, the intersection area and the union area.
[0173] In an optional implementation, the first target image acquisition module 403 includes:
[0174] A feature point acquisition unit, used to acquire a plurality of ORB feature points of the first image and a plurality of ORB feature points of the key frame;
[0175] A projection point determination unit, used to project a map point corresponding to each ORB feature point in the key frame to a coordinate system where the first image is located, to obtain a projection point corresponding to the map point;
[0176] A posture transformation matrix acquisition unit, used to acquire ORB feature points matching the projection points from the first image, and determine a posture transformation matrix based on the ORB feature points and the projection points;
[0177] a dynamic feature point determination unit, configured to determine, for each ORB feature point in the first image, whether the ORB feature point is the dynamic feature point according to the pose transformation matrix, the coordinates of the ORB feature point, and the coordinates of the ORB feature point corresponding to the ORB feature point in the key frame;
[0178] The target image acquisition unit is used to delete the multiple dynamic feature points from the dynamic detection frame after determining the multiple dynamic feature points to obtain the target image.
[0179] Based on the posture solving method disclosed in the aforementioned embodiment, the present application also discloses a posture solving device. Figure 5 A schematic diagram of the structure of a posture solving device provided in an embodiment of the present application. Figure 5 As shown, the posture solving device 500 disclosed in the present application includes:
[0180] An image acquisition module 501 is used to acquire a target image; the target image is an image acquired based on any method described in the first aspect;
[0181] The position and posture solving module 502 is used to determine the position and posture of the camera that captured the target image based on the target image.
[0182] Based on the image processing method, posture solving method and device provided in the aforementioned embodiments, the present application also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements part or all of the steps in the image processing method and posture solving method mentioned above.
[0183] Based on the image processing method and the posture solving method and device provided in the above embodiments, the present application also provides an electronic device, including:
[0184] a memory having a computer program stored thereon;
[0185] A processor is used to execute the computer program in the memory to implement part or all of the steps in the image processing method and posture solving method provided in the above embodiments.
[0186] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0187] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire a dynamic detection frame in the first image; the dynamic detection frame includes an object in motion; Determining whether the first image satisfies at least one of preset screening conditions; the preset screening conditions include: the number of the dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frame to the area of the first image is greater than or equal to a second preset value; If at least one of the preset screening conditions is met, the dynamic feature points in the dynamic detection frame are deleted to obtain a target image; the dynamic feature points are feature points that characterize the object in the motion state; If the preset screening condition is not met, all feature points in the dynamic detection frame are deleted to obtain the target image.
2. The method according to claim 1, characterized in that Before determining whether the first image satisfies at least one of the preset screening conditions, the method further includes: Acquire a target feature point pair in the first image; the target feature point pair is composed of a first feature point in the transformed image and a second feature point in the first image whose similarity to the first feature point is less than or equal to a preset similarity threshold; the transformed image is an image obtained by projecting a key frame corresponding to the first image onto a coordinate system where the first image is located; The preset screening condition also includes: the number of the target feature point pairs is less than or equal to a third preset value.
3. The method according to claim 1, characterized in that: The step of acquiring the dynamic detection frame includes: Determine a plurality of detection frames in the first image; the first image is an image captured at a first moment; each of the detection frames includes an object in motion or an object in a stationary state; For each detection frame in the multiple detection frames, determining a prediction frame corresponding to the detection frame in a second image based on a uniform Kalman filter model; the second image is an image captured at a second moment; the second moment is later than the first moment; After acquiring multiple detection frames in the second image, calculating the distance between each prediction frame in the first image and each detection frame in the second image; The dynamic detection frame is determined from multiple detection frames in the first image according to the calculated distance between each prediction frame in the first image and each detection frame in the second image through the Hungarian algorithm.
4. The method according to claim 3, characterized in that The calculating the distance between each prediction box in the first image and each detection box in the second image includes: For each prediction box in the first image and each detection box in the second image, obtaining a first distance between a center point of the prediction box and a center point of the detection box; Obtaining a union of the prediction box and the detection box as a merged box, and obtaining a diagonal length of the merged box as a second distance; Get the intersection area and union area between the prediction box and the detection box; The distance between the prediction box and the detection box is obtained according to the first distance, the second distance, the intersection area and the union area.
5. The method according to claim 2, characterized in that: The deleting the dynamic feature points in the dynamic detection frame to obtain the target image includes: Acquire a plurality of ORB feature points of the first image and a plurality of ORB feature points of the key frame; Projecting the map point corresponding to each ORB feature point in the key frame to the coordinate system where the first image is located to obtain the projection point corresponding to the map point; Acquire an ORB feature point matching the projection point from the first image, and determine a pose transformation matrix based on the ORB feature point and the projection point; For each ORB feature point in the first image, judging whether the ORB feature point is the dynamic feature point according to the pose transformation matrix, the coordinates of the ORB feature point, and the coordinates of the ORB feature point corresponding to the ORB feature point in the key frame; After determining the multiple dynamic feature points, the multiple dynamic feature points are deleted from the dynamic detection frame to obtain the target image.
6. A posture solving method, characterized in that: The method comprises: Acquire a target image; the target image is an image acquired based on the method according to any one of claims 1 to 5; Based on the target image, a position and a posture of a camera that captures the target image are determined.
7. An image processing device, characterized in that: The device comprises: A dynamic detection frame determination module, used to obtain a dynamic detection frame in the first image; the dynamic detection frame includes an object in motion; a parameter determination module, configured to determine whether the first image satisfies at least one of preset screening conditions; the preset screening conditions include: the number of the dynamic detection frames is greater than or equal to a first preset value, and the ratio of the area of the dynamic detection frames to the area of the first image is greater than or equal to a second preset value; A first target image acquisition module, configured to delete the dynamic feature points in the dynamic detection frame to obtain a target image if at least one of the preset screening conditions is met; the dynamic feature points are feature points that characterize the object in the motion state; The second target image acquisition module is used to delete all feature points in the dynamic detection frame to obtain the target image if the preset screening condition is not met.
8. A posture solving device, characterized in that: The device comprises: An image acquisition module, used to acquire a target image; the target image is an image acquired based on the method according to any one of claims 1 to 5; The position and posture solving module is used to determine the position and posture of the camera that shoots the target image based on the target image.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.
10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 6.
