Method, device and electronic device for estimating target carrier state

Abstract

This invention provides a method, apparatus, and electronic device for estimating the state of a target carrier, relating to the fields of computer vision and spatial algorithm technology. The method includes: acquiring a current image frame collected by an image sensor, and performing feature extraction on the current image frame to obtain an original feature point set; acquiring motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame; identifying dynamic feature points from the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor; removing dynamic feature points from the original feature point set to obtain a calculated feature point set for the current image frame collected by the image sensor; and performing state estimation of the target carrier based on the calculated feature point set of the current image frame collected by the image sensor, thereby improving the accuracy of the state estimation obtained from the carrier state estimation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and spatial algorithm technology, and in particular to a method, apparatus and electronic device for estimating the state of a target carrier. Background Technology

[0002] When estimating the state of a carrier in a smart wearable device, a least-squares problem concerning the carrier's state is typically solved using filtering or optimization methods. Existing filtering or optimization methods generally treat points corresponding to objects as feature points and solve the least-squares problem concerning the carrier's state based on these feature points and feature point processing rules. However, these methods cannot distinguish between feature points of dynamic and static objects, leading to significant errors in the state estimation of the carrier. Summary of the Invention

[0003] This invention provides a target carrier state estimation device and electronic device to solve the defect in the prior art where the same processing rules for the feature points of dynamic objects and the feature points of static objects lead to large deviations in the filtering or optimization results, thereby improving the accuracy of the state estimation obtained by estimating the state of the carrier.

[0004] This invention provides a method for estimating the state of a target carrier, wherein the target carrier is equipped with an image sensor, and the method includes:

[0005] The current image frame acquired by the image sensor is obtained, and feature extraction is performed on the current image frame acquired by the image sensor to obtain the original feature point set;

[0006] Acquire motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame;

[0007] Based on the motion state parameters collected by the at least one auxiliary sensor, dynamic feature points are identified from the original feature point set;

[0008] The dynamic feature points are removed from the original feature point set to obtain the solution feature point set of the current image frame acquired by the image sensor;

[0009] The state of the target carrier is estimated based on the feature point set of the current image frame acquired by the image sensor.

[0010] According to a target carrier state estimation method provided by the present invention, the step of identifying dynamic feature points from the original feature point set based on motion state parameters collected by at least one auxiliary sensor includes:

[0011] If the motion state of the target carrier is determined to be stationary based on the motion state parameters collected by the at least one auxiliary sensor, dynamic feature points are identified from the original feature point set using a first identification rule; or

[0012] Based on the motion state parameters collected by the at least one auxiliary sensor, if the motion state of the target carrier is determined to be in motion, dynamic feature points are identified from the original feature point set using a second identification rule.

[0013] According to a target carrier state estimation method provided by the present invention, the at least one auxiliary sensor includes an inertial measurement unit;

[0014] Determining the motion state of the target carrier as a motion state based on motion state parameters collected by at least one auxiliary sensor includes:

[0015] The variance of the acceleration of the target carrier is calculated based on the motion state parameters collected by the inertial measurement unit; a dynamically determined set threshold is obtained.

[0016] If the variance of the acceleration of the target carrier is less than the set threshold, the motion state of the target carrier is determined to be a stationary state; or if the variance of the acceleration of the target carrier is not less than the set threshold, the motion state of the target carrier is determined to be a moving state.

[0017] According to a target carrier state estimation method provided by the present invention, before identifying dynamic feature points from the original feature point set, the method further includes:

[0018] Based on the motion state parameters collected by the at least one auxiliary sensor, the predicted value of each feature point in the original feature point set is determined, and the prior value and measured value of each feature point in the original feature point set are determined; wherein, the prior value of the feature point is the measured value of the corresponding feature point in the previous image frame;

[0019] The step of identifying dynamic feature points from the original feature point set based on motion state parameters collected by at least one auxiliary sensor includes:

[0020] Based on the motion state parameters collected by the at least one auxiliary sensor and the prior value, measured value, and predicted value of each feature point in the original feature point set, dynamic feature points are identified from the original feature point set.

[0021] According to a target carrier state estimation method provided by the present invention, the step of identifying dynamic feature points from the original feature point set through a first identification rule includes:

[0022] The difference between the prior value and the predicted value of the feature points in the original feature point set is within a first preset range; the difference between the prior value and the measured value of the feature points in the original feature point set is within a second preset range, wherein the second preset range is greater than the first preset range; the feature point is determined to be a dynamic feature point; or

[0023] The step of identifying dynamic feature points from the original feature point set using the second identification rule includes:

[0024] The difference between the measured value and the prior value of the feature point in the original feature point set is within a third set range; the difference between the measured value and the predicted value of the feature point in the original feature point set is within a fourth set range; the angle between the first line connecting the measured value and the prior value of the feature point in the original feature point set and the second line connecting the measured value and the predicted value of the feature point in the original feature point set is within a fifth set range; the feature point is determined to be a dynamic feature point.

[0025] According to a target carrier state estimation method provided by the present invention, the predicted value of each feature point in the original feature point set is determined based on motion state parameters collected by at least one auxiliary sensor, including:

[0026] The predicted target carrier pose corresponding to the current image frame is determined based on the motion state parameters collected by the at least one auxiliary sensor, and the scene depth of each feature point in the original feature point set and the feature point measurement value of the corresponding feature point in the previous image frame are obtained.

[0027] The predicted feature point value of each feature point in the original feature point set of the current image frame is calculated and determined based on the predicted target carrier pose, the scene depth, and the feature point measurement value of the corresponding feature point in the previous image frame.

[0028] According to a target carrier state estimation method provided by the present invention, the step of determining the predicted target carrier pose corresponding to the current image frame based on motion state parameters collected by at least one auxiliary sensor includes:

[0029] The motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame and the target carrier pose corresponding to the previous image frame are determined in order to calculate and determine the predicted target carrier pose corresponding to the current image frame.

[0030] According to a target carrier state estimation method provided by the present invention, the step of determining the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame and the target carrier pose corresponding to the previous image frame, in order to calculate and determine the predicted target carrier pose corresponding to the current image frame, includes:

[0031] The inter-frame pose increment is calculated based on the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame.

[0032] The predicted target carrier pose of the current image frame is determined by calculating the inter-frame pose increment and the target carrier pose corresponding to the previous frame image.

[0033] The present invention also provides a target carrier state estimation device, wherein the target carrier is equipped with an image sensor, the device comprising:

[0034] The original feature point set acquisition module is used to acquire the current image frame collected by the image sensor and perform feature extraction on the current image frame collected by the image sensor to obtain the original feature point set;

[0035] A motion state parameter acquisition module is used to acquire motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame;

[0036] A dynamic feature point recognition module is used to identify dynamic feature points from the original feature point set based on motion state parameters collected by the at least one auxiliary sensor.

[0037] The feature point set acquisition module is used to remove the dynamic feature points from the original feature point set to obtain the feature point set of the current image frame acquired by the image sensor;

[0038] The state estimation module is used to estimate the state of the target carrier based on the solved feature point set of the current image frame acquired by the image sensor.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target carrier state estimation method as described above.

[0040] The target carrier state estimation method, apparatus, and electronic device provided by this invention utilize motion state parameters collected by at least one auxiliary sensor to obtain motion context information of the target carrier, assisting in the identification of dynamic feature points in the original feature point set. This improves the accuracy of the state estimation obtained by estimating the carrier's state and achieves precise identification. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is an exemplary system architecture diagram in which some embodiments of the present invention can be applied.

[0043] Figure 2 This is a flowchart illustrating the target carrier state estimation method provided by the present invention.

[0044] Figure 3 This is a schematic diagram illustrating an application scenario of the target carrier state estimation method provided by the present invention.

[0045] Figure 4 This is a schematic diagram of the architecture of the target carrier state estimation method provided by the present invention.

[0046] Figure 5 This is a schematic diagram of the target carrier state estimation device provided by the present invention.

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] Traditional methods for estimating the state of a carrier in smart wearable devices typically treat the points corresponding to the object as feature points and solve a least-squares problem about the state of the carrier based on the feature points and feature point processing rules.

[0050] However, directly treating the points corresponding to an object as feature points has significant limitations. Traditional methods for estimating the state of a carrier in smart wearable devices cannot distinguish points of objects in different states, resulting in the carrier pose of the smart wearable device being unpredictable.

[0051] In view of this, embodiments of the present invention provide a target carrier state estimation method, apparatus, and electronic device. The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0052] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the target carrier state estimation method or target carrier state estimation apparatus of the present invention can be applied.

[0053] like Figure 1 As shown, the system architecture 100 may include an image sensor 101, an inertial measurement unit 102, and a wheeled odometer 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the image sensor 101, the inertial measurement unit 102, the wheeled odometer 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0054] The inertial measurement unit 102 and the wheel odometer 103 are exemplary auxiliary sensors. Auxiliary sensors can also be lidar and real-time kinematic (RTK) systems, etc., without specific limitations.

[0055] Server 105 can be a remote server providing various services, such as a remote server that analyzes and processes data from image sensor 101, inertial measurement unit 102, and wheeled odometer 103. It can also be a local computing device on the target vehicle side, such as the local computing device of an XR device configured on the target vehicle, or a local computing device on a robot or autonomous vehicle, etc., without specific limitations. Thus, the target vehicle state estimation method provided in this embodiment of the invention can be executed by a remote server or by a local computing device on the target vehicle side.

[0056] It should be understood that Figure 1 The number of sensors, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of sensors, networks, and servers can be included.

[0057] Figure 2 This is a flowchart illustrating the target carrier state estimation method provided by the present invention, wherein the target carrier is equipped with an image sensor.

[0058] There are many types of image sensors, such as monocular cameras and structured light cameras, and this invention does not limit them.

[0059] like Figure 2 As shown, the method includes the following steps:

[0060] Step 201: Obtain the current image frame acquired by the image sensor, and perform feature extraction on the current image frame acquired by the image sensor to obtain the original feature point set.

[0061] It should be noted that the current image frame acquired by the image sensor may include objects in at least two states, such as a background and moving objects. The background may be a wall, table, chair, or tall building, while the moving objects may be people, cats, or vehicles. Feature extraction from objects in at least two states within the current image frame yields the original feature point set. Many methods exist for feature extraction from the current image frame, such as model extraction or image processing algorithms; this invention does not limit the specific methods used.

[0062] The original feature point set refers to the set of feature points obtained by feature extraction on the current image frame. The original feature point set includes static feature points and dynamic feature points. Static feature points can be feature points extracted from the background of the current image frame, while dynamic feature points can be feature points extracted from moving objects in the current image frame.

[0063] Step 202: Obtain motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame.

[0064] It should be noted that there are many ways to obtain motion state parameters collected by auxiliary sensors. For example, auxiliary sensors can actively upload data after triggering data collection at a set frequency, or they can upload data in response to query commands after triggering data collection by an event. This invention does not limit the methods used in this regard.

[0065] In this context, auxiliary sensors refer to sensors other than image sensors, used to assist in identifying dynamic feature points in the original feature point set. There are many types of auxiliary sensors, such as inertial measurement units (IMUs), wheeled odometers, lidar, and real-time kinematic (RTK) systems, etc., which are not limited to this invention.

[0066] Motion state parameters refer to quantitative indicators obtained by processing measurement values ​​collected by auxiliary sensors. Different auxiliary sensors will collect different motion state parameters, and this invention does not limit them.

[0067] Step 203: Identify dynamic feature points from the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor.

[0068] It should be noted that dynamic feature points can be identified from the original feature point set based on motion state parameters acquired by an auxiliary sensor, reducing computational costs; alternatively, dynamic feature points can be identified from the original feature point set by combining motion state parameters acquired by multiple auxiliary sensors, reducing the measurement error or noise influence of a single sensor, thereby improving the accuracy of subsequent judgments.

[0069] Understandably, compared to directly identifying dynamic feature points from the original feature point set based on the information of the feature points in the original feature point set, combining motion state parameters collected by auxiliary sensors can improve the accuracy of identifying dynamic feature points from the original feature point set by using the motion context information provided by the motion state parameters.

[0070] Step 204: Remove the dynamic feature points from the original feature point set to obtain the solution feature point set of the current image frame acquired by the image sensor.

[0071] It should be noted that the current image frame may include one or more dynamic objects. After removing the feature points extracted from all dynamic objects in the current image frame from the original feature point set, the solution feature point set of the current image frame is obtained.

[0072] There are many ways to remove dynamic feature points from the original feature point set. For example, dynamic feature points can be removed by assigning labels or by indexing the position of dynamic feature points. This invention does not limit the methods.

[0073] The feature point set refers to the set of feature points that are ultimately used for state estimation among the feature points obtained by feature extraction of the current image frame.

[0074] Step 205: Estimate the state of the target carrier based on the feature point set of the current image frame acquired by the image sensor.

[0075] It should be noted that there are many ways to estimate the state of a target carrier based on the calculated feature point set. The state can be estimated by using the visual-inertial odometry (VIO) algorithm based on the calculated feature point set and the motion state parameters collected by at least one auxiliary sensor, or by using a large model, etc. This invention does not limit the methods.

[0076] It is understood that the target carrier state estimation method provided in this embodiment of the invention obtains the current image frame collected by the image sensor configured on the target carrier, extracts features from the current image frame collected by the image sensor to obtain the original feature point set, obtains motion state parameters collected by at least one auxiliary sensor, and obtains motion context information of the target carrier based on the motion state parameters collected by at least one auxiliary sensor, so as to help identify dynamic feature points in the original feature point set and reduce the computational complexity of analyzing and reasoning on the original feature point set to identify dynamic feature points.

[0077] Furthermore, the target carrier state estimation method provided in this embodiment of the invention obtains the solution feature point set of the current image frame acquired by the image sensor by removing dynamic feature points from the original feature point set. It can estimate the state of the target carrier based on the feature points of static objects in the current image frame acquired by the image sensor, thereby improving the accuracy of the state estimation obtained by estimating the state of the carrier and achieving accurate identification.

[0078] The target carrier state estimation method provided in this embodiment of the invention can estimate the state of the target carrier by removing dynamic feature points from the original feature point set, thereby reducing the computational complexity of identifying dynamic feature points, so as to perform real-time and accurate state estimation of the carrier.

[0079] The target carrier is a mobile entity, such as a person, robot, or various types of vehicles. Various sensors can be directly configured on the target carrier; for example, cameras and LiDAR can be directly mounted on autonomous vehicles. Sensors can also be integrated by configuring smart devices, such as wearable devices on the human body. By configuring various sensors, image frames and motion state parameters can be collected to estimate the state of the target carrier. These wearable devices can be XR devices, smart accessories, etc. XR devices are a collective term for VR, AR, and MR devices, specifically including smart glasses and virtual reality headsets (VR headsets).

[0080] Specifically, the target carrier state estimation method for the device provided in this embodiment of the invention can be applied to, for example, Figure 3 In the application scenario shown. Figure 3 In this application scenario, the target carrier is a person. Image frames and motion state parameters are collected using smart glasses 301. The smart glasses 301 are equipped with an image sensor 302 and an auxiliary sensor 303. The smart glasses 301 can be connected to a remote server 305 via a network 304. In this case, the target carrier state estimation method provided by this embodiment of the invention can be executed by the remote server or by a local computing device on the target carrier side, etc.

[0081] According to any of the above embodiments, before identifying dynamic feature points from the original feature point set, the method further includes:

[0082] Based on the motion state parameters collected by the at least one auxiliary sensor, the predicted value of each feature point in the original feature point set is determined, and the prior value and measured value of each feature point in the original feature point set are determined; wherein, the prior value of the feature point is the measured value of the corresponding feature point in the previous image frame;

[0083] The step of identifying dynamic feature points from the original feature point set based on motion state parameters collected by at least one auxiliary sensor includes:

[0084] Based on the motion state parameters collected by the at least one auxiliary sensor and the prior value, measured value, and predicted value of each feature point in the original feature point set, dynamic feature points are identified from the original feature point set.

[0085] It should be noted that when starting the target carrier, initialization can be performed based on the initial image data acquired by the image sensor and the initial motion state parameters acquired by the auxiliary sensor. The processing result of the first image frame is used as the feature point measurement value of the corresponding feature point in the second image frame (the next image frame). The measurement value of the second image frame can be obtained by tracking measurement based on the processing result of the first image frame, and used as the feature point measurement value of the corresponding feature point in the third image frame (the next image frame).

[0086] There are many ways to determine the feature point measurement value of each feature point in the original feature point set. For example, the KLT optical flow algorithm can be used to perform optical flow tracking to determine the feature point measurement value of each feature point in the previous image frame, or it can be determined directly by the three-dimensional reconstruction method, etc. This invention does not limit the methods.

[0087] Among them, the predicted value of the feature point can be used to represent the position of each feature point in the predicted current image frame, and the measured value of the feature point can be used to represent the position of each feature point in the actual measured current image frame.

[0088] It should be noted that the motion state of the target carrier can be determined based on the motion state parameters collected by at least one auxiliary sensor. However, under different motion states, based on the prior value, measured value, and predicted value of each feature point in the same set of original feature points, different dynamic feature point recognition results may be obtained from the original feature point set.

[0089] It is understandable that the prior value, measured value, and predicted value of each feature point can reflect its position information in adjacent image frames, so as to facilitate the construction of accurate dynamic point recognition rules.

[0090] According to the above embodiments, identifying dynamic feature points from the original feature point set based on motion state parameters collected by the at least one auxiliary sensor includes:

[0091] Based on the motion state parameters collected by the at least one auxiliary sensor, if the motion state of the target carrier is determined to be stationary, dynamic feature points are identified from the original feature point set by the first identification rule.

[0092] It should be noted that, based on motion state parameters collected by at least one auxiliary sensor, the motion state of the target carrier in the current image frame can be determined to be stationary from at least two types of motion states, so that dynamic feature points can be identified through a predetermined first recognition rule corresponding to the stationary state of the target carrier. The determination of the motion state of the target carrier based on the motion state parameters collected by the auxiliary sensor can be performed in a manner appropriate to the type of auxiliary sensor, and this invention does not limit this method.

[0093] In one embodiment, identifying dynamic feature points from the original feature point set using a first identification rule includes:

[0094] The difference between the prior value and the predicted value of the feature point in the original feature point set is within a first set range; the difference between the prior value and the measured value of the feature point in the original feature point set is within a second set range, wherein the second set range is greater than the first set range; the feature point is determined to be a dynamic feature point.

[0095] It should be noted that a small difference between the prior value and the predicted value of a feature point indicates that they are essentially consistent, thus placing the difference within a first predetermined range. A large difference between the prior value and the predicted value indicates a significant shift between them, placing the difference within a second predetermined range. The second predetermined range is larger than the first predetermined range. The criteria for determining "larger" or "essentially consistent" can be set according to actual business needs, and this invention does not impose limitations on this.

[0096] It is understandable that, compared to simply judging the difference between the prior value and the measured value of the feature point, in this embodiment of the invention, if the assumption that the feature point is stationary holds true and there is a measurement error, the prior value and the predicted value of the feature point are basically consistent even when the target carrier is stationary. However, there is a large deviation between the prior value and the measured value of the feature point. By comparing the difference between the predicted value and the prior value of the feature point and the difference between the measured value and the prior value of the feature point, the recognition accuracy of dynamic points can be improved.

[0097] In one embodiment, if the motion state of the target carrier is determined to be stationary based on the motion state parameters collected by the at least one auxiliary sensor, static feature points are also identified from the original feature point set using a third identification rule.

[0098] It should be noted that the prior value of a feature point is basically consistent with the measured value of the feature point. If the displacement between the prior value and the predicted value of a feature point is large, the feature point is determined to be a static feature point.

[0099] It is understandable that identifying static feature points from the original feature point set makes it easier to mark them, thereby reducing the chance of mistakenly removing static feature points from the original feature point set when removing dynamic feature points.

[0100] According to any of the above embodiments, the step of identifying dynamic feature points from the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor includes:

[0101] Based on the motion state parameters collected by the at least one auxiliary sensor, if the motion state of the target carrier is determined to be in motion, dynamic feature points are identified from the original feature point set using a second identification rule.

[0102] It should be noted that, based on motion state parameters collected by at least one auxiliary sensor, the motion state of the target carrier in the current image frame can be determined from at least two types of motion states, so as to identify dynamic feature points through a predetermined second recognition rule corresponding to the motion state of the target carrier. The determination of the motion state of the target carrier based on the motion state parameters collected by the auxiliary sensor can be carried out in a manner appropriate to the type of auxiliary sensor, and this invention does not limit this method.

[0103] In one embodiment, identifying dynamic feature points from the original feature point set using a second identification rule includes:

[0104] The difference between the measured value and the prior value of the feature point in the original feature point set is within a third set range; the difference between the measured value and the predicted value of the feature point in the original feature point set is within a fourth set range; the angle between the first line connecting the measured value and the prior value of the feature point in the original feature point set and the second line connecting the measured value and the predicted value of the feature point in the original feature point set is within a fifth set range; the feature point is determined to be a dynamic feature point.

[0105] It should be noted that a large difference between the measured value and the prior value of a feature point indicates a significant displacement between the measured and prior values. In this case, the difference between the measured and prior values ​​of the feature points in the original feature point set falls within the third preset range. Similarly, a large difference between the measured and predicted values ​​of a feature point indicates a significant displacement between the measured and predicted values. In this case, the difference between the measured and predicted values ​​of the feature points in the original feature point set falls within the fourth preset range. A larger angle value falls within the fifth preset range. The criteria for determining a larger angle can be set according to actual business needs; this invention does not impose limitations on this.

[0106] Among them, the larger judgment criteria among the second, third, fourth and fifth setting ranges can be the same or different, and the present invention does not limit this.

[0107] Understandably, when the difference between the measured value and the prior value of a feature point is large, excluding measurement errors, and the predicted value of the feature point is inferred from the motion state parameters of the target carrier, a large displacement between the measured and predicted values ​​of the feature point can preliminarily determine that the feature point is a dynamic point. The first line connecting the measured and prior values ​​of the feature point can represent the actual motion of the feature point, while the second line connecting the measured and predicted values ​​can represent the predicted motion of the feature point at rest. Thus, the angle between the first and second lines can reflect the difference between the actual motion and the predicted motion at rest, thereby confirming the feature point as a dynamic point and improving recognition accuracy.

[0108] In one embodiment, if the motion state of the target carrier is determined to be in motion based on the motion state parameters collected by the at least one auxiliary sensor, static feature points are also identified from the original feature point set using a fourth identification rule.

[0109] It should be noted that if the displacement between the prior value and the measured value of a feature point is large, the displacement between the prior value and the predicted value of a feature point is large, and the measured values ​​of a feature point are close to each other, then the feature point is determined to be a static feature point.

[0110] The purpose of identifying static feature points when the target carrier is in motion is basically the same as the purpose of identifying static feature points when the target carrier is at rest, and will not be elaborated here.

[0111] It is understandable that, compared to using the same method to identify dynamic feature points from the original feature point set, in this embodiment of the invention, the motion state type of the target carrier is determined from at least two types of motion states by motion state parameters, thereby identifying dynamic feature points from the original feature point set by recognition rules corresponding to the motion state, which can improve recognition accuracy.

[0112] According to any of the above embodiments, determining the predicted feature point value of each feature point in the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor includes:

[0113] The predicted target carrier pose corresponding to the current image frame is determined based on the motion state parameters collected by the at least one auxiliary sensor, and the scene depth of each feature point in the original feature point set and the feature point measurement value of the corresponding feature point in the previous image frame are obtained.

[0114] The predicted feature point value of each feature point in the original feature point set of the current image frame is calculated and determined based on the predicted target carrier pose, the scene depth, and the feature point measurement value of the corresponding feature point in the previous image frame.

[0115] It should be noted that there are many ways to obtain the scene depth of each feature point in the original feature point set. For example, the scene depth of each feature point can be extracted by a deep learning model, or the scene depth of each feature point can be obtained by triangulation of the feature points in the original feature point set. This invention does not limit the methods.

[0116] It is understood that the feature point prediction value determination method provided in this embodiment of the invention determines the predicted target carrier pose of the current image frame based on motion state parameters collected by at least one auxiliary sensor, and then determines the feature point prediction value by combining the scene depth and prior value of each feature point with multi-source information, which can improve the accuracy of the obtained feature point prediction value.

[0117] According to any of the above embodiments, determining the predicted target carrier pose corresponding to the current image frame based on the motion state parameters collected by the at least one auxiliary sensor includes:

[0118] The motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame and the target carrier pose corresponding to the previous image frame are determined in order to calculate and determine the predicted target carrier pose corresponding to the current image frame.

[0119] It should be noted that the predicted target carrier pose of the current image frame can be inferred from the motion state parameters of the previous image frame, based on the target carrier pose corresponding to the previous image frame. There are many ways to determine the target carrier pose corresponding to an image frame, such as through deep learning models or backend carrier pose calculation; this invention does not limit this method.

[0120] The target carrier pose is calculated from the image frame to obtain the true pose of the target carrier. The predicted target carrier pose is the predicted pose of the target carrier based on the target carrier pose of the previous image frame.

[0121] It is understandable that, based on the target carrier pose corresponding to the previous image frame, predicting the target carrier pose corresponding to the current image frame using the motion state parameters of the previous image frame can improve the prediction accuracy of the target carrier pose by taking advantage of the continuity of the target carrier's motion.

[0122] According to any of the above embodiments, determining the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame and the target carrier pose corresponding to the previous image frame, in order to calculate and determine the predicted target carrier pose corresponding to the current image frame, includes:

[0123] The inter-frame pose increment is calculated based on the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame.

[0124] The predicted target carrier pose of the current image frame is determined by calculating the inter-frame pose increment and the target carrier pose corresponding to the previous frame image.

[0125] It should be noted that there are many ways to calculate the inter-frame pose increment based on motion state parameters, such as through deep learning models or Kalman filtering, and this invention does not limit this method. There are also many ways to determine the predicted target carrier pose for the current image frame using the target carrier pose corresponding to the previous frame and the inter-frame pose increment, such as by directly summing the two or by weighting them, and this invention does not limit this method.

[0126] Among them, the inter-frame pose increment refers to the change in the pose of the target carrier between adjacent image frames.

[0127] It is understandable that the predicted target carrier pose is determined based on the target carrier pose and image information corresponding to the previous frame image. The inter-frame pose increment determined by the motion state parameters can help determine the predicted target carrier pose and improve the prediction accuracy.

[0128] According to any of the above embodiments, the at least one auxiliary sensor includes an inertial measurement unit;

[0129] Determining the motion state of the target carrier as a motion state based on motion state parameters collected by at least one auxiliary sensor includes:

[0130] The variance of the acceleration of the target carrier is calculated based on the motion state parameters collected by the inertial measurement unit; a dynamically determined set threshold is obtained.

[0131] If the variance of the acceleration of the target carrier is less than a set threshold, the motion state of the target carrier is determined to be a stationary state; or if the variance of the acceleration of the target carrier is not less than a set threshold, the motion state of the target carrier is determined to be a moving state.

[0132] It should be noted that the motion state parameters collected by the inertial measurement unit may contain interference parameters such as high-frequency noise or instantaneous outliers. Interference parameters can be filtered out first by low-pass or median filtering, and then the acceleration variance can be calculated in order to evaluate the motion state of the target carrier as a whole by combining it with the set threshold.

[0133] The statement that the variance of the target carrier's acceleration is not less than a set threshold means that the variance of the target carrier's acceleration is greater than or equal to the set threshold.

[0134] The specific value of the dynamically determined threshold can be determined according to the actual business scenario of the target carrier, and this invention does not limit it.

[0135] It is understandable that the acceleration of a target carrier in a stationary state is mainly composed of gravitational acceleration, and its acceleration variance is relatively small. The acceleration of a target carrier in a moving state is constantly changing, and its acceleration variance is relatively large. Thus, if an appropriate threshold is determined in advance based on the actual business scenario of the target carrier, the motion state of the target carrier can be accurately determined by combining the calculated acceleration variance.

[0136] According to any of the above embodiments, calculating the variance of the acceleration of the target carrier based on the motion state parameters collected by the inertial measurement unit includes:

[0137] Acquire at least two sets of triaxial acceleration measurement values ​​collected by the inertial measurement unit within a preset time period;

[0138] The gravity vector is projected based on the target carrier's attitude at the sampling time, and each set of triaxial acceleration measurements is calibrated by combining the zero bias error of the inertial measurement unit.

[0139] The variance of the target carrier's acceleration is calculated based on the calibrated triaxial acceleration measurements.

[0140] It should be noted that multiple sets of triaxial acceleration measurement values ​​collected by the inertial measurement unit within a preset time period can be obtained through a sliding window. The gravity component is projected onto the coordinate system of the inertial measurement unit through the target carrier attitude, so as to eliminate the influence of gravity from the triaxial acceleration measurement values ​​and eliminate the influence of zero bias error from the triaxial acceleration measurement values, thus obtaining calibrated triaxial acceleration measurement values.

[0141] Among them, triaxial acceleration refers to the acceleration of the target carrier in the X-axis, Y-axis and Z-axis directions.

[0142] For example, after obtaining the calibrated triaxial acceleration measurements, the average acceleration of the target carrier can be calculated using the following formula. :

[0143]

[0144] in, Indicating the inertial measurement unit (IMU) i The triaxial acceleration measurements collected this time. Indicates the target carrier in the first... i The target carrier attitude during the next data acquisition. Represents gravitational acceleration. This represents the zero bias error of the inertial measurement unit. This indicates the total number of times data was collected.

[0145] The variance of the target vehicle's acceleration can be calculated using the following formula. :

[0146]

[0147] Figure 4 This is a schematic diagram of the architecture of the target carrier state estimation method provided by the present invention, as shown below. Figure 4 As shown, in order to illustrate the function of the target carrier state estimation method provided in this embodiment, a specific example is provided below.

[0148] The system acquires the current image frame captured by the image sensor and performs feature extraction on the current image frame to obtain the original feature point set; it also acquires motion state parameters captured by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame.

[0149] The inter-frame pose increment is calculated based on the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame; the predicted target carrier pose corresponding to the current image frame is determined based on the inter-frame pose increment and the target carrier pose corresponding to the previous image frame; the scene depth of each feature point in the original feature point set and the feature point measurement value of the corresponding feature point in the previous image frame are obtained; the predicted feature point value of each feature point in the original feature point set of the current image frame is determined based on the predicted target carrier pose, scene depth, and the feature point measurement value of the corresponding feature point in the previous image frame; the prior value and the measured value of each feature point in the original feature point set are determined; wherein, the prior value of the feature point is the measured value of the corresponding feature point in the previous image frame;

[0150] At least one auxiliary sensor includes an inertial measurement unit, which calculates the variance of the target carrier's acceleration based on the motion state parameters collected by the inertial measurement unit.

[0151] If the variance of the target carrier's acceleration is less than a set threshold, the motion state of the target carrier is determined to be a stationary state. The difference between the prior value and the predicted value of the feature point in the original feature point set is within a first set range; the difference between the prior value and the measured value of the feature point in the original feature point set is within a second set range, wherein the second set range is greater than the first set range; the feature point is determined to be a dynamic feature point.

[0152] Alternatively, if the variance of the target carrier's acceleration is not less than a set threshold, the motion state of the target carrier is determined to be a motion state, and the difference between the measured value and the prior value of the feature point in the original feature point set is within a third set range; the difference between the measured value and the predicted value of the feature point in the original feature point set is within a fourth set range; the angle between the first line connecting the measured value and the prior value of the feature point in the original feature point set and the second line connecting the measured value and the predicted value of the feature point in the original feature point set is within a fifth set range; the feature point is determined to be a dynamic feature point.

[0153] The solution feature point set of the current image frame acquired by the image sensor is obtained by identifying and removing dynamic feature points from the original feature point set; the carrier pose is calculated based on the solution feature point set of the current image frame acquired by the image sensor to estimate the state of the target carrier.

[0154] The target carrier state estimation method provided in this invention collects motion state parameters of the target carrier through at least one auxiliary sensor and assists in calculating the feature prediction values ​​of feature points. It can reduce the measurement error or noise influence of a single sensor by using multiple auxiliary sensors. Furthermore, based on the feature point prediction value, feature point prior value, and feature point measurement value of each feature point, the dynamic feature points are judged by determining the motion state of the target carrier and adopting corresponding recognition rules, which helps to improve the recognition accuracy of dynamic feature points.

[0155] The target carrier state estimation device provided by the present invention is described below. The target carrier state estimation device described below and the target carrier state estimation method described above can be referred to in correspondence.

[0156] Figure 5 This is a schematic diagram of the target carrier state estimation device provided by the present invention. The target carrier is equipped with an image sensor, such as... Figure 5 As shown, the device includes:

[0157] The original feature point set acquisition module 510 is used to acquire the current image frame collected by the image sensor and perform feature extraction on the current image frame collected by the image sensor to obtain the original feature point set.

[0158] The motion state parameter acquisition module 520 is used to acquire motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame;

[0159] The dynamic feature point recognition module 530 is used to identify dynamic feature points from the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor.

[0160] The feature point set acquisition module 540 is used to remove the dynamic feature points from the original feature point set to obtain the feature point set of the current image frame acquired by the image sensor;

[0161] The state estimation module 550 is used to estimate the state of the target carrier based on the solved feature point set of the current image frame acquired by the image sensor.

[0162] According to any of the above embodiments, the dynamic feature point recognition module 530 includes:

[0163] The first dynamic feature point recognition unit is used to identify dynamic feature points from the original feature point set according to a first recognition rule when the motion state of the target carrier is determined to be stationary based on the motion state parameters collected by the at least one auxiliary sensor; or

[0164] The second dynamic feature point recognition unit is used to identify dynamic feature points from the original feature point set by means of a second recognition rule when the motion state of the target carrier is determined to be in motion state based on the motion state parameters collected by the at least one auxiliary sensor.

[0165] According to any of the above embodiments, the device further includes a feature point parameter determination unit, configured to determine the predicted feature point value of each feature point in the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor, and to determine the prior feature point value and the measured feature point value of each feature point in the original feature point set; wherein, the prior feature point value is the measured feature point value of the corresponding feature point in the previous image frame;

[0166] The dynamic feature point recognition module 530 is specifically used to identify dynamic feature points from the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor and the prior value, measured value, and predicted value of each feature point in the original feature point set.

[0167] According to any of the above embodiments, the feature point parameter determination unit is specifically used for:

[0168] The predicted target carrier pose corresponding to the current image frame is determined based on the motion state parameters collected by the at least one auxiliary sensor, and the scene depth of each feature point in the original feature point set and the feature point measurement value of the corresponding feature point in the previous image frame are obtained.

[0169] The predicted feature point value of each feature point in the original feature point set of the current image frame is calculated and determined based on the predicted target carrier pose, the scene depth, and the feature point measurement value of the corresponding feature point in the previous image frame.

[0170] According to any of the above embodiments, the feature point parameter determination unit is specifically used to determine the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame and the target carrier pose corresponding to the previous image frame, so as to calculate and determine the predicted target carrier pose corresponding to the current image frame.

[0171] According to any of the above embodiments, the feature point parameter determination unit is specifically used for:

[0172] The inter-frame pose increment is calculated based on the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame.

[0173] The predicted target carrier pose of the current image frame is determined by calculating the inter-frame pose increment and the target carrier pose corresponding to the previous frame image.

[0174] According to any of the above embodiments, the at least one auxiliary sensor includes an inertial measurement unit;

[0175] The dynamic feature point recognition module 530 is specifically used for:

[0176] The variance of the acceleration of the target carrier is calculated based on the motion state parameters collected by the inertial measurement unit.

[0177] If the variance of the acceleration of the target carrier is less than a set threshold, the motion state of the target carrier is determined to be a stationary state; or if the variance of the acceleration of the target carrier is not less than a set threshold, the motion state of the target carrier is determined to be a moving state.

[0178] According to any of the above embodiments, the first dynamic feature point recognition unit is specifically used to determine that the difference between the prior value and the predicted value of the feature point in the original feature point set is within a first preset range; the difference between the prior value and the measured value of the feature point in the original feature point set is within a second preset range, wherein the second preset range is greater than the first preset range; and to determine that the feature point is a dynamic feature point; or

[0179] The second dynamic feature point recognition unit is specifically used to determine that the feature point is a dynamic feature point when the difference between the measured value and the prior value of the feature point in the original feature point set is within a third preset range; the difference between the measured value and the predicted value of the feature point in the original feature point set is within a fourth preset range; and the angle between the first line connecting the measured value and the prior value of the feature point in the original feature point set and the second line connecting the measured value and the predicted value of the feature point in the original feature point set is within a fifth preset range.

[0180] Figure 6 An example is a schematic diagram of the structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a target carrier state estimation method. This method includes: acquiring a current image frame collected by the image sensor and performing feature extraction on the current image frame to obtain an original feature point set; acquiring motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame; identifying dynamic feature points from the original feature point set based on the motion state parameters collected by the at least one auxiliary sensor; removing the dynamic feature points from the original feature point set to obtain a calculated feature point set for the current image frame collected by the image sensor; and performing state estimation on the target carrier based on the calculated feature point set for the current image frame collected by the image sensor.

[0181] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating the state of a target carrier, characterized in that, The target carrier is equipped with an image sensor, and the method includes: The current image frame acquired by the image sensor is obtained, and feature extraction is performed on the current image frame acquired by the image sensor to obtain the original feature point set; Acquire motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame; Based on the motion state parameters collected by the at least one auxiliary sensor, dynamic feature points are identified from the original feature point set; wherein, based on the motion state parameters collected by the at least one auxiliary sensor, the motion state of the target carrier is determined, and dynamic feature points are identified from the original feature point set based on the motion state; the at least one auxiliary sensor includes an inertial measurement unit; the variance of the acceleration of the target carrier is calculated based on the motion state parameters collected by the inertial measurement unit; a dynamically determined set threshold is obtained; if the variance of the acceleration of the target carrier is less than the set threshold, the motion state of the target carrier is determined to be a stationary state; or if the variance of the acceleration of the target carrier is not less than the set threshold, the motion state of the target carrier is determined to be a moving state. The dynamic feature points are removed from the original feature point set to obtain the solution feature point set of the current image frame acquired by the image sensor; The state of the target carrier is estimated based on the feature point set of the current image frame acquired by the image sensor.

2. The target carrier state estimation method according to claim 1, characterized in that, The process of identifying dynamic feature points from the original feature point set based on the motion state includes: When the target carrier is in a stationary state, dynamic feature points are identified from the original feature point set using a first identification rule; or Based on the motion state parameters collected by the at least one auxiliary sensor, if the motion state of the target carrier is determined to be in motion, dynamic feature points are identified from the original feature point set by the second identification rule. The first identification rule is that the difference between the prior value and the predicted value of the feature point in the original feature point set is within a first set range, and the difference between the prior value and the measured value of the feature point is within a second set range, wherein the second set range is greater than the first set range. The second identification rule is that the difference between the measured value of a feature point and the prior value of a feature point in the original feature point set is within a third set range, the difference between the measured value of a feature point and the predicted value of a feature point is within a fourth set range, and the angle between the first line connecting the measured value of a feature point and the prior value of a feature point and the second line connecting the measured value of a feature point and the predicted value of a feature point is within a fifth set range. Wherein, the predicted feature point value represents the position of each feature point in the current image frame as predicted, the measured feature point value represents the position of each feature point in the current image frame as actually measured, and the prior feature point value is the measured feature point value of the corresponding feature point in the previous image frame.

3. The target carrier state estimation method according to claim 2, characterized in that, Before identifying dynamic feature points from the original feature point set, the method further includes: Based on the motion state parameters collected by the at least one auxiliary sensor, determine the predicted value of each feature point in the original feature point set, and determine the prior value and measured value of each feature point in the original feature point set. The step of identifying dynamic feature points from the original feature point set based on motion state parameters collected by at least one auxiliary sensor includes: Based on the motion state parameters collected by the at least one auxiliary sensor and the prior value, measured value, and predicted value of each feature point in the original feature point set, dynamic feature points are identified from the original feature point set.

4. The target carrier state estimation method according to claim 3, characterized in that, The step of identifying dynamic feature points from the original feature point set using the first identification rule includes: The difference between the prior value and the predicted value of the feature points in the original feature point set is within a first preset range; the difference between the prior value and the measured value of the feature points in the original feature point set is within a second preset range, wherein the second preset range is greater than the first preset range; the feature point is determined to be a dynamic feature point; or The step of identifying dynamic feature points from the original feature point set using the second identification rule includes: The difference between the measured value and the prior value of the feature point in the original feature point set is within a third set range; the difference between the measured value and the predicted value of the feature point in the original feature point set is within a fourth set range; the angle between the first line connecting the measured value and the prior value of the feature point in the original feature point set and the second line connecting the measured value and the predicted value of the feature point in the original feature point set is within a fifth set range; the feature point is determined to be a dynamic feature point.

5. The target carrier state estimation method according to claim 3, characterized in that, Determining the predicted feature point value for each feature point in the original feature point set based on motion state parameters collected by at least one auxiliary sensor includes: The predicted target carrier pose corresponding to the current image frame is determined based on the motion state parameters collected by the at least one auxiliary sensor, and the scene depth of each feature point in the original feature point set and the feature point measurement value of the corresponding feature point in the previous image frame are obtained. The predicted feature point value of each feature point in the original feature point set of the current image frame is calculated and determined based on the predicted target carrier pose, the scene depth, and the feature point measurement value of the corresponding feature point in the previous image frame.

6. The target carrier state estimation method according to claim 5, characterized in that, Determining the predicted target carrier pose corresponding to the current image frame based on motion state parameters collected by the at least one auxiliary sensor includes: The motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame and the target carrier pose corresponding to the previous image frame are determined in order to calculate and determine the predicted target carrier pose corresponding to the current image frame.

7. The target carrier state estimation method according to claim 6, characterized in that, The step of determining the motion state parameters acquired by at least one auxiliary sensor corresponding to the previous image frame and the target carrier pose corresponding to the previous image frame, in order to calculate and determine the predicted target carrier pose corresponding to the current image frame, includes: The inter-frame pose increment is calculated based on the motion state parameters collected by at least one auxiliary sensor corresponding to the previous image frame. The predicted target carrier pose of the current image frame is determined by calculating the inter-frame pose increment and the target carrier pose corresponding to the previous frame image.

8. A target carrier state estimation device, characterized in that, The target carrier is equipped with an image sensor, and the device includes: The original feature point set acquisition module is used to acquire the current image frame collected by the image sensor and perform feature extraction on the current image frame collected by the image sensor to obtain the original feature point set; A motion state parameter acquisition module is used to acquire motion state parameters collected by at least one auxiliary sensor; wherein the motion state parameters correspond to the current image frame; A dynamic feature point recognition module is used to identify dynamic feature points from the original feature point set based on motion state parameters collected by at least one auxiliary sensor; wherein, the motion state of the target carrier is determined based on the motion state parameters collected by at least one auxiliary sensor, and dynamic feature points are identified from the original feature point set based on the motion state; the at least one auxiliary sensor includes an inertial measurement unit; the variance of the acceleration of the target carrier is calculated based on the motion state parameters collected by the inertial measurement unit; a dynamically determined set threshold is obtained; if the variance of the acceleration of the target carrier is less than the set threshold, the motion state of the target carrier is determined to be a stationary state; or if the variance of the acceleration of the target carrier is not less than the set threshold, the motion state of the target carrier is determined to be a moving state. The feature point set acquisition module is used to remove the dynamic feature points from the original feature point set to obtain the feature point set of the current image frame acquired by the image sensor; The state estimation module is used to estimate the state of the target carrier based on the solved feature point set of the current image frame acquired by the image sensor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the target carrier state estimation method as described in any one of claims 1 to 7.

Images