Pose determination method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure disclose a pose determination method, device, equipment and medium, the method comprising: acquiring a current frame image collected by a camera and inertial data collected by an inertial measurement unit; performing semantic segmentation on the current frame image to obtain a semantic segmentation map; wherein the semantic segmentation map is used to identify a dynamic pixel region belonging to a set of dynamic categories in the current frame image; determining a feature point in a set of visual feature points extracted from the current frame image as a dynamic feature point if the feature point is located in the dynamic pixel region, and filtering out the dynamic feature points in the set of visual feature points to obtain a target set of visual feature points; and determining a device pose at a current time according to the target set of visual feature points and the inertial data.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a pose determination method, apparatus, device, and medium. Background Technology

[0002] Visual-Inertial Odometry (VIO) is a core technology for achieving autonomous positioning of devices such as mobile robots, augmented reality devices, and handheld gimbals by fusing camera images with inertial measurement unit (IMU) data.

[0003] Traditional VIO methods treat all feature points extracted from an image equally. However, in real-world scenarios, there are often moving objects such as pedestrians and vehicles. These dynamic feature points can incorrectly constrain the object's own motion in pose estimation, leading to pose jumps or drifts.

[0004] Therefore, how to accurately identify and filter out dynamic feature points to improve the robustness and accuracy of VIO in dynamic environments has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this disclosure is to provide a new technical solution for pose determination.

[0006] According to a first aspect of the present disclosure, a pose determination method is provided, the method comprising: Acquire the current frame image captured by the camera and the inertial data collected by the inertial measurement unit; Semantic segmentation is performed on the current frame image to obtain a semantic segmentation map; wherein, the semantic segmentation map is used to identify dynamic pixel regions in the current frame image that belong to a set of dynamic categories; Feature points located within the dynamic pixel region from the set of visual feature points extracted from the current frame image are identified as dynamic feature points, and dynamic feature points in the set of visual feature points are filtered out to obtain the target set of visual feature points; The device pose at the current moment is determined based on the target visual feature point set and the inertial data.

[0007] Optionally, performing semantic segmentation on the current frame image to obtain a semantic segmentation map includes: The current frame image is input into the set semantic segmentation model to obtain the semantic segmentation map.

[0008] Optionally, the method further includes identifying the dynamic pixel region in the semantic segmentation map, wherein identifying the dynamic pixel region in the semantic segmentation map includes: Obtain the set of defined dynamic categories; wherein the set of defined dynamic categories includes at least one dynamic semantic category; In the semantic segmentation image, the pixel value of the pixel point whose pixel value belongs to any dynamic semantic category in the set dynamic category is set as the first value, and the remaining pixels are set as the second value to generate a dynamic object binary mask. The pixel region in the binary mask of the dynamic object that takes the value of the first value is determined as the dynamic pixel region.

[0009] Optionally, determining feature points located within the dynamic pixel region from the set of visual feature points extracted from the current frame image as dynamic feature points includes: Obtain the pixel position of each feature point in the set of visual feature points; Based on the pixel position of each feature point, the dynamic pixel region is queried; If the pixel position of the feature point is located within the dynamic pixel region, the feature point is determined to be the dynamic feature point.

[0010] Optionally, the method further includes: For any of the dynamic feature points, determine the reprojection error between the dynamic feature point and the target matrix; If the reprojection error is greater than the average reprojection error of the static feature points in the visual feature point set, the step of filtering out the dynamic feature points in the visual feature point set continues. If the reprojection error is less than or equal to the average reprojection error of the static feature points in the visual feature point set, the dynamic feature points are recalled to the visual feature point set.

[0011] Optionally, determining the device pose at the current moment based on the target visual feature point set and the inertial data includes: Based on the reprojection error of the feature points in the target visual feature point set, a first residual term is constructed; Construct a second residual term based on the inertial data; Construct the objective function based on the first residual term and the second residual term; Optimize the objective function to determine the device pose at the current moment.

[0012] Optionally, the method further includes: Feature point tracking is performed on the current frame image to obtain the set of visual feature points.

[0013] According to a second aspect of the present disclosure, a pose determination apparatus is provided, the apparatus comprising: The acquisition module is used to acquire the current frame image captured by the camera and the inertial data acquired by the inertial measurement unit; A semantic segmentation module is used to perform semantic segmentation on the current frame image to obtain a semantic segmentation map; wherein, the semantic segmentation map is used to identify dynamic pixel regions in the current frame image that belong to a set of defined dynamic categories; The filtering module is used to determine the feature points located in the dynamic pixel region from the set of visual feature points extracted from the current frame image as dynamic feature points, and filter out the dynamic feature points in the set of visual feature points to obtain the target set of visual feature points. The determination module is used to determine the device pose at the current moment based on the target visual feature point set and the inertial data.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory for storing executable computer instructions; and a processor for executing the method described in accordance with the first aspect above, under the control of the executable computer instructions.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, the computer program implementing the method according to the first aspect above when executed by a processor.

[0016] This disclosure provides a pose determination method that simultaneously acquires the current frame image and inertial data. By performing semantic segmentation on the current frame image, it identifies dynamic pixel regions belonging to a set of defined dynamic categories. Visual feature points located within these dynamic pixel regions are then filtered out as dynamic feature points, resulting in a target visual feature point set containing only static background feature points. This target visual feature point set is then fused with the inertial data for pose calculation. In dynamic environments such as pedestrian traffic and vehicle movement, this method can accurately eliminate contamination of pose estimation by moving objects at the data source, avoid introducing erroneous constraints by dynamic feature points, effectively suppress pose jumps or drifts, and significantly improve the robustness and accuracy of visual inertial odometry pose estimation in dynamic scenes.

[0017] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0019] Figure 1This is a schematic flowchart of a pose determination method provided in an embodiment of this disclosure; Figure 2 This is a schematic flowchart of a pose determination method provided in another embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of a pose determination device provided in an embodiment of this disclosure; Figure 4 This is a block diagram of the hardware configuration of an electronic device that implements a pose determination method according to an embodiment of this disclosure. Detailed Implementation

[0020] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.

[0021] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0022] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0023] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0025] <Method Example 1> This disclosure provides a pose determination method for use in electronic devices, such as mobile robots equipped with cameras and inertial measurement units (IMUs), smartphones, augmented reality (AR) / virtual reality (VR) headsets, etc. Figure 1 As shown, the pose determination method of this embodiment may include the following steps S1100 to S1400: Step S1100: Acquire the current frame image captured by the camera and the inertial data collected by the inertial measurement unit.

[0026] The current frame image is one frame in a sequence of images continuously captured by the camera during its motion. The camera can capture images of the surrounding environment at a first frame rate (e.g., 30 frames / second).

[0027] Inertial data is acquired by an inertial measurement unit (IMU), which is typically rigidly connected to the camera and synchronized in time. In practical applications, the IMU's sampling frequency is usually much higher than the camera's frame rate (e.g., 100Hz or higher), enabling the IMU to capture high-frequency motion information between two frames of the camera image. An IMU typically includes a three-axis accelerometer and a three-axis gyroscope. The accelerometer measures the acceleration of the electronic device, and the gyroscope measures its angular velocity. Correspondingly, the inertial data can be acceleration data and angular velocity data.

[0028] Step S1200: Perform semantic segmentation on the current frame image to obtain a semantic segmentation map. The semantic segmentation map is used to identify dynamic pixel regions in the current frame image that belong to a set of dynamic categories.

[0029] Semantic segmentation is a pixel-level classification task, and its output, a "semantic segmentation map," has the same size as the current frame image. In this semantic segmentation map, the pixel value (or label) of each pixel represents the object category to which the corresponding pixel belongs in the current frame image. For example, pixel value "0" can represent the category "person," pixel value "1" can represent the category "vehicle," pixel value "2" can represent the category "ground," pixel value "3" can represent the category "building," and so on.

[0030] In this context, dynamic pixel regions refer to image regions in the current frame that belong to a defined dynamic category set. This set includes, but is not limited to, object categories with independent motion capabilities such as pedestrians, cyclists, cars, buses, and motorcycles. In traditional visual inertial odometry (VIO) methods, feature points on these dynamic objects (i.e., dynamic feature points) are treated the same as feature points on a static background, participating in pose estimation. Because dynamic feature points possess real displacements independent of camera motion, they incorrectly impose the object's motion onto the camera motion estimation, leading to jumps or drifts in pose estimation. This step identifies these dynamic pixel regions through semantic segmentation, allowing feature points within these regions to be filtered out subsequently, preventing dynamic objects from contaminating pose estimation at the data source.

[0031] For example, if the current frame image is an urban road scene, the dynamic pixel regions identified through semantic segmentation can include regions of moving cars, walking pedestrians, and moving motorcycles.

[0032] In one embodiment, step S1200, which involves semantic segmentation of the current frame image to obtain a semantic segmentation map, may further include: inputting the current frame image into a set semantic segmentation model to obtain a semantic segmentation map.

[0033] This semantic segmentation model can employ a lightweight semantic segmentation neural network.

[0034] In some examples, this lightweight semantic segmentation neural network can use MobileNetV3 as the encoder and LR-ASPP (Lite Reduced Atrous Spatial Pyramid Pooling) as the decoder. In these examples, by employing a lightweight semantic segmentation neural network with MobileNetV3 as the encoder and LR-ASPP as the decoder, high-precision semantic segmentation tasks can be completed at near real-time speeds on computationally limited platforms. This effectively balances the accuracy of semantic understanding with the real-time requirements of pose estimation, avoiding the problem of excessively high overall latency caused by introducing complex semantic segmentation models.

[0035] Step S1300: The feature points located in the dynamic pixel region from the set of visual feature points extracted from the current frame image are determined to be dynamic feature points, and the dynamic feature points in the set of visual feature points are filtered out to obtain the target set of visual feature points.

[0036] The visual feature point set refers to the set of visual feature points extracted from the current frame image using traditional feature extraction methods (such as optical flow and feature matching). Each feature point includes pixel coordinates, which are used for subsequent pose calculation. In other words, the visual feature point set is obtained by tracking feature points in the current frame image using traditional feature extraction methods.

[0037] Among them, dynamic feature points refer to visual feature points located in dynamic pixel areas, that is, feature points in pixel areas corresponding to object categories with independent movement capabilities such as pedestrians, cyclists, cars, buses, and motorcycles.

[0038] The target visual feature point set refers to the set of visual feature points remaining after filtering out dynamic feature points from the visual feature point set. The feature points in the target visual feature point set are all located outside the dynamic pixel region and belong to static background feature points, which are used for subsequent pose calculation.

[0039] In one example, step S1300, which determines a feature point located within a dynamic pixel region from the set of visual feature points extracted from the current frame image as a dynamic feature point, may further include: obtaining the pixel position of each feature point in the set of visual feature points; querying the dynamic pixel region based on the pixel position of each feature point; and determining the feature point as a dynamic feature point if the pixel position of the feature point is within the dynamic pixel region, so as to directly filter out the dynamic feature points in the set of visual feature points.

[0040] In another example, to address potential boundary errors or misclassifications in semantic segmentation, step S1300 may further include a posterior verification step: for feature points identified as dynamic feature points from the set of visual feature points extracted from the current frame image, determining the reprojection error between the dynamic feature point and the target matrix, which may include an essential matrix or a homography matrix; if the reprojection error is greater than the average reprojection error of static feature points in the set of visual feature points, directly filtering out the dynamic feature point from the set of visual feature points; if the reprojection error is less than or equal to the average reprojection error of static feature points in the set of visual feature points, recalling the dynamic feature point back into the set of visual feature points.

[0041] Static feature points refer to visual feature points located outside the dynamic pixel region. They belong to the pixel region corresponding to static backgrounds (such as ground, walls, buildings, roads, and other objects that do not have independent movement capabilities), and are also called static background feature points. These static feature points are stationary relative to the world coordinate system in the real world. Static feature points provide realistic and reliable geometric constraints for camera motion estimation and are the basis for pose calculation in visual inertial odometry.

[0042] For example, if the current frame image is an urban road scene, the dynamic pixel regions identified through semantic segmentation can include regions of moving cars, walking pedestrians, and moving motorcycles. Before filtering, the set of visual feature points can include: feature points on moving cars (such as headlight corner points and license plate edge points), feature points on pedestrians (such as pedestrian outline corner points and clothing texture points), feature points on moving motorcycles (such as wheel edge points and reflective strip points), corner points on road lane lines, feature points on static vehicles on the roadside (such as unmanned vehicles parked on the roadside), feature points on the edges of traffic signs, feature points on the exterior walls of buildings, and feature points on the curbs on both sides of the road.

[0043] After filtering out dynamic feature points in step S1300, dynamic feature points located on moving cars, pedestrians, and moving motorcycles are removed. The resulting set of target visual feature points can include: corner points located on road lane lines, feature points located on static vehicles on the roadside, feature points located on the edges of traffic signs, feature points located on the exterior walls of buildings, and feature points located on the curb stones on both sides of the road. These feature points all belong to static background feature points.

[0044] In these examples, by directly determining and filtering dynamic feature points based on semantic categories, the contamination of pose estimation by dynamic objects is eliminated from the data source. At the same time, by performing posterior verification of the filtering decisions through a lightweight geometric verification step, the boundary error or misclassification problem of semantic segmentation can be effectively addressed, further improving the accuracy of filtering.

[0045] Step S1400: Determine the current device pose based on the target visual feature point set and inertial data.

[0046] Device pose typically includes the position (three-dimensional vector) and orientation (usually represented by quaternions or Euler angles) of the electronic device in the world coordinate system.

[0047] In one example, step S1400, which determines the device pose at the current moment based on the target visual feature point set and inertial data, may further include the following steps S1410 to S1440: Step S1410: Construct the first residual term based on the reprojection error of the feature points in the target visual feature point set.

[0048] The first residual term can be called the visual reprojection error term, which is used to introduce geometric constraints on visual feature points into the objective function.

[0049] Specifically, for each visual feature point in the target visual feature point set, its corresponding 3D spatial point is reprojected onto the current frame image, and the residual between the reprojected position and the actual observation position is calculated as the reprojection error of that feature point. The first residual term is the sum (or weighted sum) of the reprojection errors of all feature points in the target visual feature point set, used to constrain the reprojection residual to approach zero.

[0050] For example, if the target visual feature point set contains multiple feature points located on a static background (such as corner points on road lane lines, feature points on the edges of traffic signs, etc.), the reprojection error terms of these feature points are used to construct the first residual term, providing a reliable visual observation constraint for pose optimization.

[0051] Step S1420: Construct the second residual term based on the inertial data.

[0052] The second residual term can be called the inertial pre-integration error term, which is used to introduce short-time motion constraints of the inertial measurement unit into the objective function.

[0053] Specifically, the inertial data between two consecutive frames can be pre-integrated to obtain a pre-integrated quantity, which can be used as the actual measured value. Then, the corresponding predicted motion is calculated based on the device pose (including position and attitude) to be optimized. Next, the difference between the predicted motion and the actual measured value is used as a second residual term to ensure that the inertial pre-integration result is consistent with the pose estimation result.

[0054] Step S1430: Construct the objective function based on the first residual term and the second residual term.

[0055] The objective function can be a loss function in the form of nonlinear least squares.

[0056] Specifically, corresponding weight coefficients can be assigned to the first residual term and the second residual term respectively. Then the objective function can be expressed as the weighted sum of squares of the first residual term and the second residual term, that is: objective function = weighted square of the first residual term + weighted square of the second residual term.

[0057] Step S1440: Optimize the objective function and determine the device pose at the current moment.

[0058] In these examples, by jointly constructing the residual terms corresponding to the visual feature point reprojection error and the residual terms corresponding to the inertial measurement unit pre-integration error into a nonlinear least squares objective function and performing joint optimization, the complementary characteristics of the two types of information can be fully utilized: the visual feature point reprojection error provides accurate local geometric constraints, while the inertial pre-integration error provides short-time motion estimation and absolute scale information. The two are optimized synergistically, and since dynamic feature points have been filtered out in step S1300, the target visual feature point set only contains static background feature points, thus avoiding contamination of pose estimation by dynamic objects. This allows the device to obtain high-precision and highly robust pose estimation even in dynamic environments.

[0059] Through the embodiments of this disclosure, it can simultaneously acquire the current frame image and inertial data. By performing semantic segmentation on the current frame image, it identifies dynamic pixel regions belonging to a set of defined dynamic categories, and filters out visual feature points located within these dynamic pixel regions as dynamic feature points, obtaining a target visual feature point set containing only static background feature points. This target visual feature point set is then fused with the inertial data for pose estimation. In dynamic environments such as pedestrian traffic and vehicle movement, it can accurately remove contamination from moving objects at the data source, avoid introducing erroneous constraints by dynamic feature points, effectively suppress pose jumps or drifts, and significantly improve the robustness and accuracy of visual inertial odometry pose estimation in dynamic scenes.

[0060] <Method Example 2> Another embodiment of this disclosure provides a pose determination method, such as... Figure 2 As shown, the pose determination method of this disclosure embodiment may include the following steps S2100 to S2500: Step S2100: Acquire the current frame image captured by the camera and the inertial data collected by the inertial measurement unit.

[0061] Step S2200: Perform semantic segmentation on the current frame image to obtain a semantic segmentation map.

[0062] Step S2300: Identify dynamic pixel regions in the semantic segmentation map.

[0063] In some examples, step S2300, which identifies dynamic pixel regions in the semantic segmentation map, may further include the following steps S2310 to S2330: Step S2310: Obtain the set of dynamic categories.

[0064] The dynamic category set includes at least one dynamic semantic category. For example, the dynamic category set may include, but is not limited to, object categories with independent movement capabilities such as pedestrians, cyclists, cars, buses, and motorcycles.

[0065] Step S2320: Set the pixel value of the pixel in the semantic segmentation image that belongs to any dynamic semantic category in the dynamic category set to the first value, and set the other pixels to the second value to generate a dynamic object binary mask.

[0066] For example, for a pixel in the semantic segmentation image with the category label "pedestrian" or "car", its pixel value is set to 1 (the first value), indicating that the pixel belongs to a dynamic object; the pixel values ​​of all other pixels (such as "ground", "building", "sky", etc.) are set to 0 (the second value), indicating that the pixel belongs to the static background. The resulting binary mask for dynamic objects has the same size as the current frame image, and the value of each pixel identifies whether the location belongs to a dynamic object.

[0067] Step S2330: The pixel region with the first value in the binary mask of the dynamic object is determined as the dynamic pixel region.

[0068] Specifically, in the binary mask of a dynamic object, pixels with a value of the first value (such as 1) constitute one or more pixel regions, which are called dynamic pixel regions. Dynamic pixel regions can be spatially disconnected. For example, when multiple cars and multiple pedestrians appear simultaneously in the current frame image, the pixel region corresponding to each car and each pedestrian is an independent dynamic pixel region.

[0069] Based on steps S2310 to S2330 above, a binary mask for dynamic objects is generated based on the set of dynamic categories and the semantic segmentation map. This mask can accurately separate the pixel regions belonging to moving objects from the current frame image, providing a foundation for the rapid determination of subsequent dynamic feature points. Furthermore, the mask operation is simple, efficient, and computationally inefficient.

[0070] Step S2400: The feature points located in the dynamic pixel region from the set of visual feature points extracted from the current frame image are determined to be dynamic feature points, and the dynamic feature points in the set of visual feature points are filtered out to obtain the target set of visual feature points.

[0071] In some examples, step S2400, which determines feature points located within dynamic pixel regions from the set of visual feature points extracted from the current frame image as dynamic feature points and filters out dynamic feature points from the set of visual feature points to obtain the target set of visual feature points, may further include the following steps S2410 to S2430: Step S2410: Obtain the pixel position of each feature point in the visual feature point set.

[0072] Each feature point can include pixel coordinates (u_i, v_i), where u_i and v_i represent the column coordinates (horizontal coordinate) and row coordinates (vertical coordinate) of the feature point in the image, respectively, and are used for subsequent pose calculation.

[0073] Step S2420: Query the dynamic pixel region based on the pixel position of each feature point.

[0074] For each feature point, the mask value at the corresponding position can be found in the binary mask of the dynamic object based on its pixel coordinates (u_i, v_i).

[0075] Step S2430: If the pixel position of a feature point is located within a dynamic pixel region, the feature point is determined to be a dynamic feature point.

[0076] Specifically, if the mask value corresponding to the pixel coordinates of a feature point is the first value (e.g., 1), then the feature point is determined to be located within the dynamic pixel region, i.e., a dynamic feature point; if the corresponding mask value is the second value (e.g., 0), then the feature point is determined to be a static feature point and is retained.

[0077] Based on steps S2410 to S2430 above, the fast query and matching of the pixel position of the feature point and the dynamic pixel region can efficiently and accurately identify the dynamic feature point.

[0078] Step S2500: Determine the current device pose based on the target visual feature point set and inertial data.

[0079] Through the embodiments of this disclosure, dynamic pixel regions are identified and binary masks are generated after semantic segmentation. Based on pixel position, fast query matching is performed to accurately filter out dynamic feature points, thereby eliminating the pollution of pose estimation by dynamic objects from the source. The computational overhead is small, and the robustness and accuracy of visual inertial odometry in dynamic environments are significantly improved.

[0080] <Device Embodiment> This disclosure also provides a pose determination device, such as Figure 3 As shown, the pose determination device 300 may include an acquisition module 310, a semantic segmentation module 320, a filtering module 330, and a determination module 340.

[0081] The acquisition module 310 is used to acquire the current frame image captured by the camera and the inertial data acquired by the inertial measurement unit; The semantic segmentation module 320 is used to perform semantic segmentation on the current frame image to obtain a semantic segmentation map; wherein, the semantic segmentation map is used to identify dynamic pixel regions in the current frame image that belong to a set of dynamic categories; The filtering module 330 is used to determine the feature points located in the dynamic pixel region from the set of visual feature points extracted from the current frame image as dynamic feature points, and filter out the dynamic feature points in the set of visual feature points to obtain the target set of visual feature points. The determination module 340 is used to determine the device pose at the current moment based on the target visual feature point set and the inertial data.

[0082] In one embodiment, the semantic segmentation module 320 is used to input the current frame image into a set semantic segmentation model to obtain the semantic segmentation map.

[0083] In one embodiment, the device 300 further includes an identification module (not shown in the figure), which is used to obtain the set of dynamic categories; wherein the set of dynamic categories includes at least one dynamic semantic category; the pixel values ​​of pixels in the semantic segmentation image that belong to any dynamic semantic category in the set of dynamic categories are set to a first value, and the remaining pixels are set to a second value, thereby generating a dynamic object binary mask; the pixel regions in the dynamic object binary mask that take the first value are determined as the dynamic pixel regions.

[0084] In one embodiment, the filtering module 330 is used to obtain the pixel position of each feature point in the set of visual feature points; query the dynamic pixel region based on the pixel position of each feature point; and determine that the feature point is the dynamic feature point if the pixel position of the feature point is located within the dynamic pixel region.

[0085] In one embodiment, the determining module 340 is further configured to determine, for any one of the dynamic feature points, the reprojection error between the dynamic feature point and the target matrix; wherein the target matrix includes an essential matrix or a homography matrix; The filtering module 330 is used to continue executing the step of filtering out the dynamic feature points in the visual feature point set when the reprojection error is greater than the average reprojection error of the static feature points in the visual feature point set; and to recall the dynamic feature points to the visual feature point set when the reprojection error is less than or equal to the average reprojection error of the static feature points in the visual feature point set.

[0086] In one embodiment, the determining module 340 is configured to construct a first residual term based on the reprojection error of feature points in the target visual feature point set; construct a second residual term based on the inertial data; construct an objective function based on the first residual term and the second residual term; optimize the objective function to determine the device pose at the current moment.

[0087] In one embodiment, the acquisition module 310 is further configured to perform feature point tracking on the current frame image to obtain the set of visual feature points.

[0088] According to this embodiment, it can simultaneously acquire the current frame image and inertial data. By performing semantic segmentation on the current frame image, it identifies dynamic pixel regions belonging to a set of defined dynamic categories, and filters out visual feature points located within these dynamic pixel regions as dynamic feature points, obtaining a target visual feature point set containing only static background feature points. This target visual feature point set is then fused with the inertial data for pose estimation. In dynamic environments such as pedestrian traffic and vehicle movement, it can accurately remove contamination from moving objects at the data source, avoiding incorrect constraints introduced by dynamic feature points, effectively suppressing pose jumps or drifts, and significantly improving the robustness and accuracy of visual inertial odometry pose estimation in dynamic scenes.

[0089] <Equipment Example> This disclosure also provides an electronic device, such as Figure 4 As shown, the electronic device 400 includes a processor 410 and a memory 420.

[0090] The memory 420 can be used to store executable computer instructions.

[0091] The processor 410 can be used to execute a pose determination method according to an embodiment of the method disclosed herein, under the control of executable computer instructions.

[0092] In another embodiment, the electronic device 400 may include the above pose determination device 300.

[0093] In one embodiment, each module of the above pose determination device 300 can be implemented by the processor 410 running computer instructions stored in the memory 420.

[0094] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the pose determination method provided in this disclosure.

[0095] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0096] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0097] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0098] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0099] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0103] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A pose determination method, characterized in that, The method includes: Acquire the current frame image captured by the camera and the inertial data collected by the inertial measurement unit; Semantic segmentation is performed on the current frame image to obtain a semantic segmentation map; wherein, the semantic segmentation map is used to identify dynamic pixel regions in the current frame image that belong to a set of dynamic categories; Feature points located within the dynamic pixel region from the set of visual feature points extracted from the current frame image are identified as dynamic feature points, and dynamic feature points in the set of visual feature points are filtered out to obtain the target set of visual feature points; The device pose at the current moment is determined based on the target visual feature point set and the inertial data.

2. The method according to claim 1, characterized in that, The step of performing semantic segmentation on the current frame image to obtain a semantic segmentation map includes: The current frame image is input into the set semantic segmentation model to obtain the semantic segmentation map.

3. The method according to claim 1, characterized in that, The method further includes identifying the dynamic pixel regions in the semantic segmentation map, wherein identifying the dynamic pixel regions in the semantic segmentation map includes: Obtain the set of defined dynamic categories; wherein the set of defined dynamic categories includes at least one dynamic semantic category; In the semantic segmentation image, the pixel value of the pixel point whose pixel value belongs to any dynamic semantic category in the set dynamic category is set as the first value, and the remaining pixels are set as the second value to generate a dynamic object binary mask. The pixel region in the binary mask of the dynamic object that takes the value of the first value is determined as the dynamic pixel region.

4. The method according to claim 1, characterized in that, The step of determining the feature points located within the dynamic pixel region from the set of visual feature points extracted from the current frame image as dynamic feature points includes: Obtain the pixel position of each feature point in the set of visual feature points; Based on the pixel position of each feature point, the dynamic pixel region is queried; If the pixel position of the feature point is located within the dynamic pixel region, the feature point is determined to be the dynamic feature point.

5. The method according to claim 1, characterized in that, The method further includes: For any of the dynamic feature points, determine the reprojection error between the dynamic feature point and the target matrix; If the reprojection error is greater than the average reprojection error of the static feature points in the visual feature point set, the dynamic feature points in the visual feature point set are filtered out. If the reprojection error is less than or equal to the average reprojection error of the static feature points in the visual feature point set, the dynamic feature points are recalled to the visual feature point set.

6. The method according to claim 1, characterized in that, Determining the device pose at the current moment based on the target visual feature point set and the inertial data includes: Based on the reprojection error of the feature points in the target visual feature point set, a first residual term is constructed; Construct a second residual term based on the inertial data; Construct the objective function based on the first residual term and the second residual term; Optimize the objective function to determine the device pose at the current moment.

7. The method according to claim 1, characterized in that, The method further includes: Feature point tracking is performed on the current frame image to obtain the set of visual feature points.

8. A pose determination device, characterized in that, The device includes: The acquisition module is used to acquire the current frame image captured by the camera and the inertial data acquired by the inertial measurement unit; A semantic segmentation module is used to perform semantic segmentation on the current frame image to obtain a semantic segmentation map; wherein, the semantic segmentation map is used to identify dynamic pixel regions in the current frame image that belong to a set of defined dynamic categories; The filtering module is used to determine the feature points located in the dynamic pixel region from the set of visual feature points extracted from the current frame image as dynamic feature points, and filter out the dynamic feature points in the set of visual feature points to obtain the target set of visual feature points. The determination module is used to determine the device pose at the current moment based on the target visual feature point set and the inertial data.

9. An electronic device, characterized in that, include: Memory is used to store executable computer instructions; A processor configured to execute the method according to any one of claims 1-7, under the control of the executable computer instructions.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1-7.

Images