Unmanned aerial vehicle lightweight semantic visual inertial odometer method, medium and processing equipment

Abstract

The invention discloses an unmanned aerial vehicle lightweight semantic visual inertial odometer method, a medium and processing equipment, and the method comprises the steps: carrying out the semantic segmentation of image data through a three-branch semantic segmentation network, and obtaining a potential dynamic region mask; extracting a plurality of feature points of the image data, performing motion consistency check on the plurality of feature points in the potential dynamic region mask, screening out static feature points, performing feature tracking on the static feature points, and obtaining feature point information of the image data and a pose estimation result of the camera module; pre-integration information of the motion state data is obtained, state parameters of the camera module are optimized according to the feature point information, the pose estimation result and the pre-integration information, optimal pose estimation is obtained, and state pose information of the unmanned aerial vehicle is output according to the optimal pose estimation. According to the method and the device, high accuracy and real-time performance of positioning in a dynamic environment can be met, and the requirement on computing resources is also remarkably reduced, so that edge equipment with limited computing resources can be deployed favorably.

Description

Technical Field

[0001] The present application relates to the technical field of simultaneous positioning and mapping, and in particular to a lightweight semantic visual-inertial odometer method, medium and processing equipment for unmanned aerial vehicles. Background Art

[0002] Nowadays, Visual-Inertial Odometry (VIO) plays a vital role in drone positioning. By combining visual information and Inertial Measurement Unit (IMU) data, VIO systems are able to provide high-precision real-time position and attitude estimation. However, the system assumes that the drone is in a completely static scene, which is often inconsistent with reality because dynamic scenes are very common and the complete static nature of the environment cannot be guaranteed. In a dynamic environment, moving objects (such as moving pedestrians, moving vehicles, etc.) introduce additional image feature points, which may be confused with static feature points in the background environment, resulting in feature point matching errors. This confusion introduces incorrect visual constraints in the drone's attitude estimation process, which has a negative impact on the positioning accuracy and stability of the VIO system, which in turn leads to serious deviations in the positioning results.

[0003] In the existing technology, by introducing semantic information, the system can achieve more accurate positioning in dynamic scenes. However, the existing semantic visual inertial odometry systems usually face the problems of insufficient lightness and slow operation speed, especially the relatively complex semantic algorithm modules and high computational overhead, which makes it difficult to effectively deploy these systems on UAV platforms with limited computing resources, thus limiting their application in practical engineering.

[0004] Although some classic visual-inertial SLAM systems, such as DynaVINS and RDS-SLAM, have been applied to robot positioning research in dynamic scenes, there are still problems that need to be improved. The positioning accuracy of these systems in dynamic scenes needs to be further optimized and improved. At the same time, visual-inertial odometer systems combined with semantic information often have high requirements for computing resources. The algorithm model design of the semantic module is relatively complex and redundant, and it is difficult to ensure the real-time and stability of the algorithm. This makes it difficult to effectively deploy them on edge devices with limited computing resources, limiting their application in real dynamic scenes, thereby affecting the value of their engineering applications. Summary of the invention

[0005] The purpose of this application is to provide a lightweight semantic visual inertial odometer method, medium and processing equipment for unmanned aerial vehicles to solve the technical problems of the existing visual inertial odometer system in the prior art, such as high computing resource requirements and complex and redundant models. The preferred technical solutions among the many technical solutions provided in this application can produce many technical effects, which are described in detail below.

[0006] To achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, the present application provides a lightweight semantic visual inertial odometer method for an unmanned aerial vehicle, wherein the unmanned aerial vehicle includes a camera module for collecting image data in real time, and a sensor for collecting motion state data, the method comprising: using a three-branch semantic segmentation network to perform semantic segmentation on the image data to obtain a potential dynamic area mask; extracting multiple feature points of the image data, performing a motion consistency check on the multiple feature points in the potential dynamic area mask, screening out static feature points, performing feature tracking on the static feature points, obtaining feature point information of the image data and a pose estimation result of the camera module, wherein the three-branch semantic segmentation network is constructed based on deep separable convolution and a coordinate attention mechanism; obtaining pre-integration information of the motion state data, optimizing the state parameters of the camera module according to the feature point information, the pose estimation result and the pre-integration information to obtain an optimal pose estimation, and outputting the state pose information of the unmanned aerial vehicle according to the optimal pose estimation.

[0008] In some embodiments, extracting the plurality of feature points of the image data includes: in each frame of the image data, obtaining at least one feature point of each frame of the image using a feature extraction algorithm.

[0009] In some embodiments, the motion consistency check is performed on multiple feature points in the potential dynamic area mask to screen out static feature points, including: using a feature matching algorithm to obtain multiple matching feature point pairs of any two consecutive frame images; based on the matching feature point pairs, estimating the basic matrix using epipolar geometry constraints; using the basic matrix to calculate the epipolar line position of the corresponding matching feature point pair in the current frame image, if the distance between the position of the feature point in the current frame image and the epipolar line corresponding to the feature point is greater than or equal to a preset threshold, the feature point is output as the dynamic feature point, and if the distance between the position of the feature point in the current frame image and the epipolar line corresponding to the feature point is less than the preset threshold, the feature point is output as the static feature point.

[0010] In some embodiments, the feature tracking of the static feature points to obtain the feature point information of the image data and the pose estimation result of the camera module includes: filtering the static feature points, obtaining the feature point information based on the filtered static feature points, and obtaining the pose estimation result of the camera module based on the basic matrix of the filtered static feature points.

[0011] In some embodiments, the screening of the static feature points includes: using a sparse optical flow method to compare pixel changes between adjacent frame images to estimate the motion trajectory of the static feature points; judging whether the static feature points meet preset conditions based on optical flow estimation error, reprojection error and matching error, and screening out static feature points that meet the preset conditions.

[0012] In some embodiments, the obtaining of pre-integration information of the motion state data includes: performing pre-integration calculation on the motion state data, and obtaining measurement values ​​of position, velocity and posture increments between adjacent frame images as the pre-integration information based on the calculation results.

[0013] In some embodiments, optimizing the state parameters of the camera module to obtain the optimal pose estimation includes: optimizing the state parameters of the camera module using a graph optimization algorithm, adjusting the pose information of the camera module, and obtaining the optimal pose estimation after multiple iterations; the state parameters include the pose information, velocity information and IMU deviation information, wherein the IMU deviation information includes accelerometer deviation and gyroscope deviation.

[0014] In some embodiments, the loss function of the three-branch semantic segmentation network includes at least one of an additional semantic loss, a weighted binary cross entropy loss, a cross entropy loss, and a boundary-aware cross entropy loss.

[0015] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the lightweight semantic visual-inertial odometer method for a drone as described above is implemented.

[0016] In a third aspect, the present application provides a processing device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors are used to execute the one or more computer programs stored in the memory, so that the one or more processors execute the lightweight semantic visual inertial odometer method for unmanned aerial vehicles as described above.

[0017] Implementing one of the above technical solutions of the present application has the following advantages or beneficial effects: In the present application, a three-branch semantic segmentation network is used to semantically segment the image data, and a motion consistency check is performed on multiple feature points in the potential dynamic area mask, and static feature points are screened out for feature tracking, and finally the camera module pose is optimized in combination with the feature point information and the pose estimation result to obtain the state pose information of the drone. The three-branch semantic segmentation network can greatly reduce the computational complexity of the algorithm model while ensuring the accuracy of semantic segmentation, and significantly improve the reasoning speed. At the same time, the consistency check of the feature points, the screening and retention of static feature points can significantly improve the positioning accuracy and robustness of the drone in dynamic scenes. The embodiment of the present application can not only meet the high accuracy and real-time performance of positioning in a dynamic environment, but also significantly reduce the requirements for computing resources, which is conducive to its deployment to edge devices with limited computing resources. Compared with the prior art, the present application effectively improves the positioning accuracy and robustness of drones in dynamic scenes, and has great engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 It is a flowchart of a lightweight semantic visual inertial odometer method for a drone according to an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of the structure of a three-branch semantic segmentation network in an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of the result after semantic segmentation by a three-branch semantic segmentation network according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a motion consistency check according to an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram comparing the positioning trajectories of the embodiment of the present application with other visual-inertial SLAM systems;

[0024] Figure 6 It is another schematic diagram comparing positioning trajectories of the embodiment of the present application with other visual-inertial SLAM systems;

[0025] Figure 7 It is a structural block diagram of the processing device of an embodiment of the present application.

[0026] In the figure: 1. processing device; 10. memory; 11. processor. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present application are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, and devices that are consistent with some aspects disclosed in the present application as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present application.

[0028] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", etc. indicate the orientation or positional relationship based on the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0029] In order to illustrate the technical solution described in the present application, a specific embodiment is provided below, and only the parts related to the embodiment of the present application are shown.

[0030] like Figure 1 As shown, the present application provides a lightweight semantic visual inertial odometer method for a drone, comprising the following steps (step S1 to step S2):

[0031] S1. Use a three-branch semantic segmentation network to perform semantic segmentation on the image data to obtain a potential dynamic area mask; extract multiple feature points of the image data, perform motion consistency check on the multiple feature points in the potential dynamic area mask, screen out static feature points, perform feature tracking on the static feature points, and obtain feature point information of the image data and the pose estimation result of the camera module.

[0032] In some embodiments, the drone may include a camera module for collecting image data in real time, and a plurality of sensors for collecting motion status data.

[0033] In some embodiments, the camera module may be a camera device carried by the drone for capturing images, such as an RGB camera or a webcam. Image data may refer to a visual image of the drone's surrounding environment captured by the camera module, which may be used to identify and track feature points in the environment. Image data may include multiple consecutive frames of images captured by the camera module as the drone moves, that is, the image data may be multiple frames of images captured in real time by the camera module.

[0034] In some embodiments, a potential dynamic region mask may be used to indicate a dynamic region in an image environment. A dynamic region may represent a portion of a background that may include moving objects or changes. For example, in an application scenario where there are pedestrians in an environment, a dynamic region may refer to the region of the pedestrians. After identifying and segmenting the image data of the environment, the region of the pedestrians may be output as a potential dynamic region mask.

[0035] In some embodiments, a dynamic mask image can be obtained according to the potential dynamic region mask. The dynamic mask image is a binary image, and each mask value in the binary image can indicate whether the corresponding pixel point belongs to the dynamic region.

[0036] In some embodiments, the lightweight semantic visual inertial odometer method for drones may further include: obtaining pixel coordinates of feature points in the image, and obtaining mask values ​​in a dynamic mask map corresponding to a potential dynamic region mask according to the pixel coordinates. If the mask value corresponding to the feature point is 1, it may indicate that the feature point is located in a dynamic region, and if the mask value corresponding to the feature point is 0, it may indicate that the feature point is located in a static region. Thus, it is possible to determine whether a feature point is in a potential dynamic region according to the potential dynamic region mask.

[0037] In some embodiments, Figure 2As shown in the figure, the three-branch semantic segmentation network (LPIDNet, Lightweight Proportional-Integral-Derivative Network) can be built based on depthwise separable convolutions and coordinate attention mechanism. Compared with traditional standard convolutions, depthwise separable convolutions can significantly reduce the computational complexity of the model and improve the reasoning speed of the segmentation algorithm.

[0038] The computational complexity of standard convolution can be expressed by the following formula:

[0039] D K ·D K ·M·N·D F ·D F ,

[0040] Among them, M can represent the number of input channels of the feature, N can represent the number of output channels, and D K Can be the size of the convolution kernel, D F It can be the size of the feature map. Generally speaking, the feature map can refer to image data processed by the convolution layer and / or the pooling layer, and the size of the feature map can include the width and length of the feature map. The width of the feature map can refer to the number of pixels of the feature map in the horizontal direction, and the height of the feature map can refer to the number of pixels of the feature map in the vertical direction.

[0041] Generally speaking, standard convolution can complete channel-by-channel convolution and space-by-space convolution in one step.

[0042] The depth-wise separable convolution in the embodiment of the present application can be completed in two steps, that is, the first step of depth-wise convolution is completed first, and then the second step of point-wise convolution is completed.

[0043] In some embodiments, the depth-wise convolution may be expressed by the following formula:

[0044] D K ·D K ·M·D F ·D F ,

[0045] Among them, M can represent the number of input channels of the feature, D K Can be the size of the convolution kernel, D F It can be the size of the feature map.

[0046] In some embodiments, the point-by-point convolution may be expressed by the following formula:

[0047] M.N.D. F ·D F ,

[0048] Among them, M can represent the number of input channels of the feature, N can represent the number of output channels, and D F It can be the size of the feature map.

[0049] In some embodiments, the total computation amount of depthwise separable convolution may be:

[0050] D K ·D K ·M·D F ·D F +M·N·D F ·D F .

[0051] In some embodiments, the computational complexity of depthwise separable convolution compared to the computational complexity of standard convolution can be expressed by the following formula:

[0052]

[0053] It can be seen that the degree of reduction in computational effort is closely related to the number of output channels and the size of the convolution kernel.

[0054] In some embodiments, the convolution kernel used by the three-branch semantic segmentation network may be 3×3.

[0055] In some embodiments, when the number of output channels N is 256, the degree of reduction in the computational complexity of depthwise separable convolution compared to standard convolution can be expressed by the following formula:

[0056]

[0057] It can be seen that compared with the standard convolution, the depth of the three-branch semantic segmentation network can be reduced by 8 to 9 times. Figure 3 As shown, the three-branch semantic segmentation network can perform semantic segmentation on image data and obtain potential dynamic area masks.

[0058] In some embodiments, the loss function of the three-branch semantic segmentation network may include at least one of an additional semantic loss, a weighted binary cross entropy loss, a cross entropy loss, and a boundary-aware cross entropy loss.

[0059] In some embodiments, the loss function of the three-branch semantic segmentation network may be:

[0060] Loss=λ0l0+λ1l1+λ2l2+λ3l3,

[0061] Among them, l0 can represent additional semantic loss, l1 can represent weighted binary cross entropy loss, l2 can represent cross entropy loss, l3 can represent boundary-aware cross entropy loss, and λ0, λ1, λ2, and λ3 are preset coefficients of each loss term, respectively.

[0062] In some embodiments, the boundary-aware cross entropy loss may be expressed as follows:

[0063] l3=-∑ i,c {1:b i >t}(s i,c logs i,c ),

[0064] Where t can refer to a preset threshold, b i Can represent the output of the boundary head, s i,c It can represent the segmentation truth value of the i-th pixel of category c, s i,c It can represent the segmentation prediction value for the i-th pixel of category c.

[0065] In some embodiments, the preset coefficient λ0 of the additional semantic loss can be 0.4, the preset coefficient λ1 of the weighted binary cross entropy loss can be 20, the preset coefficient λ2 of the cross entropy loss can be 1, the boundary-aware cross entropy loss λ3 can be 1, and the preset threshold t can be 0.8.

[0066] In some embodiments, a coordinate attention mechanism can be introduced into the three-branch semantic segmentation network. Thus, the accuracy of semantic segmentation can be improved. Specifically, the coordinate attention mechanism can decompose channel attention into two one-dimensional feature encoding processes, aggregating features in the horizontal direction and the vertical direction respectively. In this case, the three-branch semantic segmentation network can capture long-distance feature dependencies while maintaining accurate position information, thereby further improving the effect of semantic segmentation.

[0067] In some embodiments, the three-branch semantic segmentation network can be trained using the Cityscapes dataset to obtain a semantic segmentation model based on the three-branch semantic segmentation network. The Cityscapes dataset is a widely used urban scene understanding dataset, mainly used for image segmentation and visual perception tasks. It contains high-resolution images from 50 different cities, focusing mainly on dynamic scenes in urban environments. The resolution of the dataset is 2048×1024, and it contains a total of 5,000 accurately labeled high-resolution images, including 2,975 training images, 500 verification images, and 1,525 test images, covering different weather conditions, times, and perspectives, which increases the challenge of the model's performance in real environments.

[0068] In some embodiments, extracting multiple feature points of the image data may include: in each frame of the image data, obtaining at least one feature point of each frame of the image using a feature extraction algorithm.

[0069] In some embodiments, the feature point may include at least one of a corner point, an edge intersection point, and a texture feature point in the image. The feature point may be a position with obvious changes in the image. Each frame of the image may correspond to multiple feature points, and the feature points may include static feature points and dynamic feature points. The dynamic area may be used to limit the screening range of the dynamic feature points.

[0070] In some embodiments, the feature extraction algorithm may be one of a Shi-Tomasi corner detection algorithm, a Harris corner detection algorithm, a SIFT (Scale Invariant Feature Transform) algorithm, or a SURF (Speeded Up Robust Features) algorithm.

[0071] In some embodiments, Figure 4 As shown, a motion consistency check is performed on multiple feature points in the potential dynamic area mask to screen out static feature points, which can include: using a feature matching algorithm to obtain multiple matching feature point pairs of any two consecutive frame images; based on the matching feature point pairs, using epipolar geometry constraints to estimate the basic matrix; using the basic matrix to calculate the epipolar line position of the corresponding matching feature point pair in the current frame, if the distance between the position of the feature point in the current frame and the epipolar line corresponding to the feature point is greater than or equal to a preset threshold, the feature point is output as a dynamic feature point, and if the distance between the position of the feature point in the current frame image and the epipolar line corresponding to the feature point is less than a preset threshold, the feature point is output as a static feature point.

[0072] In some embodiments, the matching feature point pairs may refer to corresponding feature points in two frames of images that represent the same physical position or have similar features in visual content. The matching feature point pairs may be used to estimate the geometric transformation between images, and further may be used for motion estimation.

[0073] In some embodiments, the lightweight semantic visual inertial odometry method for a drone may include: calculating feature descriptors of feature points, and obtaining matching feature point pairs according to the similarity of the feature descriptors.

[0074] In some embodiments, epipolar geometry constraints can be used to implement motion consistency checks. In this case, by analyzing the geometric relationship between the image pairs, it is ensured that the movement of the feature points is as expected, thereby improving the positioning accuracy and robustness in dynamic environments. Epipolar geometry can be used to describe the correspondence between two frames of images, where if the same camera module shoots the same object at different positions, and there is an overlapping area in the two frames of images taken, it can be considered that there is a correspondence between the two frames of images. The correspondence mainly involves the optical center of the camera, the observation of 3D points, and the coplanar relationship between 3D points.

[0075] In some embodiments, in the coordinate system of the first frame image, the spatial position of point P can be expressed as:

[0076] P = [X, Y, Z] T .

[0077] In some embodiments, an epipolar geometry constraint equation may be constructed based on epipolar geometry, which may be expressed by the following formula:

[0078] s1p1=KP

[0079] s2p2=K(RP+t)

[0080]

[0081] Wherein, P1 and P2 can be the projection points of point P on two imaging platforms, K can be the intrinsic parameter matrix of the camera module, R can be the rotation matrix, t can be the translation vector, and t ∧ R can be recorded as the essential matrix E (Essential Matrix), K -T t ∧ R -1 It can be recorded as the basic matrix F,t ∧ is the antisymmetric matrix of the translation vector t and can be used to represent the cross product operation.

[0082] In some embodiments, a plurality of matching feature point pairs may be used to construct a system of equations to solve an essential matrix and a fundamental matrix.

[0083] The essential matrix can be used to describe the relative motion of the two cameras, that is, the rotation matrix R and the displacement vector t. Therefore, the process of solving the essential matrix is ​​to restore the rotation and translation information between the camera modules through the matching feature point pairs in the image. The basic matrix F can be used to describe the geometric relationship between the matching feature point pairs between the two frames of images.

[0084] The fundamental matrix can be obtained from the intrinsic matrix through the camera module's intrinsic matrix K. Based on the fundamental matrix F and the camera module's intrinsic matrix K, the intrinsic matrix can also be restored, and then the rotation matrix R and displacement vector t between the cameras can be extracted from the essential matrix.

[0085] In summary, the present application can classify feature points based on the basic matrix, thereby being helpful in improving the accuracy of motion estimation in a dynamic environment.

[0086] In some embodiments, tracking the features of static feature points, obtaining feature point information of image data and pose estimation results of the camera module may include: screening the static feature points, obtaining feature point information based on the screened static feature points, and obtaining pose estimation results of the camera module based on the basic matrix of the screened static feature points. Dynamic feature points may be screened, dynamic feature points may be removed, and static feature points may be retained.

[0087] In some embodiments, screening of static feature points may include: using a sparse optical flow method to compare pixel changes between adjacent frame images to estimate the motion trajectory of the static feature points; judging whether the static feature points meet preset conditions based on the optical flow estimation error, reprojection error, and matching error, and screening out static feature points that meet the preset conditions.

[0088] In some embodiments, the optical flow estimation error may be expressed by the following formula:

[0089]

[0090] Among them, (u est ,v est ) is the displacement of feature points estimated by the optical flow method, (u act ,v act ) is the actual detected displacement. The optical flow estimation error can be measured by calculating the displacement difference of the feature points between two adjacent frames.

[0091] Assume that a three-dimensional point P = [X, Y, Z] is obtained by estimating the movement of the camera, and it is projected into the current image through the camera's intrinsic and extrinsic parameters to obtain the projected point p = (x, y). Then the reprojection error is the difference between the projected point and the actual detected feature point position.

[0092] In some embodiments, the reprojection error may be expressed by the following formula:

[0093]

[0094] in, is the estimated projection point, and p = [u, v] is the actual measured projection point. The reprojection error can represent the difference between the actual feature point position and the projection position estimated by the camera projection model.

[0095] In some embodiments, the matching error can be expressed by the following formula:

[0096]

[0097] Where (u1, v1) and (u2, v2) are the pixel coordinates of the matching feature points in the two images respectively. The matching error may refer to the difference between the position of the matching point found and the actual matching point when matching feature points in the images.

[0098] In some embodiments, if the optical flow estimation error is greater than a first predetermined threshold, or the matching error is greater than a second predetermined threshold, or the reprojection error is not within a preset range, it can be considered that the static feature point does not meet the preset condition.

[0099] In some embodiments, the pose estimation result of the camera module may include the position information and posture information of the camera module, that is, the rotation matrix and the translation vector.

[0100] In some embodiments, the feature point information may include location information, tracking status, and whether the feature point is a static feature point.

[0101] S2. Obtain pre-integration information of motion state data, optimize the state parameters of the camera module to obtain the optimal pose estimation based on the feature point information, pose estimation results and pre-integration information, and output the state pose information of the drone based on the optimal pose estimation.

[0102] In some embodiments, the motion state data may be IMU data, i.e., inertial measurement unit data. The motion state data may include acceleration information and angular velocity information of the drone during flight. The motion state data may be used to predict and correct the motion trajectory of the camera and determine the attitude information and position information of the drone. The sensor may include an accelerometer and a gyroscope.

[0103] In some embodiments, obtaining pre-integration information of motion state data may include: performing pre-integration calculation on the motion state data, and obtaining measurement values ​​of position, velocity, and posture increments between adjacent frame images as pre-integration information based on the calculation results.

[0104] In some embodiments, the state parameters may include posture information, velocity information, and IMU bias information, wherein the IMU bias information may include accelerometer bias and gyroscope bias.

[0105] In some embodiments, the position and orientation information may include the position and orientation of the camera module in each frame of the image. The velocity information may include the velocity vector of the camera module at each moment. The IMU bias information may be an accelerometer bias and an estimated value of the accelerometer bias. The optimal position estimation may be an optimal estimation of the position information, velocity information, and IMU bias information.

[0106] In some embodiments, optimizing the state parameters of the camera module to obtain an optimal pose estimate may include: optimizing the state parameters of the camera module using a graph optimization algorithm, adjusting the pose parameters of the camera module, and obtaining the optimal pose estimate after multiple iterations.

[0107] In some embodiments, the graph optimization algorithm may be a nonlinear optimization algorithm such as a Gauss-Newton algorithm and a Levenberg-Marquardt algorithm.

[0108] In some embodiments, the nonlinear minimization problem to be optimized can be formulated as:

[0109]

[0110] Among them, r i (X) is the residual function, and X is the parameter vector to be optimized.

[0111] In some embodiments, the formula of Gauss-Newton's nonlinear optimization algorithm can be expressed as:

[0112] X k+1 =X k -(J T J) -1 J T r,

[0113] Where J is the Jacobian matrix of the residual function and r is the residual vector.

[0114] The Levenberg-Marquardt algorithm is an improvement of the Gauss-Newton algorithm. It adds a damping factor λ to improve convergence, which can be expressed by the following formula:

[0115] X k+1 =X k -(J T J+λΙ) -1 J T r

[0116] Where λ is the damping factor used to control the size of the update, and Ι is the identity matrix. By adjusting λ, the Levenberg-Marquardt algorithm can adaptively switch between the Gauss-Newton algorithm and gradient descent to achieve better convergence.

[0117] In some embodiments, the state posture information may include the absolute position of the drone in three-dimensional space and the direction in three-dimensional space, where the absolute position can be represented by the coordinate points x, y, and z in the three-dimensional coordinate system, and the direction can be represented by a quaternion.

[0118] In some embodiments, the expression formula of quaternion can be:

[0119] q=q0+q1i+q2j+q3k,

[0120] Among them, q0 is the real part, referred to as the real part, i, j, k are the imaginary parts, referred to as the imaginary parts. And these three imaginary parts satisfy the following relationship:

[0121]

[0122] In particular, when , the quaternion is a unit quaternion.

[0123] In some embodiments, the state parameters of the camera module may be optimized in real time to obtain an optimal pose estimate. Specifically, whenever new image data and motion state data are received, the state parameters may be optimized in real time based on the new data to obtain the current optimal pose estimate.

[0124] In some embodiments, local optimization can be performed based on new data, that is, optimization and adjustment can be performed based on the epipolar geometric constraints between the current frame image and the previous frame image. In this way, it is possible to adapt to environmental changes in real time during long-term operation, maintain high accuracy and reduce error accumulation.

[0125] Simulation experiment results:

[0126] Figure 5 The comparison of positioning trajectories on the home1-3 sequence data set in the OpenLORIS-Scene dynamic scene data set is shown. The compared algorithms include the embodiments of this application and other mainstream visual inertial SLAM systems. Among them, the OpenLORIS-Scene dynamic scene data set covers different real-life scenes, including various indoor and outdoor environments such as homes, shopping malls, and offices. During the acquisition process, the changes in different time periods, lighting conditions, and viewing angles are taken into account to ensure the diversity and richness of the data, which is very suitable for verifying the accuracy and robustness of the robot system in dynamic scenes.

[0127] like Figure 5 As shown, the positioning trajectory of the embodiment of the present application is closer to the true value, and there is almost no obvious drift phenomenon, while other visual inertial SLAM systems, such as ORB-SLAM3 and DynaVINS, produce large errors visible to the naked eye. Specifically, ORB-SLAM3 will encounter tracking failures in scenes involving violent rotations and white walls, while DynaVINS will experience significant positioning drift. In contrast, the embodiment of the present application can stably estimate the drone attitude while maintaining global consistency.

[0128] Therefore, in the present application, by utilizing the fusion of IMU data, as well as a lightweight three-branch semantic segmentation network and motion consistency check, the influence of dynamic targets can be effectively eliminated. Even in an environment with severe rotation and rich dynamic features, the embodiments of the present application can still achieve accurate and stable posture estimation, and exhibit high positioning robustness in dynamic scenes.

[0129] Figure 6 The figure shows the comparison of the positioning trajectory of the embodiment of the present application and other visual inertial SLAM systems in real dynamic scene data sets. In the dynamic scene in the real world, such as Figure 6 As shown in the figure, both DynaVINS and VINS-Fusion produced obvious trajectory errors, with large deviations from the true value. However, the embodiment of the present application provides a more stable and reliable attitude estimation with minimal drift. This result shows that other algorithms have serious positioning errors and poor robustness in actual application scenarios, and their performance is lower than that of the embodiment of the present application.

[0130]

[0131] Table 1

[0132]

[0133] Table 2

[0134] Table 1 shows the test indicators of the three-branch semantic segmentation network of the embodiment of the present application and other mainstream real-time semantic segmentation algorithms on the Cityscapes dataset, including segmentation accuracy, reasoning speed, computational complexity (expressed in GFLOPs) and model parameter comparison. From the experimental results in Table 1, it can be seen that the overall performance of the three-branch semantic segmentation network of the embodiment of the present application is significantly better than other mainstream real-time semantic segmentation networks. On the Cityscapes dataset, the three-branch semantic segmentation network achieves the highest reasoning accuracy and the fastest reasoning speed, while ensuring that the model parameters and computational complexity are kept at the lowest level, thereby achieving the best balance between semantic segmentation accuracy and speed.

[0135] Table 2 shows a comparison table of algorithm module time consumption of the embodiment of the present application and other mainstream visual-inertial SLAM systems under real dynamic scene data sets. It can be seen from the simulation experiment results of Table 2 that compared with other visual-inertial SLAM systems, the embodiment of the present application has achieved the fastest running speed in each part, and can also successfully run on edge devices with limited computing resources, meeting the real-time requirements of the algorithm. Therefore, the embodiment of the present application not only has higher positioning accuracy and robustness in a dynamic environment, but also has the fastest algorithm running speed. And the embodiment of the present application can realize real-time and reliable attitude estimation of drones on platforms with limited computing resources, fully demonstrating the powerful application potential of the embodiment of the present application in real dynamic scenes, and has significant engineering application value.

[0136] In this application, a three-branch semantic segmentation network is used to perform semantic segmentation on image data, and a motion consistency check is performed on multiple feature points in the potential dynamic area mask, and static feature points are screened out for feature tracking. Finally, the camera module pose is optimized in combination with the feature point information and pose estimation results to obtain the state pose information of the drone. The three-branch semantic segmentation network can greatly reduce the computational complexity of the algorithm model while ensuring the accuracy of semantic segmentation, and significantly improve the reasoning speed. At the same time, consistency checks are performed on feature points, and static feature points are screened and retained, which can significantly improve the positioning accuracy and robustness of drones in dynamic scenes. The embodiments of the present application can not only meet the high accuracy and real-time performance of positioning in dynamic environments, but also significantly reduce the requirements for computing resources, which is conducive to its deployment on edge devices with limited computing resources. Compared with the prior art, the present application effectively improves the positioning accuracy and robustness of drones in dynamic scenes, and has great engineering application value.

[0137] Those skilled in the art can understand that all or part of the features / steps of the above-mentioned method embodiments can be implemented by methods, data processing systems or computer programs, and these features can be implemented without hardware, entirely by software, or by a combination of hardware and software. The aforementioned computer program can be stored in one or more computer-readable storage media, and the storage medium stores the computer program. When the computer program is executed (such as by a processor), the steps of the above-mentioned drone lightweight semantic visual inertial odometer method embodiment are executed.

[0138] The aforementioned storage media that can store program codes include: static hard disks, solid-state hard disks, random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), optical storage devices, magnetic storage devices, flash memory, magnetic disks or optical disks and / or combinations of the above devices, that is, they can be implemented by any type of volatile or non-volatile storage device or a combination thereof.

[0139] like Figure 7 As shown, the present application also provides an embodiment of a processing device 1, including one or more processors 11 and a memory 10; wherein the memory 10 is used to store one or more computer programs, and the one or more processors 11 are used to execute one or more computer programs stored in the memory 10, so that the processor 11 executes the features / steps of the above-mentioned drone lightweight semantic visual inertial odometer method embodiment.

[0140] The above is only the preferred embodiment of the present application. Those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application belong to the protection scope of the present application.

Claims

1. A lightweight semantic visual inertial odometry method for unmanned aerial vehicles, characterized in that: The drone includes a camera module for collecting image data in real time, and a sensor for collecting motion state data. The method includes: Perform semantic segmentation on the image data using a three-branch semantic segmentation network to obtain a potential dynamic area mask; extract multiple feature points of the image data, perform motion consistency check on the multiple feature points in the potential dynamic area mask, screen out static feature points, perform feature tracking on the static feature points, and obtain feature point information of the image data and a pose estimation result of the camera module, wherein the three-branch semantic segmentation network is constructed based on deep separable convolution and coordinate attention mechanism; Obtain pre-integration information of the motion state data, optimize the state parameters of the camera module to obtain an optimal pose estimation based on the feature point information, the pose estimation result and the pre-integration information, and output the state pose information of the drone based on the optimal pose estimation.

2. The lightweight semantic visual inertial odometer method for unmanned aerial vehicles according to claim 1 is characterized in that: The extracting of the plurality of feature points of the image data comprises: in each frame of the image data, obtaining at least one feature point of each frame of the image by using a feature extraction algorithm.

3. The lightweight semantic visual inertial odometer method for unmanned aerial vehicles according to claim 1 is characterized in that: The motion consistency check is performed on multiple feature points in the potential dynamic area mask to screen out static feature points, including: using a feature matching algorithm to obtain multiple matching feature point pairs of any two consecutive frame images; based on the matching feature point pairs, estimating a basic matrix using epipolar geometry constraints; using the basic matrix to calculate the epipolar line position of the corresponding matching feature point pair in the current frame image, if the distance between the position of the feature point in the current frame image and the epipolar line corresponding to the feature point is less than the preset threshold, then the feature point is output as the static feature point.

4. The lightweight semantic visual inertial odometer method for unmanned aerial vehicles according to claim 1, characterized in that: The feature tracking of the static feature points to obtain the feature point information of the image data and the pose estimation result of the camera module includes: screening the static feature points, obtaining the feature point information based on the screened static feature points, and obtaining the pose estimation result of the camera module based on the basic matrix of the screened static feature points.

5. The lightweight semantic visual inertial odometer method for unmanned aerial vehicles according to claim 4 is characterized in that: The screening of the static feature points includes: using a sparse optical flow method to compare pixel changes between adjacent frame images to estimate the motion trajectory of the static feature points; judging whether the static feature points meet preset conditions based on optical flow estimation errors, reprojection errors, and matching errors, and screening out static feature points that meet the preset conditions.

6. The lightweight semantic visual inertial odometer method for unmanned aerial vehicles according to claim 1, characterized in that: The obtaining of pre-integration information of the motion state data comprises: performing pre-integration calculation on the motion state data, and obtaining measurement values ​​of position, speed and posture increments between adjacent frame images as the pre-integration information according to the calculation results.

7. The lightweight semantic visual inertial odometer method for unmanned aerial vehicles according to claim 1, characterized in that: The optimizing the state parameters of the camera module to obtain the optimal posture estimation includes: optimizing the state parameters of the camera module using a graph optimization algorithm, adjusting the posture information of the camera module, and obtaining the optimal posture estimation after multiple iterations; the state parameters include the posture information, speed information and IMU deviation information, wherein the IMU deviation information includes accelerometer deviation and gyroscope deviation.

8. The lightweight semantic visual inertial odometer method for unmanned aerial vehicles according to claim 1, characterized in that: The loss function of the three-branch semantic segmentation network includes at least one of an additional semantic loss, a weighted binary cross entropy loss, a cross entropy loss, and a boundary-aware cross entropy loss.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed, implements the lightweight semantic visual-inertial odometer method for a drone as described in any one of claims 1 to 8.

10. A processing device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, and one or more of the processors for executing the one or more computer programs stored in the memory, so that one or more of the processors execute the lightweight semantic visual-inertial odometer method for a drone as described in any one of claims 1-8.

Images