Unmanned aerial vehicle lightweight semantic visual inertial odometry method, medium and processing device

Abstract

The application discloses a lightweight semantic visual inertial odometer method for unmanned aerial vehicles, a medium and a processing device. The method comprises the following steps: performing semantic segmentation on image data by using a three-branch semantic segmentation network to obtain a potential dynamic region mask; extracting a plurality of feature points of the image data, performing motion consistency checking on the plurality of feature points in the potential dynamic region mask, screening out static feature points, performing feature tracking on the static feature points, obtaining feature point information of the image data and a pose estimation result of a camera module; obtaining pre-integration information of motion state data, optimizing state parameters of the camera module according to the feature point information, the pose estimation result and the pre-integration information to obtain an optimal pose estimation, and outputting state pose information of the unmanned aerial vehicle according to the optimal pose estimation. The application can meet the high accuracy and real-time performance of positioning in a dynamic environment, and the requirement for computing resources is significantly reduced, which is beneficial to deployment to edge devices with limited computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of simultaneous localization and mapping, and particularly relates to a lightweight semantic visual-inertial odometry method for unmanned aerial vehicles, a medium and a processing device. BACKGROUND

[0002] Nowadays, visual-inertial odometry (VIO) plays a crucial role in unmanned aerial vehicle positioning. By combining visual information and inertial measurement unit (IMU) data, a VIO system can provide high-precision real-time position and attitude estimation. However, this system assumes that the unmanned aerial vehicle is in a completely static scene, but this often does not match the reality, because dynamic scenes are very common, and it is impossible to guarantee the complete staticity of the environment. In dynamic environments, moving objects such as moving pedestrians and vehicles can introduce additional image feature points, which may be confused with static feature points in the background environment, leading to feature point matching errors. This confusion introduces incorrect visual constraints in the attitude estimation process of the unmanned aerial vehicle, thereby negatively affecting the positioning accuracy and stability of the VIO system, and further causing serious deviations in the positioning results.

[0003] In the prior art, by introducing semantic information, the system can achieve more accurate positioning in dynamic scenes. However, existing semantic visual-inertial odometry systems often face the problems of insufficient lightweight and insufficient running speed, especially the relatively complex semantic algorithm module, which has a large computational overhead, making it difficult for these systems to be effectively deployed on unmanned aerial vehicle platforms with limited computing resources, thereby limiting their application in actual engineering.

[0004] Although there are some classic visual-inertial SLAM systems such as DynaVINS, RDS-SLAM, etc. that have been applied to the positioning research of robots in dynamic scenes, there are still problems to be improved. The positioning accuracy of these systems in dynamic scenes needs to be further optimized and improved. At the same time, visual-inertial odometry systems combined with semantic information often have high demands on computing resources, and the algorithm model design of the semantic module is complex and redundant, making it difficult to ensure the real-time performance and stability of the algorithm. This makes it difficult for them to be effectively deployed on edge devices with limited computing resources, limiting their application in real dynamic scenes, and thus affecting their value in engineering applications. SUMMARY

[0005] The purpose of the present application is to provide a lightweight semantic visual inertial odometer method for unmanned aerial vehicles, a medium and a processing device, to solve the technical problems of high computational resource requirements and complex and redundant models of existing visual inertial odometer systems in the prior art. The preferred technical solutions in the many technical solutions provided by the present application can produce many technical effects, which are described in detail below.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a lightweight semantic visual inertial odometer method for unmanned aerial vehicles, the unmanned aerial vehicle comprising a camera module for real-time image data acquisition and a sensor for motion state data acquisition, the method comprising: performing semantic segmentation on the image data using a three-branch semantic segmentation network to obtain a potential dynamic region mask; extracting a plurality of feature points of the image data, performing motion consistency checking on a plurality of feature points in the potential dynamic region mask, and screening out static feature points; performing feature tracking on the static feature points to obtain feature point information of the image data and pose estimation results of the camera module, wherein the three-branch semantic segmentation network is constructed based on depth separable convolution and coordinate attention mechanism; obtaining pre-integration information of the motion state data, and optimizing state parameters of the camera module according to the feature point information, the pose estimation results and the pre-integration information to obtain optimal pose estimation; and outputting state pose information of the unmanned aerial vehicle according to the optimal pose estimation.

[0008] In some embodiments, the extracting a plurality of feature points of the image data comprises: in each frame of image in the image data, using a feature extraction algorithm to obtain at least one feature point of each frame of image.

[0009] In some embodiments, the motion consistency checking on a plurality of feature points in the potential dynamic region mask and screening out static feature points comprises: using a feature matching algorithm to obtain a plurality of matching feature point pairs of any two consecutive frames of image; based on the matching feature point pairs, using epipolar geometry constraint to estimate a fundamental matrix; using the fundamental matrix to calculate the position of the corresponding matching feature point pair in the current frame of image, if the distance between the position of the feature point in the current frame of image and the corresponding epipolar line of the feature point is greater than or equal to a preset threshold, the feature point is output as the dynamic feature point, and if the distance between the position of the feature point in the current frame of image and the corresponding epipolar line of the feature point is less than the preset threshold, the feature point is output as the static feature point.

[0010] In some embodiments, the feature tracking of the static feature points, the obtaining of the feature point information of the image data, and the obtaining of the pose estimation result of the camera module include: screening the static feature points, obtaining the feature point information based on the screened static feature points, and obtaining the pose estimation result of the camera module based on the fundamental matrix of the screened static feature points.

[0011] In some embodiments, the screening of the static feature points includes: comparing pixel changes between adjacent frame images by using a sparse optical flow method to estimate motion trajectories of the static feature points; and determining whether the static feature points meet a preset condition according to an optical flow estimation error, a re-projection error, and a matching error, and screening static feature points meeting the preset condition.

[0012] In some embodiments, the obtaining of the pre-integration information of the motion state data includes: performing pre-integration calculation on the motion state data, and obtaining measurement values of position, velocity, and attitude increments between adjacent frame images as the pre-integration information according to a calculation result.

[0013] In some embodiments, the optimization of the state parameters of the camera module to obtain the optimal pose estimation includes: optimizing the state parameters of the camera module by using a graph optimization algorithm, adjusting the pose information of the camera module, and obtaining the optimal pose estimation after multiple iterations; and the state parameters include the pose information, velocity information, and IMU bias information, wherein the IMU bias information includes accelerometer bias and gyroscope bias.

[0014] In some embodiments, the loss function of the three-branch semantic segmentation network includes at least one of an additional semantic loss, a weighted binary cross-entropy loss, a cross-entropy loss, and a boundary-aware cross-entropy loss.

[0015] In a second aspect, the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed to implement the unmanned aerial vehicle lightweight semantic visual inertial odometry method as described above.

[0016] In a third aspect, the present application provides a processing device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors are configured to execute the one or more computer programs stored in the memory, so that the one or more processors perform the unmanned aerial vehicle lightweight semantic visual inertial odometry method as described above.

[0017] The application has the following advantages or beneficial effects: in the application, the image data is subjected to semantic segmentation by using a three-branch semantic segmentation network, motion consistency checking is performed on a plurality of feature points in a potential dynamic region mask, static feature points are screened out for feature tracking, and finally the camera module pose is optimized in combination with feature point information and pose estimation results to obtain the state pose information of the unmanned aerial vehicle. The three-branch semantic segmentation network can greatly reduce the calculation complexity of the algorithm model while ensuring the semantic segmentation accuracy, and significantly improve the inference speed. At the same time, the consistency checking is performed on the feature points, the static feature points are screened and retained, which can significantly improve the positioning accuracy and robustness of the unmanned aerial vehicle in a dynamic scene. The embodiments of the application not only can meet the high accuracy and real-time performance of positioning in a dynamic environment, but also significantly reduce the requirement for computing resources, which is conducive to deploying it to edge devices with limited computing resources. Compared with the prior art, the application effectively improves the positioning accuracy and robustness of the unmanned aerial vehicle in a dynamic scene, and has great engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0019] Figure 1 is a flowchart of the unmanned aerial vehicle lightweight semantic visual inertial odometer method of the embodiments of the application;

[0020] Figure 2 is a structural diagram of the three-branch semantic segmentation network of the embodiments of the application;

[0021] Figure 3 is a result diagram after semantic segmentation by the three-branch semantic segmentation network of the embodiments of the application;

[0022] Figure 4 is a schematic diagram of motion consistency checking of the embodiments of the application;

[0023] Figure 5 is a positioning trajectory comparison diagram of the embodiments of the application and other visual inertial SLAM systems;

[0024] Figure 6 is another positioning trajectory comparison diagram of the embodiments of the application and other visual inertial SLAM systems;

[0025] Figure 7 is a structural block diagram of the processing device of the embodiments of the application.

[0026] In the drawings: 1, processing device; 10, memory; 11, processor. DETAILED DESCRIPTION

[0027] In order to make the purposes, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will be described with reference to the corresponding drawings, which form part of the exemplary embodiments, and in which various exemplary embodiments that can be used to implement the present application are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. It should be understood that they are only examples of processes, methods and apparatuses, etc. consistent with some aspects of the present disclosure as detailed in the appended claims, and other embodiments can be used, or structural and functional modifications can be made to the embodiments listed herein, without departing from the scope and spirit of the present application.

[0028] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the elements referred to must have a particular orientation, be constructed and operated in a particular orientation. The terms "first", "second" and the like are only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "a plurality of" means two or more. The terms "connected", "connected" should be broadly understood, for example, it can be fixed connection, detachable connection, integral connection, mechanical connection, electrical connection, communication connection, direct connection, indirect connection through intermediate medium, internal communication of two elements or interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0029] In order to illustrate the technical solutions described in the present application, the following will be described by specific embodiments, only showing the parts related to the embodiments of the present application.

[0030] As Figure 1 As shown in the present application, a lightweight semantic visual inertial odometer method for unmanned aerial vehicle is provided, comprising the following steps (steps S1 to S2):

[0031] S1, performing semantic segmentation on the image data by using a three-branch semantic segmentation network to obtain a potential dynamic region mask; extracting a plurality of feature points of the image data, performing motion consistency checking on the plurality of feature points in the potential dynamic region mask, screening out static feature points, and performing feature tracking on the static feature points to obtain feature point information of the image data and a pose estimation result of the camera module.

[0032] In some embodiments, the UAV can include a camera module for collecting image data in real time, and a plurality of sensors for collecting motion state data.

[0033] In some embodiments, the camera module can be a photographing device, such as an RGB camera or a camera, carried by the UAV for taking images. The image data can refer to visual images of the environment around the UAV taken by the camera module, which can be used to identify and track feature points in the environment. The image data can include a plurality of continuous frames of images taken by the camera module as the UAV moves, that is, the image data can be a plurality of frames of images collected by the camera module in real time.

[0034] In some embodiments, the potential dynamic region mask can be used to indicate a dynamic region in the image environment. The dynamic region can represent a part that can include moving objects or changes in the background. For example, in an application scenario with pedestrians in the environment, the dynamic region can refer to the region of the pedestrians. After identifying and segmenting the image data of the environment, the region of the pedestrians can be output as the potential dynamic region mask.

[0035] In some embodiments, a dynamic mask graph can be obtained according to the potential dynamic region mask. The dynamic mask graph is a binary image, and each mask value in the binary image can represent whether the corresponding pixel point belongs to the dynamic region.

[0036] In some embodiments, the lightweight semantic visual-inertial odometry method for UAVs can further include: obtaining pixel coordinates of the feature points in the image, and obtaining mask values in the dynamic mask graph corresponding to the potential dynamic region mask according to the pixel coordinates. If the mask value corresponding to the feature point is 1, it can be indicated that the feature point is located in the dynamic region, and if the mask value corresponding to the feature point is 0, it can be indicated that the feature point is located in the static region. Thus, it can be determined whether the feature point is in the potential dynamic region according to the potential dynamic region mask.

[0037] In some embodiments, as Figure 2As shown, a three-branch semantic segmentation network (LPIDNet, Lightweight Proportional-Integral-Derivative Network) can be constructed based on depthwise separable convolutions and coordinate attention mechanisms. Compared with traditional standard convolutions, depthwise separable convolutions can significantly reduce the computational complexity of the model and improve the inference speed of the segmentation algorithm.

[0038] The computational amount of a standard convolution can be represented by the following formula:

[0039] D K ·D K ·M·N·D F ·D F ,

[0040] where M can represent the input channel number of the feature, N can represent the output channel number, D K may be the size of the convolution kernel, and D F may be the size of the feature map. Generally, a feature map can refer to image data processed by a convolution layer and / or a pooling layer, and the size of the feature map can include the width and length of the feature map. The width of the feature map can refer to the number of pixels of the feature map in the horizontal direction, and the height of the feature map can refer to the number of pixels of the feature map in the vertical direction.

[0041] Generally, a standard convolution can complete channel-wise convolution and spatial-wise convolution within one step.

[0042] The depthwise separable convolution of the embodiments of the present application can be completed in two steps, that is, first completing the depth-wise convolution of the first step, and then completing the point-wise convolution of the second step.

[0043] In some embodiments, the depth-wise convolution can be represented by the following formula:

[0044] D K ·D K ·M·D F ·D F ,

[0045] where M can represent the input channel number of the feature, D K may be the size of the convolution kernel, and D F may be the size of the feature map.

[0046] In some embodiments, the point-wise convolution can be represented by the following formula:

[0047] M·N·D F ·D F ,

[0048] where M can represent the number of input channels of a feature, N can represent the number of output channels, D F may be the size of a feature map.

[0049] In some embodiments, the total amount of computation of the depthwise separable convolution can be:

[0050] D K ·D K ·M·D F ·D F +M·N·D F ·D F .

[0051] In some embodiments, the proportion of the amount of computation of the depthwise separable convolution compared to the amount of computation of the standard convolution can be represented by the following formula:

[0052]

[0053] As can be seen, the degree of reduction in the amount of computation is closely related to the number of output channels and the size of the convolution kernel.

[0054] In some embodiments, the convolution kernel adopted by the three-branch semantic segmentation network can be 3x3.

[0055] In some embodiments, when the number of output channels N is 256, the degree of reduction in the amount of computation of the depthwise separable convolution compared to the standard convolution can be represented by the following formula:

[0056]

[0057] As can be seen, the depthwise separable convolution of the three-branch semantic segmentation network can reduce by 8 to 9 times compared to the standard convolution. As Figure 3 shown, the three-branch semantic segmentation network can perform semantic segmentation on image data to obtain a potential dynamic region mask.

[0058] In some embodiments, the loss function of the three-branch semantic segmentation network can include at least one of an additional semantic loss, a weighted binary cross-entropy loss, a cross-entropy loss, and a boundary-aware cross-entropy loss.

[0059] In some embodiments, the loss function of the three-branch semantic segmentation network can be:

[0060] Loss = λ0l0+ λ1l1+ λ2l2+ λ3l3,

[0061] wherein, l0may represent an extra semantic loss, l1may represent a weighted binary cross-entropy loss, l2may represent a cross-entropy loss, l3may represent a boundary-aware cross-entropy loss, and λ0, λ1, λ2, λ3may be preset coefficients of the respective loss terms.

[0062] In some embodiments, the boundary-aware cross-entropy loss can be represented by the following formula:

[0063] l3 = -∑ i,c {1:b i >t}(s i,c logs i,c ),

[0064] wherein, t can be a preset threshold, b i may represent an output of a boundary head, s i,c may represent a segmentation ground truth of the i-th pixel for the class c, and s i,c may represent a segmentation prediction value of the i-th pixel for the class c.

[0065] In some embodiments, the preset coefficient λ0of the extra semantic loss can be 0.4, the preset coefficient λ1of the weighted binary cross-entropy loss can be 20, the preset coefficient λ2of the cross-entropy loss can be 1, the boundary-aware cross-entropy loss λ3may be 1, and the preset threshold t can be 0.8.

[0066] In some embodiments, a coordinate attention mechanism can be introduced in the three-branch semantic segmentation network. In this way, the accuracy of semantic segmentation can be improved. Specifically, the coordinate attention mechanism can decompose the channel attention into two one-dimensional feature encoding processes, respectively aggregating features in the horizontal direction and the vertical direction. In this case, the three-branch semantic segmentation network can capture long-distance feature dependencies while maintaining accurate location information, thereby further improving the effect of semantic segmentation.

[0067] In some embodiments, the three-branch semantic segmentation network can be trained using the Cityscapes dataset to obtain a semantic segmentation model based on the three-branch semantic segmentation network. The Cityscapes dataset is a widely used urban scene understanding dataset, mainly used for image segmentation and visual perception tasks, which contains high-resolution images from 50 different cities, mainly focusing on dynamic scenes in urban environments. The resolution of this dataset is 2048x1024, and it contains a total of 5000 accurately labeled high-resolution images, including 2975 training images, 500 validation images and 1525 test images, covering different weather conditions, time and perspective, increasing the challenge of the model's performance in real environments.

[0068] In some embodiments, extracting multiple feature points from image data may include: obtaining at least one feature point in each frame of the image data using a feature extraction algorithm.

[0069] In some embodiments, feature points may include at least one of corner points, edge intersections, and texture feature points in the image. Feature points can be locations in the image that exhibit significant changes. Each frame of the image may correspond to multiple feature points, which may include static feature points and dynamic feature points. Dynamic regions can be used to define the filtering range of dynamic feature points.

[0070] In some embodiments, the feature extraction algorithm may be one of the following: Shi-Tomasi corner detection algorithm, Harris corner detection algorithm, SIFT (Scale Invariant Feature Transform) algorithm, or SURF (Speed ​​Robust Feature) algorithm.

[0071] In some embodiments, such as Figure 4 As shown, performing motion consistency checks on multiple feature points within a potential dynamic region mask and filtering out static feature points can include: using a feature matching algorithm to obtain multiple matching feature point pairs between any two consecutive frames; estimating the fundamental matrix based on the matching feature point pairs using epipolar geometric constraints; calculating the epipolar position of the corresponding matching feature point pair in the current frame using the fundamental matrix; if the distance between the position of the feature point in the current frame and the corresponding epipolar line is greater than or equal to a preset threshold, the feature point is output as a dynamic feature point; if the distance between the position of the feature point in the current frame and the corresponding epipolar line is less than the preset threshold, the feature point is output as a static feature point.

[0072] In some embodiments, a matching feature point pair can refer to corresponding feature points in two frames that visually represent the same physical location or have similar features. Matching feature point pairs can be used to estimate geometric transformations between images, and further, for motion estimation.

[0073] In some embodiments, the lightweight semantic visual inertial odometry method for UAVs may include: calculating feature descriptors of feature points and obtaining matching feature point pairs based on the similarity of the feature descriptors.

[0074] In some embodiments, the motion consistency check can be implemented by employing epipolar geometry constraints. In this case, by analyzing the geometric relationship between image pairs, the motion of feature points is ensured to be consistent with the expectation, thereby improving the positioning accuracy and robustness in dynamic environments. Epipolar geometry can be used to describe the correspondence between two frames of images, where if the same camera module takes the same object at different positions, there is an overlapping area in the two frames of images taken, then it can be considered that there is a correspondence between the two frames of images. The correspondence mainly involves the optical center of the camera, the observation of 3D points and the coplanar relationship between 3D points.

[0075] In some embodiments, the spatial position of point P in the coordinate system of the first frame of image can be expressed as:

[0076] P = [X, Y, Z] T .

[0077] In some embodiments, based on the epipolar geometry, an epipolar geometry constraint equation can be constructed, which can be expressed by the following formula:

[0078] s1p1 = KP

[0079] s2p2 = K(RP + t)

[0080]

[0081] where P1 and P2 can be the projection points of point P on two imaging platforms, K can be the intrinsic matrix of the camera module, R can be the rotation matrix, t can be the translation vector, and t ∧ R can be denoted as the essential matrix E (Essential Matrix), K -T t ∧ RK -1 can be denoted as the fundamental matrix F, t ∧ is the skew-symmetric matrix of the translation vector t, which can be used to represent the cross product operation.

[0082] In some embodiments, a system of equations can be constructed by using a plurality of matched feature point pairs to solve the essential matrix and the fundamental matrix.

[0083] The essential matrix can be used to describe the relative motion of two cameras, i.e. the rotation matrix R and the displacement vector t. Therefore, the process of solving the essential matrix is to recover the rotation and translation information between the camera modules through the matched feature point pairs in the images. The fundamental matrix F can be used to describe the geometric relationship between the matched feature point pairs between two frames of images.

[0084] The essential matrix can be obtained from the camera intrinsic matrix K of the camera module. Based on the essential matrix F and the camera intrinsic matrix K of the camera module, the essential matrix can also be recovered, and then the rotation matrix R and the displacement vector t between the cameras can be extracted from the essential matrix.

[0085] In summary, the application can classify feature points based on the essential matrix, thereby facilitating improvement of the accuracy of motion estimation in a dynamic environment.

[0086] In some embodiments, the feature tracking of the static feature points, the acquisition of the feature point information of the image data, and the acquisition of the pose estimation result of the camera module can include: screening the static feature points, acquiring the feature point information based on the screened static feature points, and acquiring the pose estimation result of the camera module based on the essential matrix of the screened static feature points. The dynamic feature points can be screened, the dynamic feature points can be removed, and the static feature points can be retained.

[0087] In some embodiments, the screening of the static feature points can include: comparing the pixel changes between adjacent frame images by using a sparse optical flow method to estimate the motion trajectory of the static feature points; and determining whether the static feature points meet a preset condition according to an optical flow estimation error, a re-projection error, and a matching error, and screening the static feature points that meet the preset condition.

[0088] In some embodiments, the optical flow estimation error can be represented by the following formula:

[0089]

[0090] where (u est ,v est ) is the displacement of the feature point estimated by the optical flow method, and (u act ,v act ) is the actual detected displacement. The optical flow estimation error can be measured by calculating the displacement difference of the feature points between two adjacent frame images.

[0091] Suppose that a three-dimensional point P = [X, Y, Z] is obtained by estimating the motion of the camera, and is projected into the current image by the camera intrinsic and extrinsic parameters, to obtain a projection point p = (x, y), then the re-projection error is the difference between the projection point and the actual detected feature point position.

[0092] In some embodiments, the re-projection error can be represented by the following formula:

[0093]

[0094] where, For the estimated projection point, p = [u, v] is the actually measured projection point. The re-projection error can represent the difference between the actual feature point position and the estimated projection position through the camera projection model.

[0095] In some embodiments, the matching error can be represented by the following formula:

[0096]

[0097] where (u1, v1) and (u2, v2) are the pixel coordinates of the matching feature points in the two images, respectively. The matching error can refer to the difference between the found matching point position and the actual matching point when matching the feature points in the image.

[0098] In some embodiments, if the optical flow estimation error is greater than a first predetermined threshold, or the matching error is greater than a second predetermined threshold, or the re-projection error is not within a preset range, it can be considered that the static feature point does not meet the preset condition.

[0099] In some embodiments, the pose estimation result of the camera module can include position information and attitude information of the camera module, that is, a rotation matrix and a translation vector.

[0100] In some embodiments, the feature point information can include position information, tracking state, and whether it is a static feature point of the feature point.

[0101] S2, obtaining pre-integration information of motion state data, optimizing the state parameters of the camera module according to the feature point information, the pose estimation result and the pre-integration information to obtain an optimal pose estimation, and outputting the state pose information of the unmanned aerial vehicle according to the optimal pose estimation.

[0102] In some embodiments, the motion state data can be IMU data, that is, inertial measurement unit data. The motion state data can include acceleration information and angular velocity information of the unmanned aerial vehicle during flight. The motion state data can be used to predict and correct the motion trajectory of the camera and determine the attitude information and position information of the unmanned aerial vehicle. The sensor can include an accelerometer and a gyroscope.

[0103] In some embodiments, obtaining pre-integration information of motion state data can include: performing pre-integration calculation on the motion state data, and obtaining the measurement values of the position, velocity and attitude increments between adjacent frame images as the pre-integration information according to the calculation results.

[0104] In some embodiments, the state parameters can include pose information, velocity information and IMU bias information, wherein the IMU bias information can include accelerometer bias and gyroscope bias.

[0105] In some embodiments, the position and orientation information can include a position and an orientation of the camera module in each frame of image. The velocity information can include a velocity vector of the camera module at each time instant. The IMU bias information can include an accelerometer bias and an estimation of the accelerometer bias. The optimal pose estimate can be an optimal estimate of the position and orientation information, the velocity information, and the IMU bias information.

[0106] In some embodiments, optimizing the state parameters of the camera module to obtain the optimal pose estimate can include optimizing the state parameters of the camera module using a graph optimization algorithm, adjusting the pose parameters of the camera module, and obtaining the optimal pose estimate after a plurality of iterations.

[0107] In some embodiments, the graph optimization algorithm can be a non-linear optimization algorithm such as a Gauss-Newton algorithm and a Levenberg-Marquardt algorithm.

[0108] In some embodiments, a non-linear minimization problem to be optimized can be expressed in a formula as follows:

[0109]

[0110] where r i (X) is a residual function, and X is a parameter vector to be optimized.

[0111] In some embodiments, the formula of the Gauss-Newton non-linear optimization algorithm can be expressed as follows:

[0112] X k+1 = X k - (J T J) -1 J T r,

[0113] where J is a Jacobian matrix of the residual function, and r is a residual vector.

[0114] The Levenberg-Marquardt algorithm is an improvement of the Gauss-Newton algorithm, which adds a damping factor λ to improve convergence, and can be expressed in a formula as follows:

[0115] X k+1 = X k - (J T J + λI) -1 J T r

[0116] where λ is a damping factor used to control the size of the update, I is an identity matrix. By adjusting λ, the Levenberg-Marquardt algorithm can adaptively switch between the Gauss-Newton algorithm and gradient descent to achieve better convergence.

[0117] In some embodiments, the state pose information can include an absolute position of the UAV in a three-dimensional space and a direction in the three-dimensional space, where the absolute position can be represented by coordinate points x, y, z in a three-dimensional coordinate system, and the direction can be represented by a quaternion.

[0118] In some embodiments, the expression formula of the quaternion can be:

[0119] q = q0 + q1i + q2j + q3k,

[0120] where q0 is a real part, referred to as a real component, and i, j, k are imaginary parts, referred to as imaginary components. And the three imaginary components satisfy the following relationship:

[0121]

[0122] In particular, when , the quaternion is a unit quaternion.

[0123] In some embodiments, the state parameters of the camera module can be optimized in real time to obtain the optimal pose estimation. Specifically, each time new image data and motion state data are received, the state parameters can be optimized in real time based on the new data to obtain the current optimal pose estimation.

[0124] In some embodiments, local optimization can be performed according to the new data, that is, optimization and adjustment are performed according to the epipolar geometric constraint between the current frame image and the previous frame image. In this way, the system can adapt to changes in the environment in real time in long-term operation, maintain high accuracy, and reduce the accumulation of errors.

[0125] Simulation experiment results:

[0126] Figure 5 The positioning trajectory comparison on the home1-3 sequence data set in the OpenLORIS-Scene dynamic scene data set is shown, and the comparison algorithms include the embodiments of the present application and other mainstream visual-inertial SLAM systems. The OpenLORIS-Scene dynamic scene data set covers different real-life scenarios, including home, shopping mall, office, and various indoor and outdoor environments. During the collection process, changes in different time periods, lighting conditions, and viewing angles are considered to ensure the diversity and richness of the data, which is very suitable for verifying the accuracy and robustness of the robot system in dynamic scenes.

[0127] like Figure 5 As shown, the positioning trajectory of this embodiment is closer to the true value and exhibits almost no noticeable drift, while other visual-inertial SLAM systems, such as ORB-SLAM3 and DynaVINS, produce large errors visible to the naked eye. Specifically, ORB-SLAM3 encounters tracking failures in scenarios involving sharp rotation and white walls, while DynaVINS exhibits significant positioning drift. In contrast, this embodiment can stably estimate the UAV attitude while maintaining global consistency.

[0128] Therefore, in this application, by utilizing the fusion of IMU data, as well as a lightweight three-branch semantic segmentation network and motion consistency check, the influence of dynamic targets can be effectively eliminated. Even in environments with severe rotation and rich dynamic features, the embodiments of this application can still achieve accurate and stable attitude estimation and exhibit high localization robustness in dynamic scenes.

[0129] Figure 6 This diagram illustrates a comparison of the localization trajectories of embodiments of this application with other visual-inertial SLAM systems on a real-world dynamic scene dataset. In real-world dynamic scenes, such as... Figure 6 As shown, both DynaVINS and VINS-Fusion produce significant trajectory errors, deviating considerably from the true values. In contrast, the embodiment of this application provides a more stable and reliable attitude estimation with minimal drift. This result demonstrates that other algorithms exhibit severe positioning errors and poor robustness in real-world applications, with performance inferior to the embodiment of this application.

[0130]

[0131] Table 1

[0132]

[0133] Table 2

[0134] Table 1 shows the test metrics of the three-branch semantic segmentation network of this application embodiment and other mainstream real-time semantic segmentation algorithms on the Cityscapes dataset, including a comparison of segmentation accuracy, inference speed, computational cost (expressed as GFLOPs), and number of model parameters. The experimental results in Table 1 show that the overall performance of the three-branch semantic segmentation network of this application embodiment is significantly better than other mainstream real-time semantic segmentation networks. On the Cityscapes dataset, the three-branch semantic segmentation network achieves the highest inference accuracy and the fastest inference speed, while ensuring that the number of model parameters and computational cost are kept at a minimum, thus achieving the best balance between semantic segmentation accuracy and speed.

[0135] Table 2 shows a comparison table of algorithm module time consumption of the embodiment of the present application and other mainstream visual-inertial SLAM systems under a real dynamic scene dataset. As can be known from the simulation experiment results in Table 2, compared with other visual-inertial SLAM systems, the embodiment of the present application achieves the fastest running speed in each part, can also successfully run on edge devices with limited computing resources, and meets the real-time requirements of the algorithm. Therefore, the embodiment of the present application not only has higher positioning accuracy and robustness in a dynamic environment, but also has the fastest algorithm running speed. Moreover, the embodiment of the present application can realize real-time and reliable attitude estimation of the unmanned aerial vehicle on a platform with limited computing resources, fully demonstrates the strong application potential of the embodiment of the present application in a real dynamic scene, and has significant engineering application value.

[0136] In the present application, the image data is subjected to semantic segmentation by using a three-branch semantic segmentation network, motion consistency checking is performed on a plurality of feature points in the potential dynamic region mask, static feature points are screened out for feature tracking, and finally the camera module pose is optimized in combination with the feature point information and the pose estimation result to obtain the state pose information of the unmanned aerial vehicle. The three-branch semantic segmentation network can greatly reduce the computational complexity of the algorithm model while ensuring the accuracy of semantic segmentation, and significantly improve the inference speed. At the same time, the consistency checking is performed on the feature points, the static feature points are screened and retained, which can significantly improve the positioning accuracy and robustness of the unmanned aerial vehicle in a dynamic scene. The embodiment of the present application not only can meet the high accuracy and real-time performance of positioning in a dynamic environment, but also significantly reduces the requirement for computing resources, which is conducive to deploying it to edge devices with limited computing resources. Compared with the prior art, the present application effectively improves the positioning accuracy and robustness of the unmanned aerial vehicle in a dynamic scene, and has great engineering application value.

[0137] Those skilled in the art can understand that all or part of the features / steps of the above-mentioned method embodiments can be realized by a method, a data processing system or a computer program, and these features can be realized in a manner of not using hardware, entirely using software or using a combination of hardware and software. The aforementioned computer program can be stored in one or more computer-readable storage media, and the storage medium stores the computer program, which is executed by (such as a processor) to perform the steps of the unmanned aerial vehicle lightweight semantic visual-inertial odometry method embodiment.

[0138] The aforementioned storage medium that can store program codes includes a static hard disk, a solid state disk, a random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), an optical storage device, a magnetic storage device, a flash memory, a magnetic disk or an optical disk, and / or a combination of the above devices, i.e., can be implemented by any type of volatile or non-volatile storage device or a combination thereof.

[0139] As shown in Figure 7 The application further provides a processing device 1 embodiment, including one or more processors 11 and a memory 10; wherein the memory 10 is used to store one or more computer programs, and the one or more processors 11 are used to execute the one or more computer programs stored in the memory 10, so that the processor 11 executes the features / steps of the above-mentioned UAV lightweight semantic visual-inertial odometry method embodiment.

[0140] The above only describes the preferred embodiments of the present application, and those skilled in the art know that various changes or equivalent replacements can be made to the features and embodiments without departing from the spirit and scope of the present application. In addition, the features and embodiments can be modified to adapt to specific conditions and materials under the guidance of the present application without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the protection scope of the present application.

Claims

1. A lightweight semantic visual-inertial odometry method for unmanned aerial vehicles, characterized in that, The unmanned aerial vehicle comprises a camera module for collecting image data in real time, and a sensor for collecting motion state data, and the method comprises: performing semantic segmentation on the image data by using a three-branch semantic segmentation network to obtain a potential dynamic region mask; extracting a plurality of feature points of the image data, performing motion consistency checking on the plurality of feature points in the potential dynamic region mask, screening out static feature points, performing feature tracking on the static feature points, and obtaining feature point information of the image data and pose estimation results of the camera module, wherein the three-branch semantic segmentation network is constructed based on a depth separable convolution and a coordinate attention mechanism; obtaining pre-integration information of the motion state data, optimizing state parameters of the camera module according to the feature point information, the pose estimation results and the pre-integration information to obtain optimal pose estimation, and outputting state pose information of the unmanned aerial vehicle according to the optimal pose estimation; the state parameters comprise pose information, velocity information and IMU bias information, wherein the IMU bias information comprises accelerometer bias and gyroscope bias; the pose estimation results of the camera module comprise position information and attitude information of the camera module, that is, a rotation matrix and a translation vector; the feature point information comprises position information, tracking state and whether the feature point is a static feature point. 2.The UAV lightweight semantic visual-inertial odometry method of claim 1, wherein, The extraction of the plurality of feature points of the image data comprises: in each frame of image in the image data, at least one feature point of each frame of image is obtained by using a feature extraction algorithm. 3.The UAV lightweight semantic visual-inertial odometry method of claim 1, wherein, The motion consistency checking on the plurality of feature points in the potential dynamic region mask comprises: a plurality of matching feature point pairs of any two consecutive frames of image are obtained by using a feature matching algorithm; a fundamental matrix is estimated based on the matching feature point pairs by using an epipolar geometry constraint; the position of the corresponding matching feature point pair in the current frame of image is calculated by using the fundamental matrix, and if the distance between the position of the feature point in the current frame of image and the corresponding epipolar line is less than the preset threshold, the feature point is output as the static feature point. 4.The UAV lightweight semantic visual-inertial odometry method of claim 1, wherein, The feature tracking on the static feature points comprises: the static feature points are screened, the feature point information is obtained based on the screened static feature points, and the pose estimation results of the camera module are obtained based on the fundamental matrix of the screened static feature points.

5. The lightweight semantic visual-inertial odometry method for UAVs of claim 4, wherein, The screening of the static feature points comprises: the motion trajectory of the static feature points is estimated by comparing the pixel changes between adjacent frames of image by using a sparse optical flow method; whether the static feature points satisfy a preset condition is judged according to an optical flow estimation error, a re-projection error and a matching error, and the static feature points satisfying the preset condition are screened out.

6. The lightweight semantic visual-inertial odometry method for UAVs of claim 1, wherein, The pre-integration information of the motion state data comprises: pre-integration calculation is performed on the motion state data, and the measurement values of the position, velocity and attitude increments between adjacent frames of image are obtained as the pre-integration information according to the calculation results.

7. The lightweight semantic visual-inertial odometry method for UAVs of claim 1, wherein, The optimization of the state parameters of the camera module obtains an optimal pose estimation, comprising: optimizing the state parameters of the camera module by using a graph optimization algorithm, adjusting the pose information of the camera module, and obtaining the optimal pose estimation after multiple iterations.

8. The lightweight semantic visual-inertial odometry method for UAVs of claim 1, wherein, The loss function of the three-branch semantic segmentation network comprises at least one of an additional semantic loss, a weighted binary cross-entropy loss, a cross-entropy loss, and a boundary-aware cross-entropy loss.

9. A computer readable storage medium, characterized in that, The storage medium has stored thereon a computer program, which, when executed, implements the unmanned aerial vehicle lightweight semantic visual inertial odometry method of any one of claims 1-8.

10. A processing device, characterized by Comprise: One or more processors; A memory for storing one or more computer programs, and one or more processors for executing the one or more computer programs stored in the memory to enable the one or more processors to execute the unmanned aerial vehicle lightweight semantic visual inertial odometry method of any one of claims 1-8.

Images