Target positioning method and system based on imu-lk optical flow and storage medium

Abstract

This invention discloses a target localization method, system, and storage medium based on IMU-LK optical flow. The method includes: sparse processing multiple spatiotemporally continuous sampled images to obtain multiple target images; establishing feature associations between the multiple target images by tracking target feature points through LK optical flow, and obtaining the pixel coordinates of the feature points based on the feature associations; determining the pixel coordinate system based on a monocular camera, and establishing kinematic associations based on the pixel coordinate system and the inertial coordinate system; obtaining camera pose information based on the pixel coordinate system, the inertial coordinate system, the kinematic associations, and inertial measurement unit (IMU) data; and determining the target localization result based on the feature point pixel coordinates and the camera pose information. This invention, by designing a target localization method based on IMU-LK optical flow geometric analysis, avoids the problem of losing target point height data during the calculation process of traditional geometric analysis target localization methods, and obtains better accuracy under the same sampling conditions.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a target localization method, system and storage medium based on IMU-LK optical flow. Background Technology

[0002] Image-guided missiles are among the earliest precision-guided weapons to be put into use. They are passive homing guided weapons, mainly used to destroy various enemy ground targets within a range of 5 to 10 kilometers. However, as major military powers around the world have been developing information technology, modern ground equipment has also begun to develop towards unmanned, intelligent, and collaborative capabilities. Existing image-guided missiles have revealed shortcomings in their ability to utilize sampled image information and their poor target positioning accuracy when striking enemy ground targets, which affects their ability to perceive the battlefield situation.

[0003] Traditional geometric analysis-based target localization methods primarily rely on the principle of similar triangles to calculate the spatial position of ground target points. Analysis reveals that this approach assumes the target point lies on a horizontal plane with zero height during localization. It combines IMU data from image-guided missiles with the sine and cosine relationships of right triangles to calculate the three-dimensional coordinates of the point, requiring the horizontal plane assumption to be satisfied. Because this horizontal plane assumption projects the target point, which originally possesses height, directly onto the ground at zero height along the camera-target extension line, when using traditional geometric analysis-based target localization methods to calculate the position of a target with height, the height data in the localization result will all become 0, leading to significant errors in the ground coordinates on both axes. Therefore, how to avoid losing target height data during the calculation process of traditional geometric analysis-based target localization methods while obtaining better accuracy under the same sampling conditions is a pressing problem that needs to be solved.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a target localization method, system, and storage medium based on IMU-LK optical flow, aiming to solve the technical problem of how to avoid losing target point height data during the calculation process of traditional geometric analysis target localization methods, while obtaining better accuracy under the same sampling conditions.

[0006] To achieve the above objectives, the present invention provides a target localization method based on IMU-LK optical flow, the target localization method based on IMU-LK optical flow comprising:

[0007] The ground target was sampled by a monocular camera on a quadcopter drone platform, and multiple spatiotemporally continuous sampled images were obtained.

[0008] Multiple target images are obtained by sparse processing of multiple spatiotemporally continuous sampled images;

[0009] LK optical flow tracking is used to establish feature associations between multiple target images, and the pixel coordinates of the feature points are obtained based on the feature associations.

[0010] The pixel coordinate system is determined based on the monocular camera of the quadcopter UAV platform, and a kinematic relationship is established based on the pixel coordinate system and the inertial coordinate system.

[0011] Camera pose information is obtained based on the pixel coordinate system, the inertial coordinate system, the kinematic correlation, and the inertial measurement unit (IMU) data.

[0012] The target localization result is determined based on the feature point pixel coordinates and the camera pose information.

[0013] Optionally, the step of performing sparse processing on multiple spatiotemporally continuous sampled images to obtain multiple target images includes:

[0014] Determine the frame ordering information of multiple spatiotemporally continuous sampled images;

[0015] The image selection strategy is determined based on the frame sorting information;

[0016] Multiple target images are selected from multiple spatiotemporally continuous sampled images according to the image selection strategy.

[0017] Optionally, the step of establishing feature associations among multiple target images by tracking target feature points using LK optical flow includes:

[0018] LK optical flow tracking of target feature points determines the motion state of pixels in an image sequence;

[0019] Establish feature associations between multiple images based on the motion state of pixels in the image sequence.

[0020] Optionally, the step of establishing a kinematic relationship based on the pixel coordinate system and the inertial coordinate system includes:

[0021] The coordinate system transformation relationship is determined based on the pixel coordinate system and the inertial coordinate system;

[0022] The kinematic relationship between the pixel coordinate system and the inertial coordinate system is established based on the continuous transformation relationship of the coordinate system.

[0023] Optionally, the step of determining the target localization result based on the feature point pixel coordinates and the camera pose information includes:

[0024] Determine the target point depth information in the camera coordinate system based on the feature point pixel coordinates and the camera pose information;

[0025] The three-dimensional spatial information of the target point in the inertial coordinate system is calculated based on the target point depth information through the image sampling coordinate system transformation relationship;

[0026] The target location result is determined based on the three-dimensional spatial information of the target point.

[0027] Furthermore, to achieve the above objectives, the present invention also proposes a target localization system based on IMU-LK optical flow, wherein the target localization system based on IMU-LK optical flow includes:

[0028] The sampling module is used to sample ground targets using the monocular camera of the quadcopter UAV platform to obtain multiple spatiotemporally continuous sampled images;

[0029] The processing module is used to perform sparse processing on multiple spatiotemporally continuous sampled images to obtain multiple target images;

[0030] The determination module is used to establish feature associations between multiple target images by tracking target feature points through LK optical flow, and to obtain the pixel coordinates of the feature points based on the feature associations;

[0031] A module is established to determine the pixel coordinate system based on the monocular camera of the quadcopter UAV platform, and to establish a kinematic relationship between the pixel coordinate system and the inertial coordinate system.

[0032] The determining module is further configured to obtain camera pose information based on the pixel coordinate system, the inertial coordinate system, the kinematic correlation, and the inertial measurement unit (IMU) data;

[0033] The determining module is further configured to determine the target positioning result based on the feature point pixel coordinates and the camera pose information.

[0034] Furthermore, to achieve the above objectives, the present invention also proposes a target localization device based on IMU-LK optical flow, the device comprising: a memory, a processor, and a target localization program based on IMU-LK optical flow stored in the memory and executable on the processor, the target localization program based on IMU-LK optical flow being configured to implement the steps of the target localization method based on IMU-LK optical flow as described above.

[0035] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a target localization program based on IMU-LK optical flow, wherein when the target localization program based on IMU-LK optical flow is executed by a processor, it implements the steps of the target localization method based on IMU-LK optical flow as described above.

[0036] This invention first samples the ground target using a monocular camera on a quadcopter UAV platform, obtaining multiple spatiotemporally continuous sampled images. These images are then sparsely processed to obtain multiple target images. Next, LK optical flow tracking is used to trace target feature points and establish feature relationships between the multiple target images. The pixel coordinates of the feature points are then obtained based on these relationships. Subsequently, a pixel coordinate system is determined using the monocular camera on the quadcopter UAV platform, and a kinematic relationship is established between the pixel coordinate system and the inertial coordinate system. Finally, camera pose information is obtained based on the pixel coordinate system, the inertial coordinate system, the kinematic relationship, and inertial measurement unit (IMU) data. The target localization result is then determined based on the feature point pixel coordinates and the camera pose information. This invention employs LK optical flow to track the input image, recording information such as the imaging coordinates, global number, pixel grayscale, and optical flow characteristics of each feature point within continuous spatiotemporal time. This information can be further used to calculate the spatial coordinates of the target point. Simultaneously, since image-guided missiles can record IMU data such as angular acceleration and acceleration of the missile body in real time through inertial measurement, the optical flow motion of the target feature points at continuous sampling times can be solved by combining coordinate system transformation relationships and the Coriolis equation. Finally, the depth information of the target point at each sampling time is calculated based on the grayscale invariance assumption of LK optical flow. The IMU-LK optical flow-based geometric analysis target localization method calculates the geometric coordinates of the target point entirely based on the optical flow motion of feature points in the sampled image, effectively avoiding the horizontal plane assumption and equal strength assumptions present in traditional geometric analysis target localization methods, resulting in better accuracy. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the structure of a target positioning device based on IMU-LK optical flow in the hardware operating environment involved in the embodiments of the present invention;

[0038] Figure 2 This is a flowchart illustrating the first embodiment of the target localization method based on IMU-LK optical flow of the present invention;

[0039] Figure 3 This is a schematic diagram illustrating the principle of calculating the spatial coordinates of a target point using the conventional geometric analysis method in the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0040] Figure 4 This is a schematic diagram of a sequence of multiple target images from the first embodiment of the target localization method based on IMU-LK optical flow of the present invention;

[0041] Figure 5 This is a schematic diagram of the coordinate system geometry of the first embodiment of the target localization method based on IMU-LK optical flow of the present invention;

[0042] Figure 6This is a schematic diagram of the geometric relationship between the camera coordinate system, imaging coordinate system, and pixel coordinate system in the actual camera of the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0043] Figure 7 This is a schematic diagram of the coordinate system transformation relationship of the first embodiment of the target localization method based on IMU-LK optical flow of the present invention;

[0044] Figure 8 This is a schematic diagram of the projection relationship between the ground target and the sampled image in the first embodiment of the target localization method based on IMU-LK optical flow of the present invention;

[0045] Figure 9 This is a flowchart of the geometric analysis target localization process in the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0046] Figure 10 This is a structural block diagram of the first embodiment of the target localization system based on IMU-LK optical flow of the present invention.

[0047] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] Reference Figure 1 , Figure 1 This is a schematic diagram of the target positioning device structure based on IMU-LK optical flow in the hardware operating environment of the embodiment of the present invention.

[0050] like Figure 1As shown, the IMU-LK optical flow-based target positioning device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. Optionally, the memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0051] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on IMU-LK optical flow-based target positioning devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0052] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a target positioning program based on IMU-LK optical flow.

[0053] exist Figure 1 In the target positioning device based on IMU-LK optical flow shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the target positioning device based on IMU-LK optical flow of the present invention can be set in the target positioning device based on IMU-LK optical flow. The target positioning device based on IMU-LK optical flow calls the target positioning program based on IMU-LK optical flow stored in memory 1005 through processor 1001 and executes the target positioning method based on IMU-LK optical flow provided in the embodiment of the present invention.

[0054] This invention provides a target localization method based on IMU-LK optical flow, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0055] In this embodiment, the target localization method based on IMU-LK optical flow includes the following steps:

[0056] Step S10: Sample the ground target using the monocular camera of the quadcopter UAV platform to obtain multiple spatiotemporally continuous sampled images.

[0057] It is easy to understand that the execution subject of this embodiment can be a target positioning system based on IMU-LK optical flow with functions such as data processing, network communication and program execution, or other computer devices with similar functions. This embodiment does not limit it.

[0058] refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the principle of calculating the spatial coordinates of a target point using the traditional geometric analysis method in the first embodiment of the target localization method based on IMU-LK optical flow of the present invention. The traditional geometric analysis target localization method mainly calculates the spatial position of the ground target point based on the principle of similar triangles, and its sampling imaging relationship is as follows: Figure 3 As shown. Analysis reveals that during the positioning process, it assumes the ground target spatial point... Located on a horizontal plane with zero altitude, and using IMU data from image-guided missiles and the sine and cosine relationships of right-angled triangles to calculate the three-dimensional coordinates of spatial points, the horizontal plane assumption must be satisfied. Because this horizontal plane assumption projects the target spatial point, which originally had altitude, directly onto the ground at zero altitude along the camera-target extension line, traditional geometric analysis target localization methods will result in all altitude-related data becoming 0 when calculating the target's position information, leading to significant errors in the ground coordinates. To address this issue, this embodiment proposes a geometric analysis target localization method based on IMU-LK optical flow.

[0059] It should be noted that the monocular camera of the quadcopter drone platform (DJI-M300RTK) continuously samples ground targets with height in the inertial coordinate system to obtain multiple spatiotemporally continuous sampled images.

[0060] Step S20: Perform sparse processing on multiple spatiotemporally continuous sampled images to obtain multiple target images.

[0061] Furthermore, the process of sparse processing multiple spatiotemporally continuous sampled images to obtain multiple target images involves determining the frame ordering information of the multiple spatiotemporally continuous sampled images; determining the image selection strategy based on the frame ordering information; and selecting multiple target images from the multiple spatiotemporally continuous sampled images according to the image selection strategy.

[0062] It should be understood that the frame sorting information is the sequential numbering information of multiple spatiotemporally continuous sampled images. The image selection strategy allows the user to extract one image as the target image every few frames based on the sequential numbering information.

[0063] Multiple target images refer to a sequence of target images used for localization calculations. Figure 4 , Figure 4 This is a schematic diagram of a sequence of multiple target images in the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0064] Step S30: Establish feature associations between multiple target images by tracking target feature points using LK optical flow, and obtain the pixel coordinates of feature points based on the feature associations.

[0065] It should also be noted that by using LK optical flow to track target feature points, feature associations are established between key images (i.e., multiple target images), and information such as imaging coordinates, global number, pixel grayscale, and optical flow features of each feature point are globally recorded.

[0066] Furthermore, the processing method for establishing feature associations between multiple target images by tracking target feature points through LK optical flow involves determining the motion state of pixels in the image sequence by tracking target feature points through LK optical flow; and establishing feature associations between multiple images based on the motion state of pixels in the image sequence.

[0067] In this embodiment, based on the difference in the computational scale of pixels in the image, optical flow methods can be divided into dense optical flow, which calculates the motion of all pixels, and sparse optical flow, which calculates the motion of only some pixels. Clearly, sparse optical flow is beneficial for reducing computational load. The LK (Lucas-Kanade) optical flow feature used belongs to sparse optical flow. After extracting ground target feature points, it is tracked in an image sequence with spatiotemporal continuity. Let's assume that the target feature points are... The pixel coordinates on the sampled image at time are ,exist The pixel coordinates on the sampled image at time are Then, based on the assumption that grayscale remains unchanged, we have

[0068]

[0069] To calculate pixel motion, we can perform a Taylor expansion and retain the first-order differential term, which gives us:

[0070]

[0071] in, This represents the gradient change of the gray value of a pixel in the horizontal direction of the sampled image, which can be denoted as . ; This represents the gradient change of the gray value of a pixel in the vertical direction of the sampled image, which can be denoted as [missing information]. ; This represents the change in image grayscale over time, which can be denoted as . Substituting the assumption that the grayscale remains unchanged, we get:

[0072]

[0073] Eliminate both sides simultaneously This allows us to obtain the relationship between the motion state of feature points in an image sequence and the change in pixel grayscale:

[0074]

[0075] Through formal transformation, the motion state of pixels in a spatiotemporally continuous image sequence (i.e., the motion state of pixels in the image sequence) can be analyzed, where the horizontal velocity... vertical velocity Based on this, further rearranged into matrix form, we obtain:

[0076]

[0077] Established the motion of a single pixel , The correlation between image grayscale changes and spatiotemporally continuous image sequences (i.e., feature correlation between multiple images). Since the motion of a single pixel cannot accurately reflect the true motion of the target in the sampled image, and since it contains two unknowns that cannot be solved by a single equation, we can assume that the image contains a... All pixels within the sampling block exhibit the same instantaneous motion. The number of simultaneous equations that can be established for the pixels within the sampling block is... ,Right now:

[0078]

[0079] If we denote the coefficient matrix as follows, it can be further simplified to the form shown:

[0080]

[0081]

[0082] The above is about time. In the sampled image Sampling block motion state , The least-squares solution to the overdetermined linear equation system is expressed as:

[0083]

[0084] Besides the assumption of grayscale invariance, due to the high sampling frequency of spatiotemporally continuous image sequences, the position and grayscale changes of the same pixels between consecutive images are very small. Therefore, the LK optical flow method can assume that pixel motion conforms to the small motion assumption. By performing temporal discretization processing on the image sequence, the position and grayscale changes of the sampled image at each time step can be solved. The motion state of the sampling block is used to obtain the continuous positional changes of the target object in the sampled image (i.e., the pixel coordinates of the feature points).

[0085] Step S40: Determine the pixel coordinate system based on the monocular camera of the quadcopter UAV platform, and establish a kinematic relationship between the pixel coordinate system and the inertial coordinate system.

[0086] In this embodiment, in order to describe the transformation relationship between the target point in three-dimensional space and two-dimensional image during the level flight cruise phase of an image-guided missile, the following coordinate system first needs to be established: the shooter's coordinate system (i.e., the inertial coordinate system or the world coordinate system) ), projectile coordinate system ( ), camera coordinate system ( Imaging coordinate system ), pixel coordinate system ( The geometric relationships of the above coordinate system are as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of the coordinate system geometry of the first embodiment of the target localization method based on IMU-LK optical flow of the present invention, wherein both the imaging coordinate system and the pixel coordinate system are located on the camera imaging plane. It should be noted that since the image obtained by the actual camera is not an inverted image, the imaging plane can be equivalently moved to the front of the camera and placed on the same side as the target three-dimensional spatial point.

[0087] The shooter's coordinate system, also known as the inertial reference system, is used to represent the projectile / camera pose and target position information. The origin of the shooter's coordinate system is... For image-guided missile launch points or ground control centers, Vertically upwards from the ground Perpendicular to Pointing in the direction of the target, Determined according to the right-hand rule.

[0088] The missile's coordinate system is fixedly connected to the image-guided missile, with the origin at... Located at the center of mass of the projectile, Pointing towards the front of the projectile, Perpendicular to Located within the principal plane of symmetry of the projectile, Determined according to the right-hand rule. Specifically, the principal plane of symmetry of the projectile at the moment of launch... The planes coincide.

[0089] Origin of the camera coordinate system For the camera optical center, Point in front of the camera lens, Aligned with the vertical direction of the camera's imaging plane. Aligned with the camera's imaging plane laterally. Both the imaging coordinate system and the pixel coordinate system lie within the camera's imaging plane, with the origin of the imaging coordinate system at... The point where the camera's optical axis intersects the imaging plane. and parallel, and parallel, Camera focal length ; Origin of pixel coordinate system Located in the upper left corner of the camera's imaging plane, and parallel, and Parallel. The geometric relationship of the coordinate system in a monocular camera is shown in Figure 6. Figure 6 This is a schematic diagram of the geometric relationship between the camera coordinate system, imaging coordinate system, and pixel coordinate system in the actual camera of the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0090] Furthermore, the process of establishing kinematic association based on pixel coordinate system and inertial coordinate system is as follows: determine the continuous transformation relationship of coordinate system based on pixel coordinate system and inertial coordinate system; and establish the kinematic association between pixel coordinate system and inertial coordinate system based on the continuous transformation relationship of coordinate system.

[0091] Shooter-projectile coordinate system transformation relationship:

[0092] Since image-guided missiles undergo rigid body motion during level flight cruise, the relationship between the shooter's coordinate system and the missile's body coordinate system is an Euler transformation. In an Euler transformation, for any given vector, the magnitude and direction remain unchanged across different coordinate systems; only the relative spatial position and attitude change. The Euler transformation between the shooter's and missile's body coordinate systems consists of rotation and translation: For rotational transformations, let's assume the coordinate system... The following is the orthonormal basis: coordinate system The following is the orthonormal basis: If any spatial vector In coordinate system and coordinate system The coordinates below are respectively and Then, according to the definition of coordinates, the following conditions are met:

[0093]

[0094] You might as well multiply by the left side on both sides simultaneously. Due to the orthogonality of the basis, the coefficient matrix on the left is transformed into the identity matrix, and therefore can be rewritten as:

[0095]

[0096] This represents the rotation matrix, which describes the coordinate transformation relationship of any vector in space under different coordinate systems. For the case where subscripts exist... Indicates from coordinate system Transform to coordinate system The rotation matrix is ​​such that it satisfies:

[0097]

[0098] Translation transformation of spatial vectors involves transforming the translation vector in the corresponding coordinate system. Add to rotation transformation Then, it is indicated that, for cases with subscripts, Representing the coordinate system From the coordinate system below Origin points to coordinate system The vector at the origin. In summary, for any vector in space, the rotation and translation transformations between different coordinate systems can be expressed as:

[0099]

[0100] Transform it into homogeneous coordinate form, that is:

[0101]

[0102] Rotation matrix Translation vector Using the transformation matrix This means that the entire coordinate transformation relationship is made linear. For image-guided missiles in level flight, the rotation matrix... Represents the projectile's attitude, translation vector Indicates the position of the projectile. and This is collectively referred to as missile attitude information. Since the onboard inertial measurement unit can measure parameters such as the Euler angular velocity and acceleration of the image-guided missile in real time during flight, the rotation matrix and spatial position of the image-guided missile relative to the shooter's coordinate system at any given moment can be obtained by integrating the measured data. Specifically, it is expressed as:

[0103]

[0104] in:

[0105]

[0106]

[0107] From pitch angle Yaw angle Roll angle It was jointly determined that the three Euler angles mentioned above were obtained by integrating the inertial measurement data.

[0108] Projectile-camera coordinate system transformation relationship:

[0109] Origin of the camera coordinate system Normally not related to the center of mass of the projectile Overlap, generally used This represents the fixed structural distance between the camera's optical center and the projectile's center of mass, due to the scale in the shooter's coordinate system. Too small to be considered. Camera coordinate system. and projectile coordinate system The rotational relationship between them is achieved through the yaw frame angle. and pitch frame angle structure This indicates that the yaw frame angle and missile yaw angle The positive direction is defined in the opposite way. It can be represented as:

[0110]

[0111] Therefore, the transformation relationship from the shooter's coordinate system to the camera coordinate system is as follows:

[0112]

[0113] Camera-imaging coordinate system transformation relationship:

[0114] Since a monocular camera satisfies the pinhole imaging principle, the camera coordinate system can be determined. and imaging coordinate system The conversion relationship between them is shown below:

[0115]

[0116] in The camera focal length, as shown in the diagram. .

[0117] Image-pixel coordinate system transformation relationship:

[0118] The difference between the imaging coordinate system and the pixel coordinate system lies in the location of the origin and the unit of measurement; the coordinate representations differ by a scaling factor. Direction is , Direction is ) and origin translation ( and ),Right now:

[0119]

[0120] As shown in homogeneous matrix form:

[0121]

[0122] In summary, the coordinate transformation relationships between different coordinate systems are as follows: Figure 7 As shown, Figure 7 This is a schematic diagram of the coordinate system transformation relationship of the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0123] Coordinate system transformation relationships (i.e., continuous coordinate system transformation relationships):

[0124] Let's assume that the coordinates of any point in the archer's coordinate system are... Its projection point pixel coordinates on the camera imaging plane are ,but and The conversion relationship between them is shown below:

[0125]

[0126] Step S50: Obtain camera pose information based on the pixel coordinate system, the inertial coordinate system, the kinematic correlation, and the inertial measurement unit (IMU) data.

[0127] In this embodiment, the continuous motion state (i.e., camera pose information) of the monocular camera is calculated by integrating the IMU data such as angular acceleration and acceleration recorded in real time by inertial measurement and combining the image sampling coordinate system transformation relationship.

[0128] It should also be understood that the continuous motion state of a monocular camera refers to the displacement vector and rotation matrix of the monocular camera in the inertial coordinate system. The displacement vector and rotation matrix of the monocular camera in the inertial coordinate system are determined based on the angular velocity and flight speed of the monocular camera in the shooter's coordinate system.

[0129] In practical implementation, IMU data such as angular acceleration and acceleration are recorded in real time through inertial measurement, so the angular velocity of the monocular camera in the shooter's coordinate system can be directly obtained through integration. and flight speed .

[0130] During the sampling process, the quadcopter drone maintained level flight. Based on the IMU data measured in real time by the inertial measurement unit, the position information of the quadcopter drone at each sampling moment was obtained, as shown in the table below:

[0131]

[0132] Step S60: Determine the target localization result based on the feature point pixel coordinates and the camera pose information.

[0133] Furthermore, the processing method for determining the target localization result based on feature point pixel coordinates and camera pose information involves: determining the target point depth information in the camera coordinate system based on the feature point pixel coordinates and camera pose information; calculating the target point's three-dimensional spatial information in the inertial coordinate system based on the target point depth information through image sampling coordinate system transformation relationships; and determining the target localization result based on the target point's three-dimensional spatial information. (Reference) Figure 9 , Figure 9 This is a flowchart of the geometric analysis target localization process in the first embodiment of the target localization method based on IMU-LK optical flow of the present invention.

[0134] It should also be understood that the depth information of the target point in the camera coordinate system is calculated based on the gray-level invariance assumption of LK optical flow, and the three-dimensional spatial information of the target point in the inertial frame is calculated based on the image sampling coordinate system transformation relationship.

[0135] In the actual implementation, let's assume the spatial position of the target point in the inertial frame. Spatial position in camera coordinate system And the coordinates of the target feature points in the camera coordinate system Satisfies the Coriolis equation;

[0136]

[0137] Ground target projection relationship as follows Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the projection relationship between a ground target and a sampled image in the first embodiment of the target localization method based on IMU-LK optical flow of the present invention. In the sampled image at a certain moment, the target spatial point... Corresponding feature points on the camera's imaging plane .

[0138] Based on spatial points and feature points From the coordinate transformation relationship, we can see that... The horizontal and vertical coordinates on the camera's imaging plane are respectively:

[0139]

[0140] in, The focal length of the camera does not change over time, and the spatial point Depth information (i.e., the depth information of the target point in the camera coordinate system) Let be the quantity to be solved. Since the LK optical flow tracking results satisfy the small motion assumption, therefore in Figure 3 In the series of sampling scenarios shown, the angle between adjacent sampled images and The change is a small angle. Furthermore, based on the small angle assumption, it can be known that when the angle is very small, its tangent value can be approximated by itself. Therefore, in any sampled image... , and feature points Imaging coordinates The relationships (i.e., the image sampling coordinate system transformation relationships) can all be expressed as:

[0141]

[0142] Because the angle changes to a small angle, it can be passed through. and Representing feature points Optical flow motion:

[0143]

[0144] In summary, we further solve for the optical flow components at the feature points:

[0145]

[0146] Due to feature points Satisfying the grayscale invariance assumption Then, the values ​​can be substituted into the calculation to obtain the spatial points at each sampling time. depth information :

[0147]

[0148] It should also be understood that the actual position of the ground target in the sampled image in the shooter's coordinate system (i.e., the three-dimensional spatial information of the target point in the inertial coordinate system) is... The three-dimensional spatial information of the target point in the inertial coordinate system is used as the target positioning result, which is then used as the accuracy reference for subsequent positioning solutions.

[0149] In this embodiment, the ground target is first sampled using the monocular camera of the quadcopter UAV platform to obtain multiple spatiotemporally continuous sampled images. These images are then sparsely processed to obtain multiple target images. Next, LK optical flow tracking is used to trace target feature points and establish feature relationships between the multiple target images. The pixel coordinates of the feature points are then obtained based on these relationships. Subsequently, the pixel coordinate system is determined using the monocular camera of the quadcopter UAV platform, and a kinematic relationship is established between the pixel coordinate system and the inertial coordinate system. Finally, the camera pose information is obtained based on the pixel coordinate system, the inertial coordinate system, the kinematic relationship, and inertial measurement unit (IMU) data. The target localization result is then determined based on the feature point pixel coordinates and the camera pose information. This embodiment uses LK optical flow to track the input image, recording information such as the imaging coordinates, global number, pixel grayscale, and optical flow characteristics of each feature point in continuous space-time. This information can be further used to calculate the spatial coordinates of the target point. Simultaneously, since image-guided missiles can record IMU data such as angular acceleration and acceleration of the missile body in real time through inertial measurement, the optical flow motion of the target feature points at continuous sampling times can be solved by combining coordinate system transformation relationships and the Coriolis equation. Finally, the depth information of the target point at each sampling time is calculated based on the grayscale invariance assumption of LK optical flow. The IMU-LK optical flow-based geometric analysis target localization method calculates the geometric coordinates of the target point entirely based on the optical flow motion of feature points in the sampled image, effectively avoiding the horizontal plane assumption and equal strength assumptions present in traditional geometric analysis target localization methods, resulting in better accuracy.

[0150] Reference Figure 10 , Figure 10 This is a structural block diagram of the first embodiment of the target localization system based on IMU-LK optical flow of the present invention.

[0151] like Figure 10 As shown, the target localization system based on IMU-LK optical flow proposed in this embodiment of the invention includes:

[0152] The sampling module 101 is used to sample ground targets using a monocular camera on a quadcopter UAV platform to obtain multiple spatiotemporally continuous sampled images.

[0153] Processing module 102 is used to perform sparse processing on multiple spatiotemporally continuous sampled images to obtain multiple target images;

[0154] The determination module 103 is used to establish feature associations between multiple target images by tracking target feature points through LK optical flow, and to obtain the pixel coordinates of the feature points based on the feature associations;

[0155] Module 104 is used to determine the pixel coordinate system based on the monocular camera of the quadcopter UAV platform, and to establish a kinematic relationship based on the pixel coordinate system and the inertial coordinate system.

[0156] The determining module 102 is further configured to obtain camera pose information based on the pixel coordinate system, the inertial coordinate system, the kinematic correlation, and the inertial measurement unit (IMU) data.

[0157] The determining module 102 is further configured to determine the target positioning result based on the feature point pixel coordinates and the camera pose information.

[0158] In this embodiment, the ground target is first sampled using the monocular camera of the quadcopter UAV platform to obtain multiple spatiotemporally continuous sampled images. These images are then sparsely processed to obtain multiple target images. Next, LK optical flow tracking is used to trace target feature points and establish feature relationships between the multiple target images. The pixel coordinates of the feature points are then obtained based on these relationships. Subsequently, the pixel coordinate system is determined using the monocular camera of the quadcopter UAV platform, and a kinematic relationship is established between the pixel coordinate system and the inertial coordinate system. Finally, the camera pose information is obtained based on the pixel coordinate system, the inertial coordinate system, the kinematic relationship, and inertial measurement unit (IMU) data. The target localization result is then determined based on the feature point pixel coordinates and the camera pose information. This embodiment uses LK optical flow to track the input image, recording information such as the imaging coordinates, global number, pixel grayscale, and optical flow characteristics of each feature point in continuous space-time. This information can be further used to calculate the spatial coordinates of the target point. Simultaneously, since image-guided missiles can record IMU data such as angular acceleration and acceleration of the missile body in real time through inertial measurement, the optical flow motion of the target feature points at continuous sampling times can be solved by combining coordinate system transformation relationships and the Coriolis equation. Finally, the depth information of the target point at each sampling time is calculated based on the grayscale invariance assumption of LK optical flow. The IMU-LK optical flow-based geometric analysis target localization method calculates the geometric coordinates of the target point entirely based on the optical flow motion of feature points in the sampled image, effectively avoiding the horizontal plane assumption and equal strength assumptions present in traditional geometric analysis target localization methods, resulting in better accuracy.

[0159] Other embodiments or specific implementations of the target positioning system based on IMU-LK optical flow of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0160] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0161] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0163] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A target localization method based on IMU-LK optical flow, characterized in that, The target localization method based on IMU-LK optical flow includes the following steps: The ground target was sampled by a monocular camera on a quadcopter drone platform, and multiple spatiotemporally continuous sampled images were obtained. Multiple target images are obtained by sparse processing of multiple spatiotemporally continuous sampled images; By using LK sparse optical flow tracking based on the assumptions of grayscale invariance and small motion, feature associations are established between multiple target images, and the pixel coordinates of the feature points are obtained based on the feature associations. Wherein, the LK sparse optical flow satisfies the constraint formula: In the formula, The feature point gray value is the gradient in the horizontal direction of the image. The gradient of the gray value of the feature point in the vertical direction of the sampled image. This represents the change in image grayscale over time. This represents the horizontal velocity of the feature points in the image sequence. This represents the vertical velocity of the feature points in the image sequence. The pixel coordinate system is determined based on the monocular camera of the quadcopter UAV platform, and the continuous transformation relationship between the pixel coordinate system and the inertial coordinate system is determined based on the continuous transformation relationship between the pixel coordinate system and the inertial coordinate system. The kinematic relationship between the pixel coordinate system and the inertial coordinate system is established based on the Coriolis equation. The Coriolis equation is: In the formula, is the coordinate of the target feature point in the camera coordinate system, is the flight speed, is the angular velocity; Camera pose information is obtained based on the pixel coordinate system, the inertial coordinate system, the kinematic correlation, and the inertial measurement unit (IMU) data. Based on the feature point pixel coordinates and the camera pose information, the target point depth information in the camera coordinate system is calculated based on the LK optical flow grayscale invariance assumption and the kinematic correlation solution. Based on the target point depth information, the three-dimensional spatial information of the target point in the inertial coordinate system is calculated through the coordinate system continuous transformation relationship. The target location result is determined based on the three-dimensional spatial information of the target point.

2. The method of claim 1, wherein, The step of performing sparse processing on multiple spatiotemporally continuous sampled images to obtain multiple target images includes: Determine the frame ordering information of multiple spatiotemporally continuous sampled images; The image selection strategy is determined based on the frame sorting information; Multiple target images are selected from multiple spatiotemporally continuous sampled images according to the image selection strategy.

3. The method of claim 1, wherein, The step of establishing feature associations between multiple target images by tracking target feature points using LK sparse optical flow based on the gray-level invariance assumption and the small motion assumption includes: The motion state of pixels in an image sequence is determined by tracking target feature points using LK sparse optical flow based on the assumptions of gray-level invariance and small motion. Establish feature associations between multiple images based on the motion state of pixels in the image sequence.

4. A target localization system based on IMU-LK optical flow, applied to the target localization method based on IMU-LK optical flow as described in claim 1, characterized in that, The target localization system based on IMU-LK optical flow includes: The sampling module is used to sample ground targets using the monocular camera of the quadcopter UAV platform to obtain multiple spatiotemporally continuous sampled images; The processing module is used to perform sparse processing on multiple spatiotemporally continuous sampled images to obtain multiple target images; The determination module is used to establish feature associations between multiple target images by tracking target feature points through LK optical flow, and to obtain the pixel coordinates of the feature points based on the feature associations; A module is established to determine the pixel coordinate system based on the monocular camera of the quadcopter UAV platform, and to establish a kinematic relationship between the pixel coordinate system and the inertial coordinate system. The determining module is further configured to obtain camera pose information based on the pixel coordinate system, the inertial coordinate system, the kinematic correlation, and the inertial measurement unit (IMU) data; The determining module is further configured to determine the target positioning result based on the feature point pixel coordinates and the camera pose information.

5. A storage medium, characterized by The storage medium stores a target localization program based on IMU-LK optical flow, which, when executed by a processor, implements the steps of the target localization method based on IMU-LK optical flow as described in any one of claims 1 to 3.

Images