Target positioning method and system based on IMU-LK optical flow, and storage medium

Abstract

The invention discloses a target positioning method and system based on IMU-LK optical flow and a storage medium, and the method comprises the steps: carrying out the sparse processing of a plurality of time-space continuous sampling images, and obtaining a plurality of target images; tracking the target feature points through the LK optical flow to establish feature association among the plurality of target images, and obtaining pixel coordinates of the feature points according to the feature association; determining a pixel coordinate system according to the monocular camera, and establishing kinematics association according to the pixel coordinate system and the inertial coordinate system; camera pose information is obtained according to the pixel coordinate system, the inertial coordinate system, kinematics association and inertial measurement IMU data; and determining a target positioning result according to the feature point pixel coordinates and the camera pose information. According to the target positioning method based on IMU-LK optical flow geometric analysis, the problem that height data of a target point is lost in the resolving process of a traditional geometric analysis target positioning method is avoided, and better precision is obtained under the same sampling condition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target positioning method, system and storage medium based on IMU-LK optical flow. Background Art

[0002] Image-guided missiles are one of the earliest precision-guided weapons put into use. They are passive homing weapons and are mainly used to destroy various types of enemy ground targets within a range of 5 to 10 kilometers. However, as the world's major military powers have carried out information construction, modern ground equipment has also begun to develop in the direction of unmanned, intelligent, and collaborative. The existing image-guided missiles have exposed the shortcomings of low utilization rate of sampled image information and poor target positioning accuracy when attacking enemy ground targets, which affects their ability to perceive the battlefield situation.

[0003] The traditional geometric analysis target positioning method mainly solves the spatial position of the ground target point based on the principle of similar triangles. After analysis, it is known that in the positioning process, it is considered that the ground target space point is located on a horizontal plane with a height of zero, and the three-dimensional coordinates of the space point are solved by combining the IMU data of the image-guided missile and the sine and cosine relationship of the right triangle, which needs to meet the horizontal plane assumption. Because the horizontal plane assumption directly projects the target space point with height along the camera-target extension line to the ground with zero height, when the traditional geometric analysis target positioning method is used to solve the target position information with height, the height direction data of the positioning result will all become 0, and the two-axis coordinates of the ground will also have serious errors. Therefore, how to avoid the loss of target point height data in the solution process of the traditional geometric analysis target positioning method and obtain better accuracy under the same sampling conditions is an urgent problem to be solved.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a target positioning method, system and storage medium based on IMU-LK optical flow, aiming to solve the technical problem of how to avoid the loss of target point height data during the solution process of traditional geometric analysis target positioning method and obtain better accuracy under the same sampling conditions.

[0006] To achieve the above object, the present invention provides a target positioning method based on IMU-LK optical flow, and the target positioning method based on IMU-LK optical flow includes: The monocular camera of the quad-rotor UAV platform is used to sample ground targets and obtain multiple spatiotemporal continuous sampling images; Perform sparse processing on multiple spatiotemporal continuous sampling images to obtain multiple target images; Tracking target feature points through LK optical flow establishes feature associations between multiple target images, and obtains pixel coordinates of feature points based on the feature associations; Determine a pixel coordinate system according to the monocular camera of the quadrotor drone platform, and establish a kinematic association between the pixel coordinate system and the inertial coordinate system; Obtaining camera pose information according to the pixel coordinate system, the inertial coordinate system, the kinematic association and inertial measurement IMU data; The target positioning result is determined according to the pixel coordinates of the feature points and the camera pose information.

[0007] Optionally, the step of performing sparse processing on a plurality of spatiotemporal continuous sampling images to obtain a plurality of target images comprises: Determine frame ordering information of a plurality of spatiotemporal continuous sampling images; Determining an image selection strategy according to the frame sorting information; A plurality of target images are selected from a plurality of spatiotemporal continuous sampling images according to the image selection strategy.

[0008] Optionally, the step of establishing feature associations between multiple target images by tracking target feature points through LK optical flow includes: Track the target feature points through LK optical flow to determine the motion state of the pixels in the image sequence; A feature association is established between the multiple images according to the motion states of the pixels in the image sequence.

[0009] Optionally, the step of establishing a kinematic association according to the pixel coordinate system and the inertial coordinate system comprises: Determining a coordinate system continuous transformation relationship according to the pixel coordinate system and the inertial coordinate system; A kinematic association between the pixel coordinate system and the inertial coordinate system is established according to the continuous transformation relationship of the coordinate system.

[0010] Optionally, the step of determining the target positioning result according to the feature point pixel coordinates and the camera pose information includes: Determine the depth information of the target point in the camera coordinate system according to the pixel coordinates of the feature point and the camera pose information; Calculating the three-dimensional spatial information of the target point in the inertial coordinate system according to the depth information of the target point through the image sampling coordinate system conversion relationship; The target positioning result is determined according to the three-dimensional spatial information of the target point.

[0011] In addition, to achieve the above purpose, the present invention also proposes a target positioning system based on IMU-LK optical flow, and the target positioning system based on IMU-LK optical flow includes: The sampling module is used to sample ground targets through the monocular camera of the quadrotor drone platform to obtain multiple spatiotemporal continuous sampling images; A processing module, used for performing sparse processing on a plurality of spatiotemporal continuous sampling images to obtain a plurality of target images; A determination module is used to track target feature points through LK optical flow to establish feature associations between multiple target images, and obtain pixel coordinates of feature points based on the feature associations; An establishment module is used to determine a pixel coordinate system according to the monocular camera of the quadrotor drone platform, and to establish a kinematic association between the pixel coordinate system and the inertial coordinate system; The determination module is further used to obtain camera pose information according to the pixel coordinate system, the inertial coordinate system, the kinematic association and the inertial measurement IMU data; The determination module is further used to determine the target positioning result according to the pixel coordinates of the feature points and the camera posture information.

[0012] In addition, to achieve the above-mentioned objectives, the present invention also proposes a target positioning device based on IMU-LK optical flow, the device comprising: a memory, a processor, and a target positioning program based on IMU-LK optical flow stored in the memory and executable on the processor, the target positioning program based on IMU-LK optical flow being configured to implement the steps of the target positioning method based on IMU-LK optical flow as described above.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a target positioning program based on IMU-LK optical flow is stored. When the target positioning program based on IMU-LK optical flow is executed by a processor, the steps of the target positioning method based on IMU-LK optical flow as described above are implemented.

[0014] The present invention first samples ground targets through a monocular camera of a quad-rotor unmanned aerial vehicle platform to obtain a plurality of spatiotemporal continuity sampling images, performs sparse processing on the plurality of spatiotemporal continuity sampling images to obtain a plurality of target images, then tracks target feature points through LK optical flow to establish feature associations between the plurality of target images, and obtains pixel coordinates of feature points according to the feature associations, then determines a pixel coordinate system according to the monocular camera of the quad-rotor unmanned aerial vehicle platform, and establishes a kinematic association according to the pixel coordinate system and the inertial coordinate system, and finally obtains camera pose information according to the pixel coordinate system, the inertial coordinate system, the kinematic association and inertial measurement IMU data; and determines a target positioning result according to the pixel coordinates of the feature points and the camera pose information. The present invention tracks the input image by using LK optical flow, and the imaging coordinates, global number, pixel grayscale and optical flow characteristics of each feature point in continuous time and space are recorded, which can be further used to solve the spatial coordinates of the target point. At the same time, since the image-guided missile can record the IMU data such as the angular acceleration and acceleration of the missile body in real time through inertial measurement, the optical flow motion of the target feature point at continuous sampling moments can be solved by combining the coordinate system conversion relationship and the Coriolis equation, and finally the depth information of the target point at each sampling moment is calculated by combining the grayscale invariance assumption of the LK optical flow. The geometric analysis target positioning method based on IMU-LK optical flow completely solves the geometric coordinates of the target point according to the optical flow motion of the feature point in the sampled image, effectively avoiding the strong assumption conditions such as the horizontal plane assumption existing in the traditional geometric analysis target positioning method, and has better result accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural schematic diagram of a target positioning device based on IMU-LK optical flow in a hardware operating environment involved in an embodiment of the present invention; Figure 2 It is a flowchart of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Figure 3 This is a principle diagram of calculating the spatial coordinates of the target point using a traditional geometric analysis method according to the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Figure 4 A schematic diagram of a sequence of multiple target images of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Figure 5 A schematic diagram of the geometric relationship of the coordinate system of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Figure 6 A schematic diagram of the geometric relationship between the camera coordinate system, the imaging coordinate system, and the pixel coordinate system in an actual camera of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Figure 7A schematic diagram of coordinate system transformation relationship of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Figure 8 A schematic diagram of the projection relationship between a ground target and a sampled image in the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Fig. 9 This is a geometric analysis target positioning flow chart of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention; Fig.10 This is a structural block diagram of the first embodiment of the target positioning system based on IMU-LK optical flow of the present invention.

[0016] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0018] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of a target positioning device based on IMU-LK optical flow in the hardware operating environment involved in the embodiment of the present invention.

[0019] like Figure 1 As shown, the target positioning device based on IMU-LK optical flow may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RandomAccess Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0020] Those skilled in the art will understand that Figure 1The structure shown in the figure does not constitute a limitation on the target positioning device based on IMU-LK optical flow, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0021] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a target positioning program based on IMU-LK optical flow.

[0022] exist Figure 1 In the target positioning device based on IMU-LK optical flow shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the target positioning device based on IMU-LK optical flow of the present invention can be set in the target positioning device based on IMU-LK optical flow, and the target positioning device based on IMU-LK optical flow calls the target positioning program based on IMU-LK optical flow stored in the memory 1005 through the processor 1001, and executes the target positioning method based on IMU-LK optical flow provided by the embodiment of the present invention.

[0023] The embodiment of the present invention provides a target positioning method based on IMU-LK optical flow, referring to Figure 2 , Figure 2 It is a flowchart of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention.

[0024] In this embodiment, the target positioning method based on IMU-LK optical flow includes the following steps: Step S10: sampling the ground target through the monocular camera of the quadrotor drone platform to obtain multiple spatiotemporal continuous sampling images.

[0025] It is easy to understand that the execution subject of this embodiment can be an IMU-LK optical flow-based target positioning system with functions such as data processing, network communication and program running, or other computer devices with similar functions, etc., and this embodiment is not limited.

[0026] refer to Figure 3 , Figure 3 This is a schematic diagram of the principle of calculating the spatial coordinates of the target point using the traditional geometric analysis method of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention. The traditional geometric analysis target positioning method mainly solves the spatial position of the ground target point based on the principle of similar triangles. Its sampling imaging relationship is as follows: Figure 3 As shown. After analysis, it can be seen that in the positioning process, it considers the ground target space point Located on a horizontal plane with a height of zero, and combining the IMU data of the image-guided missile and the sine and cosine relationship of the right triangle to solve the three-dimensional coordinates of the space point, the horizontal plane assumption needs to be met. Because the horizontal plane assumption directly projects the target space point that originally has a height along the camera-target extension line to the ground at zero height, when the traditional geometric analysis target positioning method is used to solve the target position information with height, the height direction data of the positioning result will all become 0, and the two-axis coordinates of the ground will also have serious errors. In order to solve this problem, this embodiment proposes a geometric analysis target positioning method based on IMU-LK optical flow.

[0027] It should be noted that the monocular camera of the quadrotor UAV platform (DJI-M300RTK) is used to continuously sample the ground targets with height in the inertial coordinate system to obtain multiple spatiotemporal continuous sampling images.

[0028] Step S20: performing sparse processing on a plurality of spatiotemporal continuous sampling images to obtain a plurality of target images.

[0029] Furthermore, a plurality of spatiotemporal continuity sampling images are sparsely processed to obtain a plurality of target images by determining frame sorting information of the plurality of spatiotemporal continuity sampling images; determining an image selection strategy based on the frame sorting information; and selecting a plurality of target images from the plurality of spatiotemporal continuity sampling images based on the image selection strategy.

[0030] It should be understood that the frame sorting information is the sequential numbering information of multiple spatiotemporal continuous sampling images, and the image selection strategy can be that the user extracts one image as the target image every few frames according to the sequential numbering information.

[0031] The multiple target images are multiple images of the target image sequence used for positioning calculation, refer to Figure 4 , Figure 4 It is a schematic diagram of a sequence of multiple target images of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention.

[0032] Step S30: Tracking target feature points through LK optical flow to establish feature associations between multiple target images, and obtaining pixel coordinates of feature points based on the feature associations.

[0033] It should also be noted that by tracking the target feature points through LK optical flow, feature associations between key images (i.e., multiple target images) are established, and the imaging coordinates, global number, pixel grayscale, optical flow features and other information of each feature point are globally recorded.

[0034] Furthermore, a processing method for establishing feature associations between multiple target images by tracking target feature points through LK optical flow is to determine the motion state of pixels in the image sequence by tracking target feature points through LK optical flow; and establish feature associations between multiple images according to the motion state of pixels in the image sequence.

[0035] In this embodiment, according to the difference in the calculation scale of pixels in the image, the optical flow method can be divided into dense optical flow that calculates the movement of all pixels and sparse optical flow that only calculates the movement of some pixels. Obviously, sparse optical flow is helpful to reduce the amount of calculation. The LK (Lucas-Kanade) optical flow feature used belongs to sparse optical flow. After extracting the ground target feature points, they are tracked in the image sequence with spatiotemporal continuity. Let's assume that the target feature points are at The pixel coordinates on the sampled image at time instant are ,exist The pixel coordinates on the sampled image at time instant are , then according to the grayscale invariance assumption, we have

[0036] In order to calculate the pixel motion, we can perform Taylor expansion and retain the first-order differential term to obtain:

[0037] in, It represents the gradient change of the gray value of the pixel in the horizontal direction of the sampled image, which can be recorded as ; It represents the gradient change of the gray value of the pixel in the vertical direction of the sampled image, which can be recorded as ; Represents the change of image grayscale over time, which can be recorded as . Then substitute the grayscale invariance assumption shown in the figure into the equation:

[0038] Eliminate both sides at the same time , we can get the relationship between the motion state of feature points in the image sequence and the change of pixel grayscale:

[0039] Through form transformation, the motion state of pixels in the spatiotemporal continuous image sequence (i.e., the motion state of pixels in the image sequence) can be analyzed, where the horizontal speed , the vertical speed . Based on this, it can be further organized into a matrix form and obtained:

[0040] Established single pixel motion , The correlation between the grayscale changes of the image in the spatiotemporal continuous image sequence (i.e., the feature correlation between multiple images). Since the movement of a single pixel is difficult to reflect the actual movement of the target in the sampled image, and the two unknown quantities to be solved cannot be solved by a single equation, it is better to assume that an image All pixels in the sampling block have the same instantaneous motion. For the pixel points in the sampling block, the simultaneous dimension of the equations that can be established is ,Right now:

[0041] Let's record the coefficient matrix as follows, which can be further simplified to the form shown below:

[0042]

[0043] The above is about the time In the sampled image Sampling block motion status , The least squares solution of the overdetermined linear equations is expressed as:

[0044] In addition to the grayscale invariance assumption, due to the high sampling frequency of the spatiotemporal continuous image sequence, the position and grayscale changes of the same pixel between consecutive images are very small. Therefore, the LK optical flow method can assume that the pixel motion conforms to the small motion assumption. By performing time discretization on the image sequence, the The motion state of the sampling block is finally used to obtain the continuous position change of the target object in the sampled image (that is, the pixel coordinates of the feature points).

[0045] Step S40: determining a pixel coordinate system according to the monocular camera of the quadrotor UAV platform, and establishing a kinematic association between the pixel coordinate system and the inertial coordinate system.

[0046] In this embodiment, in order to describe the conversion relationship between the target point in the three-dimensional space and the two-dimensional image in the level flight cruise phase of the image-guided missile, the following coordinate systems need to be established first: the shooter coordinate system (i.e., the inertial coordinate system or the world coordinate system) ( )、Projectile coordinate system( )、Camera coordinate system( )、Imaging coordinate system( )、Pixel Coordinate System( ), the geometric relationship of the above coordinate system is as follows Figure 5As shown, Figure 5 This is a schematic diagram of the geometric relationship of the coordinate system of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention, in which the imaging coordinate system and the pixel coordinate system are both located in the camera imaging plane. It should be noted that since the image obtained by the actual camera is not an inverted image, the imaging plane can be equivalently moved to the front of the camera and placed on the same side as the target three-dimensional space point.

[0047] The shooter coordinate system is the inertial reference system (i.e., inertial coordinate system), which is used to represent the missile / camera pose and target position information. It is the starting point or ground control center for image-guided missiles. Vertically upward from the ground, Perpendicular to Point to the target direction, Determined according to the right-hand rule.

[0048] The missile body coordinate system is fixed to the image-guided missile, and the origin Located at the center of mass of the projectile, Pointing to the front end of the missile, Perpendicular to Located in the main symmetry plane of the projectile, Determined by the right-hand rule. Among them, the main symmetry plane of the projectile is The planes overlap.

[0049] The origin of the camera coordinate system is the camera optical center, Point in front of the camera lens. The vertical direction is consistent with the camera imaging plane. The imaging coordinate system and the pixel coordinate system are both located in the camera imaging plane. The origin of the imaging coordinate system is is the intersection of the camera optical axis and the imaging plane, and parallel, and parallel, is the camera focal length ; Pixel coordinate system origin Located in the upper left corner of the camera imaging plane, and parallel, and Parallel. The geometric relationship of the above coordinate system in the monocular camera is shown in 6. Figure 6 It is a schematic diagram of the geometric relationship between the camera coordinate system, the imaging coordinate system, and the pixel coordinate system in the actual camera of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention.

[0050] Furthermore, the processing method for establishing kinematic association according to the pixel coordinate system and the inertial coordinate system is to determine the coordinate system continuous transformation relationship according to the pixel coordinate system and the inertial coordinate system; and establish the kinematic association between the pixel coordinate system and the inertial coordinate system according to the coordinate system continuous transformation relationship.

[0051] Shooter-projectile coordinate system conversion relationship: Since the image-guided missile performs rigid body motion in the level flight cruise phase, the relationship between the shooter coordinate system and the missile body coordinate system is an Euler transformation. In the Euler transformation, for any determined vector, the modulus and direction in different coordinate systems remain unchanged, and only the relative spatial position and attitude in each coordinate system change. The Euler transformation between the shooter coordinate system and the missile body coordinate system consists of rotation and translation: For the rotation transformation, it is assumed that the coordinate system The unit orthogonal basis under , coordinate system The unit orthogonal basis under , if any space vector In the coordinate system and coordinate system The coordinates below are and , then according to the coordinate definition:

[0052] You can multiply both sides by left , due to the orthogonal properties of the basis, the coefficient matrix on the left is transformed into the identity matrix, so it can be rewritten as:

[0053] Represents the rotation matrix, which describes the coordinate transformation relationship of any vector in the space under different coordinate systems. For the case where there is a subscript, Indicates the coordinate system Convert to coordinate system The rotation matrix of , and satisfies:

[0054] The translation transformation of a space vector is achieved by converting the translation vector in the corresponding coordinate system Add to rotation transformation Then express it, for the case with subscript, Representing coordinate system From the coordinate system Origin pointing coordinate system The vector of the origin. In summary, for any vector in space, the rotation and translation transformation between different coordinate systems can be expressed as:

[0055] Converted into homogeneous coordinate form, that is:

[0056] The rotation matrix and translation vectors Using transformation matrix Represents that the entire coordinate transformation relationship becomes a linear form. For the image-guided missile in level flight and cruise, the rotation matrix Represents the missile body posture, translation vector Indicates the projectile position. and Since the missile-borne inertial measurement element can measure the Euler angular velocity and acceleration and other parameters of the image-guided missile in real time during flight, the rotation matrix and spatial position of the image-guided missile relative to the shooter coordinate system at any time can be obtained by integrating the measured data. , specifically expressed as:

[0057] in:

[0058]

[0059] By pitch angle , yaw angle , Roll Angle It is jointly determined that the above three Euler angles are obtained by integrating the inertial measurement data.

[0060] Projectile-camera coordinate system conversion relationship: The origin of the camera coordinate system Usually not with the center of mass of the projectile Overlap, generally used Represents the fixed structural distance between the camera optical center and the projectile mass center. Too small, so it can be ignored. Camera coordinate system and the projectile body coordinate system The rotation relationship between them is determined by the yaw frame angle and pitch frame angle structure represents, where the yaw frame angle and the missile body yaw angle The positive direction of is defined in the opposite direction. It can be expressed as:

[0061] Therefore, the conversion relationship from the shooter coordinate system to the camera coordinate system is shown as follows:

[0062] Camera-imaging coordinate system transformation relationship: Since the monocular camera satisfies the pinhole imaging principle, we know that the camera coordinate system and imaging coordinate system The conversion relationship between them is shown as follows:

[0063] in is the focal length of the camera, i.e. .

[0064] Imaging-pixel coordinate system conversion relationship: The difference between the imaging coordinate system and the pixel coordinate system lies in the different origin positions and measurement units. The coordinate representation differs by a scale ( Direction , Direction ) and the origin translation ( and ),Right now:

[0065] The homogeneous matrix form is shown as:

[0066] In summary, the coordinate transformation relationship between the coordinate systems is as follows: Figure 7 As shown, Figure 7 Schematic diagram of coordinate system transformation relationship of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention.

[0067] Coordinate system transformation relationship (i.e. coordinate system continuous transformation relationship): Let's assume that the coordinates of any space point in the shooter coordinate system are , the pixel coordinates of its projection point on the camera imaging plane are ,but and The conversion relationship between them is shown as follows:

[0068] Step S50: Obtaining camera pose information according to the pixel coordinate system, the inertial coordinate system, the kinematic association and inertial measurement IMU data.

[0069] In this embodiment, the continuous motion state of the monocular camera (i.e., the camera posture information) is calculated by integrating the IMU data such as angular acceleration and acceleration recorded in real time by inertial measurement and combining the image sampling coordinate system conversion relationship.

[0070] It should also be understood that the continuous motion state of the monocular camera refers to the displacement vector and rotation matrix of the monocular camera in the inertial coordinate system, and the displacement vector and rotation matrix of the monocular camera in the inertial coordinate system are determined according to the angular velocity and flight speed of the monocular camera in the shooter coordinate system.

[0071] In the specific implementation, IMU data such as angular acceleration and acceleration are recorded in real time through inertial measurement, so the angular velocity of the monocular camera in the shooter coordinate system can be directly obtained by integration. and flight speed .

[0072] During the sampling process, the quadcopter maintained level flight. Based on the IMU data measured in real time by the inertial measurement unit, the position information of the quadcopter at each sampling time was obtained, as shown in the following table:

[0073] Step S60: determining the target positioning result according to the pixel coordinates of the feature points and the camera posture information.

[0074] Furthermore, the processing method for determining the target positioning result according to the pixel coordinates of the feature points and the camera pose information is to determine the depth information of the target point in the camera coordinate system according to the pixel coordinates of the feature points and the camera pose information; calculate the three-dimensional space information of the target point in the inertial coordinate system through the image sampling coordinate system conversion relationship according to the depth information of the target point; and determine the target positioning result according to the three-dimensional space information of the target point. Fig. 9 , Fig. 9 This is a geometric analysis target positioning flow chart of the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention.

[0075] It should also be understood that the depth information of the target point in the camera coordinate system is calculated based on the grayscale invariance assumption of the LK optical flow, and the three-dimensional spatial information of the target point in the inertial system is calculated based on the image sampling coordinate system conversion relationship.

[0076] In the specific implementation, it is assumed that the spatial position of the target point in the inertial system is , the spatial position in the camera coordinate system , and the coordinates of the target feature points in the camera coordinate system Satisfy the Coriolis equation;

[0077] The projection relationship of ground targets is as follows: Figure 8As shown, Figure 8 Schematic diagram of the projection relationship between the ground target and the sampled image in the first embodiment of the target positioning method based on IMU-LK optical flow of the present invention. In the sampled image at a certain moment, the target space point Corresponding to the feature points on the camera imaging plane .

[0078] According to the space point and feature points The coordinate transformation relationship is as follows: The horizontal and vertical coordinates on the camera imaging plane are:

[0079] in, is the focal length of the camera and does not change with time. Depth information (i.e., depth information of the target point in the camera coordinate system) is the quantity to be solved. Since the LK optical flow tracking result satisfies the small motion assumption, Figure 3 In the series of sampling scenes shown, the angle between adjacent sampling images and The change is a small angle. At the same time, according to the small angle hypothesis, when the angle is very small, its tangent value can be approximated by itself. Therefore, it can be seen that in any sampled image , and feature points Imaging coordinates The relationship (i.e., the image sampling coordinate system transformation relationship) can be expressed as:

[0080] Since the angle change is small, it can be and Representing feature points Optical flow motion:

[0081] In summary, the optical flow components of the feature points are further solved:

[0082] Because the feature points Satisfy the grayscale invariance assumption , then we can substitute it into the calculation to get the spatial point at each sampling time Depth information :

[0083] It should also be understood that the actual position of the ground target in the sampled image in the shooter coordinate system (i.e., the three-dimensional spatial information of the target point in the inertial coordinate system) is , the three-dimensional spatial information of the target point in the inertial coordinate system is used as the target positioning result, which is used as the accuracy reference of the subsequent positioning solution results.

[0084] In this embodiment, the ground target is first sampled by the monocular camera of the quadrotor UAV platform to obtain a plurality of spatiotemporal continuity sampling images, and the plurality of spatiotemporal continuity sampling images are sparsely processed to obtain a plurality of target images. Then, the target feature points are tracked by LK optical flow to establish feature associations between the plurality of target images, and the pixel coordinates of the feature points are obtained according to the feature associations. After that, the pixel coordinate system is determined according to the monocular camera of the quadrotor UAV platform, and a kinematic association is established according to the pixel coordinate system and the inertial coordinate system. Finally, the camera pose information is obtained according to the pixel coordinate system, the inertial coordinate system, the kinematic association and the inertial measurement IMU data; and the target positioning result is determined according to the pixel coordinates of the feature points and the camera pose information. This embodiment uses LK optical flow to track the input image, and the imaging coordinates, global number, pixel grayscale and optical flow characteristics of each feature point in continuous space and time are recorded, which can be further used to solve the spatial coordinates of the target point. At the same time, since the image-guided missile can record the IMU data such as the angular acceleration and acceleration of the missile body in real time through inertial measurement, the optical flow motion of the target feature point at continuous sampling moments can be solved by combining the coordinate system conversion relationship and the Coriolis equation. Finally, the depth information of the target point at each sampling moment is calculated by combining the grayscale invariance assumption of the LK optical flow. The geometric analysis target positioning method based on IMU-LK optical flow completely solves the geometric coordinates of the target point according to the optical flow motion of the feature point in the sampled image, effectively avoiding strong assumptions such as the horizontal plane assumption in the traditional geometric analysis target positioning method, and has better result accuracy.

[0085] Reference Fig.10 , Fig.10 This is a structural block diagram of the first embodiment of the target positioning system based on IMU-LK optical flow of the present invention.

[0086] like Fig.10 As shown, the target positioning system based on IMU-LK optical flow proposed in an embodiment of the present invention includes: The sampling module 101 is used to sample ground targets through a monocular camera of a quad-rotor UAV platform to obtain multiple spatiotemporal continuous sampling images; The processing module 102 is used to perform sparse processing on a plurality of spatiotemporal continuous sampling images to obtain a plurality of target images; The determination module 103 is used to track the target feature points through LK optical flow to establish feature associations between multiple target images, and obtain pixel coordinates of the feature points according to the feature associations; An establishing module 104 is used to determine a pixel coordinate system according to the monocular camera of the quadrotor drone platform, and to establish a kinematic association between the pixel coordinate system and the inertial coordinate system; The determination module 102 is further used to obtain camera pose information according to the pixel coordinate system, the inertial coordinate system, the kinematic association and the inertial measurement IMU data; The determination module 102 is further configured to determine a target positioning result according to the pixel coordinates of the feature points and the camera posture information.

[0087] In this embodiment, the ground target is first sampled by the monocular camera of the quadrotor UAV platform to obtain a plurality of spatiotemporal continuity sampling images, and the plurality of spatiotemporal continuity sampling images are sparsely processed to obtain a plurality of target images. Then, the target feature points are tracked by LK optical flow to establish feature associations between the plurality of target images, and the pixel coordinates of the feature points are obtained according to the feature associations. After that, the pixel coordinate system is determined according to the monocular camera of the quadrotor UAV platform, and a kinematic association is established according to the pixel coordinate system and the inertial coordinate system. Finally, the camera pose information is obtained according to the pixel coordinate system, the inertial coordinate system, the kinematic association and the inertial measurement IMU data; and the target positioning result is determined according to the pixel coordinates of the feature points and the camera pose information. This embodiment uses LK optical flow to track the input image, and the imaging coordinates, global number, pixel grayscale and optical flow characteristics of each feature point in continuous space and time are recorded, which can be further used to solve the spatial coordinates of the target point. At the same time, since the image-guided missile can record the IMU data such as the angular acceleration and acceleration of the missile body in real time through inertial measurement, the optical flow motion of the target feature point at continuous sampling moments can be solved by combining the coordinate system conversion relationship and the Coriolis equation. Finally, the depth information of the target point at each sampling moment is calculated by combining the grayscale invariance assumption of the LK optical flow. The geometric analysis target positioning method based on IMU-LK optical flow completely solves the geometric coordinates of the target point according to the optical flow motion of the feature point in the sampled image, effectively avoiding strong assumptions such as the horizontal plane assumption in the traditional geometric analysis target positioning method, and has better result accuracy.

[0088] Other embodiments or specific implementations of the target positioning system based on IMU-LK optical flow of the present invention can refer to the above-mentioned method embodiments, which will not be described in detail here.

[0089] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0090] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0092] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A target positioning method based on IMU-LK optical flow, characterized in that: The target positioning method based on IMU-LK optical flow includes the following steps: The monocular camera of the quad-rotor UAV platform is used to sample ground targets and obtain multiple spatiotemporal continuous sampling images; Perform sparse processing on multiple spatiotemporal continuous sampling images to obtain multiple target images; Tracking target feature points through LK optical flow establishes feature associations between multiple target images, and obtains pixel coordinates of feature points based on the feature associations; Determine a pixel coordinate system according to the monocular camera of the quadrotor drone platform, and establish a kinematic association between the pixel coordinate system and the inertial coordinate system; Obtaining camera pose information according to the pixel coordinate system, the inertial coordinate system, the kinematic association and the inertial measurement IMU data; The target positioning result is determined according to the pixel coordinates of the feature points and the camera pose information.

2. The method according to claim 1, characterized in that The step of performing sparse processing on a plurality of spatiotemporal continuous sampling images to obtain a plurality of target images comprises: Determine frame ordering information of a plurality of spatiotemporal continuous sampling images; Determining an image selection strategy according to the frame sorting information; A plurality of target images are selected from a plurality of spatiotemporal continuity sampling images according to the image selection strategy.

3. The method according to claim 1, characterized in that The step of establishing feature associations between multiple target images by tracking target feature points through LK optical flow includes: Track the target feature points through LK optical flow to determine the motion state of the pixels in the image sequence; A feature association is established between the multiple images according to the motion states of the pixels in the image sequence.

4. The method according to claim 3, characterized in that The step of establishing a kinematic association based on the pixel coordinate system and the inertial coordinate system comprises: Determining a coordinate system continuous transformation relationship according to the pixel coordinate system and the inertial coordinate system; A kinematic association between the pixel coordinate system and the inertial coordinate system is established according to the continuous transformation relationship of the coordinate system.

5. The method according to claim 4, characterized in that The step of determining the target positioning result according to the feature point pixel coordinates and the camera pose information comprises: Determine the depth information of the target point in the camera coordinate system according to the pixel coordinates of the feature point and the camera pose information; Calculating the three-dimensional spatial information of the target point in the inertial coordinate system according to the depth information of the target point through the image sampling coordinate system conversion relationship; The target positioning result is determined according to the three-dimensional spatial information of the target point.

6. A target positioning system based on IMU-LK optical flow, characterized in that: The target positioning system based on IMU-LK optical flow includes: The sampling module is used to sample ground targets through the monocular camera of the quadrotor drone platform to obtain multiple spatiotemporal continuous sampling images; A processing module, used for performing sparse processing on a plurality of spatiotemporal continuous sampling images to obtain a plurality of target images; A determination module is used to track target feature points through LK optical flow to establish feature associations between multiple target images, and obtain pixel coordinates of feature points based on the feature associations; An establishment module is used to determine a pixel coordinate system according to the monocular camera of the quadrotor drone platform, and to establish a kinematic association between the pixel coordinate system and the inertial coordinate system; The determination module is further used to obtain camera pose information according to the pixel coordinate system, the inertial coordinate system, the kinematic association and the inertial measurement IMU data; The determination module is further used to determine the target positioning result according to the pixel coordinates of the feature points and the camera posture information.

7. A storage medium, characterized in that: The storage medium stores a target positioning program based on IMU-LK optical flow, and when the target positioning program based on IMU-LK optical flow is executed by the processor, the steps of the target positioning method based on IMU-LK optical flow as described in any one of claims 1 to 5 are implemented.

Images