A warhead detection method based on the joint spatiotemporal characteristics of strongly correlated targets

Abstract

The present invention discloses a method for detecting warheads in a falling area based on the joint spatiotemporal characteristics of strongly correlated targets. First, camera parameter calibration and three-dimensional spatial coordinate reconstruction are performed. The internal parameters of the camera are calibrated based on a black and white checkerboard pattern, and external parameter calibration is completed in combination with the pixel-to-world coordinate conversion process. The least squares method is used to calculate the intersection of multiple optical axes to achieve three-dimensional reconstruction of spatial coordinate points under multiple fields of view. Then, a preliminary detection of the warhead motion state is performed under low perception accuracy conditions based on optical flow tracking. Finally, the warhead motion state is corrected based on the spatiotemporal characteristics of the strongly correlated targets and the projectile-target intersection parameters, and the preliminary detection results of the warhead are corrected to complete the accurate detection of the warhead. The present invention realizes the accurate detection of warheads in the complex dynamic environment conditions of field training, reduces costs and operational difficulty, and realizes large-scale rapid deployment and application in training sites.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a method for detecting warheads in a falling area based on the joint spatiotemporal characteristics of strongly correlated targets. Background Art

[0002] Accurately detecting weapon warheads and then obtaining the projectile-target intersection parameters such as the terminal trajectory, landing velocity, landing posture, and landing point of the warhead in the landing zone is fundamental to testing weapon strike accuracy, identifying warhead performance, and evaluating damage effects. As the test and training missions of various types of weapons and equipment continue to advance in the direction of actual combat and scale, stringent requirements are placed on the rapid maneuverability and wide-area deployment capabilities of test equipment. The natural environment at the test and training site is harsh and the scene content is complex. Interference targets with constantly changing spatiotemporal characteristics such as thick smoke, wind and sand, clouds, streamers, drones, and target debris often appear. Accurately detecting warheads under these conditions is quite challenging. Therefore, constructing a landing zone warhead detection method suitable for the test and training site, which can complete the measurement task in real time and efficiently while taking into account the economic and reliability of hardware costs, meets the actual needs of large-scale application at the test and training site, is of great engineering significance.

[0003] Currently, warhead detection in the impact zone is primarily based on radar echoes, acoustic / seismic signals, and binocular stereo vision. Radar-based warhead detection suffers from issues such as bulky, expensive, technically complex, difficult to maintain, and low absolute measurement accuracy. Currently, it is only applicable to trajectory detection at offshore ranges, hindering widespread adoption. Impact point detection based on acoustic / seismic signals suffers from low detection accuracy and an inability to identify warhead trajectory and posture. Binocular stereo vision-based warhead detection has been widely used due to its simplicity, ease of operation, high target positioning accuracy, and fast system response. For example, Zhang Hui et al. designed a binocular vision-based miss distance measurement system for offshore targets using high-speed cameras. This system can automatically measure miss distances in real time, achieving a miss distance uncertainty of less than 3.79 meters under experimental conditions. In his master's thesis, Xu Fengliang detailed a binocular stereo vision-based missile-target spatial intersection measurement system. Using inter-frame differencing and background modeling methods, the system achieved moving target detection in laboratory conditions and measured missile-target intersection parameters.

[0004] Due to the poor image quality and low frame rate of general high-definition or industrial cameras, the current warhead detection based on binocular stereo vision relies on professional high-speed cameras in hardware. These cameras are expensive and difficult to deploy on a large scale and over a wide area. In terms of processing methods, they rely on traditional image difference processing methods, which have poor anti-interference capabilities and are difficult to adapt to the complex dynamic environment of the test training site. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention provides a method for detecting warheads in the falling area based on the joint spatiotemporal characteristics of strongly correlated targets. First, camera parameter calibration and three-dimensional spatial coordinate reconstruction are performed; the calibration of the camera's internal parameters is completed based on a black and white checkerboard, and the external parameter calibration is completed in combination with the pixel-to-world coordinate conversion process; the least squares method is used to calculate the intersection of multiple optical axes to achieve three-dimensional reconstruction of spatial coordinate points under multiple fields of view; then, a preliminary detection of the warhead's motion state is performed under low perception accuracy conditions based on optical flow tracking. Finally, based on the warhead's motion state correction and the missile-target intersection parameters of the strongly correlated spatiotemporal characteristics, the preliminary detection results of the warhead are corrected to complete the accurate detection of the warhead. The present invention achieves accurate detection of warheads under complex dynamic environmental conditions in field training, reduces costs and operational difficulty, and realizes large-scale rapid deployment and application at training sites.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step 1: Arrange the camera;

[0008] Use the two cameras to observe the target area at a set angle to ensure that the target area can be observed by both cameras;

[0009] Step 2: Camera parameter calibration;

[0010] Step 2-1: Given a checkerboard calibration plate, detect the pixel coordinates of each corner point on the calibration plate image Assuming that the world coordinate system is fixed on the chessboard, the physical coordinates of any point on the chessboard are Since the size of each grid on the calibration plate is known, the physical coordinates of each corner point in the world coordinate system are calculated using the Harris corner detection algorithm based on the grayscale changes of the black and white grid edges. ;

[0011] Step 2-2: Solve the camera's internal and external parameters;

[0012] According to the pixel coordinates of each corner point , the physical coordinates of each corner point in the world coordinate system , solve the transformation relationship between pixel and world coordinate system, obtain the camera's intrinsic parameter matrix and extrinsic parameter matrix to complete the camera calibration; the transformation relationship between pixel and world coordinate system is:

[0013] (1)

[0014] in, is the internal parameter matrix, is the external parameter matrix; represents the image distance, It represents the angle between the horizontal and vertical edges of the photographic plate. Respectively represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system, Indicates the physical length of a pixel on the camera's photosensitive plate in both directions. represents the rotation matrix, represents the translation vector;

[0015] Solve the camera's intrinsic parameter matrix based on Zhang Zhengyou's calibration;

[0016] For external parameters, several spatial points in a graph are known P i The three-dimensional coordinates of P i = ( X i , Y i , Z i ) and its corresponding pixel coordinates x i = ( u i , v i ), find R and T;

[0017] The origin of the world coordinate system is the center of the experimental site, with due east as the positive direction of the first axis, due north as the second positive direction, and height as the third positive direction. The specific values are measured by the total station. R and T , first construct an augmented matrix A = [R|T], according to the projection principle:

[0018]

[0019] Eliminate s using the last line to get

[0020]

[0021] make

[0022] 、 、 、

[0023] get

[0024]

[0025]

[0026] Assuming there are N characteristic points, list the equations:

[0027]

[0028] Solving the above system of equations yields , which is the specific value of the augmented matrix A = [R|T]; the external parameters describe the position and angle of the camera relative to the origin of the world coordinate system;

[0029] Step 3: 3D space coordinate reconstruction;

[0030] Given the pixel coordinates of the target at multiple angles ( , )、( , ), establish an optical axis convergence model between the pixel point and the camera position;

[0031] According to the pixel coordinates of the target ( , )、( , ), for the three-dimensional coordinates ( , , ) Using the optical axis convergence model, we can solve the problem by simultaneously solving multiple optical axis rays:

[0032] First, according to the camera model

[0033]

[0034] Formula (1) is simplified to:

[0035] (2)

[0036] Where Q is a 3*4 matrix, and Equation (2) can be written as two equations:

[0037] (3)

[0038] (4)

[0039] in, Represents the i-th row and j-th column element of the first camera matrix;

[0040] Similarly, for another camera, two equations are obtained:

[0041] (5)

[0042] (6)

[0043] Finally, the least squares method is used to obtain the three-dimensional coordinates ( , , );

[0044] Step 4: Low-perception-precision warhead detection based on optical flow tracking and multi-target motion patterns;

[0045] Step 4-1: Perform inter-frame difference processing on the warhead strike process image sequence captured by the camera to extract all feature points to be tracked in adjacent image frames;

[0046] Step 4-2: Use the Lucas–Kanade optical flow method to track optical flow points and calculate the optical flow of each frame of the video :

[0047]

[0048] in, Represents two frames of images in the video. Described t time N The position change relationship of each corner point; filter out the corner points whose position change exceeds the set value and moves downward to obtain the pixel coordinates of the warhead center point ( , ).

[0049] Step 4-3: After obtaining the pixel coordinates of the center point of the missile warhead in different cameras, substitute them into the formula in step 3 to obtain its three-dimensional coordinates ( , , ).

[0050] The beneficial effects of the present invention are as follows:

[0051] 1. Based on the imaging characteristics of high-definition or industrial cameras, this paper proposes a warhead motion state detection method under low perception accuracy conditions based on optical flow tracking. The detection results are corrected based on the joint spatiotemporal characteristics of strongly correlated targets, achieving accurate detection of warheads under complex dynamic environment conditions in field training.

[0052] 2. The present invention is based on general or industrial cameras and combines highly automated processing methods to reduce costs and operational difficulties, and achieve large-scale rapid deployment and application in training venues. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the method of the present invention.

[0054] Figure 2 The original video data of the embodiment of the present invention, (a) frame 0, (b) frame 1, and (c) frame 2.

[0055] Figure 3 This is the image difference result of the embodiment of the present invention.

[0056] Figure 4 This is a preliminary detection diagram of the warhead under low perception accuracy conditions in an embodiment of the present invention.

[0057] Figure 5 This is a diagram showing the final warhead detection results based on the joint spatiotemporal characteristics of strongly correlated targets according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to the accompanying drawings and examples.

[0059] In order to solve the problems in the prior art, the present invention proposes a warhead detection method in the falling area based on the joint spatiotemporal characteristics of strongly correlated targets under field test conditions, which realizes rapid and high-precision detection of warheads while taking economy into consideration.

[0060] The present invention discloses a method for detecting warheads in the falling area under field test conditions based on the joint spatiotemporal characteristics of strongly correlated targets. This method uses economical and reliable general-purpose high-definition or industrial cameras as observation equipment. Aiming at their imaging characteristics and combining them with optical flow tracking technology, a long-distance, small-target tracking method based on the spatiotemporal structural characteristics of warheads is designed. This method solves the problem of warhead detection being difficult and susceptible to interference in complex dynamic field environments, ultimately achieving real-time and accurate detection of warheads and their large-scale rapid deployment and application in test and training venues.

[0061] like Figure 1 As shown, the technical solution of the present invention includes two major modules, namely camera parameter calibration and three-dimensional space coordinate reconstruction, and warhead detection under complex dynamic environmental conditions. The specific technical solution is as follows.

[0062] (1) General high-definition or industrial camera parameter calibration and 3D space coordinate reconstruction;

[0063] 1) Internal and external parameter calibration: The camera’s internal parameters are calibrated based on a black and white checkerboard pattern, and the external parameter calibration is completed by combining the pixel-to-world coordinate conversion process;

[0064] 2) 3D spatial coordinate reconstruction: Dynamically model the mapping relationship between pixel points and spatial coordinates, and use the least squares method to calculate the intersection of multiple optical axes to achieve 3D reconstruction of spatial coordinate points under multiple fields of view.

[0065] (2) Warhead detection under complex dynamic environmental conditions

[0066] 1) Preliminary detection of warhead motion state under low perception accuracy conditions based on optical flow tracking

[0067] Based on the imaging characteristics of general-purpose high-definition or industrial cameras, the system combines optical flow information to model the warhead's behavior under low perception accuracy. This provides a basis for visual target behavior discrimination, eliminates interfering targets with slowly changing spatiotemporal characteristics such as clouds and smoke, and completes preliminary detection of the warhead's motion state at the current moment in the video image sequence.

[0068] 2) Warhead motion state correction and missile-target rendezvous parameter measurement based on the spatiotemporal characteristics of strongly correlated targets

[0069] According to the differences in the spatiotemporal characteristics between the warhead and the interference target and the spatiotemporal characteristics correlation between strongly correlated targets, the interference target with similar motion law to the warhead is suppressed, the preliminary detection results of the warhead are corrected, the accurate detection of the warhead is completed, and based on this, the measurement of the missile-target intersection parameters is realized. Specific embodiment:

[0071] 1. Arrange the camera

[0072] Place two general-purpose high-definition or industrial cameras at a certain angle to observe the target area, ensuring that the target area can be observed by both cameras. The captured image is as follows: Figure 2 shown.

[0073] 2. Camera parameter calibration

[0074] Given a checkerboard calibration plate, detect the pixel coordinates of each corner point on the calibration plate image Assuming that the world coordinate system is fixed on the chessboard, the physical coordinates of any point on the chessboard are Since the size of each grid on the calibration plate is known, the physical coordinates of each corner point in the world coordinate system can be calculated using the Harris corner detection algorithm based on the grayscale changes of the black and white grid edges. .

[0075] According to the pixel coordinates of each corner point , the physical coordinates of each corner point in the world coordinate system , solve the pixel-world coordinate system transformation relationship, obtain the camera's intrinsic and extrinsic matrix to complete the camera calibration. The pixel-world coordinate system transformation relationship is:

[0076] (1)

[0077] Based on Zhang Zhengyou's calibration method, Z. Zhang, "A flexible new technique for camera calibration," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, Nov. 2000, doi: 10.1109 / 34.888718. Solve for the camera's intrinsic parameter matrix.

[0078] For external parameters, several spatial points in a graph are known P i The three-dimensional coordinates of P i = ( X i , Y i , Z i ) and its corresponding pixel coordinates x i = ( u i , v i ), find R and T;

[0079] The three-dimensional coordinates here are artificially defined, with the origin of the world coordinate system as the center of the experimental site, due east as the positive direction of the first axis, due north as the second positive direction, and height as the third positive direction. The specific values are measured by the total station. To obtain R and T, first construct an augmented matrix A = [R|T] According to the projection principle,

[0080]

[0081] Eliminate s using the last line to get

[0082]

[0083] make

[0084] 、 、 、

[0085] get

[0086]

[0087]

[0088] Assuming there are N (N>6) characteristic points, we can list the equations

[0089]

[0090] Solving the above system of equations yields , which is the specific value of the augmented matrix A = [R|T]. The external parameters describe the position and angle of a camera relative to the origin of the world coordinate system.

[0091] 3. 3D space coordinate reconstruction

[0092] After the camera's internal and external parameters are calibrated, the transformation relationship from the camera coordinate system to the world coordinate system is obtained. Given the pixel coordinates of the target at multiple angles ( , ), ( , ), by establishing an optical axis model between the pixel point and the camera position:

[0093] For the target to be reconstructed into three-dimensional coordinates, according to its pixel coordinates ( , ), ( , ), its real coordinates ( , , ) can be used as an optical axis convergence model, by solving multiple optical axis rays simultaneously:

[0094] (3)

[0095] (4)

[0096] (5)

[0097] (6)

[0098] Finally, the least squares method is used to calculate the optimal solution of the convergence intersection to achieve the reconstruction of the xx spatial coordinate points under multiple fields of view.

[0099] 4. Low-perception-precision warhead based on optical flow tracking and multi-target motion patterns

[0100] like Figure 3 and Figure 4 As shown, first, inter-frame difference processing is performed on the striking process image sequence captured by the camera to extract all feature points to be tracked in adjacent image frames.

[0101] The Lucas–Kanade optical flow method is used to track optical flow points. According to Lucas B and Kanade T. An Iterative Image Registration Technique with an Application to Stereo Vision. Proc. Of 7th International Joint Conference on Artificial Intelligence (IJCAI), pp.674-679, the optical flow of each frame of the video is calculated.

[0102]

[0103] Represents two frames of images in the video. Describes the position change relationship of N corner points at time t. Since the warhead usually moves downward quickly, we can filter out the corner points with large position changes and downward movement to obtain the pixel coordinates of the center point of the missile warhead ( , ).

[0104] By solving the above equation using the least squares method, we can get the optical flow at this point. x and y The speed in the axis direction, i.e. the pixel point ( x, y ). Since interfering targets, such as clouds and smoke, whose spatiotemporal characteristics are constantly changing, move slowly, while warheads move quickly, a criterion for distinguishing between interfering targets and warheads is established based on this, preliminarily suppressing misdetected interfering targets.

[0105] 5. Warhead detection result correction and missile-target rendezvous parameter measurement based on the joint spatiotemporal characteristics of strongly correlated targets

[0106] like Figure 5 As shown in the figure, considering that the above preliminary detection results still contain a large number of interfering targets with similar motion patterns to the warhead (such as rapidly shaking streamers in strong winds, windblown sand, and target debris), it is necessary to further correct the detection results based on the spatiotemporal structure characteristics of the warhead. Specifically, the warhead and the explosion flame show a strong correlation in terms of temporal characteristics. This temporal characteristics can be used to filter out a large number of interfering targets generated after the flame explosion (such as explosion fragments and target fragments).

[0107] In addition, there are many characteristics of the warhead in terms of spatial distribution: (a) The optical flow point of the warhead always appears above the explosion flame. and are the coordinates of a certain optical flow point and the coordinates of the center point of the bottom edge of the flame, that is, they should satisfy ; (b) For the center point of the bottom edge of the flame Pointing to the warhead optical flow point A vector of y Mapping vectors on axes Instead of x Mapping vectors on axes Angle between ( Always greater than a certain threshold H (It varies depending on the physical characteristics and mission of different warheads).

[0108] The spatial distribution characteristics described above can be used to suppress interference targets that appear before the flame explosion and have similar motion patterns to the warhead, and further enhance the algorithm's detection performance based on the preliminary measurements above. Combined with the warhead detected, the optical flow information at different times can be processed to obtain the correct warhead trajectory. The measurement of projectile-target intersection parameters can be directly obtained based on the warhead trajectory combined with simple geometric operations. The warhead landing point is directly obtained by taking the coordinates of the end point of the trajectory, and the falling speed is calculated based on the timing and trajectory position information of the warhead during its fall. The landing posture can be determined by combining the warhead trajectory, landing point, and falling speed.

Claims

1. A method for detecting warheads in the falling area based on the joint spatiotemporal characteristics of strongly correlated targets, characterized in that: The steps include: Step 1: Arrange the camera; Use the two cameras to observe the target area at a set angle to ensure that the target area can be observed by both cameras; Step 2: Camera parameter calibration; Step 2-1: Given a checkerboard calibration plate, detect the pixel coordinates of each corner point on the calibration plate image Assuming that the world coordinate system is fixed on the chessboard, the physical coordinates of any point on the chessboard are Since the size of each grid on the calibration plate is known, the physical coordinates of each corner point in the world coordinate system are calculated using the Harris corner detection algorithm based on the grayscale changes of the black and white grid edges. ; Step 2-2: Solve the camera's internal and external parameters; According to the pixel coordinates of each corner point , the physical coordinates of each corner point in the world coordinate system , solve the transformation relationship between pixel and world coordinate system, obtain the camera's intrinsic parameter matrix and extrinsic parameter matrix to complete the camera calibration; the transformation relationship between pixel and world coordinate system is: (1) in, is the internal parameter matrix, is the external parameter matrix; represents the image distance, It represents the angle between the horizontal and vertical edges of the photographic plate. Respectively represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system, Indicates the physical length of a pixel on the camera's photosensitive plate in both directions. represents the rotation matrix, represents the translation vector; Solve the camera's intrinsic parameter matrix based on Zhang Zhengyou's calibration; For external parameters, several spatial points in a graph are known P i The three-dimensional coordinates of P i = ( X i , Y i , Z i ) and its corresponding pixel coordinates x i = ( u i , v i ), find R and T; The origin of the world coordinate system is the center of the experimental site, with due east as the positive direction of the first axis, due north as the second positive direction, and height as the third positive direction. The specific values are measured by the total station. R and T , first construct an augmented matrix A = [R|T], according to the projection principle: Eliminate s using the last line to get make 、 、 、 get Assuming there are N characteristic points, list the equations: Solving the above equations we can get , which is the specific value of the augmented matrix A = [R|T]; the external parameters describe the position and angle of the camera relative to the origin of the world coordinate system; Step 3: 3D space coordinate reconstruction; Given the pixel coordinates of the target at multiple angles ( , )、( , ), establish an optical axis convergence model between the pixel point and the camera position; According to the pixel coordinates of the target ( , )、( , ), the optical axis convergence model is used for the three-dimensional coordinates (X, Y, Z), and multiple optical axis rays are solved simultaneously: First, according to the camera model Formula (1) is simplified to: （2） Where Q is a 3*4 matrix, and Equation (2) can be written as two equations: (3) (4) in, Represents the i-th row and j-th column element of the first camera matrix; Similarly, for another camera, two equations are obtained: (5) (6) Finally, the least squares method is used to obtain the three-dimensional coordinates ( , , ); Step 4: Low-perception-precision warhead detection based on optical flow tracking and multi-target motion patterns; Step 4-1: Perform inter-frame difference processing on the warhead strike process image sequence captured by the camera to extract all feature points to be tracked in adjacent image frames; Step 4-2: Use the Lucas–Kanade optical flow method to track optical flow points and calculate the optical flow of each frame of the video : in, Represents two frames of images in the video. Described t time N The position change relationship of each corner point; filter out the corner points whose position change exceeds the set value and moves downward to obtain the pixel coordinates of the warhead center point ( , ); Step 4-3: After obtaining the pixel coordinates of the warhead center point in different cameras, substitute them into the formula in step 3 to obtain its three-dimensional coordinates ( , , ).

Images