Method and system for detecting moving targets in multi-temporal images based on epipolar constraints

Abstract

The present invention provides a method and system for detecting moving targets in multi-phase images based on epipolar constraints. First, the ASIFT algorithm is used to detect image feature points, and the RANSAC algorithm is used to eliminate mismatched points. Epipolar constraints are introduced to calculate the basic matrix. Secondly, points are calibrated on the region of interest in the original image, and the epipolar lines corresponding to the calibrated points in the region of interest are obtained in the target image using the basic matrix. Finally, a robust and efficient template matching method based on deep convolution features is used to calibrate the template image area to be matched in the original image, and template matching is performed in the target image. By comparing the positional relationship between the epipolar lines and the rectangular box of the template matching result in the target image, it is determined whether the target has moved. The present invention can effectively detect moving targets with high accuracy, has the characteristics of simple method, obvious effect, and easy engineering implementation, and is of great significance in actual production and life.

Description

Technical Field

[0001] The present invention relates to the technical field of moving target detection, and in particular to a method and system for detecting moving targets in multi-temporal images based on epipolar constraints. Background Art

[0002] Computer vision technology has a wide range of applications in target detection, target recognition and tracking, SLAM (Simultaneous Localization and Mapping), and three-dimensional environment construction. Among them, moving object detection is a core technology in computer vision, widely used in both military and civilian fields. In the military, it can be combined with radar technology to analyze enemy targets. In civilian applications, it can be used for autonomous driving, vehicle monitoring, and visual analysis of human motion. Current moving object detection techniques typically process video images, extracting moving objects from video or image sequences, and obtaining their characteristic information, such as color, shape, and outline. Video-based moving object detection algorithms essentially identify differences in continuous image sequences and detect and extract them. Depending on whether the camera is in motion, the analysis can be categorized into two main types: those with and without camera motion. Common methods for the non-motion case include frame-to-frame difference, background difference, optical flow, and motion energy, which are relatively mature. For the motion case, methods based on motion compensation, grid voxel-based methods, geometric constraints, and deep learning are available. However, these moving object detection algorithms rely on video frame processing and require continuous detection. However, in some scenarios in actual production and life, such as disaster monitoring, there is no need to continuously monitor video sequences. Instead, it is necessary to detect specific areas at certain intervals and sample the image information, and find the moving targets of interest by comparing the two images. At this time, the traditional video-based motion target detection algorithm no longer meets the needs of actual tasks. Summary of the Invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for detecting moving targets in multi-temporal images based on epipolar constraints.

[0004] According to one aspect of the present invention, a method for detecting moving objects in multi-temporal images based on epipolar constraints is provided, comprising:

[0005] Perform ASIFT feature point detection and matching on the acquired first image and the second image to obtain matching feature point pairs;

[0006] Eliminating mismatches on the matching feature point pairs to obtain optimized feature point pairs;

[0007] Calculating a fundamental matrix using the optimized feature point pairs;

[0008] Selecting a random point in the region of interest in the first image, outputting the pixel coordinates of the random point, and calculating an epipolar line corresponding to a point in the second image corresponding to the random point based on the value of the fundamental matrix and the pixel coordinates;

[0009] Cutting out a region of interest from the first image as a template image for representing the target to be detected; using the template image to search for an image region in the second image that is most similar to the template image, and marking the region in the second image with a labeling box;

[0010] The positional relationship between the epipolar lines and the annotation box in the second image is compared. If the epipolar lines above the set threshold intersect with the annotation box, it is determined that the target to be detected has not moved. Otherwise, it is determined that the target to be detected has moved.

[0011] Optionally, performing ASIFT feature point detection and matching on the first image and the second image, using a feature point-based bidirectional matching method, includes:

[0012] Extracting scale-invariant feature points of the first image and the second image respectively using the ASIFT algorithm;

[0013] The first image is used as the reference image and the second image is used as the image to be matched; the feature points in the first image are obtained, and the most similar feature points are found in the second image to match them, forming a feature point pair; this step is repeated to complete the matching of other feature points, and a matching feature point pair Ω1 is obtained;

[0014] The second image is used as the reference image and the first image is used as the image to be matched; the feature points in the second image are obtained, and the most similar feature points are found in the first image to match them, forming a feature point pair; this step is repeated to complete the matching of other feature points, and a matching feature point pair Ω2 is obtained;

[0015] The common matching point pairs belonging to the matching feature point pair Ω1 and the matching feature point pair Ω2 are counted and used as the final matching feature point pair Ω.

[0016] Optionally, mismatching is eliminated from the matching feature point pairs using a RANSAC algorithm, including:

[0017] Randomly selecting a number of matching feature point pairs from the first image and the second image to form a matching feature point pair dataset, and using the matching feature point pairs to solve the homography transformation matrix H, including:

[0018] Assume that the coordinates of the feature points in the first image are (x, y), the coordinates of the feature points in the second image are (x1, y1), and the homography matrix H is The corresponding solution formula is:

[0019]

[0020] Using the matching feature point pairs, the corresponding equations are listed according to the solution formula, and the unknown parameters h in the homography matrix H are solved according to the equations. 11 、h 12 、h 13 、h 21 、h 22 、h 23 、h 31 、h 32 ;

[0021] Calculate the projection error between all data in the matching feature point pair data set and the homography matrix H. If the error is less than a threshold, add the inlier set I;

[0022] If the number of elements in the current interior point set I is greater than the number of elements in the optimal interior point set I_best, then I_best is updated to I, and the number of iterations k is updated at the same time; wherein the initial value of the optimal interior point set I_best is 0; the calculation formula for the number of iterations k is:

[0023]

[0024] Where p is the confidence level; w is the ratio of internal points; m is the minimum number of matching feature point pairs required to calculate the homography matrix H;

[0025] If the actual number of iterations is greater than k, the projection error calculation is exited; otherwise, the number of iterations is increased by 1, and the steps of projection error calculation are repeated;

[0026] Based on the final obtained inlier point set I, the remaining matching feature point pairs that do not belong to the inlier point set are eliminated to achieve false matching elimination.

[0027] Optionally, the p value is 0.995.

[0028] Optionally, m is 4.

[0029] Optionally, the method further includes: obtaining the number of the optimized feature point pairs, and when the number is greater than or equal to a set quantity threshold, performing a subsequent step of calculating a basic matrix.

[0030] Optionally, calculating a basic matrix using the optimized feature point pairs includes:

[0031] Determine that the spatial geometric constraint relationship satisfied between the first image and the second image and used to represent the correspondence relationship between the target point to be detected and the epipolar line is:

[0032]

[0033] Among them, p l and p r are the projection points of the target point p to be detected under the first camera and the second camera respectively; F is the basic matrix used to represent the relationship between the epipolar line and the point in the constraint relationship, and the basic matrix F is expressed as:

[0034]

[0035] Among them, M l and M r are the internal parameter matrices of the first camera and the second camera respectively; E is the intrinsic matrix;

[0036] Randomly selecting four pairs of feature point pairs from the optimized feature point pairs to calculate the basic matrix F to obtain an initial value of the basic matrix F;

[0037] The basic matrix F is iteratively calculated using other feature point pairs in the optimized feature point pair, and the value of the basic matrix F is refined, ultimately reflecting the value of the basic matrix F of the optimal spatial geometric constraint relationship between the first image and the second image.

[0038] Optionally, selecting a random point in the region of interest in the first image, outputting pixel coordinates of the random point, and calculating an epipolar line corresponding to a point corresponding to the random point in the second image based on a value of the fundamental matrix and the pixel coordinates, includes:

[0039] Selecting a random portion of pixel points in the region of interest in the first image, and outputting pixel coordinate values of the pixel points;

[0040] The pixel coordinate values of all selected pixel points are stored in the form of a matrix to obtain the pixel coordinate value matrix;

[0041] The coordinate value matrix of the pixel points is called, and the spatial geometric constraint relationship between the first image and the second image is used to solve the epipolar lines corresponding to the pixel points in the second image.

[0042] Optionally, using the template image to search in the second image for an image region that is most similar to the template image, a robust template matching method using scale-adaptive deep convolution features is employed, including:

[0043] Cutting out a region of interest from the first image as a target to be detected and saving it as a template image img;

[0044] The template image img and the second image are passed through a pre-trained VGG-Net model to extract scale-adaptive deep convolution feature vectors, and the distance between the features of the template image img and the corresponding features of the second image is measured using the NCC algorithm. Based on a set similarity threshold, the image area of the second image whose similarity measure with the template image img is within the set similarity threshold is detected, which is the image area in the second image that is most similar to the template image;

[0045] Output pixel coordinate values of the upper left corner and the lower right corner of the image area, and mark the second image.

[0046] Optionally, the threshold value of the polar lines is set to 0.95, that is, when more than 95% of the polar lines intersect with the annotation box, it is determined that the target to be detected has not moved; otherwise, it is determined that the target to be detected has moved.

[0047] According to another aspect of the present invention, a system for detecting moving objects in multi-temporal images based on epipolar constraints is provided, comprising:

[0048] An ASIFT algorithm module is used to perform ASIFT feature point detection and matching on the acquired first image and the second image to obtain matching feature point pairs;

[0049] A feature point optimization module, which is used to eliminate mismatches of the matching feature point pairs to obtain optimized feature point pairs;

[0050] A basic matrix calculation module, which is used to calculate the basic matrix using the optimized feature point pairs;

[0051] an epipolar line calculation module, configured to select a random point in a region of interest in the first image, output pixel coordinates of the random point, and calculate an epipolar line corresponding to a point in the second image corresponding to the random point based on a value of the fundamental matrix and the pixel coordinates;

[0052] a target region annotation module, configured to intercept a region of interest in the first image as a template image representing the target to be detected; use the template image to search for an image region in the second image that is most similar to the template image, and annotate the region in the second image using a annotation box;

[0053] The movement detection module is used to compare the positional relationship between the epipolar lines and the annotation box in the second image. If the epipolar lines above the set threshold intersect with the annotation box, it is determined that the target to be detected has not moved; otherwise, it is determined that the target to be detected has moved.

[0054] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above-described methods or to execute the above-described system when executing the program.

[0055] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute any of the above-mentioned methods, or run the above-mentioned system.

[0056] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0057] The method and system for detecting moving targets in multi-phase images based on epipolar constraints provided by the present invention meet the needs of detecting moving targets in image pairs with long intervals and taken from arbitrarily different angles, and solve the problem of needing to compare consecutive frames in traditional video-based image motion target detection technology. It has good flexibility and does not require data sets for training, thus avoiding the problem of large workload of deep learning. It has wide application value in actual production and life.

[0058] The present invention provides a method and system for detecting moving targets in multi-temporal images based on epipolar constraints. Epipolar constraints are introduced and combined with image features. By manually selecting the target object to be detected, moving target detection is achieved in two images taken at different times and angles.

[0059] The multi-temporal image moving target detection method and system based on epipolar constraints provided by the present invention can effectively detect moving targets in videos with high accuracy, solving the problem of traditional video-based motion target detection technology requiring real-time monitoring of video or image sequences, and is of great significance in actual production and life. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0061] Figure 1 FIG. 4 is a flowchart of a method for detecting moving objects in multi-temporal images based on epipolar constraints in one embodiment of the present invention.

[0062] Figure 2 Schematic diagram of the components of a multi-temporal image moving object detection system based on epipolar constraints in one embodiment of the present invention.

[0063] Figure 3Schematic diagram of longitude and latitude angles in the ASIFT algorithm in a preferred embodiment of the present invention.

[0064] Figure 4 Schematic diagram of latitude and longitude simulation sampling in the ASIFT algorithm in a preferred embodiment of the present invention; wherein (a) is a perspective view along the y-axis direction, and (b) is a perspective view along the z-axis direction.

[0065] Figure 5 Schematic diagram of epipolar constraints in a preferred embodiment of the present invention.

[0066] Figures 6 to 10 They are schematic diagrams of image pairs taken from different perspectives in a specific application example of the present invention, wherein (a) is the first image img1 and (b) is the second image img2.

[0067] Figure 11 Schematic diagram of the success rate curves and AUC values of the target detection method provided by an embodiment of the present invention and four other methods.

[0068] Figures 12 to 16 These are the second image epipolar line drawing and matching result images and the target movement judgment results from image pairs taken from different perspectives, where (a) is the second image epipolar line drawing and matching result image, and (b) is the target movement judgment result. DETAILED DESCRIPTION

[0069] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention.

[0070] In order to meet the needs of detecting moving targets in images with long intervals and taken from any different angles, and to solve the problem of requiring comparison of consecutive frames in traditional video-based image motion target detection technology, an embodiment of the present invention provides a multi-phase image moving target detection method based on epipolar constraints. This method has good flexibility and does not require data sets for training, thus avoiding the problem of large workload of deep learning. It has wide application value in actual production and life.

[0071] like Figure 1 As shown, the method for detecting moving objects in multi-temporal images based on epipolar constraints provided by this embodiment may include:

[0072] S1, performing ASIFT feature point detection and matching on the acquired first image and the second image to obtain a pair of matching feature points. In a preferred embodiment, performing ASIFT feature point detection and matching on the first image and the second image using a feature point-based bidirectional matching method may include:

[0073] S11, using the ASIFT algorithm to extract scale-invariant feature points of the first image and the second image respectively;

[0074] S12, taking the first image as the reference image and the second image as the image to be matched; obtaining the feature points in the first image, and searching for the most similar feature points in the second image to match them, forming a feature point pair; repeating this step to complete the matching of other feature points, and obtaining the matching feature point pair Ω1;

[0075] S13, taking the second image as the reference image and the first image as the image to be matched; obtaining feature points in the second image, and searching for the most similar feature points in the first image to match them, forming a feature point pair; repeating this step to complete the matching of other feature points, and obtaining a matching feature point pair Ω2;

[0076] S14: Count the common matching point pairs belonging to the matching feature point pair Ω1 and the matching feature point pair Ω2, and use them as the final matching feature point pair Ω.

[0077] S2, eliminating false matches from the matched feature point pairs to obtain optimized feature point pairs. In a preferred embodiment, this step may further include: obtaining the number of optimized feature point pairs obtained, and when the number is greater than or equal to a set threshold, performing a subsequent step of calculating the basic matrix F. In a preferred embodiment, eliminating false matches from the matched feature point pairs using the RANSAC algorithm may include:

[0078] S21, randomly select a number of matching feature point pairs from the first image and the second image to form a matching feature point pair dataset, and use these feature matching point pairs to solve the homography transformation matrix H, specifically:

[0079] Suppose the coordinates of the feature points in the first image are (x, y), the coordinates of the feature points in the second image are (x1, y1), and the homography matrix H is The corresponding solution formula is:

[0080]

[0081] Using the feature matching point pairs, the corresponding equations are listed according to the solution formula, and the unknown parameters h in the homography matrix H can be solved according to the equations. 11 、h 12 、h 13 、h21 、h 22 、h 23 、h 31 、h 32 ;

[0082] S22, calculate the projection error between the matching feature points and all data in the dataset and the homography matrix H. If the error is less than a threshold, add the inliers to the set I;

[0083] S23, if the number of elements in the current interior point set I is greater than the number of elements in the optimal interior point set I_best, then I_best is updated to I, and the number of iterations k is updated at the same time; wherein, the initial value of the optimal interior point set I_best is 0; the calculation formula for the number of iterations k is:

[0084]

[0085] Where p is the confidence level, preferably 0.995; w is the ratio of inliers; m is the minimum number of matching feature point pairs required to calculate the homography matrix H, preferably 4;

[0086] S24, if the actual number of iterations is greater than k, exit; otherwise, increase the number of iterations by 1 and repeat steps S22 and S23;

[0087] S25, based on the finally obtained inlier point set I, the remaining matching feature point pairs that do not belong to the inlier point set are eliminated to achieve mismatch elimination.

[0088] S3, using the optimized feature point pairs to calculate the basic matrix F. In a preferred embodiment, using the optimized feature point pairs to calculate the basic matrix F may include:

[0089] S31, determining that the spatial geometric constraint relationship satisfied between the first image and the second image for representing the correspondence relationship between the target point to be detected and the epipolar line is:

[0090]

[0091] Among them, p l and p r are the projection points of the target point p to be detected under the first camera and the second camera respectively; F is the basic matrix used to represent the relationship between the polar line and the point in the constraint relationship. The basic matrix F is expressed as:

[0092]

[0093] Among them, M l and M r are the internal parameter matrices of the first camera and the second camera respectively; E is the intrinsic matrix;

[0094] S32, randomly selecting four pairs of feature point pairs from the optimized feature point pairs to calculate the basic matrix F, and obtaining an initial value of the basic matrix F;

[0095] S33, using other feature point pairs in the optimized feature point pair to iteratively calculate the basic matrix F, and refine the value of the basic matrix F, and finally reflect the value of the basic matrix F of the optimal spatial geometric constraint relationship between the first image and the second image.

[0096] S4, selecting a random point in the region of interest in the first image, outputting the pixel coordinates of the random point, and calculating the epipolar line corresponding to the point corresponding to the random point in the second image based on the value of the fundamental matrix F and the pixel coordinates. In a preferred embodiment, selecting a random point in the region of interest in the first image, outputting the pixel coordinates of the random point, and calculating the epipolar line corresponding to the point corresponding to the random point in the second image based on the value of the fundamental matrix F and the pixel coordinates may include:

[0097] S41, selecting a random portion of pixel points in the region of interest in the first image, and outputting pixel coordinate values of the pixel points;

[0098] S42, storing the pixel coordinate values of all selected pixel points in the form of a matrix to obtain a pixel coordinate value matrix;

[0099] S43, calling the coordinate value matrix of the pixel points, and using the spatial geometric constraint relationship between the first image and the second image, solving the epipolar lines corresponding to the pixel points in the second image.

[0100] S5: Extracting a region of interest from the first image as a template image representing the target to be detected; using the template image, searching for an image region in the second image that is most similar to the template image, and marking the region in the second image with a labeling box. In a preferred embodiment, using the template image, searching for an image region in the second image that is most similar to the template image, using a robust template matching method with scale-adaptive deep convolution features, may include:

[0101] S51, intercepting a region of interest in the first image as a target to be detected, and saving it as a template image img;

[0102] S52, the template image img and the second image are passed through a pre-trained VGG-Net model to extract scale-adaptive deep convolution feature vectors, and the NCC algorithm is used to measure the distance between the features of the template image img and the corresponding features of the second image. According to a set similarity threshold, the image area of the second image whose similarity measure with the template image img is within the set similarity threshold is detected, which is the image area in the second image that is most similar to the template image;

[0103] S53: Output pixel coordinate values of the upper left corner and the lower right corner of the image area, and mark the second image.

[0104] S6, comparing the positional relationship between the epipolar lines in the second image and the labeled frame. If all epipolar lines above a set threshold intersect the labeled frame, it is determined that the target to be detected has not moved. Otherwise, it is determined that the target to be detected has moved. In a preferred embodiment, the set threshold for the epipolar lines can be set to 0.95, that is, if more than 95% of the epipolar lines intersect the labeled frame, it is determined that the target to be detected has not moved. Otherwise, it is determined that the target to be detected has moved.

[0105] In a preferred embodiment, the target detection method provided in this embodiment may further include:

[0106] S0, acquiring a multi-phase image, where the multi-phase image includes a first image and a second image.

[0107] The technical solutions provided by the above embodiments of the present invention are further described below in conjunction with the accompanying drawings and preferred embodiments.

[0108] A preferred embodiment of the present invention provides a method for detecting moving objects in multi-temporal images based on epipolar constraints, the method comprising the following steps:

[0109] Step 1: Use the feature point-based bidirectional matching method to implement the ASIFT algorithm to detect and match the feature points of the first image and the second image, and obtain matching feature point pairs. This includes the following steps:

[0110] Step (1): extract ASIFT feature points of the first image and the second image.

[0111] Step (2): Use the first image as the reference image and the second image as the image to be matched. Extract the feature points in the first image and find the most similar feature points in the second image to match them to form a feature point pair. Then repeat this step to complete the matching of other feature points, and output the number of successful feature point matches and record the successfully matched feature point pairs Ω1.

[0112] Step (3), using the second image as the reference image and the first image as the image to be matched, completing feature point matching with the second image as the reference image using the method in step (2), and recording the feature point pairs Ω2 that have successfully completed the matching;

[0113] Step (4): Count the common matching point pairs belonging to Ω1 and Ω2 and use them as the final matching feature point pair Ω.

[0114] The implementation steps of the ASIFT algorithm are as follows:

[0115] (1) Constructing an affine transformation model

[0116] The ASIFT algorithm first uses horizontal and vertical angles to simulate all possible affine distortions and transform the image. According to the affine transformation model, there are six unknown parameters in the model, two of which represent translation and can be simulated using mathematical methods. The remaining four unknown parameters a1, a2, a3, and a4 can be expressed by the following formula:

[0117]

[0118] Where k represents the focal length of the camera, Represents the rotation of the image to be matched relative to the reference image, φ represents the observation angle, φ∈[0,π], t and 1 represent the two eigenvalues of the diagonal matrix respectively; Figure 3 As shown in the figure, an affine camera model of an aerial image is shown, where k represents the scaling factor and t = 1 / cosθ.

[0119] (2) Sampling longitude and latitude. Longitude Φ and latitude θ are two key parameters that determine the deformation of the camera optical axis. Excessive tilt Therefore, the latitude can be sampled according to the geometric progression t=1, a1, a2...an. Since it is necessary to meet the sampling accuracy and sparsity at the same time, the sampling interval is set to Take samples. Figure 4 As shown, the figure shows the sampling interval The sampling interval perspective under and observation angle φ. Figure 4 The points in represent sampling points. Figure 4 (a) is a perspective view along the y-axis. Figure 4 (b) is a perspective view along the z-axis.

[0120] (3) Sampling the image

[0121] The ASIFT algorithm processes images using a dual-resolution model. This approach improves the algorithm's computational efficiency, stability, and robustness, enabling it to better solve image matching problems at different resolutions. When encountering images of different resolutions, the ASIFT algorithm rescales them to the lower resolution.

[0122] Resolution reduction involves subsampling the query image u and the search image v using the KxK sampling factor to obtain u' = SK*GK*u and v' = SK*GK*v. SK is the subsampling factor, and GK is the anti-aliasing Gaussian discrete filter used to perform anti-aliasing and smoothing on the image, minimizing image distortion during the downsampling process.

[0123] (4) Perform SIFT feature point detection and matching on the image to obtain matching feature point pairs.

[0124] In step 2, the RANSAC algorithm is used to eliminate false matches of the matching feature point pairs in step 1. At the same time, the number of matching feature point pairs after false matches are eliminated can be displayed. If the number is less than the set threshold, subsequent processing cannot be completed. If there are enough feature points, subsequent processing can continue.

[0125] In stereo vision, the RANSAC algorithm uses the positional constraints that pixels in images taken from different perspectives should satisfy when the object's position remains unchanged. This algorithm removes feature matching points from objects that have moved and excludes them from subsequent calculations. This prevents errors in the solution of the fundamental matrix F and improves the accuracy of subsequent epipolar solutions. The RANSAC algorithm's constraint is the homography matrix H between the two images, and mismatched points are removed based on H.

[0126] The specific steps of the RANSAC algorithm to remove mismatched points in two images are as follows:

[0127] (1) Randomly select four pairs of matching feature points from the first image and the second image to form a matching feature point pair data set, and use these four pairs of matching feature points to solve the homography transformation matrix (short for homography matrix) H. Specifically, let the pixel coordinates in the first image be (x, y), and the pixel coordinates in the second image be (x1, y1), and the homography matrix H is The corresponding solution formula is:

[0128]

[0129] Using four pairs of matching feature points, the corresponding four sets of equations are listed according to the solution formula. Based on these four sets of equations, the eight unknown parameters h in the homography matrix can be solved. 11 、h 12 、h 13 、h 21 、h 22 、h 23 、h 31 、h 32 ;

[0130] (2) Calculate the projection error between all the data in the matching feature point dataset and the homography matrix H. If the error is less than the threshold, add the inlier set I;

[0131] (3) If the number of elements in the current interior point set I is greater than the number of elements in the optimal interior point set I_best, then I_best is updated to I, and the number of iterations k is updated at the same time; the initial value of the optimal interior point set I_best is 0 and is continuously updated in the subsequent steps; the calculation formula for the number of iterations k is:

[0132]

[0133] Where p is the confidence level, which is generally set to 0.995; w is the ratio of inliers; m is the minimum number of matching feature point pairs required to calculate the homography matrix H = 4;

[0134] (4) If the actual number of iterations is greater than k, exit; otherwise, increase the number of iterations by 1 and repeat steps (2) and (3) above;

[0135] (5) Based on the inlier point set I finally obtained in the above steps, the remaining matching feature point pairs that do not belong to the inlier point set are eliminated to achieve the elimination of mismatched points.

[0136] Step 3: Calculate the basic matrix F using the feature point pairs after mismatching has been eliminated.

[0137] As is known, in binocular vision, epipolar geometry refers to the spatial geometric constraints that should be satisfied between two images of the same object taken at different locations. This constraint reflects the correspondence between points and lines. The line corresponding to the point is called the epipolar line. At the same time, the relationship between the epipolar line and the point is represented by the fundamental matrix F. Figure 5 As shown, it means from O l and O r Observe and photograph the same location. l , O r Represents the position of the left and right cameras; I l and I r Represents the imaging plane of the left and right cameras; e l Indicates O r The projection point on the left imaging plane, e r Indicates O l The projection point on the right imaging plane, e l 、e r It is called the extreme point; L l and L r is the polar line passing through the extreme point; p l and p r is the observed point p on camera O l and camera O r The projection point below.

[0138] (1) The formula for determining the spatial geometric constraint relationship (epipolar constraint) is as follows:

[0139]

[0140] Where F represents the basic matrix, and its calculation formula is:

[0141]

[0142] Among them, M l and M r Represents camera O l and camera O r The internal parameter matrix of ; E represents the eigenvalue matrix.

[0143] (2) Calculate the basic matrix F

[0144] The solution to the basic matrix F is as follows: Since F is a 3x3 matrix with eight unknown parameters, including the intrinsic and extrinsic parameters of the camera, at least eight linear equations are required to solve. Corresponding to the point position information, since one set of point pairs can list two equations, at least four sets of point pairs are required. In order to improve the accuracy of solving the basic matrix F, this algorithm uses the RANSAC algorithm to eliminate mismatched point pairs and obtain optimized feature point pairs. It then performs continuous iterative calculations, continuously removing inliers with large distances from the epipolar lines, and trying to increase the number of inliers that meet the constraints of the basic matrix to improve the accuracy of the basic matrix; until the sum of the distances of all remaining inliers to the epipolar lines is within a certain threshold range, the final basic matrix F is calculated using all the inlier points.

[0145] First, four pairs of optimized feature point pairs are randomly selected from the first and second images to calculate the basic matrix F to obtain the initial value of the basic matrix F. Then, iterative calculations are performed on other optimized matching point pairs in the first and second images to gradually refine the value of the basic matrix F, and finally a matrix value that most accurately reflects the pixel position constraint relationship in the image pair is obtained.

[0146] Step 4: Use the human-computer interactive mouse to select a random point in the area of interest in the first image, output the pixel coordinates of the point, and calculate the epipolar line corresponding to the point in the second image 2 based on the value of the basic matrix F in step 3 and the pixel coordinates of the point in the output first image, and mark it in the figure.

[0147] The value of the fundamental matrix F is calculated in step 3. When the pixel coordinates of a specified point in the first image are known, the positional relationship restored to the second image is expressed as a straight line using the fundamental matrix F, that is, the correspondence between the point and the line, that is, the positional relationship between the pixel point and the epipolar line. The specific steps for drawing the epipolar line of a specified point are as follows:

[0148] (1) Use the mouse to select a random pixel point in the area you want to detect in the first image, and output the pixel coordinate value [x i ,yi ];

[0149] (2) storing the coordinate values of all selected pixel points in the form of a matrix;

[0150] (3) Call the pixel coordinate matrix of the midpoint in step (2) and use the formula The epipolar lines corresponding to the pixel points in the second image are solved and marked in the second image.

[0151] Step 5: Use a rectangular frame to capture the region of interest in the first image and save it as a template image to reflect the target to be detected; use the template image to find the most similar image area in the second image using the robust template matching method of scale-adaptive deep convolution features, and mark it with a rectangular frame in the second image.

[0152] The specific steps of implementing the template matching method based on scale-adaptive deep convolution feature extraction are as follows:

[0153] (1) Scale-adaptive deep convolution feature extraction

[0154] In the scale-adaptive feature extraction method, VGG-Net is used to extract feature vectors from the template and input image. Unlike common CNN-based methods, this method does not scale the template or image to a specific size, such as 224×224. Instead, it adaptively identifies the target layer of the VGG-Net and extracts feature vectors from the target layer. Each layer of CNNs has a receptive field of Rfl×Rfl, where the width of the receptive field of the lth layer Rfl is defined as:

[0155]

[0156] For simplicity, the number of layers l is set depending on their order, e.g., conv1-1 is 1 layer, conv1-2 is 2 layers, and pool-1 is 3 layers. l Represents the size of the l-th layer filter, s i Represents the step size of the i-th layer. If the template size is smaller than the receptive field size of the target layer, then the layer processes a meaningless template zero-filled outer area. Therefore, the size of the target layer receptive field is limited to be smaller than or equal to the size of the template. Therefore, the target layer l* layer is expressed as:

[0157] l*=max(lk,1)stRfl≤min(w,h)

[0158] To meet the conditions in the formula, k represents a constant coefficient that should be greater than or equal to 0. In this algorithm, k is uniformly set to 3 to facilitate consistency in subsequent operations. The receptive field size of layers 1-3 is approximately half of min(w, h), where w represents the pixel width of the template image and h represents the pixel height of the template image. Because there is a pooling layer between layers 1 and 1-3. The template image T and the target image I are then input into CNNs, and the template features N and image feature map M are extracted from the target layer. Using this method, the convolution feature vector only needs to be calculated once for each template and image, which is much more efficient than searching the image using a naive sliding window method.

[0159] (2) Similarity measurement method based on NCC

[0160] The most similar patch is found using the NCC (normalized cross correlation) between M and N. First, by:

[0161]

[0162] Calculate the NCC between M and N: Represents the feature map of the area to be matched in the image, with a width of w f and height h f The size is equal to N. Then the position (i*, j*) with the maximum NCC value is calculated using this formula.

[0163] (3) Location optimization

[0164] Since this method is mapped on the convolutional feature layer, the step size of the sliding window is assumed to be the size of the target layer. Therefore, in order to obtain higher accuracy, the position of the feature domain is optimized. First, the initial box position is set by the position (i*, j*) with the largest NCC value. in is the upper left corner position of the initial box, is the lower right corner of the initial box. Then, we refine the image by taking a patch near the maximum position (i*, j*) on the NCC. The detailed steps are: if we want to get the refined x1, we use a 3x4 patch, whose weight is expressed as:

[0165]

[0166] Among them, u and v represent the value ranges [-1,1] and [-2,1] respectively, corresponding to the size of 3x4 patch.

[0167] In this method, we first capture the region of interest in the first image as the target to be detected and save it as a template image img. Then, we use the pre-trained VGG-Net to extract the scale-adaptive deep convolution feature vector from the template image img and the second image. Then, we use NCC to measure the distance between the features of the template image img and the corresponding features of the second image. We specify the threshold value and detect the area where the similarity measure between the second image and the template image img is within the given threshold range. The area that meets this threshold range is designated as the successfully matched area, and the pixel coordinates of the upper left corner and lower right corner of the area are output.

[0168] Step 6: Compare the positional relationship between the epipolar lines and the rectangular frame in the second image. If more than 95% of the epipolar lines intersect with the rectangular frame, it is determined that the target to be detected has not moved. Otherwise, it is determined that the target to be detected has moved.

[0169] Experiments have shown that the template matching algorithm provided in step five has a high matching accuracy and can output the actual position of the template image in the second image more accurately. Since the successfully matched template image in the second image reflects the actual position of the target to be detected in the second image, the epipolar lines drawn in step four reflect the theoretical position of the object in the second image if it has not moved. Therefore, the positional relationship between the epipolar lines and the rectangular frame reflects the change in the object's position. Therefore, by judging the positional relationship between the epipolar lines and the rectangular frame of the successfully matched template image in the second image, it can be determined whether the object has moved. The threshold is set to 0.95; if more than 95% of the total number of epipolar lines intersect with the rectangular frame, it is determined that the object is still in place and has not moved; otherwise, if most of the epipolar lines show a trend of separation from the positional relationship of the rectangular frame, it is determined that the object has moved.

[0170] The technical solution provided by the above embodiment of the present invention is further described below with reference to a specific application example.

[0171] This specific application example is for five image pairs of the same area taken by a mobile phone at different viewing angles, such as Figures 6 to 10 As shown in (a) and (b), the specific implementation of the above embodiment of the present invention is described as follows.

[0172] In this specific application example, the method for detecting moving objects in multi-temporal images based on epipolar constraints includes the following steps:

[0173] In step 1, the ASIFT algorithm is used to extract and match the image feature points. The number of feature points extracted and matched and the time spent in the matching stage are shown in Table 1.

[0174] Table 1 Number of feature points extracted and matched by ASIFT algorithm and matching time

[0175]

[0176] Step 2: Use the RANSAC algorithm to remove mismatches from the matching feature points. The number of inliers after removing mismatches using RANSAC is shown in Table 2.

[0177] Table 2 Number of inliers after RANSAC mismatch removal

[0178] img1-img2 matching inner points (a) 4131 (b) 40 (c) 172 (d) 7807 (e) 10180

[0179] Step 3: Calculate the basic matrix F using the feature point pairs after mismatching has been eliminated.

[0180] Step 4: Use the interactive mouse to select a random point in the region of interest in img1 and output the pixel coordinates of the point. Based on the value of the fundamental matrix F from step 3 and the pixel coordinates of the point in img1, calculate the epipolar line corresponding to the point in img2 and display it in the image using random colors.

[0181] Step five, use a rectangular frame to capture the region of interest in img1 and save it as a template image; use the template image to find the most similar image region in img2 using the robust template matching method of scale-adaptive deep convolution features, and mark it with a rectangular frame in image 2. In this example, 105 template-image pairs are first sampled using 35 videos (3 pairs per video) in the tracking dataset. The template is randomly selected, and 20 frames of the image are sampled after the template is captured. For each pair, the intersection-over-union (IoU) between the ground truth and the predicted box is measured, and then different IoU thresholds and area under the curve (AUC) are used for quantitative comparison, and the template matching method based on scale-adaptive deep convolution feature extraction adopted in this paper is compared with four methods: SSD, NCC, HM, and BBS, as shown in the following figure. Figure 11 shown.

[0182] At the same time, the IoU is used to quantitatively analyze the effect of the template matching algorithm of the scale-adaptive deep convolution feature extraction on the five images selected in this example, and compared with the other four template matching methods. The comparison results are shown in Table 3. It can be seen that the algorithm has the highest matching accuracy and has good adaptability to different types of images. It can accurately detect the position of the template in img2 and has the best effect.

[0183] Table 3 IoU comparison of each algorithm for five images

[0184]

[0185] Step 6: Compare the positional relationship between the epipolar lines and the rectangular frame in image 2. If more than 95% of the epipolar lines intersect with the rectangular frame, it is determined that the object has not moved. Otherwise, it is determined that the object has moved. Since the ASIFT algorithm and the template matching method based on scale-adaptive deep convolution feature extraction adopted in the above-mentioned embodiment of the present invention have good performance, the target detection method provided by the above-mentioned embodiment of the present invention has a high accuracy. In this example, the final output of img2 of the five image pairs and the judgment results of the program are as follows: Figures 12 to 16 As shown in (a) and (b), both correctly output the changes in the object's position. It can be seen that the content of the present invention has a good application effect and can be applied in production and life.

[0186] An embodiment of the present invention provides a system for detecting moving objects in multi-temporal images based on epipolar constraints.

[0187] like Figure 2 As shown, the multi-temporal image moving target detection system based on epipolar constraints provided by this embodiment may include:

[0188] An ASIFT algorithm module, which is used to perform ASIFT feature point detection and matching on the first image and the second image to obtain matching feature point pairs;

[0189] Feature point optimization module, which is used to eliminate mismatches of matching feature point pairs and obtain optimized feature point pairs;

[0190] A basic matrix calculation module, which is used to calculate the basic matrix F using the optimized feature point pairs;

[0191] an epipolar calculation module for selecting a random point in the region of interest in the first image, outputting the pixel coordinates of the random point, and calculating the epipolar line corresponding to the point corresponding to the random point in the second image based on the value of the fundamental matrix F and the pixel coordinates;

[0192] The target region annotation module is used to intercept the region of interest in the first image as a template image for representing the target to be detected; use the template image to find the image region most similar to the template image in the second image, and annotate it with a labeling box in the second image;

[0193] The movement detection module is used to compare the positional relationship between the epipolar lines and the annotation box in the second image. If the epipolar lines above the set threshold intersect with the annotation box, it is determined that the target to be detected has not moved; otherwise, it is determined that the target to be detected has moved.

[0194] In a preferred embodiment, the target detection system provided in this embodiment may further include:

[0195] The image acquisition module is used to acquire multi-phase images, which include a first image and a second image.

[0196] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the method can be understood as preferred examples of constructing the system, which will not be elaborated here.

[0197] An embodiment of the present invention provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the terminal can be used to execute any one of the methods described in the foregoing embodiments of the present invention, or to execute any one of the systems described in the foregoing embodiments of the present invention.

[0198] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0199] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0200] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0201] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0202] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0203] The epipolar constraint-based method and system for detecting moving targets in multi-temporal images, provided by the aforementioned embodiments of the present invention, address the problem that most current moving target detection and tracking algorithms are based on video or image sequences, requiring continuous monitoring and extraction of multiple video frames or multiple sequential images for comparison. This method and system introduces epipolar constraints and combines them with image features. By manually selecting the target object to be detected, it achieves moving target detection in two images taken at different times and angles. The method and system first detects image feature points using the ASIFT algorithm and eliminates mismatched points using the RANSAC algorithm. Combined with stereo vision, epipolar constraints are introduced to determine the fundamental matrix F between the image pairs. Next, points are calibrated in the region of interest in the original image, and the epipolar lines corresponding to the calibrated points in the region of interest are obtained in the target image using the fundamental matrix F. Finally, a robust and efficient template matching method based on deep convolutional features is used to calibrate the template image region to be matched in the original image. Template matching is then performed in the target image. The positional relationship between the epipolar lines and the template matching result rectangle in the target image is compared to determine whether the target has moved. The results show that this method and system can effectively detect moving targets with high accuracy, solving the problem of real-time monitoring of video or image sequences in video-based moving target detection algorithms. It has the characteristics of simple method, obvious effect, and easy engineering implementation, and is of great significance in actual production and life.

[0204] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered a hardware component, and the devices included therein for implementing the various functions can also be considered as structures within the hardware component; the devices for implementing the various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0205] Matters not mentioned in the above embodiments of the present invention are well known in the art.

[0206] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for detecting moving objects in multi-temporal images based on epipolar constraints, characterized by: include: Perform ASIFT feature point detection and matching on the acquired first image and the second image to obtain matching feature point pairs; Eliminate mismatches of the matching feature point pairs and use the RANSAC algorithm to obtain optimized feature point pairs; Calculating a fundamental matrix using the optimized feature point pairs; Selecting a random point in the region of interest in the first image, outputting the pixel coordinates of the random point, and calculating an epipolar line corresponding to a point in the second image corresponding to the random point based on the value of the fundamental matrix and the pixel coordinates; intercepting a region of interest from the first image as a template image for reflecting the target to be detected; Using the template image, searching for an image region in the second image that is most similar to the template image, and marking the image region in the second image with a marking frame; Comparing the positional relationship between the epipolar lines and the marked frame in the second image, if all the epipolar lines above a set threshold intersect with the marked frame, it is determined that the target to be detected has not moved; otherwise, it is determined that the target to be detected has moved; Using the template image to search for an image region in the second image that is most similar to the template image, a robust template matching method using scale-adaptive deep convolution features is employed, comprising: Cutting out a region of interest from the first image as a target to be detected and saving it as a template image img; The template image img and the second image are passed through a pre-trained VGG-Net model to extract scale-adaptive deep convolution feature vectors, and the distance between the features of the template image img and the corresponding features of the second image is measured using the NCC algorithm. Based on a set similarity threshold, the image area of the second image whose similarity measure with the template image img is within the set similarity threshold is detected, which is the image area in the second image that is most similar to the template image; Output pixel coordinate values of the upper left corner and the lower right corner of the image area, and mark the second image.

2. The method for detecting moving objects in multi-temporal images based on epipolar constraints according to claim 1, characterized in that: Performing ASIFT feature point detection and matching on the first image and the second image, using a feature point-based bidirectional matching method, including: Extracting scale-invariant feature points of the first image and the second image respectively using the ASIFT algorithm; The first image is used as the reference image and the second image is used as the image to be matched; the feature points in the first image are obtained, and the most similar feature points are found in the second image to match them, forming a feature point pair; this step is repeated to complete the matching of other feature points, and a matching feature point pair Ω1 is obtained; The second image is used as the reference image and the first image is used as the image to be matched; the feature points in the second image are obtained, and the most similar feature points are found in the first image to match them, forming a feature point pair; this step is repeated to complete the matching of other feature points, and a matching feature point pair Ω2 is obtained; The common matching point pairs belonging to the matching feature point pair Ω1 and the matching feature point pair Ω2 are counted and used as the final matching feature point pair Ω.

3. The method for detecting moving objects in multi-temporal images based on epipolar constraints according to claim 1, characterized in that: Mismatches are eliminated from the matching feature point pairs using the RANSAC algorithm, including: Randomly selecting a number of matching feature point pairs from the first image and the second image to form a matching feature point pair dataset, and using the matching feature point pairs to solve the homography transformation matrix H, including: Assume that the coordinates of the feature points in the first image are (x, y), the coordinates of the feature points in the second image are (x1, y1), and the homography matrix H is The corresponding solution formula is: Using the matching feature point pairs, the corresponding equations are listed according to the solution formula, and the unknown parameters h in the homography matrix H are solved according to the equations. 11 、h 12 、h 13 、h 21 、h 22 、h 23 、h 31 、h 32 ; Calculate the projection error between all data in the matching feature point pair data set and the homography matrix H. If the error is less than a threshold, add the inlier set I; If the number of elements in the current interior point set I is greater than the number of elements in the optimal interior point set I_best, then I_best is updated to I, and the number of iterations k is updated at the same time; wherein the initial value of the optimal interior point set I_best is 0; the calculation formula for the number of iterations k is: Where p is the confidence level; w is the ratio of internal points; m is the minimum number of matching feature point pairs required to calculate the homography matrix H; If the actual number of iterations is greater than k, the projection error calculation is exited; otherwise, the number of iterations is increased by 1, and the steps of projection error calculation are repeated; Based on the final obtained inlier point set I, the remaining matching feature point pairs that do not belong to the inlier point set are eliminated to achieve false matching elimination.

4. The method for detecting moving objects in multi-temporal images based on epipolar constraints according to claim 3, characterized in that: Also includes: The number of the optimized feature point pairs is obtained, and when the number is greater than or equal to a set number threshold, a subsequent step of calculating a basic matrix is performed.

5. The method for detecting moving objects in multi-temporal images based on epipolar constraints according to claim 1, characterized in that: Calculating a basic matrix using the optimized feature point pairs includes: Determine that the spatial geometric constraint relationship satisfied between the first image and the second image and used to represent the correspondence relationship between the target point to be detected and the epipolar line is: Among them, p l and p r are the projection points of the target point p to be detected under the first camera and the second camera respectively; F is the basic matrix used to represent the relationship between the epipolar line and the point in the constraint relationship, and the basic matrix F is expressed as: Among them, M l and M r are the internal parameter matrices of the first camera and the second camera respectively; E is the intrinsic matrix; Randomly selecting four pairs of feature point pairs from the optimized feature point pairs to calculate the basic matrix F to obtain an initial value of the basic matrix F; The basic matrix F is iteratively calculated using other feature point pairs in the optimized feature point pair, and the value of the basic matrix F is refined, ultimately reflecting the value of the basic matrix F of the optimal spatial geometric constraint relationship between the first image and the second image.

6. The method for detecting moving objects in multi-temporal images based on epipolar constraints according to claim 1, characterized in that: Selecting a random point in a region of interest in the first image, outputting pixel coordinates of the random point, and calculating an epipolar line corresponding to a point corresponding to the random point in the second image based on a value of the fundamental matrix and the pixel coordinates, including: Selecting a random portion of pixel points in the region of interest in the first image, and outputting pixel coordinate values of the pixel points; The pixel coordinate values of all selected pixel points are stored in the form of a matrix to obtain the pixel coordinate value matrix; The coordinate value matrix of the pixel points is called, and the spatial geometric constraint relationship between the first image and the second image is used to solve the epipolar lines corresponding to the pixel points in the second image.

7. The method for detecting moving objects in multi-temporal images based on epipolar constraints according to claim 1, characterized in that: The threshold value of the epipolar lines is set to 0.95, that is, when more than 95% of the epipolar lines intersect with the marking frame, it is determined that the target to be detected has not moved; otherwise, it is determined that the target to be detected has moved.

8. A multi-temporal image moving target detection system based on epipolar constraints, characterized by: include: An ASIFT algorithm module is used to perform ASIFT feature point detection and matching on the acquired first image and the second image to obtain matching feature point pairs; A feature point optimization module is used to eliminate mismatches of the matching feature point pairs and obtain optimized feature point pairs using the RANSAC algorithm; A basic matrix calculation module, which is used to calculate the basic matrix using the optimized feature point pairs; an epipolar line calculation module, configured to select a random point in a region of interest in the first image, output pixel coordinates of the random point, and calculate an epipolar line corresponding to a point in the second image corresponding to the random point based on a value of the fundamental matrix and the pixel coordinates; a target region labeling module, the module being configured to intercept a region of interest in the first image as a template image for representing a target to be detected; Using the template image, searching for an image region in the second image that is most similar to the template image, and marking the image region in the second image with a marking frame; a motion detection module, configured to compare the positional relationship between the epipolar lines and the labeled frame in the second image; if all epipolar lines above a set threshold intersect the labeled frame, it is determined that the target to be detected has not moved; otherwise, it is determined that the target to be detected has moved; Using the template image to search for an image region in the second image that is most similar to the template image, a robust template matching method using scale-adaptive deep convolution features is employed, comprising: Cutting out a region of interest from the first image as a target to be detected and saving it as a template image img; The template image img and the second image are passed through a pre-trained VGG-Net model to extract scale-adaptive deep convolution feature vectors, and the distance between the features of the template image img and the corresponding features of the second image is measured using the NCC algorithm. Based on a set similarity threshold, the image area of the second image whose similarity measure with the template image img is within the set similarity threshold is detected, which is the image area in the second image that is most similar to the template image; Output pixel coordinate values of the upper left corner and the lower right corner of the image area, and mark the second image.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 7, or run the system according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 7, or to run the system according to claim 8.

Images