A target tracking method and system combining correlation filtering and potential matching

Abstract

The application discloses a target tracking method and system combining correlation filtering and potential matching, and relates to the technical field of computer vision. The method generates a correlation filter and an image template by using a target image, and generates a boundary potential field, a depth field and a mask template of a point cloud template by using a target point cloud. Then, the method performs correlation operation on the image template and an environment image by using the correlation filter to obtain a potential region of the target. Then, the boundary potential field and the depth field in the potential region are rasterized, and local statistical information is calculated. Finally, potential matching is performed on the point cloud template and the potential region by using the local statistical information to obtain an accurate position of the target. The application fuses a visual sensor and a laser radar, and has high tracking accuracy and stability in scenes such as target deformation, illumination change and motion blur. Meanwhile, the application adopts a 'from coarse to fine' tracking mode, realizes the complementary advantages of vision and point cloud, and has high execution efficiency and reliability.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a target tracking method and system that combines correlation filtering and potential energy matching. Background Technology

[0002] Object tracking is one of the most valuable and commercially viable tasks in computer vision, widely used in video surveillance, autonomous driving, navigation, and guidance. Currently, mainstream object tracking algorithms use cameras as sensors, leveraging color and texture information to distinguish the target from the background. However, vision-based object tracking algorithms are susceptible to factors such as target deformation, lighting variations, and motion blur.

[0003] LiDAR (Light Detection and Ranging) is a sensor that emits a detection signal (laser beam) towards a target to detect its spatial position, geometric shape, and other parameters. The point cloud sampled by LiDAR can provide structural and positional information of the target and is insensitive to changes in illumination. Therefore, point cloud-based target tracking algorithms have become a hot research topic. However, the sparsity and disorder of LiDAR point clouds can affect the accuracy and robustness of target tracking algorithms.

[0004] Therefore, how to improve the accuracy and robustness of target tracking is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a target tracking method and system that combines correlation filtering and potential energy matching. By utilizing visual sensors and lidar, the algorithm ensures stability under target deformation, illumination changes and motion blur conditions. It makes reasonable use of the advantages of visual information and point cloud information while avoiding their disadvantages, effectively reducing computation time and improving the accuracy and robustness of target tracking.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A target tracking method combining correlation filtering and potential energy matching includes the following steps:

[0008] Step 1: Obtain the target image and target point cloud, generate an image template using the HOG feature map of the target image, train relevant filters using the image template, and generate the image template. Use the target point cloud to generate the boundary potential field, depth field, and mask template of the point cloud template.

[0009] Target images and target point clouds are selected manually or using algorithms.

[0010] Step 2: Acquire the environmental image captured by the visual sensor and the environmental point cloud captured by the LiDAR at the current moment;

[0011] Step 3: Use a correlation filter to perform correlation operations on the image template and the environmental image to obtain the potential region of the target;

[0012] Step 4: Extract the point cloud that can be projected onto the potential region from the environmental point cloud, generate the boundary potential energy field, depth field and mask template of the point cloud corresponding to the potential region in the environmental point cloud, and rasterize it, and calculate the local statistical information in each raster region of the boundary potential energy field, depth field and mask template.

[0013] Step 5: Using local statistical information, perform potential energy matching on the potential regions of the point cloud template and the environmental point cloud to obtain the precise location of the target, and update the image template and point cloud template to complete target tracking at the current moment. Return to step 2 at the next moment.

[0014] Preferably, the specific implementation process of step 1 is as follows:

[0015] Step 11: Based on the selected target image, calculate the HOG feature map of the target image, use the HOG feature map as the image template of the target, and use the image template to train the correlation filter to obtain the correlation coefficient of the correlation filter.

[0016] Step 12: For each point in the target point cloud, use singular value decomposition to fit the plane containing the corresponding point cloud, and extract the normal vector of the plane.

[0017] Step 13: Project the target point cloud onto the pixel coordinate system, and use the normal vector and depth value of the point cloud to generate the boundary potential field, depth field and mask template.

[0018] Preferably, in step 11, the correlation filter is extended to a nonlinear space using a Gaussian kernel function, the weights of the correlation filter are linearly represented using an image template and correlation coefficients, and the correlation coefficients are subjected to discrete Fourier transform.

[0019] The Gaussian kernel function of the correlation filter is expressed as:

[0020]

[0021] In the formula, σ is the standard deviation of the Gaussian kernel function; x and z are the two input signals of the Gaussian kernel function; ||*|| represents taking the 2-norm; dft -1 (*) indicates the inverse discrete Fourier transform; ⊙ indicates the dot product; ^ indicates the result of the discrete Fourier transform; * indicates taking the conjugate value; Let c be the conjugate value of the input signal x after the discrete Fourier transform of its channel c. The result is the discrete Fourier transform of the input signal's z-channel c.

[0022] The correlation coefficient is expressed as a discrete Fourier transform:

[0023]

[0024] In the formula, This is the result of the discrete Fourier transform of the correlation coefficient α; The discrete Fourier transform result of the Gaussian response peak; ||*|| indicates taking the 2-norm; n is the image template; Represents the Gaussian kernel function k nn The discrete Fourier transform result; λ is the regularization coefficient, and λ is taken as 0.0001.

[0025] Preferably, in step 13, the target point cloud is first projected onto the pixel coordinate system, and a boundary potential energy field, a depth field, and a mask template are generated. For each pixel, when one or more point clouds are projected onto the pixel, the boundary potential energy field of the pixel at that position is the mean of the normalized normal vectors of these point clouds, the depth field is the depth value of the point, and the mask template is 1. When no point cloud is projected onto the pixel, the boundary potential energy field of the pixel at that position is a zero vector, the depth field is 0, and the mask template is 0. In subsequent calculations, only pixels with a mask template of 1 participate in the calculation.

[0026] Preferably, in step 3, the HOG feature map of the environment image is calculated, and the correlation response of the HOG feature map of the environment image is calculated using the correlation filter trained in step 1; the center of the potential region is selected at the point where the correlation response is maximum; the width of the potential region is set according to the width of the image template, and the length of the potential region is set according to the length of the image template. The potential region width is 2.5 times the width of the image template, and the potential region length is 2.5 times the length of the image template, thus generating a rectangular potential region of the target.

[0027] The correlation response between the correlation filter and the HOG feature map of the environment image is calculated as follows:

[0028]

[0029] In the formula, response is the relevance response; The result is the discrete Fourier transform of the correlation coefficient obtained in step 1; The HOG feature map z of the environmental image and the discrete Fourier transform result of the Gaussian kernel function of the image template n obtained in step 1 are used; ⊙ represents the dot product.

[0030] Preferably, the specific implementation process of step 4 is as follows:

[0031] Step 41: Based on the potential region obtained in Step 3, extract the point cloud that can be projected into the potential region, and generate the boundary potential energy field, depth field and mask template of the point cloud in the same way as in Step 1; for each point cloud, fit the plane where the corresponding point cloud is located by singular value decomposition, and extract the normal vector of the plane; project the extracted point cloud onto the pixel coordinate system, and generate the boundary potential energy field, depth field and mask template using the normal vector and depth value of the point cloud;

[0032] Step 42: Rasterize the potential region, and simultaneously rasterize the boundary potential energy field, depth field and mask template generated from the point cloud corresponding to the potential region in the environmental point cloud in Step 41. Calculate the mean and dispersion of each grid in the rasterized boundary potential energy field, depth field and mask template, and use the mean and dispersion to represent the local statistical information within the current grid area.

[0033] The mean and dispersion of the rasterized boundary potential field are calculated as follows:

[0034]

[0035]

[0036] In the formula, Represents a grid (g) u ,g v The mean of the boundary potential energy of the potential region; E(i,j) represents the value of the boundary potential energy field at position (i,j), and must satisfy the following conditions: 1) the coordinate (i,j) is gridded (g u ,g v ) contains, 2) the mask template of the potential region is 1 at (i,j); n represents the raster (g u ,g v Within ) the number of pixels with a mask template of 1; v E (g u ,g v ) represents raster (g u ,g v The boundary potential energy dispersion;

[0037] The mean and dispersion of the rasterized depth field are calculated as follows:

[0038]

[0039]

[0040] In the formula, The coordinates are (g) u ,g vThe depth mean of the raster; D(i,j) represents the value of the depth field of the potential region at position (i,j), and must satisfy the following conditions: 1) the coordinate (i,j) is covered by the raster (g u ,g v ) contains, 2) the mask template of the potential region is 1 at (i,j); v D (g u ,g v ) indicates that the coordinates are (g u ,g v The depth dispersion of the raster; ω is the discrete scaling factor, ω=0.25; tanh(*) is the hyperbolic tangent function; |*| indicates taking the absolute value.

[0041] Preferably, the specific implementation process of step 5 includes:

[0042] Step 51: Based on the boundary potential energy field and depth field of the point cloud template generated in Step 1, and the rasterized boundary potential energy field and depth field of the potential region generated in Step 4, calculate the difference between the point cloud template and the potential region, and perform potential energy matching.

[0043] Step 52: Based on the potential energy matching result of Step 51, generate the corresponding iteration step size, so that the point cloud template moves towards the grid with smaller difference, return to Step 51, until the maximum number of iterations is reached or the matching converges, and take the center position of the boundary potential energy field of the point cloud template at the time of iteration as the target precise position.

[0044] Step 53: Based on the difference σ between the point cloud template and the potential region from the last iteration in Steps 51 and 52. l Extract all that satisfy σ l A potential region raster with a value <0.7 is used to extract all environmental point clouds that can be projected onto these raster grids, generating a point cloud group G. new The point cloud group G is segmented using Euclidean clustering. new Divided into sub-point cloud groups N represents the number of sub-point cloud groups; for each sub-point cloud group Reproject the entire point cloud into the potential region and calculate the sub-point cloud group. The average difference; generate a new point cloud template from the sub-point cloud group with the smallest average difference.

[0045] The average degree of difference is calculated as follows:

[0046]

[0047] In the formula, Sub-point cloud group The average degree of difference; Sub-point cloud group The k-th point can be projected onto the grid (g) u ,g v ) difference σ l The value of m is the sub-point cloud group. The number of point clouds;

[0048] Step 54: Project the new point cloud template onto the pixel coordinate system, select the center position of the new point cloud template as the precise target position, select the smallest rectangular bounding box occupied by the new point cloud template as the precise target bounding box, and then crop the corresponding position of the environment image according to the precise target position and the target bounding box to obtain a new image template, and return to step 2 for subsequent visual tracking algorithms.

[0049] Preferably, the specific implementation process of step 51 is as follows:

[0050] Step 511: Place the center of the boundary potential field and depth field of the point cloud template at the center of the potential region, and use this as the initial value for subsequent potential matching and iterative updates.

[0051] Step 512: Potential energy matching is based on the difference between the boundary potential energy field and the depth field. Using the boundary potential energy field and depth field of the point cloud template, as well as the rasterized boundary potential energy field and depth field of the potential region, the difference between the central grid and its neighboring grids to which each point cloud falls in the point cloud template is calculated. The grid to which the point cloud is projected is selected as the central grid, and the four grids in the neighborhood (top, bottom, left, right) of the central grid are selected as the neighboring grids.

[0052] The difference in each point cloud consists of boundary potential energy difference and depth difference, calculated as follows:

[0053] σ=γ D σ D +γ E σ E ;

[0054]

[0055]

[0056] In the formula, σ represents the degree of difference; D For depth difference; γ D σ is the depth difference correction coefficient; E The difference in boundary potential energy; γ E This is a correction coefficient for the difference in boundary potential energy. v E (g u ,g v ), v D (g u ,gv These are the mean boundary potential energy, boundary potential energy dispersion, mean depth, and depth dispersion obtained in step 4, respectively. The coordinates are (g) u ,g v The mean boundary potential energy of the grid; v E (g u ,g v ) indicates that the coordinates are (g u ,g v The boundary potential energy dispersion of the grid; The coordinates are (g) u ,g v The depth mean of the raster; v D (g u ,g v ) indicates that the coordinates are (g u ,g v The depth dispersion of the raster;

[0057] When the difference satisfies σ≤0.5, it is considered that the point cloud of the current point cloud template is similar to that of the potential region (g). u ,g v If the grid potential energy match is successful, otherwise the potential energy match is considered to have failed.

[0058] Preferably, in step 52, based on the potential energy matching result calculated in step 51, a corresponding iteration step size is generated, causing the point cloud template to move towards the grid where the potential energy matching is successful. The specific process is as follows:

[0059] Step 521: Based on the potential energy matching results of each point cloud in the point cloud template in Step 51, determine the iteration step size provided by the point cloud in this iteration;

[0060] When the point cloud successfully matches the potential energy of the central grid, the point cloud does not provide an iteration step size; when the point cloud fails to match the potential energy of the central grid, but successfully matches the potential energy of one or more neighboring grids, the point cloud provides an iteration step size to move the point cloud toward the grid where the potential energy is successfully matched.

[0061] The iteration step ξ of the i-th point that fails to match the potential energy of the central grid and successfully matches the potential energy of the neighboring grids. i The calculation is as follows:

[0062]

[0063] In the formula, pos(n) is the current position of the neighboring grid n that has successfully matched the potential energy; pos(c) is the current position of the center grid c;

[0064] Step 522: Calculate the iteration step size provided by each point cloud in the point cloud template obtained in step 521, and obtain the total iteration step size;

[0065] The total iteration step size is calculated as follows:

[0066]

[0067] In the formula, ξ t+1 ξ is the total iteration step size at time t+1; t ξ is the total iteration step size at time t; i t+1 Let be the iteration step size of the i-th point cloud at time t+1; α is the momentum coefficient, taken as α = 0.6; n p For the number of matching points;

[0068] Step 523: Translate the boundary potential field and depth field of the point cloud template along the current iteration total step size direction to complete this round of step size update;

[0069] Step 524: After completing the step size update, perform the operation in step 51 to obtain a new step size. Then, iterate and update repeatedly until the maximum number of iterations is reached or the proportion of the number of point clouds with successfully matched potential energy in the point cloud template is match_per > 0.5, at which point the iteration stops. After the iteration ends, the center of the boundary potential energy field of the point cloud template is taken as the precise target position.

[0070] A target tracking system combining correlation filtering and potential energy matching, applicable to a target tracking algorithm combining correlation filtering and potential energy matching, comprising:

[0071] The sensor module is used to obtain environmental images and environmental point clouds of the current environment;

[0072] The initialization module is used to initialize the relevant filter parameters, image template, point cloud template, and the boundary potential field, depth field, and mask template of the point cloud template.

[0073] The information management module is used to store and manage relevant filter parameters, image templates, point cloud templates, and the boundary potential energy field, depth field, and mask template of the point cloud templates. It also updates the relevant filter parameters, image templates, point cloud templates, and the boundary potential energy field, depth field, and mask template of the point cloud templates based on the tracking results.

[0074] The visual tracking module is used to perform correlation calculations on the image template and the environmental image using correlation filters to obtain the potential region of the target;

[0075] The point cloud tracking module is used to extract point clouds within a potential region, generate corresponding boundary potential energy fields, depth fields, and mask templates, and rasterize them. It calculates local statistical information within each raster region and uses this local statistical information to perform potential energy matching between the point cloud template and the potential region to obtain the precise location of the target and the tracking result.

[0076] As can be seen from the above technical solution, compared with the prior art, this invention discloses a target tracking method and system that combines correlation filtering and potential energy matching. It employs a camera and a lidar as sensors, and utilizes color texture information provided by the camera and geometric structure information provided by the lidar to achieve a "coarse-to-fine" tracking mode, realizing the correction of visual tracking results by point cloud matching results. This invention, on the one hand, avoids the influence of target deformation, illumination changes, and motion blur, effectively improving the tracking accuracy and stability in these situations; on the other hand, it also solves the influence of sparse point clouds and high-frequency noise interference, reducing the computation time for point cloud processing and improving the algorithm's execution speed. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0078] Figure 1 The attached figure is a flowchart of a target tracking method combining correlation filtering and potential energy matching provided by the present invention;

[0079] Figure 2 The attached figure is a flowchart of the visual tracking calculation based on correlation filtering provided by the present invention;

[0080] Figure 3 The attached figure is a flowchart of the point cloud tracking calculation based on potential energy matching provided by the present invention;

[0081] Figure 4 The attached figure is a schematic diagram of a target tracking system that combines correlation filtering and potential energy matching provided by the present invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Example 1

[0084] This invention discloses a target tracking method combining correlation filtering and potential energy matching, such as... Figures 1-3 As shown, it includes the following steps:

[0085] S1: Obtain the target image and target point cloud, train relevant filters using the HOG feature map of the target image, and generate an image template. Use the target point cloud to generate the boundary potential field and depth mask template of the point cloud template.

[0086] Target images and target point clouds are selected manually or using algorithms.

[0087] S11: Based on the selected target image, calculate the HOG feature map of the target image, use the HOG feature map as the image template of the target, and use the image template to train the correlation filter to obtain the correlation coefficient of the correlation filter.

[0088] The correlation filter is extended to a nonlinear space using a Gaussian kernel function. The weights of the correlation filter are linearly represented using an image template and the correlation coefficient, and the correlation coefficient is subjected to a discrete Fourier transform.

[0089] The Gaussian kernel function of the correlation filter is as follows:

[0090]

[0091] In the formula, σ is the standard deviation of the Gaussian kernel function; x and z are the two input signals of the kernel function; ||*|| denotes taking the 2-norm; dft -1 (*) indicates the inverse discrete Fourier transform; ⊙ indicates the dot product; ^ indicates the result of the discrete Fourier transform; the superscript * indicates taking the conjugate value; Let c be the conjugate value of the input signal x after the discrete Fourier transform of its channel c. The result is the discrete Fourier transform of the input signal's z-channel c.

[0092] The correlation coefficient is processed by Discrete Fourier Transform as follows:

[0093]

[0094] In the formula, This is the result of the discrete Fourier transform of the correlation coefficient α; The discrete Fourier transform result of the Gaussian response peak; ||*|| indicates taking the 2-norm; n is the image template generated in step S31; Represents the Gaussian kernel function k nn The discrete Fourier transform result; λ is the regularization coefficient, and λ is taken as 0.0001;

[0095] S12: For each point in the target point cloud, use singular value decomposition to fit the plane where the point is located, and extract the normal vector of the plane. Use the normal vector to represent the boundary potential energy of the point.

[0096] S13: Project the target point cloud onto the pixel coordinate system, and generate the boundary potential field, depth field, and mask template using the normal vector and depth value of the point cloud;

[0097] First, the target point cloud is projected onto the pixel coordinate system, and a boundary potential field, a depth field, and a mask template are generated. For each pixel, when one or more point clouds are projected onto that pixel, the boundary potential field at that location is the mean of the normalized normal vectors of these point clouds, the depth field is the depth value of that point, and the mask template is set to 1. When no point clouds are projected onto that pixel, the boundary potential field at that location is a zero vector, the depth field is 0, and the mask template is 0. In subsequent calculations, only pixels with a mask template of 1 participate in the calculation.

[0098] S2: Use a visual sensor and a lidar to obtain the current environmental image and environmental point cloud, respectively;

[0099] S3: Use a correlation filter to perform correlation operations on the image template and the environmental image to obtain the potential region of the target;

[0100] The HOG feature map of the environment image is calculated, and the relevance response of the HOG feature map of the environment image is calculated using the filter trained by S1. The center of the potential region is taken as the point with the maximum relevance response, the width of the potential region is 2.5 times the width of the image template, and the length of the potential region is 2.5 times the length of the image template, thus generating a rectangular potential region of the target.

[0101] The correlation response between the filter and the HOG feature map of the environment image is calculated as follows:

[0102]

[0103] In the formula, response is the relevance response; The discrete Fourier transform result of the correlation coefficient obtained from S1; The discrete Fourier transform of the Gaussian kernel function of the image template n is obtained from the HOG feature maps z and S1 of the environmental image; ⊙⊙ represents dot product;

[0104] S4: Extract point clouds that can be projected onto potential regions from the environmental point cloud, generate the boundary potential energy field, depth field and mask template of the point cloud corresponding to the potential region in the environmental point cloud and rasterize them, calculate the local statistical information of each raster region in the boundary potential energy field, depth field and mask template.

[0105] S41: Based on the potential region obtained in S3, extract the point cloud that can be projected onto the potential region. Generate the boundary potential energy field, depth field, and mask template of the point cloud in the same way as in S13. For each point cloud, fit the plane where the corresponding point cloud is located using singular value decomposition and extract the normal vector of the plane. Project the extracted point cloud onto the pixel coordinate system and generate the boundary potential energy field, depth field, and mask template using the normal vector and depth value of the point cloud.

[0106] S42: Rasterize the potential region, and simultaneously rasterize the boundary potential energy field, depth field, and mask template of the potential region generated in the environmental point cloud of S41. Calculate the mean and dispersion of each grid in the rasterized boundary potential energy field, depth field, and mask template, and use the mean and dispersion to represent the local statistical information within the current grid region.

[0107] The mean and dispersion of the rasterized boundary potential field are calculated as follows:

[0108]

[0109]

[0110] In the formula, Represents a grid (g) u ,g v The mean of the boundary potential energy of the potential region; E(i,j) represents the value of the boundary potential energy field at position (i,j), and must satisfy the following conditions: 1) the coordinate (i,j) is gridded (g u ,g v ) contains, 2) the mask template of the potential region is 1 at (i,j); n represents the raster (g u ,g v Within ) the number of pixels with a mask template of 1; v E (g u ,g v ) represents raster (g u ,g v The boundary potential energy dispersion;

[0111] The mean and dispersion of the rasterized depth field are calculated as follows:

[0112]

[0113]

[0114] In the formula, The coordinates are (g) u ,g vThe depth mean of the raster; D(i,j) represents the value of the depth field of the potential region at position (i,j), and must satisfy the following conditions: 1) the coordinate (i,j) is covered by the raster (g u ,g v ) contains, 2) the mask template of the potential region is 1 at (i,j); v D (g u ,g v ) indicates that the coordinates are (g u ,g v The depth dispersion of the raster; ω is the discrete scaling factor, ω=0.25; tanh(*) is the hyperbolic tangent function; |*| indicates taking the absolute value.

[0115] S5: Using local statistical information, potential energy matching is performed on the point cloud template and potential region to obtain the precise location of the target. The image template and point cloud template are updated according to the precise location to complete the target tracking at the current moment.

[0116] S51: Calculate the difference between the point cloud template and the potential region by comparing the potential energy field and depth field of the point cloud template generated in S1 with the potential energy field and depth field of the rasterized boundary of the potential region generated in S4, and perform potential energy matching.

[0117] The center of the potential energy field and depth field of the point cloud template boundary is placed at the center of the potential region, and this point is used as the initial value for subsequent potential energy matching and iterative updates. Potential energy matching is based on the difference between the boundary potential energy field and the depth field. The difference between the central grid and its neighboring grids of each point cloud in the point cloud template is calculated using the boundary potential energy field and depth field of the point cloud template, as well as the rasterized boundary potential energy field and depth field of the potential region. The grid to which the point cloud is projected is selected as the central grid, and the four grids in the neighborhood (top, bottom, left, right) of the central grid are selected as the neighboring grids.

[0118] The difference in each point cloud consists of boundary potential energy difference and depth difference, calculated as follows:

[0119] σ=γ D σ D +γ E σ E ;

[0120]

[0121]

[0122] In the formula, σ represents the degree of difference; D For depth difference; γ D σ is the depth difference correction coefficient; E The difference in boundary potential energy; γ EThis is a correction coefficient for the difference in boundary potential energy. v E (g u ,g v ), v D (g u ,g v The mean and dispersion obtained from S4 are: The coordinates are (g) u ,g v The mean boundary potential energy of the grid; v E (g u ,g v ) indicates that the coordinates are (g u ,g v The boundary potential energy dispersion of the grid; The coordinates are (g) u ,g v The depth mean of the raster; v D (g u ,g v ) indicates that the coordinates are (g u ,g v The depth dispersion of the raster;

[0123] When the difference satisfies σ≤0.5, it is considered that the point cloud of the current point cloud template is similar to that of the potential region (g). u ,g v If the grid potential energy match is successful, otherwise the potential energy match is considered to have failed.

[0124] S52: Based on the matching result of S51, generate the corresponding iteration step size, so that the point cloud template moves to the grid with smaller difference. After the step size is updated, perform the operation of S51 again to obtain a new step size, until the maximum number of iterations is reached or the matching converges.

[0125] Based on the potential energy matching result of each point in the point cloud template in S51, the step size provided by the point cloud in this iteration is determined. When the potential energy of the point cloud is successfully matched with the central grid, the point cloud does not provide an iteration step size. When the potential energy of the point cloud fails to match with the central grid, but the point cloud is successfully matched with the potential energy of one or more neighboring grids, the point cloud provides an iteration step size to move the point cloud toward the grid with successfully matched potential energy.

[0126] The iteration step ξ of the i-th point that fails to match the potential energy of the central grid and successfully matches the potential energy of the neighboring grids. i The calculation is as follows:

[0127]

[0128] In the formula, pos(n) is the current position of the neighboring grid n that has successfully matched the potential energy; pos(c) is the current position of the center grid c;

[0129] The iteration step size provided by each point cloud in the point cloud template is obtained through statistical calculation, and the total iteration step size is obtained. The potential energy field and depth field of the point cloud template boundary are translated along the current total step size direction to complete the step size update for this round. After the step size update is completed, the S51 operation is performed again to obtain a new step size. The iteration update is then repeated until the maximum number of iterations is reached or the percentage of points with successfully matched potential energy match_per > 0.5, at which point the iteration stops. After the iteration ends, the boundary potential energy center of the point cloud template is taken as the precise location of the target.

[0130] The total update step size is calculated as follows:

[0131]

[0132] In the formula, ξ t+1 ξ is the total iteration step size at time t+1; t Let be the total iteration step size at time t; α is the momentum coefficient, taken as α = 0.6; n p To match the number of points.

[0133] S53: The difference σ between the point cloud template and the potential region based on the last iteration in steps 51 and 52. l Extract all that satisfy σ l A potential region raster with a value <0.7 is used to extract all environmental point clouds that can be projected onto these raster grids, generating a point cloud group G. new The point cloud group G is segmented using Euclidean clustering. new Divided into sub-point cloud groups N represents the number of sub-point cloud groups; for each sub-point cloud group Will Reproject the entire point cloud into the potential region and calculate the sub-point cloud group. The average difference; generate a new point cloud template from the sub-point cloud group with the smallest average difference.

[0134] The average degree of difference is calculated as follows:

[0135]

[0136] In the formula, Sub-point cloud group The average degree of difference; Sub-point cloud group The k-th point can be projected onto the grid (g) u ,g v ) difference σ l The value of m is the sub-point cloud group. The number of point clouds;

[0137] S54: Based on the projection of the new point cloud template onto the pixel coordinate system, select the center position of the new point cloud template as the precise target position, select the smallest rectangular bounding box occupied by the new point cloud template as the precise target bounding box, and then crop the corresponding position of the environment image according to the precise target position and the target bounding box to obtain a new image template for subsequent visual tracking algorithms.

[0138] The visual / laser fusion target tracking algorithm combining correlation filtering and potential energy matching provided in this invention optimizes the visual target tracking results based on correlation filtering along with the point cloud target tracking results based on potential energy matching to obtain a final accurate solution. First, using correlation filtering, the correlation response of the image search region in the current frame is calculated using the same image template to generate a potential region based on texture information. Then, the point cloud within the potential region is extracted to generate a rasterized boundary potential energy field, depth field, and mask template. Finally, local statistical information is used to perform potential energy matching with the point cloud template to obtain the precise location of the target. This invention integrates a visual sensor and a lidar, exhibiting high tracking accuracy and stability under conditions of target deformation, illumination changes, and motion blur. Simultaneously, it employs a "coarse-to-fine" tracking mode, enabling the correction of visual tracking results by point cloud matching, thereby improving the algorithm's execution speed.

[0139] Example 2

[0140] One embodiment provides a target tracking system that combines correlation filtering and potential matching, such as Figure 4 As shown, the system is applicable to a target tracking method that combines correlation filtering and potential energy matching, including: sensor module 1, information management module 2, visual tracking module 3, point cloud tracking module 4, and initialization module 5.

[0141] Sensor module 1 includes camera 11 and 3D LiDAR 12, which provide real-time environmental images and environmental point clouds, and are used in subsequent visual tracking module and point cloud tracking module.

[0142] Initialization module 5 is used to initialize the relevant filter parameters 21, image template 22, point cloud template 23, and the boundary potential field, depth field, and mask template of the point cloud template;

[0143] The information management module 2 is used to store and manage the relevant filter parameters 21, image template 22, point cloud template 23, and the boundary potential energy field, depth field, and mask template of the point cloud template. It also updates the relevant filter parameters 21, image template 22, point cloud template 23, and the boundary potential energy field, depth field, and mask template of the point cloud template 23 according to the output of the visual tracking module 3 and the point cloud tracking module 4.

[0144] Visual tracking module 3 uses a correlation filter to perform correlation operations on the image template and the environment image to obtain the potential region of the target. The potential region calculated by visual tracking module 3 provides the initial values ​​for iteration of point cloud tracking module 4;

[0145] Point cloud tracking module 4 is used to extract point clouds within potential regions, generate rasterized boundary potential energy fields, depth fields, and mask templates, calculate local statistical information within each raster region, and use this local statistical information to perform potential energy matching between the point cloud template and the potential region to obtain the precise location of the target. The calculated precise location is applied to information management module 2 to complete the tracking task for the current frame.

[0146] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0148] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method combining correlation filtering and potential energy matching, characterized in that, Includes the following steps: Step 1: Acquire the target image and target point cloud, generate an image template using the target image, train relevant filters using the image template, and generate the boundary potential field, depth field, and mask template of the point cloud template using the target point cloud. Step 2: Acquire the environmental image captured by the visual sensor and the environmental point cloud captured by the LiDAR at the current moment; Step 3: Use a correlation filter to perform correlation operations on the image template and the environmental image to obtain the potential region of the target; Step 4: Extract the point cloud that can be projected onto the potential region from the environmental point cloud, generate the boundary potential energy field, depth field and mask template of the point cloud corresponding to the potential region in the environmental point cloud, and rasterize it, and calculate the local statistical information in each raster region of the boundary potential energy field, depth field and mask template. Step 5: Using local statistical information, perform potential energy matching between the potential regions of the point cloud template and the environmental point cloud to obtain the precise location of the target, and update the image template and point cloud template. Return to step 2 at the next moment.

2. The target tracking method combining correlation filtering and potential energy matching according to claim 1, characterized in that, The specific implementation process of step 1 is as follows: Step 11: Based on the selected target image, calculate the HOG feature map of the target image, use the HOG feature map as the image template of the target, and use the image template to train the correlation filter to obtain the correlation coefficient of the correlation filter. Step 12: For each point in the target point cloud, use singular value decomposition to fit the plane containing the corresponding point cloud, and extract the normal vector of the plane. Step 13: Project the target point cloud onto the pixel coordinate system, and use the normal vector and depth value of the point cloud to generate the boundary potential field, depth field and mask template.

3. The target tracking method combining correlation filtering and potential energy matching according to claim 2, characterized in that, In step 11, the correlation filter is extended to a nonlinear space using a Gaussian kernel function. The weights of the correlation filter are linearly represented using an image template and the correlation coefficient, and the correlation coefficient is subjected to a discrete Fourier transform. The Gaussian kernel function of the correlation filter is expressed as: ； In the formula, The standard deviation of the Gaussian kernel function; and These are the two input signals of the Gaussian kernel function; This indicates taking the 2-norm; Indicates the inverse discrete Fourier transform; Dot product; This represents the result of the discrete Fourier transform. This indicates taking the conjugate value; Input signal The conjugate value of channel c after discrete Fourier transform; Input signal The result of the discrete Fourier transform of channel c; The correlation coefficient is expressed as a discrete Fourier transform: ； In the formula, Correlation coefficient The discrete Fourier transform results; The discrete Fourier transform results of the Gaussian response peak; This indicates taking the 2-norm; Image template; Represents the Gaussian kernel function The discrete Fourier transform results; This is the regularization coefficient.

4. The target tracking method combining correlation filtering and potential energy matching according to claim 2, characterized in that, Step 3 calculates the HOG feature map of the environment image, and uses the correlation filter trained in Step 1 to calculate the correlation response of the HOG feature map of the environment image; selects the point with the maximum correlation response as the center of the potential region, sets the width of the potential region according to the width of the image template, sets the length of the potential region according to the length of the image template, and generates a rectangular potential region of the target. The correlation response between the correlation filter and the HOG feature map of the environment image is calculated as follows: ； In the formula, For relevance response; The result is the discrete Fourier transform of the correlation coefficient obtained in step 1; HOG feature map of environmental image The image template obtained in step 1 The discrete Fourier transform results of the Gaussian kernel function; This indicates dot product.

5. The target tracking method combining correlation filtering and potential energy matching according to claim 1, characterized in that, The specific implementation process of step 4 is as follows: Step 41: Extract point clouds that can be projected onto the potential region based on the potential region. For each point cloud, use singular value decomposition to fit the plane where the corresponding point cloud is located and extract the normal vector of the plane. Project the extracted point cloud onto the pixel coordinate system and use the normal vector and depth value of the point cloud to generate the boundary potential energy field, depth field and mask template. Step 42: Rasterize the potential region, and simultaneously rasterize the boundary potential energy field, depth field and mask template generated from the point cloud corresponding to the potential region in the environmental point cloud in Step 41. Calculate the mean and dispersion of each grid in the rasterized boundary potential energy field, depth field and mask template, and use the mean and dispersion to represent the local statistical information within the current grid area. The mean and dispersion of the rasterized boundary potential field are calculated as follows: ； ； In the formula, Represents grid The mean of the boundary potential energy; This indicates the boundary potential field of the potential region. The location value must meet the following conditions: 1) Coordinates The area is gridded The included, 2) mask template of the potential region in The value is 1; Represents grid The number of pixels with a mask template of 1 within the inner area; Represents a grid The boundary potential energy dispersion; The mean and dispersion of the rasterized depth field are calculated as follows: ； ； In the formula, Indicates coordinates as Mean depth of the raster; The depth field of the potential region is represented in The location value must meet the following conditions: 1) Coordinates The area is gridded The included, 2) mask template of the potential region in The value is 1; Indicates coordinates as The depth dispersion of the raster; For discrete scaling factors, ; It is the hyperbolic tangent function; This indicates taking the absolute value.

6. The target tracking method combining correlation filtering and potential energy matching according to claim 1, characterized in that, The specific implementation process of step 5 includes: Step 51: Based on the boundary potential energy field and depth field of the point cloud template generated in Step 1, and the rasterized boundary potential energy field and depth field of the potential region generated in Step 4, calculate the difference between the point cloud template and the potential region, and perform potential energy matching. Step 52: Based on the potential energy matching result, generate the corresponding iteration step size so that the point cloud template moves towards the grid with small difference. Return to step 51 until the maximum number of iterations is reached or the matching converges. Take the center position of the boundary potential energy field of the point cloud template when the iteration is completed as the target precise position. Step 53: Based on the difference between the point cloud template and the potential region from the last iteration in Steps 51 and 52 Extract all that satisfy The potential region raster is used to extract all environmental point clouds that can be projected onto these raster grids, generating a point cloud cluster. Point cloud groups are segmented using Euclidean clustering. Divided into sub-point cloud groups N represents the number of sub-point cloud groups; for each sub-point cloud group For i=1, ..., N, reproject all point clouds onto the potential region and calculate the sub-point cloud group. The average difference; generate a new point cloud template from the sub-point cloud group with the smallest average difference. The average degree of difference is calculated as follows: ； In the formula, Sub-point cloud group The average degree of difference; Sub-point cloud group The Middle Each point can be projected onto the grid. Difference The value of ; For the children, dot the clouds The number of point clouds; Step 54: Project the new point cloud template onto the pixel coordinate system, select the center position of the new point cloud template as the precise target position, select the smallest rectangular bounding box occupied by the new point cloud template as the precise target bounding box, and then crop the corresponding position of the environment image according to the precise target position and the precise target bounding box to obtain a new image template, and return to step 2.

7. The target tracking method combining correlation filtering and potential energy matching according to claim 6, characterized in that, The specific implementation process of step 51 is as follows: Step 511: Place the center of the boundary potential field and depth field of the point cloud template at the center of the potential region, and use this point as the initial value for iteration; Step 512: Potential energy matching is based on the difference between the boundary potential energy field and the depth field. Using the boundary potential energy field and depth field of the point cloud template, as well as the rasterized boundary potential energy field and depth field of the potential region, the difference between the central grid and its neighboring grids to which each point cloud falls in the point cloud template is calculated. The grid to which the point cloud is projected is selected as the central grid, and the four grids in the neighborhood of the central grid are selected as the neighboring grids. The difference in each point cloud consists of boundary potential energy difference and depth difference, calculated as follows: ； ； ； In the formula, For the degree of difference; For depth difference; This is a depth difference correction coefficient; The degree of difference in boundary potential energy; This is a correction coefficient for the difference in boundary potential energy. , , , These are the mean boundary potential energy, boundary potential energy dispersion, mean depth, and depth dispersion obtained in step 4, respectively. Indicates coordinates as Mean boundary potential energy of the grid; Indicates coordinates as Boundary potential energy dispersion of the grid; Indicates coordinates as Mean depth of the raster; Indicates coordinates as The depth dispersion of the raster; When the difference satisfies When this happens, it is considered that the point cloud in the current point cloud template is related to the potential region. If the grid potential energy match is successful, then the potential energy match is considered to have failed.

8. The target tracking method combining correlation filtering and potential energy matching according to claim 6, characterized in that, In step 52, based on the difference calculated in step 51, a corresponding iteration step size is generated, causing the point cloud template to move towards the grid with smaller difference. The specific process is as follows: Step 521: Based on the potential energy matching results of each point cloud in the point cloud template in Step 51, determine the iteration step size provided by the point cloud in this iteration; When the point cloud successfully matches the potential energy of the central grid, the point cloud does not provide an iteration step size; when the point cloud fails to match the potential energy of the central grid, but successfully matches the potential energy of one or more neighboring grids, the point cloud provides an iteration step size to move the point cloud toward the grid where the potential energy is successfully matched. The first grid cell failed to match the potential energy of the central grid cell, but successfully matched the potential energy of a neighboring grid cell. Iteration step size of each point The calculation is as follows: ； In the formula, The current position of the neighboring grid n where the potential energy is successfully matched; The current position of the center grid c; Step 522: Calculate the iteration step size provided by each point cloud in the point cloud template obtained in Step 521, and obtain the total iteration step size. ； In the formula, Let be the total iteration step size at time t+1; Let be the total iteration step size at time t; The iteration step size provided for the i-th point cloud in the point cloud template at time t+1; The momentum coefficient; For the number of matching points; Step 523: Translate the boundary potential field and depth field of the point cloud template along the current total iteration step direction to complete the step size update for this round; Step 524: Return to step 51 to obtain the new iteration step size, until the maximum number of iterations is reached or the percentage of point clouds with successfully matched potential energy in the point cloud template is reached. When the iteration stops, the exact location of the target is determined by the center of the boundary potential field of the point cloud template.

9. A target tracking system combining correlation filtering and potential energy matching, characterized in that, The target tracking method combining correlation filtering and potential energy matching according to any one of claims 1-8 includes: The sensor module is used to obtain environmental images and environmental point clouds of the current environment; The initialization module is used to initialize the relevant filter parameters, image template, point cloud template, and the boundary potential field, depth field, and mask template of the point cloud template. The information management module is used to store and manage relevant filter parameters, image templates, point cloud templates, and the boundary potential energy field, depth field, and mask template of the point cloud templates. It also updates the relevant filter parameters, image templates, point cloud templates, and the boundary potential energy field, depth field, and mask template of the point cloud templates based on the tracking results. The visual tracking module is used to perform correlation calculations on the image template and the environmental image using correlation filters to obtain the potential region of the target; The point cloud tracking module is used to extract point clouds within a potential region, generate corresponding boundary potential energy fields, depth fields, and mask templates, and rasterize them. It calculates local statistical information within each raster region and uses this local statistical information to perform potential energy matching between the point cloud template and the potential region to obtain the precise location of the target and the tracking result.

Images