Open pit coal mine vehicle tracking method based on feature fusion
By using feature fusion and nuclear-related filter training methods in the tracking of open-pit coal mine vehicles and combining subpixel positioning technology, the problem of open-pit coal mine vehicles being unable to accurately identify and track in complex environments is solved, and the tracking accuracy and robustness are improved.
Patent Information
- Application Number
- CN202411826837.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-22
AI Technical Summary
Open-pit coal mine vehicles cannot accurately identify and track in complex environments. Traditional vehicle tracking algorithms only consider single features and do not consider the correlation and fusion between features, resulting in low tracking accuracy in non-paved roads, dust environments and different lighting conditions.
Using a feature fusion-based method, the direction gradient histogram features, local binary mode features and SIFT features of the vehicle are extracted, and the input core-related filters are trained through adaptive feature fusion, and the vehicle position is precisely positioned in combination with sub-pixel positioning technology.
It improves the accuracy of vehicle tracking of open-pit coal mines, effectively overcomes the problems of missing and blurred vehicle image information or complex background information in complex environments, and enhances the robustness and positioning accuracy of the tracking model.
Smart Images
Figure CN119863771A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of open-pit mine vehicle tracking, and in particular to an open-pit coal mine vehicle tracking method based on feature fusion. Background Art
[0002] Since open-pit coal mines have always been in a priority development position in the coal industry, their related technologies have also been rapidly developed. In the actual production of open-pit coal mines, a certain number of mining trucks are used. How to improve safety and productivity while saving labor costs has become the main research direction of open-pit coal mine intelligence. Open-pit coal mine vehicle tracking technology is a relatively important research content in the field of mine unmanned driving in recent years. Although vehicle tracking technology has been developed for many years, it still faces many challenges in actual application, such as the influence of non-paved roads, dusty environments, rain, fog and snow, changes in perspectives under multiple cameras, different lighting and interference with similar targets. The video images of the same vehicle in different scenes may show large differences in appearance, which brings challenges to the tracking of vehicle targets. In addition, most traditional vehicle tracking algorithms only target and extract a single feature of the target, without considering the correlation between different features of the same target and the fusion between features. In addition, due to the influence of factors such as the complex environment of open-pit coal mines, there will be certain random interference in the video images of mine vehicles, which will blur or lose the features of the original vehicle image itself, resulting in the inability to accurately identify and track mine vehicles. Summary of the invention
[0003] In order to overcome the deficiencies of the above-mentioned prior art, the present invention proposes an open-pit coal mine vehicle tracking method based on feature fusion, which improves the accuracy of open-pit coal mine vehicle tracking and solves the problem that accurate identification and tracking cannot be performed in complex environments.
[0004] The present invention proposes an open-pit coal mine vehicle tracking method based on feature fusion, and its technical key points are as follows: S1: acquiring a monitoring video of an open-pit coal mine vehicle and obtaining an initial frame position and size of a target vehicle;
[0005] S2: extract the directional gradient histogram features, local binary pattern features and SIFT features of the target vehicle, and preprocess the features;
[0006] S3: Perform feature fusion according to the adaptive feature fusion method, and input the fused features into the kernel correlation filter for training; S4: Perform sub-pixel positioning on the response map obtained by training;
[0007] S5: Output the vehicle position for subsequent tracking.
[0008] Furthermore, in step S1, a monitoring video of an open-pit coal mine vehicle is obtained, and the monitoring video is a video within a set certain time period; obtaining the initial frame position of the target vehicle refers to selecting a frame in which the vehicle target is clearer in the first few frames of the video as the initial frame for tracking, using a rectangular frame to mark the target in the initial frame, and obtaining the position and size of the vehicle target through the position and size of the rectangular frame.
[0009] Furthermore, the preprocessing described in step S2 refers to: using principal component analysis to reduce the dimension of the directional gradient histogram features, local binary pattern features and SIFT features, and then performing feature extraction after the dimension reduction.
[0010] Furthermore, the feature fusion described in step S3 refers to: fusing the LBP feature and SIFT feature of the target in parallel to obtain the FUS feature, and then training the kernel correlation filter with the HOG feature and the FUS feature respectively.
[0011] Furthermore, the step S3 of inputting the fused features into the kernel correlation filter for training comprises the following steps: S3.1: inputting the target features to train the kernel correlation filter;
[0012] S3.2: Perform weighted fusion on the responses of the two features to obtain the final response;
[0013] S3.3: Update the filter model.
[0014] Furthermore, the steps of inputting the target features in step S3.1 to train the kernel correlation filter are as follows:
[0015] In the process of training the kernel correlation filter, for the feature φ with the number of channels K, the range W×H, and the unit size C, the feature x φ The size is Its objective function is as follows:
[0016]
[0017] If the vehicle target sample is x=[x 1 ,x 2 ,x 3 ,…,x n ], then the circulant matrix C(x) is generated by cyclic shift, which is expressed as follows:
[0018]
[0019] Diagonalizing the matrix yields:
[0020]
[0021] in, is the DFT transform of x, FH is the conjugate transpose of F;
[0022] The Gaussian kernel function is used to transform the vehicle target sample from the low-dimensional space to the high-dimensional space. The Gaussian kernel function is expressed as follows:
[0023]
[0024] Among them, x′ represents the image to be detected, σ represents the bandwidth of the Gaussian kernel function;
[0025] For most kernel functions, This can be represented using a circulant matrix:
[0026]
[0027] The kernel correlation filters of HOG features and FUS features are trained separately, and the responses of the two features are:
[0028]
[0029] in, and Represent the correlation filters of two feature trainings, and Represents the Gaussian kernel between the target sample and the candidate sample of the two features, f HOG(Z) and f FUS(Z) Respectively represent relevant responses.
[0030] Furthermore, the steps of weighted fusion of the responses of the two features described in step S3.2 to obtain the final response are as follows:
[0031] According to the designed peak sidelobe ratio PSR and the target response maximum value f max (t) method to calculate the feature fusion weight and obtain the feature confidence score The feature fusion weights of HOG features and FUS features can be defined as:
[0032]
[0033] η HOG =min(ξ HOG ,0.6);
[0034] η FUS =1-η HOG ;
[0035] The final response of the target vehicle can be obtained by weighting the responses of the two features, expressed as:
[0036] f(z)=η HOG f HOG(z)+η FUS f FUS (z).
[0037] Furthermore, the steps of updating the filter model described in step S3.3 are as follows:
[0038] During the tracking process, drift will occur when the filter model is updated. The exponential decay method is used to weight the filter model for update. The model update formula of the feature φ is:
[0039]
[0040] Among them, θ φ is the exponential decay learning rate of feature φ, the current frame is t, and is the appearance model and filter model of the previous frame, and Appearance features and filter models calculated for the current frame.
[0041] Furthermore, the step of performing sub-pixel positioning on the response map obtained through training in S4 includes:
[0042] The peak value of the correlation response is approximated by the Taylor formula, and the estimated deviation value ε is obtained at the same time. The peak value of the correlation response is expressed as:
[0043] f(m+ε)=f(m)+f′(m)ε+(ε 2 );
[0044] Taking the first-order derivative of both sides of the expression at the peak of the continuous response, we can obtain:
[0045] f′(m+ε)=f′(m)+f″(m)ε;
[0046]
[0047] Among them, α is the step size, and the value is 0.08;
[0048] The true peak position of the response is:
[0049]
[0050] Among them, x t ,y t is the horizontal and vertical coordinate value of the true response peak, ε x , ε y is the deviation value of ε on the x and y axes.
[0051] Compared with the prior art, the present invention has the following beneficial technical effects:
[0052] The present invention can improve the accuracy of the description of the appearance of the vehicle target by extracting the HOG, LBP and SIFT features of the vehicle target and adaptively fusing the features, and can effectively overcome the problem of missing and blurred target image information or complex background information, thereby improving the tracking accuracy. The present invention improves the robustness of the tracking model by inputting the fused features into the kernel correlation filter for training, and proposes an exponential decay learning rate to weight the tracking model update. By performing sub-pixel positioning on the response map obtained by training and correcting the deviation of the kernel correlation filter, the accuracy of vehicle target positioning is improved, which is more beneficial to the tracking of the vehicle target. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. The features and advantages of the present invention will be more clearly understood by referring to the drawings. The drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0054] Figure 1 A schematic diagram of the steps of the open-pit coal mine vehicle tracking method according to the embodiment disclosed in the present invention;
[0055] Figure 2 The present invention is a flowchart of the overall process of open-pit coal mine vehicle tracking according to the disclosed embodiment of the present invention. DETAILED DESCRIPTION
[0056] The following will be combined with the attached embodiment of the present invention Figure 1 and 2 , the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] The embodiment of the present invention proposes a method for tracking vehicles in an open-pit coal mine based on feature fusion, comprising the following steps:
[0058] S1: Obtain the surveillance video of the open-pit coal mine vehicle and obtain the initial frame position and size of the target vehicle. The specific steps are: obtain the surveillance video of the open-pit coal mine vehicle, and the surveillance video takes the video within a certain period of time. The period of time is set by the customer according to the needs. Obtaining the initial frame position of the target vehicle means selecting a frame with a clearer vehicle target (such as a resolution of about 200*200) in the first few frames of the video as the initial frame for tracking, using a rectangular frame to mark the target in the initial frame, and obtaining the position and size of the vehicle target through the position and size of the rectangular frame.
[0059] S2: Extract the HOG, LBP and SIFT features of the target vehicle and preprocess the features. The specific steps include:
[0060] The target vehicle's Histogram of Directed Gradients (HOG), Local Binary Pattern (LBP) and SIFT features are extracted. During feature extraction, the extracted features have a high dimension, which will affect the real-time performance of tracking. Therefore, the principal component analysis (PCA) is used to reduce the dimension of the features.
[0061] When extracting LBP features, the image is first divided into several small blocks, LBP features are extracted for each small block, and the gradient information of the pixels in the small block is calculated. Then, the LBP features and gradient information are fused to obtain the fused LBP features of the small block. Finally, the fused LBP features of all small blocks are spliced to obtain the LBP features of the entire image.
[0062] When extracting SIFT features, first extract the key feature points of the entire image, then intercept the feature points in the target tracking frame, and resize the intercepted key points and match and unify the pixel values. Since SIFT features are relatively high in dimension, SIFT features are first reduced to 1 dimension.
[0063] S3: According to the designed adaptive feature fusion method, feature fusion is performed, and the fused features are input into the kernel correlation filter for training. The specific steps include:
[0064] First, the LBP feature and SIFT feature of the target are fused in parallel to obtain the FUS feature, and then the HOG feature and FUS feature are used to train the kernel correlation filter respectively;
[0065] S3.1: Input target features to train the kernel correlation filter;
[0066] In the process of training the kernel correlation filter, for the feature φ with the number of channels K, the range W×H, and the unit size C, the feature x φ The size is Its objective function is as follows:
[0067]
[0068] Among them, y φ is x φ The regression results of
[0069] If the vehicle target sample is x=[x 1 ,x 2 ,x 3 ,…,x n ], then the circulant matrix C(x) is generated by cyclic shift, which is expressed as follows:
[0070]
[0071] Diagonalizing the matrix yields:
[0072]
[0073] in, is the DFT transform of x, F is the Fourier matrix, F H is the conjugate transpose of F;
[0074] The Gaussian kernel function is used to transform the vehicle target sample from the low-dimensional space to the high-dimensional space. The Gaussian kernel function is expressed as follows:
[0075]
[0076] Among them, k xx′ is the Gaussian kernel, x′ represents the image to be detected, represents the DFT transform of the image to be detected, is x * DFT transform, σ represents the bandwidth of the Gaussian kernel function;
[0077] For most kernel functions, This can be represented using a circulant matrix:
[0078]
[0079] Among them, λ is the regularization coefficient, is the DFT transform of the filter coefficients, is the self-kernel correlation of vector x in the Fourier domain;
[0080] The kernel correlation filters of HOG features and FUS features are trained separately, and the responses of the two features are:
[0081]
[0082] in, and Represent the correlation filters of two feature trainings, and Represents the Gaussian kernel between the target sample and the candidate sample of the two features, f HOG(Z) and f FUS(Z) Respectively represent the relevant responses;
[0083] S3.2: Perform weighted fusion on the responses of the two features to obtain the final response;
[0084] According to the design based on PSR (peak sidelobe ratio) and the target response maximum value f max (t) method to calculate the feature fusion weight, the process is as follows:
[0085] First normalize the two and multiply them to get the feature confidence score It is expressed as:
[0086]
[0087] Among them, f max ( i) Indicates the maximum target response of the i-th frame, PSR (i) represents the peak sidelobe ratio of the i-th frame;
[0088] Based on feature confidence score The feature fusion weights of HOG features and FUS features can be defined as:
[0089]
[0090] η HOG =min(ξ HOG ,0.6);
[0091] η FUS =1-η HOG ;
[0092] Among them, η HOG and η FUS They are the fusion weights of HOG features and FUS features respectively;
[0093] The final response of the target vehicle can be obtained by weighting the responses of the two features, expressed as:
[0094] f(z)=η HOG f HOG (z)+η FUS f FUS (z);
[0095] S3.3: Update the filter model
[0096] During the tracking process, drift will occur when the filter model is updated, so the exponential decay method is used to weight the filter model. The exponential decay learning rate θ is expressed as:
[0097]
[0098] Among them, β is the basic learning rate, is the confidence score of the feature response graph, σ is the set threshold, and ε is the attenuation coefficient; the model update formula of feature φ is:
[0099]
[0100] Among them, θ φ is the exponential decay learning rate of feature φ, the current frame is t, and is the appearance model and filter model of the previous frame, and Appearance features and filter models calculated for the current frame.
[0101] The fused features obtained in step S3 are input into the kernel correlation filter for training. The description of the appearance of the vehicle target by a single feature is limited and cannot show more information of the target image. In addition, due to the complex background environment in which the vehicle target is located, the target image information may be missing or blurred. The method in the above process is adopted to extract the HOG, LBP and SIFT feature information of the vehicle target, train the kernel filter with the two fused features respectively, and perform weighted fusion on the two obtained responses, which can effectively overcome the problem of missing and blurred target image information or complex background information. S4: Perform sub-pixel positioning on the response map obtained by training. The specific steps include:
[0102] After obtaining the relevant response graph of the vehicle target, the peak position of the response in the current frame is obtained as m according to the response graph, and the peak value of the response is f(m). Then the true peak position of the response should be m+ε, and the true peak value of the response should be f(m+ε), where ε is the estimated deviation value.
[0103] The peak value of the correlation response can be approximately expressed by Taylor's formula, and the estimated deviation value ε can be obtained at the same time. The peak value of the correlation response is expressed as:
[0104] f(m+ε)=f(m)+f′(m)ε+(ε 2 );
[0105] Taking the first-order derivative of both sides of the expression at the peak of the continuous response, we can obtain:
[0106] f′(m+ε)=f′(m)+f″(m)ε;
[0107]
[0108] Among them, α is the step size, and the value is 0.08;
[0109] The true peak position of the response is:
[0110]
[0111] Among them, x m ,y m is the horizontal and vertical coordinate value of the peak value of the response in the response graph, x t ,y t is the horizontal and vertical coordinate value of the true response peak, ε x , ε y is the deviation value of ε on the x and y axes.
[0112] The final position of the vehicle target is determined by estimating the maximum position of the relevant response. However, the calculation of the maximum position may be affected by the accumulated error of the vehicle target moving frame by frame, causing the tracking frame to drift and reduce the tracking accuracy. The proposed sub-pixel positioning method can correct the deviation and calculate the position of the true peak in the response, thereby improving the tracking accuracy.
[0113] S5: Output the vehicle position for subsequent tracking. The specific steps include: taking the vehicle position after sub-pixel positioning as the vehicle position of the current frame, and repeating the above steps S2, S3 and S4 for subsequent tracking.
[0114] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included in the scope of protection of the present invention.
Claims
1. A vehicle tracking method for open-pit coal mine based on feature fusion, characterized in that: The following steps are involved: S1: Obtain the monitoring video of the open-pit coal mine vehicle and obtain the initial frame position and size of the target vehicle; S2: extract the directional gradient histogram features, local binary pattern features and SIFT features of the target vehicle, and preprocess the features; S3: Perform feature fusion according to the adaptive feature fusion method, and input the fused features into the kernel correlation filter for training; S4: Perform sub-pixel positioning on the response map obtained through training; S5: Output the vehicle position for subsequent tracking.
2. The open-pit mine vehicle tracking method based on feature fusion according to claim 1, characterized in that: In step S1, a monitoring video of an open-pit coal mine vehicle is obtained, and the monitoring video is a video within a set time period; Obtaining the initial frame position of the target vehicle means selecting a frame in which the vehicle target is clearer in the first few frames of the video as the initial frame for tracking, marking the target in the initial frame with a rectangular frame, and obtaining the position and size of the vehicle target through the position and size of the rectangular frame.
3. The open-pit mine vehicle tracking method based on feature fusion according to claim 1 is characterized in that: The preprocessing described in step S2 refers to: using the principal component analysis method to reduce the dimension of the directional gradient histogram features, local binary pattern features and SIFT features, and then performing feature extraction after the dimension reduction.
4. The open-pit mine vehicle tracking method based on feature fusion according to claim 1, characterized in that: The feature fusion described in step S3 refers to: fusing the LBP feature and SIFT feature of the target in parallel to obtain the FUS feature, and then training the kernel correlation filter with the HOG feature and the FUS feature respectively.
5. The open-pit mine vehicle tracking method based on feature fusion according to claim 1, characterized in that: Step S3, in which the fused features are input into the kernel correlation filter for training, comprises the following steps: S3.1: Input target features to train the kernel correlation filter; S3.2: Perform weighted fusion on the responses of the two features to obtain the final response; S3.3: Update the filter model.
6. The open-pit mine vehicle tracking method based on feature fusion according to claim 5, characterized in that: The steps of inputting the target features in step S3.1 to train the kernel correlation filter are as follows: In the process of training the kernel correlation filter, for the feature φ with the number of channels K, the range W×H, and the unit size C, the feature x φ The size is Its objective function is as follows: If the vehicle target sample is x=[x1,x2,x3,…,x n ], then the circulant matrix C(x) is generated by cyclic shift, which is expressed as follows: Diagonalizing the matrix yields: in, is the DFT transform of x, F H is the conjugate transpose of F; The Gaussian kernel function is used to transform the vehicle target sample from the low-dimensional space to the high-dimensional space. The Gaussian kernel function is expressed as follows: Among them, x′ represents the image to be detected, σ represents the bandwidth of the Gaussian kernel function; For most kernel functions, This can be represented using a circulant matrix: The kernel correlation filters of HOG features and FUS features are trained separately, and the responses of the two features are: in, and Represent the correlation filters of two feature trainings, and Represents the Gaussian kernel between the target sample and the candidate sample of the two features, f HOG(Z) and f FUS(Z) Respectively represent relevant responses.
7. The open-pit mine vehicle tracking method based on feature fusion according to claim 5, characterized in that: The steps of weighted fusion of the responses of the two features described in step S3.2 to obtain the final response are as follows: According to the designed peak sidelobe ratio PSR and the target response maximum value f max (t) method to calculate the feature fusion weight and obtain the feature confidence score The feature fusion weights of HOG features and FUS features can be defined as: or HOG =min(ξ HOG ,0.6); or FUS =1st HOG ; The final response of the target vehicle can be obtained by weighting the responses of the two features, expressed as: f(z)=η HOG f HOG (z)+η FUS f FUS (z)。 8. The open-pit mine vehicle tracking method based on feature fusion according to claim 5, characterized in that: The steps for updating the filter model described in step S3.3 are as follows: During the tracking process, drift will occur when the filter model is updated. The exponential decay method is used to weight the filter model for update. The model update formula of the feature φ is: Among them, θ φ is the exponential decay learning rate of feature φ, the current frame is t, and is the appearance model and filter model of the previous frame, and Appearance features and filter models calculated for the current frame.
9. The open-pit mine vehicle tracking method based on feature fusion according to claim 1, characterized in that: The step of performing sub-pixel positioning on the response map obtained through training in S4 comprises: The peak value of the correlation response is approximated by the Taylor formula, and the estimated deviation value ε is obtained at the same time. The peak value of the correlation response is expressed as: f(m+ε)=f(m)+f′(m)ε+(ε 2 ); Taking the first-order derivative of both sides of the expression at the peak of the continuous response, we can obtain: f′(m+ε)=f′(m)+f″(m)ε; Among them, α is the step size, and the value is 0.08; The true peak position of the response is: Among them, x t ,y t is the horizontal and vertical coordinate value of the true response peak, ε x , ε y is the deviation value of ε on the x and y axes.