Unmanned mine car front car detection method based on visual tracking

Abstract

The invention provides an unmanned mine car front car detection method based on visual tracking, and the method comprises the following steps: S1, front car detection: collecting an image through a vehicle-mounted camera, locking all front cars from a historical video frame through a front car detection algorithm, and recording the front cars as detection cars; predicting and tracking the state of the detection vehicle at the current moment, and recording the state as vehicle prediction; s2, data association: performing ID matching on the detected vehicle and the vehicle prediction by using a data association algorithm; s3, state prediction updating: updating the tracking vehicle state corresponding to the detection vehicle according to the obtained pairing relationship; and S4, tracking the vehicle: transmitting the information of the tracked vehicle to a path planning module. Through a front vehicle detection algorithm, the detection precision can be greatly improved, the false alarm rate and the missing report rate of target detection can be reduced, and the robustness of a sensing function in an unmanned driving system in a complex and severe environment can be improved; and meanwhile, the requirement of real-time tracking is met.

Description

Technical Field

[0001] The present invention relates to the technical field of vision-assisted automatic driving, and in particular to a method for detecting a leading vehicle of an unmanned mining vehicle based on vision tracking. Background Art

[0002] According to statistics, rear-end collisions are the main safety issue for driverless cars. Vehicle detection and tracking technology based on camera vision is one of the safety assistance technologies to prevent forward vehicle collisions. Accurate vehicle tracking is closely related to collision warning. By analyzing the position and movement status of the vehicle in front in real time, driverless cars can make corresponding driving decisions to ensure driving safety.

[0003] In recent years, the visual front vehicle tracking technology based on vehicle-mounted cameras has been successfully applied in autonomous driving L2 and above, which is of great significance for the distance judgment and collision warning of the vehicle in front of the unmanned car. The traditional visual tracking method is mainly divided into generative and discriminative methods, specifically:

[0004] The generative approach mainly describes the appearance features of the target by constructing a target model. Representative algorithms include sparse coding, online density estimation, and principal component analysis. However, these methods usually have high accuracy but high computational complexity, making it difficult to meet real-time requirements.

[0005] The discriminative method transforms the front vehicle tracking into a binary classification problem by training a classifier to distinguish between the front vehicle and the background. Although this method has lower computational complexity, it is easily disturbed in complex scenes, resulting in tracking failure.

[0006] Although the existing visual vehicle tracking technology has achieved certain results, it still faces many challenges in practical applications. Especially in the application scenario of unmanned mining vehicles in open-pit mines, how to maintain tracking accuracy under bad weather and complex lighting conditions has become an urgent problem to be solved. In addition, due to the limited computing power of the on-board computing platform, how to ensure high-speed, low-latency processing capabilities while meeting the needs of real-time tracking is also a major problem facing current technology. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a method for detecting a leading vehicle of an unmanned mining vehicle based on visual tracking, the method comprising the following steps:

[0008] S1. Front vehicle detection: Use the vehicle-mounted camera to collect images, and use the front vehicle detection algorithm to lock all front vehicles from historical video frames, which are recorded as detected vehicles; predict and track the status of the detected vehicles at the current moment, which is recorded as vehicle prediction;

[0009] S2, data association: using a data association algorithm to perform ID matching between the detected vehicle and the predicted vehicle;

[0010] S3, state prediction update: updating the tracking vehicle state corresponding to the detected vehicle according to the obtained pairing relationship;

[0011] S4. Tracking vehicle: passing the information of the tracked vehicle to the path planning module.

[0012] Furthermore, the output of the front vehicle detection algorithm includes the vehicle category, the center of the vehicle captured image, the vehicle image scaling scale, and the rotation angle. The output of the front vehicle detection algorithm is represented by a state vector S as follows:

[0013] S = [classID, c x , c y ,w,h,θ] T

[0014] Among them, classID represents the category of the preceding vehicle, (c x , c y ) represents the center coordinates of the front vehicle image, (w, h) represents the width and height of the front vehicle image frame, and θ represents the rotation angle of the front vehicle image relative to the vehicle image.

[0015] Furthermore, the S2 further includes:

[0016] S21. The optimization model of the data association algorithm satisfies the following expression:

[0017]

[0018] Among them, w(S i , T j ) indicates S i With T j The similarity of j represents the predicted state vector of the vehicle i, T j represents the state vector of the vehicle j in front of the measurement;

[0019] The constraints of the optimization model of the data association algorithm are:

[0020]

[0021] Wherein, m(i, j) represents the matching relationship between the predicted front vehicle i and the measured front vehicle j, m(i, j)=1 represents a successful match, and m(i, j)=0 represents an unsuccessful match;

[0022] S22, the distance calculation formula between the predicted front vehicle and the measured front vehicle satisfies:

[0023]

[0024] in, d(S i , T j ) indicates S i With T i The Euclidean distance between the subvectors of the two state vectors after removing the classID component;

[0025] The S i With T i The weight w(S i , T j )The calculation formula of similarity is:

[0026]

[0027] Among them, d max is the maximum distance threshold set in advance;

[0028] If the preceding vehicle is of a different type or the preceding vehicle is farther away than d max When the preceding vehicle fails to match, that is, w(S i , T j )=-1;

[0029] If the preceding vehicle is of the same type and the distance to the preceding vehicle is not greater than d max When w(S i , T j ) is negatively correlated with the distance between the two.

[0030] Furthermore, the S3 also includes:

[0031] S31, the transfer equation for tracking the vehicle state satisfies the formula:

[0032]

[0033] in, Indicates the speed of the image center in the x and y directions, represents the angular velocity of the acquired image frame, represents the transferred Gaussian noise, and the vector subscript t represents the time corresponding to the tth frame of the video;

[0034] The observation equation of the front vehicle motion satisfies the formula:

[0035]

[0036] in, represents the center and turning angle measurements of the front vehicle at the t frame time obtained by the front vehicle detection algorithm, To measure Gaussian noise.

[0037] Further, the S4 includes:

[0038] The tracked front vehicle is divided into potential front vehicles and real front vehicles; wherein the potential front vehicle is detected continuously but the number of times does not reach T lock The potential front vehicle is detected continuously T lock The frame is considered to be real, that is, it becomes the real preceding vehicle;

[0039] If the real front vehicle fails to match the detected front vehicle, it is considered that the front vehicle has driven out or failed to track or was missed;

[0040] If the real preceding vehicle continues T lost When frame matching fails, it is considered that tracking is lost; wherein, the T lock With T lost The value is set manually.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention can greatly improve the detection accuracy through the leading vehicle detection algorithm, reduce the false alarm rate and missed alarm rate of target detection, and still maintain tracking accuracy in open-pit mines, thereby improving the robustness of the perception function in the mine unmanned driving system; at the same time, under the condition of limited computing power of the on-board computing platform, it can achieve high-speed and low-latency processing capabilities to meet the needs of real-time tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The present invention is a flowchart of the present invention. DETAILED DESCRIPTION

[0044] In order to make the technical scheme and technical effect of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0045] The present invention aims to provide a method for detecting a leading vehicle in an unmanned mining vehicle based on visual tracking, so as to improve the tracking accuracy in complex and harsh environments, and meet the needs of real-time tracking under the condition of limited computing power of the on-board computing platform.

[0046] refer to Figure 1 The method provided by the present invention mainly comprises the following steps:

[0047] The onboard camera of the unmanned mining car continuously collects images and outputs the real state vector of the front vehicle at the tth frame. Among them, the front vehicle tracking refers to locking the front vehicle from the historical video, including the dangerous vehicle set and the safe vehicle set. Among them, the dangerous vehicle set and the safe vehicle set in the initial stage are both empty sets.

[0048] The vision-based front vehicle tracking task consists of four steps: front vehicle detection, data association, state prediction update, and vehicle tracking. Specifically:

[0049] S1, front vehicle detection

[0050] The vehicle-mounted camera is used to collect images, and the front vehicle detection algorithm is used to lock all the front vehicles from the historical video frames, which are recorded as detected vehicles; the state of the detected vehicle at the current moment is predicted and tracked, which is recorded as vehicle prediction.

[0051] The output of the front vehicle detection algorithm includes vehicle category, vehicle captured image center, vehicle image scaling, rotation angle, etc.

[0052] The output of the front vehicle detection algorithm is expressed as a state vector S:

[0053] S = [classID, c x , c y ,w,h,θ] T

[0054] Among them, classID represents the category of the preceding vehicle, (c x , c y ) represents the center coordinates of the front vehicle image, (w, h) represents the width and height of the front vehicle image frame, and θ represents the rotation angle of the front vehicle image relative to the vehicle image.

[0055] S2. Data Association

[0056] The current state of the detected vehicle in the prediction tracking S1 is called vehicle prediction. The detected vehicle is matched with the vehicle prediction by ID using the data association algorithm.

[0057] Furthermore, the preceding vehicle prediction data is associated with the measured data through maximum weight matching of a weighted bipartite graph; wherein, preceding vehicle prediction refers to identifying the vehicle from the image frame captured by the current camera; preceding vehicle tracking refers to predicting and tracking the preceding vehicle at the current moment by locking the preceding vehicle in the historical video.

[0058] S21. The optimization model of the data association algorithm satisfies the following expression:

[0059]

[0060] Among them, w(S i , T j ) indicates S i With T j The similarity of j represents the predicted state vector of the vehicle i, T j represents the state vector of the vehicle j in front of the measurement;

[0061] The constraints of the optimization model of the data association algorithm are:

[0062]

[0063] Among them, m(i, j) represents the matching relationship between the predicted front vehicle i and the measured front vehicle j, m(i, j) = 1 represents a successful match, and m(i, j) = 0 represents an unsuccessful match; the objective function is to maximize the sum of the similarities of the matching results.

[0064] S22. The distance calculation formula between the predicted front vehicle and the measured front vehicle satisfies:

[0065]

[0066] in, d(S i , T j ) indicates S i With T i The Euclidean distance between the subvectors of two state vectors after removing the classID component.

[0067] S i With T i The weight w(S i , T j )The calculation formula of similarity is:

[0068]

[0069] Among them, d max is the maximum distance threshold set in advance;

[0070] If the preceding vehicle is of a different type or the preceding vehicle is farther away than d max When the preceding vehicle fails to match, that is, w(S i , T j )=-1;

[0071] If the preceding vehicle is of the same type and the distance to the preceding vehicle is not greater than d max When w(S i , T j ) is negatively correlated with the distance between the two.

[0072] S3. Status prediction update

[0073] According to the obtained pairing relationship, the tracking vehicle status corresponding to the detected vehicle is updated.

[0074] The center coordinates of the front vehicle image (c x , c y ) changes with the rotation angle θ at a constant speed, tracking the transfer equation of the vehicle state, satisfying the formula:

[0075]

[0076] in, Indicates the speed of the image center in the x and y directions, represents the angular velocity of the acquired image frame, represents the transferred Gaussian noise, and the vector subscript t represents the time corresponding to the tth frame of the video;

[0077] The observation equation of the front vehicle motion satisfies the formula:

[0078]

[0079] in, represents the center and turning angle measurements of the front vehicle at the t frame time obtained by the front vehicle detection algorithm, To measure Gaussian noise.

[0080] S4. Tracking Vehicles

[0081] Pass the tracking vehicle information to the path planning module.

[0082] In order to solve the problem of false detection and missed detection of the front vehicle with collision risk in real time, the tracked front vehicle is divided into potential front vehicle and real front vehicle; among them, the potential front vehicle is the one that is detected continuously but the number of times does not reach T. lock The vehicle ahead may be a real vehicle with a collision risk, or a falsely detected vehicle without a collision risk. lock The frame after that is considered to be real, that is, it becomes the real preceding vehicle.

[0083] If the real front vehicle fails to match the detected front vehicle, it is considered that the front vehicle has driven out or failed to track or was missed;

[0084] If the real front vehicle continues T lost When frame matching fails, it is considered as tracking loss; where T lock With T lost The value can be set manually.

[0085] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting a leading vehicle of an unmanned mining vehicle based on visual tracking, characterized in that: The method comprises the following steps: S1. Front vehicle detection: Use the vehicle-mounted camera to collect images, and use the front vehicle detection algorithm to lock all front vehicles from historical video frames, which are recorded as detected vehicles; predict and track the status of the detected vehicles at the current moment, which is recorded as vehicle prediction; S2, data association: using a data association algorithm to perform ID matching between the detected vehicle and the predicted vehicle; S3, state prediction update: updating the tracking vehicle state corresponding to the detected vehicle according to the obtained pairing relationship; S4. Tracking vehicle: passing the information of the tracked vehicle to the path planning module.

2. The method for detecting the leading vehicle of an unmanned mining vehicle based on visual tracking according to claim 1 is characterized in that: The output of the front vehicle detection algorithm includes the vehicle category, the center of the vehicle acquisition image, the vehicle image scaling scale, and the rotation angle. The output of the front vehicle detection algorithm is represented by a state vector S as follows: S=[classID,c x ,c y ,w,h,θ] T Among them, classID represents the category of the preceding vehicle, (c x , c y ) represents the center coordinates of the front vehicle image, (w, h) represents the width and height of the front vehicle image frame, and θ represents the rotation angle of the front vehicle image relative to the vehicle image.

3. The method for detecting the leading vehicle of an unmanned mining vehicle based on visual tracking according to claim 2 is characterized in that: The S2 further includes: S21. The optimization model of the data association algorithm satisfies the following expression: Among them, w(S i , T j ) indicates S i With T j The similarity of j represents the predicted state vector of the vehicle i, T j represents the state vector of the vehicle j in front of the measurement; The constraints of the optimization model of the data association algorithm are: Wherein, m(i, j) represents the matching relationship between the predicted front vehicle i and the measured front vehicle j, m(i, j)=1 represents a successful match, and m(i, j)=0 represents an unsuccessful match; S22, the distance calculation formula between the predicted front vehicle and the measured front vehicle satisfies: in, d(S i , T j ) indicates S i With T i The Euclidean distance between the subvectors of the two state vectors after removing the classID component; The S i With T i The weight w(S i , T j )The calculation formula of similarity is: Among them, d max is the maximum distance threshold set in advance; If the preceding vehicle is of a different type or the preceding vehicle is farther away than d max When the preceding vehicle fails to match, that is, w(S i , T j )=-1; If the preceding vehicle is of the same type and the distance to the preceding vehicle is not greater than d max When w(S i , T j ) is negatively correlated with the distance between the two.

4. The method for detecting the leading vehicle of an unmanned mining vehicle based on visual tracking according to claim 3 is characterized in that: The S3 further includes: S31, the transfer equation for tracking the vehicle state satisfies the formula: in, Indicates the speed of the image center in the x and y directions, represents the angular velocity of the acquired image frame, represents the transferred Gaussian noise, and the vector subscript t represents the time corresponding to the tth frame of the video; The observation equation of the front vehicle motion satisfies the formula: in, represents the center and turning angle measurements of the front vehicle at the t frame time obtained by the front vehicle detection algorithm, To measure Gaussian noise.

5. The method for detecting the leading vehicle of an unmanned mining vehicle based on visual tracking according to claim 4 is characterized in that: The S4 includes: The tracked front vehicle is divided into potential front vehicles and real front vehicles; wherein the potential front vehicle is detected continuously but the number of times does not reach T lock The potential front vehicle is detected continuously T lock The frame is considered to be real, that is, it becomes the real preceding vehicle; If the real front vehicle fails to match the detected front vehicle, it is considered that the front vehicle has driven out or failed to track or was missed; If the real preceding vehicle continues T lost When frame matching fails, it is considered that tracking is lost; wherein, the T lock With T lost The value is set manually.

Images