Monocular ranging system and method for unmanned downhole vehicle

Abstract

This invention discloses a monocular ranging system and method for underground unmanned vehicles, comprising the following steps: During the vehicle's travel route, images of the preceding vehicle are acquired via sensors, and detection box information and the type information of the preceding vehicle are automatically identified; the vehicle pose of the preceding vehicle is determined based on the type information and detection box information; the pixel coordinates of the closest point on the preceding vehicle to the sensor are obtained in the preceding vehicle image based on the vehicle pose; the measured distance between the closest point and the sensor is obtained, and the measured distance is then predicted and updated to obtain the vehicle distance. By real-time perception of the vehicle body, rear, and wheel hubs, a multi-target information fusion method is proposed based on the detection results, providing a prerequisite for accurate ranging. Then, a two-stage accurate ranging method is adopted, combining a traditional geometric ranging model with a Kalman filter algorithm to further improve the accuracy of the final result, thereby improving the stability and robustness of the ranging of the underground unmanned vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned perception in a mine, and in particular to a monocular distance measuring system and method for an unmanned vehicle in a mine. BACKGROUND

[0002] In the face of growing energy demand, high labor costs due to skilled labor shortages, and severe international competitive pressures, mining companies and energy groups are actively exploring the intelligent construction of mining areas. Due to the harsh working environment in the mining area and the need for around-the-clock operation, unmanned transportation technology in the mining area has become a key point in promoting intelligent construction. Currently, there are many problems in underground transportation, such as difficulty in recruiting workers, low efficiency, and high safety risks.

[0003] Unmanned driving is the current mainstream research direction, and unmanned transportation in the mining area can fundamentally solve the safety problem of driving personnel, reduce labor costs, and greatly improve production and transportation efficiency. In particular, for underground mines, compared to open-pit mines, the working environment is more severe and the labor cost is higher, so the demand for intelligent and unmanned transportation is also stronger. The perception system, especially the monocular vision distance measuring technology for unmanned vehicles in a mine, has important research value.

[0004] Current monocular distance measuring methods are mainly used for urban roads and highways. Patent CN113720300A proposes a monocular distance measuring method based on a target recognition neural network. This method obtains the distance by bringing the detection results into the optical relationship formula after collecting the near and far images of the same target. Patent CN111046843A proposes a monocular distance measuring method in an intelligent driving environment. This method uses a detection box to segment and extract numerical information from the object to be measured. Patent CN114046769A proposes a monocular distance measuring method based on multi-dimensional reference information. This method uses the vanishing point and the similar triangle method to measure the distance of the detected object. However, the above monocular distance measuring methods have some problems: (1) The underground roadway is mostly irregular, and the uneven ground conditions cause large errors in distance measurement; (2) Since the target detection box is two-dimensional, it is impossible to obtain accurate depth information of the vehicle relying on a single detection box. SUMMARY

[0005] The present application aims to provide a monocular distance measuring system and method for an unmanned vehicle in a mine to solve or improve at least one of the above technical problems.

[0006] Therefore, the first aspect of the present application provides a monocular distance measuring method for an unmanned vehicle in a mine.

[0007] The second aspect of the present application provides a monocular distance measuring system for an unmanned vehicle in a mine.

[0008] The first aspect of the present application provides a monocular ranging method of an unmanned downhole vehicle, comprising the following steps: S1, in the vehicle driving route, collecting a forward vehicle image through a sensor, and automatically identifying detection box information and type information of the forward vehicle; S2, judging the vehicle pose of the forward vehicle through the type information and the detection box information; S3, obtaining the pixel coordinates of the closest point on the forward vehicle to the sensor in the forward vehicle image according to the vehicle pose; S4, bringing the pixel coordinates into a monocular geometric ranging model to obtain the measured distance of the closest point to the sensor, and then using a Kalman filtering algorithm to predict and update the vehicle distance; wherein the type information includes body information, tail information and hub information, and each corresponds to the detection box information one by one.

[0009] The present application provides a monocular ranging system and method of an unmanned downhole vehicle, which is applied to solve the above problems. The present application designs a multi-information fusion method based on a convolutional neural network. The method uses a convolutional neural network to detect vehicle information in real time, and performs information fusion according to the detection results, thereby providing conditions for subsequent acquisition of vehicle pose information. In addition, the present application designs a pose estimation module to judge the vehicle pose and obtain the pixel coordinates of the closest point of the forward vehicle. Finally, a two-stage precise ranging method is designed. The vehicle distance is first obtained based on a monocular geometric ranging model, and then the distance is predicted and updated using a Kalman filter, thereby ensuring the robustness and stability of the ranging.

[0010] The present application provides a monocular ranging system and method of an unmanned downhole vehicle, which is applied to solve the above problems. The present application designs a multi-information fusion method based on a convolutional neural network. The method uses a convolutional neural network to detect vehicle information in real time, and performs information fusion according to the detection results, thereby providing conditions for subsequent acquisition of vehicle pose information. In addition, the present application designs a pose estimation module to judge the vehicle pose and obtain the pixel coordinates of the closest point of the forward vehicle. Finally, a two-stage precise ranging method is designed. The vehicle distance is first obtained based on a monocular geometric ranging model, and then the distance is predicted and updated using a Kalman filter, thereby ensuring the robustness and stability of the ranging.

[0011] The present application provides a monocular ranging system and method of an unmanned downhole vehicle, which is applied to solve the above problems. The present application designs a multi-information fusion method based on a convolutional neural network. The method uses a convolutional neural network to detect vehicle information in real time, and performs information fusion according to the detection results, thereby providing conditions for subsequent acquisition of vehicle pose information. In addition, the present application designs a pose estimation module to judge the vehicle pose and obtain the pixel coordinates of the closest point of the forward vehicle. Finally, a two-stage precise ranging method is designed. The vehicle distance is first obtained based on a monocular geometric ranging model, and then the distance is predicted and updated using a Kalman filter, thereby ensuring the robustness and stability of the ranging.

[0012] The present application provides a monocular ranging system and method of an unmanned downhole vehicle, which is applied to solve the above problems. The present application designs a multi-information fusion method based on a convolutional neural network. The method uses a convolutional neural network to detect vehicle information in real time, and performs information fusion according to the detection results, thereby providing conditions for subsequent acquisition of vehicle pose information. In addition, the present application designs a pose estimation module to judge the vehicle pose and obtain the pixel coordinates of the closest point of the forward vehicle. Finally, a two-stage precise ranging method is designed. The vehicle distance is first obtained based on a monocular geometric ranging model, and then the distance is predicted and updated using a Kalman filter, thereby ensuring the robustness and stability of the ranging.

[0013] In addition, the technical scheme provided by the embodiments of the present application can have the following additional technical features:

[0014] In any of the above technical solutions, the forward vehicle image is obtained frame by frame from a real-time video stream captured by the sensor; the sensor is installed on the windshield of the vehicle, and the image acquisition surface is set in the same direction as the vehicle's driving direction; wherein, the sensor is an infrared camera.

[0015] In this technical solution, sensors can continuously capture images of vehicles ahead to obtain real-time images for subsequent algorithm calculations. By acquiring images frame by frame, the most accurate and timely images can be obtained from the continuous video stream, so that the calculated distance can match the actual distance as closely as possible.

[0016] By installing the sensor on the windshield of the vehicle, and specifically inside the vehicle, the windshield provides some dust protection and interference prevention, ensuring that the sensor can work continuously and effectively. Furthermore, by setting the camera acquisition surface in the same direction as the vehicle's driving direction, it is possible to directly capture images of vehicles traveling in front of the vehicle in the same direction as the vehicle's driving direction.

[0017] The use of an infrared camera makes this method applicable to the dim working environment downhole, ensuring the stability of the overall operation.

[0018] In any of the above technical solutions, when vehicle information is detected ahead, it is determined that there is a vehicle moving forward. The step of determining the vehicle position of the vehicle moving forward specifically includes: when only vehicle information is detected, but no rear information or wheel hub information is detected, it is determined that the vehicle moving forward is directly in front of the vehicle; when vehicle information and rear information are detected, but no wheel hub information is detected, it is determined that the vehicle moving forward is directly in front of the vehicle; when vehicle information, rear information, and one wheel hub information are detected simultaneously, it is determined that the vehicle moving forward is directly in front of the vehicle; when vehicle information, rear information, and two wheel hub information are detected simultaneously, it is determined that the vehicle moving forward is located to the side front of the vehicle.

[0019] In this technical solution, when a vehicle detects a vehicle ahead while it is traveling, it is necessary to determine the position and orientation of the vehicle ahead so that the accuracy of the data can be guaranteed in the subsequent distance calculation.

[0020] When only vehicle body information is detected, it is determined that the detected vehicle is in front. However, since the rear of the vehicle and wheel hubs are not detected, the nearest point of the vehicle in front is considered to be the pixel coordinates of the bottom midpoint of the vehicle body detection box.

[0021] When the vehicle body and rear are detected but the wheel hubs are not, it means that the wheel hubs of the vehicle in front are obscured by the vehicle body in the forward-looking direction of the vehicle. Therefore, it is determined that the vehicle in front detected at this time is located directly in front of the vehicle.

[0022] When vehicle body information, rear information, and wheel hub information are detected simultaneously, it means that one wheel hub of the vehicle in front is deflected relative to the vehicle body at an angle. Therefore, the vehicle's sensors only detect one wheel hub, which means that the vehicle in front is at the side front of the vehicle, but the angle is small.

[0023] When vehicle body information, rear information, and two wheel hub information are detected simultaneously, it indicates that the forward vehicle's direction of travel is different from that of the vehicle in front. Therefore, both wheel hubs on one side are detected, indicating that the forward vehicle is located at the side front of the vehicle body.

[0024] In any of the above technical solutions, when it is determined that the forward vehicle is located at the side front of the vehicle, the pixel coordinates p1(x1,y1) and p2(x2,y2) of the bottom midpoint of the detection box information of the two wheels are obtained; when (x1-x2)(y1-y2)<0, it is determined that the vehicle is at the left front; when (x1-x2)(y1-y2)>0, it is determined that the vehicle is at the right front.

[0025] In this technical solution, when the forward vehicle is located to the side front of the vehicle, it is necessary to determine the left and right deviation of the forward vehicle. By comparing the pixel coordinates of the bottom midpoint of the detection box information of the two wheels, the left and right deviation of the vehicle body can be determined. That is, when (x1-x2)(y1-y2)<0, it is determined that the vehicle is to the left front; when (x1-x2)(y1-y2)>0, it is determined that the vehicle is to the right front.

[0026] In any of the above technical solutions, let the pixel coordinates of the nearest point be A(x) a ,y a S3 specifically includes: when the forward vehicle is directly in front of the vehicle, obtaining the pixel coordinates of the bottom midpoint of the detection frame information of the vehicle body information as the pixel coordinates of the nearest point A; when the forward vehicle is to the side front of the vehicle, obtaining the pixel coordinates Q(x) of the lower right or lower left corner of the detection frame information of the rear of the vehicle at this time. q ,y q At this point, the lines containing p1 and p2 are:

[0027]

[0028] The coordinates of the nearest point A are:

[0029] In this technical solution, when the forward vehicle is directly in front of the vehicle, the pixel coordinates of the bottom midpoint of the detection box information corresponding to the vehicle body information are directly set as the nearest point. When the forward vehicle is to the left front of the vehicle, the lower right pixel coordinates of the detection box information corresponding to the rear information are obtained, or when the forward vehicle is to the right front of the vehicle, the lower left pixel coordinates of the detection box information corresponding to the rear information are obtained. Based on the above formula, the coordinates of the nearest point when the forward vehicle is at the front left or right front of the vehicle are calculated.

[0030] By employing different selection methods for the nearest point based on the different attitudes of the vehicle in front relative to the vehicle, the final location information can be made more accurate and more adaptable.

[0031] In any of the above technical solutions, in step S4, the nearest point A(x) is... a ,y a Substituting the pixel coordinates of () into the monocular geometric ranging model, the following formula can be obtained:

[0032]

[0033] h is the height of the sensor above the ground; γ is the downward angle of the sensor; y a It is the vertical coordinate of the point where the forward-moving vehicle is closest to the sensor in the pixel coordinates of the image; v0, f v These are the camera's internal parameters, where v0 is the pixel ordinate corresponding to the principal point, and f... v The length of the focal length in the y-axis direction is described using pixels and can be obtained by Zhang Zhengyou's calibration method; d is the distance to be calculated.

[0034] This technical solution utilizes the existing monocular geometric ranging model, which allows for the derivation of the expression for the measured distance d, and the solution to obtain the specific value of d based on the expression.

[0035] In any of the above technical solutions, the Kalman filter algorithm includes: a prediction part model and an update part model, wherein the output of the prediction part model is used as the input condition of the update part model.

[0036] In this technical solution, the calculated measurement value d is further refined by predicting a partial model and updating a partial model, thereby improving the accuracy of the final output value.

[0037] In any of the above technical solutions, the prediction model includes the following formula:

[0038]

[0039] P t - =FP t-1 FT +Q; (2)

[0040] in, Let u be the state variable predicted at time t. t Let F be the control variable at time t, F be the state matrix, and B be the control matrix. Let P be the error covariance predicted at time t. t_1 For the error covariance updated at time t-1, Let Q be the state variable predicted at time t-1, and let Q be the observation noise.

[0041] In this technical solution, formula (1) is the prior estimation formula for the state variable, and the state variable at time t-1... Given that the control matrix B is known, and the control variable u at time t-1 is known. t-1 Given the given conditions, the prior estimate at time t can be obtained. Formula (2) is the formula for estimating the prior error covariance. At time t-1, the error covariance p t-1 Given the state matrix F and the observation noise, the prior error covariance at time t can be calculated.

[0042] In any of the above technical solutions, the updated part of the model includes the following formula:

[0043] K t =P t - H T HP t - H T +R) -1 (3)

[0044]

[0045] P t =(IK t H)P t - (5)

[0046] Among them, P t Let K be the error covariance, Q be the observation noise, and K be the error covariance. t Here, H is the gain coefficient, and z is the gain matrix. t Let I be the observed variable, H be the identity matrix, and I be the identity matrix. T Let H be the transpose of H and R be the noise covariance.

[0047] In this technical solution, formula (3) is the formula for calculating the Kalman gain coefficient, formula (4) is the formula for posterior estimation of the state variables, and formula (5) is the formula for posterior estimation of the error covariance. The gain coefficient K at time t is then calculated. t The observed distance z obtained by the traditional geometric ranging model at time t t and predicted state variables The updated value of the state variable at time t can be calculated. That is, the distance to the nearest point in the final output.

[0048] A second aspect of the present invention provides a monocular ranging system for an unmanned underground vehicle, comprising a storage device and a processor, wherein the storage device stores a computer program, and the processor executes the program to implement the steps of any of the above-described technical solutions.

[0049] In this technical solution, since the processor contained therein can implement the steps of any of the above-described technical solutions and methods, all the technical effects of the monocular ranging system for an unmanned underground vehicle provided by the second aspect of the present invention will not be repeated here.

[0050] Existing monocular ranging methods do not account for errors introduced by changes in vehicle pose and exhibit poor robustness in unstructured underground scenarios. To address these issues, this invention proposes a real-time, accurate monocular ranging method for unmanned underground vehicles, which offers the following advantages compared to other methods:

[0051] This invention uses a target detection algorithm based on a convolutional neural network to achieve real-time perception of the vehicle body, rear of the vehicle, and wheel hubs. Based on the detection results, it proposes a method for multi-target information fusion, which provides a prerequisite for accurate distance measurement.

[0052] This invention designs a vehicle pose estimation module to obtain the relative pose information of the vehicle in front, and calculates the pixel coordinates of the nearest point on the image using the pose information. The design of this module improves the ranging accuracy compared with traditional methods.

[0053] This invention proposes a two-stage accurate ranging method that combines a traditional geometric ranging model with a Kalman filter algorithm, thereby improving the stability and robustness of ranging for unmanned underground vehicles.

[0054] Additional aspects and advantages of embodiments of the invention will become apparent in the following description or may be learned by practice of embodiments of the invention. Attached Figure Description

[0055] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] Figure 1 This is a flowchart illustrating the overall solution of the present invention;

[0057] Figure 2 This is a scene diagram for the ranging of the present invention;

[0058] Figure 3 This is a scene diagram of the detection vehicle being directly in front of you according to the present invention;

[0059] Figure 4 This is a scene diagram of the detection vehicle when it is in the left front position according to the present invention;

[0060] Figure 5 This is a scene diagram of the detection vehicle when it is in the right front position according to the present invention;

[0061] Figure 6 This is a flowchart of the multi-information fusion decision-making process of the present invention;

[0062] Figure 7 This is the geometric ranging model for monocular vision in this invention;

[0063] Figure 8 This is a flowchart of the Kalman filtering algorithm of the present invention. Detailed Implementation

[0064] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0065] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0066] Please see Figures 1-8 The first aspect of the present invention provides a monocular ranging method for an unmanned underground vehicle, comprising the following steps: S1, acquiring images of a forward-facing vehicle via a sensor along the vehicle's driving route, and automatically identifying detection box information and type information of the forward-facing vehicle; S2, determining the vehicle pose of the forward-facing vehicle based on the type information and detection box information; S3, obtaining the pixel coordinates of the closest point on the forward-facing vehicle to the sensor in the forward-facing vehicle image based on the vehicle pose; S4, substituting the pixel coordinates into a monocular geometric ranging model to obtain the measured distance between the closest point and the sensor, and then using a Kalman filter algorithm to predict and update the measured distance to obtain the vehicle distance; wherein, the type information includes: vehicle body information, rear information, and wheel hub information, and each corresponds one-to-one with the detection box information.

[0067] This invention provides a monocular ranging system and method for an unmanned underground vehicle. This invention provides a prerequisite for accurate ranging by real-time perception of the vehicle body, rear, and wheel hubs, and proposes a multi-target information fusion method based on the detection results. Specifically, a convolutional neural network target detection algorithm is used to achieve real-time perception of the vehicle body, rear, and wheel hubs.

[0068] This invention designs a method to obtain the relative pose information of the vehicle in front and calculate the pixel coordinates of the nearest point on the image using the pose information. This design improves the ranging accuracy compared to traditional methods. Specifically, it uses a commercially available vehicle pose estimation module to obtain the pose of the vehicle in front of the parking space.

[0069] This invention proposes a two-stage accurate ranging method that combines a traditional geometric ranging model with a Kalman filtering algorithm to further improve the accuracy of the final result of the distance measured by the traditional geometric ranging model, thereby enhancing the stability and robustness of ranging for unmanned underground vehicles.

[0070] Due to the unique characteristics of underground environments, general monocular ranging methods exhibit poor robustness and low accuracy. This invention aims to propose a monocular ranging method applicable to unmanned underground driving scenarios. Specifically, considering the numerous unstructured roads and uneven ground conditions underground, as well as the urgent need for safety assurance and cost reduction, and addressing the instability and limitations of traditional monocular ranging methods, this method designs a multi-information fusion module, a pose estimation module, and a two-stage accurate ranging module based on convolutional neural networks. This solves the problem of accurately perceiving the distance information of vehicles ahead during underground unmanned vehicle operation, thereby enhancing the safety and reliability of underground unmanned transportation.

[0071] In any of the above embodiments, as Figures 1-8 As shown, the forward-facing vehicle image is obtained frame by frame from a real-time video stream captured by a sensor; the sensor is installed on the windshield of the vehicle, and the image acquisition surface is set in the same direction as the vehicle's driving direction; the sensor is an infrared camera.

[0072] In this embodiment, the sensor can continuously capture images of the vehicle in front to obtain real-time images for subsequent algorithm calculations. By acquiring images frame by frame, the most accurate and timely images can be obtained from the continuous video stream, so that the calculated distance can match the actual distance as closely as possible.

[0073] By installing the sensor on the windshield of the vehicle, and specifically inside the vehicle, the windshield provides some dust protection and interference prevention, ensuring that the sensor can work continuously and effectively. Furthermore, by setting the camera acquisition surface in the same direction as the vehicle's driving direction, it can directly capture images of vehicles in front of the vehicle in the same direction as the vehicle's driving direction.

[0074] The use of an infrared camera makes this method applicable to the dim working environment downhole, ensuring the stability of the overall operation.

[0075] In any of the above embodiments, such as Figures 1-8 As shown, when vehicle information is detected ahead, the steps for determining the vehicle's position include: if only vehicle information is detected, but no rear or wheel hub information is detected, the vehicle is determined to be directly in front of the vehicle; if vehicle and rear information are detected, but no wheel hub information is detected, the vehicle is determined to be directly in front of the vehicle; if vehicle, rear, and one wheel hub information are detected simultaneously, the vehicle is determined to be directly in front of the vehicle; and if vehicle, rear, and two wheel hub information are detected simultaneously, the vehicle is determined to be to the side front of the vehicle.

[0076] In this embodiment, when a vehicle detects a vehicle ahead while traveling, it is necessary to determine the pose of the vehicle ahead to ensure the accuracy of the data in subsequent distance calculations.

[0077] When only vehicle body information is detected, it is determined that the detected vehicle is in front. However, since the rear of the vehicle and wheel hubs are not detected, the nearest point of the vehicle in front is considered to be the pixel coordinates of the bottom midpoint of the vehicle body detection box.

[0078] When the vehicle body and rear are detected but the wheel hubs are not, it means that the wheel hubs of the vehicle in front are obscured by the vehicle body in the forward-looking direction of the vehicle. Therefore, it is determined that the vehicle in front detected at this time is located directly in front of the vehicle.

[0079] When vehicle body information, rear information, and wheel hub information are detected simultaneously, it means that one wheel hub of the vehicle in front is deflected relative to the vehicle body at an angle. Therefore, the vehicle's sensors only detect one wheel hub, which means that the vehicle in front is at the side front of the vehicle, but the angle is small.

[0080] When vehicle body information, rear information, and two wheel hub information are detected simultaneously, it indicates that the forward vehicle's direction of travel is different from that of the vehicle in front. Therefore, both wheel hubs on one side are detected, indicating that the forward vehicle is located at the side front of the vehicle body.

[0081] In any of the above embodiments, such as Figures 1-8 As shown, when it is determined that the vehicle in front is located to the side front of the vehicle, the pixel coordinates p1(x1,y1) and p2(x2,y2) of the bottom midpoint of the detection box information of the two wheels are obtained; when (x1-x2)(y1-y2)<0, it is determined that the vehicle is to the left front; when (x1-x2)(y1-y2)>0, it is determined that the vehicle is to the right front.

[0082] In this embodiment, when the forward vehicle is located to the side front of the vehicle, it is necessary to determine the left and right deviation of the forward vehicle. By comparing the pixel coordinates of the bottom midpoint of the detection box information of the two wheels, the left and right deviation of the vehicle body can be determined. That is, when (x1-x2)(y1-y2)<0, it is determined that the vehicle is to the left front; when (x1-x2)(y1-y2)>0, it is determined that the vehicle is to the right front.

[0083] In any of the above embodiments, as Figures 1-8 As shown, let the pixel coordinates of the nearest point be A(x). a ,y a S3 specifically includes: when the vehicle is directly in front of the vehicle, the pixel coordinates of the bottom midpoint of the detection box for obtaining vehicle body information are the pixel coordinates of the nearest point A; when the vehicle is to the side front of the vehicle, the pixel coordinates Q(x) of the bottom right or bottom left corner of the detection box for obtaining rear vehicle information are... q ,y q At this point, the lines containing p1 and p2 are:

[0084]

[0085] The coordinates of the nearest point A are:

[0086] In this embodiment, when the forward vehicle is directly in front of the vehicle, the pixel coordinates of the bottom midpoint of the detection box information corresponding to the vehicle body information are directly set as the nearest point. When the forward vehicle is to the left front of the vehicle, the lower right pixel coordinates of the detection box information corresponding to the rear information are obtained, or when the forward vehicle is to the right front of the vehicle, the lower left pixel coordinates of the detection box information corresponding to the rear information are obtained. The coordinates of the nearest point when the forward vehicle is at the front left or right front of the vehicle are calculated according to the above formula.

[0087] By employing different selection methods for the nearest point based on the different attitudes of the vehicle in front relative to the vehicle, the final location information can be made more accurate and more adaptable.

[0088] In any of the above embodiments, as Figures 1-8 As shown in Figure S4, the nearest point A(x) is... a ,y a Substituting the pixel coordinates of () into the monocular geometric ranging model, the following formula can be obtained:

[0089]

[0090] h is the height of the sensor above the ground; γ is the downward angle of the sensor; y a It is the vertical coordinate of the point where the forward-moving vehicle is closest to the sensor in the pixel coordinates of the image; v0, fv These are the camera's internal parameters, where v0 is the pixel ordinate corresponding to the principal point, and f... v The length of the focal length in the y-axis direction is described using pixels and can be obtained by Zhang Zhengyou's calibration method; d is the distance to be calculated.

[0091] In this embodiment, the existing monocular geometric ranging model is used to derive an expression for the measured distance d, and the specific value of d is solved based on the expression.

[0092] In any of the above embodiments, as Figures 1-8 As shown, the Kalman filter algorithm includes a prediction part model and an update part model, with the output of the prediction part model serving as the input condition for the update part model.

[0093] In this embodiment, the calculated measurement value d is further refined by predicting the partial model and updating the partial model, respectively, so as to improve the accuracy of the final output value.

[0094] In any of the above embodiments, as Figures 1-8 As shown, the prediction part of the model includes the following formulas:

[0095]

[0096] P t - =FP t-1 F T +Q; (2)

[0097] in, Let u be the state variable predicted at time t. t Let F be the control variable at time t, F be the state matrix, and B be the control matrix. Let P be the error covariance predicted at time t. t_1 For the error covariance updated at time t-1, Let Q be the state variable predicted at time t-1, and let Q be the observation noise.

[0098] In this embodiment, formula (1) is the prior estimation formula for the state variable, and the state variable at time t-1 is... Given that the control matrix B is known, and the control variable u at time t-1 is known. t-1 Given the given conditions, the prior estimate at time t can be obtained. Formula (2) is the formula for estimating the prior error covariance. At time t-1, the error covariance p t-1 Given the state matrix F and the observation noise, the prior error covariance p at time t can be calculated. t - .

[0099] In any of the above embodiments, as Figures 1-8 As shown, the updated part of the model includes the following formulas:

[0100] K t =P t - H T HP t - H T +R) -1 (3)

[0101]

[0102] P t =(IK t H)P t - (5)

[0103] Among them, P t For error covariance, K t Here, H is the gain coefficient, and z is the gain matrix. t Let I be the observed variable, H be the identity matrix, and I be the identity matrix. T Let H be the transpose of H and R be the noise covariance.

[0104] In this embodiment, formula (3) is the formula for calculating the Kalman gain coefficient, formula (4) is the formula for posterior estimation of the state variables, and formula (5) is the formula for posterior estimation of the error covariance. The gain coefficient K at time t is then calculated. t The observed distance z obtained by the traditional geometric ranging model at time t t and predicted state variables The updated value of the state variable at time t can be calculated. That is, the distance to the nearest point in the final output.

[0105] In any of the above embodiments, the prediction part model and the update part model are derived using the following steps:

[0106] During parameter selection, when t=0, the default initial distance is... That is, the closest detection distance of the camera (blind zone distance), speed. dt = 0.33, predicting the covariance matrix Distance measurement noise R=1, system process covariance noise

[0107] (1) After obtaining the distance information d, the target's state variable is therefore... Selected as Target observation z tThe distance d is selected as the distance obtained from the first stage measurement, and the sampling period is the time dt for the camera to acquire one frame of image.

[0108] (2) Predict the current distance. The state variable prediction equation is:

[0109]

[0110] Since both the vehicle-mounted camera and the obstacle in front are moving, and their speeds and accelerations differ, the obstacle ranging information is actually the relative distance between them. Idealizing their relative motion as uniformly accelerated motion, we have...

[0111] distance t =distance t-1 +velocity t-1 gdt+agdt 2 / 2

[0112] velocity t =velocity t-1 +agdt

[0113] Where velocity is the relative velocity and a is the relative acceleration. Let u represent distance and v represent velocity, then we have...

[0114]

[0115] at this time, u t =[a].

[0116] (3) Predict the covariance matrix. The error covariance equation is:

[0117] P t - =FP t-1 F T +Q

[0118] Based on prior estimates, there are system parameters.

[0119]

[0120] The system process covariance noise Q is

[0121]

[0122] Since distance noise and velocity noise are independent, we can obtain

[0123]

[0124] Among them, the variances of distance noise and velocity noise are constants, which can be given empirically or calculated.

[0125] Let the previous prediction covariance matrix be...

[0126]

[0127] The predicted covariance matrix is ​​as follows:

[0128]

[0129] P t - and P t-1 Substituting into the formula for predicting covariance, we have

[0130]

[0131] Organizing can yield

[0132]

[0133] (4) Establish the measurement equations. The system measurement equations are as follows:

[0134] z t =Hu t +V

[0135] Because the system output u t Given the distance information to the obstacles, we have H = [1 0].

[0136] (5) Calculate the Kalman gain. The equation for the Kalman gain coefficient is as follows:

[0137] K t =P t - H T HP t - H T +R) -1

[0138] Organizing can yield

[0139]

[0140] The distance measurement noise R is a constant.

[0141] (6) Calculate the current optimal estimate. Based on the optimal estimate equation...

[0142]

[0143] Substitute parameters

[0144]

[0145] Organizing can yield

[0146]

[0147]

[0148] at this time This is the final distance to the nearest point, which is the distance to the vehicle in front.

[0149] (7) Update the covariance matrix. Calculate the error covariance using the formula...

[0150] P t =(IK t H)P t -

[0151] Substitute parameters

[0152]

[0153] A second aspect of the present invention provides a monocular ranging system for an unmanned underground vehicle, including a storage device and a processor. The storage device stores a computer program, and when the processor executes the program, it implements the steps of the method as described in any of the above embodiments.

[0154] In this embodiment, since the processor included therein can implement the steps of the method as described in any of the above embodiments, all the technical effects of the monocular ranging system for an unmanned underground vehicle provided by the second aspect of the present invention will not be repeated here.

[0155] Example 1

[0156] like Figure 1 , 6 As shown in 7 and 8

[0157] An infrared camera is installed on the windshield of the trackless rubber-wheeled vehicle, with its viewing angle facing the direction of travel. This allows for the collection of environmental data in front of the vehicle.

[0158] Step 1: Convolutional Neural Network Performs Object Detection

[0159] The infrared camera installed in step 1 is used for data acquisition. The acquired data is a real-time video stream. Target detection is performed on each frame of the video using a convolutional neural network. The detection content includes the body, rear, and wheel hub of the front rubber-wheeled vehicle (since the underground mine tunnels only allow one vehicle to pass at a time, only the rear of the vehicle is detected, not the front). The detection results include the type information of the detected target and the detection bounding box information.

[0160] Step 2: Multi-target information fusion

[0161] Based on the position of the forward-facing detection vehicle, three scenarios are defined: the detection vehicle is directly in front, the detection vehicle is to the left front, and the detection vehicle is to the right front. These three scenarios will be described in detail below. Figure 3 , Figure 4 , Figure 5 Further explanation is provided. The fusion judgment steps are as follows:

[0162] The test results are input into the multi-information fusion decision-making process;

[0163] (1) Determine whether a vehicle body is detected. If no, determine that there is no vehicle in front. If yes, continue to determine.

[0164] (2) Determine whether the rear of the vehicle has been detected. If not, determine that the vehicle is located directly in front. If yes, continue the determination.

[0165] (3) Determine whether a wheel has been detected. If no, determine that the car is in front. If yes, detect the number of wheels.

[0166] (4) If one wheel is detected, the vehicle is determined to be directly in front; if two wheels are detected, the vehicle is determined to be to the side front.

[0167] (5) When the vehicle is located to the side front, firstly, determine the vehicle's position based on the relative positions of p1 and p2.

[0168] When (x1-x2)(y1-y2)<0, it is determined that the vehicle is at the left front.

[0169] When (x1-x2)(y1-y2)>0, it is determined that the vehicle is at the right front.

[0170] Step 3: Pose estimation to obtain the nearest point

[0171] The method for calculating the nearest pixel coordinates differs depending on whether the vehicle is to the side or directly in front. Let the nearest pixel coordinates be A(x... a ,y a When the vehicle is directly in front, the pixel coordinates of the midpoint of the bottom edge of the vehicle body detection frame are obtained as the pixel coordinates of the nearest point A. When the vehicle is to the side front, taking the vehicle being to the left front as an example, the pixel coordinates Q(x) of the bottom right corner of the rear detection frame are obtained. q ,y q At this point, the line containing p1 and p2 is...

[0172]

[0173] The coordinates of the nearest point A can be obtained as follows:

[0174] Step 4: Calculate distance using a monocular geometric ranging model

[0175] In the first stage, the pixel coordinates of the nearest point A are input into the monocular geometric ranging model to obtain the distance d between the nearest point and the camera in the real world.

[0176]

[0177] h is the height of the camera above the ground, γ is the camera's downward angle, and y a These are the pixel coordinates of the nearest point in the image, v0, f v These are internal parameters of the camera, which can be obtained through calibration, and d is the distance to be determined.

[0178] Step 5: Calculate distance using the monocular geometric ranging model

[0179] The second stage involves Kalman prediction and updating. The Kalman filter model consists of two parts: prediction and updating. The prediction part of the model is as follows:

[0180]

[0181] P t - =FP t-1 F T +Q

[0182] Update part of the model as

[0183] K t =P t - H T HP t - H T +R) -1

[0184]

[0185] P t =(IK t H)P t -

[0186] Step six, output the vehicle distance, which is...

[0187] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0188] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A monocular ranging method for an unmanned underground vehicle, characterized in that, Includes the following steps: S1, In the vehicle's driving route, images of the vehicles ahead are collected by sensors, and convolutional neural networks are used to automatically identify the detection box information and the type information of the vehicles ahead; S2, determine the vehicle pose of the forward vehicle based on the type information and the detection box information; S3, based on the vehicle pose, obtain the pixel coordinates of the closest point on the forward vehicle to the sensor in the forward vehicle image; S4, Substitute the pixel coordinates into the monocular geometric ranging model to obtain the measured distance between the nearest point and the sensor, and then use the Kalman filter algorithm to predict and update the measured distance to obtain the vehicle distance; The type information includes: vehicle body information, rear information and wheel hub information, and each corresponds one-to-one with the detection frame information; When vehicle information ahead is detected, it is determined that there is a vehicle moving forward. The step of determining the vehicle position of the vehicle moving forward specifically includes: When only vehicle body information is detected, but no rear or wheel hub information is detected, it is determined that the forward vehicle is located directly in front of the vehicle. When vehicle body information and rear information are detected, but wheel hub information is not detected, it is determined that the forward vehicle is located directly in front of the vehicle. When vehicle body information, rear information, and wheel hub information are detected simultaneously, it is determined that the forward vehicle is located directly in front of the vehicle. When vehicle body information, rear information, and two wheel hub information are detected simultaneously, it is determined that the forward vehicle is located at the side front of the vehicle. When it is determined that the forward vehicle is located to the side front of the vehicle, the pixel coordinates p1(x1,y1) and p2(x2,y2) of the bottom midpoint of the detection box information of the two wheels are obtained. When (x1-x2)(y1-y2)<0, it is determined that the vehicle is at the left front. When (x1-x2)(y1-y2)>0, it is determined that the vehicle is currently in the right front position. Let the pixel coordinates of the nearest point be A(x) a ,y a S3 specifically includes: When the vehicle in front is directly in front of the vehicle, the pixel coordinates of the bottom midpoint of the detection box information for obtaining the vehicle body information are the pixel coordinates of the nearest point A. When the preceding vehicle is located to the side and in front of the vehicle, the lower right or lower left pixel coordinates Q(x) of the detection frame information used to obtain the rear information of the vehicle at this time are determined. q ,y q At this point, the lines containing p1 and p2 are: The coordinates of the nearest point A are: The image of the forward-facing vehicle is obtained frame by frame from a real-time video stream captured by the sensor. The sensor is installed on the windshield of the vehicle, and the camera acquisition surface is set in the same direction as the vehicle's driving direction; The sensor is an infrared camera.

2. The monocular ranging method for an unmanned underground vehicle according to claim 1, characterized in that, In S4, the nearest point A(x) a ,y a Substituting the pixel coordinates of () into the monocular geometric ranging model, the following formula can be obtained: h is the height of the sensor above the ground; γ is the downward angle of the sensor; y a It is the vertical coordinate of the point where the forward-moving vehicle is closest to the sensor in the pixel coordinates of the image; v0, f v These are the camera's internal parameters, where v0 is the pixel ordinate corresponding to the principal point, and f... v The length of the focal length in the y-axis direction is described using pixels and can be obtained by Zhang Zhengyou's calibration method; d is the distance to be calculated.

3. The monocular ranging method for an unmanned underground vehicle according to claim 1, characterized in that, The Kalman filter algorithm includes a prediction part model and an update part model, wherein the output of the prediction part model is used as the input condition of the update part model.

4. The monocular ranging method for an unmanned underground vehicle according to claim 3, characterized in that, The prediction model includes the following formulas: P t - =FP t-1 F T +Q; in, Let u be the state variable predicted at time t. t Let F be the control variable at time t, F be the state matrix, B be the control matrix, and P be the control variable at time t. t - Let P be the error covariance predicted at time t. t-1 For the error covariance updated at time t-1, Let Q be the state variable predicted at time t-1, and let Q be the observation noise.

5. The monocular ranging method for an unmanned underground vehicle according to claim 4, characterized in that, The updated part of the model includes the following formula: K t =P t - H T (HP t - H T +R) -1 ; P t =(I-K t H)P t - ; Among them, P t Let K be the error covariance, Q be the observation noise, and K be the error covariance. t Here, H is the gain coefficient, and z is the gain matrix. t Let I be the observed variable, H be the identity matrix, and I be the identity matrix. T Let H be the transpose of H and R be the noise covariance.

Images