A method for material pile identification and localization based on the fusion of machine vision and satellite positioning

Abstract

This invention discloses a method for material pile identification and positioning based on the fusion of machine vision and satellite positioning. The data processing unit simultaneously collects ranging information from a machine vision device and positioning information from a satellite receiver for engineering vehicles. It obtains the distance between the target material pile and the engineering vehicle measured by the machine vision device within adjacent sampling times, as well as the movement distance of the engineering vehicle. It also obtains the movement distance of the engineering vehicle located by satellite positioning within adjacent sampling times. When the distance between the engineering vehicle and the target material pile is less than a set distance, the distance directly measured by the machine vision device is used as the measured value between the target material pile and the engineering vehicle. When the distance is greater than the set distance, the distance corrected for machine vision ranging error obtained through a fusion algorithm of machine vision and satellite positioning is used as the measured value between the target material pile and the engineering vehicle. This invention can significantly improve the positioning accuracy of target material piles and is particularly suitable for intelligent engineering machinery.

Description

Technical Field

[0001] This invention relates to a material pile identification and positioning method, specifically a material pile identification and positioning method based on the fusion of machine vision and satellite positioning, applicable to intelligent engineering machinery, and belongs to the field of intelligent engineering machinery technology. Background Technology

[0002] The rapid development of automation and intelligence in construction machinery has driven the important application of material pile identification and positioning technology in the construction machinery field. In traditional construction machinery operations, material pile identification and positioning usually rely on manual operation or a single sensor. This traditional approach often suffers from insufficient positioning accuracy and poor adaptability in complex environments, restricting the efficiency and safety of construction machinery operations. Taking unmanned intelligent loaders as an example, current unmanned material pile identification and positioning methods for loaders mostly use a single sensor. Machine vision ranging methods, including binocular cameras, can identify and locate material piles. However, machine vision ranging and positioning are easily affected by external environmental interference. Long distances, excessively strong or weak light conditions can significantly impact the positioning accuracy of machine vision. Even when using millimeter-wave radar ranging, there are problems with accurately identifying and locating material piles under complex working conditions.

[0003] Therefore, existing technologies have developed methods for material pile identification and positioning that combine machine vision and satellite positioning. Method one involves first using machine vision to detect the target material pile, and then using satellite positioning to move the pan-tilt unit to calculate the target pile's position. This method ignores errors caused by changes in the external environment during machine vision target detection. While this method is usable when only the position of the target material pile needs to be obtained, its error rate is insufficient for unmanned operation of construction machinery. Method two uses a fixed-position camera to detect the target material pile, and then uses satellite positioning to move the pan-tilt unit to calculate the target pile's position. However, this method cannot move the camera with the construction machinery, making it unsuitable for changing workplaces. Furthermore, it requires the application of positioning stickers for target identification, making the operation process more complex. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a material pile identification and positioning method based on the fusion of machine vision and satellite positioning. This method can automatically identify target material piles and greatly improve the positioning accuracy of machine vision for target material piles, making it particularly suitable for intelligent engineering machinery.

[0005] To achieve the above objectives, the material pile identification and positioning system based on the fusion of machine vision and satellite positioning used in this material pile identification and positioning method includes a satellite receiving device, a machine vision device, and a data processing unit. The data processing unit is electrically connected to the machine vision device and the satellite receiving device, respectively.

[0006] The specific method for material pile identification and localization based on the fusion of machine vision and satellite positioning is as follows:

[0007] The data processing unit simultaneously collects ranging information fed back by the machine vision device and positioning information of the engineering vehicle fed back by the satellite receiver. It obtains the distance between the target material pile and the engineering vehicle measured by the machine vision device and the moving distance of the engineering vehicle within adjacent sampling time. It also obtains the moving distance of the engineering vehicle located by the satellite within adjacent sampling time. When the distance between the engineering vehicle and the target material pile is less than the set distance, the distance directly measured by the machine vision device is used as the measurement value between the target material pile and the engineering vehicle. When the distance between the engineering vehicle and the target material pile is greater than the set distance, the distance after correction of the machine vision ranging error obtained by the machine vision and satellite positioning fusion algorithm is used as the measurement value between the target material pile and the engineering vehicle.

[0008] The specific algorithm for fusing machine vision and satellite positioning is as follows:

[0009] First, the machine vision device measures the distance the engineering vehicle has traveled. Subtract the distance the engineering vehicle has traveled as currently measured by satellite positioning. Obtain machine vision ranging error Then, Kalman filtering is applied to... Perform filtering to obtain machine vision ranging compensation correction values. Finally, Incorporate the distance between the target stockpile and the engineering vehicle directly measured by the machine vision device to obtain the measured value between the stockpile and the engineering vehicle. .

[0010] Furthermore, Kalman filtering is used to... The specific process of filtering is as follows:

[0011] ① Define the state variable at time t as follows: ;

[0012] In the formula: Let be the machine vision ranging error at time t. ; Let be the rate of change of error at time t;

[0013] ②Establish the state transition equation:

[0014] Simplified to a linear change over time, Simplifying to a uniform change, the state transition equation is expressed as: ; ;

[0015] In the formula: and They are respectively Machine vision ranging error and error rate of change at any given time;

[0016] Converting it to matrix form, it can be represented as follows: ;

[0017] In the formula: Let A be the state matrix; For process noise, denoted as Where Q is the process noise covariance matrix, set as: ;

[0018] In the formula: Machine vision ranging error The variance of the corresponding process noise; Error change rate The variance of the corresponding process noise;

[0019] ③Establish the observation equation:

[0020] Observations for Then its observation equation can be expressed as: ;

[0021] In the formula: H represents the observed values; H is the observation matrix. ; Indicates the error observation value; For observation noise, it is denoted as Where R is the observation noise covariance matrix, set as: ;

[0022] In the formula: To observe noise The corresponding variance;

[0023] ④ The following is a Kalman filter mathematical model for calculating the distance between the engineering vehicle and the target stockpile: ; ; ; ; ;

[0024] In the formula: This is the predicted value at the current moment; This is the optimal estimate from the previous moment; The sampling period; This is the estimated error covariance matrix of the prediction at the current time. Let be the estimation error covariance matrix of the previous time step; Kalman gain; This is the optimal state estimate at the current moment; The value measured at the current moment; Let be the estimation error covariance matrix at the current time.

[0025] Furthermore, before identifying and locating the target material pile, the machine vision device is calibrated, and then images of the material pile under different lighting and angles are collected as a training dataset. The YOLOv8 model is then used to train the material pile recognition model.

[0026] Furthermore, when the data processing unit simultaneously collects ranging information from the machine vision device and positioning information of the engineering vehicle from the satellite receiver, the data processing unit's acquisition frequency period is set to a common multiple of the detection frequency period of the machine vision device and the detection frequency period of the satellite receiver.

[0027] Furthermore, before using machine vision devices to locate and detect the target material pile, the acquired images are preprocessed and then stereo matching is performed.

[0028] Furthermore, the machine vision device is fixedly installed on the top of the engineering vehicle and located at the geometric center of the engineering vehicle; the satellite receiving device is fixedly installed on the top of the engineering vehicle and is symmetrically arranged with respect to the geometric center of the engineering vehicle; the data processing unit is located in the driver's cab of the engineering vehicle.

[0029] Preferably, the machine vision device includes a binocular camera; the satellite receiver is a satellite receiver that receives positioning signals from the BeiDou positioning system.

[0030] Compared with existing technologies, machine vision devices have a large ranging error at longer distances, while satellite positioning systems have higher positioning accuracy. Therefore, when the distance between the engineering vehicle and the target material pile is less than the set distance, this material pile identification and positioning method based on the fusion of machine vision and satellite positioning uses the distance directly measured by the machine vision device as the measurement value between the target material pile and the engineering vehicle. When the distance between the engineering vehicle and the target material pile is greater than the set distance, this material pile identification and positioning method based on the fusion of machine vision and satellite positioning uses the more accurate distance after correction of the machine vision ranging error obtained by the fusion algorithm of machine vision and satellite positioning as the measurement value between the target material pile and the engineering vehicle. This can greatly improve the positioning accuracy of machine vision for the target material pile, and is particularly suitable for intelligent engineering machinery. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a material pile identification and positioning system based on the fusion of machine vision and satellite positioning installed on an unmanned intelligent loader;

[0032] Figure 2This is a flowchart of the machine vision and satellite positioning fusion algorithm of the present invention;

[0033] Figure 3 This is a schematic diagram of the machine vision and satellite positioning fusion algorithm of the present invention;

[0034] Figure 4 This is a photograph of the test stockpile in an embodiment of the present invention;

[0035] Figure 5 This is a comparison chart of the actual distance measurement value and the measured value using binocular camera distance measurement in an embodiment of the present invention;

[0036] Figure 6 This is a comparison chart of the actual and measured distance values ​​obtained by using BeiDou and binocular fusion ranging in an embodiment of the present invention.

[0037] In the diagram: 1. Loader body, 2. Satellite receiver, 3. Machine vision device, 4. Data processing unit;

[0038] Let t be the distance between the material pile and the unmanned intelligent loader measured by the binocular camera. for The distance between the material pile and the unmanned intelligent loader is measured by the binocular camera at all times. This refers to the distance traveled by the unmanned intelligent loader as measured by BeiDou within a given time period. The distance traveled by the unmanned intelligent loader as measured by the binocular camera within a given time period. Detailed Implementation

[0039] The following example, using the installation of a material pile identification and positioning system based on the fusion of machine vision and satellite positioning on an unmanned intelligent loader, further illustrates the present invention with reference to the accompanying drawings.

[0040] like Figure 1 As shown, the material pile identification and positioning system based on the fusion of machine vision and satellite positioning includes a satellite receiving device 2, a machine vision device 3, and a data processing unit 4. The machine vision device 3 is fixedly installed on the top of the loader body 1 and is located at the geometric center of the loader body 1. The machine vision device 3 may include a binocular camera or a depth camera, preferably a binocular camera. The satellite receiving device 2 is fixedly installed on the top of the loader body 1 and is symmetrically arranged with respect to the geometric center of the loader body 1. The satellite positioning system is preferably a Beidou positioning system with a positioning accuracy of centimeter level, that is, the satellite receiving device 2 is a Beidou positioning signal receiving device. The data processing unit 4 is located in the cab of the loader body 1. The binocular camera of the machine vision device 3 is electrically connected to the data processing unit 4 through a USB connection cable, and the satellite receiving device 2 is electrically connected to the data processing unit 4 through a USB to 232 connection cable.

[0041] This material pile identification and positioning method based on the fusion of machine vision and satellite positioning specifically includes the following steps:

[0042] Step 1, Target stockpile identification and positioning preparation:

[0043] Step 1-1, Camera Calibration: Before using the binocular camera of the machine vision device 3 to identify and locate the target material pile, the binocular camera is calibrated using the Zhang Zhengyou calibration method to determine internal parameters (such as focal length, principal point, distortion coefficient, etc.) and external parameters (relative position of the two cameras) in order to reduce image distortion.

[0044] Using the Bouguet algorithm in OpenCV (Open Computer Vision Library, a cross-platform computer vision library) to calibrate the stereo camera, two stereo images taken by different cameras are adjusted through geometric transformations to make them coplanar and with parallel optical axes. After stereo calibration, the computational complexity can be significantly reduced and the matching accuracy and depth estimation accuracy can be improved.

[0045] Step 1-1-2, Material pile recognition model training: Collect 5000 images of material piles under different lighting and angles as training datasets, and use the YOLOv8 model to train the material pile recognition model.

[0046] Step 2, Target stockpile distance detection:

[0047] like Figure 2 As shown, the data processing unit 4 simultaneously acquires the detection information from the binocular camera of the machine vision device 3 and the satellite receiving device 2. The acquisition frequency period of the data processing unit 4 can be set to a common multiple of the detection frequency period of the machine vision device 3 and the detection frequency period of the satellite receiving device 2. In this embodiment, the acquisition frequency period of the data processing unit 4 is set to 100ms to perform positioning detection on the target material pile.

[0048] Step 2-1, Image Preprocessing:

[0049] ① Grayscale processing: To ensure the accuracy of material pile identification and positioning, the image is grayscale processed before locating the material pile to eliminate color interference.

[0050] The grayscale conversion process uses a weighted average method, which calculates the weighted average of the RGB values ​​of each pixel based on the weights of different channels. The weights for the red, green, and blue channels are set to 0.299, 0.587, and 0.114, respectively. The specific calculation formula is as follows: ;

[0051] In the formula: Gray represents grayscale; R represents the red channel; G represents the green channel; B represents the blue channel.

[0052] ② Gaussian filtering: Gaussian filtering and smoothing are performed using the built-in Gaussian filtering function cv2.GaussianBlur() in OpenCV to ensure that subsequent processing is not affected by noise.

[0053] ③ Gamma Transform: Using the OpenCV built-in gamma_correction() function, a gamma value of 0.7 is selected to perform gamma transformation on the Gaussian filtered image of the stockpile to adjust the image brightness and make the image more suitable for processing.

[0054] ④ Histogram equalization: The histogram equalization operation is performed on the stockpile image after gamma correction using the cv2.equalizeHist() function to enhance the contrast and details of the image.

[0055] ⑤ Edge detection: The outline, structure and boundary of the material pile are accurately delineated using the cv2.Canny() edge detection algorithm function provided by OpenCV.

[0056] Step 2-2, Stereo Matching:

[0057] The Semi-Global Block Matching (SGBM) stereo matching algorithm is employed to obtain disparity information from stereo images, thereby improving the quality of the generated disparity map and the accuracy of binocular camera ranging. This algorithm consists of four key steps: preprocessing, cost calculation, cost aggregation, and post-processing.

[0058] ① Preprocessing uses the Sobel operator (edge ​​detection operator) to extract edge information from the image and reduce the impact of low-texture areas.

[0059] ② The cost calculation adopts an absolute method based on gradient, combined with the Birchfield-Tomasi (BT) cost function to improve the accuracy and robustness of matching.

[0060] ③ Cost aggregation selects four paths in different directions, optimizes pixel matching cost through neighborhood information, reduces noise and mismatches, and maintains the integrity of depth edge information.

[0061] ④ The post-processing adopts a disparity map optimization and reconstruction algorithm based on WLS filtering post-processing. This algorithm combines a weighted least squares edge-preserving filter with bilateral filtering, which effectively reduces noise in the depth map while preserving the edge details of the object.

[0062] Step 3, Fusion of machine vision and satellite positioning:

[0063] like Figure 3 As shown, where , (The distance values ​​currently measured by the binocular cameras) are respectively at time t and The distance between the material pile and the unmanned intelligent loader is measured by the binocular camera at all times. , The distances traveled by the unmanned intelligent loader were measured by Beidou and the binocular camera respectively within the sampling period. Since the distance measurement accuracy of the binocular camera is relatively large at long distances, while Beidou has higher positioning accuracy, Beidou is used to correct the distance measured by the binocular camera when the unmanned intelligent loader is far away from the material pile.

[0064] The specific correction method is as follows: the moving distance of the unmanned intelligent loader measured by the binocular camera. Subtract the distance traveled by the unmanned intelligent loader as measured by Beidou. , get value (Updated every 100ms), this value is the ranging error of the stereo camera. A Kalman filter is used to filter this value to obtain a more accurate error correction value, denoted as [value to be inserted here]. The machine vision ranging compensation value is then added to the distance directly measured by the stereo camera to obtain a more accurate measurement value. Therefore, the error correction model is: .

[0065] The specific process of ranging data processing based on Kalman filtering is as follows:

[0066] ① Define the state variable at time t as follows: ;

[0067] In the formula: Let be the machine vision ranging error at time t, i.e. ; Let be the rate of change of error at time t.

[0068] ②Establish the state transition equation:

[0069] To simplify the calculations, we assume the engineering vehicle is moving at a constant speed, and the distance measurement error is... This can be understood as a linear change over time, with the error rate of change being... This can also be interpreted as a uniform change, from which the state transition equation can be obtained as follows: ; ;

[0070] In the formula: and They are respectively Machine vision ranging error and error rate of change at any given time.

[0071] Converting it to matrix form, it can be represented as follows: ;

[0072] In the formula: Let A be the state matrix; The process noise follows a normal distribution and can be expressed as: Where Q is the process noise covariance matrix, set as: ;

[0073] In the formula: Distance measurement error The variance of the corresponding process noise reflects the ranging error. The degree of change; Error change rate The smaller the variance of the corresponding process noise, the more stable the error rate of change.

[0074] ③Establish the observation equation:

[0075] Observations for Then its observation equation can be expressed as: ;

[0076] In the formula: H is the observation matrix, ; Indicates the error observation value; The observation noise, which follows a normal distribution, can be represented as: Where R is the observation noise covariance matrix, set as: ;

[0077] In the formula: To observe noise The corresponding variance reflects the magnitude of measurement noise; the larger the value, the less trust there is in the measurement.

[0078] ④ The following is the mathematical model for calculating the loader-stock distance using Kalman filtering:

[0079] Prior estimation formula: ;

[0080] Formula for the prior covariance matrix: ;

[0081] Kalman gain formula: ;

[0082] Update state estimation formula: ;

[0083] Formula for updating the covariance matrix: ;

[0084] In the formula: This is the predicted value at the current moment; This is the optimal estimate from the previous moment; The sampling period; This is the estimated error covariance matrix of the prediction at the current time. Let be the estimation error covariance matrix of the previous time step; Kalman gain; This is the optimal state estimate at the current moment; The value measured at the current moment; Let be the estimation error covariance matrix at the current time.

[0085] Example: Actual picture of the test stockpile. Figure 4 As shown, distance measurement was performed using a binocular camera. The test distance started at 2000mm and increased in 500mm increments, covering a total of 17 different measurement points. The distance measurement data from the binocular camera is shown in Table 1 below, and the comparison graph between the actual distance measurement value and the measured value is shown in the figure below. Figure 5 As shown.

[0086]

[0087] Using BeiDou and binocular fusion ranging, the experimental ranging range was set from 2000mm to 10000mm. A material pile identification ranging test was conducted every 500mm, for a total of 17 sets of material pile identification ranging tests at different locations. The BeiDou and binocular fusion ranging data are shown in Table 2 below. A comparison chart of the fusion ranging values ​​and the measured values ​​is shown below. Figure 6 As shown.

[0088]

[0089] The experimental data above show that the absolute and relative errors of the ranging data obtained by the fusion of BeiDou and binocular cameras are both smaller than those of the ranging data obtained by using only binocular cameras. The ranging data obtained by using the fusion of BeiDou and binocular cameras is more accurate.

[0090] This material pile identification and positioning method based on the fusion of machine vision and satellite positioning uses the distance directly measured by the binocular camera of the machine vision device 3 as the measurement value between the target material pile and the engineering vehicle when the distance between the unmanned intelligent loader and the material pile is less than 2m. When the distance between the unmanned intelligent loader and the material pile is greater than 2m, the more accurate distance after correction of the machine vision ranging error obtained by the fusion algorithm of machine vision and satellite positioning is used as the measurement value between the target material pile and the engineering vehicle. This method can greatly improve the positioning accuracy of the target material pile by machine vision.

Claims

1. A material pile identification and positioning method based on the fusion of machine vision and satellite positioning, wherein the material pile identification and positioning system based on the fusion of machine vision and satellite positioning includes a satellite receiving device (2), a machine vision device (3) and a data processing unit (4), wherein the data processing unit (4) is electrically connected to the machine vision device (3) and the satellite receiving device (2) respectively. Its features are, The specific method for material pile identification and localization based on the fusion of machine vision and satellite positioning is as follows: The data processing unit (4) simultaneously collects the ranging information fed back by the machine vision device (3) and the positioning information of the engineering vehicle fed back by the satellite receiving device (2), obtains the distance between the target material pile and the engineering vehicle measured by the machine vision device (3) and the moving distance of the engineering vehicle within adjacent sampling time, obtains the moving distance of the engineering vehicle located by the satellite within adjacent sampling time, and when the distance between the engineering vehicle and the target material pile is less than the set distance, the distance directly measured by the machine vision device (3) is used as the measurement value between the target material pile and the engineering vehicle, and when the distance between the engineering vehicle and the target material pile is greater than the set distance, the distance after the machine vision ranging error correction obtained by the machine vision and satellite positioning fusion algorithm is used as the measurement value between the target material pile and the engineering vehicle. The specific algorithm for fusing machine vision and satellite positioning is as follows: First, the machine vision device (3) measures the current moving distance ΔL of the engineering vehicle. stero Subtract the distance ΔL that the engineering vehicle has traveled as currently measured by satellite positioning. Beidou Obtain the machine vision ranging error e t Then, e is processed using a Kalman filter. t Perform filtering to obtain machine vision ranging compensation correction values. Finally By incorporating a machine vision device (3), the measured value L between the target stockpile and the engineering vehicle is obtained above the distance currently being directly measured between the stockpile and the engineering vehicle. ′ t+1 ; e is processed by Kalman filtering t The specific process of filtering is as follows: ① Define the state variable X at time t. t as follows: In the formula: e t Let e ​​be the machine vision ranging error at time t. t =ΔL stero -ΔL Beidou V t Let be the rate of change of error at time t; ②Establish the state transition equation: e t Simplified to a linear change over time, V t Simplifying to a uniform change, the state transition equation is expressed as: And t+1 =and t +V t ·Δt V t+1 =V t In the formula: e t+1 and V t+1 These represent the machine vision ranging error and the rate of change of error at time t+1, respectively. Converting it to matrix form, it can be represented as follows: In the formula: Let A be the state matrix; W be the state matrix. t The process noise is denoted as W. t ~N(0,Q), where Q is the process noise covariance matrix, defined as: In the formula: machine vision ranging error e t The variance of the corresponding process noise; The error change rate V t The variance of the corresponding process noise; ③Establish the observation equation: Observation Z t For ΔL stero -ΔL Beidou Then its observation equation can be expressed as: Z t =HX t +V t In the formula: Z t X represents the observed values; H is the observation matrix, H = [1 0]; X represents the observed values. t V represents the error observation value; t The observation noise is denoted as V ~ N(0,R), where R is the observation noise covariance matrix, defined as: In the formula: For observing noise V t The corresponding variance; ④ The following is a Kalman filter mathematical model for calculating the distance between the engineering vehicle and the target stockpile: In the formula: This is the predicted value at the current moment; The value is the optimal estimate from the previous moment; Δt is the sampling period. P is the estimation error covariance matrix predicted at the current time. k-1 K is the estimation error covariance matrix of the previous time step; k Kalman gain; Z(K) is the optimal state estimate at the current time; Z(K) is the measured value at the current time; P k Let be the estimation error covariance matrix at the current time.

2. The material pile identification and positioning method based on the fusion of machine vision and satellite positioning according to claim 1, characterized in that, Before identifying and locating the target material pile, the machine vision device (3) is calibrated, and then images of the material pile under different lighting and angles are collected as training datasets. The YOLOv8 model is used to train the material pile recognition model.

3. The material pile identification and positioning method based on the fusion of machine vision and satellite positioning according to claim 1, characterized in that, When the data processing unit (4) simultaneously collects the ranging information fed back by the machine vision device (3) and the positioning information of the engineering vehicle fed back by the satellite receiving device (2), the acquisition frequency period of the data processing unit (4) is set to the common multiple of the detection frequency period of the machine vision device (3) and the detection frequency period of the satellite receiving device (2).

4. The material pile identification and positioning method based on the fusion of machine vision and satellite positioning according to claim 1, characterized in that, Before the target material pile is located and detected by the machine vision device (3), the acquired image is preprocessed and then stereo matching is performed.

5. The material pile identification and positioning method based on machine vision and satellite positioning fusion according to claim 1, characterized in that, The machine vision device (3) is fixedly installed on the top of the engineering vehicle and is located at the geometric center of the engineering vehicle; the satellite receiving device (2) is fixedly installed on the top of the engineering vehicle and is symmetrically arranged with respect to the geometric center of the engineering vehicle; the data processing unit (4) is located in the cab of the engineering vehicle.

6. The material pile identification and positioning method based on machine vision and satellite positioning fusion according to claim 1, characterized in that, The machine vision device (3) includes a binocular camera; the satellite receiver (2) is a satellite receiver that receives positioning signals from the Beidou positioning system.

Images