Unmanned intelligent power station wheel type inspection robot obstacle detection method and system based on multi-sensor fusion

Abstract

The invention provides an unmanned intelligent power station wheel type inspection robot obstacle detection method and system based on multi-sensor fusion, and belongs to the technical field of obstacle detection. The detection method comprises the steps of firstly obtaining sensor data of the inspection robot, performing type conversion and coordinate conversion on depth data in the sensor data, and finally fusing the depth data and laser radar data by using a BP neural network algorithm to obtain more accurate obstacle information. A detection system comprises a data acquisition module used for acquiring depth data and laser radar point cloud data of an area in front of a robot, a first data processing module used for converting the depth data into point cloud data, a second data processing module used for performing coordinate transformation on the processed depth data, and a data fusion module. And the depth data and the point cloud data are fused by using a BP neural network algorithm to obtain more accurate obstacle information. The beneficial effects of the invention are that the method can improve the range and precision of obstacle detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of obstacle detection, and particularly relates to an obstacle detection method and system for a wheeled inspection robot of an unmanned intelligent power station based on multi-sensor fusion. Background Art

[0002] Power station equipment is an important part to ensure the safe and stable operation of the power grid. However, the equipment in the power station is often eroded by natural environments such as strong winds, sunlight, and rain, which accelerates the aging of the power station equipment, increases the failure rate of the equipment, and seriously affects the safety and stability of the power grid. Therefore, it is very necessary to conduct regular inspections on the power station and eliminate potential hazards in advance.

[0003] The traditional inspection method of the power station is manual inspection, which not only has low inspection efficiency and high maintenance costs, but also cannot guarantee the safety of the inspection personnel. If encountering bad weather, it will increase the difficulty of inspection. Therefore, using a safer and more efficient inspection robot for inspection has gradually become a research hotspot.

[0004] During the inspection process, the inspection robot inevitably needs to avoid obstacles. Therefore, it is crucial to detect obstacles. However, the commonly used detection sensors at present are limited by the detection range and accuracy, and cannot meet the needs of inspection. Summary of the Invention

[0005] To solve the problems in the above background art, the present invention provides an obstacle detection method and system for a wheeled inspection robot of an unmanned intelligent power station based on multi-sensor fusion, which can get rid of the limitations of a single sensor and improve the detection range and accuracy.

[0006] To achieve the above object, the present invention provides the following technical solutions: In the first aspect, an obstacle detection method for a wheeled inspection robot of an unmanned intelligent power station based on multi-sensor fusion is provided. The method includes the following steps:

[0007] S1. Obtain the sensor data of the inspection robot, including lidar and depth camera data;

[0008] S2. Convert the depth data of the depth camera into point cloud data, and change the converted depth data from the depth camera coordinate system to the lidar coordinate system;

[0009] S3. Use the BP neural network algorithm to fuse the depth camera data and the lidar data to obtain more accurate obstacle information.

[0010] In a preferred solution, in the step S1, the lidar selected is the SlAMTEC laser scanning ranging radar;

[0011] The depth camera selected is the LLVision depth camera.

[0012] Further, in the step S2, the conversion between the depth data and the point cloud data is the conversion between the image coordinate system O-xyz and the world coordinate system O w -x w y w z w , and the conversion relationship is as follows:

[0013]

[0014] where f is the camera focal length and d is the depth value.

[0015] Further, in the step S2, the conversion relationship between the depth camera coordinate system and the lidar coordinate system is as follows:

[0016] Taking the optical center O of the depth camera c as the origin, a depth camera coordinate system O c -x c y c z c is established. Taking the lidar center O r as the origin, a lidar coordinate system O c -x r y r z r is established. The height difference between O c and O r is H, and the horizontal distance is L;

[0017] For the point P, its coordinates in the depth camera coordinate system are (x c , y c , z c ), and its coordinates in the lidar coordinate system are (x r , y r , z r ). The relationship between the two is as follows:

[0018]

[0019] Rewriting the above formula into matrix form, we have:

[0020]

[0021] Further, in the step S3, the BP neural network algorithm includes an input layer, a hidden layer, and an output layer. The input layer includes depth camera data and lidar data. The hidden layer includes m1 neurons, and the output layer outputs the updated obstacle information;

[0022] The steps for the BP neural network algorithm to update the obstacle information are as follows:

[0023] S1 Import the input variables. The i-th neuron in the input layer is xi where x1 is the depth camera data, x2 is the lidar data, and the two together form the input vector X = (x1, x2) T ;

[0024] S2 calculates the input value i and output value of the i-th neuron h in the hidden layer The calculation process is as follows:

[0025]

[0026] where n is the number of neurons in the input layer, is the weight of the hidden layer, b1 is the bias of the hidden layer, and f in (·) is the activation function of the hidden layer;

[0027] S3 calculates the input value y and output value y of the neuron y in the output layer in The calculation process is as follows: out where

[0028]

[0029] is the weight of the output layer, b2 is the bias of the output layer, and f (·) is the activation function of the output layer; out (·) is the activation function of the output layer;

[0030] S4 When the set number of iterations is reached, the output value y out is the updated obstacle information.

[0031] Furthermore, in the step S3, the method of training the neural network selects the gradient descent method, and the iterative formulas for the weights and biases of the output layer are:

[0032]

[0033] where w out ′ and b2′ are the values of the weights and biases of the output layer to be updated after iteration, η is the learning rate, E(m) is the evaluation function, O(m) is the output value of the m-th neuron in the output layer, and O1(m) is the expected output value of the m-th neuron in the output layer;

[0034] Judge the magnitude of E(m). When E(m) ≥ 0.5, update the weights and biases of the output layer, that is, let:

[0035]

[0036] Furthermore, in the step S32, the activation function f in (·) of the hidden layer selects the hyperbolic tangent function, and its expression is:

[0037]

[0038] Furthermore, in the step S33, the activation function f of the hidden layer out (·) selects the sigmoid function, and its expression is:

[0039]

[0040] Furthermore, for the BP neural network algorithm, the fireworks algorithm is first used to optimize the initial values of its parameters;

[0041] The optimization steps are as follows:

[0042] S1) The parameters to be optimized for the BP neural network are: the weights and biases of the hidden layer, the weights and biases of the output layer, and the learning rate;

[0043] S2) Encode the above parameters into the form of fireworks individuals, and determine the dimension D of each fireworks x fi as:

[0044] D = n IW(1,1) + n b(1,1) + n IW(2,1) + n b(2,1) ;

[0045] Among them, n IW (1,1) is the number of weights between the input layer and the hidden layer of the neural network, n b (1,1) is the number of thresholds of the hidden layer neurons, n IW (2,1) is the number of weights between the hidden layer and the output layer of the neural network, n b (2,1) is the number of thresholds of the output layer neurons. Randomly initialize each fireworks x fi and set the dimension range of each fireworks to [-1, 1];

[0046] S3) Use the sum of squared errors to calculate the fitness value f(x fi ) of each fireworks:

[0047]

[0048] Among them, t is the output expectation of the neural network parameters, y is the output value of the neural network parameters, and s is the number of output layer neurons;

[0049] S4) The explosion radius A i of the fireworks explosion and the number of sparks S i generated are:

[0050]

[0051] Among them, N is the initial number of the fireworks population, and E r and E n are parameters for controlling the explosion radius and quantity respectively, f min and f max are the minimum and maximum fitness values respectively, and ε is a very small real number used to avoid a zero denominator;

[0052] S5) Randomly select a firework and perform Gaussian mutation operation on each of its dimensions:

[0053] x fik = x fik g;

[0054] Among them, x fik is the position of the i-th individual in the k-th dimension, and g is a random number obeying a Gaussian distribution with a mean of 1 and a variance of 1;

[0055] Map the fireworks beyond the boundary range to within the boundary range:

[0056]

[0057] Among them, are the upper and lower bounds of the k-th dimension respectively.

[0058] S6) Combine the initial fireworks, explosion sparks, and Gaussian mutation sparks to form a population set K. Retain the optimal solution in each round of iteration, and select the remaining N - 1 sparks according to the Euclidean distance between each firework and the optimal firework. The selection strategy is as follows:

[0059]

[0060] Among them, R(x fi ) is the Euclidean distance between the remaining fireworks and the optimal firework, and P(x fi ) is the probability that the firework x fi is selected;

[0061] S7) When the number of iterations reaches the set value, the optimization ends.

[0062] In a second aspect, a wheeled inspection robot obstacle detection system for an unmanned intelligent power station based on multi-sensor fusion is provided to implement the method for detecting obstacles of a wheeled inspection robot for an unmanned intelligent power station based on multi-sensor fusion described in any one of the above. The system includes a data acquisition module, a first data processing module, a second data processing module, and a data fusion module;

[0063] The data acquisition module uses a depth camera to collect depth data of the area in front of the robot, uses a lidar to collect point cloud data of the area in front of the robot, and sends the collected data to the first data processing module;

[0064] The first data processing module is used to receive the data collected by the data acquisition module, convert the depth data of the depth camera among them into point cloud data, and then send the data to the second data processing module;

[0065] The second data processing module is used to receive the data of the first data processing module, change the already converted depth data from the depth camera coordinate system to the lidar coordinate system, and then send the data to the data fusion module;

[0066] The data fusion module is used to receive the data of the second data processing module, and fuse the converted depth data with the point cloud data of the lidar using the optimized BP neural network algorithm to obtain more accurate obstacle information.

[0067] The beneficial effects of the present invention are: by fusing the data of multiple sensors, the detection limitation of a single sensor can be overcome, and the range and accuracy of obstacle detection can be improved. Description of the Drawings

[0068] The present invention will be further described below in conjunction with the drawings and embodiments:

[0069] Figure 1 It is a schematic flowchart of the detection method of the present invention;

[0070] Figure 2 It is a block diagram of the BP neural network fusion of the present invention;

[0071] Figure 3 It is a block diagram of the detection system of the present invention. Detailed Embodiments

[0072] Embodiment 1

[0073] As Figures 1 - 2 shown, a method for detecting obstacles of a wheeled inspection robot for an unmanned intelligent power station based on multi-sensor fusion, the steps are as follows:

[0074] S1. Obtain the sensor data of the inspection robot, including the data of the lidar and the depth camera;

[0075] S2. Convert the depth data of the depth camera into point cloud data. The conversion between the depth data and the point cloud data is the conversion between the image coordinate system O-xyz and the world coordinate system O w -x w y w z w The conversion relationship is:

[0076]

[0077] Among them, f is the camera focal length and d is the depth value.

[0078] Transform the converted depth data from the depth camera coordinate system to the lidar coordinate system. Establish a depth camera coordinate system O with the optical center O of the depth camera as the origin c -x c y c z c Establish a lidar coordinate system O with the lidar center Or as the origin c -x r y r z r The height difference between O and O is H, and the lateral distance is L; r O c and O r For point P, its coordinates in the depth camera coordinate system are (x

[0079] , y c , z c ), and its coordinates in the lidar coordinate system are (x c , y r , z r ). The relationship between the two is as follows:

[0080]

[0081] Rewrite the above formula in matrix form, we get:

[0082]

[0083] S3. First, use the fireworks algorithm to optimize the initial values of the parameters of the BP neural network. The process is as follows:

[0084] S1) Select the parameters to be optimized for the BP neural network: the weights and biases of the hidden layer, the weights and biases of the output layer, and the learning rate;

[0085] S2) Encode the above parameters into the form of fireworks individuals, and determine the dimension D of each fireworks x fi as:

[0086] D = n IW(1,1) + n b(1,1) + n IW(2,1) + n b(2,1) + n IW ;

[0087] where n b (1,1) is the number of weights between the input layer and the hidden layer of the neural network, n IW (1,1) is the number of thresholds of the hidden layer neurons, n b (2,1) is the number of weights between the hidden layer and the output layer of the neural network, n fi (2,1) is the number of thresholds of the output layer neurons. For each fireworks xfi Perform random initialization and set the dimensional range of each firework to [-1, 1];

[0088] S3) Calculate the fitness value f(x of each firework using the sum of squared errors fi ):

[0089]

[0090] where t is the output expectation of the neural network parameters, y is the output value of the neural network parameters, and s is the number of neurons in the output layer;

[0091] S4) The explosion radius A of the firework explosion i and the number of sparks S generated i are:

[0092]

[0093] where N is the initial number of the firework population, E r and E n are the parameters controlling the explosion radius and the number respectively, f min and f max are the minimum and maximum fitness respectively, and ε is a very small real number used to avoid the denominator being zero;

[0094] S5) Randomly select a firework and perform Gaussian mutation operation on each of its dimensions:

[0095] x ik = x ik g;

[0096] where x fik is the position of the i-th individual in the k-th dimension, and g is a random number obeying the Gaussian distribution with a mean of 1 and a variance of 1;

[0097] Map the fireworks beyond the boundary range to within the boundary range:

[0098]

[0099] where, are the upper and lower bounds of the k-th dimension respectively.

[0100] S6) Combine the initial fireworks, explosion sparks, and Gaussian mutation sparks to form a population set K. Retain the optimal solution in each round of iteration, and select the remaining N - 1 sparks according to the Euclidean distance between each firework and the optimal firework. The selection strategy is as follows:

[0101]

[0102] where R(x fi) is the Euclidean distance between the remaining fireworks and the optimal firework, and P(x fi ) is the probability that the firework x fi is selected;

[0103] S7) When the number of iterations reaches the set value, the optimization ends;

[0104] Use the optimized parameter values as the initial values of the BP neural network, and fuse the transformed depth camera data and lidar point cloud data. The steps are as follows:

[0105] S31 Import the input variables. The i-th neuron in the input layer is x i , where x1 is the depth camera data and x2 is the lidar data, and the two together form the input vector X = (x1, x2) T ;

[0106] S32 Calculate the input value i and output value of the i-th neuron h in the hidden layer. The calculation process is as follows:

[0107]

[0108] where n is the number of neurons in the input layer, is the weight of the hidden layer, b1 is the bias of the hidden layer, and f in (·) is the activation function of the hidden layer;

[0109] f in (·) selects the hyperbolic tangent function, and its expression is:

[0110]

[0111] S33 Calculate the input value y in and output value y out of the neuron y in the output layer. The calculation process is as follows:

[0112]

[0113] where m1 is the number of neurons in the hidden layer, is the weight of the output layer, b2 is the bias of the output layer, and f out (·) is the activation function of the output layer;

[0114] f out (·) selects the sigmoid function, and its expression is:

[0115]

[0116] S34 When the set number of iterations is reached, the output value yout This is the updated obstacle information.

[0117] Furthermore, in step S3, the method for training the neural network is the gradient descent method, and the iterative formulas for the weights and biases of the output layer are as follows:

[0118]

[0119] where w out ′ and b2′ are the values of the weights and biases of the output layer to be updated after iteration, η is the learning rate, E(m) is the evaluation function, O(m) is the output value of the m-th neuron in the output layer, and O1(m) is the expected output value of the m-th neuron in the output layer;

[0120] Judge the magnitude of E(m). When E(m) ≥ 0.5, update the weights and biases of the output layer, that is, let:

[0121]

[0122] In the preferred solution, in step 1, the lidar used is the SlAMTEC laser scanning ranging radar;

[0123] The depth camera used is the Leap Motion depth camera.

[0124] Embodiment 2

[0125] As Figure 3 shown, in combination with Embodiment 1, a wheeled inspection robot obstacle detection system for an unmanned intelligent power station based on multi-sensor fusion is further described to implement the method for detecting obstacles of a wheeled inspection robot for an unmanned intelligent power station based on multi-sensor fusion described in the above embodiment. The system includes a data acquisition module, a first data processing module, a second data processing module, and a data fusion module;

[0126] The data acquisition module uses a depth camera to collect depth data of the area in front of the robot, uses a lidar to collect point cloud data of the area in front of the robot, and sends the collected data to the first data processing module;

[0127] The first data processing module is used to receive the data collected by the data acquisition module, convert the depth data of the depth camera into point cloud data, and then send the data to the second data processing module;

[0128] The second data processing module is used to receive the data from the first data processing module, change the converted depth data from the depth camera coordinate system to the lidar coordinate system, and then send the data to the data fusion module;

[0129] A data fusion module, configured to receive data from the second data processing module, and fuse the converted depth data with the point cloud data of the lidar using an optimized BP neural network algorithm to obtain more accurate obstacle information.

[0130] The above embodiments are only the preferred technical solutions of the present invention and should not be regarded as limitations to the present invention. The protection scope of the present invention should be the technical solutions recorded in the claims, including equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. An obstacle detection method for a wheeled inspection robot of an unmanned intelligent power station based on multi-sensor fusion, characterized in that: The method includes the following steps: S1. Obtain the sensor data of the inspection robot, including lidar and depth camera data; S2. Convert the depth data of the depth camera into point cloud data, and transform the converted depth data from the depth camera coordinate system to the lidar coordinate system; S3. Use the BP neural network algorithm to fuse the depth camera data and lidar data to obtain more accurate obstacle information.

2. The obstacle detection method for the wheeled inspection robot of the unmanned intelligent power station based on multi-sensor fusion according to claim 1, wherein: In step S1, the lidar selected is the Slan laser scanning ranging radar; The depth camera selected is the Lobei Zhongguang depth camera.

3. The obstacle detection method for the wheeled inspection robot of the unmanned intelligent power station based on multi-sensor fusion according to claim 1, characterized in that: In the said step S2, the conversion between the depth data and the point cloud data is the conversion between the image coordinate system O-xyz and the world coordinate system O w -x w y w z w The conversion relationship is as follows: Where f is the camera focal length and d is the depth value.

4. The obstacle detection method of the wheeled inspection robot for the unmanned intelligent power station based on multi-sensor fusion according to claim 1, wherein: In step S2, the conversion relationship between the depth camera coordinate system and the lidar coordinate system is: Taking the optical center O of the depth camera c as the origin, establish the depth camera coordinate system O c -x c y c z c , taking the center O of the lidar r as the origin, establish the lidar coordinate system O r -x r y r z r , the height difference between O c and O r is H, and the horizontal distance is L; For point P, its coordinates in the depth camera coordinate system are (x c , y c , z c ), and its coordinates in the lidar coordinate system are (x r , y r , z r ). The relationship between the two is as follows: Rewrite equation (2) in matrix form, we get:

5. The obstacle detection method for the wheeled inspection robot of the unmanned intelligent power station based on multi-sensor fusion according to claim 1, characterized in that: In step S3, the BP neural network algorithm includes an input layer, a hidden layer, and an output layer. The input layer includes depth camera data and lidar data. The hidden layer includes m1 neurons, and the output layer outputs the updated obstacle information; The steps for the BP neural network algorithm to update the obstacle information are as follows: S31 imports input variables. The i-th neuron in the input layer is x i , where x1 is depth camera data and x2 is lidar data, and the two together form the input vector X = (x1, x2) T ; S32 calculates the input value of the i-th neuron h in the hidden layer i and the output value The calculation process is as follows: The calculation process is as follows: where n is the number of neurons in the input layer, is the weight of the hidden layer, b1 is the bias of the hidden layer, and f in (·) is the activation function of the hidden layer; S33 calculates the input value y of neuron y in the output layer in and the output value y out , and the calculation process is as follows: Among them, is the weight of the output layer, b2 is the bias of the output layer, and f out (·) is the activation function of the output layer; S34 When the set number of iterations is reached, output the value y out That is the updated obstacle information.

6. The obstacle detection method for the wheeled inspection robot of the unmanned intelligent power station based on multi-sensor fusion according to claim 5, wherein: In step S3, the method for training the neural network selects the gradient descent method. The iterative formulas for the weights and biases of the output layer are: where, w out′ , b2′ are the values of the output layer weights and biases to be updated after iteration, η is the learning rate, E(m) is the evaluation function, O(m) is the output value of the m-th output layer neuron, and O1(m) is the expected output value of the m-th output layer neuron; Judge the magnitude of E(m). When E(m)≥0.5, update the weights and biases of the output layer, that is, let:

7. The obstacle detection method for the wheeled inspection robot of the unmanned intelligent power station based on multi-sensor fusion according to claim 5, characterized in that: In the said step S32, the activation function f in (·) of the hidden layer selects the hyperbolic tangent function, and its expression is:

8. The obstacle detection method of the wheeled inspection robot for unmanned intelligent power station based on multi-sensor fusion according to claim 5, characterized in that: In the described step S33, the activation function f out (·) of the hidden layer selects the sigmoid function, and its expression is:

9. The obstacle detection method for a wheeled inspection robot of an unmanned intelligent power station based on multi-sensor fusion according to claim 1 or 5, characterized in that: For the BP neural network algorithm, first optimize the initial values of its parameters using the fireworks algorithm; The optimization steps are as follows: S1) Select the parameters to be optimized for the BP neural network: the weights and biases of the hidden layer, the weights and biases of the output layer, and the learning rate; S2) Encode the above parameters in the form of individual fireworks, and determine that the dimension D of each firework x fi is: D = n IW(1,1) + n b(1,1) + n IW(2,1) + n b(2,1) (10); Among them, n IW (1, 1) is the number of weights between the input layer and the hidden layer of the neural network, n b (1, 1) is the number of thresholds of the hidden layer neurons, n IW (2, 1) is the number of weights between the hidden layer and the output layer of the neural network, n b (2, 1) is the number of thresholds of the output layer neurons. For each firework x fi perform random initialization, and set the dimensional range of each firework to [-1, 1]; S3) Calculate the fitness value f(x of each firework using the sum of squared errors fi ) Where t is the output expectation of the neural network parameters, y is the output value of the neural network parameters, and s is the number of neurons in the output layer; S4) Explosion radius A of the fireworks explosion i and the number of sparks S generated i are as follows: Among them, N is the initial number of the fireworks population, E r and E n are parameters for controlling the explosion radius and quantity respectively, f min and f max are the minimum and maximum fitness values respectively, and ε is a very small real number used to avoid a zero denominator; S5) Randomly select a firework and perform Gaussian mutation operations on each of its dimensions: x fik = x fik g(13); where x fik is the position of the i-th individual in the k-th dimension, and g is a random number following a Gaussian distribution with a mean of 1 and a variance of 1; Map the fireworks beyond the boundary range to within the boundary range: wherein, are respectively the upper bound and the lower bound of the k-th dimension. S6) Combine the initial fireworks, explosion sparks, and Gaussian mutation sparks to form a population set K. Retain the optimal solution in each iteration, and select the remaining N - 1 sparks according to the Euclidean distance between each firework and the optimal firework. The selection strategy is as follows: Among them, R(x fi ) is the Euclidean distance between the remaining fireworks and the optimal firework, and P(x fi ) is the probability that the firework x fi is selected; S7) When the number of iterations reaches the set value, the optimization ends.

10. An obstacle detection system for a wheeled inspection robot of an unmanned intelligent power station based on multi-sensor fusion, which is used to implement the obstacle detection method for the wheeled inspection robot of the unmanned intelligent power station based on multi-sensor fusion according to any one of claims 1 to 9, characterized in that: It includes a data acquisition module, a first data processing module, a second data processing module, and a data fusion module; The data acquisition module uses the depth camera to collect the depth data of the area in front of the robot, uses the lidar to collect the point cloud data of the area in front of the robot, and sends the collected data to the first data processing module; The first data processing module is used to receive the data collected by the data acquisition module, convert the depth data of the depth camera into point cloud data, and then send the data to the second data processing module; The second data processing module is used to receive the data from the first data processing module, transform the already converted depth data from the depth camera coordinate system to the lidar coordinate system, and then send the data to the data fusion module; The data fusion module is used to receive the data from the second data processing module, and fuse the converted depth data with the lidar point cloud data using the optimized BP neural network algorithm to obtain more accurate obstacle information.

Images