Remote monitoring control method and system for fan blade maintenance robot based on self-built ecology

Abstract

A kind of remote monitoring control method and system of fan blade maintenance robot based on self-built ecology, comprising: maintenance robot collects fan blade image, and collects environmental information by sensor;Improved ORB algorithm is used to extract fan blade image features;Data structure is constructed and efficient traversal search is carried out, similar data points and new data points are quickly found with original map, and real-time map updating is completed;Environment information and updated map are synchronized to terminal equipment;Terminal carries out danger early warning and risk assessment to fan blade according to environment information and updated map;Control instruction is generated according to danger early warning result and risk assessment result, and fan blade maintenance robot works;The present application completes real-time map updating by improving ORB algorithm and FLANN, combines with multi-sensor self-built virtual ecological environment, relies on terminal application program to remotely monitor the real-time state of fan blade, improves fan blade maintenance efficiency and operation safety.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot control, and particularly relates to a fan blade maintenance robot remote monitoring control method and system based on self-built ecology. BACKGROUND

[0002] The detection and maintenance of fan blades are difficult. The fan blades are usually located at a height of dozens of meters or even hundreds of meters. Artificial detection and maintenance of the fan blades need to use a basket, an elevator and other equipment. The working environment is harsh, the working strength is large, and the safety risk is high. The existing nondestructive testing technology mainly adopts a method of combining artificial detection with nondestructive testing equipment. The detection method can be divided into artificial non-contact observation and inspection and contact measurement and detection according to the contact form. The traditional detection method is that an inspector uses a detection device to enter the inside of a wind turbine, and mainly uses artificial visual inspection, supplemented by a crack measuring instrument, a camera and other tools to obtain the actual state of the fan blade assembly. With the continuous development of unmanned aerial vehicle technology and low-altitude remote sensing technology, an unmanned aerial vehicle system carrying a visual and remote sensing detection device starts to be applied to the inspection and detection of wind turbines. This blade inspection method mainly relies on an unmanned aerial vehicle, a sensor and a camera to perform remote non-contact detection, and obtains information such as the surface morphology and crack damage of the blade by collecting and analyzing the images of the bridge.

[0003] Such unmanned aerial vehicle blade detection technology mainly focuses on remote non-contact aerial observation and inspection, has a small application range, low detection precision and difficulty in detection operation in complex outdoor environments and working conditions. At the same time, it cannot replace detection personnel to perform close contact detection by holding professional detection devices such as an ultrasonic non-metal internal defect detector and a crack detector, which limits the application of the unmanned aerial vehicle in holographic detection of the blade. With the increase in the number of wind turbines, the industry has higher requirements for the maintenance and management of wind power systems, and the reliability maintenance of the fan blades gradually becomes the focus of the industry development. The traditional fan maintenance adopts a detection-then-maintenance mode, which is time-consuming and difficult to ensure the safety of the operation. SUMMARY

[0004] To solve the problems in the prior art, the application provides a remote monitoring control method for a fan blade maintenance robot based on a self-built ecology, which comprises the following steps: a maintenance robot collects fan blade images and collects environmental information through a sensor; an improved ORB algorithm is used to extract features of the fan blade images; a data structure is constructed, efficient traversal search is performed through the features of the fan blade images, similar data points and new data points are quickly found, and real-time map updating is completed; wind power, temperature, bird flocks and other environmental information and the updated map are synchronously sent to a terminal; the terminal performs danger early warning and risk assessment on the fan blades according to the environmental information and the new map; control instructions are generated according to the danger early warning result and the risk assessment result, and the fan blade maintenance robot is controlled according to the control instructions.

[0005] A remote monitoring control system for a fan blade maintenance robot based on a self-built ecology, which comprises an information collection module, a data interaction module, a feature extraction module, a risk assessment module and an execution control module.

[0006] The information collection module is used to collect fan blade image information, wind speed information, temperature information and other surrounding environmental information.

[0007] The data interaction module is used to realize data interaction between a sensor and a cloud server and between the cloud server and a terminal.

[0008] The feature extraction module is used to extract feature point information of a picture and update a map according to the extracted feature point information.

[0009] The risk assessment module is used to perform robot operation feasibility assessment in combination with environmental information of a current robot.

[0010] The execution control module is used to control a robot according to an instruction issued by a user on a terminal.

[0011] The application has the following beneficial effects:

[0012] The improved ORB algorithm and FLANN are used to extract, screen and match feature points of fan blade images, the extraction process of the feature points is optimized, and the accuracy of map information updating is improved. The application combines environmental information collected by other sensors on a robot to build a virtual ecological environment, and relies on a terminal application program to remotely monitor the real-time state of fan blades; the maintenance feasibility of the fan blades is determined, safe and effective maintenance of the fan blades is realized, and therefore the maintenance efficiency and operation safety of the fan blades are improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 A flow chart of the remote monitoring control system for the fan blade maintenance robot based on the self-built ecology in the embodiments of the application;

[0014] Figure 2 SVM-based fan blade damage classification flowchart in the embodiment of the present application;

[0015] Figure 3 total system structure framework in the embodiment of the present application;

[0016] Figure 4 terminal device interface schematic diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0018] A fan blade maintenance robot remote detection control method based on self-built ecology, as shown in Figure 1 the method comprises the following steps: a maintenance robot collects fan blade images and collects environmental information through a sensor; an improved ORB algorithm is used to extract fan blade image features; a data structure is constructed by using FLANN and high-efficiency traversal search is performed to quickly find similar data points and newly-added data points in the original map, and the map is updated in real time; the collected environmental information and the updated map are synchronized to a terminal device; the terminal performs danger early warning and risk assessment on the fan blade according to the environmental information and the updated map; control instructions are generated according to the danger early warning result and the risk assessment result, and the fan blade maintenance robot is controlled according to the control instructions. The present application builds an ecological environment on the terminal by collecting information through multiple sensors, and performs safety early warning on the fan according to the actual situation, so as to remotely control the robot to act, realize remote monitoring and maintenance of the fan blade, and effectively enhance the flexibility and safety of the robot maintenance work.

[0019] In the present embodiment, an RGB-D camera installed at the front end of the robot is used to obtain the distance information of each pixel point in the scene to the camera, provide the contour information in the image and the three-dimensional coordinates of each point, and be used for three-dimensional reconstruction of the object.

[0020] An ultrasonic wind speed sensor is used to measure the wind speed, and a wind vane sensor is used to determine the direction of the wind by detecting the direction of the wind vane. A thermistor sensor is used to measure the ambient temperature around the fan blade, and an infrared sensor is used to collect the bird group information around the fan blade.

[0021] In the embodiment, the image collected by RGB-D is processed by histogram equalization to expand the pixel contrast of the image. Then the scale space information of the feature points of the image is obtained by the image Gaussian pyramid, and finally the image contrast system is constructed to obtain the adaptive threshold of the feature points. The histogram equalization changes the gray scale distribution of the image, and enhances the difference of the gray scale values of different regions of the image. According to the principle of histogram equalization, the relationship between the input image and the output image gray scale is:

[0022]

[0023] Wherein, h and w are the height and width of the image respectively, k represents the gray scale of the image, J i represents the gray scale of the i-th pixel point of the image, D i is the output gray scale of the i-th pixel point, and hist(k) is the number of pixel points with gray scale k.

[0024] In the embodiment, the image Gaussian pyramid module processes the enhanced map, including: determining the scaling factor of each layer of image in the pyramid to realize the scale invariance of the feature points; calculating the total area of the pyramid; and calculating the number of feature points to be allocated per unit area according to the total area of the pyramid.

[0025] The scaling factor of each layer of image in the pyramid is determined to realize the scale invariance of the feature points. The scaling factor is set to 1.3, each layer is reduced by 1.3 times according to the size of the previous layer, and a 5-layer image pyramid is constructed.

[0026] The total area of the pyramid is:

[0027] S=h*w*(s 2 ) 0 +h*w*(s 2 ) 1 +…+h*w*(s 2 ) (n-1)

[0028] The number of feature points to be allocated per unit area is:

[0029] N a =N / S

[0030] The number of feature points to be allocated in the i-th layer is:

[0031]

[0032] Wherein, s is the scaling factor of the image, n is the number of layers of the pyramid, and N is the number of feature points required.

[0033] The system compares the difference between the average value of the pixel matrix of the image and the gray scale value of each point, constructs the image contrast system to obtain the adaptive threshold of the image, and the expression is:

[0034]

[0035]

[0036] wherein τ is a proportional coefficient, 0 < τ < 1, h is the image height, l(x, y) is the image selected point gray value, v lh×w is the average value of the gray matrix pixels.

[0037] In the embodiment, the real-time updating of the map is completed by matching and comparing the feature points. The matching and comparing of the feature points of the fan blade image and the map comprises: constructing a data structure by using FLANN, efficiently traversing and searching the structure, using an improved FLANN matching method combined with a PROSAC algorithm, quickly finding similar data points and new data points in the original map, and specifically comprising: FLANN mainly realizes feature matching according to KD-TREE, the space to which the feature points belong is an m-dimensional real vector space R m , the Euclidean distances between the image feature point p and each feature point of the sorting image are identified, the corresponding feature point of the feature point p is found, and the following formula is calculated:

[0038]

[0039] wherein Q is the Euclidean distance of two feature points, (x, y) is the pixel coordinate of the feature point, X v is the pixel coordinate value in the v-th space, all Euclidean distances Q are stored by using the KD-TREE structure, the matching point pair set of the feature point p is obtained, the matching point pair set is searched recursively from top to bottom, the feature point p corresponding to the minimum Q is obtained, the matching point pair (p, q) is obtained, and the matching point pair is obtained by performing the above operation on each feature point.

[0040] In the embodiment, the PROSAC algorithm is combined to remove the false matching point pairs, and the steps are as follows:

[0041] (1) initialize the related parameters, input the maximum iteration number D max , the error limit E lim of all points within the judgment range, and the number threshold S lim .

[0042] (2) arrange the ratio of the nearest neighbor distance and the second nearest neighbor distance in the matching point pairs in descending order of matching quality, select the first 5 groups of matching point pairs, calculate the matching point error, if the matching point error is greater than E lim , the matching point is an outlier, otherwise, the matching point is an inlier, and all the matching point pairs are traversed.

[0043] (3) judge the iteration number, if the iteration number is less than D max , continue to iterate, otherwise, stop and output the matching point pair set with the most inliers in the statistical data.

[0044] (4) Compare the number of inliers with S lim , if the number is less than S lim , continue iteration, otherwise output the inlier set, complete the optimization of the matching method. Keep the matching successful position recognition feature point data, save this part of the data point Γ i together to the database and replace the prior data point Γ i-1 to obtain the latest map file.

[0045] In this embodiment, the wind, temperature, and bird group information collected by the robot sensor are synchronized in real time with the updated map on the terminal, and data exchange is realized between the robot and the terminal. The robot packages the collected sensor data, map information, etc. into a message and sends it to the cloud server and then to the terminal through the communication protocol. The terminal displays the map, including real-time updated wind, stability, and bird group information, and users can intuitively understand the environmental information perceived by the robot through the map interface.

[0046] The cloud server receives data from the robot and stores it in the database. At the same time, the cloud server analyzes historical data and generates a report and sends it to the terminal. Through the cloud server, a two-way connection is established between the terminal and the robot, and the terminal updates the new message sent by the cloud server in real time.

[0047] In this embodiment, the terminal is built-in with a danger warning model, which analyzes the state parameters collected by the sensor to detect the presence of danger in the wind turbine blade and implement an alarm. Support Vector Machine (SVM) is used to classify the extracted parameter features, outputting 0, 1, 2, 3, 4, and five results. Among them, 0 means no damage to the wind turbine blade, 1 means paint damage to the wind turbine blade, 2 means cracks in the wind turbine blade, 3 means asymmetric deformation of the wind turbine blade, and 4 means others.

[0048] T(s) = (m, n) s i + b

[0049] where s i is the input data, T(s) is the output result with values 0, 1, 2, 3, or 4, m is the weighted vector value, n is the feature vector, and b is the bias value.

[0050] When the classification result T(s) = 0, it is judged as no damage.

[0051] When the classification result T(s) = 1, it is judged as paint damage.

[0052] When the classification result T(s) = 2, it is judged as cracks in the wind turbine blade.

[0053] When the classification result T(s) = 3, it is judged as asymmetric deformation of the wind turbine blade.

[0054] According to the classification of the fan fault by the SVM, the terminal displays a corresponding alarm. According to the wind speed, temperature and whether there are birds on the fan blade, threshold values are set for risk assessment. The wind speed threshold is set as α1, if the wind speed v < α1, the safety factor β1 = 1; the temperature threshold is set as α2, if the temperature t < α2, the safety factor β2 = 1; if there are birds, the safety factor β3 = 1, if β1, β2 and β3 are all 1, it is determined that the operation environment is safe.

[0055] In the embodiment, the technician controls the robot action through the terminal to finally realize the fan blade maintenance work, a user-friendly interface is designed in the terminal to allow the user to select the maintenance task, set the action plan of the robot and monitor the real-time data. The robot receives the instruction from the terminal and analyzes the corresponding control parameters, according to the analyzed instruction, the robot control system controls the robot movement to execute the maintenance task.

[0056] The fan blade damage classification flowchart based on the SVM established by the application is shown in Figure 2 The total system structure framework is shown in Figure 3 The terminal interface is shown in Figure 4

[0057] A fan blade maintenance robot remote monitoring control system based on self-built ecology, comprising: an information acquisition module, a data interaction module, a feature extraction module, a risk assessment module and an execution control module;

[0058] The information acquisition module is used to acquire fan blade image information, wind speed information, temperature information and other surrounding environment information;

[0059] The data interaction module is used to realize the data interaction of the sensor, the cloud server and the terminal;

[0060] The feature extraction module is used to extract the feature point information of the picture and update the map according to the extracted feature point information;

[0061] The risk assessment module is used to evaluate the operation feasibility of the robot in combination with the current environment information of the robot.

[0062] The execution control module is used to control the robot according to the instruction issued by the user on the terminal.

[0063] The system embodiment of the application is the same as the method embodiment.

[0064] ​The above examples further illustrate the objects, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A wind turbine blade maintenance robot remote monitoring control method based on self-built ecology, characterized in that, The application relates to a wind turbine blade maintenance robot system. The robot collects wind turbine blade images and environmental information through sensors on the robot; An improved ORB algorithm is used to extract wind turbine blade image features; a data structure is constructed, efficient traversal search is carried out through wind turbine blade image features, similar data points and new data points in an original map are quickly found, and real-time map updating is completed; the collected environmental information and the updated map are synchronously sent to a terminal; the terminal carries out danger early warning and risk assessment on the wind turbine blade according to the environmental information and the updated map; Control instructions are generated according to the danger early warning result and the risk assessment result, and the wind turbine blade maintenance robot is controlled according to the control instructions; The matching and comparison of the fan blade image feature points and the map comprises: constructing a data structure by using FLANN, performing efficient traversal search on the structure, and using an improved FLANN matching method combined with a PROSAC algorithm to quickly find similar data points and new data points in the original map, wherein the improved FLANN matching method realizes feature matching through KD-TREE, that is, the space to which the feature points belong is an m-dimensional real number vector space R m , the Euclidean distance between the image feature point p and each feature point of the sorting image is identified, the corresponding feature point of the feature point p is found, and the following formula is calculated: wherein Q is the Euclidean distance of two feature points, (x, y) is the pixel coordinate of the feature point, X v is the pixel coordinate value in the vth space, all the Euclidean distances Q are stored by a KD-TREE structure to obtain a matching point pair set of the feature point p, a feature point p corresponding to the minimum Q is obtained by recursively searching the matching point pair set from top to bottom, and a matching point pair (p, q) is obtained, and the above operation is performed on each feature point to obtain a matching point pair; The quick finding of similar data points and new data points in the original map comprises the following steps: (1) initialize relevant parameters, input the maximum iteration number D max , error limit E of all points in the judging range lim , number threshold S lim ; (2) The ratio of the nearest and the second nearest distance in the matched point pair is arranged in descending order of matching quality, and the first 5 matched point pairs are selected to calculate the matching point error. If the matching point error is greater than E lim , it is an outlier, otherwise it is an inlier, until all matched point pairs are traversed. (3) judge the iteration number, if less than D max then continue iteration, otherwise stop and output the matched point pair set with the most inner points in the statistical data; (4) compare the number of inliers with S lim , if the number is less than S lim , continue iteration, otherwise output the inlier set, complete the optimization of the matching method; keep the feature point data of the matching successful position, save this part of data points Γ i to the database together and replace the prior data points Γ i-1 to obtain the latest map file; The danger early warning and risk assessment on the wind turbine blade comprise the following steps: the wind turbine blade image features are acquired, the wind turbine blade image features are input into a support vector machine model, and the wind turbine damage category is obtained; a threshold value is set, the threshold value comprises a wind speed threshold value alpha 1, a temperature threshold value alpha 2 and whether there are bird groups or not; the set threshold value is compared with the current environmental information, if the wind speed v is less than alpha 1, the safety coefficient beta 1 is 1; the temperature threshold value is set as alpha 2, if the temperature t is less than alpha 2, the safety coefficient beta 2 is 1; if there are bird groups, the safety coefficient beta 3 is 1, if beta 1, beta 2 and beta 3 are all 1, it is determined that the operation environment is safe; otherwise, the environment is dangerous; when the environment is safe, the robot control instructions are generated, otherwise, the robot control instructions are not generated.

2. The remote monitoring and control method for the fan blade maintenance robot based on self-built ecology according to claim 1, characterized in that, The environmental information comprises wind power, temperature and bird group information around the wind turbine. 3.The remote detection and control method of the wind turbine blade maintenance robot based on self-built ecology according to claim 1, characterized in that, The improved ORB algorithm used to extract the wind turbine blade image features comprises the following steps: the collected wind turbine blade image is subjected to histogram equalization processing; the processed image is input into an image Gaussian pyramid module to obtain scale space information of picture feature points; a picture comparison system is constructed according to the scale space information of the picture feature points to obtain a feature point adaptive threshold value; and the feature points of the image are extracted according to the feature point adaptive threshold value.

4. The remote monitoring and control method for the fan blade maintenance robot based on self-built ecology according to claim 3, characterized in that, The histogram equalization processing of the collected wind turbine blade image comprises the following steps: a histogram equalization method is used to enhance the difference of the gray scale values of different regions of the wind turbine blade image, wherein the relationship between the input image and the output image gray scale levels is as follows: where h and w are the height and width of the image, respectively, k represents the gray level of the image, J i represents the gray level of the i-th pixel point of the image, D i is the output gray level of the i-th pixel point, and hist(k) is the number of pixel points with a gray level of k.

5. The method according to claim 3, wherein, The image Gaussian pyramid module processes the enhanced map, which comprises the following steps: the scaling factor of each layer of images in the pyramid is determined to realize the scale invariance of the feature points; the total area of the pyramid is calculated; the number of feature points to be distributed per unit area is calculated according to the total area of the pyramid; the total area of the pyramid is as follows: S = h * w * (s 2 ) 0 + h * w * (s 2 ) 1 +... + h * w * (s 2 ) (n-1) The number of feature points to be distributed per unit area is as follows: N a = N / S The number of feature points to be distributed in the i-th layer is as follows: Wherein, s is the scaling factor of the image, n is the number of pyramid layers, and N is the number of required feature points.

6. The method according to claim 3, wherein, The feature point adaptive threshold value is as follows: where τ is a proportional coefficient, h is the image height, and l(x, y) is the gray value of the selected point of the image, is the average value of the pixels of the gray matrix. 7.The remote monitoring and control method of the fan blade maintenance robot based on self-built ecology according to claim 1, characterized in that, The synchronization of the environmental information and the updated map to the terminal comprises the following steps: the robot packages the collected data information and the map into a message and sends the message to a cloud server through a communication protocol; the cloud server analyzes the received information and generates a report; the generated report is sent to a user terminal; and the user terminal updates the information sent by the cloud server in real time.

8. A remote monitoring control system for a self-ecological fan blade maintenance robot, the system being configured to perform the self-ecological fan blade maintenance robot remote monitoring control method according to any one of claims 1 to 7. ​ ​ The information collection module is used for acquiring fan blade image information, wind speed information, temperature information and other surrounding environment information; The data interaction module is used for realizing data interaction of the sensor, the cloud server and the terminal; The feature extraction module is used for extracting picture feature point information and updating the map according to the extracted feature point information; The risk assessment module is used for combining current environment information of the robot to evaluate the operation feasibility of the robot; The execution control module is used for controlling the robot according to the instruction issued by the user on the terminal.

Images