Fan blade online safety monitoring system based on image recognition

Abstract

The invention discloses a fan blade online safety monitoring system based on image recognition, and particularly relates to the field of computer vision, a five-module collaborative system is constructed around fan blade collision safety risk prevention and control, a data acquisition and preprocessing module is provided with a camera, and video images of a blade scanning area are continuously acquired and preprocessed; the intelligent sensing and target recognition module marks the preprocessed data, and accurately recognizes obstacle types in different environments; the early-warning and decision-making module tracks the movement track of an obstacle in real time, evaluates the collision possibility and the risk level, and outputs differential early-warning signals in a matching manner; the closed loop and model optimization module divides risk intervals, collects fuzzy and failure assessment cases to construct a difficult case library, and the difficult case library is used for model training iteration after correction; and when the risk level exceeds a threshold value, the deployment and operation module matches a historical processing result and executes a protection action, a complete closed loop is formed, and the operation safety of the fan is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and more particularly, to an online safety monitoring system for fan blades based on image recognition. BACKGROUND

[0002] The prior art mainly uses millimeter wave radar for obstacle detection to achieve the purpose of safety protection. In the fan blade anti-collision system, the millimeter wave radar is used as the core sensing unit. The radar sensor is installed and calibrated on the top or both sides of the fan cabin to ensure that the scanning sector can completely cover the entire swept area of the blade. The radar actively transmits millimeter wave signals to the detection area at a specific frequency and synchronously receives the echoes reflected by the front obstacles. The original radio frequency signals are processed by the built-in processor to generate raw point clouds containing distance, azimuth angle and radial velocity information. Then these sparse point clouds are aggregated into individual target objects, and each target is assigned a unique ID. High-frequency updated data, including the real-time distance, radial velocity, azimuth angle and motion trajectory trend of the target, are combined with the real-time rotation position data of the fan blade through a high-speed communication interface to accurately calculate the relative speed between the obstacle and the blade, predict the future minimum distance, and ultimately serve as the most critical and reliable input for the collision possibility and risk level assessment model.

[0003] However, it still has some disadvantages in actual use, such as 1. Poor adaptability to bad weather. Millimeter wave radar is easily affected by rain, snow, heavy fog and other weather conditions, resulting in attenuation or clutter of echo signals, reducing the accuracy and stability of target detection; 2. Single target information dimension. Radar mainly provides distance, speed, azimuth angle and other information of the target, and cannot identify the specific type of obstacle, which makes the risk assessment model rely only on motion parameters, not comprehensive and accurate; 3. The model is rigid and difficult to continuously optimize. The detection and evaluation algorithms of traditional systems are usually fixed and lack effective feedback and learning mechanisms. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the present application provides an online safety monitoring system for fan blades based on image recognition, which solves the problems raised in the background art by the following scheme.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme: an online safety monitoring system for fan blades based on image recognition, comprising a data acquisition and preprocessing module: through the deployment of cameras on the fan cabin and the tower drum, continuously acquiring video and image data covering the swept area of the fan blades, and preprocessing the images or videos;

[0006] Intelligent perception and target recognition module: collects pre-processed images or videos, annotates image or video data under different environmental conditions, and identifies different obstacle types;

[0007] Early warning and decision-making module: Tracks the identified obstacles in real time, obtains their movement trajectory, assesses the probability of the obstacle colliding with the blade and the collision safety risk level, and provides different early warning signals for different collision safety risk levels;

[0008] Closed-loop and model optimization module: Divide into three risk ranges based on the probability of collision, collect cases in the fuzzy range and cases that failed to be evaluated to build a difficult case library, correct the cases in the difficult case library, and output the corrected cases to train the model;

[0009] Deployment and Operation Module: When the calculated collision safety risk level is higher than the threshold, historical processing results are matched, and protective actions are executed based on the matched historical processing results.

[0010] The technical effects and advantages of this invention are as follows:

[0011] 1. Higher recognition accuracy in complex environments: Through multimodal image enhancement and precise image correction, the system can effectively overcome the impact of severe weather and perform stable and accurate recognition and classification of obstacles; 2. Stronger obstacle type recognition capability: The system can not only detect obstacles but also accurately identify their specific types, providing a key basis for subsequent risk level assessment; 3. The system has self-optimization and closed-loop learning capabilities: Through the closed-loop and model optimization modules, the system can automatically collect cases for iterative training of the model. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall structure of the present invention.

[0013] Figure 2 This is a schematic diagram of the operation process of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] like Figure 1 The image recognition-based online safety monitoring system for wind turbine blades includes a data acquisition and preprocessing module, an intelligent sensing and target recognition module, an early warning and decision-making module, a closed-loop and model optimization module, and a deployment and operation and maintenance module.

[0016] like Figure 2 The system shown is an online safety monitoring system for wind turbine blades based on image recognition. A camera is mounted on the wind turbine blade to collect image information. The collected image information undergoes three preprocessing stages: image enhancement, image correction, and static background subtraction, to identify the location and type of obstacles.

[0017] Data acquisition and preprocessing module: Through cameras deployed on the wind turbine nacelle and tower, it continuously acquires video and image data covering the area swept by the wind turbine blades, and preprocesses the images or videos.

[0018] High-definition cameras are deployed on the top of the wind turbine nacelle and in the middle of the tower to accurately identify potential obstacles that could endanger the blades. The cameras form an intersecting field of view, fully covering the blade sweep area. Obstacle identification is performed on the camera-captured images. The average value of the initial N interference-free images is then used to construct the background model. Where B(x,y) is the pixel value of the background model at (x,y), I(x,y) is the pixel value of the k-th frame at (x,y), (x,y) is the image pixel coordinate, and the grayscale value of the pixel at (x,y) ∈ [0, 255]; calculate the pixel difference D(x,y) = |I(x,y) - B(x,y)| between the previous frame and the background model, and calculate the average difference value of the region. Among them, Ω i Let i be the set of pixels in the i-th connected region. The larger the area, the higher the probability that the area is an obstacle. In this example, it is based on Numerical recognition of images, if The image does not contain any obstacle-free frames. If images are manually identified, If the image contains obstacles, then image preprocessing is performed on the image containing obstacles.

[0019] Preprocessing includes an image enhancement stage, an image correction stage, and a static background subtraction stage. The input image or video stream enters the image enhancement stage, where contrast-limited adaptive histogram equalization is used to improve the overall contrast under adverse weather conditions, and gamma correction is used to adjust the brightness of overexposed or underexposed areas. The enhanced image or video enters the image correction stage, where iterative optimization based on a calibration board is used to correct image distortion edges. The enhanced and corrected image is then used for static background modeling and subtraction.

[0020] The image enhancement stage divides the input image I(x, y) into M×N non-overlapping sub-blocks, I ij (x, y) = I(x, y), Where: ω=W / M, h=H / N, are the width and height of the sub-block, respectively, and W×H is the width and height of the original image.

[0021] For each sub-block I ij Calculate the grayscale histogram: k = 0, 1, ..., L-1, where L is the number of gray levels and δ is the Dirac function; the histogram of each sub-block is cropped to limit contrast. Among them, T c The cropping threshold, β is the clipping constraint parameter, which is taken as 1.5 in this example; calculate the cumulative distribution function (CDF). Gray-scale mapping function for generating sub-blocks Block artifacts are eliminated by interpolating the mapping functions of the surrounding four sub-blocks. Wherein the weighting coefficients are: Where (x, y) are the coordinates of this sub-block, (x...y...) m y n ) represents the coordinates of the four surrounding sub-blocks, and ω is the final coordinate. mn The average of the weight coefficients of the four sub-blocks; normalize the processed image to the range [0, 1]. Gamma correction is applied to the image, I g (x,y)=[I n (x,y)] γ For overexposed areas (mean image brightness μ > 0.7), γ is set to 1.5–2.5; for underexposed areas (mean image brightness μ < 0.3), γ is set to 0.4–0.7; and for normal areas (mean image brightness 0.3 ≤ μ ≤ 0.7), γ is set to 0.8–1.2. A high-precision checkerboard or dot array calibration board of known size is used.

[0022] In the image correction stage, multiple images of the calibration board are acquired from different perspectives and positions; in this example, at least 10 images are used to ensure coverage of all areas of the image. Feature points of the calibration board in each image are detected to obtain pixel coordinates (u). ij v ij ), where i is the i-th image, j is the j-th feature point in the i-th image, and the calibration plate plane is defined as the world coordinate system Z. w =0 plane, establish the world coordinates (X) of the corner points j Y j ,0); then a camera imaging model was established. Where s is the scale factor, K is the intrinsic parameter matrix, representing the camera's inherent parameters, and f x f y The focal lengths in the x and y directions are respectively, and c x c yHere, R represents the coordinates of the optical center, R is a 3×3 extrinsic parameter matrix, and t is the translation vector; normalized coordinates are then calculated. [X c ,Y c Z c ] T =R[X,Y,Z] T +t, perform distortion model calculation, the calculation formula is as follows

[0023] Where r 2 =x 2 +y 2 k1, k2, and k3 are radial distortion coefficients, describing the distortion caused by the radial curvature of the lens; p1 and p2 are tangential distortion coefficients, describing the distortion caused by lens mounting misalignment; x d y d The normalized coordinates are obtained after distortion; the obtained normalized coordinates after distortion are projected onto pixel coordinates. The intrinsic parameter matrix K and the extrinsic parameters R and t of each image are initially estimated using the least squares method, and the objective parameters are optimized using a nonlinear function. Where θ is the optimization variable, θ={K,k1,k2,k3,p1,p2,R1,t1,...,R N , t N}, containing intrinsic parameters, distortion parameters, and extrinsic parameters for all images, u ij v ij Let be the actual detected pixel coordinates of the j-th calibration board corner point in the i-th image.

[0024] The input consists of a sequence of enhanced and corrected clear images. A reference model representing a static scene is built. In this example, the average background model is used. Multiple frames of images are acquired over a period of time, ensuring that no moving targets appear during this period. For each pixel location, the average intensity of the pixel across all frames in the sequence is calculated. This average intensity constitutes the background model for that pixel. The average intensity of all pixels in the entire image is combined to form the complete background image. After the background model is built, each new input frame is compared pixel by pixel with the background model. In this example, the absolute difference is calculated, which is the absolute difference between the brightness value of each pixel in the current frame and the corresponding pixel value in the background model. The larger the difference, the greater the likelihood of a change in the pixel. Pixels with differences greater than a threshold are identified as having obstacles.

[0025] Intelligent sensing and target recognition module: collects pre-processed images or videos, annotates image or video data under different environmental conditions, and identifies different obstacle types; the annotation label consists of time + identification device serial number + weather conditions + obstacle type.

[0026] Identifying specific obstacle types is crucial for further analysis of obstacle collision safety with blades. This requires binarizing image regions identified as containing obstacles, followed by morphological processing and contour analysis of the binarized results. The YOLO deep learning object detection model is then used for end-to-end feature extraction and classification of these pixel regions, ultimately outputting a label for each obstacle type. Data label generation is an automated acquisition process. The system directly collects timestamps and unique serial numbers from the device's internal clock and hardware information to obtain time tags and device serial number tags. Weather condition tags are obtained by directly accessing authoritative historical weather data released by meteorological departments.

[0027] Early warning and decision-making module: Tracks identified obstacles in real time, obtains their trajectory, assesses the probability of collision with the blades and the level of collision safety risk, and provides different early warning signals for different levels of collision safety risk.

[0028] Using the Kalman filter algorithm, the system predicts the location of an obstacle in the new frame based on its existing motion state. It then calculates the geometric distance between the predicted location and all actual detection locations in the current frame; the closer the distance, the higher the matching probability. A convolutional neural network extracts high-dimensional feature vectors from each detection box. Cosine similarity between the current detection features and historical features of each trajectory is used to determine appearance consistency; the more similar the appearance, the higher the matching probability. Combining motion distance and appearance similarity, the system obtains a total association cost through weighted summation, which is then optimized using the Hungarian algorithm to find the optimal matching solution between the detection box and existing trajectories in the current frame. After data association, the sequence of observation points belonging to the same trajectory is connected in time to generate a smooth and accurate trajectory, and unobserved state variables such as velocity are estimated. Finally, the system outputs obstacle trajectory information, along with the instantaneous velocity and direction of motion estimated by the filtering algorithm. The motion trajectory is represented by the state vector [x, y, z, w, h, d, v]. x v y v z v w v h v d ] T Express the following: x is the horizontal coordinate of the obstacle's center, y is the vertical coordinate of the obstacle's center, z is the depth coordinate of the obstacle's center, w is the width of the obstacle, h is the height of the obstacle, d is the depth of the obstacle, and v is the width of the obstacle. x Let v be the velocity of the center of the obstacle in the X direction. y Let v be the velocity of the obstacle's center in the Y direction. z Let v be the velocity of the obstacle in the Z direction. w v is the rate of change of the obstacle's width. h v is the rate of change of the obstacle's height. dLet d be the rate of change of the dimension d of the obstacle in the depth direction.

[0029] The probability of an obstacle colliding with the wind turbine blades is assessed using the following formula: Among them, D min The calculated future closest distance is obtained by combining the obstacle's moving speed and direction with the wind turbine blade rotation speed; r o +r t V is the sum of the equivalent safety radii of the obstacle and the blade; o The speed of the obstacle is the magnitude of its movement. The faster the speed, the shorter the reaction time and the higher the risk. α and β are adjustable parameters calculated based on massive historical sample data. α is the weighting coefficient of the speed effect, and β is a smoothing constant to prevent the denominator from being 0 when the speed is 0.

[0030] The formula for assessing collision safety risk level is R = P c ×S, where P is obtained from the collision probability assessment, and S is the severity of the collision consequences, which is determined by both the type of obstacle and the collision energy; the severity of consequences S is determined by two factors: obstacle type and relative kinetic energy, one determining the inherent destructive force and the other determining the intensity of the impact; W t The collision damage coefficient is determined based on the identified obstacle type. Collision energy is proportional to kinetic energy, and kinetic energy is related to the square of velocity. Therefore, the square of the relative velocity is used to amplify the influence of the motion state. The collision kinetic energy coefficient is calculated as follows: Among them, V r The relative velocity between the obstacle and the blade is the magnitude of the obstacle's velocity V. o The magnitude of the vector sum of the blade tip velocities, V n To normalize the reference velocity, the severity S of the collision consequence is calculated using the formula S = W. t ×(1+K f) Based on the numerical value of the collision safety risk level assessment, determine the collision safety risk level of the obstacle. When R ≥ 6.0, it is judged as an extremely high-risk obstacle, and the collision may cause structural damage to the wind turbine, even catastrophic accidents; when 3.0 < R < 6.0, it is judged as a high-risk obstacle, and the collision may cause serious damage to the blades, resulting in significant economic losses and long-term shutdowns; when 1.5 < R ≤ 3.0, it is judged as a medium-risk obstacle, and the collision may cause surface damage to the blades and requires maintenance; when R < 1.5, it is judged as a low-risk obstacle, and the collision safety risk is relatively low, only causing temporary vibration or fouling; issue a red visual alarm for extremely high-risk obstacles, immediately issue a shutdown command, and send a text message or email warning; issue an orange visual alarm for high-risk obstacles, and issue a shutdown command after obtaining manual approval; issue a yellow visual alarm for medium-risk obstacles, issue a screen warning, and remotely prompt the staff; issue a blue status prompt and screen information prompt for low-risk obstacles, and do not issue additional prompts.

[0031] Closed-loop and model optimization module: Divide three risk intervals according to the collision possibility, collect cases in the fuzzy interval and cases with failed evaluations to build a difficult case library, correct the cases in the difficult case library, and output the corrected cases to train the model.

[0032] Based on the collision possibility P c The value is divided into a safe interval, a fuzzy interval, and a dangerous interval. P c ≥ 0.6 is the dangerous interval, 0.3 < P c < 0.6 is the fuzzy interval, P c ≤ 0.3 is the safe interval; P c If the value of P is in the safe interval, it is considered that the obstacle will not collide with the wind turbine blade. If it is in the dangerous interval, it is considered that the obstacle will collide with the wind turbine blade; if a collision does not occur in a case in the dangerous interval or a collision occurs in a case in the safe interval, the collision assessment of this case is incorrect; collect cases in the fuzzy interval and cases with verified incorrect collision assessments of obstacles to build a difficult case library. Each case package completely records the sensor data, image sequence, and the original output of the model at that time; these cases are pushed to the manual review platform, and the staff conducts a review, arbitrates the cases in the fuzzy interval, clearly marks them as either a collision will occur or will not occur, and corrects the previous misjudged cases. Finally, the corrected cases are output for iterative training of the model, allowing the model to learn from the cases in the difficult case library, correct the coefficients in the model calculation process, reduce the uncertainty of the future system, and gradually improve its judgment accuracy and reliability in complex scenarios.

[0033] Deployment and operation and maintenance module: When the calculated collision safety risk level is higher than the threshold, perform a match with the historical processing results and execute protection actions based on the matched historical processing results.

[0034] When the system calculates in real time that the collision safety risk level of an obstacle has entered the danger zone, it will immediately trigger the historical processing plan call process based on tag matching. First, it reads the current risk event tag and uses the tag as the search condition to start a multi-dimensional similarity matching algorithm in the historical case library to find the past cases with the highest tag matching degree. After a successful match, the system directly reads the final processing strategy and its execution result recorded in the high similarity historical cases. If the matched case shows that a certain strategy has successfully avoided collisions multiple times, the system will prioritize locking that strategy as the execution instruction for this time. Conversely, for strategies that have been marked as failures or insufficient responses in the past, the system will add them to the avoidance list.

[0035] Ultimately, the system integrates the matched historical best solutions with the preset emergency plans to generate precise protection instructions for the current scenario, which are then automatically sent to the wind turbine control system for execution. This process ensures that every hazard response is based on historical experience, guaranteeing the reliability of decisions while continuously optimizing the system's intelligent decision-making capabilities through the accumulation of case data.

[0036] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0037] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image recognition based online safety monitoring system for wind turbine blades, characterized in that, Comprise: Data acquisition and preprocessing module: through the deployment of camera on the fan cabin and tower, continuously collect video and image data covering the fan blade sweeping area, and pre-process the image or video; Intelligent perception and target identification module: collect pre-processed images or videos, label image or video data under different environmental conditions, and identify different obstacle types; Early warning and decision-making module: real-time tracking of identified obstacles, obtaining their motion trajectories, evaluating the possibility of obstacle collision with blades and the collision safety risk level, giving different warning signals for different collision safety risk levels; Closed loop and model optimization module: divide into three risk intervals according to the collision possibility, collect cases in the fuzzy interval and failed evaluation cases to build a difficult case library, correct the cases in the difficult case library, and output the corrected cases to train the model; Deployment and operation module: when the collision safety risk level is higher than the threshold, match the historical processing results, and execute the protection action based on the matched historical processing results.

2. The image recognition based online safety monitoring system for wind turbine blades as claimed in claim 1, wherein: The preprocessing includes image enhancement stage, image correction stage and static background reduction stage; the input image or video stream enters the image enhancement stage, the overall contrast under bad weather is improved by using limited contrast adaptive histogram equalization, and the brightness of overexposed or insufficient light area is adjusted by gamma correction; the enhanced image or video enters the image correction stage, the image deformation edge is corrected based on the iterative optimization method of calibration board; the clear image after enhancement and correction is modeled and reduced for static background.

3. The image recognition based online safety monitoring system for wind turbine blades as claimed in claim 2, wherein: The intelligent perception and target identification module labels the tag composition as time + identification device serial number + weather condition + obstacle type.

4. The image recognition based online safety monitoring system for fan blades as claimed in claim 1 wherein: The motion trajectory is expressed by a state vector [x, y, z, w, h, d, v x , v y , v z , v w , v h , v d ] T , where x is the horizontal coordinate of the center of the obstacle, y is the vertical coordinate of the center of the obstacle, z is the depth coordinate of the center of the obstacle, w is the width of the obstacle, h is the height of the obstacle, d is the depth of the obstacle, v x is the velocity of the center of the obstacle in the X direction, v y is the velocity of the center of the obstacle in the Y direction, v z is the velocity of the obstacle in the Z direction, v w is the rate of change of the width of the obstacle, v h is the rate of change of the height of the obstacle, and v d is the rate of change of the depth dimension d of the obstacle.

5. The image recognition based online safety monitoring system for fan blades as claimed in claim 1 wherein: The collision possibility evaluation calculation formula is Wherein, D min is the calculated future closest distance, which is calculated based on the obstacle movement speed, direction and fan blade rotation speed; r o + r t is the sum of the equivalent safety radius of the obstacle and the blade; V o is the magnitude of the movement speed of the obstacle, the faster the speed, the shorter the reaction time, the higher the risk, and α and β are adjustable parameters calculated according to a large amount of historical sample data, α is the weight coefficient of the speed influence, and β is a smoothing constant to prevent the denominator from being 0 when the speed is 0.

6. The image recognition based online safety monitoring system for wind turbine blades as claimed in claim 1, wherein: The collision safety risk level evaluation formula is R=P c X S, wherein P is obtained by evaluating the collision possibility, and S is the collision consequence severity, which is determined by the obstacle type and the collision energy.

7. The image recognition based online safety monitoring system for wind turbine blades as claimed in claim 6, wherein: The collision consequence severity S is calculated by the formula S = W t ×(1+K f ), where W t is a collision damage coefficient and K f is a collision kinetic energy coefficient.

8. The image recognition based online safety monitoring system for wind turbine blades as claimed in claim 1, wherein: The risk interval is based on the collision possibility P c The value size is divided into a safe interval, a fuzzy interval and a dangerous interval, P c ≧0.6 is a dangerous interval, 0.3 c <0.6 is a fuzzy interval, P c ≦0.3 is a safe interval.

9. The image recognition based online safety monitoring system for fan blades as claimed in claim 1, wherein: The closed loop and model optimization module determines the collision safety risk level of the obstacle based on the collision safety risk level evaluation value, when R≧6.0, it is judged as an extremely high risk obstacle; When 3.0<R<6.0, it is judged as a high risk obstacle; When 1.5<R≦3.0, it is judged as a medium risk obstacle; and when R<1.5, it is judged as a low risk obstacle.

Images