Mixed tower drum monitoring method and device based on combination of laser vision and YOLOV5 model, electronic equipment, storage medium and program

Abstract

The invention provides a mixed tower drum monitoring method and device based on combination of laser vision and a YOLOV5 model, electronic equipment, a storage medium and a program, and relates to the technical field of mixed tower drum swing monitoring. S10, an original image is acquired, the original image is preprocessed, and a preprocessed ROI image is output; s20, inputting the preprocessed ROI image into a pre-trained YOLOv5 model, and outputting a cross light spot dynamic parameter sequence; s30, outputting a synchronous six-dimensional time sequence data matrix according to the cross light spot dynamic parameter sequence; and S40, outputting a 28-dimensional normalized feature vector according to the synchronized 6-dimensional time sequence data matrix. Through the cooperation of the above structures, the device has the following beneficial effects: 1, through a non-contact measurement mode of laser projection and image acquisition, the damage of a traditional sensor to a tower drum structure is avoided; and secondly, top laser projection can be combined with an AI model to identify millimeter-level displacement change, so that a traditional monitoring blind area is made up, and the instability risk of the top structure is warned in advance.

Description

Technical Field

[0001] The present invention relates to the technical field of hybrid tower tube swing monitoring, and in particular to a hybrid tower tube monitoring method, device, electronic equipment, storage medium and program based on the combination of laser vision and YOLOV5 model. Background Art

[0002] In the field of wind power generation, hybrid towers, as the key structure supporting wind turbines, are crucial for monitoring their dynamic response under complex wind loads. Existing monitoring technologies primarily rely on traditional sensors such as strain gauges, inclinometers, and GPS positioning devices. Strain gauges monitor tower stress changes, inclinometers detect the tower's tilt angle, and GPS captures displacement data at specific locations on the tower. These sensors are typically concentrated in a limited area in the lower middle portion of the tower, providing single-point or localized monitoring of the tower's condition.

[0003] Hybrid towers sway under the influence of wind. If this sway exceeds a certain range, it may affect the structural safety and service life of the tower. In short, existing wind turbine hybrid tower monitoring technology has the following shortcomings: First, dynamic response monitoring blind spots: Traditional sensors (such as strain gauges and inclinometers) are concentrated in the middle and lower part of the tower. This means that traditional inclination sensors or GPS can only detect single-directional displacement and cannot simultaneously capture the combined motion characteristics of the tower's horizontal sway and axial torsion. It is also difficult to capture transient impacts under extreme working conditions (such as typhoons and blade emergency braking). Second, real-time performance and data loss: Existing systems only provide displacement data. Single-point monitoring cannot capture the tower's three-dimensional torsional motion and lacks intelligent identification and early warning of sway modes (such as vortex-induced vibration and galloping). Third, data isolation and weak analysis capabilities: Multi-dimensional data (vibration, inclination, stress) lacks coordinated analysis. Traditional manual inspections rely on experience and lack real-time dynamic analysis capabilities. Therefore, how to accurately and real-time monitor the sway and torsion status of hybrid towers has become a technical problem that needs to be solved. Summary of the Invention

[0004] In view of the shortcomings of the above-mentioned prior art, the technical problem to be solved by the present invention is to provide a mixed tower tube monitoring method, device, electronic equipment, storage medium and program based on the combination of laser vision and YOLOV5 model, which can realize the composite motion monitoring of mixed tower swing and torsion in a low-cost and high-reliability manner.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: the present invention provides a mixed tower monitoring method based on the combination of laser vision and YOLOV5 model, comprising the following steps:

[0006] S10, acquiring an original image, performing preprocessing on the original image, and outputting a preprocessed ROI image;

[0007] S20, input the pre-processed ROI image into the pre-trained YOLOv5 model, and output the cross spot dynamic parameter sequence;

[0008] S30, outputting a synchronized 6-dimensional time series data matrix according to the cross spot dynamic parameter sequence;

[0009] S40, outputting a 28-dimensional normalized feature vector according to the synchronized 6-dimensional time series data matrix;

[0010] S50, build a 1D-CNN / LSTM hybrid model, input the 28-dimensional feature vector into the hybrid model, and output the probability distribution of 7 types of vibration modes;

[0011] S60: Output a graded warning signal according to the probability of the seven vibration modes.

[0012] In the preferred solution, the specific steps of step S10 are: grayscale conversion and Gaussian filtering are performed on the acquired original image, a 300×300 pixel ROI area is dynamically intercepted based on the spot position of the previous frame, the ambient light brightness L is obtained through the ambient light sensor, and the gamma value is dynamically adjusted according to the ambient light brightness L, and a chroma separation filter, non-local mean noise reduction and morphological processing are combined to output the pre-processed ROI image;

[0013] If the spot position moves out of the current ROI area, a full-frame scan is started and the ROI center coordinates are adjusted based on the YOLOv5 detection results. The 1:1 mapping between physical coordinates and pixel coordinates is determined through the preset target grid.

[0014] When L>50klux, chroma separation filtering is used to suppress high-frequency noise;

[0015] When L<10klux, non-local means noise reduction is used.

[0016] In the preferred solution, the specific steps of step S20 are as follows:

[0017] S21, input layer optimization: scale the preprocessed ROI image to 640×640 and retain the original resolution 300×300 ROI area of the target grid;

[0018] S22, Backbone Network Optimization: Replace three layers of 3×3 convolution with dilated convolution in CSPDarknet53, expand the receptive field to a 7×7 ROI area of 300×300 of the original resolution covering the target grid, and insert the ECA-Net channel attention module after the C3 layer;

[0019] S23, multi-task detection head: branch 1 outputs ΔX and ΔY, and branch 2 outputs θ.

[0020] In the preferred solution, the specific steps of step S30 are: linearly interpolate the accelerometer data to 200 Hz, synchronize the timestamp with the image data, verify the validity by calculating the correlation coefficient of the data before and after interpolation, and fill in 0 if the acceleration data is missing and mark it as an invalid dimension to generate a synchronized 6-dimensional time series matrix: [ΔX t ,ΔY t ,θ t ,a x (t),a y (t),a z (t)].

[0021] In the preferred solution, the specific steps of step S40 are as follows:

[0022] S41. Time domain features: Calculate the standard deviation, range, skewness, and PCA ellipticity of ΔX, ΔY, and θ. The formula is:

[0023]

[0024] Where λ1 and λ2 are the two eigenvalues of the covariance matrix, and λ1 ≥ λ2;

[0025] S42, frequency domain features: Perform 1024-point FFT on ΔX, ΔY, and θ to extract the main frequency amplitude ratio and frequency band energy proportion, and divide the frequency bands into 0.1-1Hz, 1-3Hz, and 3-5Hz;

[0026] Decompose the ΔX sequence into 3 layers to obtain 8 sub-bands, calculate the energy entropy, and take the entropy values of the 3 sub-bands with the largest energy

[0027] S43, nonlinear characteristics: calculate the ΔX sequence, iteratively calculate the maximum Lyapunov exponent, and determine chaotic vibration if the threshold is greater than 0;

[0028] Construct a recurrence matrix by calculating the recurrence rate and determinism;

[0029] S44, feature normalization: Calculate the mean and standard deviation of the 28-dimensional features, process them dimension by dimension, and output a normalized vector.

[0030] In the preferred solution, the specific steps of step S40 are: constructing a 1D-CNN / LSTM hybrid model, inputting a 20-dimensional feature vector into the 1D-CNN / LSTM hybrid model, with a convolution kernel size of [5, 3, 3] and a number of channels of [32, 64, 128], extracting local temporal patterns, ReLU activation, and a MaxPooling layer to reduce the dimension of the feature map from (1000, 32) to (250, 32), and then performing subsequent convolution and pooling operations to finally output a (250, 128) feature map;

[0031] The output 64-channel feature map is passed into LSTM after dimensionality reduction by MaxPooling;

[0032] The output of 1D-CNN is processed by 128 hidden units of Bi-LSTM to generate forward and reverse hidden states, which are concatenated into (250, 256) to capture long-term dependencies.

[0033] The concatenated 1D-CNN and Bi-LSTM outputs are (250,384), and the weights of each time step are calculated through a learnable attention matrix to generate a context vector and highlight key features.

[0034] The fully connected layer is mapped to 7 categories, Softmax is activated, FocalLoss handles category imbalance, and outputs a probability distribution of 7 categories of swing patterns;

[0035] Among them, the probability distribution of seven types of oscillation modes includes steady-state wind load, vortex-induced vibration, galloping, resonance, chaotic oscillation, sensor failure, and other abnormalities.

[0036] In a preferred embodiment, the present invention further provides a device for use in any of the above-mentioned methods for monitoring a mixed tower based on the combination of laser vision and the YOLOV5 model, comprising:

[0037] Laser positioning module, the laser positioning module includes a cross laser generator fixed to the top of the tower by a double-rope suspension system or rigid connector, which is used to project a cross light spot onto the target curtain at the bottom of the tower;

[0038] Dynamic target and image acquisition module, which includes a 3m×3m high-reflectivity flexible screen with a millimeter-level grid printed on the surface, an industrial camera, and an ambient light sensor. It is used to transmit the target image captured by the industrial camera in real time to the image pre-processing unit;

[0039] An image preprocessing unit integrates a ROI dynamic interception module, an adaptive noise reduction module, and a morphological processing module. The unit defines a 300×300 pixel ROI based on the spot position of the previous frame in step S10, dynamically selects a noise reduction algorithm according to the ambient light brightness L, and outputs a preprocessed ROI image.

[0040] The target detection and analysis module is equipped with an improved YOLOv5 model to detect the center offset (ΔX, ΔY) and cross angle θ of the light spot in real time, and output a 200Hz dynamic parameter sequence.

[0041] The multimodal data fusion module is used to linearly interpolate the 100Hz triaxial acceleration data to 200Hz, time-align it with the image modality data, and construct a 6-dimensional time series matrix;

[0042] The feature extraction and analysis unit is used to extract 28-dimensional time-frequency domain and nonlinear features from the 6-dimensional matrix of the time window, and output them to the hybrid AI prediction and warning module after Z-score normalization;

[0043] Hybrid AI prediction and warning module, which includes a 1D-CNN / LSTM hybrid model that outputs the probability distribution of seven types of vibration patterns through convolutional layers, Bi-LSTM, and spatiotemporal attention mechanisms to trigger hierarchical warning logic;

[0044] Feedback control execution unit, the feedback control execution unit is used to receive early warning signals, update early warning thresholds and store abnormal data for model online learning.

[0045] In a preferred embodiment, the present invention also provides a computer device / equipment / system, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that the processor executes the computer program to implement any of the steps of the above-mentioned mixed tower monitoring method based on the combination of laser vision and YOLOV5 model.

[0046] In a preferred embodiment, the present invention further provides a computer non-transitory readable storage medium on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed by a processor, the steps of the mixed tower monitoring method based on the combination of laser vision and YOLOV5 model as described above are implemented.

[0047] In a preferred embodiment, the present invention further provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by one or more processors, the steps of the mixed tower monitoring method based on the combination of laser vision and YOLOV5 model are implemented as described in any of the above items.

[0048] The present invention provides a method, device, electronic device, storage medium, and program for monitoring a mixed tower based on a combination of laser vision and the YOLOV5 model. The coordination of the above structures has the following beneficial effects:

[0049] First, the non-contact measurement method of laser projection and image acquisition avoids damage to the tower structure caused by traditional sensors, reduces maintenance costs, and supports the rapid installation and modification of existing hybrid towers.

[0050] Second, top laser projection can be combined with AI models to identify millimeter-level displacement changes, filling in the blind spots of traditional monitoring and providing early warning of the risk of top structural instability. It also integrates multimodal data such as accelerometers and ambient light sensors, and reduces the false alarm rate through time domain, frequency domain and nonlinear feature extraction. It can accurately distinguish seven types of vibration modes such as vortex-induced vibration, galloping, and resonance and their root causes, and can accurately and in real time monitor the swing and torsional state of the hybrid tower barrel and resist environmental interference.

[0051] Third, the 1D-CNN / LSTM hybrid model and attention mechanism are used, combined with a hierarchical warning strategy to achieve rapid response. Level 1 warnings automatically reduce wind turbine power, and Level 2 triggers an emergency shutdown and activates the hydraulic damper, significantly improving structural safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below with reference to the accompanying drawings and examples:

[0053] Figure 1 It is a timing structure diagram of the present invention;

[0054] Figure 2 is a schematic diagram of the image capture and preprocessing process of the present invention;

[0055] Figure 3 is a flow chart of step S20 and step S30 of the present invention;

[0056] Figure 4 is a flow chart of step S40 of the present invention;

[0057] Figure 5 It is a schematic diagram of the structure of the computer device / equipment / system of the present invention. DETAILED DESCRIPTION

[0058] In order to better understand the purpose, system architecture and functional implementation of the present embodiment, the embodiments in the present application and the features in the embodiments may be combined with each other without conflict. The exemplary embodiments disclosed in the present embodiment will be described below in conjunction with the accompanying drawings, which contain specific technical details disclosed in the present embodiment to assist understanding, but these details should be regarded as exemplary rather than restrictive. Therefore, it should be understood by those skilled in the art that various improvements and adjustments may be made to the embodiments described herein without departing from the scope and core ideas of the present invention. Similarly, for clarity of expression, a detailed description of well-known technologies, functions and structures (such as standard image processing algorithms, general communication protocols) is omitted in the following description.

[0059] In the field of engineering monitoring, artificial intelligence models are widely used in intelligent analysis and decision support of multimodal data. Through technologies such as deep learning, the model can extract common feature representations from multi-source data, significantly reducing dependence on labeled data and improving prediction accuracy. With the development of technology, artificial intelligence models have expanded from single modalities (such as text or image analysis) to multimodal fusion scenarios to meet the challenges of complex engineering tasks. For example, multimodal artificial intelligence models can integrate multi-source information such as laser vision and accelerometer data to achieve joint modeling in a unified feature space, which is generally better than traditional single-modal methods. However, in scenarios where reasoning requires combining external knowledge (such as environmental parameters, historical vibration patterns), existing technologies have significant limitations:

[0060] The modal processing dimension is limited, that is, traditional methods only perform knowledge injection on a single modality (such as text or pure image data) and fail to fully integrate multi-physics field information;

[0061] The knowledge type is single, that is, external knowledge is usually only introduced in the form of text fragments, which makes it difficult to capture dynamic temporal characteristics and nonlinear pattern associations.

[0062] It's important to note that existing technologies for monitoring mixed-use towers rely on single-sensor data (such as accelerometers) or static text rule bases, making it difficult to simultaneously analyze implicit knowledge (such as the chaotic characteristics of vortex-induced vibration) in multimodal data such as laser spot displacement and vibration spectrum characteristics. This limitation makes it difficult for models to accurately identify and dynamically warn of complex motions (swing + torsion).

[0063] The core improvement of this embodiment is to align the time and space of laser vision and physical sensor data to construct a 6-dimensional time series feature matrix containing displacement, angle, and acceleration, providing multi-physics field correlation information for the model;

[0064] Secondly, a spatiotemporal attention module is introduced into hybrid AI models (such as 1D-CNN / LSTM) to dynamically integrate environmental parameters (wind speed, temperature) with historical vibration patterns, improving the ability to recognize complex patterns such as chaotic oscillations and resonances.

[0065] Finally, the model parameters are continuously optimized through the online learning module to avoid the obsolescence problem of traditional static knowledge base.

[0066] Example 1

[0067] like Figures 1 to 5 As shown in the figure, a mixed tower monitoring method based on the combination of laser vision and YOLOV5 model includes the following steps:

[0068] Step 10: Image capture and preprocessing: Obtain the original image using a high-speed industrial camera, perform preprocessing on the original image, and output the preprocessed ROI image;

[0069] Input: original camera image (resolution 1280×1024 and frame rate 200fps)

[0070] Output: Preprocessed ROI image

[0071] The specific steps are:

[0072] Step 11: Dynamic ROI capture:

[0073] Sliding window mechanism: A 300×300 pixel area is defined based on the spot position of the previous frame (the initial position is preset by the target grid). If the spot moves out of the ROI, a full-frame scan is initiated (the window is adjusted based on the real-time target detection results of YOLOv5).

[0074] Coordinate mapping: Bind the ROI physical coordinates to the image pixel coordinates using a preset target grid (1mm / pixel).

[0075] Step 12: Adaptive noise reduction: Use the ambient light sensor to transmit the ambient light brightness L to the FPGA via the SPI bus to dynamically adjust the image gamma value. The specific formula for the gamma value is as follows:

[0076] gamma = 1.0-0.02 × L;

[0077] Among them, when L>50klux, gamma≤0.5; when L<10klux, gamma≥0.8, which is achieved through a fixed-point arithmetic unit.

[0078] Process the ROI image pixel by pixel. Output to the noise reduction unit;

[0079] When L > 50 klux, which is a strong light environment, the RGB channels are separated by a hardware splitter, and a 5 × 5 Gaussian low-pass filter is applied to each channel (the hardware multiplier implements the convolution operation) to suppress high-frequency noise, and then the channels are merged.

[0080] L < 10klux, that is, a low-light environment: the search window is 15×15, the similarity block is 7×7, the sum of squares of pixel differences is calculated through a parallel comparator, and the weight accumulator generates the denoised pixel value, preserving the edge details of the light spot.

[0081] Perform an opening operation (3×3 elliptical kernel) on the spot area to eliminate small noise points and ensure the integrity of the cross spot structure.

[0082] Step 20: Light spot positioning and angle detection based on YOLOv5: Input the preprocessed ROI image into the pre-trained YOLOv5 model and output the cross light spot dynamic parameter sequence;

[0083] Input: Preprocessed ROI image.

[0084] Algorithm: Spot center offset coordinates (ΔX, ΔY), crosshair angle θ

[0085] The specific steps are as follows:

[0086] Step 21. Input layer optimization: Load the pre-trained YOLOv5 model into the GPU memory and scale the pre-processed ROI image to 640×640. The output layer contains the spot center coordinates (xyxy format) and the cross angle θ (radian value). Input the pre-processed ROI image into the pre-trained YOLOv5 model, retaining the original resolution of the target grid area (300×300 ROI) to avoid loss of small target (cross spot) information.

[0087] Mosaic data augmentation is used: During the training phase, four target images under different lighting conditions (L = 5klux, 30klux, 80klux, and 15klux) are fused, cropped, and spliced into 640×640 training samples to enhance the model's robustness to strong / weak light environments.

[0088] Step 22: Backbone network optimization:

[0089] In the backbone network CSPDarknet53, the three-layer 3×3 convolution is replaced by the dilated convolution (dilation=2), and the receptive field is expanded to 7×7, covering the 300×300 ROI global features; the ECA-Net channel attention module dynamically weights the feature channels after the C3 layer to enhance the response of the spot area.

[0090] The ECA-Net channel attention module is introduced to dynamically weight the feature channels after the C3 layer to enhance the feature response of the spot area.

[0091] Step 23: Multi-task detection head:

[0092] Branch 1: Spot center detection, using CIoU Loss, confidence threshold 0.9, output center point coordinates (x center ,y center ), converted to an offset ΔX=x relative to the target origin center -2000, ΔY=y center -2000 (unit: pixel, 1 pixel = 1 mm).

[0093] Branch 2: Cross angle regression, Smooth L1 Loss combined with cosine similarity constraint, output θ (radian angle, accuracy ±2°).

[0094] Here, θ∈[80°,100°] is forced.

[0095] Step 30, multimodal data fusion: input the cross spot dynamic parameter sequence and output a synchronized 6-dimensional time series data matrix;

[0096] Input: Image modality: ΔX, ΔY, θ (200Hz).

[0097] Physical mode: accelerometer data a x 、a y 、a z (100Hz, optional).

[0098] Output: synchronized 6-dimensional time series data matrix (200 Hz × 6 dimensions).

[0099] The specific steps are as follows:

[0100] Step 31, time alignment: The camera and accelerometer are synchronized through the hardware timer. The timestamp is generated by the STM32 internal RTC module (accuracy 1μs) to ensure that the image and acceleration data are time aligned.

[0101] Linear interpolation implementation: Acceleration data (100Hz) is stored in the circular buffer acc_buffer with timestamps [t0, t0+10ms, t0+20ms, ...].

[0102] For the 200Hz time point t0+5ms of the image data, the calculation formula is as follows:

[0103] a x (t0+5ms)=a x (t0)+(a x (t0+10ms)-a x (t0))×0.5;

[0104] The results are stored.

[0105] The hardware multiplier implements floating-point operations, and the interpolated data is transferred to the fusion unit via DMA.

[0106] The STM32 floating-point unit (FPU) is used to calculate the Pearson correlation coefficient of the acceleration series before and after interpolation. If it is less than 0.98, cubic spline interpolation is triggered, which is implemented through an external DSP coprocessor to improve accuracy.

[0107] Step 32, exception handling: If the accelerometer SPI communication error (error flag SPI_ERR = 1), the fusion module fills [0,0,0] to the corresponding dimension and sets the missing flag to 1. This flag is output to the feature extraction unit along with the 6-dimensional matrix to mask invalid data.

[0108] Step 33, matrix construction: the data format for each 200Hz time point is [ΔX t ,ΔY t ,θ t ,a x (t),a y (t),a z (t)], stored in the STM32 chip with a matrix size of 1000 × 6 (corresponding to a 5s time window), and output to the DSP through the UART serial port (baud rate 115200).

[0109] Step 40: Time-frequency domain and nonlinear feature extraction:

[0110] Input: synchronized 6-dimensional time series data matrix (time window T = 5s, 1000 sampling points).

[0111] Output: 28-dimensional normalized feature vector.

[0112] The specific steps of feature calculation are as follows:

[0113] Step 41, time domain features (10 dimensions): For the ΔX sequence, the standard deviation, range, and skewness are calculated using the DSP multiplier-accumulator, and the results are stored in the time_features array.

[0114] Perform PCA (hardware covariance matrix calculation unit) on (ΔX, ΔY) to extract the eigenvalues λ1 and λ2. The ellipticity formula is:

[0115]

[0116] Among them, λ1 and λ2 are the two eigenvalues of the covariance matrix, and λ1 ≥ λ2, reflecting the directionality of the swing trajectory (0 is a circle, 1 is a straight line).

[0117] Step 42, frequency domain features (12 dimensions): perform a 1024-point FFT on the θ sequence, use the hardware spectrum analysis unit to identify the main frequency, and calculate the amplitude ratio (DC is the f=0 component);

[0118] The frequency bands are divided into 0.1-1Hz, 1-3Hz, and 3-5Hz, and the energy proportion of each band is accumulated.

[0119] Decompose the ΔX sequence into 3 layers to obtain 8 sub-bands. Calculate the energy entropy and take the entropy values of the 3 sub-bands with the largest energy.

[0120] Step 43, nonlinear characteristics (6 dimensions): Wolf algorithm hardware implementation module, input ΔX sequence, iteratively calculate the maximum Lyapunov exponent, and determine chaotic vibration if the threshold is >0.

[0121] A recursion matrix (threshold = 0.5 × ΔX standard deviation) is constructed, and the recursion rate (RR, the proportion of recursion points) and the determinism (DET, the proportion of consecutive recursion points) are calculated using a hardware comparator.

[0122] Step 44, feature normalization: Calculate the mean and standard deviation of the 28-dimensional features (implemented by moving average filter), process them dimension by dimension, and output a normalized vector (data type: float32, range [-3, 3]).

[0123] Step 50: Hybrid model classification prediction and vibration pattern recognition: Load the 1D-CNN / LSTM hybrid model into the edge computing device CPU;

[0124] Input: 28-dimensional feature vector (time series T=5s).

[0125] Output: Probability distribution of 7 types of vibration modes.

[0126] The specific steps are as follows:

[0127] Step 51, model architecture:

[0128] 1D-CNN / LSTM time series modeling:

[0129] 1D-CNN:

[0130] Input: 20-dimensional feature vector, time step T = 1000 (corresponding to 5s data).

[0131] Convolution kernel: 3 layers of convolution (kernel=5,3,3), channel number [32,64,128], extract local temporal patterns, ReLU activation, MaxPooling layer to reduce the dimension of feature map from (1000,32) to (250,32), and then through subsequent convolution and pooling operations, the final output is (250,128) feature map

[0132] Output: 64-channel feature map, which is passed to LSTM after dimensionality reduction by MaxPooling.

[0133] Bi-LSTM: 128 hidden units, processes 1D-CNN output, generates forward and reverse hidden states, concatenated into (250, 256), and captures long-term dependencies.

[0134] Attention fusion: The output of 1D-CNN and Bi-LSTM is concatenated into (250, 384), and the weight of each time step is calculated through the learnable attention matrix to generate a context vector (384,) to highlight key features.

[0135] Step 52: Classification output:

[0136] Output layer: The fully connected layer is mapped to 7 categories, with Softmax activation and Focal Loss (α=0.25,γ=2) to handle category imbalance, and outputs a probability distribution of 7 categories of swing patterns.

[0137] The probability distribution of the seven types of oscillation modes includes steady-state wind load, vortex-induced vibration, galloping, resonance, chaotic oscillation, sensor failure, and other anomalies.

[0138] Step 53: Probability calibration:

[0139] Temperature scaling optimizes the model output confidence. We set the temperature parameter T = 0.5, scale the logits, and output the calibrated probability after Softmax to reduce overfitting bias.

[0140] Step 60: Gradual warning and feedback control

[0141] Input: Probability of 7 types of vibration modes.

[0142] Output: graded warning signal (Level 0 / 1 / 2).

[0143] The specific steps are as follows:

[0144] Step 61: Determine the probability threshold:

[0145] Level 0 is normal working condition: all categories have a probability < 0.7, no action, and the status indicator is always green.

[0146] Level 1 is the attention level: the probability of a single category is ≥ 0.7 and lasts for 5 seconds. A command is sent to the fan controller via the Modbus protocol (function code 0x06, register address 40001, value 90), the power is reduced to 90%, and the status indicator flashes yellow.

[0147] Level 2 is the dangerous level: probability ≥ 0.9 or Lyapunov index > 0 and 1-3Hz energy ratio > 0.6. The hardware interrupt triggers the relay to disconnect the main circuit. At the same time, the PWM module outputs a high-level signal to start the hydraulic damper. The status indicator is solid red and triggers the local sound and light alarm.

[0148] Step 62: Adaptive Optimization

[0149] Dynamic threshold adjustment: At 0:00 every day, the microcontroller calculates the mean μ and standard deviation σ of the past 24 hours, writes them to the edge computing device register through the serial port, and updates the probability threshold based on the sliding window (24 hours):

[0150] P threshold =μ prob +2.5σ prob ;

[0151] Among them, μ prob is the average probability, 2.5σ prob is the standard deviation

[0152] Online learning: When an abnormal event occurs (Level 2 trigger), the original images, sensor data, and feature vectors for the previous and subsequent 10 seconds are stored in a local SSD and transmitted to the cloud via Ethernet. This triggers the EWC algorithm to fine-tune the model and lock key weights (such as the convolution kernels of the first three layers of the backbone network).

[0153] Example 2

[0154] The following describes a device for a mixed tower monitoring method based on a combination of laser vision and a YOLOV5 model provided by the present invention. The device for a mixed tower monitoring method based on a combination of laser vision and a YOLOV5 model described below and the mixed tower monitoring method based on a combination of laser vision and a YOLOV5 model described above can be referred to each other and further described in conjunction with Example 1.

[0155] The apparatus for monitoring a mixed-use tower using laser vision combined with the YOLOV5 model, provided in this application embodiment, comprises a laser positioning module, a dynamic target and image acquisition module, an image preprocessing unit, a multimodal data fusion module, a feature extraction and analysis unit, an AI analysis and warning module, and a feedback control execution unit. These units work together to accurately monitor the vibration status of the mixed-use tower and provide timely warnings.

[0156] In this embodiment, the laser positioning module includes the following:

[0157] Suspension unit: The suspension unit is fixed to the steel beam at the top of the tower using a rigid connector or a double-rope suspension system to prevent deflection or torsion caused by external environmental influences.

[0158] Rigid Connector: A high-strength aluminum alloy bracket secures the laser generator to the tower's mounting platform with bolts. The bracket is designed with adjustable angle and height to accommodate installation requirements on different tower models. Shock-absorbing rubber pads are installed at the bracket-to-tower connection to reduce the impact of tower vibration on the laser generator.

[0159] The dual-rope suspension system utilizes two high-strength, low-elongation stainless steel wire ropes with a diameter of 6mm. One end of the wire rope is connected to two symmetrical suspension points on the top of the laser generator via a custom hook, while the other end is secured to a steel beam at the top of the tower via a tension adjustment device. This tension adjustment device allows for fine-tuning of the wire rope tension based on actual conditions, ensuring the laser generator remains stable while suspended and preventing measurement deviations caused by twisting of a single rope.

[0160] Laser emitting unit: adopts cross laser generator, fixed with rigid connector or double rope suspension system to avoid single rope twisting.

[0161] Parameters: Laser wavelength 532nm (green light), power ≤5mW, spot diameter ≤2mm, ensuring clear imaging within a height range of 200 meters.

[0162] In this embodiment, the dynamic target and image acquisition module includes the following:

[0163] Target screen: A high-reflectivity flexible screen (size 3m×3m) is fixed at the bottom of the tower as the projection surface of the laser light spot. It is made of special reflective material to enhance the reflection effect of the laser spot. The surface is required to be flat and have moderate reflectivity. The surface is printed with millimeter-level grid scale to assist in the positioning and calibration of the center of the light spot, enhance the positioning accuracy of the light spot, and facilitate the camera to capture the light spot position.

[0164] Camera unit: A high-speed industrial camera is installed directly below the target screen. The high-speed industrial camera is a Sony IMX535 global shutter CMOS (frame rate ≥ 100 fps, resolution ≥ 4000 × 3000 pixels);

[0165] The camera is equipped with an optical lens, which uses a large aperture, low distortion wide-angle lens. The focal length of the lens is 12mm and the field of view angle reaches 120°. By covering the entire area of the screen through the wide-angle lens, it can accurately capture the position changes of the laser light spot on the screen.

[0166] Ambient Light Sensor: An ambient light sensor is installed near the camera to monitor ambient light intensity (L) in real time. The sensor uses a digital output with a measurement range of 0-100 klux and an accuracy of ±5%. Ambient light data is transmitted via the SPI bus to the control module of the image acquisition and preprocessing unit for parameter adjustment of the adaptive noise reduction algorithm.

[0167] In this embodiment, the image pre-processing unit includes the following:

[0168] A field-programmable gate array (FPGA) is used as the core hardware for image preprocessing. The FPGA integrates algorithm modules such as ROI dynamic interception, adaptive noise reduction, and morphological processing. Through hardware acceleration, the original image captured by the camera can be quickly preprocessed. The FPGA and the camera are connected via a high-speed CameraLink interface, with a data transmission rate of up to 10Gbps, ensuring real-time transmission and processing of image data. The processed ROI image is output to the YOLOv5-based spot positioning and analysis device through a high-speed serial port.

[0169] In this embodiment, the target detection and analysis module includes the following:

[0170] Edge computing devices: Select an ARM-based edge computing device with a high-performance CPU and GPU capable of running the improved YOLOv5. The edge computing device is equipped with 8GB of RAM and a 128GB solid-state drive to store model parameters and temporary data. The device supports multiple network connections, including Ethernet and Wi-Fi, for data communication with a host computer or the cloud.

[0171] Improved YOLOv5 Model Hardware Acceleration: The improved YOLOv5 model is hardware accelerated on the GPU of the edge computing device. By optimizing the model's computational graph, some computational tasks are distributed to the GPU's parallel computing cores, significantly improving the model's inference speed. Furthermore, the GPU's memory management mechanism is utilized to efficiently store and access model parameters and intermediate calculation results, reducing data transmission overhead. The detected dynamic parameter sequence, such as the crosshair center offset coordinates (ΔX, ΔY) and the crosshair angle θ, is output to the multimodal data fusion module via a high-speed serial port.

[0172] In this embodiment, the multimodal data fusion module includes the following:

[0173] Includes accelerometer and data synchronization and fusion unit.

[0174] Accelerometer: A three-axis MEMS accelerometer is used with a measurement range of ±16g, a resolution of 16 bits, and a sampling frequency of 100Hz. The accelerometer is fixed to the rigid part of the tower bottom with epoxy resin glue or screws to collect the vibration acceleration data of the tower. x 、a y 、a z The accelerometer adopts a low-power design with an operating current of less than 1mA to reduce the overall power consumption of the system.

[0175] Data synchronization and fusion unit: This module is built on the STM32 microprocessor and is responsible for the image modality data (ΔX, ΔY, θ) and physical modality data (a x 、a y 、a z ) for time alignment and feature-level fusion. Through hardware timers and software algorithms, the acceleration data is linearly interpolated to increase its sampling frequency to 200Hz, consistent with the frequency of the image data. Then, the multimodal data at each time point is constructed into a 6-dimensional time series data matrix [ΔX t ,ΔY t ,θ t ,a x (t),a y (t),a z(t)]. This module has a data cache function, which can store multimodal data within a certain period of time for subsequent feature extraction and analysis, and output it to the feature extraction and analysis unit through the high-speed serial port.

[0176] In this embodiment, the feature extraction and analysis unit includes the following:

[0177] Digital Signal Processor (DSP): A high-performance DSP chip is used as the core computing device for feature extraction. The DSP chip possesses powerful digital signal processing capabilities and can quickly complete time-domain, frequency-domain, and nonlinear feature extraction operations. The DSP integrates multiple hardware multipliers and adders, enabling parallel processing of large amounts of data and improving computational efficiency.

[0178] Feature extraction algorithm library: A library of time-domain, frequency-domain, and nonlinear feature extraction algorithms is pre-programmed into the DSP. Time-domain feature extraction includes calculating statistics such as the standard deviation, range, and skewness of ΔX, ΔY, and θ, as well as calculating the displacement covariance matrix and extracting ellipticity using PCA.

[0179] Frequency domain feature extraction uses the FFT algorithm to perform 1024-point fast Fourier transform on ΔX, ΔY, and θ to extract features such as the main frequency amplitude ratio and the frequency band energy proportion. At the same time, the DB4 wavelet is used to perform three-layer wavelet packet decomposition and calculate the energy entropy of each sub-band.

[0180] The Wolf algorithm is used to calculate the maximum Lyapunov exponent of the ΔX sequence for nonlinear feature extraction, and the recursion rate, certainty and other indicators are calculated through recursive quantification analysis (RQA).

[0181] Feature normalization module: A hardware circuit is used to implement the Z-score normalization algorithm to normalize the extracted 28-dimensional feature vectors, eliminating the dimensionality effect and ensuring that different features have the same weight during model training and inference. The normalized feature vectors are transmitted to the hybrid AI prediction and warning module via a high-speed serial port.

[0182] In this embodiment, the hybrid AI prediction and warning module includes the following:

[0183] Continue to deploy the 1D-CNN / LSTM hybrid model on edge computing devices.

[0184] 1D-CNN / LSTM hybrid model implementation: Run the 1D-CNN and Bi-LSTM models on the CPU of the edge computing device and perform feature fusion and collaborative inference with the YOLOv5 model.

[0185] 1D-CNN model: Input is a 20-dimensional feature vector (features with strong time series correlation are selected from the 28-dimensional vector) with a time step of T = 1000 (corresponding to 5 seconds of data). A three-layer convolution (kernel = 5, 3, 3) with channels of [32, 64, 128] is used to extract local temporal patterns. The output is a 64-channel feature map, which is then fed into the LSTM after dimensionality reduction using MaxPooling.

[0186] Bi-LSTM: Set 128 hidden units to capture long-term dependencies. Concatenate the 1D-CNN and Bi-LSTM outputs, and weight the key features through the spatiotemporal attention module to obtain the context vector.

[0187] The probability distribution of the seven types of vibration modes is output to the hierarchical early warning and feedback control device through the network interface.

[0188] In this embodiment, the feedback control execution unit includes the following:

[0189] Warning signal output interface: This interface, designed based on the RS485 bus standard, can output graded warning signals (Level 0 / 1 / 2) as digital signals to the wind turbine control system or other external devices. The interface has electrical isolation, effectively preventing external interference from affecting the system.

[0190] Wind turbine power regulation and damper control module: Upon receiving a Level 1 warning signal, the control module sends a command to the wind turbine control system, reducing wind turbine power to 90%. The control module utilizes industrial-grade relays and drivers to reliably control the wind turbine's power regulation. Upon receiving a Level 2 warning signal, the control module immediately triggers the hydraulic damper's activation signal and simultaneously sends an emergency shutdown command to the wind turbine control system. The hydraulic damper's control signal is precisely controlled using PWM (pulse-width modulation) technology, ensuring rapid damper response and effective suppression of tower vibration.

[0191] Threshold dynamic adjustment and online learning module: This module is built on a microcontroller and is responsible for updating the probability threshold based on the historical data within 24 hours of the sliding window. By calculating the mean μ of the probability data in real time prob and standard deviation σ prob , dynamically adjust the warning threshold to adapt to the monitoring needs under different working conditions.

[0192] At the same time, the module has online learning capabilities, which can store and transmit abnormal event data to edge computing devices, and use the elastic weight merging (EWC) algorithm to fine-tune the model to prevent catastrophic forgetting and improve the long-term monitoring performance of the model.

[0193] Example 3

[0194] Further illustrate with reference to Example 1, Figure 5 The structure shown, Figure 5 This is a schematic diagram of the structure of a computer device / equipment / system provided in an embodiment of the present application. The computer device / equipment / system includes:

[0195] A processor, memory, a communication bus, and a computer program stored in the memory and executable on the processor.

[0196] The processor can call the computer program in the memory, and when executing the program, implement the mixed tower monitoring method, device, electronic device, storage medium and program based on the combination of laser vision and YOLOV5 model provided in the above embodiments. The method includes: S10, obtaining the original image, performing preprocessing on the original image, and outputting the preprocessed ROI image; S20, inputting the preprocessed ROI image into the pre-trained YOLOv5 model, and outputting the cross spot dynamic parameter sequence; S30, outputting a synchronized 6-dimensional time series data matrix according to the cross spot dynamic parameter sequence; S40, outputting a 28-dimensional normalized feature vector according to the synchronized 6-dimensional time series data matrix; S50, constructing a 1D-CNN / LSTM hybrid model, inputting the 28-dimensional feature vector into the hybrid model, and outputting 7 types of vibration mode probability distribution; S60, outputting a graded warning signal according to the 7 types of vibration mode probabilities.

[0197] Furthermore, the computer device / equipment / system also includes:

[0198] Communications Interface: used for communication between memory and processor.

[0199] The memory may include a high-speed RAM memory and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0200] If the memory, processor, and communication interface are implemented independently, the communication interface, memory, and processor can be interconnected via a bus to facilitate communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, Figure X uses only a single thick line, but this does not imply a single bus or type of bus.

[0201] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0202] A processor may include one or more processing units. For example, a processor may include an application processor (AP), an application-specific integrated circuit (ASIC), a modem processor, a central processing unit (CPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may be a neural network center or command center. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a high-speed cache memory. This memory can store instructions or data that have just been used or are being recycled by the processor. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0204] A display device is used to display images, videos, etc. The display device may include a display panel, which may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), MiniLED, MicroLed, Micro-oLed, or a quantum dot light-emitting diode (QLED).

[0205] Optionally, in a specific implementation, if the memory, processor, and communication interface are integrated on a chip, the memory, processor, and communication interface can communicate with each other through an internal interface.

[0206] On the other hand, an embodiment of the present application also provides a computer non-transitory readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned mixed tower monitoring method based on the combination of laser vision and YOLOV5 model. The method includes: S10, obtaining the original image, performing preprocessing on the original image, and outputting the preprocessed ROI image; S20, inputting the preprocessed ROI image into the pre-trained YOLOv5 model, and outputting the cross spot dynamic parameter sequence; S30, outputting a synchronized 6-dimensional time series data matrix according to the cross spot dynamic parameter sequence; S40, outputting a 28-dimensional normalized feature vector according to the synchronized 6-dimensional time series data matrix; S50, constructing a 1D-CNN / LSTM hybrid model, inputting the 28-dimensional feature vector into the hybrid model, and outputting 7 types of vibration mode probability distributions; S60, outputting a graded warning signal according to the 7 types of vibration mode probabilities.

[0207] On the other hand, an embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. The computer program can run computer instructions. When the computer program is executed by a processor, the computer can execute the mixed tower monitoring method based on the combination of laser vision and YOLOV5 model provided by the above methods. The method includes: S10, obtaining the original image, performing preprocessing on the original image, and outputting the preprocessed ROI image; S20, inputting the preprocessed ROI image into the pre-trained YOLOv5 model, and outputting a cross spot dynamic parameter sequence; S30, outputting a synchronized 6-dimensional time series data matrix according to the cross spot dynamic parameter sequence; S40, outputting a 28-dimensional normalized feature vector according to the synchronized 6-dimensional time series data matrix; S50, constructing a 1D-CNN / LSTM hybrid model, inputting the 28-dimensional feature vector into the hybrid model, and outputting 7 types of vibration mode probability distributions; S60, outputting a graded warning signal according to the 7 types of vibration mode probabilities.

[0208] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0209] For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use with or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, the computer-readable medium can even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting, or otherwise processing it in a suitable manner as needed, and then storing it in a computer memory.

[0210] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0211] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0212] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0213] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0214] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0215] The above specific embodiments do not constitute a limitation on the scope of protection of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limitations of the present application. Those skilled in the art can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A mixed tower monitoring method based on the combination of laser vision and YOLOV5 model, characterized in that: The following steps are involved: S10, acquiring an original image, performing preprocessing on the original image, and outputting a preprocessed ROI image; S20, input the pre-processed ROI image into the pre-trained YOLOv5 model, and output the cross spot dynamic parameter sequence; S30, outputting a synchronized 6-dimensional time series data matrix according to the cross spot dynamic parameter sequence; S40, outputting a 28-dimensional normalized feature vector according to the synchronized 6-dimensional time series data matrix; S50, build a 1D-CNN / LSTM hybrid model, input the 28-dimensional feature vector into the hybrid model, and output the probability distribution of 7 types of vibration modes; S60: Output a graded warning signal according to the probability of the seven vibration modes.

2. The method for monitoring a mixed tower based on laser vision and YOLOV5 model according to claim 1 is characterized in that: The specific steps of step S10 are as follows: grayscale conversion and Gaussian filtering are performed on the acquired original image, a 300×300 pixel ROI area is dynamically intercepted based on the spot position of the previous frame, the ambient light brightness L is obtained through the ambient light sensor, and the gamma value is dynamically adjusted according to the ambient light brightness L. Combined with chroma separation filtering, non-local mean noise reduction and morphological processing, the pre-processed ROI image is output; If the spot position moves out of the current ROI area, a full-frame scan is started and the ROI center coordinates are adjusted based on the YOLOv5 detection results. The 1:1 mapping between physical coordinates and pixel coordinates is determined through the preset target grid. When L>50klux, chroma separation filtering is used to suppress high-frequency noise; When L<10klux, non-local means noise reduction is used.

3. The method for monitoring a mixed tower based on laser vision and YOLOV5 model according to claim 1 is characterized in that: The specific steps of step S20 are as follows: S21, input layer optimization: scale the preprocessed ROI image to 640×640 and retain the original resolution 300×300 ROI area of the target grid; S22, Backbone Network Optimization: Replace three layers of 3×3 convolution with dilated convolution in CSPDarknet53, expand the receptive field to a 7×7 ROI area of 300×300 of the original resolution covering the target grid, and insert the ECA-Net channel attention module after the C3 layer; S23, multi-task detection head: branch 1 outputs ΔX and ΔY, and branch 2 outputs θ.

4. The method for monitoring a mixed tower based on laser vision and YOLOV5 model according to claim 1 is characterized in that: The specific steps of step S30 are: linearly interpolate the accelerometer data to 200Hz, synchronize the timestamp with the image data, verify the validity by calculating the correlation coefficient of the data before and after interpolation, and fill in 0 if the acceleration data is missing and mark it as an invalid dimension to generate a synchronized 6-dimensional time series matrix: [ΔX t ,ΔY t ,θ t ,a x (t),a y (t),a z (t)].

5. The method for monitoring a mixed tower based on laser vision and YOLOV5 model according to claim 1 is characterized in that: The specific steps of step S40 are as follows: S41. Time domain features: Calculate the standard deviation, range, skewness, and PCA ellipticity of ΔX, ΔY, and θ. The formula is: Where λ1 and λ2 are the two eigenvalues of the covariance matrix, and λ1 ≥ λ2; S42, frequency domain features: Perform 1024-point FFT on ΔX, ΔY, and θ to extract the main frequency amplitude ratio and frequency band energy proportion, and divide the frequency bands into 0.1-1Hz, 1-3Hz, and 3-5Hz; Decompose the ΔX sequence into 3 layers to obtain 8 sub-bands, calculate the energy entropy, and take the entropy values of the 3 sub-bands with the largest energy S43, nonlinear characteristics: calculate the ΔX sequence, iteratively calculate the maximum Lyapunov exponent, and determine chaotic vibration if the threshold is greater than 0; Construct a recurrence matrix by calculating the recurrence rate and determinism; S44, feature normalization: Calculate the mean and standard deviation of the 28-dimensional features, process them dimension by dimension, and output a normalized vector.

6. The method for monitoring a mixed tower based on laser vision and YOLOV5 model according to claim 1 is characterized in that: The specific steps of step S40 are as follows: constructing a 1D-CNN / LSTM hybrid model, inputting a 20-dimensional feature vector into the 1D-CNN / LSTM hybrid model, using a convolution kernel size of [5, 3, 3] and a number of channels of [32, 64, 128], extracting local temporal patterns, performing ReLU activation, and using a MaxPooling layer to reduce the dimension of the feature map from (1000, 32) to (250, 32). After subsequent convolution and pooling operations, the feature map is finally outputted as (250, 128). The output 64-channel feature map is passed into LSTM after dimensionality reduction by MaxPooling; The output of 1D-CNN is processed by 128 hidden units of Bi-LSTM to generate forward and reverse hidden states, which are concatenated into (250, 256) to capture long-term dependencies. The concatenated 1D-CNN and Bi-LSTM outputs are (250,384), and the weights of each time step are calculated through a learnable attention matrix to generate a context vector and highlight key features. The fully connected layer is mapped to 7 categories, Softmax is activated, FocalLoss handles category imbalance, and outputs a probability distribution of 7 categories of swing patterns; Among them, the probability distribution of seven types of oscillation modes includes steady-state wind load, vortex-induced vibration, galloping, resonance, chaotic oscillation, sensor failure, and other abnormalities.

7. A device for use in the method for monitoring a mixed tower based on laser vision combined with a YOLOV5 model as described in any one of claims 1 to 6, characterized in that: include: Laser positioning module, the laser positioning module includes a cross laser generator fixed to the top of the tower by a double-rope suspension system or rigid connector, which is used to project a cross light spot onto the target curtain at the bottom of the tower; Dynamic target and image acquisition module, which includes a 3m×3m high-reflectivity flexible screen with a millimeter-level grid printed on the surface, an industrial camera, and an ambient light sensor. It is used to transmit the target image captured by the industrial camera in real time to the image pre-processing unit; An image preprocessing unit integrates a ROI dynamic interception module, an adaptive noise reduction module, and a morphological processing module. The unit defines a 300×300 pixel ROI based on the spot position of the previous frame in step S10, dynamically selects a noise reduction algorithm according to the ambient light brightness L, and outputs a preprocessed ROI image. The target detection and analysis module is equipped with an improved YOLOv5 model to detect the center offset (ΔX, ΔY) and cross angle θ of the light spot in real time, and output a 200Hz dynamic parameter sequence. The multimodal data fusion module is used to linearly interpolate the 100Hz triaxial acceleration data to 200Hz, time-align it with the image modality data, and construct a 6-dimensional time series matrix; The feature extraction and analysis unit is used to extract 28-dimensional time-frequency domain and nonlinear features from the 6-dimensional matrix of the time window, and output them to the hybrid AI prediction and warning module after Z-score normalization; Hybrid AI prediction and warning module, which includes a 1D-CNN / LSTM hybrid model that outputs the probability distribution of seven types of vibration patterns through convolutional layers, Bi-LSTM, and spatiotemporal attention mechanisms to trigger hierarchical warning logic; Feedback control execution unit, the feedback control execution unit is used to receive early warning signals, update early warning thresholds and store abnormal data for model online learning.

8. A computer device / apparatus / system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the computer program to implement the steps of the mixed tower monitoring method based on the combination of laser vision and YOLOV5 model as described in any one of claims 1 to 6.

9. A computer non-transitory readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the mixed tower monitoring method based on the combination of laser vision and YOLOV5 model described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by one or more processors, the steps of the mixed tower monitoring method based on the combination of laser vision and YOLOV5 model as described in any one of claims 1 to 6 are implemented.

Images