Multi-modal data fusion wind turbine hybrid tower defect identification method and system

Abstract

The application discloses a wind turbine mixed tower defect identification method and system based on multi-modal data fusion, relates to the technical field of defect identification, and comprises the following steps: three-dimensional modeling is performed, a wind turbine mixed tower three-dimensional model is generated, and a multi-modal sensor group is arranged according to the wind turbine mixed tower three-dimensional model; mixed tower multi-modal detection data streams are synchronously collected; feature extraction and data fusion are performed to obtain a mixed tower multi-modal fusion feature set; a mixed tower defect identification model is established; the mixed tower multi-modal fusion feature set is identified and positioned by using the mixed tower defect identification model; a mixed tower defect feature distribution heat map is generated; defect grade evaluation is performed based on the mixed tower defect feature distribution heat map; the current mixed tower defect identification result is determined; and defect prediction and early warning are performed through the current mixed tower defect identification result. The application solves the technical problem of insufficient accuracy of wind turbine mixed tower defect identification in the prior art, and achieves the technical effect of improving the accuracy of wind turbine mixed tower defect identification.

Description

Technical Field

[0001] This invention relates to the field of defect identification technology, specifically to a method and system for identifying mixed tower defects in wind turbines using multimodal data fusion. Background Technology

[0002] Currently, wind turbine hybrid towers are prone to defects such as cracks, erosion, and abnormal connections during long-term service. Related inspections usually rely on a single type of inspection method to obtain defect information. Since different defects differ in their manifestation, distribution location, and evolution characteristics, a single data source cannot fully reflect the true state of the hybrid tower. It is also easily affected by environmental interference, limited collection range, and incomplete feature information, which leads to problems such as missed detection, false detection, or inaccurate positioning in the defect identification process, resulting in insufficient accuracy in identifying defects in wind turbine hybrid towers. Summary of the Invention

[0003] This application provides a method and system for identifying mixed tower defects in wind turbines using multimodal data fusion, which addresses the technical problem of insufficient accuracy in identifying mixed tower defects in wind turbines in the prior art.

[0004] In view of the above problems, this application provides a method and system for identifying mixed tower defects in wind turbines using multimodal data fusion.

[0005] The first aspect of this application provides a method for identifying mixed-tower defects in wind turbines based on multimodal data fusion, the method comprising: Based on the structural design information of the wind turbine hybrid tower, a 3D model is generated to produce a 3D model of the wind turbine hybrid tower. A multimodal sensor array is then deployed based on this model. Multimodal detection data streams are synchronously acquired using these sensor arrays. Feature extraction and data fusion are performed on the multimodal detection data streams to obtain a multimodal fusion feature set. A hybrid tower defect identification model is established, and this model is used to identify and locate defects within the multimodal fusion feature set, generating a heatmap of hybrid tower defect feature distribution. Defect level assessment is performed based on the heatmap to determine the current defect identification result, and defect prediction and early warning are then conducted based on this result.

[0006] A second aspect of this application provides a multimodal data fusion-based wind turbine tower defect identification system, the system comprising: The modeling module is used to perform 3D modeling based on the structural design information of the wind turbine hybrid tower, generate a 3D model of the wind turbine hybrid tower, and deploy a multimodal sensor group according to the 3D model of the wind turbine hybrid tower; the acquisition module is used to synchronously acquire the multimodal detection data stream of the hybrid tower through the multimodal sensor group, extract features and fuse data from the multimodal detection data stream to obtain a multimodal fusion feature set of the hybrid tower; the identification and positioning module is used to establish a hybrid tower defect identification model, use the hybrid tower defect identification model to identify and locate the multimodal fusion feature set of the hybrid tower, and generate a heat map of the distribution of hybrid tower defect features; the defect prediction and early warning module is used to evaluate the defect level based on the heat map of the distribution of hybrid tower defect features, determine the current hybrid tower defect identification result, and perform defect prediction and early warning based on the current hybrid tower defect identification result.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application uses the structural design information of the wind turbine hybrid tower to perform 3D modeling, generating a 3D model of the wind turbine hybrid tower. Based on this 3D model, a multimodal sensor group is deployed. The multimodal sensor group synchronously collects multimodal detection data streams of the hybrid tower, performs feature extraction and data fusion on these data streams, and obtains a multimodal fused feature set of the hybrid tower. A hybrid tower defect identification model is established, and this model is used to identify and locate the multimodal fused feature set of the hybrid tower, generating a heat map of the hybrid tower defect feature distribution. Based on the heat map, a defect level assessment is performed to determine the current defect identification result, and a defect prediction and early warning are then performed based on this result. This invention solves the technical problem of insufficient accuracy in identifying wind turbine hybrid tower defects in existing technologies. By extracting and fusing features from the multimodal detection data of the hybrid tower and identifying and locating defects based on the defect identification model, the technical effect of improving the accuracy of wind turbine hybrid tower defect identification is achieved. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A schematic flowchart of a method for identifying mixed-tower defects in wind turbines using multimodal data fusion, provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of the wind turbine mixed tower defect identification system based on multimodal data fusion provided in this application embodiment.

[0010] Figure labeling: Modeling module 11, data acquisition module 12, identification and location module 13, defect prediction and early warning module 14. Detailed Implementation

[0011] This application provides a method and system for identifying mixed tower defects in wind turbines by using multimodal data fusion. It addresses the technical problem of insufficient accuracy in identifying mixed tower defects in existing technologies by extracting and fusing features from multimodal detection data of mixed towers and identifying and locating defects based on a defect identification model, thereby improving the accuracy of identifying mixed tower defects in wind turbines.

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0013] It should be noted that any variation of the terms "comprising" and "having" is intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0014] Example 1, as Figure 1 As shown, this application provides a method for identifying mixed-tower defects in wind turbines based on multimodal data fusion, the method comprising: Step S100: Based on the structural design information of the wind turbine hybrid tower, perform three-dimensional modeling to generate a three-dimensional model of the wind turbine hybrid tower, and deploy a multi-modal sensor group according to the three-dimensional model of the wind turbine hybrid tower.

[0015] In this embodiment of the application, when performing 3D modeling based on the structural design information of the wind turbine hybrid tower, the pre-stored structural design information of the wind turbine hybrid tower is first extracted. The structural design information includes at least the tower height, dimensions of each tower section, wall thickness parameters, connection node positions, internal component layout, and material properties. Then, based on the structural design information, the tower base section, transition section, and tower body section of the hybrid tower are parametrically modeled in the 3D modeling software to construct a 3D geometric entity consistent with the actual hybrid tower structure. The spatial positional relationship, external contour, and key connection areas of each structural part are correspondingly calibrated to generate a 3D model of the wind turbine hybrid tower.

[0016] Next, based on the 3D model of the wind turbine tower, a multimodal sensor array is deployed. In this process, based on the 3D model of the wind turbine tower, key areas are identified according to the frequency of tower defect events, resulting in a set of key tower areas. A detection requirement analysis is performed on this set of key areas to obtain tower detection requirement parameters. Then, sensor layout planning and selection are conducted for the key area set according to these parameters to determine the tower detection sensor layout parameters. Finally, based on these sensor layout parameters, the multimodal sensor array is deployed on the working area of ​​the wind turbine tower.

[0017] Furthermore, in the method provided in the application embodiments, the deployment of a multi-modal sensor group based on the three-dimensional model of the wind turbine tower further includes: Based on the frequency of occurrence of mixed tower defect events, key areas of the three-dimensional model of the wind turbine mixed tower are identified to obtain a set of key mixed tower areas. Detection requirements are analyzed for the set of key mixed tower areas to obtain mixed tower detection requirement parameters. Sensor layout planning and selection are performed on the set of key mixed tower areas according to the mixed tower detection requirement parameters to determine the mixed tower detection sensor layout parameters. Based on the mixed tower detection sensor layout parameters, a multimodal sensor group is deployed on the working area of ​​the wind turbine mixed tower.

[0018] In this embodiment, when identifying key areas of the 3D modeling results of the mixed tower of a wind turbine based on the frequency of mixed tower defect events, the historical inspection and maintenance records of the mixed tower are first read. Defect events are then classified and statistically analyzed according to the tower base section, transition section, tower body section, and connection parts to obtain the number of defect events occurring in each structural part within a preset statistical period. The number of defect events occurring in each structural part is then mapped to the corresponding position in the 3D modeling results. Subsequently, the number of defect events is compared with preset identification conditions. The preset identification conditions can be preset frequency thresholds, such as the number of defect events occurring more than or equal to 3 within a preset statistical period, or being in the top 30% of the total number of occurrences of all structural parts. Areas that meet the preset identification conditions are marked as key areas. Adjacent key areas are then merged according to continuous boundaries to obtain a set of key areas for the mixed tower.

[0019] Next, a detection requirements analysis is conducted for the key areas of the mixed-tower system. Specifically, for each key area, the main defect type corresponding to that area is first determined, and then the detection content is determined based on the defect type. When the detection content is surface cracks, the crack width recognition accuracy and image resolution requirements are determined; when the detection content is internal damage, the detection depth and sensitivity requirements are determined; when the detection content is abnormal vibration, the sampling frequency and continuous acquisition duration requirements are determined. Simultaneously, considering the installation space, power supply conditions, and communication distance of the area, installation constraint parameters are determined. The above information is then compiled to form the mixed-tower detection requirement parameters, which at least include the detection range, detection accuracy, sampling frequency, detection depth, installation height, and communication distance.

[0020] Subsequently, based on the mixed-tower detection requirements, sensor layout planning and selection were performed for the key areas of the mixed-tower detection system, determining the sensor layout parameters. Specifically, first, a sensor correspondence was established: image acquisition requirements corresponded to vision sensors, internal defect detection requirements to ultrasonic sensors, and vibration status monitoring requirements to vibration sensors. Then, the number of sensors was calculated based on their effective coverage area. For example, if a single vision sensor can cover a 2-meter height range, at least three vision sensors would be configured for a 6-meter height key area. The installation point and direction of each sensor were determined based on the length, width, and installation location of the key area. Next, it was checked whether there were any detection gaps between adjacent sensors. If gaps existed, the number of sensors was increased or the installation spacing was adjusted. After completing these steps, the mixed-tower detection sensor layout parameters were obtained, which at least included sensor type, installation location, installation direction, number of sensors, installation spacing, and coverage area.

[0021] Finally, based on the hybrid tower detection sensor layout parameters, a multimodal sensor group was deployed in the working area of ​​the wind turbine hybrid tower. Specifically, according to the sensor type, installation location, and installation direction given in the hybrid tower detection sensor layout parameters, different types of sensors were fixed in their corresponding working areas. After installation, each sensor was numbered, and its location coordinates were recorded. Then, time synchronization was set for all sensors to ensure that the data collected by each sensor has a unified time reference. Subsequently, power-on tests and data acquisition tests were conducted to check whether the coverage area of ​​each sensor was consistent with the corresponding key area. If not, the positions were adjusted according to the hybrid tower detection sensor layout parameters. After location recording, time synchronization, and data acquisition tests, the deployment of the multimodal sensor group was completed.

[0022] Step S200: Synchronously acquire multimodal detection data streams of the mixed tower through the multimodal sensor group, extract features and fuse data from the multimodal detection data streams to obtain a multimodal fusion feature set of the mixed tower.

[0023] In this embodiment, when synchronously acquiring multimodal detection data streams of the hybrid wind turbine tower using a multimodal sensor array, various sensors deployed on different working areas of the hybrid wind turbine tower are simultaneously activated according to a unified time reference to synchronously detect the operating status and defect-related information of the hybrid tower and output corresponding detection data. The data acquired by different sensors may include image data, vibration data, acoustic data, or other detection data. This data from different sensing modes is recorded and aggregated at the same acquisition time or corresponding time period, thereby forming a multimodal detection data stream that collectively reflects the hybrid tower's status information.

[0024] Next, feature extraction and data fusion are performed on the multimodal detection data stream. In this process, firstly, a multimodal feature extraction algorithm is selected based on the modal characteristics of the multimodal detection data stream; then, the multimodal feature extraction algorithm is used to extract features from the multimodal detection data stream to obtain a multimodal mixed-tower perception feature set; finally, the multimodal mixed-tower perception feature set is aligned and fused to obtain a mixed-tower multimodal fusion feature set.

[0025] Furthermore, in the method provided in the application embodiments, feature extraction and data fusion are performed on the multimodal detection data stream to obtain a mixed-tower multimodal fusion feature set, which further includes: Based on the modal characteristics of the multimodal detection data stream, a multimodal feature extraction algorithm is selected; the multimodal feature extraction algorithm is used to extract features from the multimodal detection data stream to obtain a multimodal mixed-tower perception feature set; the multimodal mixed-tower perception feature set is aligned and fused to obtain a mixed-tower multimodal fusion feature set.

[0026] In this embodiment, when selecting a multimodal feature extraction algorithm based on the modal characteristics of the multimodal detection data stream, the synchronously acquired multimodal detection data stream is first classified according to sensor type, dividing the data into image data, vibration data, and acoustic data. Then, the information representation of each type of data is analyzed. Modal characteristics refer to the features of different data in information expression; image data is reflected in pixel distribution and edge changes, vibration data in waveform changes, and acoustic data in sound wave spectrum changes. Subsequently, a corresponding multimodal feature extraction algorithm is selected for each type of data according to its modal characteristics. Specifically, the Canny edge detection algorithm is selected for image data, the fast Fourier transform algorithm is selected for vibration data, and the Mel frequency cepstral coefficient extraction algorithm is selected for acoustic data. This completes the corresponding selection between the multimodal detection data stream and the multimodal feature extraction algorithm.

[0027] Next, a multimodal feature extraction algorithm is used to extract features from the multimodal detection data stream. In this process, image data is input into the Canny edge detection algorithm to extract crack edge location and morphological features; vibration data is input into the Fast Fourier Transform algorithm to extract dominant frequency distribution features; acoustic data is input into the Mel frequency cepstral coefficient extraction algorithm to extract acoustic spectrum features. After feature extraction of each type of data, sensor numbers, acquisition times, and detection location identifiers are added to the extracted features. Then, the modal features are aggregated according to the same acquisition cycle to form a multimodal mixed-tower sensing feature set. The multimodal mixed-tower sensing feature set refers to the set of feature data extracted from image data, vibration data, and acoustic data respectively, which can jointly characterize the mixed-tower state information.

[0028] Finally, the multimodal mixed-tower sensing feature set is aligned and fused. In this process, the multimodal mixed-tower sensing feature set is first aligned according to timestamps and spatial coordinates to obtain a usable multimodal mixed-tower sensing feature set; then, the reliability of each modal sensor in the multimodal sensor group is evaluated to determine the multimodal sensor reliability coefficient set; finally, based on the multimodal sensor reliability coefficient set, the multimodal mixed-tower sensing feature set is fused using feature weighting to obtain the mixed-tower multimodal fused feature set.

[0029] Furthermore, in the method provided in the application embodiments, the alignment and feature fusion of the multimodal mixed-tower sensing feature set to obtain a mixed-tower multimodal fused feature set further includes: The multimodal mixed-tower sensing feature set is aligned according to timestamps and spatial coordinates to obtain a usable multimodal mixed-tower sensing feature set; the reliability of each modal sensor in the multimodal sensor group is evaluated to determine the multimodal sensor reliability coefficient set; based on the multimodal sensor reliability coefficient set, the multimodal mixed-tower sensing feature set is fused with feature weighting to obtain a mixed-tower multimodal fusion feature set.

[0030] In this embodiment, when aligning the multimodal mixed-tower sensing feature set according to timestamps and spatial coordinates, the feature records output by the image sensor, vibration sensor, and acoustic sensor are first uniformly organized so that each feature record contains a timestamp, spatial coordinates, and feature value. Then, a time window matching method is used for time alignment, that is, all feature records are grouped according to a preset acquisition cycle, so that feature records within the same acquisition cycle are grouped into the same time unit. After completing the time grouping, a region mapping method is used for spatial alignment, that is, according to the installation position and detection coverage of each sensor, the spatial coordinates of each feature record are mapped to a preset detection area, so that feature records located in the same detection area are grouped into the same spatial unit. Then, each spatial unit in each time unit is compared item by item, and image features, vibration features, and acoustic features that simultaneously satisfy the time correspondence and spatial correspondence are retained, while feature records that lack time correspondence or lack spatial correspondence are removed, thereby obtaining a usable multimodal mixed-tower sensing feature set.

[0031] Next, the reliability of each modal sensor in the multimodal sensor group is evaluated to determine the reliability coefficient set of the multimodal sensor. Specifically, a proportional calculation method is used to statistically analyze the image sensor, vibration sensor, and acoustic sensor respectively. First, the number of acquired features is counted, which refers to the total number of feature records that each modal sensor should output according to the preset acquisition cycle within the preset statistical period. Then, the number of valid features is counted, which refers to the number of feature records with complete timestamps, complete spatial coordinates, and no missing feature values. Subsequently, the number of abnormal features is counted, which refers to the number of feature records in the valid feature records that have data mutations, data interruptions, or exceed the preset range. Finally, the number of features that have achieved dual temporal and spatial alignment is counted, which refers to the number of feature records that simultaneously satisfy the temporal correspondence and spatial correspondence after alignment processing and are retained. After completing the above statistics, the completeness is obtained by dividing the number of effective features by the number of collected features; the stability is obtained by subtracting the number of abnormal features from the number of effective features and then dividing by the number of effective features; and the alignment is obtained by dividing the number of features that have achieved both temporal and spatial alignment by the number of effective features. Then, the completeness, stability, and alignment are added together and divided by 3 to obtain the reliability coefficient of the corresponding modal sensor. After performing the above steps on each modal sensor, the reliability coefficient set of the multimodal sensor is obtained.

[0032] Finally, feature weighting and fusion are performed on the multimodal hybrid tower sensing feature set based on the multimodal sensor reliability coefficient set. In this process, the aligned image features, vibration features, and acoustic features are first combined according to the same time unit and the same spatial unit to form feature groups to be fused. Then, the different modal features in each feature group to be fused are uniformly processed to ensure that each modal feature is within a consistent data range. Next, the reliability coefficients corresponding to each modal feature are read from the multimodal sensor reliability coefficient set, and each modal feature value is multiplied by its corresponding reliability coefficient to obtain the weighted result of each modal feature. Then, the weighted results of each modal feature are summed, and the corresponding reliability coefficients are summed. Finally, the sum of the aforementioned weighted results is divided by the sum of the reliability coefficients to obtain the fused feature value corresponding to that time unit and that spatial unit. After performing the above steps sequentially on all feature groups to be fused, the obtained fused feature values ​​are aggregated to obtain the hybrid tower multimodal fusion feature set.

[0033] Step S300: Establish a mixed tower defect identification model, use the mixed tower defect identification model to identify and locate the mixed tower multimodal fusion feature set, and generate a heat map of mixed tower defect feature distribution.

[0034] In this embodiment of the application, when establishing the mixed tower defect identification model, the historical mixed tower defect detection dataset is first collected and the mixed tower historical defect detection dataset is labeled with types to obtain the mixed tower defect detection sample set; then the defect model architecture is selected, the backbone network of the defect model architecture adopts ResNet50, the classification output layer adopts a fully connected layer + Softmax, and the defect type probability is output; finally, the defect model architecture is used to train the mixed tower defect detection sample set for defect identification to establish the mixed tower defect identification model.

[0035] Next, a hybrid tower defect identification model is used to identify and locate the hybrid tower multimodal fusion feature set. In this process, the hybrid tower defect identification model is first used to identify and locate the hybrid tower multimodal fusion feature set, obtaining a hybrid tower defect feature type set and a corresponding hybrid tower defect feature location set; then, based on the hybrid tower defect feature type set and the corresponding hybrid tower defect feature location set, the defect distribution of the wind turbine hybrid tower 3D model is visualized, generating a hybrid tower defect feature distribution heat map.

[0036] Furthermore, the method provided in the application embodiments, in establishing the mixed tower defect identification model, further includes: Collect a historical defect detection dataset of mixed towers, and label the dataset to obtain a mixed tower defect detection sample set. Select a defect model architecture, with ResNet50 as the backbone network and a fully connected layer + Softmax as the classification output layer to output the probability of defect type. Use the defect model architecture to train the mixed tower defect detection sample set for defect recognition and establish a mixed tower defect recognition model.

[0037] In this embodiment, when establishing the hybrid tower defect identification model, a historical hybrid tower defect detection dataset is first collected, and the dataset is then labeled with different types to obtain a hybrid tower defect detection sample set. The historical hybrid tower defect detection dataset consists of historical detection data obtained during each hybrid tower inspection of the wind turbine, containing defect-related detection results and their corresponding defect information. After organizing the historical hybrid tower defect detection dataset, data with missing time information, missing location information, or unidentifiable defect categories are removed. Then, type labeling is performed based on the defect manifestations reflected in each historical detection data point, ensuring that each historical detection data point corresponds to a clear defect category label. This forms a hybrid tower defect detection sample set with defect category labels, thus completing the conversion from the historical hybrid tower defect detection dataset to the hybrid tower defect detection sample set.

[0038] After obtaining the mixed-tower defect detection sample set, a defect model architecture is selected. The backbone network of the defect model architecture uses ResNet50, and the classification output layer uses a fully connected layer + Softmax to output the defect type probability. ResNet50 is used to extract features layer by layer from the input samples in the mixed-tower defect detection sample set, so that the features that can represent the differences in defect categories are extracted from the input samples. After ResNet50 completes feature extraction, the extracted feature results are input into the fully connected layer, which establishes the correspondence between the feature results and each defect category. Then, the output result of the fully connected layer is input into Softmax to normalize the output value of each defect category, thereby obtaining the defect type probability corresponding to each defect category. The defect type probability is used to represent the probability distribution of the input sample belonging to different defect categories, thus completing the determination of the defect model architecture.

[0039] After determining the defect model architecture, it is used to train the mixed tower defect detection sample set for defect recognition, thus establishing a mixed tower defect recognition model. The mixed tower defect detection sample set is input into the defect model architecture, where ResNet50 extracts defect features from each sample. These features are then passed through a fully connected layer and Softmax to output the defect type probability of the corresponding sample. This probability is compared with the corresponding defect category label to determine the classification error between the current recognition result and the type labeling result. The network parameters in the defect model architecture are then updated based on the classification error. This process of sample input, feature extraction, probability output, error comparison, and parameter update is repeated until the defect model architecture's recognition result for the mixed tower defect detection sample set meets the preset training conditions, completing the defect recognition training and obtaining the mixed tower defect recognition model. This achieves a continuous processing flow from collecting historical mixed tower defect detection datasets, type labeling, determining the defect model architecture, to defect recognition training.

[0040] Furthermore, in the method provided in the application embodiments, the method uses the hybrid tower defect identification model to identify and locate the hybrid tower multimodal fusion feature set, and generates a heat map of hybrid tower defect feature distribution, and further includes: The hybrid tower defect identification model is used to identify and locate the multimodal fusion feature set of the hybrid tower, thereby obtaining the hybrid tower defect feature type set and the corresponding hybrid tower defect feature location set; based on the hybrid tower defect feature type set and the corresponding hybrid tower defect feature location set, the defect distribution of the three-dimensional model of the wind turbine hybrid tower is visualized, and a heat map of hybrid tower defect feature distribution is generated.

[0041] In this embodiment, when using a hybrid tower defect identification model to identify and locate the hybrid tower multimodal fusion feature set, the hybrid tower multimodal fusion feature set is input into the hybrid tower defect identification model according to a preset input order. The hybrid tower defect identification model identifies the defect category of each fusion feature and outputs the corresponding defect type probability. The defect type probability is used to characterize the probability distribution of the current fusion feature belonging to each defect category. Subsequently, the corresponding spatial location identifier is read from each hybrid tower multimodal fusion feature, and the defect category corresponding to the highest probability in the defect type probability is determined as the identification result of that fusion feature. Then, the identification result is associated with the corresponding spatial location identifier to form the defect type and defect location corresponding to a single fusion feature. After performing defect category identification, probability determination, and location association on all hybrid tower multimodal fusion features in sequence, a hybrid tower defect feature type set and a corresponding hybrid tower defect feature location set are obtained. The hybrid tower defect feature type set is the set of defect type results obtained by the hybrid tower defect identification model, and the hybrid tower defect feature location set is the set of location results corresponding to each defect type result.

[0042] Next, when visualizing the defect distribution of the 3D model of the wind turbine's mixed tower based on the mixed tower defect feature type set and the corresponding mixed tower defect feature location set, the 3D model of the wind turbine's mixed tower is first divided into multiple visualization statistical units, each corresponding to a part of the mixed tower. Then, the location results in the mixed tower defect feature location set are converted into the corresponding locations in the 3D model of the wind turbine's mixed tower, and the visualization statistical unit to which each location result belongs is determined. Then, the defect type result corresponding to the location result is written into the corresponding visualization statistical unit. After completing all location mapping and type writing, the defect records written in each visualization statistical unit are counted, and the number of defect records is used as the heat value of the visualization statistical unit; that is, if a visualization statistical unit corresponds to 1 defect record, the heat value of the visualization statistical unit is 1, and if it corresponds to 2 defect records, the heat value of the visualization statistical unit is 2. Next, the display intensity is assigned based on the heat values ​​of each visualization statistical unit. A visualization statistical unit with a heat value of 0 corresponds to the first display intensity, a unit with a heat value of 1 corresponds to the second display intensity, a unit with a heat value of 2 corresponds to the third display intensity, and so on, ensuring that different visualization statistical units display results according to their respective heat values. Finally, the display results of each visualization statistical unit are loaded into the corresponding area of ​​the wind turbine hybrid tower 3D model, completing the visualization of defect distribution and generating a heat map of hybrid tower defect feature distribution.

[0043] Step S400: Based on the heat map of the mixed tower defect feature distribution, perform defect level assessment, determine the current mixed tower defect identification result, and perform defect prediction and early warning based on the current mixed tower defect identification result.

[0044] In this embodiment of the application, when evaluating the defect level based on the heat map of the defect feature distribution of the mixed tower, the heat value corresponding to each visualization statistical unit and the defect type result corresponding to the visualization statistical unit are read first, and then the determination is made according to the preset level classification rules. Specifically, visualization statistical units with a thermal value of 0 and no defect type results are identified as defect-free areas; visualization statistical units with thermal values ​​of 1 to 2 corresponding to minor cracks or minor wear are identified as Level 1 defect areas; visualization statistical units with thermal values ​​of 3 to 4 corresponding to obvious cracks or localized peeling are identified as Level 2 defect areas; and visualization statistical units with thermal values ​​greater than or equal to 5 corresponding to through cracks or severe damage are identified as Level 3 defect areas. After determining the level of all visualization statistical units, the defect level results of each visualization statistical unit are summarized. When a Level 3 defect area exists, the current mixed tower defect identification result is determined to be a severe defect; when no Level 3 defect area exists but a Level 2 defect area exists, the current mixed tower defect identification result is determined to be a moderate defect; when only a Level 1 defect area exists, the current mixed tower defect identification result is determined to be a minor defect; and when all visualization statistical units are defect-free areas, the current mixed tower defect identification result is determined to be defect-free.

[0045] Next, defect prediction and early warning are performed based on the current mixed tower defect identification results. In this process, firstly, the defect expansion trend is predicted based on the current mixed tower defect identification results to obtain the mixed tower defect expansion trend prediction results; then, the early warning level is matched with the mixed tower defect expansion trend prediction results to determine the mixed tower defect early warning level, and defect early warning processing is performed based on the mixed tower defect early warning level.

[0046] Furthermore, the method provided in the application embodiment, which performs defect prediction and early warning based on the current mixed tower defect identification result, further includes: Based on the current mixed tower defect identification results, the defect expansion trend is predicted to obtain the mixed tower defect expansion trend prediction result; the mixed tower defect expansion trend prediction result is matched with the early warning level to determine the mixed tower defect early warning level, and the defect early warning is processed according to the mixed tower defect early warning level.

[0047] In this embodiment, when predicting the expansion trend of defects based on the current mixed-tower defect identification results, the defect type, defect location, defect level, and thermal value of the corresponding area in the current mixed-tower defect identification results are first read. Then, the defect identification results of the same detection area in the previous detection cycle are retrieved and matched according to the defect location. The defect expansion trend prediction is a process of judging the development status of the same defect in adjacent detection cycles. Subsequently, the thermal value, defect level, and defect distribution range of the same defect location are compared between the current and previous detection cycles. The thermal value comparison is used to determine whether the number of defect records at that location has changed; the defect level comparison is used to determine whether the severity of the defect at that location has changed; and the defect distribution range comparison is used to determine whether the number of visual statistical units corresponding to that location has changed. After the comparison, when the thermal value, defect level, and defect distribution range remain unchanged, the trend state corresponding to the defect location is determined to be a stable trend. When the thermal value increases, the defect level increases, or the defect distribution range expands, the trend state corresponding to the defect location is determined to be an expanding trend. After performing the above comparisons on all defect locations in sequence, the trend states corresponding to each defect location are aggregated to obtain the mixed tower defect expansion trend prediction result; wherein, the mixed tower defect expansion trend prediction result is a set of trend state results corresponding one-to-one with each defect location.

[0048] After obtaining the predicted trend of mixed tower defects, the prediction trend is matched with the warning level to determine the warning level of the mixed tower defects, and then the defects are processed according to the warning level. Specifically, a warning level matching rule is first established, and the defect level in the current mixed tower defect identification result is matched with the trend state in the mixed tower defect expansion trend prediction result; the warning level matching is the process of mapping the defect level and trend state to the preset warning level. Then, the judgment is made according to the preset rule: when the defect level is level one and the trend state is stable, the mixed tower defect warning level is determined to be level one; when the defect level is level one and the trend state is expanding, or when the defect level is level two and the trend state is stable, the mixed tower defect warning level is determined to be level two; when the defect level is level two and the trend state is expanding, or when the defect level is level three and the trend state is stable, the mixed tower defect warning level is determined to be level three; when the defect level is level three and the trend state is expanding, the mixed tower defect warning level is determined to be level four. After matching the warning levels for all defect locations, the defect warning processing is performed according to the corresponding mixed tower defect warning level. Specifically, Level 1 warning corresponds to defect record archiving, Level 2 warning corresponds to maintenance reminder output, Level 3 warning corresponds to time-limited maintenance warning output, and Level 4 warning corresponds to immediate shutdown inspection warning output. This completes the determination of the mixed tower defect warning level and the defect warning processing.

[0049] In summary, the embodiments of this application have at least the following technical effects: This application uses the structural design information of the wind turbine hybrid tower to perform 3D modeling, generating a 3D model of the wind turbine hybrid tower. Based on this 3D model, a multimodal sensor group is deployed. The multimodal sensor group synchronously collects multimodal detection data streams of the hybrid tower, performs feature extraction and data fusion on these data streams, and obtains a multimodal fused feature set of the hybrid tower. A hybrid tower defect identification model is established, and this model is used to identify and locate the multimodal fused feature set of the hybrid tower, generating a heat map of the hybrid tower defect feature distribution. Based on the heat map, a defect level assessment is performed to determine the current defect identification result, and a defect prediction and early warning are then performed based on this result. This invention solves the technical problem of insufficient accuracy in identifying wind turbine hybrid tower defects in existing technologies. By extracting and fusing features from the multimodal detection data of the hybrid tower and identifying and locating defects based on the defect identification model, the technical effect of improving the accuracy of wind turbine hybrid tower defect identification is achieved.

[0050] Example 2, based on the same inventive concept as the wind turbine mixed tower defect identification method using multimodal data fusion in the aforementioned examples, such as... Figure 2 As shown, this application provides a multimodal data fusion-based wind turbine tower defect identification system. The system and method embodiments in this application are based on the same inventive concept. The system includes: Modeling module 11 is used to perform three-dimensional modeling based on the structural design information of the wind turbine hybrid tower, generate a three-dimensional model of the wind turbine hybrid tower, and deploy a multi-modal sensor group according to the three-dimensional model of the wind turbine hybrid tower; Acquisition module 12 is used to synchronously acquire the hybrid tower multi-modal detection data stream through the multi-modal sensor group, extract features and fuse data from the multi-modal detection data stream to obtain a hybrid tower multi-modal fusion feature set; Identification and positioning module 13 is used to establish a hybrid tower defect identification model, use the hybrid tower defect identification model to identify and locate the hybrid tower multi-modal fusion feature set, and generate a hybrid tower defect feature distribution heat map; Defect prediction and early warning module 14 is used to evaluate the defect level based on the hybrid tower defect feature distribution heat map, determine the current hybrid tower defect identification result, and perform defect prediction and early warning based on the current hybrid tower defect identification result.

[0051] Furthermore, the system is also used to implement the following functions: Based on the frequency of occurrence of mixed tower defect events, key areas of the three-dimensional model of the wind turbine mixed tower are identified to obtain a set of key mixed tower areas. Detection requirements are analyzed for the set of key mixed tower areas to obtain mixed tower detection requirement parameters. Sensor layout planning and selection are performed on the set of key mixed tower areas according to the mixed tower detection requirement parameters to determine the mixed tower detection sensor layout parameters. Based on the mixed tower detection sensor layout parameters, a multimodal sensor group is deployed on the working area of ​​the wind turbine mixed tower.

[0052] Furthermore, the system is also used to implement the following functions: Based on the modal characteristics of the multimodal detection data stream, a multimodal feature extraction algorithm is selected; the multimodal feature extraction algorithm is used to extract features from the multimodal detection data stream to obtain a multimodal mixed-tower perception feature set; the multimodal mixed-tower perception feature set is aligned and fused to obtain a mixed-tower multimodal fusion feature set.

[0053] Furthermore, the system is also used to implement the following functions: The multimodal mixed-tower sensing feature set is aligned according to timestamps and spatial coordinates to obtain a usable multimodal mixed-tower sensing feature set; the reliability of each modal sensor in the multimodal sensor group is evaluated to determine the multimodal sensor reliability coefficient set; based on the multimodal sensor reliability coefficient set, the multimodal mixed-tower sensing feature set is fused with feature weighting to obtain a mixed-tower multimodal fusion feature set.

[0054] Furthermore, the system is also used to implement the following functions: Collect a historical defect detection dataset of mixed towers, and label the dataset to obtain a mixed tower defect detection sample set. Select a defect model architecture, with ResNet50 as the backbone network and a fully connected layer + Softmax as the classification output layer to output the probability of defect type. Use the defect model architecture to train the mixed tower defect detection sample set for defect recognition and establish a mixed tower defect recognition model.

[0055] Furthermore, the system is also used to implement the following functions: The hybrid tower defect identification model is used to identify and locate the multimodal fusion feature set of the hybrid tower, thereby obtaining the hybrid tower defect feature type set and the corresponding hybrid tower defect feature location set; based on the hybrid tower defect feature type set and the corresponding hybrid tower defect feature location set, the defect distribution of the three-dimensional model of the wind turbine hybrid tower is visualized, and a heat map of hybrid tower defect feature distribution is generated.

[0056] Furthermore, the system is also used to implement the following functions: Based on the current mixed tower defect identification results, the defect expansion trend is predicted to obtain the mixed tower defect expansion trend prediction result; the mixed tower defect expansion trend prediction result is matched with the early warning level to determine the mixed tower defect early warning level, and the defect early warning is processed according to the mixed tower defect early warning level.

[0057] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for identifying mixed-tower defects in wind turbines based on multimodal data fusion, characterized in that, The method includes: Three-dimensional modeling is performed based on the structural design information of the wind turbine hybrid tower to generate a three-dimensional model of the wind turbine hybrid tower, and a multi-modal sensor group is deployed according to the three-dimensional model of the wind turbine hybrid tower. The multimodal sensor group synchronously collects the multimodal detection data stream of the mixed tower, performs feature extraction and data fusion on the multimodal detection data stream, and obtains the multimodal fusion feature set of the mixed tower; A mixed tower defect identification model is established, and the mixed tower multimodal fusion feature set is identified and located using the mixed tower defect identification model to generate a heat map of mixed tower defect feature distribution. Based on the heat map of the mixed tower defect feature distribution, the defect level is evaluated, the current mixed tower defect identification result is determined, and the defect prediction and early warning are performed based on the current mixed tower defect identification result.

2. The method for identifying mixed tower defects in wind turbines using multimodal data fusion as described in claim 1, characterized in that, Based on the aforementioned three-dimensional model of the wind turbine tower, a multimodal sensor array is deployed, including: Based on the frequency of occurrence of mixed tower defect events, the key areas of the three-dimensional model of the mixed tower of the wind turbine are identified to obtain the set of key areas of the mixed tower. The detection requirements of the key areas of the mixed tower are analyzed to obtain the detection requirements parameters of the mixed tower. Based on the required parameters for mixed tower detection, the sensor layout is planned and selected for the key area set of the mixed tower, and the sensor layout parameters for mixed tower detection are determined. Based on the aforementioned hybrid tower detection sensor layout parameters, a multimodal sensor group is deployed on the working area of ​​the hybrid tower of the wind turbine.

3. The method for identifying mixed-tower defects in wind turbines using multimodal data fusion as described in claim 1, characterized in that, Feature extraction and data fusion are performed on the multimodal detection data stream to obtain a multimodal fusion feature set, including: Based on the modal characteristics of the multimodal detection data stream, a multimodal feature extraction algorithm is selected; The multimodal feature extraction algorithm is used to extract features from the multimodal detection data stream to obtain a multimodal mixed tower sensing feature set; Alignment and feature fusion are performed on the multimodal mixed tower sensing feature set to obtain the mixed tower multimodal fusion feature set.

4. The method for identifying mixed tower defects in wind turbines using multimodal data fusion as described in claim 3, characterized in that, The multimodal mixed-tower sensing feature set is aligned and fused to obtain a multimodal mixed-tower fused feature set, including: The multimodal mixed tower sensing feature set is aligned according to timestamps and spatial coordinates to obtain a usable multimodal mixed tower sensing feature set; The reliability of each modal sensor in the multimodal sensor group is evaluated to determine the set of reliability coefficients for the multimodal sensors. Based on the reliability coefficient set of the multimodal sensors, the multimodal hybrid tower sensing feature set is fused by feature weighting to obtain the hybrid tower multimodal fusion feature set.

5. The method for identifying mixed tower defects in wind turbines using multimodal data fusion as described in claim 1, characterized in that, Establish a defect identification model for mixed towers, including: Collect a historical defect detection dataset of mixed towers, and label the dataset with different types to obtain a sample set of mixed tower defect detections; The defect model architecture is selected, with ResNet50 as the backbone network and a fully connected layer + Softmax as the classification output layer, which outputs the probability of defect type. The defect model architecture is used to train the mixed tower defect detection sample set for defect identification, and a mixed tower defect identification model is established.

6. The method for identifying mixed tower defects in wind turbines using multimodal data fusion as described in claim 1, characterized in that, The hybrid tower defect identification model is used to identify and locate the multimodal fusion feature set of the hybrid tower, generating a heat map of the hybrid tower defect feature distribution, including: The mixed tower defect identification model is used to identify and locate the mixed tower multimodal fusion feature set, thereby obtaining the mixed tower defect feature type set and the corresponding mixed tower defect feature location set; Based on the set of mixed tower defect feature types and the corresponding set of mixed tower defect feature locations, the defect distribution of the three-dimensional model of the wind turbine mixed tower is visualized, and a heat map of the mixed tower defect feature distribution is generated.

7. The method for identifying mixed tower defects in wind turbines using multimodal data fusion as described in claim 1, characterized in that, Defect prediction and early warning are performed based on the current mixed tower defect identification results, including: Based on the current mixed tower defect identification results, the defect expansion trend is predicted to obtain the mixed tower defect expansion trend prediction results. The prediction results of the mixed tower defect expansion trend are matched with the early warning level to determine the mixed tower defect early warning level, and the defect early warning is processed according to the mixed tower defect early warning level.

8. A multimodal data fusion-based wind turbine tower defect identification system, characterized in that, The system is used to execute the wind turbine tower defect identification method based on multimodal data fusion as described in any one of claims 1-7, the system comprising: The modeling module is used to perform three-dimensional modeling based on the structural design information of the wind turbine hybrid tower, generate a three-dimensional model of the wind turbine hybrid tower, and deploy a multi-modal sensor group according to the three-dimensional model of the wind turbine hybrid tower. The acquisition module is used to synchronously acquire the multimodal detection data stream of the mixed tower through the multimodal sensor group, and to perform feature extraction and data fusion on the multimodal detection data stream to obtain the multimodal fusion feature set of the mixed tower. The identification and localization module is used to establish a mixed tower defect identification model, use the mixed tower defect identification model to identify and locate the mixed tower multimodal fusion feature set, and generate a heat map of mixed tower defect feature distribution. The defect prediction and early warning module is used to evaluate the defect level based on the heat map of the mixed tower defect feature distribution, determine the current mixed tower defect identification result, and perform defect prediction and early warning based on the current mixed tower defect identification result.

Images