Sea condition classification method for offshore wind plant

Abstract

The invention relates to a sea condition classification method for an offshore wind plant. The method comprises the following steps: S1, carrying out key frame extraction on an original ocean monocular vision in-situ video; s2, performing sea condition grade labeling based on sea surface features in the key frame picture to obtain a sea condition data set; s3, constructing a sea condition classification model; s4, preprocessing the images in the sea condition data set to obtain a model input sequence integrated with an image block embedding sequence, a category vector and a position coding vector; and S5, inputting the model input sequence into the sea condition classification model to carry out fine tuning training on the model until the model converges, and obtaining the trained sea condition classification model. According to the invention, the sea condition classification model based on the ViT network is constructed, the 2D interpolation method is used to enable the model to adapt to the high-resolution sea condition image input, the multi-head self-attention mechanism of the model is used to capture the global features of the wave form, all-weather real-time classification is realized, and the precision, speed and robustness of sea condition classification are significantly improved.

Description

Technical Field

[0001] The invention belongs to the technical field of ocean observation, and in particular relates to a method for classifying sea conditions in an offshore wind farm. Background Art

[0002] The sea state table developed by the World Meteorological Organization is an important standard for assessing the current sea conditions of offshore wind farms. It can describe the wave conditions in wind farms, thereby providing a reference for weather forecasting and marine engineering, and effectively optimizing the working window for offshore construction operations.

[0003] With the development of computer vision and image model technology, the task of classifying sea conditions has gradually shifted from analyzing the response data of wave measuring instruments to image classification tasks. Its purpose is to invert the real sea condition level within the visual range through information such as the texture, fragmentation, and white cap of the ocean surface. As for the method of realizing sea condition classification through image recognition technology and image classification model, there are also some designs in the existing technology. However, the existing recognition and classification methods generally face the following problems: (1) Most existing datasets are small-scale and non-standardized, with insufficient samples of extreme sea conditions, which can easily lead to model training bias and low credibility of classification results; (2) The original sea condition images usually have a high resolution, while the existing feature extraction and image classification models can only accept input images with a low resolution and cannot directly process the original sea condition images. If the original sea condition images are directly reduced in size and input into the model for recognition and classification, the wave morphology in the image will be seriously affected, thereby affecting the effect of sea condition classification.

[0004] Therefore, there is an urgent need for a sea condition classification method that can process high-resolution original sea condition images and whose classification accuracy and speed can meet the requirements of all-weather real-time sea condition classification. Summary of the invention

[0005] The purpose of the present invention is to solve one of the above technical problems and provide a sea condition classification method for an offshore wind farm. By fusing HSV color histogram and Canny edge feature to extract key frames, a balanced data set covering multiple levels of sea conditions is constructed, and a sea condition classification model based on a Vision Transformer network is constructed. The model is adapted to high-resolution sea condition image input through a 2D Ranchos interpolation method, and the multi-head self-attention mechanism of the model is used to capture the global characteristics of wave morphology, so as to realize all-weather real-time classification and significantly improve the accuracy, speed and robustness of sea condition classification.

[0006] To achieve the above object, the technical solution adopted by the present invention is: A method for classifying sea conditions in an offshore wind farm comprises the following steps: S1: Obtain the original ocean monocular vision in-situ video data of the offshore wind farm and extract key frames from the video; S2: Obtain the significant wave height and wind speed of the current sea area when the key frame image is taken, and mark the key frame with the sea state level based on the significant wave height and wind speed to obtain a sea state data set based on sea state level classification; S3: Build a sea state classification model based on the Vision Transformer network; the sea state classification model includes an encoder and a multi-layer perceptron classification head; S4: Preprocess the images in the sea state dataset to obtain a model input sequence consisting of an image block embedding sequence, a category vector, and a position encoding vector; S5: Input the model input sequence into the sea state classification model to fine-tune the model, so that the model input sequence is continuously forward-transferred in the encoder to extract the features corresponding to the category vector therein, and input the features into the multi-layer perceptron classification head to obtain the classification result of the category vector. During the training process, the model parameters are adjusted until the model converges to obtain the trained sea state classification model; S6: Use the trained sea state classification model to classify sea conditions.

[0007] In some embodiments of the present invention, the method of preprocessing the images in the data set in step S4 includes the following steps: S41: adjusting the image in the sea state dataset to a predetermined size; S42: performing image block embedding processing on the adjusted image, dividing the image into a plurality of image blocks of fixed sizes, and generating an image block embedding sequence; S43: setting a learnable embedding vector as a category vector for classification; S44: Use sine and cosine functions to perform two-dimensional position encoding on each image block to obtain an original two-dimensional position encoding vector; perform a 2D Ranchos interpolation operation on the original two-dimensional position encoding vector, calculate the position encoding of the interpolation position in each dimension, and merge the position encoding of all dimensions to obtain a complete position encoding vector.

[0008] S45: Concatenate the image patch embedding sequence, the category vector, and the complete position encoding vector to generate the model input sequence.

[0009] In some embodiments of the present invention, in step S44, the method of using sine and cosine functions to perform two-dimensional position encoding on each image block is: For any position in each image block (x,y) , the calculation formula for the position encoding of even dimensions is: ; ; The calculation formula for the position encoding of odd dimensions is: ; ; Among them, x and y are the horizontal and vertical coordinates in the current image block, respectively, and PE(x,2i) is the position x In the 2nd i Dimensional encoding, i and j For dimension-dependent indexes, D is the embedding dimension.

[0010] In some embodiments of the present invention, the formula for calculating the position encoding of the interpolation position in each dimension is: ; in, is the encoding result of the interpolation position at the new position, is the target pixel position in each dimension, is the original two-dimensional position encoding vector in The value at m and n is the pixel distance of the traversal Ranchos window, and They are x and y Scaling factors in both dimensions, is the Ranchos window parameter, which is used to control the interpolation range. is the Ranchos function; in, and The calculation formulas are: ; ; in, W old and H old are the numbers of the original two-dimensional position encoding in the width and height dimensions, W new and H new The numbers of position encodings in width and height dimensions required for the model input image are respectively; The calculation formula of the Ranchos function is: .

[0011] In some embodiments of the present invention, the fine-tuning training process uses Focal Loss The loss function accelerates the convergence of the model. The expression of the Focal Loss loss function is: ; in, For the k The predicted probability of the class sample, the sum of the predicted probabilities of all classes is 1, For the real label One-Hot Encoding, that is, the actual category corresponding to , otherwise 0, For Category k The weight of is the focus factor, used to magnify role.

[0012] In some embodiments of the present invention, the following steps are further included: Calculate the macro-average of the classification results of the sea state classification model; in the classification results, each classification category corresponds to a sea state level; the macro-average includes the arithmetic mean of the precision, recall and F1-Score of each category, and the macro-average calculation formulas are: ; ; ; in, is the arithmetic mean of the precision rate, is the arithmetic mean of the recall rate, is the arithmetic mean of F1-Score; Calculate the micro-average of the classification results of the sea state classification model; the micro-average includes the precision, recall and F1-Score calculated based on the sum of TP, FP and FN of each category. The micro-calculation formulas are: ; ; ; in, Calculated based on the sum of TP, FP, and FN of each category, is the recall calculated based on the sum of TP, FP, and FN of each category, The F1-Score is calculated based on the sum of TP, FP, and FN of each category. TP means that the positive sample is successfully predicted as positive; TN means that the negative sample is successfully predicted as negative; FP means that the negative sample is incorrectly predicted as positive; and FN means that the positive sample is incorrectly predicted as negative. l i The model predicts i The number of samples with correct predictions. m i The model predicts i The number of categories, ni is the number of items that actually belong to the i-th category; Calculate the classification accuracy of each category in the classification results of the model: ; The classification results of the trained sea state classification model were evaluated based on the macro-average, micro-average and classification accuracy of each category.

[0013] In some embodiments of the present invention, the method for extracting key frames from a video specifically includes the following steps: S11: extracting the HSV histogram feature vector and the canny edge contour feature vector of each frame image of the original ocean monocular vision in-situ video; S12: Concatenate the HSV histogram features and the canny edge contour features to obtain a fused feature vector; S13: setting a predetermined unit video duration; using an adaptive mean clustering method to sequentially classify each frame image within the unit video duration into a corresponding cluster at each predetermined unit video duration, and updating the cluster center of each cluster in real time; S14: extracting the image frames closest to the respective cluster centers as the key frames of the respective clusters.

[0014] In some embodiments of the present invention, the method for extracting the HSV histogram feature vector of each frame of the original ocean monocular vision in-situ video specifically includes the following steps: Convert the image from RGB domain to HSV domain and then draw its color histogram; H represents hue, S represents saturation, and V represents brightness; in HSV domain, hue is used to represent color components, and the three RGB channels are compressed into one channel for color segmentation and brightness processing; After obtaining the color histogram, the histograms of the three dimensions are normalized respectively and concatenated into the HSV histogram feature vector.

[0015] In some embodiments of the present invention, the method for extracting the canny edge contour feature vector of each frame image of the original ocean monocular vision in-situ video specifically includes the following steps: Convert the image to grayscale and use The convolution kernel performs Gaussian filtering to remove Gaussian noise and salt and pepper noise in the image; The Sobel operator is used to calculate the magnitude S and direction of the x-axis and y-axis gradients. , the calculation formulas are: ; ; in, Iis the image to be processed, S x is the transverse Sobel operator, S y is the longitudinal Sobel operator; Use non-maximum suppression to retain the maximum gradient in the area and suppress other extractions. After completing non-maximum suppression, a binary image is obtained, in which the grayscale values ​​of non-edge points are all 0, and the grayscale values ​​of points that may be edges are 255; Perform double threshold screening on the binary image, set the predetermined upper and lower thresholds, and identify the pixels in the image that are larger than the upper threshold as being the boundary, and the pixels in the image that are smaller than the lower threshold as not being the boundary. The pixels between the upper and lower thresholds are identified as candidates, and if they are connected to the boundary, they are retained, otherwise they are discarded.

[0016] In some embodiments of the present invention, step S13 specifically includes the following steps: Perform initialization operation, randomly select a feature vector as the initial cluster center, calculate the minimum distance between each unselected data point and the existing cluster center, and select the next cluster center according to the probability distribution of the distance; Assign data points and calculate each feature vector using cosine similarity B With cluster center A The similarity is calculated by , and it is assigned to the most similar cluster center; the calculation formula of cosine similarity is: ; in, A is the cluster center, B is the feature vector; Update the cluster center. For each cluster, calculate the average value of the data points in the cluster to obtain a new cluster center. The operations of allocating data points and updating cluster centers are repeated until a predetermined maximum number of iterations is reached.

[0017] The beneficial effects of the present invention are: 1. The present invention uses sine and cosine functions and 2D Ranchos interpolation method to position encode the extracted key frame images, thereby inputting the key frame images into the sea condition classification model for training at a higher resolution without losing the wave information in the image.

[0018] 2. The present invention applies the Vision Transformer network structure to the monocular vision sea condition classification task, and realizes the accurate classification of multi-level sea conditions by constructing a sea condition classification model based on the Vision Transformer network structure. Compared with the existing sea condition classification model, it significantly improves the classification speed while ensuring the classification accuracy; 3. The present invention extracts key frames based on HSV color histogram, Canny edge detection and clustering method, and establishes a multi-level sea condition classification data set based on monocular vision images for training the sea condition classification model, so that the images in the sea condition data set can cover sea surface conditions at different angles and different lighting conditions, thereby improving the applicability of the trained sea condition classification model in different time periods and realizing all-weather sea condition classification.

[0019] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a flow chart of a sea state classification method for offshore wind farms; Figure 2 An algorithm flow chart of a key frame extraction method provided by an embodiment of the present invention; Figure 3 A schematic diagram of an HSV transformation image of a key frame image provided by an embodiment of the present invention; Among them, a is the original ocean monocular vision in-situ image, and b is the image after HSV domain transformation; Figure 4 A color histogram corresponding to the HSV transformed image provided in an embodiment of the present invention; Among them, a is the hue histogram, b is the saturation histogram, and c is the brightness histogram; Figure 5 A schematic diagram of the network structure of the sea condition classification model provided by an embodiment of the present invention; Figure 6 A schematic diagram of a self-attention mechanism provided by an embodiment of the present invention; Figure 7 A schematic diagram of a 2D interpolation method provided by an embodiment of the present invention; Figure 8 A schematic diagram of the hardware topology of the offshore wind farm sea condition classification system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0025] The technical solution of the present invention is described in detail below in conjunction with specific embodiments and the accompanying drawings.

[0026] As attached Figure 1 -Attached Figure 8 As shown, in an illustrative embodiment of a method for classifying sea conditions in an offshore wind farm of the present invention, the method for classifying sea conditions includes the following steps.

[0027] S1: The original ocean monocular vision in-situ video data of the offshore wind farm is collected by a monocular vision camera installed on the offshore platform, and key frames are extracted from the video.

[0028] In some embodiments of the present invention, as shown in the attached Figure 2 As shown, the method for extracting key frames from a video specifically includes the following steps: S11: extracting the HSV histogram feature vector of each frame image of the original ocean monocular vision in-situ video, and extracting the canny edge contour feature vector of each frame image using the Canny edge detection algorithm.

[0029] S12: Concatenate the HSV histogram features and the canny edge contour features to obtain a fused feature vector.

[0030] S13: Setting a predetermined unit video duration; using the K-Means++ clustering method to classify each frame image within the unit video duration into a corresponding cluster at each predetermined unit video duration, and updating the cluster center of each cluster in real time.

[0031] S14: extracting the image frames closest to the respective cluster centers as the key frames of the respective clusters.

[0032] In some embodiments of the present invention, the method for extracting the HSV histogram feature vector of each frame image in the original ocean monocular vision in-situ video specifically includes the following steps: Convert the image from the RGB domain to the HSV domain and then draw its color histogram; where H represents hue, S represents saturation, and V represents brightness. In the HSV domain, the color component is represented by hue to make it more consistent with human perception of color. The color component includes red, green, and blue components, and the three RGB channels are compressed into one channel for color segmentation and brightness processing, making it less susceptible to brightness changes.

[0033] It should be noted that the color histogram is a global feature of an image. It provides a concise representation of the overall color characteristics of the image by counting the frequency of each color in the image. At night, the color histogram is converted into a grayscale histogram, which ensures that the key frame extraction of night videos can proceed normally.

[0034] In this embodiment, the HSV transformation result of the original ocean monocular vision in-situ image and its corresponding color histogram are shown in the attached figure. Figure 3 -Attached Figure 4 As shown, the color distribution in the image can be more clearly seen from the HSV image, and it has a stronger color contrast.

[0035] After obtaining the color histogram, the histograms of the three dimensions are normalized respectively and concatenated into the HSV histogram feature vector.

[0036] In actual operation, the calculation complexity needs to be simplified as much as possible while ensuring the hue information. Therefore, in this embodiment, the number of bins of the hue channel is reduced from 180 to 90, and the number of bins of the other two channels is reduced from 256 to 64, so as to obtain a feature vector with smaller dimension and easier to calculate.

[0037] In some embodiments of the present invention, the method for extracting the canny edge contour feature vector of each frame image of the original ocean monocular vision in-situ video specifically includes the following steps: Convert the image to grayscale and use The convolution kernel performs Gaussian filtering to remove Gaussian noise and salt and pepper noise in the image.

[0038] In image processing, the gradient can well reflect the change of pixels. The greater the gradient change, the greater the difference between adjacent pixels, which means there is an edge in the area. Therefore, the Sobel operator is used in the Canny edge detection algorithm to calculate the amplitude S and direction of the x-axis and y-axis gradients. , the calculation formulas are: ; ; in, I is the image to be processed, S x is the transverse Sobel operator, S y is the longitudinal Sobel operator; The Sobel edge obtained by the above calculation is still rough, and its edge is formed by the overlap of multiple edges. Therefore, non-maximum suppression is further used to retain the largest gradient in the area and suppress other extractions to achieve more accurate edge detection.

[0039] After completing non-maximum suppression, a binary image is obtained, in which the grayscale values ​​of non-edge points are all 0, and the grayscale values ​​of points that may be edges are 255.

[0040] Since the binary image obtained still contains many false edges caused by noise and other reasons, the binary image is further subjected to double threshold screening. The predetermined upper and lower thresholds are set, and the pixels in the image that are greater than the upper threshold are identified as boundaries, and the pixels in the image that are less than the lower threshold are identified as not boundaries. The pixels between the upper and lower thresholds are identified as candidates. If they are connected to the boundary, they are retained, otherwise they are discarded.

[0041] In the above exemplary embodiment, the color histogram and the Canny edge operator combine the global color information and local texture information of the current frame, and can perform a more comprehensive evaluation of the image frames in the original ocean monocular vision in-situ video.

[0042] In some embodiments of the present invention, the K-Means++ clustering method is used to extract key frames from the original ocean monocular vision in-situ video. It should be noted that clustering is an unsupervised learning method in the field of machine learning. It can mine the inherent laws and structures in the data without prior knowledge and labels, and divide the data into multiple categories. Objects in each category have high similarity, and objects in different categories show obvious differences.

[0043] Specifically, step S13 includes the following steps: Step 1: Perform initialization operation, randomly select a feature vector as the initial cluster center, calculate the minimum distance between each unselected data point and the existing cluster center, and select the next cluster center according to the probability distribution of the distance.

[0044] Step 2: Assign data points, use cosine similarity to calculate the similarity between each feature vector B and cluster center A, and assign it to the most similar cluster center; the calculation formula of cosine similarity is: ; Among them, A is the cluster center and B is the feature vector.

[0045] Step 3: Update the cluster center. For each cluster, calculate the average value of the data points in the cluster to obtain a new cluster center. Step 4: Iterate until the conditions are met, and repeat the operations of allocating data points and updating cluster centers, that is, repeat the above steps 2 and 3 until the predetermined maximum number of iterations is reached.

[0046] In this embodiment, the predetermined unit video duration is set to five minutes, three cluster centers are set during clustering, the fusion feature vectors of all frames are calculated, and the K-Means++ clustering algorithm is used to calculate the key frames within the unit video duration, that is, three key frames are extracted every five minutes in the original ocean monocular vision in-situ video. Therefore, from the 1272 videos with a duration of one hour acquired, a total of 45792 key frames are extracted as data samples. After data cleaning and deletion of meaningless key frames such as distortion, overexposure, underexposure, lens flare, etc., 42790 key frames remain for subsequent processing.

[0047] S2: The wave radar and wind direction and speed meter installed on the offshore platform are used to obtain the significant wave height and wind speed of the current sea area at the time of key frame image shooting, and then the key frames are labeled with sea state grades based on the significant wave height and wind speed to obtain a sea state dataset based on sea state grade classification. In the dataset, the resolution of each key frame image is .

[0048] In this embodiment, based on the wave radar and wind direction and anemometer data, the upper limit of the significant wave height is set to 293 cm, and the upper limit of the wind speed is set to 12.97 m / s, corresponding to the sixth level of sea conditions in the "Offshore Wind Force Level and Performance", to judge the significant wave height and wind speed in the current sea area, and use the wave height and wind speed to assess the sea condition for each key frame, respectively, and produce a sea condition classification data set containing sea condition images from level 1 to level 6.

[0049] In this embodiment, the specific method for determining the significant wave height and wind speed of the current sea area is: through the image recognition model, based on the sea surface features in the current key frame picture, the wind force level and performance table on the sea surface as shown in Table 1 is searched to obtain the wave height level and wind speed level. If the two levels are the same, the level is the sea condition level of the current key frame; if the two levels are different, manual judgment is performed according to the sea condition feature description of "Offshore Wind Force Level and Performance" based on the visual features of the current key frame.

[0050] Table 1 Wind force levels and performance on the sea surface

[0051] Before determining the significant wave height and wind speed of the current sea area based on the sea surface features in the key frame image, in order to solve the problem of imbalance of category samples, in some embodiments of the present invention, step S2 further includes the following steps: classifying the extracted key frames, and processing key frames of different categories by undersampling of majority class samples and data enhancement of minority class samples.

[0052] Among them, the purpose of undersampling the majority class samples is to reduce the number of majority class samples. Specifically, in this embodiment, the key frames with wind speed between 0-7.5 m / s and significant wave height between 0-100 cm are defined as majority class samples, and they are divided into three groups in equal proportions. Clustering operation is performed in each group, and the number of cluster centers in each group is 1 / 3 of the number of samples. In this way, the number of majority class samples can be reduced to one-fourth of the original number, totaling 14893 key frames. The use of clustering method can reduce the number of samples while minimizing information loss as much as possible.

[0053] The significance of data enhancement is to generate minority class samples to increase their proportion in the total samples. At the same time, data enhancement can also improve the diversity of training data, alleviate the overfitting of the network and improve the generalization ability of the network. In computer vision, image data enhancement mainly focuses on geometric transformation, color transformation, noise addition and blurring. However, for monocular ocean images, it is necessary to ensure that the features such as white crowns, highlight areas, wave shapes, crests, and sea surface colors in the image are complete. In this embodiment, six enhancement methods are selected from the 9 commonly used image data enhancement methods shown in Table 2, and each key frame of the minority class sample is randomly enhanced twice, expanding the sample to three times the original, for a total of 15796 key frames. The six selected enhancement methods are horizontal flipping, adding salt and pepper noise, image sharpening, perspective transformation, histogram equalization, and blurring.

[0054] Table 2 Data enhancement methods and effects

[0055] In this embodiment, among all 30689 key frames, the resolution of each key frame is , of which 70% are used as training sets, 20% as validation sets, and 10% as test sets. Among them, there are 6764 first-level sea conditions, accounting for 22.04%; 4145 second-level sea conditions, accounting for 13.51%; 7072 third-level sea conditions, accounting for 23.04%; 4387 fourth-level sea conditions, accounting for 14.30%; 3275 fifth-level sea conditions, accounting for 10.67%; and 5046 sixth-level sea conditions, accounting for 16.44%. The data samples of each level of sea conditions are basically balanced.

[0056] S3: Construct a sea state classification model ViT-L / 32 based on the Vision Transformer network; the structure of the sea state classification model is shown in the attached Figure 5 As shown, it includes an encoder and a multi-layer perceptron classification head.

[0057] It should be noted that the Vision Transformer (ViT) network is inspired by the Transformer deep learning model architecture. It splits images into patches and uses the linear embedding sequence of these patches as input to train the classification model in a supervised manner based on the self-attention mechanism. Compared with traditional convolutional neural networks, the self-attention mechanism can effectively capture the global contextual information in the image, which is very important for sea state classification because the characteristics of the marine environment are often global. Since the ViT network does not rely on convolution operations but splits the image into patches for processing, it can reduce the model's bias and increase sensitivity to different sea state characteristics.

[0058] The encoder consists of alternating multi-head self-attention layers and multi-layer perceptrons (MLPs), with layer normalization applied before the two layers and residual connections after the two layers. Layer normalization converts the activation values ​​of each layer into a distribution with a mean of 0 and a standard deviation of 1, and then scales and offsets the results to keep the input of each layer stable and alleviate the problem of gradient vanishing or gradient exploding. The calculation formula is: .

[0059] in, and are the mean and variance respectively, and are learnable scaling and offset parameters.

[0060] The core of the self-attention mechanism is to convert the two-dimensional feature input into a query, key, and value matrix, and then calculate the attention scores based on the matching of the key-value pairs, and weight the value matrix based on the weight matrix to finally obtain the attention-weighted output. The principle diagram is shown in the attached figure. Figure 6 As shown, the specific calculation formula is: ; .

[0061] in, Q h , K h , V h They are query matrix, key matrix, and value matrix respectively. The softmax activation function is used to normalize the weights and convert them into probability distribution, making the self-attention score more interpretable. d k is the dimension of the key matrix, divided by To prevent If the value of is too large, the output value of softmax will be too small, causing the model to fall into the problem of gradient disappearance.

[0062] This operation can enhance the stability of the gradient and the training effect of the model. Secondly, compared with RNN and convolution kernel, the self-attention mechanism can consider all input features, thereby increasing the model's receptive field for the two-dimensional features of the input at that moment.

[0063] The multi-head self-attention mechanism includes multiple self-attention layers: for the same input, after passing it to different self-attention layers, it undergoes h linear transformations to generate h sets of queries, keys, and values. The output matrices are concatenated together, and then passed to a linear layer to map the concatenated output to the final output space. The calculation formula is: .

[0064] in, , is a learnable weight matrix.

[0065] The multilayer perceptron contains two fully connected layers and is an inverted bottleneck structure. The first fully connected layer increases the feature dimension from D to 4D, and the second fully connected layer restores the dimension. In the middle, the Gaussian error linear unit (GeLU) is used as the activation function. Its approximate calculation formula is: .

[0066] The residual structure adds the input directly to the output of a certain layer, which is mainly used to alleviate the gradient vanishing problem in deep networks. In the Transformer encoder, both the multi-head self-attention layer and the feedforward network layer have residual connections, and the output calculation formula of the two layers is: .

[0067] in, X is the input feature, Y 1 is the output after being processed by the multi-head self-attention layer, Y 2 is the result of processing through the entire encoder layer.

[0068] S4: Preprocess the images in the sea state dataset to obtain a model input sequence consisting of an image block embedding sequence, a category vector, and a position encoding vector.

[0069] It should be noted that the resolution of the input image in the pre-training of the conventional Vision Transformer network model is , and the resolution of the key frame images in the dataset is , if we directly reduce the sea state monocular image to The size of the image will seriously affect the wave shape in the image, thus affecting the effect of sea condition classification. If the sea condition monocular image is directly input into the network, it will seriously affect the sea condition classification effect of the model. Therefore, the images in the dataset need to be preprocessed before sea condition classification.

[0070] In some embodiments of the present invention, the method of preprocessing the images in the data set in step S4 includes the following steps: S41: Adjust the image in the sea state data set to a predetermined size. In this embodiment, the image is proportionally reduced to a size of 1280×720.

[0071] S42: Perform image block embedding processing on the adjusted image, divide the image into multiple image blocks of fixed size, and generate an image block embedding sequence. In this embodiment, a block segmentation of 32×32 size is used, and the new block number calculation formula is: .

[0072] The original number of position codes, the number of position codes required for the input image .

[0073] S43: Set a learnable embedding vector as a category vector for classification.

[0074] S44: Use sine and cosine functions to perform two-dimensional position encoding on each image block to obtain an original two-dimensional position encoding vector; perform a 2D Ranchos interpolation operation on the original two-dimensional position encoding vector, calculate the position encoding of the interpolation position in each dimension, and merge the position encoding of all dimensions to obtain a complete position encoding vector.

[0075] S45: Concatenate the image patch embedding sequence, the category vector, and the complete position encoding vector to generate the model input sequence.

[0076] In some embodiments of the present invention, in step S44, the method of using sine and cosine functions to perform two-dimensional position encoding on each image block is: For any position in each image block (x,y) , the calculation formula for the position encoding of even dimensions is: ; ; The calculation formula for the position encoding of odd dimensions is: ; ; in, x and y are the horizontal and vertical coordinates in the current image block, respectively. PE(x,2i) For location x In the 2nd i Dimensional encoding, i and j For dimension-dependent indexes, D is the embedding dimension.

[0077] In some embodiments of the present invention, in step S44, as shown in the attached Figure 7 As shown, the method for performing a 2D Ranchos interpolation operation specifically includes the following steps: The original two-dimensional position code is obtained using the sin-cos encoding method.

[0078] The number of original two-dimensional position codes and new position codes is calculated to determine the spatial relationship.

[0079] The positional encoding of the interpolated position in each dimension is computed using 2D Ranchos interpolation.

[0080] Merge all dimension positional encodings and category vectors.

[0081] Among them, the formula for calculating the position encoding of the interpolation position in each dimension is: ; in, is the encoding result of the interpolation position at the new position, is the target pixel position in each dimension, is the original two-dimensional position encoding vector in The value at m and n is the pixel distance of the traversal Ranchos window, and They are x and y A two-dimensional scaling factor, i.e., a ratio of the width of the image adjusted to a predetermined size to the width of the segmented image block and a ratio of the height of the image adjusted to a predetermined size to the height of the segmented image block, is the Ranchos window parameter, which is used to control the interpolation range. is the Ranchos function; in, and The calculation formulas are ; ; in, W old and H old Respectively represent the number of original position encoding in width and height dimensions, and W new and H new They represent the number of positional encodings required for the input image in the width and height dimensions respectively; The calculation formula of the Ranchos function is: .

[0082] It should be noted that the input image size of the existing image classification model based on the Vision Transformer network structure is usually 224×224, and the segmented image block size is 32×32, so it can be divided into 49 image blocks, that is, a total of 49 original position codes. The sea condition image in this embodiment is 2560×1440. If it is directly reduced to 224×224, it will seriously damage the image ratio and affect the accuracy of sea condition classification. The use of 2D interpolation method in this application to expand the number of position codes can avoid distortion of wave morphology, better preserve wave morphology, and improve the accuracy of sea condition classification.

[0083] In the ViT-based sea state classification model, the accuracy of position encoding directly affects the model's understanding of the spatial relationship between image blocks. Compared with other interpolation methods, the 2D Ranchos method has higher interpolation accuracy, ensuring that the position information of each image block remains high-precision, thereby improving the performance of the model.

[0084] Furthermore, the original positional encoding is sin-cos Coding is a nonlinear coding, which may introduce discontinuity during discretization or interpolation, resulting in a sawtooth effect. In this embodiment, the 2D Ranchos interpolation method is used in conjunction with it. On the one hand, it has less distortion when processing nonlinear features, ensuring the accuracy of position coding, and on the other hand, it can effectively reduce the sawtooth effect and ensure the continuity and smoothness of position coding.

[0085] In order to further suppress the problem of imbalanced samples in different categories of data sets, in some embodiments of the present invention, the fine-tuning training process uses Focal Loss The loss function accelerates model convergence by reducing the weight of samples that are easy to classify and increasing the weight of samples that are difficult to classify. Focal Loss The expression of the loss function is: ; in, For the k The predicted probability of the class sample, the sum of the predicted probabilities of all classes is 1, For the real label One-Hot Code, that is, the actual category corresponding to , otherwise 0, For Category k The weight of is the focus factor, used to magnify role.

[0086] S5: Input the model input sequence into the sea state classification model to fine-tune the model. During the training process, the model input sequence is continuously forward-passed in the encoder, and all serial stacks constitute the encoder block to extract the features corresponding to the category vector for image classification. After obtaining the features, the features are further input into the multi-layer perceptron classification head to obtain the classification results of the category vector. During the training process, the model parameters are adjusted until the model converges, and the trained sea state classification model is obtained, and the model is used for sea state classification.

[0087] S6: Use the trained sea state classification model to classify sea conditions.

[0088] In some embodiments of the present invention, the following steps are further included: S7: Calculate the macro-average of the classification results of the sea state classification model; in the classification results, each classification category corresponds to a sea state level; the macro-average includes the arithmetic mean of the precision, recall and F1-Score of each category, and the macro-average calculation formulas are: ; ; .

[0089] in, is the arithmetic mean of the precision rate, is the arithmetic mean of the recall rate, It is the arithmetic mean of F1-Score.

[0090] Calculate the micro-average of the classification results of the sea state classification model; the micro-average includes the precision, recall and F1-Score calculated based on the sum of TP, FP and FN of each category. The micro-calculation formulas are: ; ; .

[0091] in, is the accuracy calculated based on the sum of TP, FP, and FN of each category, is the recall calculated based on the sum of TP, FP, and FN of each category, The F1-Score is calculated based on the sum of TP, FP, and FN of each category; TP (True Positive) is the number of samples correctly predicted as positive by the model, FP (False Positive) is the number of negative samples incorrectly predicted as positive by the model, and FN (False Negative) is the number of positive samples incorrectly predicted as negative by the model; l i The model predicts i The number of samples with correct predictions. m i The model predicts i The number of categories, n i is the number of items that actually belong to the i-th category; Calculate the classification accuracy of each category in the classification results of the model: ; Among them, TN is the number of samples correctly predicted by the model as negative class; The classification results of the trained sea state classification model were evaluated based on the macro-average, micro-average and classification accuracy of each category.

[0092] Specifically, when the macro average is much larger or smaller than the micro average, it indicates that the model is seriously unbalanced in classification. We need to introduce the weighted F1-Score to measure the balance of samples. The weighted F1-Score is the result of multiplying the F1-Score by the proportion of the class and then adding them together. Different weights are assigned according to the proportion of each class.

[0093] In this embodiment, in the training and verification task of the sea state classification model based on the Vision Transformer network, the number of training set samples is 21482, the number of verification set samples is 6138, and the number of test set samples is 3069. The data set covers monocular images under different shooting angles, different time periods, and different lighting conditions. The hyperparameters of the sea state classification model are summarized in Table 3.

[0094] Table 3 Hyperparameters of sea state classification task

[0095] Some embodiments of the present invention further provide an in-situ wave element extraction system, the hardware topology of which is as shown in the attached Figure 8 As shown, the system at least includes a monocular vision camera and a data processing terminal.

[0096] Among them, the monocular vision camera is installed on the offshore wind turbine platform to capture the monocular vision video of the in-situ impact waves of the offshore structure. It is connected to the burner and industrial computer and transmits data via Ethernet.

[0097] The data processing terminal includes a trained sea condition classification model, which communicates with a monocular vision camera to obtain the monocular video data shot by the camera, extracts key frames from the video data, and classifies the sea conditions of the key frame images.

[0098] In some embodiments of the present invention, the system further comprises a wave radar.

[0099] The wave radar is installed on the same offshore wind turbine platform as the monocular vision camera to obtain the significant wave height and spectral peak frequency in the sea area within the range of the monocular vision camera. It is connected to the data processing terminal via RS485 and transmits the data to the industrial computer via Ethernet.

[0100] The data processing terminal communicates with the wave radar to obtain the wave radar data.

[0101] In some embodiments of the present invention, the system further includes an industrial computer.

[0102] The industrial computer is connected to the monocular vision camera and wave radar, receives the data sent by the monocular vision camera and wave radar, controls the monocular vision camera and wave radar, and synchronizes the time for all connected devices to ensure the consistency of the acquired data in time. The industrial computer is also connected to the shore-based data center to transmit the acquired data back to the shore-based data center through the network.

[0103] In a specific embodiment of the present invention, a monocular vision camera is installed at the top of the outer platform of the column of the offshore wind turbine platform, with a height of 14.02m from the average water level, and a wave radar is installed at the bottom of the outer platform, with a height of 12.89m from the average water level.

[0104] The monocular vision camera comes from Hikvision, model is hemispherical camera DS-2DE2204IW-DE3 / W / XM, which can output 30fps video with a maximum resolution of 1920×1080. Its CMOS size is 1 / 1.28 inches, the focal length f is variable in the range of 2.8-12mm, the horizontal field of view (FOV) ranges from 25° to 100°, the vertical FOV ranges from 14.1° to 56.3°, and the diagonal FOV ranges from 28.7° to 114.7°. Its variable focal length and large FOV can better meet the needs of sea surface observation tasks in large scenes.

[0105] The wave radar is WaveGuide 5 Direction WG5-DR-CP produced by Radac of the Netherlands, with a sampling frequency of 10Hz. The measurement accuracy of the current sea surface elevation can reach ±3mm, and it can provide a wave height data every minute, with a wave height measurement accuracy of ±1cm, and the measurement accuracy of the wave propagation direction is ±2°.

[0106] Finally, it should be noted that: the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0107] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A method for classifying sea conditions of an offshore wind farm, characterized in that: The following steps are involved: S1: Obtain original ocean monocular vision in-situ video data, and extract key frames from the video; S2: Obtain the significant wave height and wind speed of the current sea area when the key frame image is taken, and mark the key frame with the sea state level based on the significant wave height and wind speed to obtain a sea state data set based on sea state level classification; S3: constructing a sea state classification model based on the Vision Transformer network; the sea state classification model includes an encoder and a multi-layer perceptron classification head; S4: preprocessing the images in the sea state dataset to obtain a model input sequence including an image block embedding sequence, a category vector, and a position encoding vector; S5: inputting the model input sequence into the sea condition classification model to fine-tune the model, so that the model input sequence is continuously forward-transferred in the encoder to extract features corresponding to the category vector therein, and inputting the features into the multi-layer perceptron classification head to obtain the classification result of the category vector, adjusting the model parameters during the training process until the model converges, and obtaining the trained sea condition classification model; S6: Use the trained sea state classification model to classify sea conditions.

2. The offshore wind farm sea condition classification method according to claim 1, characterized in that: The method for preprocessing the images in the data set in step S4 comprises the following steps: S41: adjusting the image in the sea state data set to a predetermined size; S42: performing image block embedding processing on the adjusted image, dividing the image into a plurality of image blocks of fixed sizes, and generating an image block embedding sequence; S43: setting a learnable embedding vector as a category vector for classification; S44: performing two-dimensional position encoding on each image block using sine and cosine functions to obtain an original two-dimensional position encoding vector; performing a 2D Ranchos interpolation operation on the original two-dimensional position encoding vector, calculating the position encoding of the interpolation position in each dimension, and combining the position encodings of all dimensions to obtain a complete position encoding vector; S45: Connect the image block embedding sequence, the category vector and the complete position encoding vector to generate a model input sequence.

3. The offshore wind farm sea condition classification method according to claim 1, characterized in that: In step S44, the method of using sine and cosine functions to perform two-dimensional position encoding on each image block is: For any position in each image block (x,y) , the calculation formula for the position encoding of even dimensions is: ； ； The calculation formula for the position encoding of odd dimensions is; ； ； in, x and y are the horizontal and vertical coordinates in the current image block, respectively. PE(x,2i) For location x In the 2nd i Dimensional encoding, i and j is the dimension-dependent index and D is the dimension of the embedding.

4. The offshore wind farm sea condition classification method according to claim 3, characterized in that: The formula for calculating the positional encoding of the interpolation position in each dimension is: ； in, is the encoding result of the interpolation position at the new position, is the target pixel position in each dimension, is the original two-dimensional position encoding vector in The value at m and n is the pixel distance of the traversal Ranchos window, and They are x and y Scaling factor in two dimensions, is the Ranchos window parameter, which is used to control the interpolation range. is the Ranchos function; in, and The calculation formulas are: ； ； in, W old and H old are the numbers of the original two-dimensional position encoding in the width and height dimensions, respectively. W new and H new The number of positions required for the model input image to be encoded in the width and height dimensions respectively; The calculation formula of the Ranchos function is: 。 5. The method for classifying sea conditions of an offshore wind farm according to claim 1, characterized in that: The fine-tuning training process uses Focal Loss The loss function accelerates model convergence. Focal Loss The expression of the loss function is: ； in, For the k The predicted probability of the class sample, the sum of the predicted probabilities of all classes is 1, is the One-Hot encoding of the true label, that is, the corresponding , otherwise 0, For Category k The weight of is the focus factor, used to magnify role.

6. The method for classifying sea conditions of an offshore wind farm according to claim 1, characterized in that: Further comprising the steps of: Calculate the macro-average value of the classification results of the sea state classification model; in the classification results, each classification category corresponds to a sea state level; the macro-average value includes the arithmetic mean of the precision rate, recall rate and F1-Score of each category, and the macro-average value calculation formulas are: ； ； ； in, is the arithmetic mean of the precision rate, is the arithmetic mean of the recall rate, is the arithmetic mean of F1-Score; Calculate the micro-average of the classification results of the sea state classification model; the micro-average includes the precision, recall and F1-Score calculated based on the sum of TP, FP and FN of each category, and the micro-calculation formulas are: ； ； ； in, is the accuracy calculated based on the sum of TP, FP, and FN of each category, is the recall calculated based on the sum of TP, FP, and FN of each category, The F1-Score is calculated based on the sum of TP, FP, and FN for each category; TP is the number of samples correctly predicted as positive by the model, FP is the number of negative samples incorrectly predicted as positive by the model, and FN is the number of positive samples incorrectly predicted as negative by the model; l i The model predicts i The number of samples with correct predictions. m i The model predicts i The number of categories, n i is the number of items that actually belong to the i-th category; Calculate the classification accuracy of each category in the classification results of the model: ； Among them, TN is the number of samples correctly predicted by the model as negative class; The classification result of the trained sea state classification model is evaluated based on the macro average value, the micro average value and the classification accuracy of each category.

7. The method for classifying sea conditions of an offshore wind farm according to claim 1, characterized in that: The method for extracting key frames from the video specifically comprises the following steps: S11: extracting the HSV histogram feature vector and the canny edge contour feature vector of each frame image of the original ocean monocular vision in-situ video; S12: concatenate the HSV histogram feature and the canny edge contour feature to obtain a fused feature vector; S13: setting a predetermined unit video duration; using an adaptive mean clustering method to sequentially classify each frame image within the unit video duration into a corresponding cluster at each predetermined unit video duration, and updating the cluster center of each cluster in real time; S14: extracting the image frames closest to the respective cluster centers as the key frames of the respective clusters.

8. The method for classifying sea conditions of an offshore wind farm according to claim 7, characterized in that: The method for extracting the HSV histogram feature vector of each frame image of the original ocean monocular vision in-situ video specifically includes the following steps: Convert the image from the RGB domain to the HSV domain and then draw its color histogram; where H represents hue, S represents saturation, and V represents brightness; in the HSV domain, the color component is represented by hue, and the three RGB channels are compressed into one channel to perform color segmentation and brightness processing; After obtaining the color histogram, the histograms of the three dimensions are normalized respectively and concatenated into the HSV histogram feature vector.

9. The method for classifying sea conditions of an offshore wind farm according to claim 7, characterized in that: The method for extracting the canny edge contour feature vector of each frame image of the original ocean monocular vision in-situ video specifically comprises the following steps: Convert the image to grayscale and use The convolution kernel performs Gaussian filtering to remove Gaussian noise and salt and pepper noise in the image; The Sobel operator is used to calculate the magnitude S and direction of the x-axis and y-axis gradients. , the calculation formulas are: ； ； in, I is the image to be processed, S x is the transverse Sobel operator, S y is the longitudinal Sobel operator; Use non-maximum suppression to retain the maximum gradient in the area and suppress other extractions. After completing non-maximum suppression, a binary image is obtained, in which the grayscale values ​​of non-edge points are all 0, and the grayscale values ​​of points that may be edges are 255; The binary image is subjected to double threshold screening, and a predetermined upper threshold and a lower threshold are set. Pixels in the image that are larger than the upper threshold are identified as being boundaries, and pixels in the image that are smaller than the lower threshold are identified as not being boundaries. Pixels between the upper threshold and the lower threshold are identified as candidates. If they are connected to the boundary, they are retained, otherwise they are discarded.

10. The method for classifying sea conditions of an offshore wind farm according to claim 7, characterized in that: The step S13 specifically includes the following steps: Perform initialization operation, randomly select a feature vector as the initial cluster center, calculate the minimum distance between each unselected data point and the existing cluster center, and select the next cluster center according to the probability distribution of the distance; Assign data points and calculate each feature vector using cosine similarity B With cluster center A and assign it to the most similar cluster center; the calculation formula of the cosine similarity is: ;in, A is the cluster center, B is the feature vector; Update the cluster center. For each cluster, calculate the average value of the data points in the cluster to obtain a new cluster center. The operations of allocating data points and updating cluster centers are repeated until a predetermined maximum number of iterations is reached.

Images