Near-shore wave breaking type classification method based on video image deep learning

Abstract

The invention discloses a near-shore wave breaking type classification method based on video image deep learning. The method comprises the steps that video image data of a near-shore area are collected based on fixed or temporary video collection equipment, and the visual angle of the video image data covers the high position of a near-shore wave breaking zone; preprocessing the video image data, and marking overflow waves, curled waves and collapse waves according to the data; constructing a training set based on the marked data, and training a neural network model based on the training set; and based on the trained neural network model, carrying out feature extraction and time sequence modeling on the coastal wave image sequence, and predicting a wave breaking type corresponding to the coastal wave image sequence. According to the method, the time sequence video data is collected, and the deep learning model is constructed for training and reasoning, so that wave breaking type identification is automatically completed. The method is efficient and reliable, can adapt to various sea conditions, and has high practical value in practical application.

Description

Technical Field

[0001] The present invention relates to the field of ocean observation technology, and in particular to a method for classifying nearshore wave breaking types based on deep learning of video images. Background Art

[0002] Wave breaking refers to the process in which waves collapse and generate foam, splashes, and intense turbulence when they propagate to shallow nearshore waters under the combined effects of factors such as seabed topography, wind force, and wave steepness ratio. As the core mechanism for nearshore wave energy dissipation, it plays an important role in driving the evolution of nearshore topography and landforms. The type of wave breaking (referred to as wave breaking type) is one of the key characteristics of the wave breaking process. Different types of wave breaking not only reflect the pattern of wave energy release, but also directly affect the nearshore hydrodynamic environment and landform changes. Therefore, achieving real-time and efficient wave breaking type identification is of great significance for coastal monitoring, wave dynamics research, and nearshore landform evolution analysis.

[0003] Traditionally, the classification of wave breaking types has relied on empirical formulas and extensive field observations, which have been unable to meet the demand for high-resolution spatial and temporal observations of coastal wave breaking processes. Numerous observations and laboratory simulations have shown that wave breaking type is primarily controlled by nearshore slope (tanβ) and wave steepness (H / L). This relationship can be expressed by calculating the Iribarren number (ξ):

[0004]

[0005]

[0006] Here, tanβ represents the beach slope, H is the wave height, and L is the wavelength. The subscript o represents offshore conditions, and the subscript b represents the breakpoint. The ξ value can be used to classify wave breaking types and is closely related to beach conditions. However, its calculation relies on intensive observations of environmental parameters such as slope and waves in the study area. Moreover, in areas with complex topography (such as sandbars and beach corners), ξ may not be an ideal predictor of wave breaking type, and a comprehensive analysis often requires integration with other dynamic geomorphological parameters.

[0007] With the rapid development of remote sensing and computer vision technologies, coastal dynamic geomorphology monitoring based on optical imaging has become an efficient, non-contact observation method, showing broad potential for wave feature extraction. Continuous multi-frame time-series video images can more comprehensively capture the spatiotemporal evolution of wave breaking processes.

[0008] Traditional methods rely on manual observation, which can lead to inefficiencies and insufficient precision. Using video imaging technology to accurately and automatically classify wave breaking types has become a pressing technical challenge. This problem involves multiple aspects: First, the nearshore environment is complex and dynamic, and wave breaking morphologies vary significantly, necessitating the construction of a representative video dataset. Second, wave breaking is a dynamic process, and extracting a single spatial feature cannot accurately capture its temporal variations. Furthermore, the varying topographical conditions and sea conditions in different sea areas pose challenges to the generalization of the model. Furthermore, the sheer volume of real-time video data poses a significant challenge in achieving fast and accurate classification inference within limited computing resources. These interrelated sub-problems collectively constitute the technical challenge of automatically identifying nearshore wave breaking types. Developing an efficient, accurate, and adaptable automatic identification method to meet the needs of real-time monitoring and early warning for coastal engineering projects remains a major challenge in this field. Summary of the Invention

[0009] To this end, this paper proposes a nearshore wave breaking type classification method based on deep learning of video imagery. By collecting time-series video data, a deep learning model is constructed for training and inference, automatically identifying wave breaking types. This method is efficient and reliable, adaptable to a variety of sea conditions, and possesses high practical value in practical applications.

[0010] According to one aspect of the present invention, a method for classifying nearshore wave breaking types based on deep learning of video images is provided, the method comprising:

[0011] Using fixed or temporary video acquisition equipment, collect video image data of the nearshore area, where the video image data covers a high position in the nearshore wave-breaking zone;

[0012] Preprocessing the video image data, and marking overflow waves, curl waves and collapse waves according to the data;

[0013] Constructing a training set based on the labeled data, and training a neural network model based on the training set;

[0014] Based on the trained neural network model, feature extraction and time series modeling are performed on the nearshore wave image sequence to predict the wave breaking type corresponding to the nearshore wave image sequence.

[0015] In the above-mentioned technical solution, this method performs a series of processing and modeling operations on the collected nearshore video images to efficiently distinguish the three common wave breaking types: overflow waves, roll waves and collapse waves. This method is of great significance to many fields such as marine research and coastal management, and can provide more accurate wave breaking information.

[0016] In this case, both fixed and temporary video capture equipment was used to acquire nearshore video images, maximizing the ability to capture the complete and representative wave breaking process and related characteristics. Fixed equipment provides long-term, continuous monitoring of specific nearshore areas, acquiring a large amount of stable data that reflects the long-term patterns of wave breaking in that area. Temporary equipment offers flexibility and can be quickly deployed at various nearshore locations to capture specific scene data based on research needs.

[0017] The neural network model is trained on a high-quality, annotated training set. Leveraging deep learning algorithms, the neural network automatically learns the characteristic patterns of different wave breaking types from this large amount of annotated data. During training, the model continuously adjusts its internal parameters to minimize the error between the predicted wave breaking type and the actual annotated type, gradually acquiring the ability to accurately distinguish between the three wave breaking types.

[0018] Unlike traditional methods of visually determining wave breaking types, automated classification using a deep learning model avoids subjective differences caused by human factors and can more objectively process and classify large amounts of wave breaking data. Once the model is trained, it can quickly classify new nearshore wave imagery. As new data continues to be acquired to expand the training set, the model has the potential for further optimization and improvement, adapting to the classification needs of nearshore wave breaking in different regions and at different times.

[0019] In some embodiments, pre-processing the video image data and labeling overflow, curl, and collapse waves according to the data include:

[0020] The video image data is divided into multiple video slices with each segment being 1 to 30 seconds; the segmented time segments are cropped to focus on the wave breaking area to obtain cropped video slices;

[0021] Annotating the video slices and classifying wave breaking types into overflow waves, curl waves, and collapse waves based on visual features;

[0022] The annotated video slices are subjected to frame extraction processing to generate image sequences suitable for model input.

[0023] In this technical solution, the captured nearshore video images are segmented into segments of 1 to 30 seconds in length, enabling detailed processing and analysis of the wave breaking process. This 1-30 second duration not only covers the key wave breaking processes but also effectively avoids the additional computational effort and data storage pressure associated with longer durations. This allows for the complete cycle of a specific wave breaking event, from wave formation to breaking, to be better utilized for subsequent wave breaking feature extraction and modeling.

[0024] Furthermore, the segmented time segments are cropped to precisely focus on the wave breaking area, removing irrelevant background information and interference factors in the video, such as beaches and coastal facilities. This not only improves annotation efficiency and accuracy, allowing annotators to more closely observe the key features of wave breaking, but also makes the cropped video slices more compatible with subsequent model input requirements, reducing the model's focus on irrelevant information and ultimately improving the model's learning of wave breaking features.

[0025] In addition, the annotated video slices are subjected to frame extraction to generate an image sequence that meets the model input requirements. Since deep learning models usually require input of fixed size and frame rate, the frame extraction operation can unify video slices of different lengths and frame rates into a format that meets the model requirements. At the same time, frame extraction can reduce the amount of data, simplify computational complexity, and improve the efficiency of model training and inference. The keyframe-based frame extraction method can better highlight the critical moment of wave breaking and help the model focus on the key features of wave breaking. In actual operation, the appropriate frame extraction strategy can be flexibly selected according to the model requirements and the characteristics of wave breaking.

[0026] In some embodiments, pre-processing the video image data and labeling overflow, curl, and collapse waves according to the data include:

[0027] The video image data is segmented into multiple video slices with each segment ranging from 1 to 30 seconds. The segmented time segments are cropped at multiple scales to retain the small-scale foam details and medium- and large-scale wave morphology in the wave breaking area, generating multi-scale video slices.

[0028] Perform feature enhancement on multi-scale video slices to generate feature-enhanced video slices;

[0029] Annotate feature-enhanced video slices to generate annotated wave-breaking images;

[0030] The annotated video slices are subjected to frame extraction processing to generate image sequences suitable for model input.

[0031] In the above technical solution, the segmented time segments are cropped at multiple scales, preserving small-scale foam details and medium- and large-scale wave morphology to comprehensively capture the complete characteristics of wave breaking at different scales. Small-scale foam details can reflect the microscopic movement and energy dissipation of the water body during wave breaking, which is important for distinguishing different types of breaking. Medium- and large-scale wave morphology provides information on the overall propagation and breaking trends of waves, which helps to understand the macroscopic process of wave breaking. The video slices generated by multi-scale cropping can more comprehensively reflect the characteristics of wave breaking, providing a richer data foundation for subsequent feature enhancement and model training. Implementing multi-scale cropping requires determining the appropriate scale range and cropping strategy. On the one hand, an image pyramid-based method can be used to generate image sequences of different resolutions and then crop them at each scale. On the other hand, a sliding window technique can be used to move the window at different scales to capture the key areas of wave breaking.

[0032] The purpose of feature enhancement is to improve the recognizability of wave breaking features, making different types of wave breaking more visually distinct and facilitating subsequent annotation and model learning. Common feature enhancement methods include contrast enhancement, edge detection, and filtering. Contrast enhancement can highlight the difference between the wave breaking area and the surrounding background, making the outline of the wave breaking clearer; edge detection can emphasize the moving boundaries of the water body during wave breaking, helping to capture the key characteristics of wave breaking; and filtering techniques can remove noise interference and enhance the stability of wave breaking features.

[0033] In some embodiments, performing feature enhancement on multi-scale video slices to generate feature-enhanced video slices includes:

[0034] The feature enhancement of multi-scale video slices is performed. The Canny edge detection operator is used to strengthen the peak contour, the brightness gain coefficient is set to highlight the foam contrast, the local binary pattern texture feature is extracted to enhance the dynamic texture, and the feature-enhanced multi-scale video slices are generated.

[0035] In this technical solution, the Canny edge detection operator, used during multi-scale cropping of the segmented time segments, accurately locates the edges of wave crests in wave-breaking images. This operator calculates the image gradient magnitude and direction, identifies edges based on gradient changes, and outlines the wave crests as they break, making them prominent and facilitating subsequent identification. It also suppresses noise, reducing the interference of meaningless noise points generated by wave motion, resulting in a clear and coherent crest contour curve, improving feature enhancement accuracy.

[0036] By setting the brightness gain coefficient, the brightness of the foam area can be increased to make it contrast sharply with dark backgrounds such as sea water and beaches, helping labelers to accurately identify the location and range of the foam and providing a clear data basis for the model to learn foam characteristics.

[0037] The Local Binary Pattern (LBP) operator effectively captures local texture features in the wave-breaking region, such as foam graininess and ripples on the wave surface. These features reflect the water's motion and energy dissipation, and are crucial for distinguishing different types of wave-breaking. Extracting and enhancing these texture features provides the model with more detailed information, improving classification performance. The LBP algorithm is computationally simple and efficient, making it suitable for processing large amounts of data from multi-scale video slices. It can quickly extract and enhance texture features without increasing the computational burden of the preprocessing process, ensuring the feasibility and practicality of the method.

[0038] In some embodiments, overflow waves, roll waves, and collapse waves are labeled according to the data, including:

[0039] The type of wave breaking is determined based on the significant visual features in the slices. If it shows gentle flow and low wave crest, it is marked as overflow wave. If it shows tubular curling and white foam, it is marked as curl wave. If it shows splashing and breaking, it is marked as collapse wave.

[0040] In the above technical solution, when annotating video slices, waves exhibiting gentle flow and low crests are labeled as overflow waves. In this case, the wave energy is low, and it propagates slowly along the shoreline upon breaking. The crest is low and gentle, with no noticeable curling or collapse. For example, when a wave breaks on a gently sloping coast in shallow water, the annotator must carefully observe the wave shape and flow characteristics to determine whether it is an overflow wave. If the wave front forms a curled tubular structure that gradually closes during propulsion and produces a large amount of white foam, it is labeled as a curl wave. Curl waves exhibit high energy upon breaking and are commonly seen on sandy beaches with moderate seabed slopes. A barrel wave is a typical example of a curl wave. Annotators should focus on the curling shape of the wave front and the distribution of foam generated. If the wave top suddenly collapses upon breaking, dumping toward the shore with strong splashing and water dispersion, it is labeled as a collapse wave. Collapse waves typically occur on steeply sloping coasts, where the wave energy is rapidly released upon encountering the steep slope. Annotators must observe the instantaneous water movement and the shape of the waves after breaking to determine whether they are collapse waves.

[0041] In some embodiments, when the segmented time segments are cropped, the time interval between wave-breaking slices at similar positions exceeds 1 minute.

[0042] In this technical solution, given the complexity and variability of the ocean environment, wave breaking at similar locations can exhibit different characteristics at different times due to factors such as tides, winds, and currents. Interval selection avoids the sample bias caused by selecting only continuous slices over a short period of time, enabling the model to learn the characteristics of wave breaking under different conditions, thereby improving its adaptability in practical applications.

[0043] If the time interval between wave-breaking slices at similar locations is too short, the wave breaking characteristics of adjacent slices may be similar, resulting in a limited feature pattern learned by the model. Setting a time interval of more than one minute can reduce the correlation between samples, allowing each sample to independently reflect the wave breaking characteristics at different moments. This helps the model more accurately learn the essential characteristics of different wave breaking types, thereby improving its generalization performance.

[0044] The increased time interval makes the visual characteristics of different wave-breaking slices more distinct, allowing annotators to more easily identify wave types based on prominent visual features. For example, the smooth flow and low crest of overflow waves, the tubular curl and white foam of curl waves, and the splashing breakup of break waves become more prominent and easier to distinguish. This reduces feature ambiguity and annotation ambiguity caused by the continuity of wave breaking, reduces the difficulty and workload of annotation, and improves annotation accuracy and efficiency.

[0045] Because the features of slices at different times are relatively independent and distinct, annotators can make judgments based on a unified annotation standard when annotating different slices at similar locations. This reduces subjective judgment differences that may arise from temporal continuity and helps improve the consistency of annotation results. It also makes it easier to review and verify subsequent annotation results, enabling more accurate detection and correction of annotation errors and ensuring the quality of the annotated data.

[0046] In some embodiments, based on the trained neural network model, feature extraction and time series modeling are performed on the nearshore wave image sequence to predict the wave breaking type corresponding to the nearshore wave image sequence, including:

[0047] Using a pre-trained convolutional network as a feature extractor to encode the image sequence;

[0048] The encoded sequence is decoded through the long short-term memory network to generate a temporal feature representation;

[0049] A classification label of the wave breaking type is determined according to the time series feature representation.

[0050] In the above technical solution, when processing nearshore wave image sequence data, using a pre-trained convolutional network (CNN) as a feature extractor is an efficient and effective method. The rich feature hierarchy structure obtained by pre-training the convolutional network on large-scale image data can efficiently encode the spatial features in the wave breaking image. Through multi-layer convolution and pooling operations, CNN automatically learns different levels of features including edges, textures, shapes, etc., and can identify key features such as crest contours, foam distribution, and wave morphology in wave breaking images. After encoding processing, each frame of the image is converted into a feature vector of fixed length. These feature vectors not only retain important information related to the wave breaking type in the original image, but also reduce the data dimension and redundancy, thereby improving the efficiency of subsequent processing.

[0051] To further capture the temporal characteristics of the wave breaking process, a long short-term memory (LSTM) network is used to model the encoded feature sequence. LSTM is a neural network structure specifically designed for processing sequential data, effectively capturing long-term dependencies in the data. In the wave breaking type classification task, wave breaking is a temporally continuous process, and the wave breaking features between previous and subsequent frames are correlated. Utilizing its internal gating mechanism (input gate, forget gate, and output gate), LSTM selectively retains and updates important temporal features related to the wave breaking type, while forgetting irrelevant or redundant information. In this way, LSTM generates a temporal feature representation that reflects the dynamics of the wave breaking process.

[0052] According to another aspect of the present invention, a nearshore wave breaking type classification system based on deep learning of video images is provided, based on the above method, comprising:

[0053] Acquisition module: used to collect video image data of the nearshore area based on fixed or temporary video acquisition equipment, and the video image data covers the high position of the nearshore wave zone;

[0054] Preprocessing module: used for preprocessing the video image data and marking overflow wave, curl wave and collapse wave according to the data;

[0055] Training module: used to construct a training set based on the labeled data and train a neural network model based on the training set;

[0056] Prediction module: used to perform feature extraction and time series modeling on the nearshore wave image sequence based on the trained neural network model, and predict the wave breaking type corresponding to the nearshore wave image sequence.

[0057] In the above technical solution, in order to better use the above method, this application proposes a nearshore wave breaking type classification system based on deep learning of video images. Each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0058] According to another aspect of the present invention, a device for classifying nearshore wave breaking types based on deep learning of video images is provided, comprising:

[0059] at least one processor and a memory communicatively coupled to the at least one processor;

[0060] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0061] In the above technical solution, in order to better run and process the method, the above method is stored in a memory and a processor is used to execute the stored method. It should be noted that the principle and effect of each step have been described above and will not be further explained here.

[0062] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0063] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium and implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be further explained here. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 This is a flow chart of an embodiment of a method for classifying nearshore wave breaking types based on deep learning of video images according to the present invention;

[0066] Figure 2 This is a flow chart of another embodiment of a method for classifying nearshore wave breaking types based on deep learning of video images according to the present invention;

[0067] Figure 3Schematic diagram of video image acquisition equipment according to another embodiment of a method for classifying nearshore wave breaking types based on deep learning of video images of the present invention;

[0068] Figure 4 This is a schematic diagram of a data set preparation process for another embodiment of a method for classifying nearshore wave breaking types based on deep learning of video images of the present invention;

[0069] Figure 5 This is a schematic diagram of a CRNN deep learning network architecture of another embodiment of a method for classifying nearshore wave breaking types based on deep learning of video images of the present invention;

[0070] Figure 6 This is a schematic diagram of the training results of a video image wave breaking classification task of a CRNN model according to another embodiment of a method for classifying nearshore wave breaking types based on deep learning of video images of the present invention;

[0071] Figure 7 This is a structural diagram of an embodiment of a nearshore wave breaking type classification system based on deep learning of video images of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It is particularly noted that the following examples are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. Similarly, the following examples are only some embodiments of the present invention and are not intended to be exhaustive. All other embodiments obtained by those of ordinary skill in the art without creative effort are intended to fall within the scope of protection of the present invention.

[0073] To this end, this paper proposes a nearshore wave breaking type classification method based on deep learning of video imagery. By collecting time-series video data, a deep learning model is constructed for training and inference, automatically identifying wave breaking types. This method is efficient and reliable, adaptable to a variety of sea conditions, and possesses high practical value in practical applications.

[0074] Example 1

[0075] See also Figure 1 A nearshore wave breaking type classification method based on deep learning of video images, the method includes:

[0076] S1. Using fixed or temporary video acquisition equipment, collect video image data of the nearshore area, with the video image data covering the high position of the nearshore wave zone;

[0077] In this embodiment, nearshore video image data is acquired. This video image data is captured by fixed or mobile video capture equipment deployed in the target coastal area, covering the nearshore breakwater zone and capturing dynamic information about the wave breaking process. Fixed or mobile video capture equipment deployed in the target coastal area captures video image data of the nearshore breakwater zone, generating a high-resolution video sequence. Data preprocessing is performed on the captured video image data, using a pixel resolution of at least 1920×1080 and a frame rate of at least 10 frames per second, to generate a standardized video frame sequence.

[0078] For example, fixed or mobile video capture equipment deployed in the target coastal area captures video data of the nearshore breaking zone. A camera with a pixel resolution of 1920×1080 and a frame rate of 30 frames per second is used to ensure high definition and continuity of the video sequence. The captured video data is preprocessed using the OpenCV library to generate a video frame sequence that meets the input requirements of the deep learning model. If the video frame sequence contains redundant background information, the U-Net image segmentation algorithm is used to isolate the nearshore breaking zone and extract video clips focusing on the wave breaking process.

[0079] S2. Preprocess the video image data and mark overflow waves, curl waves and collapse waves according to the data;

[0080] In this embodiment, S2, pre-processing the video image data and marking overflow, curl and collapse waves according to the data, including:

[0081] S21, dividing the video image data into a plurality of video slices, each of which is 1 to 30 seconds; cropping the segmented time segments, focusing on the wave breaking area, to obtain cropped video slices;

[0082] Exemplarily, a fixed or temporary video capture device captures nearshore wave video data at a resolution of 1920×1080 at a frame rate of at least 10 frames per second, generating a raw video stream. Using a sliding window-based video segmentation technique, the video stream is segmented into multiple segments with a 5-second time window, ensuring that each segment contains at least one complete wave breaking cycle. The inter-frame difference algorithm in the OpenCV library is used to calculate the pixel differences between adjacent frames. The wave breaking start frame is determined when the difference value for 10 consecutive frames exceeds a threshold of 50, and the wave breaking end frame is determined when the difference value falls below the threshold for 5 consecutive frames. Based on the start and end frame markers, the video segment is precisely cropped using the FFmpeg tool, retaining the critical 20-30 frames of the breaking process. If a time segment contains a wave breaking region, the YOLOv5 object detection model is used to identify the wave breaking region and determine its bounding box coordinates. Regions are cropped for each time segment using the bounding box coordinates, and the OpenCV library is used to extract the image content focused on the wave breaking region. The resolution was adjusted to 224 × 224 pixels according to the cropped image content, and the bilinear interpolation algorithm was used to obtain standardized wave-breaking image slices.

[0083] S22, labeling the video slices and classifying the wave breaking types into overflow waves, curl waves, and collapse waves based on visual features;

[0084] In this embodiment, overflow waves, curl waves and collapse waves are marked respectively according to the data, including: judging the wave breaking type according to the significant visual features in the slice, if it shows gentle flow and low wave crest, it is marked as overflow wave, if it shows tubular curling and white foam, it is marked as curl wave, and if it shows splashing and breaking, it is marked as collapse wave.

[0085] Specifically, a spilling breaker has a gradually steepening crest, with white foam forming on top and energy dissipating slowly. This is common on gently sloping seafloors. A plunging breaker has a crest that tilts forward, forming a curved cavity, accompanied by a dramatic release of energy. This occurs on moderately sloping seafloors. A surging breaker has a wave that crashes directly into the shore with little noticeable curling, concentrating its energy in a narrow area. This is common on steep beaches.

[0086] S23. Perform frame extraction on the annotated video slices to generate an image sequence suitable for model input.

[0087] As an optional embodiment, S2, pre-processing the video image data, and marking overflow, curl, and collapse waves according to the data, including:

[0088] S21, segmenting the video image data into multiple video slices with each segment being 1 to 30 seconds, performing multi-scale cropping on the segmented time segments, retaining small-scale foam details and medium- and large-scale wave morphologies in the wave breaking area, and generating multi-scale video slices;

[0089] In this embodiment, the original video stream data is segmented to generate 30-second video clips. A 224×224 pixel window is used, combined with multi-scale cropping, with cropping scales of 1x, 0.5x, and 0.25x, and a window sliding step of 112 pixels. Small-scale foam details and medium- and large-scale wave morphologies are retained in the wave-breaking region to generate a multi-scale wave-breaking image. A 224×224 pixel window is used to crop the segmented 30-second video clips, with cropping scales of 1x, 0.5x, and 0.25x, and a window sliding step of 112 pixels. Multi-scale cropping is performed on the wave-breaking region. If the cropped region contains foam details, the small-scale cropping result is retained; if the cropped region contains wave morphologies, the medium- and large-scale cropping result is retained. A multi-scale wave-breaking image is generated, ensuring that small-scale foam details and medium- and large-scale wave morphologies are fully retained.

[0090] For example, the original video stream is first segmented in the temporal dimension. Using a video stream with a fixed frame rate of 30fps, FFmpeg's segment command is used to generate 30-second segments at intervals of 900 frames, ensuring that the temporal accuracy of each segment is controlled within ±0.03 seconds. In the spatial processing stage, after reading frame by frame using OpenCV's VideoCapture, Gaussian pyramid downsampling is first performed to generate a multi-scale image group: three scale layers with the original resolution of 224×224 (1x), 112×112 (0.5x), and 56×56 (0.25x). The sliding window operation adopts a double loop structure. The outer loop traverses the original scale image with a step size of 112 pixels, and the inner loop synchronously processes the corresponding areas of the three scale layers, maintaining geometric consistency through bicubic interpolation.

[0091] S22, performing feature enhancement on the multi-scale video slices to generate feature-enhanced video slices;

[0092] In this embodiment, S22, performing feature enhancement on the multi-scale video slices to generate feature-enhanced video slices, includes:

[0093] S221. Feature enhancement is performed on multi-scale video slices. The Canny edge detection operator is used to strengthen the peak contour, the brightness gain coefficient is set to highlight the foam contrast, the local binary pattern texture feature is extracted to enhance the dynamic texture, and the feature-enhanced multi-scale video slices are generated.

[0094] S222, annotating the feature-enhanced video slices to generate an annotated wave-breaking image;

[0095] S223: Perform frame extraction on the annotated video slices to generate an image sequence suitable for model input.

[0096] Exemplarily, the cropped image sequence is subjected to multi-scale processing to generate a multi-scale wave-breaking image, the crest contour is extracted using the Canny edge detection operator, the brightness gain coefficient is set to 1.2 to adjust the image brightness, enhance the contrast between the foam area and the background, extract the local binary pattern texture features, capture the dynamic texture information in the wave breaking process, and fuse the crest contour, brightness enhanced image and dynamic texture features to generate a feature-enhanced wave-breaking image.

[0097] During image processing, the multi-scale wave-breaking image was first subjected to Gaussian pyramid decomposition. Downsampling with a scale factor of 1.2 was used to generate three image layers of varying resolutions. Bilinear interpolation was used to normalize the layers to their original size and then superimposed and fused to enhance the image's multi-scale features. The Canny edge detection operator was then applied with a Gaussian filter kernel size of 5×5 and high and low thresholds of 120 and 60, respectively, to extract crest contour information. Morphological closing (3×3 rectangular structuring elements) was then used to connect the breaking edges and enhance the continuity of the wave structure. To enhance the contrast of the foam region, a linear brightness gain of 1.2 was applied to the V channel of the HSV color space, while limiting pixel values to no more than 255. Histogram equalization (CLAHE algorithm, grid size 8×8, contrast limit 2.0) was also used to enhance local contrast. During the texture feature extraction phase, the image was divided into 16×16 local blocks using the LBP operator with a circular neighborhood radius of 2 pixels to calculate uniform pattern features. After gamma correction (γ = 0.8) was performed to reduce the effects of illumination, the texture feature map was weightedly fused with the original image (weight coefficient 0.7). Finally, a non-local mean filter (search window 15×15, similarity window 5×5, attenuation parameter h = 10) was used to eliminate the noise introduced during the enhancement process. This resulted in a feature-enhanced wave image, in which the gradient amplitude of the wave contour increased by approximately 35%, the contrast standard deviation of the foam region increased by 22%, and the discriminability of the texture features improved by 18%.

[0098] In this embodiment, when the segmented time segments are cropped, the time interval between wave-breaking slices at similar positions exceeds 1 minute.

[0099] S3. Constructing a training set based on the labeled data, and training a neural network model based on the training set;

[0100] For example, a deep learning classification model is constructed based on the training dataset, using a convolutional-recurrent neural network architecture to extract spatiotemporal features. If overfitting is detected during model training, model parameters are adjusted using dropout and regularization techniques to obtain an optimized model structure. Using a preset optimizer and loss function, the deep learning classification model is iteratively trained, and model parameters are updated to obtain a trained classification model. The trained classification model is then evaluated using multi-fold cross-validation to determine its classification accuracy and generalization capability. Based on the evaluation results, the trained classification model is deployed to an edge computing device or cloud server to perform real-time wave breaking type classification inference.

[0101] A deep learning classification model was constructed based on the training dataset, using a convolutional-recurrent neural network architecture to extract spatiotemporal features. If overfitting was detected during model training, model parameters were adjusted using a dropout rate of 0.5 and L2 regularization to achieve an optimized model architecture. The deep learning classification model was iteratively trained using the Adam optimizer and the cross-entropy loss function, updating the model parameters to obtain the trained classification model. The performance of the trained classification model was evaluated using a five-fold cross-validation method, calculating the classification accuracy and F1 score to determine the model's classification precision and generalization ability. Based on the evaluation results, the trained classification model was deployed to an edge computing device such as the NVIDIA Jetson Xavier or a cloud server such as an AWS EC2 instance to perform real-time wave breaking type classification inference.

[0102] S4. Based on the trained neural network model, feature extraction and time series modeling are performed on the nearshore wave image sequence to predict the wave breaking type corresponding to the nearshore wave image sequence.

[0103] In this embodiment, S4, based on the trained neural network model, performs feature extraction and time series modeling on the nearshore wave image sequence to predict the wave breaking type corresponding to the nearshore wave image sequence, including:

[0104] S41. Use the pre-trained convolutional network as a feature extractor to encode the image sequence;

[0105] S42. Decode the encoded sequence through a long short-term memory network to generate a temporal feature representation;

[0106] S43. Determine the classification label of the wave breaking type based on the time series feature representation.

[0107] For example, a convolutional-recurrent neural network is used to extract spatiotemporal features from the keyframe sequence. The convolutional layer uses a 3×3 convolution kernel and the recurrent layer uses LSTM units to obtain the dynamic feature vector of the wave breaking process. The dynamic feature vector is then inferred using a classification model, and the Softmax function is used to calculate the probability distribution of the wave breaking type. If the highest probability value in the probability distribution is greater than 0.8, the corresponding wave breaking type label is output.

[0108] Based on the above embodiments, the present invention has the following advantages:

[0109] This method uses fixed or mobile cameras to capture video data from the nearshore wave-breaking zone, segmenting, cropping, and annotating the video to construct a time-series image dataset. A deep learning model is then used to extract spatial and temporal features for automatic classification of wave breaking types. The method uses a deep learning model that combines convolutional neural networks and recurrent neural networks to perform inference processing on real-time nearshore video, output wave breaking type labels, and apply the classification results to coastal engineering monitoring and early warning. Through transfer learning technology, the method can adapt to diverse terrain and sea conditions, improving classification accuracy and generalization capabilities, and providing efficient and reliable technical support for coastal status monitoring and analysis.

[0110] Example 2

[0111] Based on the method described in one embodiment, this embodiment will be further explained with a specific example. Figure 2 :

[0112] A1. Nearshore video image collection

[0113] According to observation requirements, deploy fixed or temporary video acquisition equipment in the target coastal area ( Figure 3 In the figure, A is a schematic diagram of a fixed network camera; B is a mobile camera (such as a GoPro). The camera should be installed at an elevated position that covers the nearshore wave-breaking zone, ensuring good visual range and stability. Through continuously acquired video image sequences, dynamic information about the wave breaking process in the nearshore area is obtained, providing basic data support for subsequent image processing and classification. To ensure the accuracy of wave-breaking observations and the effectiveness of deep learning model classification, the video images must have a pixel resolution of at least 1920×1080 and a frame rate of at least 10 frames per second.

[0114] A2. Wave-breaking classification dataset creation

[0115] The original video usually has a large field of view, and the wave-breaking area occupies a small proportion of the picture. In addition, the length of a single video is usually about 10 minutes, and the frame rate exceeds 10 frames per second. Directly using it for deep learning model training may cause excessive computational burden or interference from irrelevant information. To improve the usability of the data, we performed four main processing steps on the original video: video segmentation, cropping, labeling, and frame extraction ( Figure 2 ).

[0116] The example dataset of this invention has been published on Figshare

[0117] (https: / / figshare.com / articles / dataset / VID_WBT_A_Video_Imagery_Dataset_for_Nearshore_Wave_Breaking_Type_Classification / 28814993), which contains two subsets: Set A and Set B.

[0118] -Set A: Video slices for nearshore wave breaking classification, containing 9,000 wave-breaking slices. Each slice is 30 seconds long and shows a relatively complete wave-breaking process. Slices are labeled according to the most prominent wave-breaking type as "Spilling," "Plunging," or "Surging," with the category indicated directly in the first word of the file name.

[0119] -Set B: Time series image data extracted at a 2Hz frequency based on the wave-breaking slices in Set A. Each set of 60 frames is placed in a separate folder named after the wave-breaking slice in Set A. This folder can be used as direct input for deep learning models.

[0120] In this example, see Figure 4 ,A2, wave breaking classification dataset production, the specific steps are as follows:

[0121] A21. Video Segmentation

[0122] First, we segmented the original 10-minute video into 30-second segments. It's important to note that nearshore, wind-induced gravity waves with a period of 1 to 30 seconds have the highest energy contribution. Therefore, choosing a 30-second segmentation unit ensures coverage of the complete wave-breaking process while reducing data redundancy.

[0123] A22, cutting

[0124] Based on the segmented video clips, we visually interpret the wave breaking process and crop it using a 224×224 pixel window to obtain wave-breaking slices. The cropping operation not only effectively focuses on the target area but also minimizes the consumption of computing resources. It should be noted that the 224×224 image size is the standard input size for the ResNet deep learning network. At the same time, this size has been proven in our practice to be sufficient to cover the spatiotemporal scale of the nearshore wave breaking process. Users can also customize the cropping window size according to research needs (the scale of the observation area and the deep learning model used).

[0125] Furthermore, to ensure the relative independence of the dataset, we cropped multiple segments generated from the same original video, ensuring that the time interval between wave-breaking slices at similar locations exceeded one minute. This prevented the same wave-breaking process from being repeatedly identified in the dataset, thereby preventing data leakage and improving the generalization ability of the trained model.

[0126] A23, marking

[0127] Appropriate wave-breaking slices were selected and labeled as one of three types: spilling, plunging, and surging, based on the most prominent visual features of the wave-breaking process in the slice. To ensure a balanced dataset, the number of slices of the three types of wave-breaking slices was strictly controlled in a 1:1:1 ratio during the labeling process.

[0128] A24, frame extraction

[0129] Finally, to facilitate the deep learning model's classification of wave types, we frame-decimated the wave-breaking footage. In this example, we used a 2Hz frame rate. This frequency is common in coastal video monitoring and provides sufficient temporal resolution to fully capture the temporal variations of different wave types. Users can adjust the frame rate to suit different application scenarios and analysis objectives based on their specific research needs.

[0130] A3. Deep Learning Wave Breaking Classification Model Design

[0131] In order to verify the feasibility of deep learning models for wave breaking classification in video images, we used the CNN+RNN (CRNN) model of temporal deep learning to perform an exemplary classification task. CRNN combines the spatial feature extraction capability of CNN with the temporal information modeling capability of RNN, and is suitable for processing the dynamic evolution of wave breaking. In addition, the CRNN model structure is relatively lightweight and suitable for different computing resource environments, making it easy to reproduce and widely deploy. The model structure of CRNN is as follows: Figure 5 shown.

[0132] -CNN as encoder, through:

[0133] f CNN (x (t) )=z (t) #(3)

[0134] For each 2D image x (t) , encoded (i.e., compressed dimension) as a 1D vector z (t) .

[0135] -RNN acts as a decoder, receiving the sequence input vector z from the CNN encoder (t) , and output another 1D sequence h (t) Finally, a fully connected neural network is connected to the output for wave type prediction. The CNN encoder used for validation is a ResNet-50 model pre-trained on the ILSVRC-2012-CLS image dataset. After removing the original fully connected layers, the convolutional layers are retained as spatial feature extractors. The RNN decoder uses a long short-term memory (LSTM) network. Our experiments show that this model design strikes a good balance between computational efficiency and feature expression capabilities.

[0136] A4. Model Training

[0137] During the training of the example model, the Adam optimizer (with a learning rate of 0.001) was used for gradient updates, and the cross-entropy loss function was selected as the model's loss function. To alleviate overfitting, a dropout mechanism was introduced with a dropout rate set to 0.3. Furthermore, to comprehensively evaluate the model's generalization capabilities, a 5-fold cross-validation strategy was employed, ensuring that each sample was used for both training and validation in five training cycles. This approach effectively mitigated the impact of discrepancies in data partitioning, thereby improving the robustness and reliability of model performance evaluation.

[0138] Figure 6 The training and validation results of 5-fold cross-validation are shown (A is the training loss result; B is the training accuracy result), highlighting the performance of the model in terms of stability and generalization ability. Figure 6 As shown in A, the training loss is always kept at a low level. Although the validation loss peaks in the fourth fold, the overall trend is reasonable, indicating that the model can effectively learn features under different data partitions without serious overfitting. Figure 6As shown in Figure 2, the training accuracy consistently remains above 95%, and the validation accuracy remains above 93% from the first to third folds. Although there is a brief dip in the fourth fold, it recovers in the fifth fold, demonstrating the model's good adaptability and strong generalization capabilities. By employing 5-fold cross-validation, this scheme effectively reduces the impact of random data partitioning on experimental results, maximizes the use of limited data resources, and improves the reliability of model evaluation. Furthermore, this scheme includes frame-by-frame wave breaking information, allowing users to flexibly design data augmentation strategies and frame-level sampling schemes based on their research needs. Overall, the dataset and model framework constructed by this scheme provide a solid data support and technical foundation for the task of wave breaking type classification.

[0139] A5. Model deployment and application

[0140] After model training and validation, the trained deep learning model is deployed to the nearshore video surveillance system, enabling real-time classification and identification of breaking wave types. Depending on application needs, deployment can be performed using either edge computing devices (such as embedded GPU modules) or cloud server architectures to meet the data processing and response requirements of different scenarios.

[0141] The specific application process includes:

[0142] 1) Receive real-time nearshore video streams collected by front-end camera equipment;

[0143] 2) Preprocess the video frames (segmentation, cropping, frame extraction, etc.) according to the A2 standard process;

[0144] 3) Use the deployed model to infer the frame sequence and output the corresponding wave-breaking type label;

[0145] 4) Display the classification results in real time or upload them to a remote platform for applications such as coastal status monitoring, early warning system triggering, and geomorphological analysis.

[0146] Furthermore, this method exhibits excellent versatility and scalability. In applications targeting diverse terrain, climate, or hydrodynamic environments, existing models can be fine-tuned through transfer learning, or targeted training can be performed with data from new regions, further improving classification accuracy and adaptability. This feature enables this method to flexibly address diverse coastal observation needs, promising broad prospects for engineering application.

[0147] Based on this embodiment, the present invention has the following advantages:

[0148] (1) Automated processing and high recognition efficiency: Traditional wave breaking type classification relies on manual observation or rule-driven empirical methods, which are not only inefficient and labor-intensive, but also highly subjective and unstable. The present invention uses a deep learning model to automatically analyze video images, achieving real-time, unattended recognition of wave breaking types, greatly improving efficiency and automation.

[0149] (2) High classification accuracy and robustness: This method models video frame sequences by constructing a deep learning network (such as a CRNN, 3D CNN, LSTM, or Transformer) for spatiotemporal feature extraction. This model can effectively capture the dynamic characteristics of wave breaking. In multi-fold cross-validation experiments, the model demonstrated high classification accuracy and stability, demonstrating good generalization capabilities.

[0150] (3) Supports diverse sea conditions and terrain conditions, and has strong adaptability: Compared with traditional methods that rely on empirical formulas such as the Iribarren number, the present invention does not rely on complex physical parameter calculations and exhibits stronger adaptability to areas with complex terrain or restricted measurements. Through transfer learning or regional fine-tuning, it can also be quickly deployed in new areas to improve model accuracy. Combined with high-frequency video acquisition and intelligent recognition systems, the present invention can be widely used in coastal engineering design, marine disaster warning, ecological environment monitoring and other fields, and has significant engineering promotion and practical application value.

[0151] Embodiment 3

[0152] See also Figure 7 A nearshore wave breaking type classification system based on deep learning of video images, based on the method of embodiment 1 or embodiment 2, includes:

[0153] Acquisition module: used to collect video image data of the nearshore area based on fixed or temporary video acquisition equipment. The video image data covers the high position of the nearshore wave zone;

[0154] Preprocessing module: used to preprocess video image data and mark overflow waves, curl waves and collapse waves according to the data;

[0155] Training module: used to construct a training set based on the labeled data and train a neural network model based on the training set;

[0156] Prediction module: used to extract features and perform time series modeling on the nearshore wave image sequence based on the trained neural network model, and predict the wave breaking type corresponding to the nearshore wave image sequence.

[0157] In the above technical solution, in order to better use the method described in Example 1 or Example 2, the present application proposes a nearshore wave breaking type classification system based on deep learning of video images. Each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0158] Example 4

[0159] A device for classifying nearshore wave breaking types based on deep learning of video images, comprising:

[0160] at least one processor and a memory communicatively coupled to the at least one processor;

[0161] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method described in one or two embodiments.

[0162] In the above technical solution, in order to better run and process the method described in Embodiment 1 or Embodiment 2, the above method is stored in a memory and executed by a processor. It should be noted that the principles and effects of each step have been described above and will not be further explained here.

[0163] Embodiment 5

[0164] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method described in one or two embodiments.

[0165] In the above technical solution, in order to better run and use the method described in Example 1 or Example 2, the above method is stored in a computer-readable storage medium and implemented using a processor. It should be noted that the principles and effects of each step have been described above and will not be further explained here.

[0166] The above are only some embodiments of the present invention and do not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for classifying nearshore wave breaking types based on deep learning of video images, characterized in that: The method comprises: Using fixed or temporary video acquisition equipment, collect video image data of the nearshore area, where the video image data covers a high position in the nearshore wave-breaking zone; Preprocessing the video image data, and marking overflow waves, curl waves and collapse waves according to the data; Constructing a training set based on the labeled data, and training a neural network model based on the training set; Based on the trained neural network model, feature extraction and time series modeling are performed on the nearshore wave image sequence to predict the wave breaking type corresponding to the nearshore wave image sequence.

2. The method for classifying nearshore wave breaking types based on deep learning of video images according to claim 1, characterized in that: Preprocessing the video image data and labeling overflow waves, curl waves and collapse waves according to the data, including: The video image data is divided into multiple video slices with each segment being 1 to 30 seconds; the segmented time segments are cropped to focus on the wave breaking area to obtain cropped video slices; Annotating the video slices and classifying wave breaking types into overflow waves, curl waves, and collapse waves based on visual features; The annotated video slices are subjected to frame extraction processing to generate image sequences suitable for model input.

3. The method for classifying nearshore wave breaking types based on deep learning of video images according to claim 1, characterized in that: Preprocessing the video image data and labeling overflow waves, curl waves and collapse waves according to the data, including: The video image data is segmented into multiple video slices with each segment ranging from 1 to 30 seconds. The segmented time segments are cropped at multiple scales to retain the small-scale foam details and medium- and large-scale wave morphology in the wave breaking area, generating multi-scale video slices. Perform feature enhancement on multi-scale video slices to generate feature-enhanced video slices; Annotate feature-enhanced video slices to generate annotated wave-breaking images; The annotated video slices are subjected to frame extraction processing to generate image sequences suitable for model input.

4. The method for classifying nearshore wave breaking types based on deep learning of video images according to claim 3, characterized in that: Perform feature enhancement on multi-scale video slices to generate feature-enhanced video slices, including: The feature enhancement of multi-scale video slices is performed. The Canny edge detection operator is used to strengthen the peak contour, the brightness gain coefficient is set to highlight the foam contrast, the local binary pattern texture feature is extracted to enhance the dynamic texture, and the feature-enhanced multi-scale video slices are generated.

5. A method for classifying nearshore wave breaking types based on deep learning of video images according to any one of claims 1 to 3, characterized in that: Label overflow waves, roll waves, and collapse waves according to the data, including: The type of wave breaking is determined based on the significant visual features in the slices. If it shows gentle flow and low wave crest, it is marked as overflow wave. If it shows tubular curling and white foam, it is marked as curl wave. If it shows splashing and breaking, it is marked as collapse wave.

6. A method for classifying nearshore wave breaking types based on video image deep learning according to claim 2 or 3, characterized in that: When the segmented time segments are cropped, the time interval between wave-breaking slices at similar positions exceeds 1 minute.

7. The method for classifying nearshore wave breaking types based on video image deep learning according to claim 1, characterized in that: Based on the trained neural network model, feature extraction and time series modeling are performed on the nearshore wave image sequence to predict the wave breaking type corresponding to the nearshore wave image sequence, including: Using a pre-trained convolutional network as a feature extractor to encode the image sequence; The encoded sequence is decoded through the long short-term memory network to generate a temporal feature representation; A classification label of the wave breaking type is determined according to the time series feature representation.

8. A nearshore wave breaking type classification system based on deep learning of video images, characterized by: Based on the method according to any one of claims 1 to 7, comprising: Acquisition module: used to collect video image data of the nearshore area based on fixed or temporary video acquisition equipment, and the video image data covers the high position of the nearshore wave zone; Preprocessing module: used for preprocessing the video image data and marking overflow wave, curl wave and collapse wave according to the data; Training module: used to construct a training set based on the labeled data and train a neural network model based on the training set; Prediction module: used to perform feature extraction and time series modeling on the nearshore wave image sequence based on the trained neural network model, and predict the wave breaking type corresponding to the nearshore wave image sequence.

9. A device for classifying nearshore wave breaking types based on deep learning of video images, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images