Ice recognition and segmentation method and system fusing relative position information and GLCM features

Abstract

The invention discloses an ice recognition and segmentation method and system fusing relative position information and GLCM features, and relates to the technical field of deep learning and image recognition, and the method comprises the steps: obtaining an RGB aerial image, carrying out the preprocessing of the RGB aerial image, and building a data set for the preprocessed RGB aerial image; performing texture feature extraction on the data in the data set through a gray-level co-occurrence matrix; establishing a graph convolutional network, modeling nodes in the RGB aerial image according to the graph convolutional network, and obtaining node features of ice; and feature fusion is carried out based on the texture features, the node features and the RGB channels of the RGB aerial images, fusion features are obtained, multi-scale features of the fusion features are extracted through a neural network, and a detection result is obtained. According to the method, on the basis of the yolk v11 feature, the relative position information of the ice and the texture feature extracted by the GLCM gray level co-occurrence matrix are fused, and the recognition and segmentation precision of the model on the ice is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and image recognition, and more specifically to an ice recognition and segmentation method and system integrating relative position information and GLCM features. Background Art

[0002] Ice is the solid formed by water condensing at 0℃ or below 0℃, and ice accumulation is ice. Ice can be divided into three categories: water ice, sedimentary ice, and glacial ice. Water ice is formed by the direct freezing of water, including river ice, lake ice, and sea ice. When the temperature continues to drop and the water temperature reaches and slightly below 0℃, the river begins to freeze on the shore and in the water. River ice freezes in moving water, which is different from lake ice. Lake water freezing is limited to the surface of the water body, and the deep water body still maintains a temperature above 0℃. Due to the turbulent mixing of water flow in rivers, the heat loss of water bodies is almost simultaneous in the entire water body. Therefore, rivers not only form thin ice and shore ice on the surface of the water, but also form underwater ice in the water and at the bottom of the river.

[0003] The traditional river ice monitoring method based on manual observation has limitations such as high workload, insufficient adaptability, and lagging monitoring capabilities. In aerial images, floating ice has the characteristics of dense distribution, similar characteristics, and large differences in the number and scale of different types of floating ice. Ordinary neural networks often cannot fully extract the characteristics of floating ice, making it difficult for neural networks to accurately distinguish different types of floating ice.

[0004] Therefore, how to improve the recognition and segmentation accuracy of ice flow is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides an ice recognition and segmentation method and system that integrate relative position information and GLCM features. Aiming at the problem that ordinary neural networks cannot fully extract ice features, resulting in poor effect of model on ice recognition and segmentation, a YOLO V11 ice recognition and segmentation method that integrates relative position information and GLCM features is proposed. On the basis of YOLO V11 features, the relative position information of ice and the texture features extracted by GLCM gray-level co-occurrence matrix are integrated to improve the recognition and segmentation accuracy of the model for ice.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An ice recognition and segmentation method integrating relative position information and GLCM features, comprising:

[0008] Acquire RGB aerial images, preprocess the RGB aerial images, and establish a data set for the preprocessed RGB aerial images;

[0009] Extracting texture features from the data in the data set by using a gray level co-occurrence matrix;

[0010] Establishing a graph convolutional network, modeling nodes in the RGB aerial image according to the graph convolutional network, and obtaining node features of ice;

[0011] Feature fusion is performed based on the texture features, node features and RGB channels of the RGB aerial image to obtain fused features, and multi-scale features of the fused features are extracted through a neural network to obtain detection results.

[0012] Preferably, the gray level co-occurrence matrix specifically includes:

[0013] P d,θ (m,n)=#{((x,y),(x',y′))||I(x,y)=m,I(x',y′)=n};

[0014] Where d is the distance, θ is the direction, (x', y') is the neighborhood of the pixel coordinate (x, y), and P d,θ (m,n) is the co-occurrence frequency matrix of grayscale values ​​m and n at distance d and direction θ.

[0015] Preferably, in the texture feature extraction, the extracted texture features include contrast, correlation, entropy, energy and uniformity.

[0016] Preferably, the establishment of the graph convolution network specifically includes: taking the ice area and surrounding related objects as nodes v i , establish a graph G = (V, E), where V represents the node set, E represents the edge set, and node v i ∈V, edge e i,j ∈E; Initialize each node v i The eigenvector of

[0017] The update formula of the graph convolutional network is:

[0018]

[0019] in, represents the feature vector of node i in the kth layer, is the set of neighbor nodes of node i, c ij is the normalization coefficient, is the weight matrix of the kth layer, and σ is the activation function.

[0020] Preferably, the acquiring of fusion features specifically includes:

[0021]

[0022] in, is the texture feature, is the node feature, is the RGB channel feature map, For fusion features.

[0023] Preferably, the neural network adopts the yolov11 network.

[0024] Preferably, the convolution operation of the L+1th layer in the yolov11 network is:

[0025]

[0026] in, It is the general term for all feature maps in the Lth layer. represents the convolution kernel weight, * represents the convolution operation, σ is the activation function, and b is the bias term.

[0027] An ice recognition and segmentation system integrating relative position information and GLCM features, comprising:

[0028] A data set establishment module is used to obtain RGB aerial images, preprocess the RGB aerial images, and establish a data set with the preprocessed RGB aerial images;

[0029] A texture feature extraction module extracts texture features from the data in the dataset through a gray level co-occurrence matrix;

[0030] A node extraction module is used to establish a graph convolutional network, model the nodes in the RGB aerial image according to the graph convolutional network, and obtain the node features of ice;

[0031] The detection module performs feature fusion based on the texture features, node features and RGB channels of the RGB aerial image to obtain fused features, extracts multi-scale features of the fused features through a neural network, and obtains detection results.

[0032] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an ice recognition and segmentation method and system that integrates relative position information and GLCM features. By integrating relative position information, GLCM (gray-level co-occurrence matrix) features and RGB channel features, the characteristics of ice can be comprehensively characterized from multiple dimensions. Relative position information helps to determine the layout relationship of ice in the image, GLCM features can extract the texture details of ice, and RGB channel features contain intuitive visual information such as color. This multi-dimensional feature fusion makes the description of ice richer and more accurate, and improves the reliability of recognition and segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0034] Figure 1 The improved YOLO v11 model structure diagram provided by the present invention;

[0035] Figure 2 The yolo v11 ice recognition and segmentation flow chart of the present invention that integrates relative position information and GLCM features;

[0036] Figure 3 Some pictures and label data provided by the present invention;

[0037] Figure 4a A grayscale image provided by the present invention;

[0038] Figure 4b The GLCM matrix diagram provided by the present invention;

[0039] Figure 5 The GCN feature matrix diagram provided by the present invention;

[0040] Figure 6 A node distance matrix diagram provided by the present invention;

[0041] Figure 7 The feature matrix diagram after splicing provided by the present invention;

[0042] Figure 8 The identification PR curve diagram of the model provided by the present invention;

[0043] Fig. 9 The segmentation PR curve diagram of the model provided by the present invention;

[0044] Fig.10 This is a partial flow segmentation result diagram provided by the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The embodiment of the present invention discloses an ice recognition and segmentation method integrating relative position information and GLCM features, comprising:

[0047] Acquire RGB aerial images, preprocess the RGB aerial images, and establish a data set for the preprocessed RGB aerial images;

[0048] The texture features of the data in the dataset are extracted through the gray level co-occurrence matrix;

[0049] A graph convolutional network is established, and nodes in RGB aerial images are modeled based on the graph convolutional network to obtain the node features of ice.

[0050] Feature fusion is performed based on texture features, node features and RGB channels of RGB aerial images to obtain fused features. Multi-scale features of the fused features are extracted through a neural network to obtain detection results.

[0051] Before training, the input RGB aerial images are preprocessed, including image normalization, cropping, rotation, translation and other data enhancement operations to ensure that the model can effectively learn in a diverse data environment. The enhanced dataset contains icy images of different sizes, shapes and background environments.

[0052] GLCM (Gray Level Co-occurrence Matrix) can effectively capture the texture features of an image by calculating the gray level co-occurrence frequency of pixels in an image. The surface texture of icicles has certain regularities, such as roughness, cracks, etc. Using GLCM can help the model extract these detailed texture features, thereby better distinguishing different types of icicles. For each image, especially the areas where icicles exist, texture features are extracted by calculating the Gray Level Co-occurrence Matrix (GLCM). During the calculation process, multiple directions and different distances are considered to capture the different texture features of icicles. Commonly used GLCM features include contrast, correlation, entropy, energy, and homogeneity. These texture features can effectively describe information such as the surface roughness and edge clarity of icicles. The following formula is used to calculate each GLCM feature and use it as a feature vector Helps the model distinguish different types of ice (such as floating ice, shore ice, slush ice, etc.).

[0053] For a given grayscale image I(x,y) and grayscale level L, GLCM can be obtained by the following steps:

[0054] In a specific embodiment, the gray level co-occurrence matrix specifically includes:

[0055] P d,θ(m,n)=#{((x,y),(x',y′))|I(x,y)=m,I(x',y′)=n};

[0056] Where d is the distance, θ is the direction (such as horizontal, vertical or 45°), (x', y') is the neighborhood of the pixel coordinate (x, y), and P d,θ (m,n) is the co-occurrence frequency matrix of grayscale values ​​m and n at distance d and direction θ.

[0057] In a specific embodiment, in the texture feature extraction, the extracted texture features include contrast, correlation, entropy, energy and uniformity.

[0058] Using the co-occurrence matrix P d,θ (m,n) calculates commonly used texture features. Commonly used GLCM features are defined as follows:

[0059] Contrast is used to measure the intensity of grayscale changes in an image:

[0060]

[0061] Correlation is used to measure the linear correlation between pixels, where μ is the standardized mean, which is used to describe the expected value of the gray level, and σ is the standard deviation, which is used to describe the range of gray level variation:

[0062]

[0063] Entropy is used to measure the texture uncertainty of an image:

[0064]

[0065] Energy, used to reflect the uniformity of grayscale distribution:

[0066]

[0067] Homogeneity is used to measure the smoothness of pixel grayscale distribution:

[0068]

[0069] In a specific embodiment, a graph convolutional network (GCN) can effectively model the spatial relationship between ice and its surrounding objects. Given an ice area and other surrounding objects as nodes, the ice area and surrounding related objects (such as river banks, other ice blocks, etc.) are nodes v i , establish a graph G = (V, E), where V represents the node set, E represents the edge set, and node v i∈V, the relationship between nodes (such as distance and adjacency) is the edge e i,j ∈E; Initialize each node v i The eigenvector of Including the texture features of ice GLCM , shape features, and relative position. The features of each node (ice cube or other object) are composed of multiple dimensions: the color of the ice cube, texture features (from GLCM), the shape and size of the ice cube, and the relative position of the ice cube to the surrounding objects.

[0070] Graph Convolutional Network (GCN) Modeling: Use graph convolutional networks to propagate nodes in the graph and learn the relationship between nodes. In the graph convolution layer, each node v i Update the features of neighboring nodes through their information. This allows each node (ice cube or other object) to learn not only its own features, but also the features of surrounding objects related to it. The update formula of the graph convolutional network is:

[0071]

[0072] in, represents the feature vector of node i in the kth layer, is the set of neighbor nodes of node i, c ij is the normalization coefficient, is the weight matrix of the Kth layer, and σ is the activation function, such as ReLU.

[0073] In a specific embodiment, GLCM features are added to the basic architecture of YOLO v11 And the output of GCN The RGB channels of the input image and the texture features extracted by GLCM And the node features output by the graph convolutional network Combined into multi-channel data as the original input of the yolov11 network:

[0074]

[0075] in, is the texture feature, is the node feature, is the RGB channel feature map, is the fusion feature. Then the extracted GLCM features And the output of GCN The feature map output by the second C3k2 module of the yolo v11 backbone network is spliced, and the spliced ​​feature map is fused with the upsampled feature of the yolov11 neck and used as the input of the next C3k2 module to improve the recognition effect of small target objects. Figure 1 As shown in the figure, the entire network consists of three parts: backbone, neck and head. The backbone is used for deep feature extraction, the neck is mainly used for sampling and fusion of features extracted by the backbone, and the head is used for target detection and segmentation. The three detection heads process large, medium and small targets respectively. The input of RGB images is divided into two paths. One path is input to the graph convolution module and the gray-level co-occurrence matrix module to generate GLCM features. And GCN output The second path is directly obtained by the first path Splicing to get fusion features in Continue to pass in the backbone network of yolo v11, after two convolutional layers (Conv), a C3k2 feature extraction module, a convolutional layer and a C3k2 feature extraction module, the obtained feature map is combined with the GLCM features generated by the graph convolution module and the gray-level co-occurrence matrix module And GCN output Concatenate them to get the feature map F1. It is passed down to the C2PSA module of the backbone network, and the obtained feature map F2 is then passed to the yolo v11 neck part, and after upsampling, it is spliced ​​with the feature map F3 output by the third C3k2 module of the backbone network to obtain the spliced ​​feature map F4. F4 is then passed up, and after passing through the C3k2 module and upsampling, it is spliced ​​with F1 to obtain the feature map F5. After passing through the C3k2 module, F5 is divided into two paths for further transmission, one of which is directly passed to the detection head for target detection and segmentation, and the other is spliced ​​with the feature map of F4 after passing through the C3k2 module to obtain the feature map F6. After passing through the C3k2 module, F6 is also divided into two paths, one of which is passed to the detection head for target detection and segmentation, and the other is passed down. Finally, after being spliced ​​with F2, it is passed to the detection head through the C3k2 module for target detection and segmentation.

[0076] In a specific embodiment, the neural network adopts the YOLOv11 network.

[0077] In a specific embodiment, multi-scale feature extraction is performed in the convolution layer of YOLO v11. Assume that the feature map obtained after L layers of convolution operation is Convolution operations are performed on different channels in each layer, so as to learn multiple feature information during feature extraction. The convolution operation of the L+1 layer in the yolov11 network is:

[0078]

[0079] in, It is the general term for all feature maps in the Lth layer. represents the convolution kernel weight, * represents the convolution operation, σ is the activation function, and b is the bias term.

[0080] After multiple operations such as dimensionality increase, dimensionality reduction, and splicing, YOLOv11 outputs the target detection results and segmentation mask This includes the type, location, and segmentation of each icicle:

[0081]

[0082] The overall structure is as follows Figure 1 As shown in the figure, the input is an RGB three-channel image input into the GCN module and the GLCM module to obtain the GCN features and GLCM features. The obtained GCN features and GLCM features are concatenated with the RGB channels of the image and input into the network together as the original data. At the same time, the obtained GCN features and GLCM features are also fused with the output features of the second C3k2 module. The fused features are then fused with the upsampled features in the neck structure as the input of the next module. (In the original yolo v11 structure, the output of the second C3k2 module is directly fused with the upsampled features in the neck structure). Represents the final output of the network, such as Figure 1 There are three "head" structures in it.

[0083] An ice recognition and segmentation system integrating relative position information and GLCM features, comprising:

[0084] The data set establishment module obtains RGB aerial images, preprocesses the RGB aerial images, and establishes a data set with the preprocessed RGB aerial images;

[0085] Texture feature extraction module, which extracts texture features from data in the dataset through gray-level co-occurrence matrix;

[0086] The node extraction module builds a graph convolutional network, models the nodes in the RGB aerial image based on the graph convolutional network, and obtains the node features of ice.

[0087] The detection module performs feature fusion based on texture features, node features and RGB channels of RGB aerial images to obtain fused features, and extracts multi-scale features of the fused features through a neural network to obtain detection results.

[0088] The present invention can effectively improve the recognition and segmentation accuracy of the model for different types of ice by integrating the ice RGB image features extracted by yolo v11, the relative position information of ice and the GLCM features of the image. The measured recognition mAP@0.5 is 90.7% and the segmentation mAP@0.5 is 87.2%. Compared with the original yolo v11 model, the improved model has the characteristics of high recognition and segmentation accuracy and fast reasoning speed.

[0089] In a specific embodiment, data preprocessing first performs data enhancement operations such as normalization, cropping, rotation, and translation on the image to increase data diversity, and then uses the labelme annotation tool to create image segmentation labels to form an ice recognition and segmentation dataset. Some images and label data are as follows: Figure 3 shown.

[0090] GLCM feature extraction, traverse the RGB aerial images in the dataset, first use opencv to convert the image into grayscale, and then use skimage's greycomatrix() function to calculate the GLCM matrix of each image at different angles and distances. Figure 2 Take the first image of as an example, in the figure, ① is the dataset composed of aerial ice flow images and corresponding labels, ② is the fusion splicing module, ③ is the yolo v11 backbone network, ④ is the neck part of the yolo v11 network, ⑤ is the ice recognition and segmentation module. ⑥ is the graph network GCN, ⑦ is the GLCM feature extraction module; the grayscale image and the calculated GLCM matrix are shown in Figure 4a and 4b shown.

[0091] Common texture features are extracted from the GLCM matrix, including contrast, correlation, energy, homogeneity, etc., which can be used to describe the texture properties of ice, as shown in Table 1:

[0092] Table 1 Common texture features extracted from the GLCM matrix

[0093] Contrast Relevance energy Homogeneity [[228.77967787]] [[0.92110806]] [[0.05595256]] [[0.42249167]]

[0094] The relative position information of icicles is constructed using a graph convolutional network (GCN).

[0095] ① Build a graph structure: Each piece of ice or related object is regarded as a node, and the edges between nodes represent their spatial relationship. The weight of the edge can be calculated by distance, direction or other features.

[0096] ② Calculate relative position relationship: Use Euclidean distance, direction, proximity relationship and other information to calculate the weight of the edge.

[0097] ③Build a node feature matrix: Node features can include object category, size, location and other information.

[0098] ④ Use GCN for feature transfer: Input the graph structure and feature matrix into GCN, and extract the relative position features through graph convolution operation; the GCN feature matrix is ​​as follows Figure 5 As shown, the edge weight (node ​​distance matrix) is as follows Figure 6 shown.

[0099] The obtained GLCM features and CGN features are integrated into the YOLOv11 model and concatenated with the features of the RGB image extracted by the YOLO backbone network, as shown in Figure 7 shown.

[0100] The concatenated features are used to train the neural network, and the training is iterated 1500 times to obtain the flow ice recognition and segmentation model. The recognition PR curve of the model is as follows: Figure 8 As shown, the segmentation PR curve is as follows Fig. 9 Some test results are shown in Fig.10 shown.

[0101] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0102] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An ice recognition and segmentation method integrating relative position information and GLCM features, characterized in that: include: Acquire RGB aerial images, preprocess the RGB aerial images, and establish a data set for the preprocessed RGB aerial images; Extracting texture features from the data in the data set by using a gray level co-occurrence matrix; Establishing a graph convolutional network, modeling nodes in the RGB aerial image according to the graph convolutional network, and obtaining node features of ice; Feature fusion is performed based on the texture features, node features and RGB channels of the RGB aerial image to obtain fused features, and multi-scale features of the fused features are extracted through a neural network to obtain detection results.

2. The ice recognition and segmentation method integrating relative position information and GLCM features according to claim 1, characterized in that: The gray level co-occurrence matrix specifically includes: P d,θ (m,n)=#{((x,y),(x′,y′))||I(x,y)=m,I(x′,y′)=n}; Where d is the distance, θ is the direction, (x', y') is the neighborhood of the pixel coordinate (x, y), and P d,θ (m,n) is the co-occurrence frequency matrix of grayscale values ​​m and n at distance d and direction θ.

3. The ice recognition and segmentation method integrating relative position information and GLCM features according to claim 1, characterized in that: In the texture feature extraction, the extracted texture features include contrast, correlation, entropy, energy and uniformity.

4. The ice recognition and segmentation method integrating relative position information and GLCM features according to claim 1, characterized in that: The establishment of the graph convolution network specifically includes: taking the ice area and surrounding related objects as nodes v i , establish a graph G = (V, E), where V represents the node set, E represents the edge set, and node v i ∈V, edge e i,j ∈E; Initialize each node v i The eigenvector of The update formula of the graph convolutional network is: in, represents the feature vector of node i in the kth layer, is the set of neighbor nodes of node i, c ij is the normalization coefficient, and is the weight matrix of the kth layer, and σ is the activation function.

5. The ice recognition and segmentation method integrating relative position information and GLCM features according to claim 1, characterized in that: The obtaining of fusion features specifically includes: in, is the texture feature, is the node feature, is the RGB channel feature map, For fusion features.

6. The ice recognition and segmentation method integrating relative position information and GLCM features according to claim 5, characterized in that: The neural network adopts the yolov11 network.

7. The ice recognition and segmentation method integrating relative position information and GLCM features according to claim 6, characterized in that: The convolution operation of the L+1th layer in the yolov11 network is: in, It is the general term for all feature maps in the Lth layer. represents the convolution kernel weight, * represents the convolution operation, σ is the activation function, and b is the bias term.

8. An ice recognition and segmentation system integrating relative position information and GLCM features, using an ice recognition and segmentation method integrating relative position information and GLCM features as described in any one of claims 1 to 7, characterized in that: include: A data set establishment module is used to obtain RGB aerial images, preprocess the RGB aerial images, and establish a data set with the preprocessed RGB aerial images; A texture feature extraction module extracts texture features from the data in the dataset through a gray level co-occurrence matrix; A node extraction module is used to establish a graph convolutional network, model the nodes in the RGB aerial image according to the graph convolutional network, and obtain the node features of ice; The detection module performs feature fusion based on the texture features, node features and RGB channels of the RGB aerial image to obtain fused features, extracts multi-scale features of the fused features through a neural network, and obtains detection results.

Images