A straw image segmentation method based on a special neural network and image preprocessing

Abstract

The present application provides a straw image segmentation method based on a special neural network and image preprocessing, to improve the accuracy and efficiency of straw image segmentation, and realize the balance of complexity and accuracy. It aims to solve the task of straw agronomy image segmentation. The algorithm first collects color RGB images by unmanned aerial vehicle, and uses random clipping technology to limit the image size to 512x384, and makes Straw320 dataset for subsequent network training and testing. Next, a straw image graying method is proposed, aiming to maintain the discriminability of the image while reducing the complexity of the subsequent model. Finally, a straw mixed network (SMN) is introduced to process the preprocessed image. The network includes a position encoding module to enhance the relative information of the straw image, and adopts a hybrid feature extraction module that balances complexity and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a straw image segmentation method based on a special neural network and image preprocessing. This method can be used for straw processing and utilization in the field of agriculture, and provides accurate straw image segmentation results. BACKGROUND

[0002] Straw has important significance in agriculture. First, the protective work of straw is crucial for soil protection. Straw mulching can reduce water evaporation, reduce soil erosion, and prevent soil quality from declining. Second, as a kind of organic matter, straw can be returned to the field, effectively increasing the organic matter content of the soil and improving the soil structure and fertility. Therefore, the processing and utilization of straw are of great significance to agricultural production and environmental protection.

[0003] Straw image segmentation plays a fundamental role in the subsequent processing and utilization of straw. By segmenting the straw image, the area and shape information of the straw can be accurately extracted, providing a reference for subsequent processing and utilization. For example, segmenting the straw image can help determine how to collect, compact, and stack the straw, and also help assess the quality of the straw and related agricultural research.

[0004] However, the task of straw image segmentation faces some challenges. First, straw images often have complex texture, color, and shape variations, making conventional image segmentation algorithms less accurate when segmenting straw images. Second, current segmentation neural networks are not specifically designed for straw images, lacking effective learning and expression capabilities for straw features, resulting in further degradation of segmentation accuracy. In addition, the balance between the complexity and accuracy of the straw image segmentation algorithm is also a challenging problem.

[0005] Therefore, it is necessary to propose a straw image segmentation method based on a special neural network and image preprocessing to improve the accuracy and efficiency of straw image segmentation and achieve a balance between complexity and accuracy. By introducing a feature learning and expression mechanism for straw images, this algorithm can better handle the complexity of straw images and provide accurate reference results for subsequent straw processing and utilization. SUMMARY

[0006] The present application proposes a straw image segmentation method based on a special neural network and image preprocessing, mainly including the following contents:

[0007] Step 1, collect color RGB images by unmanned aerial vehicle, make Straw320 dataset, and normalize the dataset.

[0008] Step 2, a straw image grayscale method is designed to maintain the discriminability of the image while reducing the complexity of the subsequent model.

[0009] Straw is usually yellow, and the background is usually black. In order to improve the discriminability of straw and background, and reduce the complexity of the network. According to the corresponding value of yellow in RGB space (255, 255, 0), the following grayscale coefficient can be obtained: 0.5, 0.5, 0. This means that the pixel value of the red channel is weighted average with the pixel value of the green channel, and the pixel value of the blue channel is ignored. Therefore, the following grayscale formula can be used:

[0010] P = 0.5 * R + 0.5 * G + 0 * B

[0011] Where P represents the pixel value after grayscale, R, G and B represent the pixel values of red, green and blue channels in the original color image respectively. In this way, the information of the yellow channel is enhanced, and the complexity of the network processing is reduced, so that the subsequent image segmentation task is more efficient and accurate.

[0012] Step 3, a straw mixed network (SMN) is proposed to process the preprocessed image. The network includes a position encoding module, which aims to enhance the relative information of the straw image, and adopts a mixed feature extraction module balancing complexity and accuracy.

[0013] The SMN network draws on the classic U-Net segmentation network structure and makes some key improvements to better adapt to the characteristics of straw images. The overall structure is as shown in Figure 2 Firstly, the position encoding module is introduced to provide the position information of straw and background in the straw image. By introducing the position information, the network can better understand the spatial distribution of the straw and improve the accuracy of segmentation. Secondly, the mixed feature extraction module balancing complexity and accuracy is designed. This module combines the skip connection mechanism of U-Net, while keeping the network complexity controllable, it extracts multi-scale feature information. By integrating different scale features and performing feature fusion and selection, the network can better distinguish straw and background and improve the accuracy of segmentation results. Finally, the SMN network adopts down-sampling and up-sampling operations to transfer information and recover features between the encoder and the decoder. The down-sampling operation reduces the feature size and parameter quantity through the max-pooling layer, and speeds up the training speed. The up-sampling operation uses bilinear interpolation method to restore the spatial resolution of the features, and retains more detailed information.

[0014] Step 3-1, relative position encoding module. In the field of image processing, there is rich interaction information between pixels, and effective extraction of point-to-point information can improve the accuracy of the overall task.

[0015] The present application proposes a relative position encoding module to couple the relative position coordinates with the original image in a simple way. In order to reduce the complexity of the algorithm, the present application divides the original image into small blocks (i.e. patches) of the same size, and then fuses the relative position information with the pixel values of each small block. Assuming that the width of the input image is W, the height is H, and it is divided into small blocks of patch_size x patch_size. Then the horizontal and vertical coordinate range of each small block can be expressed as:

[0016] col_indices = [0, 1, …, W / (patch_size-1)]

[0017] row_indices = [0, 1, …, H / (patch_size-1)]

[0018] In order to perform relative position encoding, the present application defines two encoding values, col_co and row_co, which represent the relative position of the column and row respectively. These encoding values are obtained by dividing col_indices and row_indices by (W / patch_size) and (H / patch_size), respectively, as follows:

[0019] col_co = col_indices / (W / patch_size)

[0020] row_co = row_indices / (H / patch_size)

[0021] Then, the present application adds these encoding values to the original data to fuse the relative position information. Since the original data has been normalized and the value range is between (0, 1), directly adding col_co and row_co encoding may mask the original image data information, leading to data imbalance and affecting the convergence of the network. Therefore, the present application introduces a flexible parameter β to limit the influence of the relative position information, as follows:

[0022] P' = P + β(col_co + row_co)

[0023] where P ∈ R W×H×1 , P' ∈ R W×H×1 .

[0024] Step 3-2, the mixed-extract feature module plays a key role in the straw segmentation task, aiming to achieve a balance between accuracy and complexity. To address this challenge, the present invention proposes a mixed-extract feature module based on the design principle of Encoder-Decoder, which combines convolution operations and adaptive attention mechanisms to achieve more effective image feature extraction.

[0025] Firstly, the module uses convolution operations to extract local features from the image. Convolution operations can capture the spatial relationships between adjacent features in the image while maintaining the feature dimension. In this way, the present invention can fully utilize the local information of the image and extract rich local features.

[0026] Secondly, the mixed-extract feature module introduces an adaptive attention mechanism to extract the relevance between global features of the image. The adaptive attention mechanism can automatically adjust the importance of different features by learning the weights between features, thus more accurately capturing the global context information of the image. This global feature extraction capability enables the module to better understand the relevance between different regions in the image, thus improving the performance of image processing tasks. To reduce the complexity of attention calculation, the present invention proposes to increase the number of hidden layers to achieve a balance between accuracy and complexity.

[0027] In the design of the module, the present invention simplifies the Encoder function as Z = Encoder(Input) and the Decoder function as Predict = Decoder(Z), which can also be regarded as Z = Decoder(Predict). Due to the similarity of Encoder and Decoder networks, the present invention maintains consistent design principles in the mixed-extract feature module. Specifically, when processing features, the Encoder part first performs convolution operations and then applies an adaptive attention mechanism, while the Decoder part first uses an adaptive attention mechanism and then performs convolution operations. The calculation process of the adaptive attention mechanism is as follows:

[0028] First, the input features are represented as a tensor with shape (B, H c ,W c ), where B is the batch size, H c and W c are the height and width of the features, respectively.

[0029] To reduce the memory consumption of attention calculation, the present invention divides the input features into token x token dimensions, divides H c and W c by token, respectively, to obtain a tensor with shape (B, (H c / token) x (W c / token), token x token). This way, the features are split into smaller chunks, making it easier for subsequent computations.

[0030] However, directly computing attention on the token x token dimension can lead to memory overflow. To solve this problem, the invention introduces a hidden layer hidden_size, which reduces the dimension of the features by adding a hidden layer. The token x token dimension is reduced to hidden_size, resulting in a feature representation of shape (B, (H c / token) x (W c / token), hidden_size), which reduces the computational load and memory consumption. Next, the invention computes attention on the reduced dimension features. First, the features are linearly mapped to Q, K, V tensors, representing query, key, and value, respectively. Then, the attention layer output Y = sof tmax(Q, K) V is computed using the softmax function, resulting in an attention layer output of shape (B, (H c / token) x (W c / token), hidden_size).

[0031] Finally, the invention restores the attention layer output Y to the original dimension, converting the tensor of shape (B, (H c / token) x (W c / token), hidden_size) to a feature representation of shape (B, H c , W c ). In this way, the invention obtains a feature representation that has been processed by an adaptive attention mechanism, which reduces the computational load of the network by adding a hidden layer.

[0032] Through the above calculation steps, the adaptive attention mechanism can reduce the memory consumption of attention computation while using smaller tokens. This mechanism has important application value in processing image tasks, as it can reduce computational complexity while maintaining accuracy, making model training and inference more convenient and efficient.

[0033] Step 3-3, the down-sampling module uses max-pooling.

[0034] Step 3-4, the up-sampling module uses bilinear interpolation.

[0035] Step 3-5, the SMN neural network is built using the relative position encoding module, mixed feature extraction module, down-sampling module, and up-sampling module designed in steps 3-1 to 3-4.

[0036] Step 3-6, the SMN network set in step 3-5 is trained and tested using the preset Straw320 dataset. Cross-entropy is used as the loss function, Adam optimizer is used, and learning rate cosine scheduler is used.

[0037] The straw image segmentation method based on the special neural network and image preprocessing provided by the application has the following technical effects:

[0038] 1. Improve image segmentation accuracy: through the grayscale method and the position encoding module, the application enhances the yellow channel information in the straw image, making the straw and the background more distinct. The position encoding module provides the relative position information of the straw and the background in the straw image, which helps the network better understand the spatial distribution of the straw, thereby improving the accuracy of image segmentation.

[0039] 2. Balance complexity and accuracy: the hybrid feature extraction module designed in the application combines convolution operation and adaptive attention mechanism, realizing effective extraction of local features and global features of straw images. By increasing the hidden layer and reducing the feature dimension, the attention mechanism improves the performance of image processing tasks while maintaining a small amount of calculation and memory consumption. This balanced design of complexity and accuracy makes the network efficient while obtaining accurate segmentation results.

[0040] 3. Improved U-Net structure: the application is improved based on the U-Net segmentation network structure, adding the position encoding module and the hybrid feature extraction module, which adapts to the characteristics of the straw image. Through the jump connection mechanism and the fusion and selection of multi-scale features, the network can better distinguish the straw and the background, improving the accuracy of the segmentation result.

[0041] 4. Efficient image processing: downsampling and upsampling operations are used to transfer information and recover features between the encoder and the decoder, speeding up the training and preserving more detailed information. The bilinear interpolation upsampling method can restore the spatial resolution of the features, maintaining the integrity of the image details.

[0042] 5. Comprehensive experimental verification: the application uses the preset Straw320 dataset for training and testing, uses cross-entropy as the loss function, and uses Adam optimizer and learning rate cosine scheduler. Through experimental verification, the method of the application has achieved good results in the task of straw image segmentation, verifying its accuracy and feasibility.

[0043] In summary, the straw image segmentation method proposed by the application can effectively realize accurate segmentation of straw and background through image preprocessing and special neural network design, improving the efficiency and accuracy of image processing. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A flowchart of a straw image segmentation method based on a special neural network and image preprocessing.

[0045] Figure 2 A flowchart of the SMN structure. DETAILED DESCRIPTION

[0046] The embodiments of the present application will be described in detail below with reference to the accompanying drawings, so that the purposes, technical solutions, characteristics and the like of the present application are easier to understand. Obviously, the described embodiments are only a part of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application. The embodiments are only used to explain the present application, but not to limit the present application.

[0047] The present application relates to the field of image processing, and in particular, to a straw image segmentation method based on a special neural network and image preprocessing. The method can be used for straw processing and utilization in the field of agriculture, providing accurate straw image segmentation results. The overall flowchart is shown in Figure 1 The specific implementation can be described by the following steps:

[0048] Step 1, collect color RGB images by a drone, and use random clipping technology to limit the image size to 512x384, and make a Straw320 dataset for subsequent network training and testing.

[0049] The images of the present application are collected by DJI drone MINI2 at a fixed height in a specific location. A total of 320 images with 4K resolution (1Kx400) are collected. Due to hardware limitations, it is not possible to input complete 4K data to the graphics card for training at the same time. Therefore, for each image, the present application randomly clips a region of 384x512 pixels, and makes a corresponding Mask label according to the region. In the label, "1" represents the straw part, and "0" represents the background (land) part.

[0050] In order to conduct further experiments and evaluation, the present application randomly divides the dataset into a training set, a validation set and a test set according to a ratio of 6:2:2. Such division helps to monitor the performance of the model during training, and to evaluate its generalization ability on new data.

[0051] In order to improve the effect and convergence speed of model training, the present application normalizes the original RGB data, limiting the pixel value to the range of (0, 1). Such preprocessing step helps to eliminate the scale difference in the data, making it easier for the model to learn meaningful features.

[0052] Step 2: A straw image grayscale method is designed to maintain the discriminability of the image while reducing the complexity of the subsequent model.

[0053] In the task of straw image segmentation, color images captured by unmanned aerial vehicle cameras or robot cameras are usually used as input. A color image consists of three channels: red (R), green (G), and blue (B). Each channel's pixel value represents the intensity of that color in the image. However, for the task of straw image segmentation, the invention focuses on the discriminability between straw and background, rather than the detailed information of color. Therefore, converting the color image to a grayscale image can reduce the complexity of the subsequent network while retaining sufficient discriminability.

[0054] Straw is usually yellow, and the background is usually black. To improve the discriminability between straw and background and reduce the complexity of the network, according to the corresponding value of yellow in the RGB space (255, 255, 0), the following grayscale coefficients can be obtained: 0.5, 0.5, 0. This means that the pixel values of the red channel and the green channel are weighted and averaged, while the pixel values of the blue channel are ignored. Therefore, the invention can use the following grayscale formula:

[0055] P = 0.5 * R + 0.5 * G + 0 * B

[0056] In the formula, P represents the pixel value after grayscale, and R, G, and B represent the pixel values of the red, green, and blue channels in the original color image, respectively. In this way, the invention enhances the information of the yellow channel while reducing the complexity of the network processing, making the subsequent image segmentation task more efficient and accurate.

[0057] Step 3: A straw mixed network (SMN) is proposed to process the preprocessed image. The network includes a position encoding module to enhance the relative information of the straw image, and a mixed feature extraction module that balances complexity and accuracy.

[0058] The SMN network draws on the classic U-Net segmentation network structure and makes some key improvements to better adapt to the characteristics of straw images. The overall structure is as follows: Figure 2As shown in the figure, the numbers in the figure represent the feature dimensions input into the module. First, the present application introduces a position encoding module to provide the position information of the straw and the background in the straw image. By introducing position information, the network can better understand the spatial distribution of the straw and improve the accuracy of segmentation. Second, the present application designs a hybrid feature extraction module that balances complexity and accuracy. This module combines the skip connection mechanism of U-Net to maintain controllable network complexity while extracting multi-scale feature information. By integrating features of different scales and performing feature fusion and selection, the network can better distinguish straw and background and improve the accuracy of the segmentation result. Finally, the SMN network uses down-sampling and up-sampling operations to transfer information and recover features between the encoder and the decoder. The down-sampling operation reduces the feature size and the number of parameters through the max-pooling layer to speed up the training. The up-sampling operation uses bilinear interpolation to restore the spatial resolution of the features and retains more detailed information.

[0059] Step 3-1, relative position encoding module. In the field of image processing, there is rich interaction information between pixels, and effective extraction of point-to-point information can improve the accuracy of the overall task. However, the traditional convolution operation is limited by the size of the convolution kernel and cannot fully extract the mutual correlation features between pixels that are too far apart, which is one of the reasons for the poor segmentation effect at present.

[0060] The present application proposes a relative position encoding module that couples relative position coordinates with the original image in a simple way. In order to reduce the complexity of the algorithm, the present application divides the original image into small blocks (i.e. patches) of the same size, and then fuses the relative position information with the pixel values of each small block. Assuming that the input image has a width of W and a height of H, and is divided into small blocks of patch_size x patch_size. Then the horizontal and vertical coordinate ranges of each small block can be expressed as:

[0061] col_indices=[0,1,...,W / (patch_size-1)]

[0062] row_indices=[0,1,...,H / (patch_size-1)]

[0063] In order to perform relative position encoding, the present application defines two encoding values, col_co and row_co, which represent the relative positions of the columns and rows, respectively. These encoding values are obtained by dividing col_indices and row_indices by (W / patch_size) and (H / patch_size), respectively, and the formula is as follows:

[0064] col_co=col_indices / (W / patch_size)

[0065] row_co = row_indices / (H / patch_size)

[0066] Then, the application adds these encoding values to the original data to fuse relative position information. Since the original data is normalized and the value range is between (0, 1), directly adding col_co and row_co encoding may mask the original image data information, leading to data imbalance, and further affecting the convergence of the network. Therefore, the application introduces a flexible parameter β to limit the influence of relative position information, as follows:

[0067] P' = P + β (col_co + row_co)

[0068] where P ∈ R W×H×1 , P' ∈ R W×H×1 .

[0069] Step 3-2, the mixed feature extraction module (Mixed-extract feature module) plays a key role in the straw segmentation task, aiming to achieve a balance between accuracy and complexity. To address this 12 challenge, the application proposes a mixed feature extraction module based on the design principle of Encoder-Decoder, which combines convolutional operations and adaptive attention mechanisms to achieve more effective image feature extraction.

[0070] The module first uses convolutional operations to extract local features from the image. Convolutional operations can capture the spatial relationships between adjacent features in the image and maintain the feature dimension unchanged. In this way, the application can fully utilize the local information of the image and extract rich local features.

[0071] Secondly, the mixed feature extraction module introduces an adaptive attention mechanism to extract the relevance between global features of the image. The adaptive attention mechanism can automatically adjust the importance of different features by learning the weights between features, so as to more accurately capture the global context information of the image. This global feature extraction capability enables the module to better understand the relevance between different regions in the image, thereby improving the performance of image processing tasks. In order to reduce the complexity of attention calculation, the application proposes to increase the number of hidden layers to achieve a balance between accuracy and complexity.

[0072] In the design of the module, the application simplifies the Encoder function as Z = Encoder (Input) and the Decoder function as Predict = Decoder (Z), which can also be regarded as Z = Decoder (Predict). Due to the similarity of the Encoder and Decoder networks, the application maintains the consistent design principle in the hybrid feature extraction module. Specifically, when processing features, the Encoder part first performs convolution operation and then applies adaptive attention mechanism; while the Decoder part first uses adaptive attention mechanism and then performs convolution operation.

[0073] 1. The convolution layer is a convolution operation with a dilated convolution kernel. Dilated convolution is a convolution operation that introduces a dilation factor to increase the receptive field range based on traditional convolution. Dilated convolution can help the network increase the perception area while maintaining computational efficiency, and is suitable for processing images or features with large-scale spatial structure.

[0074] The calculation formula of dilated convolution is as follows:

[0075]

[0076] Wherein:

[0077] · y[i,j] is the value of the i,j position of the output feature map.

[0078] · x[i+r·k,j+r·l] is the value of the i+r·k,j+r·l position of the input feature map, where r is the dilation factor.

[0079] · w[k,l,m] is the k,l,m weight value of the convolution kernel, and K,L,M is the size of the convolution kernel.

[0080] The main feature of dilated convolution is to expand the receptive field range by introducing a dilation factor r. Specifically, when r = 1, dilated convolution degenerates into traditional convolution operation; when r > 1, there will be a gap between the sampling points on the input feature map, so that the receptive field is expanded. Therefore, by adjusting the size of the dilation factor, the perception range can be increased while maintaining the computational efficiency.

[0081] 2、Adaptive attention layer: ViT usually divides the input image into multiple groups of tokens and uses adaptive attention mechanism to calculate the correlation between these tokens to extract the effective information of the input features. However, if the size of the token is set too small, it may cause excessive attention calculation, resulting in memory overflow and other problems. To solve this problem, the present invention proposes an adaptive attention mechanism. This mechanism reduces the intermediate dimension by increasing the number of hidden layers, thereby directly reducing the complexity of attention calculation while maintaining a small token size.

[0082] The calculation process of the adaptive attention mechanism is as follows:

[0083] First, the input feature is represented as a tensor with shape (B, H c ,W c ), where B is the batch size, H c and W c are the height and width of the feature, respectively.

[0084] To reduce the memory consumption of attention calculation, the present invention divides the input feature into tokenxtoken dimensions, and divides H c and W c by token, respectively, to obtain a feature representation with shape (B, (H c / token)x(W c / token), tokenxtoken). This way, the feature can be divided into smaller blocks for subsequent calculations.

[0085] However, directly performing attention calculation on the tokenxtoken dimension may cause memory overflow. To solve this problem, the present invention introduces a hidden layer hidden_size, which reduces the dimension of the feature by increasing the hidden layer. Reduce the dimension of tokenxtoken to hidden_size to obtain a feature representation with shape (B, (H c / token)x(W c / token), hidden_size), which can reduce the calculation amount and memory consumption.

[0086] Next, the present invention performs attention calculation on the reduced dimension feature. First, linearly map the feature to Q, K, and V tensors, representing query, key, and value, respectively. Then, calculate the attention layer output Y = sof tmax(Q, K)V by the softmax function to obtain a shape (B, (H c / token)x(W cthe attention layer output of shape (B, (H

[0087] Finally, the application restores the attention layer output Y to the original dimension, and the shape of (B, (H c / token) x (W c / token), hidden_size) is converted into a feature representation of shape (B, H c , W c ). In this way, the application obtains a feature representation processed by the adaptive attention mechanism, wherein the attention mechanism reduces the computational load of the network by increasing the hidden layer.

[0088] Through the above calculation steps, the adaptive attention mechanism can reduce the memory consumption of attention calculation while using smaller tokens. This mechanism has important application value in processing image tasks, can reduce the computational complexity while maintaining accuracy, and brings convenience and efficiency to model training and inference.

[0089] Step 3-3, downsampling module, the mathematical formula of using max-pooling in the downsampling module is as follows:

[0090] Let the input feature be X d , the output feature be Y d , and the dimension be H d x W d x C.

[0091] The mathematical formula of the max-pooling operation is as follows:

[0092] Y d [i,j,c] =

[0093] max(X d [2i,2j,c],X d [2i,2j+1,c],X d [2i+1,2j,c],X d [2i+1,2j+1,c])

[0094] where i and j represent the position index of the output feature map Y d , and c represents the channel index.

[0095] This formula describes the max-pooling operation, for each position (i, j) of the output feature map, the maximum value of the corresponding four input feature map positions is selected as the output value. In this way, the spatial dimension of the output feature map will be halved (height and width divided by 2), and the number of channels remains unchanged.

[0096] Step 3-4, the mathematical formula of Bilinear Interpolation used in the upsampling module is as follows:

[0097] Input feature map: X u , dimension H u ×W u ×C, where Hu is the height, Wu is the width, and C is the number of channels.

[0098] Output feature map of upsampling operation: Y u , dimension 2H u ×2W u ×C.

[0099] For each position (i,j) in the output feature map, the calculation is as follows:

[0100]

[0101] where c represents the channel index, and f(i,j,m,n) is the interpolation function of Bilinear Interpolation.

[0102] The calculation of Bilinear Interpolation function f(i,j,m,n) is as follows:

[0103]

[0104] This formula describes the Bilinear Interpolation operation, which calculates the output value for each position (i,j) in the output feature map by weighted average of the input feature map. The interpolation function is weighted according to the distance of the relative position, so as to obtain a smoother upsampling effect.

[0105] Step 3-5, build SMN neural network according to the relative position encoding module, mixed feature extraction module, downsampling module and upsampling module designed in 3-1 to 3-4. The SMN network configuration is as follows:

[0106] 1. Convert the input P∈R W×H×1 into P′∈R W×H×1 by using the relative position encoding module of step 3-1.

[0107] 2. Upgrade the dimension of input P′ to P″∈R W×H×64 by using the convolution module.

[0108] 3. Extract features from input P″ to P (3) ∈R W×H×64 by using the mixed feature extraction module of step 3-2.

[0109] 4. Reduce the dimension of input P (3) to

[0110] 5. Extract features from P(5) by using the mixed feature extraction module of step 3-2. (4) to P(6)

[0111] 6. Reduce dimensionality of P(6) by using the down-sampling module of step 3-3. (5) to P(7)

[0112] 7. Extract features from P(6) by using the mixed feature extraction module of step 3-2.

[0113] 8. Reduce dimensionality of P(7) by using the down-sampling module of step 3-3. (7) to P(8)

[0114] 9. Extract features from P(6) by using the mixed feature extraction module of step 3-2. (8) to P(9)

[0115] 10. Reduce dimensionality of P(9) by using the down-sampling module of step 3-3. (9) to P(10)

[0116] 11. Extract features from P(6) by using the mixed feature extraction module of step 3-2. (10) to P(11)

[0117] 12. Increase dimensionality of P(11) by using the up-sampling module of step 3-4. (11) to P(12) and jump-connect with P(8). (9)

[0118] 13. Extract features from P(6) by using the mixed feature extraction module of step 3-2. (12) to P(13)

[0119] 14. Increase dimensionality of P(13) by using the up-sampling module of step 3-4. (13) to P(14) and jump-connect with P(9). (7)

[0120] 15. Extract features from P(6) by using the mixed feature extraction module of step 3-2. (14) to P(15)

[0121] 16. Increase dimensionality of P(15) by using the up-sampling module of step 3-4. (15) to P(16)​​ and jump connection with P (5) Jump connection is performed.

[0122] 17, by using the mixed feature extraction module of step 3-2, input P (16) extract features to P

[0123] 18, by using the up-sampling module of step 3-4, input P (17 ) to P (18) ∈R W×H×64 , and jump connection with P(3).

[0124] 19, by using the mixed feature extraction module of step 3-2, input P (18) extract features to P (19) ∈R W×H×64 .

[0125] 20, by using the convolution module, input P (19) extract features to P (20) ∈R W×H×2 .

[0126] Step 3-6, the SMN network set by step 3-5 is trained and tested using the preset Straw320 data set. Cross entropy is used as the loss function, Adam optimizer is used, and learning rate cosine scheduler is used.

Claims

1. The present application proposes a straw image segmentation method based on a special neural network and image preprocessing, mainly including the following steps: Step 1, collect color RGB images by unmanned aerial vehicle, make Straw320 dataset, and normalize the dataset; Step 2, a straw image graying method is designed to maintain the discrimination of the image while reducing the complexity of the subsequent model. In this step, a straw image graying method is designed to maintain the discrimination of the image while reducing the complexity of the subsequent model; Straw is usually yellow, and the background is usually black. In order to improve the discrimination of straw and background, and reduce the complexity of the network, according to the corresponding value of yellow in RGB space 255, 255, 0, the following graying coefficients can be obtained: 0.5, 0.5, 0, which means that the pixel value of red channel and green channel are weighted average, and the pixel value of blue channel is ignored. The following graying formula can be used: P = 0.5×R + 0.5×G+ 0×B wherein P represents the pixel value after graying, R, G and B represent the pixel values of red, green and blue channels in the original color image respectively. Through the above method, the information of yellow channel is enhanced, and the complexity of network processing is reduced, so that the subsequent image segmentation task is more efficient and accurate; Step 3, a straw mixed network SMN is proposed to process the preprocessed image. The network includes a position encoding module to enhance the relative information of straw image, and a mixed feature extraction module to balance the complexity and accuracy. The SMN network draws on the classical U-Net segmentation network structure, Firstly, the position encoding module is introduced to provide the position information of straw and background in the straw image. By introducing the position information, the network understands the spatial distribution of straw, and improves the accuracy of segmentation; Secondly, through the mixed feature extraction module balancing complexity and accuracy, the module combines the skip connection mechanism of U-Net, extracts multi-scale feature information while keeping the network complexity controllable, and integrates and selects different scale features to make the network accurately distinguish straw and background, thereby improving the accuracy of segmentation results; Finally, the SMN network uses down-sampling and up-sampling operations to transfer information and recover features between encoder and decoder. Down-sampling operation reduces feature size and parameter quantity through max-pooling layer to speed up training, and up-sampling operation uses bilinear interpolation method to restore spatial resolution of features and retain more detailed information; Step 3-2, the mixed feature extraction module in straw segmentation task aims to balance the accuracy and complexity. According to the design principle of Encoder-Decoder, a mixed feature extraction module is proposed, which combines convolution operation and adaptive attention mechanism to realize more effective image feature extraction; The calculation process of the adaptive attention mechanism is as follows: Firstly, input features are represented as a tensor with shape (B, Hc, Wc), where B is the batch size, and Hc and Wc are the height and width of the features, respectively; In order to reduce the memory consumption of attention calculation, the input features are divided into tokenxtoken dimensions, and Hc and Wc are divided by token to obtain a feature representation with shape (B, (Hc / token)x(Wc / token), tokenxtoken), which can divide the features into smaller blocks for subsequent calculation; However, directly calculating attention on tokenxtoken dimensions may cause memory overflow, so a hidden layer hidden_size is introduced to reduce the dimension of the features by increasing the hidden layer to reduce the dimension of tokenxtoken to hidden_size, obtaining a feature representation with shape (B, (Hc / token)x(Wc / token), hidden_size) to reduce the amount of calculation and memory consumption; Next, the reduced dimension features are calculated by attention, first, the features are linearly mapped to Q, K, V tensors representing query, key and value, then the attention layer output Y=softmax(Q, K)V is calculated by the softmax function, obtaining an attention layer output with shape (B, (Hc / token)x(Wc / token), hidden_size); Finally, the attention layer output Y is restored to the original dimension, and the tensor with shape (B, (Hc / token)x(Wc / token), hidden_size) is converted to a feature representation with shape (B, Hc, Wc), so that the feature representation processed by the adaptive attention mechanism is obtained, and the attention mechanism reduces the computational complexity of the network by increasing the hidden layer; Through the above calculation steps, the adaptive attention mechanism can reduce the memory consumption of attention calculation while using smaller tokens, and this mechanism has important application value in processing image tasks, which can reduce the computational complexity while maintaining accuracy, and brings convenience and efficiency to model training and inference.

2. The straw image segmentation method based on a special neural network and image preprocessing according to claim 1, characterized in that, Step 3-1, relative position encoding module, in the field of image processing, there is rich interaction information between pixels, and effective extraction of point-to-point information can improve the accuracy of the overall task; The relative position encoding module couples the relative position coordinates with the original image in a simple way. In order to reduce the complexity of the algorithm, the original image is divided into small patches of the same size, and then the relative position information is fused with the pixel value of each small patch. Assuming that the width of the input image is W, the height is H, and it is divided into small patches of patch_size x patch_size, then the horizontal and vertical coordinate range of each small patch can be expressed as: col_indices=[0,1,...,W / (patch_size−1)] row_indices [0,1,...,H / (patch_size −1)] In order to encode the relative position, two encoding values, col_co and row_co, are defined, which represent the relative position of the column and row respectively. These encoding values are obtained by dividing col_indices and row_indices by (W / patch_size) and (H / patch_size), respectively, as follows: col_co = col_indices / (W / patch_size) row_co = row_indices / (H / patch_size) Then, these encoding values are added to the original data to fuse the relative position information. Since the original data has been normalized and the value range is between 0 and 1, directly adding col_co and row_co encoding may mask the original image data information, leading to data imbalance and affecting the convergence of the network. Therefore, a flexible parameter β is introduced to limit the influence of the relative position information, as follows: P=P+β(col_co+row_co) where P∈ RW ×H×1, P′∈RW ×H×1.

3. The straw image segmentation method based on a special neural network and image preprocessing according to claim 1 or 2, characterized in that, In the design of the module, the Encoder function is simplified as Z = Encoder(Input), and the Decoder function is Predict = Decoder(Z), which can also be regarded as Z = Decoder(Predict). Due to the similarity of the Encoder and Decoder networks, the consistent design principle in the hybrid feature extraction module is maintained. In processing features, the Encoder part first performs convolution operation, and then applies the adaptive attention mechanism; while the Decoder part first uses the adaptive attention mechanism, and then performs convolution operation.

Images