Rice image segmentation model construction method
By combining the FPM module and the Swin Transformer module, using the multi-scale self-attention module for feature fusion, the problem of Swin Transformer insufficient local feature capture in complex adhesion rice segmentation tasks is solved, and higher segmentation accuracy and semantic expression capabilities are achieved.
Patent Information
- Application Number
- CN202411954772.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-22
AI Technical Summary
In the prior art, Swin Transformer is used to achieve the feature extraction method in complex adhesion rice segmentation tasks, which weakly captures subtle local features, resulting in the problem of inaccurate segmentation.
By constructing a rice image segmentation model, using the combination of FPM module and Swin Transformer module, features of different scales are processed separately, and features are fused through multi-scale self-attention modules to improve segmentation accuracy.
It effectively improves the segmentation accuracy of adhesion rice, solves the problem of undersegment when adhesion is severe or blurred boundaries, balances details and overall structure, and improves the model's semantic expression and recognition ability in adhesion rice segmentation.
Smart Images
Figure CN119863789A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of grain quality detection, and in particular relates to a method for constructing a rice image segmentation model. Background Art
[0002] Strengthening grain quality control and optimizing its production and processing process are of great significance to improving the efficiency of grain production. Especially in the field of rice quality inspection, accurate appearance inspection is the key to ensuring product quality and improving production efficiency. However, most rice appearance inspection instruments on the market rely on vertical separators to pre-process the spatial position of sample grains. This method is highly dependent on human intervention and cannot meet the automation needs in actual inspections. Rice is usually scattered randomly in its natural state, with characteristics such as similar shapes, varying degrees of adhesion, and blurred boundaries. Traditional segmentation algorithms, such as algorithms based on a priori designs such as watersheds, thresholds, and edges, are difficult to handle the complex situation of sticky rice, and often have problems with under-segmentation or over-segmentation. How to solve the problem of rice adhesion through advanced image segmentation technology is a key link in improving the level of intelligence and automation in rice appearance inspection.
[0003] In recent years, convolutional neural networks (CNNs), as an important method of deep learning, have performed well in tasks such as image classification, object detection, and image segmentation. The two-stage detection model represented by Mask R-CNN generates candidate regions through a region proposal network, and then predicts categories, bounding boxes, and binary masks. Although models such as PANet, Mask Score R-CNN, and Cascade Mask R-CNN have made significant progress in instance segmentation, there are still problems with inaccurate edges and difficulty in capturing global relationships in the segmentation task of adherent rice. In addition, CNN has limitations in capturing global features and is difficult to effectively deal with the rice segmentation task with complex random scattering and adhesion situations. In contrast, the Transformer model shows unique advantages in global feature modeling and complex spatial relationship processing. In particular, the Swin Transformer effectively improves the processing efficiency and performance of high-resolution images by introducing a sliding window mechanism and hierarchical design, and is suitable for the feature extraction part of small target detection and instance segmentation tasks in adherent scenes.
[0004] However, although Swin Transformer has significant advantages in global feature modeling and processing complex adhesion rice segmentation tasks, it still faces challenges in capturing fine-grained local features, computational efficiency, and accurate segmentation of small targets. For example, although Swin Transformer has a strong ability to capture global information, it is weak in capturing subtle local features, which easily leads to inaccurate segmentation when the extracted features are applied to process the adhesion boundaries of rice grains. Summary of the invention
[0005] The object of the present invention is to provide a rice image segmentation model construction method, which is used to solve the problem that the method of using Swin Transformer in the prior art to realize feature extraction in complex adhesion rice segmentation tasks is weak in capturing subtle local features, so that the extracted features are applied to process the adhesion boundaries of rice grains, resulting in inaccurate segmentation.
[0006] In order to achieve the above object, the present invention provides a method for constructing a rice image segmentation model, wherein the generated rice image segmentation model is trained according to the rice images and segmentation labels in the training set to obtain the constructed model; the processing steps after the rice image is input into the model include:
[0007] 1) Take the first feature layer as the current layer; extract features from the input image to obtain adjacent feature maps of different scales as the input of the current layer;
[0008] 2) The FPM module of the current layer processes the input of the current layer to obtain enhanced images of different scales, which are fused with the output of the Swin Transformer module of the previous feature layer through the feature fusion of the first fusion module of the current layer and then with the output of the Swin Transformer module of the previous feature layer, and then input into the Swin Transformer module of the current layer, and then fused with the enhanced image of the current layer through the feature fusion of the second fusion module of the current layer and then with the enhanced image of the corresponding scale of the current layer to obtain the output of the current layer; if there is no output of the Swin Transformer module of the previous feature layer, the result of the input image after path encoding and position encoding is used;
[0009] If the next feature layer does not exist in the current layer, instance segmentation is completed according to the fusion result of the output of the current layer and the output of the Swin Transformer module of the current layer; otherwise, the output of the current layer is used as the input of the next feature layer; the next feature layer is used as the new current layer, and 2) is repeated until instance segmentation is completed;
[0010] The FPM module processes the input of the current layer in the following manner: using convolution kernels of different scales to process features of different scales corresponding to the input of the current layer.
[0011] Furthermore, at least one feature fusion method in the first and second fusion modules includes: fusing the output of the Swin Transformer module of the previous feature layer with an enhancement map of a set scale in enhancement maps of different scales, and processing the fused enhancement map and other enhancement maps through multi-scale self-attention modules respectively.
[0012] Furthermore, the number of adjacent feature maps of different scales is equal to 3.
[0013] Furthermore, the scales of the convolution kernels of different scales used are 3×3, 5×5, and 7×7.
[0014] Furthermore, a method of completing instance segmentation according to the fusion result includes: performing instance segmentation on the fusion result through an instance segmentation framework including an FPN module, a ROIAlign module, a Box branch, and a Mask branch to obtain an instance segmentation result.
[0015] Furthermore, the method of inputting the rice image into the model includes: converting the rice image to be input into the rice image segmentation model into a tensor format and then inputting the format into the model.
[0016] Furthermore, the method for obtaining the rice image to be input into the rice image segmentation model includes: adjusting the collected original rice image to be segmented to a uniform size, normalizing the pixel values of the resized image, and then performing image enhancement on the processed image to obtain the rice image to be input into the rice image segmentation model.
[0017] Furthermore, the image enhancement includes performing one or more enhancement operations of random flipping, rotation, blurring and brightness adjustment on the image.
[0018] Furthermore, the method of training the generated rice image segmentation model according to the rice images and segmentation labels in the training set includes: inputting the rice images in the training set into the rice image segmentation model to obtain the instance segmentation result and the value of the loss function obtained by the segmentation label, and using the AdamW optimization algorithm to iteratively update the model parameters of the rice image segmentation model during the training process.
[0019] Furthermore, the methods of using convolution kernels of different scales to process features of different scales corresponding to the input of the current layer include:
[0020] The different scale features corresponding to the input of the current layer are processed by linear layers respectively, and then convolutional layers containing deep convolution kernels of different scales are used to enhance the results of processing the different scale features by linear layers respectively, and then the enhanced results of different scales are processed by linear layers again.
[0021] The above technical solution of the present invention provides a new rice image segmentation model construction method, and its beneficial effects include:
[0022] The rice image segmentation model set up can process features of different scales through the convolution kernels of different scales of the FPM module to enhance the context feature capture capability. Thus, the FPM module can enhance the capture capability of different rice grain sizes and shapes, and can effectively capture fine-grained local information, effectively improve the segmentation accuracy of sticky rice, and solve the problem of under-segmentation when the sticking is serious or the boundary is blurred. At the same time, based on the original Swin Transformer module network structure, two fusion modules are used to respectively realize the fusion of the features extracted by the Swin Transformer module of the previous feature layer and the FPM module of the current feature layer for feature extraction of the Swin Transformer module of the current feature layer, and the fusion of the features extracted by the Swin Transformer module of the current feature layer and the FPM module of the current feature layer for feature extraction of the FPM module of the next feature layer. Thus, each feature layer based on the Swin Transformer module can fully combine the fine-grained local features extracted by the FPM module and the global features extracted by the Swin Transformer module, and the Swin Transformer modules of other feature layers after the first feature layer can also combine the local features extracted by the FPM module of the feature layer itself and the Swin of the previous feature layer. The fusion features of the global features extracted by the Transformer module are used for global feature extraction, which not only balances the details and the overall structure and improves the semantic expression and recognition ability of the model in the segmentation of adhered rice, but also integrates the features of different feature layers so that the extracted features contain more details of different scales. Therefore, it can solve the problem that the feature extraction method in the complex adhesion rice segmentation task has weak capture of subtle local features, which makes the extracted features not accurate enough when applied to process the adhesion boundaries of rice grains, and at the same time does not pay too much attention to the detailed features that lead to mis-segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A flowchart of a rice image segmentation model in an embodiment of a method for constructing a rice image segmentation model of the present invention;
[0024] Figure 2 It is a schematic diagram of the overall principle of the rice image segmentation model in the rice image segmentation model construction method embodiment of the present invention;
[0025] Figure 3 A schematic diagram of the network structure principle of a rice image segmentation model in an embodiment of a method for constructing a rice image segmentation model of the present invention;
[0026] Figure 4 A schematic diagram of the structural principles of the first and second fusion modules in an embodiment of the method for constructing a rice image segmentation model of the present invention;
[0027] Figure 5 The figure is a schematic diagram of the structural principle of the FPM module in the embodiment of the method for constructing a rice image segmentation model of the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0029] Rice image segmentation model construction method embodiment
[0030] The present embodiment provides a technical solution for a method for constructing a rice image segmentation model. The method is implemented by training a rice image segmentation model. The rice image segmentation model is based on an existing Swin Transformer module framework. The FPM module that can extract more detailed local features through different convolution scales is provided to capture fine-grained local information and improve the accuracy of boundary segmentation. A fusion unit that is provided in a feature layer to fuse the global features of the Swin Transformer module and the local features of the FPM module is used to achieve a balance between local features and global features, so that the extracted features can improve segmentation accuracy when applied to complex adhesion rice segmentation tasks.
[0031] Reference Figure 1 , the generated rice image segmentation model is trained according to the rice images and segmentation labels in the training set to obtain the constructed rice image segmentation model; the processing steps after the data corresponding to the rice image is input into the rice image segmentation model refer to Figure 2 ,include:
[0032] 1) The first feature layer (i.e. Figure 2 The content in the bottom dashed box in Figure 2 Each dotted box in the figure represents the content of a feature layer) as the current layer; extract features from the input image to obtain adjacent feature maps of different scales as the input of the current layer;
[0033] 2) The FPM module of the current layer processes the input of the current layer to obtain enhanced images of different scales, which are passed through the first fusion module of the current layer (i.e. Figure 2 The features of the FIF module in the lower part of each dotted box in the figure are fused and then fused with the output of the Swin Transformer module of the previous feature layer, input into the Swin Transformer module of the current layer, and then pass through the second fusion module of the current layer (i.e. Figure 2The features of the FIF module (FIF module in each dotted box above) are fused and then fused with the enhancement map of the corresponding scale of the current layer to obtain the output of the current layer; if there is no output of the Swin Transformer module of the previous feature layer, the result of the input image after path encoding (Path Embedding) and position encoding (Position Embedding) is used as the output of the Swin Transformer module of the previous feature layer;
[0034] If the next feature layer does not exist in the current layer, instance segmentation is completed according to the fusion result of the output of the current layer and the output of the Swin Transformer module of the current layer; otherwise, the output of the current layer is used as the input of the next feature layer; the next feature layer is used as the new current layer, and 2) is repeated until instance segmentation is completed;
[0035] The FPM module processes the input of the current layer in the following manner: using convolution kernels of different scales to process features of different scales corresponding to the input of the current layer.
[0036] The rice image segmentation model construction method can process features of different scales through the convolution kernels of different scales of the FPM module through the set rice image segmentation model to enhance the context feature capture capability. In summary, it can effectively improve the segmentation accuracy of adhered rice and solve the problem of under-segmentation when adhesion is severe or the boundary is blurred. At the same time, based on the original Swin Transformer module network structure, two fusion modules are used to respectively realize the fusion of the features extracted by the Swin Transformer module of the previous feature layer and the FPM module of the current feature layer for feature extraction of the Swin Transformer module of the current feature layer, and the fusion of the features extracted by the Swin Transformer module of the current feature layer and the FPM module of the current feature layer for feature extraction of the FPM module of the next feature layer. Therefore, each feature layer based on the Swin Transformer module can fully combine the fine-grained local features extracted by the FPM module and the global features extracted by the Swin Transformer module, and the Swin Transformer modules of other feature layers after the first feature layer can also combine the fusion features of the local features extracted by the FPM module containing the feature layer itself and the global features extracted by the Swin Transformer module of the previous feature layer to extract global features, which not only balances the details and the overall structure, improves the semantic expression and recognition ability of the model in the segmentation of adhered rice, but also integrates the features of different feature layers, so that the extracted features contain more details of different scales. In addition, after analysis, the Swin Transformer module may have a reduced segmentation accuracy when facing severely sticky millet grains due to the limited processing ability of the sliding window mechanism for small targets; therefore, in this embodiment, refer to Figure 4, the feature fusion method of at least one of the first and second fusion modules includes: fusing the output of the Swin Transformer module of the previous feature layer with the enhancement map of a set scale in the enhancement maps of different scales, and processing the fused enhancement map and other enhancement maps through the multi-scale self-attention module respectively. Therefore, by introducing the multi-scale self-attention mechanism into the fusion module, the small target segmentation performance can be effectively improved, the shortcomings of the sliding window mechanism of the Swin Transformer module in the processing ability of small targets can be compensated as much as possible, and the model's recognition ability of highly adhered millet grains can be enhanced. In addition, since the FIF module efficiently integrates features at different levels through cross-scale feature fusion instead of processing the features of each layer separately, this fusion method reduces the repeated operation of a single feature map, thereby reducing the computational complexity; on this basis, the FIF module also adopts a multi-scale self-attention mechanism for selective weighting instead of simple addition or splicing, so that operation redundancy can be reduced: additional up and down sampling or layer-by-layer convolution operations are avoided, and memory overhead and computational complexity are reduced; and the use of the self-attention mechanism to weight features of different scales can highlight important features and weaken irrelevant features. This selective processing avoids the waste of calculation of invalid features and improves the computational efficiency of the model; Combining the above advantages, the first and second fusion modules (i.e., FIF modules) used are compared with the speed and efficiency of ordinary feature fusion modules to fuse Transformer features and CNN features. The computing resource usage is reduced by 30%, and the reasoning speed is increased by 40%, which enables the segmentation model constructed in this embodiment to meet the real-time detection requirements and enhance its practical application value.
[0037] Reference Figure 5 In this embodiment, the number of adjacent feature maps of different scales is equal to 3. This design ensures that the scales of the extracted local details are more, and the network complexity and computational cost are lower. After analysis, the scales of the three convolution kernels of different scales used to process the enhanced images of different scales are 3×3, 5×5 and 7×7 respectively. In this case, the effect of local detail extraction is relatively the best. In addition, the method of extracting features from the input image is to downsample layer by layer to obtain a feature pyramid. The feature maps corresponding to the adjacent downsampling layers in the feature pyramid are the adjacent feature maps of different scales. In this embodiment, the feature maps of adjacent different scales use C3, C4 and C5, that is, the feature maps extracted after 3 layers of downsampling, 4 layers of downsampling and 5 layers of downsampling respectively. The Stem convolution module (that is, Figure 3 The Stem convolution module uses three convolution layers (3×3 convolution) and a maximum pooling layer; in other embodiments, other existing feature extraction methods can also be used to obtain adjacent feature maps of different scales.
[0038] like Figure 5 As shown, the FPM module in this embodiment uses convolution kernels of different scales to process the different scale features corresponding to the input of the current layer. The specific method is to firstly process the different scale features corresponding to the input of the current layer through a linear layer (i.e. Figure 5 The linear mapping in , Linear-Projection) is processed, and then a convolution layer containing deep convolution kernels of different scales (i.e. Figure 5 DWConv in the above algorithm enhances the results of processing different scale features through linear layers respectively, and then processes the enhanced results of different scales through linear layers again.
[0039] In this embodiment, Figure 4 As shown, for the operation of fusing the output of the Swin Transformer module of the previous feature layer with the enhancement map of a set scale among enhancement maps of different scales in the FIF module, the enhancement map of the set scale adopted is the enhancement map F4 with the same scale as C4, that is, the enhancement map with the middle scale among the three adjacent enhancement maps of different scales; in other embodiments, other enhancement maps among the adjacent enhancement maps of different scales may also be selected, and no special limitation is made here.
[0040] Reference Figure 3 , ⊕ represents the element-by-element addition operation, where Figure 3 The element-by-element addition operation in the dashed box in the yellow background area on the right specifically refers to the element-by-element addition of each enhanced image output by the FPM module and the feature fusion result output by the FIF module with the same scale, for example Figure 5 The enhanced images F3, F4, and F5 output by the FPM module are respectively Figure 4 The feature fusion result S3 output by the FIF module with the same scale as F3, the feature fusion result S4 output by the FIF module with the same scale as F4, and the feature fusion result S5 output by the FIF module with the same scale as F5 are added element by element; Figure 3 The element-by-element addition operation in the dashed box in the blue background area on the left specifically refers to the element-by-element addition of the output of the Swin Transformer module of the previous feature layer and the feature fusion result of the FIF module with the same scale as the enhanced image of the set scale. For example, the output of the Swin Transformer module of the previous feature layer and Figure 5 The feature fusion result S4 output by the FIF module with the same scale as F4 is added element by element, while S3 and S5 do not perform the element-by-element addition operation.
[0041] Reference Figure 2The method of completing instance segmentation according to the fusion result includes: performing instance segmentation on the fusion result through an instance segmentation framework including an FPN module, an ROIAlign module, a Box branch, and a Mask branch to obtain an instance segmentation result. Since the instance segmentation framework including the FPN module, the ROIAlign module, the Box branch, and the Mask branch belongs to the prior art, it will not be described here.
[0042] In the present embodiment, for the convenience of inputting the data corresponding to the rice image into the neural network for feature extraction, the method of inputting the rice image into the rice image segmentation model in the present embodiment includes: after the rice image to be input into the rice image segmentation model is converted into a tensor format, the rice image segmentation model is input. Thus, the input data can be more matched with the rice image segmentation model (neural network), and the effect of optimizing the rice image segmentation model processing the data corresponding to the input rice image.
[0043] Furthermore, in order to improve the diversity of data and the training effect of the model, the method of obtaining the rice image to be input into the rice image segmentation model includes: adjusting the collected original rice image to be segmented to a uniform size, and normalizing the pixel values of the resized image, and then performing image enhancement on the processed image, including performing one or more enhancement operations of random flipping, rotation, blurring and brightness adjustment on the image, and finally obtaining the rice image to be input into the rice image segmentation model. Specifically, firstly, the collected adhesion rice image is adjusted to a uniform size to ensure the standardization of the input data; then the pixel values of each image of the uniform size are normalized to scale the range of its pixel values from [0,255] to [0,1], thereby improving the training stability and convergence speed of the model; finally, a variety of enhancement operations are performed on the normalized image, including random flipping, rotation, blurring, brightness adjustment, etc., to increase the diversity of the training set and help the model improve the generalization ability of new samples.
[0044] In the present embodiment, the process of training the rice image segmentation model generated according to the rice image and the segmentation label in the training set is specifically: according to the rice image in the training set, the instance segmentation result obtained by the rice image segmentation model is input and the value of the loss function obtained by the segmentation label, and the model parameters of the rice image segmentation model are iteratively updated during the training process using the AdamW optimization algorithm. The AdamW optimization algorithm is the optimization algorithm that has the best training effect on the generated rice image segmentation model after analysis. In other embodiments, other iteratively updated optimization algorithms can also be used to achieve the training of the rice image segmentation model.
[0045] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, but do not constitute a limitation to the present invention.
Claims
1. A rice image segmentation model construction method, characterized in that, The generated rice image segmentation model is trained according to the rice images and segmentation labels in the training set to obtain the constructed model; the processing steps after the rice image is input into the model include: 1) Take the first feature layer as the current layer; extract features from the input image to obtain adjacent feature maps of different scales as the input of the current layer; 2) The FPM module of the current layer processes the input of the current layer to obtain enhanced images of different scales, which are fused with the output of the Swin Transformer module of the previous feature layer through the feature fusion of the first fusion module of the current layer and then with the output of the Swin Transformer module of the previous feature layer, and then input into the Swin Transformer module of the current layer, and then fused with the enhanced image of the current layer through the feature fusion of the second fusion module of the current layer and then with the enhanced image of the corresponding scale of the current layer to obtain the output of the current layer; if there is no output of the Swin Transformer module of the previous feature layer, the result of the input image after path encoding and position encoding is used; If the next feature layer does not exist in the current layer, instance segmentation is completed according to the fusion result of the output of the current layer and the output of the Swin Transformer module of the current layer; otherwise, the output of the current layer is used as the input of the next feature layer; the next feature layer is used as the new current layer, and 2) is repeated until instance segmentation is completed; The FPM module processes the input of the current layer in the following manner: using convolution kernels of different scales to process features of different scales corresponding to the input of the current layer.
2. The rice image segmentation model construction method according to claim 1, characterized in that, The method of fusing at least one feature in the first and second fusion modules includes: fusing the output of the Swin Transformer module of the previous feature layer with the enhanced map of a set scale in the enhanced maps of different scales, and processing the fused enhanced map and the other enhanced maps through the multi-scale self-attention module respectively.
3. The rice image segmentation model construction method according to claim 1 or 2, characterized in that, The number of adjacent feature maps of different scales is equal to 3.
4. The rice image segmentation model construction method according to claim 3, wherein The convolution kernels of different scales used are 3×3, 5×5 and 7×7.
5. The rice image segmentation model construction method according to claim 1 or 2, characterized in that, The method of completing instance segmentation according to the fusion result includes: performing instance segmentation on the fusion result through an instance segmentation framework including an FPN module, a ROIAlign module, a Box branch and a Mask branch to obtain an instance segmentation result.
6. The rice image segmentation model construction method according to claim 1 or 2, characterized in that, The method of inputting the rice image into the model includes: converting the rice image to be input into the rice image segmentation model into a tensor format and then inputting the format into the model.
7. The rice image segmentation model construction method according to claim 6, characterized in that: The method of obtaining a rice image to be input into a rice image segmentation model includes: adjusting the collected original rice image to be segmented to a uniform size, normalizing the pixel values of the resized image, and then performing image enhancement on the processed image to obtain the rice image to be input into the rice image segmentation model.
8. The rice image segmentation model construction method according to claim 7, characterized in that: The image enhancement includes performing one or more enhancement operations of random flipping, rotation, blurring and brightness adjustment on the image.
9. The rice image segmentation model construction method according to claim 1 or 2, characterized in that, The method of training the generated rice image segmentation model according to the rice images and segmentation labels in the training set includes: inputting the rice images in the training set into the rice image segmentation model to obtain the instance segmentation result and the value of the loss function obtained by the segmentation label, and using the AdamW optimization algorithm to iteratively update the model parameters of the rice image segmentation model during the training process.
10. The rice image segmentation model construction method according to claim 1 or 2, characterized in that, The methods of using convolution kernels of different scales to process the different scale features corresponding to the input of the current layer include: The different scale features corresponding to the input of the current layer are processed by linear layers respectively, and then convolutional layers containing deep convolution kernels of different scales are used to enhance the results of processing the different scale features by linear layers respectively, and then the enhanced results of different scales are processed by linear layers again.