Grain grain segmentation method and system based on vector polygon attention mechanism

Abstract

The invention relates to a grain kernel segmentation method and system based on a vector polygon attention mechanism, and belongs to the technical field of grain intelligent processing. According to the method, a visual Mangbar network is adopted as a basic network architecture to construct a segmentation model, a vector polygon attention mechanism is added in an encoder part in the visual Mangbar network, and modeling is performed on a grain grain image through the vector polygon attention mechanism to obtain a geometrical relationship and feature interaction in a local polygon region, so that the image segmentation accuracy is improved. According to the method, the focus of the encoder on the key area in the image is enhanced, the boundary and shape information of the grain grains can be extracted more accurately, feature representation with higher discrimination is provided for a subsequent segmentation task, and the segmentation precision of the grain grains is further improved.

Description

TECHNICAL FIELD

[0001] The application relates to a grain kernel segmentation method and system based on a vector polygon attention mechanism, and belongs to the technical field of intelligent grain processing. BACKGROUND

[0002] During the image acquisition of corn kernels, external factors such as changes in environmental light, slight disturbances of the test table and calibration errors of the equipment can easily interfere with the acquisition system, thereby affecting the image data quality and system stability, and limiting the development of kernel quality detection towards high-efficiency automation. Traditional quality evaluation methods mainly rely on manual or semi-automatic means, such as subjective judgment by visually observing color, smelling odor, and manually judging fullness. Such methods not only have low detection efficiency, but are also easily affected by human experience and subjective bias, and have poor accuracy and consistency, making it difficult to meet the actual needs of large-scale detection and standardized evaluation.

[0003] With the continuous progress of sensing technology, optical imaging, computer technology, image processing methods and grain informatization technology, image segmentation, as a core technology in the field of computer vision, has been widely deployed in multiple application scenarios, especially in the field of grain information processing, showing a rapid development trend. Through image acquisition of corn kernels by image sensing devices, combined with image filtering, denoising and enhancement preprocessing methods, the image quality and subsequent analysis accuracy and stability can be significantly improved. On this basis, with the help of a classifier constructed by a deep neural network, multiple tasks such as automatic segmentation, classification, identification and quality evaluation of corn kernels can be realized, thereby effectively promoting the development of grain quality detection towards intelligence and automation. This technical system is gradually gaining widespread attention and in-depth application in practical application scenarios such as agricultural production, grain storage and transportation and commercial detection, showing good development prospects and practical value.

[0004] The existing classifier constructed by a deep neural network for self-supervised segmentation generally has weak structure perception ability and fuzzy target boundary, making it difficult to accurately depict targets with complex geometric shapes. In particular, when dealing with images of grains with clear edges and shape features, traditional methods are difficult to effectively model the polygon structure of the grains, especially in scenes where the surface texture of the grains is complex and the structural differences are subtle, which easily leads to fuzzy segmentation boundaries between target regions and background regions, thereby affecting the segmentation result and resulting in low grain kernel segmentation accuracy. SUMMARY

[0005] The purpose of the present application is to provide a grain kernel segmentation method and system based on a vector polygon attention mechanism to solve the problem of low segmentation accuracy caused by the inability to effectively model the polygon structure of the grain kernels in the current self-supervised segmentation.

[0006] The application provides a grain kernel segmentation method based on a vector polygon attention mechanism, and the method comprises the following steps:

[0007] 1) obtaining a grain kernel image to be segmented;

[0008] 2) establishing a segmentation model, and performing image segmentation on the obtained grain kernel image to be segmented by using the established segmentation model; the segmentation model adopts a visual Mamba network as a basic network architecture, a vector polygon attention mechanism is arranged in an encoder part in the visual Mamba network, the vector polygon attention mechanism is used for performing vector transformation on each grid divided in an input image, combining a vector transformation result with a window position, fusing a combination result with an original input image, and performing attention mechanism calculation on the image to obtain vector polygon attention, and a decoder in the visual Mamba network is used for decoding processing of features output by the encoder to obtain a segmentation result.

[0009] Further, the decoder in the visual Mamba network adopts a prototype-image alignment method: a feature vector representing each region is extracted from the features output by the encoder as a region prototype, a feature vector of each pixel in the image is extracted, each feature vector in the image is compared with the region prototype, and the pixel is classified into the region prototype closest to the pixel, so that semantic division is realized.

[0010] Further, the vector transformation is to convert each basic window into a vector quadrilateral by using projection transformation with the basic window as a reference.

[0011] Further, relative coordinates are used as the input of each window when the projection transformation is performed, and the relative coordinates refer to coordinates relative to the center of the window.

[0012] After obtaining the relative coordinates of each target quadrilateral, a grid-sampling function is used to extract features at corresponding positions from Kw and Vw in the window respectively to generate a query key K and a value V for attention calculation, and self-attention calculation is performed.

[0013] Further, if the learned quadrilateral covers a region outside the feature map, a regularization term is used to constrain the projected quadrilateral to cover more effective regions in the feature map.

[0014] Further, the region prototype is obtained by a mask average pooling method, a mask is used to select a region of interest, and features of the region of interest selected by the mask are subjected to average pooling, and the determination process of the region prototype is as follows:

[0015] The support features are converted into coarse prototypes by the mask average pooling operation.

[0016] A similarity graph is generated by using the similarity between the coarse prototype and the query feature, a coarse query mask is generated based on the similarity graph, and the query feature is converted into a query prototype by using mask average pooling;

[0017] The coarse prototype and the query prototype are weighted and summed, and the result is the final prototype, i.e., the region prototype.

[0018] Further, when comparing each feature vector in the image with the region prototype, the feature vector and the region prototype are aligned, which improves the accuracy of feature matching and the region recognition ability.

[0019] Further, the comparison process of each feature vector in the image with the region prototype includes:

[0020] The region prototype is taken as a query, the foreground support features are taken as key and value inputs, an attention matrix between the query and the key is calculated to measure the correlation between the region prototype and each foreground support feature.

[0021] Further, the loss function of the segmentation model at each training round is:

[0022] L = L proto + L align

[0023]

[0024] Wherein M final is the query prediction result.

[0025] The application also provides a grain kernel segmentation system based on a vector polygon attention mechanism, which comprises a processor, and executable program instructions are stored in the processor, and the executable program instructions are used to be executed to realize the above-mentioned grain kernel segmentation method based on the vector polygon attention mechanism.

[0026] The application has the following beneficial effects: as an improved invention, the application uses a visual mamba network as a basic network architecture to construct a segmentation model, and a vector polygon attention mechanism is added to the encoder part of the visual mamba network, the vector polygon attention mechanism is used to model the grain kernel image to obtain the geometric relationship and feature interaction in the local polygon region, the attention degree of the encoder to the key region in the image is enhanced, the boundary and shape information of the grain kernel can be more accurately extracted, more discriminative feature representation is provided for the subsequent segmentation task, and the segmentation accuracy of the grain kernel is improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a schematic diagram of an image acquisition system used by the application to acquire the grain kernel image to be segmented;

[0028] Figure 2 is a flow chart of the grain kernel segmentation method based on the vector polygon attention mechanism of the present application;

[0029] Figure 3 is an architecture diagram of the segmentation model used in the grain kernel segmentation method based on the vector polygon attention mechanism of the present application;

[0030] Figure 4 is a schematic diagram of the vector polygon attention mechanism in the encoder stage of the segmentation model used in the grain kernel segmentation method based on the vector polygon attention mechanism of the present application;

[0031] Figure 5 is a schematic diagram of the vector quadrilateral derivation used in the embodiment of the present application;

[0032] Figure 6 is a schematic diagram of the vector quadrilateral calculation in the grain kernel segmentation method based on the vector polygon attention mechanism of the present application;

[0033] Figure 7 is a schematic diagram of the prototype-image alignment mechanism introduced by the decoder in the grain kernel segmentation method based on the vector polygon attention mechanism of the present application;

[0034] Figure 1 shows a computer; 2 represents a corn kernel image acquisition sensor; 3 represents an industrial camera; 4 represents a transverse fine-tuning dovetail slot displacement slider; 5 represents a longitudinal fine-tuning dovetail slot displacement slider; 6 represents an XYZ three-dimensional displacement lifting kernel placement platform. DETAILED DESCRIPTION

[0035] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings.

[0036] Embodiment of the grain kernel segmentation method based on the vector polygon attention mechanism

[0037] A visual Mamba network is used as the basic network architecture to construct the segmentation model. The vector polygon attention mechanism is added to the encoder part of the visual Mamba network. The vector polygon attention mechanism is used to solve the feature expression discontinuity problem caused by the window attention boundary, enhance the model's perception of local details, and improve the accuracy and efficiency of image segmentation. The decoder part of the visual Mamba network performs clustering space unsupervised division on the dense features obtained by encoding, and then completes semantic segmentation by combining the prototype-image alignment strategy.

[0038] 2. Establish a segmentation model.

[0039] The segmentation model of this invention uses a Visual Mamba Network (VMUnet) as its basic network architecture. The encoder part of the VMUnet network performs four layers of downsampling on the input image, with each layer extracting features through multiple convolutional and pooling layers, gradually reducing the spatial resolution to capture high-level semantic information. The decoder part then performs four layers of upsampling, with each layer gradually restoring the spatial resolution of the image through convolutional operations, generating the segmentation result. However, VMUnet may overload computational resources when processing high-resolution images, and ViT's self-attention may lead to information redundancy, affecting the accuracy of object boundaries. Therefore, this invention adds a vector polygon attention mechanism to the encoder part of the VMUnet network and adds clustering and prototype-image alignment to the decoder part of the VMUnet network, such as... Figure 3 As shown below, the encoder and decoder sections will be explained separately.

[0040] 1) Encoder

[0041] Encoders in VMUnet networks, such as Figure 4 As shown, it includes four downsampling layers (Stage 1, Stage 2, Stage 3, and Stage 4). Each Stage contains a PatchMerging module and VPP VSS Blocks. The traditional state-space model (SSM) used includes a structured state-space sequence model. The SSM expresses the input sequence x(t)∈R through the hidden state h(t)∈R. N The continuous linear system is as follows:

[0042] h′(t)=Ah(t)+Bx(t),

[0043] y(t)=Ch(t)+Dx(t)

[0044] Where A∈R N×N It is the state transition matrix, B∈R N×1 This represents the input mapping to the state, C∈R 1×N This represents the mapping from state to output, D∈R 1×1 This is the feedback term. To apply SSM in deep learning, it is assumed that the input remains constant at each time step Δt. The continuous system is discretized using the Zero-Order Preservation (ZOH) method, and the discretized system is represented as follows:

[0045]

[0046] After discretization, the system is represented in the following form:

[0047]

[0048] Traditional state-space models (SSMs) typically employ fixed parameters, limiting their adaptability to complex or high-dimensional data (such as images). To overcome this limitation, Mamba introduces an input-dependent mechanism, allowing parameters B, C, and the discretization ratio Δ to dynamically change with the input. This enhances the model's ability to model diverse data structures. Simultaneously, a gating mechanism is used to dynamically adjust feature representations, selectively preserving relevant information. Although Visual Mamba demonstrates good performance in image segmentation tasks, it can still lead to computational resource overload when processing high-resolution images. Therefore, this invention introduces a vector polygon attention mechanism in the encoder section to avoid feature representation discontinuities caused by window attention boundaries, enhancing the model's perception of local details and improving the accuracy and efficiency of image segmentation. This mechanism effectively highlights the geometric features and boundary information of the target region, thereby improving the accuracy and computational efficiency of image segmentation, especially exhibiting superior segmentation performance in complex backgrounds and structurally diverse scenes.

[0049] like Figure 4 As shown, a vector polygon attention mechanism is introduced in the encoder part of the Visual Mamba VMUnet. The core idea of ​​this mechanism is to assign weights based on the similarity between the query and the keys, and to perform weighted convergence of the input features to highlight key information. The attention mechanism in the vector polygon attention mechanism was originally used in NLP to enhance the modeling ability of long-distance dependencies, and is now widely used in 1D to nD tasks. This mechanism calculates a weighted sum of the input features, where the weights (or attention scores) are determined by a similarity metric between the query and a set of keys. The mechanism is defined as follows:

[0050]

[0051] in, Show the query matrix, Represents the key matrix, Represents a value matrix. d k The term is the dimension of the key vector. The softmax function is applied to the similarity between the query and the key, standardizes the weights, and allows the model to focus on the most relevant parts of the input.

[0052] Specifically, the vector polygon attention mechanism of this invention is set in VPP VSS Blocks. Based on the size of the input image, this invention divides it into n×m non-overlapping patch grids, and uses these as a reference to provide a base rectangle for subsequent polygon transformations, such as... Figure 5As shown, the working principle of the vector polygon attention mechanism is to perform vector transformation on each patch grid, the vector transformation is to transform the rectangle according to the specific content of the image, the transformation result is combined with the window position, and the combined result is fused with the input original image, the fused result is the target image after the vector polygon mask, and finally the attention mechanism calculation is performed on the image to obtain the vector polygon attention. As one of the embodiments, the grain of the present application is corn kernel, therefore, the vector polygon attention mechanism adopts a vector quadrilateral attention mechanism. The forming principle of the vector polygon attention mechanism is described below.

[0053] The vector polygon is a closed figure composed of a series of continuous head-to-tail vectors, which collectively describe the boundary of the polygon. In order to meet the requirements of the model, the vector polygon must meet two conditions.

[0054] Condition one: the head-to-tail edge vectors form a closed path. Suppose the ordered vertices of the polygon are P1, P2,...,Pn. n , define the edge vector as:

[0055] And

[0056] Where i = 1 , 2,...,n.

[0057] Condition two: the cross product (or cross product) of all adjacent edges is consistent in sign. For any two adjacent edges and , calculate their two-dimensional vector cross product (cross product):

[0058]

[0059] Where i = 1, 2,...,n, for convex polygons, all z i have the same sign (sgn(z1) = sgn(z2) =... = sgn(z n )).

[0060] Since a vector of one edge and two edges cannot form a polygon in geometric structure. The geometric structure of a vector of three edges forms a triangle, which has obvious advantages in structural stability and geometric coplanarity. However, the expression of the subdivided surface on the curved surface structure is not good, and the information expressed in the topological structure is single. The vector quadrilateral shows better performance in high-quality modeling and geometric expression, and is convenient to operate. Therefore, in a data-driven manner, the model determines the position, size, direction and shape of the window with dynamic parameters. By increasing the vector polygon module, the basic structure of the visual manba is slightly modified.

[0061] In the visual mamba architecture, the weighted memory state block is the core module to build the model memory and state dynamics. It is based on the state space model, dynamically adjusts the model parameters through input-dependent manner, so as to realize efficient modeling of long sequence. The weighted memory state block is a residual network composed of two branches. One branch uses deep convolutional layers to extract features, and performs S6 Block calculation for scanning in each direction; the other branch is composed of linear mapping and activation layer, and finally combines them to obtain the mamba module. As shown in Figure 4 the feature is extracted by the deep convolutional layer first, which is mainly performed in the SS2DBlock, and the 8D Scan scanning is used to realize the S6 calculation of 8 directions (S6 calculation refers to the processing of S6 Block), and finally the rich feature extraction is obtained, but the features extracted by this part cannot effectively retain the feature information of the image edge. Then through the linear mapping and activation layer, this part makes up for the feature discontinuity problem of the previous branch at the edge of the image target, effectively extracts the features at the transition between the grain and the background boundary by using the vector polygon mechanism. Through the joint action of the two branches, effective feature extraction is realized.

[0062] Since the vector polygon attention mechanism performs the same and independent operation on each attention head. Assuming that a given input feature map is first divided into multiple windows X with a predefined window size w, which is the same as the window-based attention, these windows are called basic windows, and the query, key and value tokens are obtained from the function of each window, i.e. Q, K, V = Linear(X).

[0063] The vector polygon attention calculation is directly performed using the query token, i.e. Qw = Q, while the key and value tokens are reshaped into feature maps for vector polygon attention calculation. The basic window is regarded as a reference, and each basic window is converted into a vector quadrilateral by using the projection transformation. It is assumed that the projection transformation does not preserve parallelism, length and direction, so the obtained vector quadrilateral is very flexible in position, size, direction and shape. The definition of the projection transformation is as follows:

[0064]

[0065] wherein, and The matrix represents the transformation of scaling, rotation and shearing, [c1,c2] T represents translation, and [a3,b3] represents the projection vector, which changes the perception of the object when the viewpoint of the observer changes in the depth dimension. Assuming that a given basic window X wThe vector polygon attention uses a quadrilateral prediction module to predict the projection transformation about the base window in the paper. However, the eight parameters of the direct projection matrix are not easy. Instead, the projection transformation is decomposed into multiple basic transformations, and the parameters of each basic transformation are predicted accordingly. Specifically, the quadrilateral transformation parameters t i where i = 1, 2,..., 9, are composed of an average pooling layer, a ReLU activation layer, and an average pooling layer:

[0066] t i = Conv·LeakyReLU·AvgPool(X w )

[0067] Then, the basic transformation T is obtained by combining the outputs t of the above formula, T ∈ R 3×3 , including scaling T s , shearing T h , rotation T r , translation T t and projection T p :

[0068]

[0069] where β1= W / a and β2= W / b represent scaling factors to help the model adapt to different input images, and finally all the transformations are multiplied in order to obtain the transformation matrix T, that is,

[0070] T = T s · T h · T r · T t · T p

[0071] Given the estimated projection matrix, the position of the projection point is obtained through the standard projection process, that is, given the coordinates (x, y) of a point, the conversion can be completed through the transformation matrix T:

[0072] T out = T·[x, y, 1] T

[0073] The calculation is performed in parallel for each point in the base window to obtain the target position in the projection vector quadrilateral. The calculation principle is shown in Figure 6 , where the left side of Figure 6 is the four basic transformations of the vector quadrilateral, and the right side of Figure 6 is the original vector quadrilateral. First, the original image is divided into m*n grids, then the divided rectangles are translated and scaled (emphasis on enlargement), then affine transformation and translation are performed, and finally the quadrilateral polygon that meets the effective feature extraction is obtained.

[0074] Assuming ti When T = 0, s T h T r T t and T p Since the identity matrix is ​​used, vector polygon attention degenerates into a window-based attention mechanism. However, directly using absolute coordinates when generating vector quadrilaterals can introduce ambiguity. For two windows located at different positions, even if they have the same projection transformation matrix, there may still be significant translation differences between the window far from the origin and the window near the origin. This difference increases the optimization difficulty of the regression module during training. To overcome this problem, this invention no longer uses absolute coordinates when performing projection transformation, but introduces relative coordinates as the input for each window, assuming that the coordinates of the marked points within a given window are {(x...}...}. i ,y i Let |i = 0, 1, ..., M}, M = a × b. Convert their coordinates to relative coordinates as follows:

[0075]

[0076] Where, x center y center Indicates the center of the window, (x relative ,y relative The coordinates (x, y) are relative to the center of the window. After the transformation matrix, according to the coordinates (x, y) in the above formula... relative ,y relative ) Obtain the relative coordinates (x, y) of each word in the target quadrilateral. q,relative ,y q,relative Then, the transformed absolute coordinates are obtained, as shown below:

[0077] (x q,relative ,y q,relative )=(x i,q,relative +x center ,y i,q,relative +y center )

[0078] After obtaining the coordinates of each target quadrilateral token, the grid-sampling function is used to extract the features at the corresponding positions from Kw and Vw within the window, respectively, to generate the query key K and value V for attention calculation. Thanks to the data-driven window configuration mechanism in VPA, the network learns multiple quadrilateral regions with diverse shapes and overlapping positions, thereby facilitating information exchange across windows. However, this design also comes with a potential problem, i.e., some generated quadrilaterals may extend beyond the boundaries of the feature map, resulting in invalid or erroneous sampling. To solve this problem, on the one hand, bilinear interpolation is used to complete the sampling coordinates within the feature map; on the other hand, zero vectors are used as their sampling values for coordinates outside the feature map. Finally, self-attention calculation is performed on Kw, Vw and the original Qw.

[0079] Specifically, as shown in the upper half of FIG. 1, Figure 5 The calculation process of the traditional window attention mechanism includes: sequentially performing grid division on the original image using a method called "grid-sampling", and then serializing and straightening the grid image; then, determining the specific position coordinates of each target quadrilateral (which can be understood as irregular small regions); then, extracting the features corresponding to these quadrilateral regions from the two feature maps Kw and Vw in the window, and these extracted features are finally used as the query, key and value for attention calculation. As shown in the lower half of FIG. 1, Figure 5 The VPA method is used to automatically adjust the shape and position of the window according to the characteristics of the data, so that the network learns these quadrilateral regions with diverse shapes and overlapping positions to help better exchange information between different windows.

[0080] However, this flexible design also brings a small problem: some quadrilaterals may extend beyond the boundaries of the feature map, and the out-of-boundary part cannot be normally sampled, resulting in errors or invalidity. To avoid this problem, the present application has two preparations: on the one hand, the part of the quadrilateral within the feature map normally uses the bilinear interpolation method to obtain the features; on the other hand, the out-of-boundary part simply uses all zeros instead, so as not to introduce error information. Finally, the extracted Kw, Vw and the original Qw are used for self-attention calculation. If the learned quadrilateral covers the area outside the feature map, the gradient of this area is always zero, which will limit the learning of the quadrilateral regression module. To solve this problem, a regularization term is used to constrain the projection quadrilateral to cover more effective areas in the feature map. Specifically, given the token coordinates (x q,relative ,y q,relative ) within the quadrilateral, a penalty function is defined to penalize the coordinates outside the feature map:

[0081]

[0082] The regularization loss of all token coordinates is:

[0083]

[0084] where λ is a hyperparameter.

[0085] Integrating the vector polygon attention mechanism into the Mamba decoder, a Mamba architecture named vector polygon attention is created. The architecture gradually down-samples the feature map in four stages, i.e. the four stages in Figure 4 Swin-UMamba is used as the reference architecture, and the conditional position embedding in CPVT is adopted before the attention layer to complete the encoding of spatial information into the model, i.e.

[0086] X = Z t-1 + CPE(Z t-1 )

[0087] The number of Mamba layers and channel size of each stage are the same as Swin-UMamba, but the window shift operation is cancelled for simplicity.

[0088] 2) Decoder part

[0089] After the vector quadrilateral attention calculation in the encoder stage, in order to realize effective segmentation of the target in the decoder, the prototype-image alignment method is proposed to train a customized Mamba decoder and a set of learnable prototypes in a self-supervised manner. By learning a set of class prototypes, the representative features of different semantic categories in the image are captured, and the pixel features in the image are aligned with the prototypes to achieve more accurate semantic segmentation. Specifically, the "center" feature vector of each category is obtained by clustering. Then, the features of each pixel in the image are extracted using the encoder of Mamba. Finally, each feature vector in the image is compared with the class prototype (such as calculating the similarity), and the pixel is classified into the prototype closest to it, thereby realizing semantic division.

[0090] In the image segmentation and prototype learning task, the model needs to extract a feature vector (i.e. prototype) representing the category from a certain region of the image. However, the image often contains background, noise or other category interference information. Directly performing global average pooling on the whole image will pollute the features obtained by the irrelevant regions, resulting in inaccurate prototype representation. Therefore, the mask average pooling selects only the region of interest for feature average pooling through a mask.

[0091] Assuming that the given support image Isupport and query image Iquery, the dense feature representation is obtained using the Mamba encoder f(x), and the support feature extraction Fs = f(I support ) and query feature extraction F q = f(I query ). The traditional prototype generation method generates a prototype by averaging all foreground features, which is robust in feature information. However, this method is not good at processing detailed information. Therefore, in the prototype calculation, first, the support feature F s is converted into a rough prototype P s by mask average pooling operation, calculated as follows:

[0092]

[0093] where ε represents a non-zero constant, c represents a class, M support represents a support mask, and (x, y) represents the spatial position index of the feature. η is used to determine whether the parameter is true, and if true, the return value is 1, otherwise the return value is 0. Then, the improved prototype P q is generated by fusing high-confidence query features, and the initial prediction mask M is generated using the prototype. Further, by extracting high-confidence query information, the region prototype expression is improved. In this paper, an adaptive query information extraction module is proposed to realize dynamic mining and enhancement of foreground semantic information.

[0094] First, the cosine similarity between the rough prototype P s and the query feature F q is calculated to generate a similarity graph S sim ; then the rough query mask is generated as follows:

[0095] M q = 1-sigmoid(S sim -ε)

[0096] Next, the query feature F q is converted into an improved query prototype P q by mask average pooling method, and information fusion is performed to obtain the final prototype P as follows:

[0097]

[0098] where ε is a non-zero constant, represents an adaptive adjustment factor, and parameters λ1 and λ2 are weights representing the adjustment information fusion ratio, which are set to 0.5 by default.

[0099] To ensure the effective alignment of the decoding stage prototype and image features, the present application first refines the region prototype, and then completes its alignment with the image features. Although the existing region prototype generation method can highlight the local region features, it is difficult to effectively model the global semantics, and relying only on the region prototype is usually difficult to fully improve the segmentation performance. Therefore, the present application proposes a prototype refinement module, which aims to enhance the local region representation while retaining as much global context information as possible. This module introduces the Transformer structure and the mask attention mechanism to build the global association between the region prototype and the foreground features in the support image, so as to achieve more effective foreground feature weighted aggregation. The support information obtained through this mechanism will be more inclined to the features with higher similarity (such as inner product) to the current prototype, thereby enhancing the consistent modeling of the region semantics. At the same time, since this aggregation process is attention weighted for all foreground features, it retains a certain degree of global semantic information. On this basis, the prototype refinement module introduces a prototype alignment mechanism, which realizes dynamic alignment of similar regions through the interaction between the prototype and the query feature, thereby improving the accuracy of feature matching and the region recognition ability.

[0100] As shown in Figure 7 , the prototype refinement module mainly consists of prototype enhancement and prototype alignment. The present application takes the "region prototype" as the query, which can be understood as the representative feature that we are interested in and want to further confirm or optimize. Then, the "support features" are taken as the key and the value, which are equivalent to reference data providing additional information. Next, the attention matrix between the query (that is, the region prototype) and the key (the support feature) is calculated. The role of this matrix is to determine how relevant and similar the region prototype is to each foreground support feature. Through this process, the region prototype can more accurately combine and align the support features, achieving the effect of further refining and enhancing its expression ability.

[0101] Suppose the region prototype P i is taken as the query, and the support feature F s is taken as the input of the key and the value. The attention matrix between the query and the key is calculated to measure the correlation between the region prototype and each foreground support feature. The calculation is as follows:

[0102]

[0103] Where d represents the channel dimension of the input sequence. In the prototype alignment operation, in order to effectively limit the attention range of the foreground region in the attention map, the present application performs a flattening operation on the binary support mask, aligning it with the support feature in the sequence dimension, and finally realizing the precise positioning of the foreground region, thereby guiding the attention mechanism to focus on the region with significant semantic information.

[0104] The grain seed image in the training set is input into the segmentation model to be trained, and the segmentation model is trained according to the output of the segmentation model and the corresponding loss function.

[0105] Suppose S region prototypes are generated, and the cosine similarity S between each region prototype and the query feature is calculated sim The final query prediction result M final The formula is as follows:

[0106] M final = 1-sigmoid(argmax(S sim -epsilon)

[0107] For the grain seed image segmentation task, the present application uses a binary cross-entropy loss function to measure the error between the predicted mask and the standard true value, thereby completing the prototype loss calculation, and the formula is as follows:

[0108]

[0109] Where N represents the number of all pixels participating in the calculation (i.e., the number of sampling points), M q (i,j) represents the coarse query mask, i,j represents the pixel position; represents the prediction result of the final query; x,y represents the spatial position index of the feature.

[0110] By using the query image as the support set to reversely predict the label of the support image, a prototype alignment regularization term is constructed, and the calculation formula is as follows:

[0111]

[0112] As described above, the loss function of each training round is defined as follows:

[0113] L=L proto +L align

[0114] The loss function represents the segmentation loss of the entire model, including the prototype loss and the prototype alignment loss.

[0115] The training data used is grain seed image data after image preprocessing, and the image preprocessing includes denoising, filtering, enhancement and normalization operations. Denoising can remove random noise in the image to improve image clarity; filtering can use Gaussian filtering or median filtering to smooth the image and reduce unnecessary details or noise; enhancement can adjust parameters such as contrast and brightness to enhance the features of the region of interest and ensure the distinguishability of the target object.

[0116] The trained segmentation model can be used to segment the to-be-segmented grain kernel image, wherein the to-be-segmented grain kernel image is also an image after image preprocessing.

[0117] Therefore, the present application introduces a vector polygon attention mechanism in the encoder stage, which enhances the attention of the encoder to the key regions in the image by explicitly modeling the geometric relationship and feature interaction within the local polygon region, especially in a complex background, can more accurately extract the boundary and shape information of the grain kernel, thereby improving the precision and computational efficiency of image segmentation. In the decoder stage, unsupervised partitioning in the clustering space is performed, and the prototype-image alignment strategy is used to complete semantic segmentation.

[0118] In one embodiment, the grain kernel used is a corn kernel, and the image segmentation process used is as shown in Figure 2 The specific process is as follows:

[0119] 1) Obtain the corn kernel image and perform preprocessing.

[0120] This embodiment uses a grain kernel image acquisition device to obtain the corn kernel image, as shown in Figure 1 , wherein 1 represents a collection computer; 2 represents a corn kernel image acquisition sensor; 3 represents an industrial camera; 4 represents a horizontal fine-tuning dovetail slot displacement slider; 5 represents a vertical fine-tuning dovetail slot displacement slider; 6 represents an XYZ three-dimensional displacement lifting seed placement platform. The horizontal fine-tuning dovetail slot displacement slider is used to accurately adjust the position of the camera or the sample platform in the horizontal direction, facilitating the adjustment of the field of view to obtain the best imaging angle; the vertical fine-tuning dovetail slot displacement slider is used to realize fine displacement control in the vertical direction, ensuring that the sample is aligned with the imaging plane, and improving the image clarity; the XYZ three-dimensional displacement lifting platform is used to provide high-precision adjustment capability in three-dimensional space, supporting precise positioning and height adjustment of the industrial camera or the sample platform, so that the target region is accurately placed in the camera field of view; the hard aluminum optical breadboard serves as the installation base of the device, has good rigidity and standardized hole positions, and ensures the stability and compatibility of the overall structure; the planar backlight platform is used to provide a uniform backlighting environment, effectively enhancing the visibility of the corn kernel edges, optimizing the image quality, and facilitating subsequent segmentation processing; the industrial camera and the camera acquisition software system are used for high-resolution image acquisition and saving, ensuring that the image details are rich, and facilitating subsequent feature extraction and analysis; the camera clamp is used to stably install the industrial camera, ensuring that the view angle is fixed and the position is stable during the acquisition process, improving data consistency and repeatability. The components of the grain kernel image acquisition device are all composed of standard optical elements, have good adjustability and modular characteristics, and can provide reliable data support for corn kernel image segmentation and phenotype feature analysis.

[0121] The preprocessing of the image is specifically realized by image denoising, filtering, enhancement, normalization and other operations on the raw image of the corn kernel collected by the camera, which will not be described here.

[0122] 2) Training the model to be trained using the training data to obtain a segmentation model.

[0123] Specifically, the training data can be the corn kernel image data obtained by step 1), or obtained by other means. Regardless of the way the corn kernel image is obtained, the training data is input into the segmentation model to be trained, and the training is performed according to the output result and the real label. Since the structure and working principle of the segmentation model have been described in detail above, they will not be described here.

[0124] 3) Evaluate the trained segmentation model.

[0125] For the segmentation model obtained after training, the trained segmentation model is evaluated before being used for actual image segmentation. In this embodiment, the data in the test set is input into the trained segmentation model, the segmentation result output by the segmentation model according to the input sample is restored to the resolution consistent with the original image, and then compared with the real label result of the sample in the verification set for evaluation, and the performance indicators are calculated. The mDSC similarity coefficient indicator and the mIoU intersection over union indicator are used for quantitative analysis, and the calculation formula of the indicator is:

[0126]

[0127] Wherein, T represents the correct judgment of the classifier, F represents the wrong judgment of the classifier, P represents the positive sample judged by the classifier, and N represents the negative sample judged by the classifier. TP represents that the classifier judges the sample as positive, and the sample is actually positive, which is correct; TN represents that the classifier judges the sample as negative, and the sample is actually negative, which is correct; FP represents that the classifier judges the sample as positive, and the sample is actually negative, which is wrong; FN represents that the classifier judges the sample as negative, and the sample is actually positive, which is wrong.

[0128] Subjectively, the consistency of the segmentation result and the original image can also be evaluated by artificial visual evaluation, focusing on observing the performance of the model in the kernel edge, shape restoration and diversity scene, and judging whether the segmentation result meets the visual intuition and actual demand.

[0129] Table 1. Comparison results of corn kernel experimental data in different segmentation methods

[0130]

[0131] Through comparison experiments, it can be concluded that the method proposed in the application is obviously superior to other similar methods in mIoU and mDSC indexes.

[0132] 4) For the trained segmentation model after performance evaluation, the grain seed image to be segmented can be directly input into the trained segmentation model to obtain a segmentation result; the acquisition mode of the corn seed image to be segmented is consistent with the acquisition mode of step 1).

[0133] Grain seed segmentation system based on vector polygon attention mechanism

[0134] The embodiment provides a grain seed segmentation system based on a vector polygon attention mechanism, the segmentation system comprising a processor, the processor storing executable program instructions, and the executable program instructions being used to be executed to implement the above-mentioned grain seed segmentation method based on the vector polygon attention mechanism. Since the working principle and working mode of the segmentation system in the embodiment have been described in detail in the above-mentioned grain seed segmentation method based on the vector polygon attention mechanism, no further description is given here.

Claims

1. A method for grain segmentation based on a vector polygon attention mechanism, characterized in that, The method includes the following steps: 1) Obtain the image of the grain to be segmented; 2) Establish a segmentation model and use the established segmentation model to segment the acquired grain images to be segmented. The segmentation model adopts a visual mamba network as the basic network architecture. In the encoder part of the visual mamba network, a vector polygon attention mechanism is set. The vector polygon attention mechanism is used to perform vector transformation on each grid of the input image, combine the vector transformation result with the window position, fuse the combined result with the original input image, and perform attention mechanism calculation on the image to obtain the vector polygon attention. The decoder in the visual mamba network is used to decode the features output by the encoder to obtain the segmentation result.

2. The grain segmentation method based on vector polygon attention mechanism according to claim 1, characterized in that, The decoder in the Visual Mamba network uses a prototype-image alignment method: extracting feature vectors representing each region from the features output by the encoder as region prototypes, extracting feature vectors for each pixel in the image, comparing each feature vector in the image with the region prototype, and classifying the pixel to the region prototype closest to it, thereby achieving semantic segmentation.

3. The grain grain segmentation method based on vector polygon attention mechanism according to claim 1, characterized in that, The vector transformation described above uses a base window as a reference and employs projection transformation to convert each base window into a vector quadrilateral.

4. The grain segmentation method based on vector polygon attention mechanism according to claim 3, characterized in that, When performing projection transformation, relative coordinates are used as the input for each window. The relative coordinates refer to the coordinates relative to the center of the window. After obtaining the relative coordinates of each target quadrilateral, the grid-sampling function is used to extract the features at the corresponding positions from Kw and Vw in the window, respectively, to generate the query key K and value V for attention calculation, thereby performing self-attention calculation.

5. The grain segmentation method based on vector polygon attention mechanism according to claim 3, characterized in that, If the learned quadrilateral covers areas outside the feature map, then a regularization term is used to constrain the projected quadrilateral to cover more effective areas in the feature map.

6. The grain segmentation method based on vector polygon attention mechanism according to claim 2, characterized in that, The region prototype is obtained through masked average pooling. A mask is used to select the region of interest, and average pooling is applied to the features of the selected region of interest. The process of determining the region prototype is as follows: Supporting features are transformed into coarse prototypes through masked average pooling operations. A similarity graph is generated by utilizing the similarity between the coarse prototype and the query features. A coarse query mask is generated based on the similarity graph. The query features are then transformed into a query prototype by average pooling of the mask. The final prototype, or region prototype, is obtained by weighted summation of the coarse prototype and the query prototype.

7. The grain segmentation method based on vector polygon attention mechanism according to claim 2, characterized in that, When comparing each feature vector in the image with the region prototype, an alignment operation is performed between the feature vector and the region prototype to improve the accuracy of feature matching and the ability to identify regions.

8. The grain segmentation method based on vector polygon attention mechanism according to claim 7, characterized in that, The process of comparing each feature vector in the image with the region prototype includes: Using the region prototype as the query and the foreground support features as the key and value inputs, we compute an attention matrix between the query and the key to measure the correlation between the region prototype and each foreground support feature.

9. The grain segmentation method based on vector polygon attention mechanism according to any one of claims 1-8, characterized in that, The loss function of the segmentation model in each training round is: L=L proto +L align Where M final To query the prediction results.

10. A grain segmentation system based on a vector polygon attention mechanism, comprising a processor, wherein the processor stores executable program instructions, characterized in that, The executable program instructions are executed to implement the grain segmentation method based on vector polygon attention mechanism as described in any one of claims 1-9.

Images