Leaf vegetable image segmentation method, system, device and medium

Abstract

The invention discloses a leafy vegetable image segmentation method, system and device and a medium, and relates to the technical field of target segmentation, and the method comprises the steps: collecting leafy vegetable image data, and extracting a multi-scale initial feature map; convolution is carried out on the multi-scale initial feature map to obtain a multi-scale feature map, new feature mapping is generated for the multi-scale feature map from bottom to top, and the new feature mapping is fused to obtain a final feature map; and generating an initial prediction mask based on the final feature map, carrying out blocking, DCT transformation, correction and boundary optimization operation on the initial prediction mask, outputting a multi-dimensional DCT vector containing mask information, and taking the final multi-dimensional DCT vector as a final prediction mask to obtain a segmentation result of the leaf vegetable image. On the basis of the method, the local features of the leaf vegetables are captured, multi-scale feature fusion is carried out, core information of leaf vegetable images is reserved, and the influence of environmental factors is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of target segmentation, and in particular to a method, system, equipment and medium for segmenting leafy vegetable images. Background Art

[0002] Leafy vegetables are an important part of the vegetable industry, providing rich vitamins and minerals. They have the characteristics of a short growing period, a high multiple cropping index, and a wide distribution of planting areas, making them an indispensable agricultural product in people's daily diet. With the continuous advancement of agricultural technology and the diversification of vegetable planting varieties, consumers' demand for fresh, high-quality vegetables continues to increase. During the growth process of vegetables, the color, shape and size of vegetables vary greatly. When researchers conduct provenance identification of vegetables, they face many challenges in identification and segmentation due to the different growth stages and changes in phenotypic characteristics of vegetables, resulting in low production efficiency and unstable product quality. For tourists and consumers, their understanding of the cultivation characteristics and variety characteristics of vegetables is limited, and they are often unable to judge the type and growth stage of vegetables by surface characteristics.

[0003] Under natural field conditions, leafy vegetables grow in a complex and diverse environment, influenced by factors such as lighting variations, background weeds, and soil. Furthermore, the leaves of vegetables of the same or different species are often highly similar, posing a challenge to the automated classification and segmentation of leafy vegetables. Traditional vegetable classification and segmentation methods typically rely on manually designed feature extraction and image processing techniques. For example, methods such as color and shape feature analysis, texture analysis, and traditional machine learning methods (such as support vector machines and K-nearest neighbor algorithms) are used to classify vegetables, while edge detection algorithms, threshold segmentation, and morphological operations are used for segmentation. These methods are significantly affected by lighting, irregularities in vegetable phenotypes, and image quality. They are therefore unstable and time-consuming when used in natural or highly variable environments.

[0004] In recent years, deep learning technology has made significant progress in image classification and segmentation, particularly convolutional neural networks (CNNs), which have multiple hidden layers between the input and output layers and possess powerful feature extraction and representation capabilities. This has made vegetable classification and segmentation more efficient, less complex, and less subject to human influence. Key challenges in leaf classification and segmentation include the variability of leaf posture and structure over growth, the phenotypic similarity of leaves between varieties, and overlap and occlusion by other leaves. Instance segmentation, a key model in convolutional neural networks, can detect objects pixel by pixel, effectively addressing the challenges of classification and segmentation caused by leaf adhesion and occlusion, while simultaneously capturing both the variety and morphological structure of the vegetable. Currently, instance segmentation algorithms can be broadly categorized into single-stage and two-stage approaches. Single-stage approaches directly predict classification scores and masks based on predefined anchor points. Two-stage approaches first detect a sparse set of proposals and then predict masks based on them, further extending the approaches to top-down approaches based on object detection and bottom-up approaches based on semantic segmentation.

[0005] Traditional vegetable classification and segmentation algorithms can achieve good recognition results for leafy vegetables with significant phenotypic differences because the extracted leafy vegetable features are relatively basic. However, when processing sub-categorized leafy vegetables with high phenotypic similarity, the recognition accuracy is significantly reduced. In addition, leafy vegetable images collected in natural environments are easily disturbed by factors such as changes in light intensity, leaf overlap and occlusion, which affect the extraction of phenotypic features of leafy vegetables and lead to reduced recognition accuracy. Summary of the Invention

[0006] The purpose of the present invention is to provide a leafy vegetable image segmentation method, system, device and medium to address the deficiencies in the prior art described above, so as to solve the problems in the prior art.

[0007] The present invention specifically provides the following technical solution, a leafy vegetable image segmentation method, comprising:

[0008] Collect leafy vegetable images and extract multi-scale initial feature maps of the leafy vegetable images;

[0009] Perform several convolutions on the multi-scale initial feature map to obtain a global information matrix, downsample the global information matrix, multiply the downsampled vector with the initial feature map, and obtain an information-enhanced multi-scale feature map through convolution. Generate a new feature map for the multi-scale feature map in a bottom-up path, and fuse the new feature maps to obtain the final feature map;

[0010] Based on the final feature map, an initial prediction mask is generated. After dividing the initial prediction mask into multiple image patches, the image patches are subjected to block discrete cosine transform (DCT) transformation. Foreground and background image patches that are incorrectly predicted after the DCT transformation are corrected, and the boundaries of image patches where both foreground and background exist are optimized. A multidimensional DCT vector containing mask information is output, and the multidimensional DCT vector is concatenated with features of the multi-scale feature map. The concatenated features are subjected to DCT transformation, and then the multidimensional DCT vector is obtained multiple times again, with the final multidimensional DCT vector being used as the final prediction mask.

[0011] Based on the final prediction mask, a segmentation result of the leafy vegetable image is obtained.

[0012] Preferably, generating a new feature map from the multi-scale feature map in a bottom-up manner includes:

[0013] In the deep learning model Mask R-CNN, the FPN feature map [P2, P3, P4, P5] is used. The spatial size is downsampled with a stride of 2 starting from P2 through a 3×3 convolutional layer to obtain the downsampled map. [P2, P3, P4, P5] is the level of the multi-scale feature map, representing feature maps of different spatial resolutions.

[0014] The i+1th feature map P i+1 Each element of is horizontally connected to the down-sampled image to obtain the fused feature map, which is then processed through a 3×3 convolution layer to generate the i+1th new feature map N i+1 , and loop until P5 to generate all new feature maps [N2, N3, N4, N5].

[0015] Preferably, after the final multi-dimensional DCT vector is used as the final prediction mask, the final prediction mask is optimized by a boundary loss function, specifically:

[0016] The boundary loss function is obtained by weighted fusion of binary cross entropy loss and Dice loss, and combined with classification loss, bounding box regression loss and mask loss. The final predicted mask is optimized through the boundary loss function.

[0017] Among them, the binary cross entropy loss is used to obtain the average loss of all leafy vegetable samples, and the Dice loss is used to measure the degree of overlap between the predicted vegetable segmentation mask and the actual vegetable segmentation mask.

[0018] Preferably, the extraction of the multi-scale initial feature map of the leafy vegetable image is specifically as follows:

[0019] Preferably, a lightweight feature extraction network is constructed comprising a lightweight classifier backbone network of multiple inverted residual blocks, a CA attention mechanism, and a fast pyramid pooling module FPPM; wherein each inverted residual block corresponds to a CA attention mechanism;

[0020] Multi-scale initial feature maps of leafy vegetable images are extracted through a lightweight feature extraction network.

[0021] Preferably, the construction includes a lightweight classifier backbone network of multiple inverted residual blocks, a CA attention mechanism, and a lightweight feature extraction network of a fast pyramid pooling module FPPM, specifically:

[0022] The inverted residual block structure of MobileNetv2 is introduced into Mask R-CNN as the backbone network of the lightweight classifier. Specifically, the inverted residual block structure of MobileNetv2 is as follows: the last average pooling layer, fully connected layer, and softmax layer of the MobileNetv2 network are deleted, and the MobileNetv2 network structure is divided into five inverted residual blocks C1 to C5. C1 contains one standard convolution and one depthwise separable convolution, C2 and C3 contain two depthwise separable convolutions respectively, C4 contains six depthwise separable convolutions, and C5 contains two depthwise separable convolutions.

[0023] The depth-wise separable convolution of each inverted residual block includes sequentially connected point-by-point convolution PW, depth-wise convolution DW and point-by-point convolution PW. The depth-wise convolution DW includes 3×3 convolution, normalization BN and ReLU activation function, and the CA attention mechanism is introduced after the ReLU activation function; the fast pyramid pooling module FPPM is introduced after the last inverted residual block.

[0024] Preferably, the extracting of multi-scale initial feature maps of leafy vegetable images by a lightweight feature extraction network comprises:

[0025] In each inverted residual block, features of leafy vegetable images are extracted Figure X , and use pooling to check features Figure X Encode the horizontal coordinate and vertical coordinate of to obtain channel output;

[0026] Based on the channel output, features are aggregated along two spatial directions, namely, feature dependency and feature position information, to obtain a pair of direction-aware feature maps, the pair of direction-aware feature maps are spliced, and a 1×1 convolution operation is performed on the spliced ​​direction-aware feature maps using a convolution transformation function F1 to obtain an intermediate feature map of two directional spatial information;

[0027] Decompose the intermediate feature map into two independent tensors in the height and width directions along the spatial dimension, and convert the two independent tensors into two tensors with the feature Figure X Tensors with the same channel number;

[0028] The two converted tensors are used as attention weights and multiplied with the input feature map to obtain the feature representation of channel attention. The feature representation of channel attention is then passed through a 1×1 convolution operation and normalization operation to output the initial feature maps of branches of different scales.

[0029] The initial feature maps of branches of different scales are upsampled separately and processed by 1×1 convolution and 3×3 convolution in sequence. The convolution processing results of multiple branches of different scales are mapped to feature maps and weighted fused by 1×1 convolution to obtain multi-scale initial feature maps.

[0030] The present invention provides a leafy vegetable image segmentation system, comprising:

[0031] A data acquisition module is used to collect leafy vegetable images and extract multi-scale initial feature maps of the leafy vegetable images;

[0032] The feature map acquisition module is used to perform several convolutions on the multi-scale initial feature map to obtain a global information matrix, downsample the global information matrix, multiply the downsampled vector with the initial feature map, and obtain an information-enhanced multi-scale feature map through convolution. A new feature map is generated for the multi-scale feature map in a bottom-up path, and the new feature maps are fused to obtain the final feature map;

[0033] A mask prediction module is configured to generate an initial prediction mask based on the final feature map, divide the initial prediction mask into multiple image patches, perform block discrete cosine transform (DCT) on the image patches, correct foreground and background image patches that are incorrectly predicted after DCT, and optimize the boundaries of image patches where both foreground and background exist. The module then outputs a multidimensional DCT vector containing mask information, concatenates the multidimensional DCT vector with features of the multiscale feature map, performs DCT on the concatenated features, and then re-acquires the multidimensional DCT vector multiple times, using the final multidimensional DCT vector as the final prediction mask.

[0034] The output module is used to obtain the segmentation result of the leafy vegetable image based on the final prediction mask.

[0035] The present invention provides a computer device, comprising a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the above-mentioned leafy vegetable image segmentation method.

[0036] The present invention provides a storage medium storing a computer program, wherein the computer program implements the steps of the above-mentioned leafy vegetable image segmentation method when executed by a processor.

[0037] Compared with the prior art, the present invention has the following significant advantages:

[0038] The present invention performs information enhancement processing on the multi-scale initial feature map, generates a new feature map for the multi-scale feature map in a bottom-up path, and fuses the new feature map to obtain the final feature map. In the process of feature fusion, global information is fully utilized, and the problem of loss of useful channel information in high-level feature mapping of vegetable data sets in FPN is solved. The low-level precise positioning signal is more effectively transmitted to the entire feature pyramid, and the positioning capability of the feature pyramid for the shape and structure of vegetables is enhanced. By introducing block discrete cosine transform, the final mask is obtained. In the case of leafy vegetables with high similarity, low-frequency and high-frequency information in the image can be effectively extracted, the local features of leafy vegetables can be captured, and multi-scale feature fusion can be performed to retain the core information of the leafy vegetable image, reduce the influence of environmental factors, especially important details such as leaf edges and contours, thereby reducing data redundancy and achieving the effect of improving segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a partial data set diagram of the present invention; wherein, Figure 1 (a1) to (c1) are the seedling stage photos of Guotai 181, 702-1-4 and Youfeng 70 Jianye 3, respectively. Figure 1 (a2) to (c2) are the growth periods of the above varieties. Figure 1 (a3) to (c3) are the bolting periods of the above varieties. Figure 1 (a4) to (c4) are the flowering periods of the above varieties;

[0040] Figure 2 This is the Mask R-CNN network architecture diagram of the present invention;

[0041] Figure 3 This is the overall structural diagram of the EPA-FPN of the present invention;

[0042] Figure 4 It is a structural diagram of the GEM of the present invention;

[0043] Figure 5 It is the structure diagram of PatchDCT of the present invention;

[0044] Figure 6 is a multi-stage PatchDCT flow chart of the present invention;

[0045] Figure 7is a graph showing the segmentation accuracy and loss changes of the validation set during the training process of the present invention; Figure 8 (a) is the AP change, Figure 8 (b) is the AP50 change, Figure 8 (c) is the AP75 change, Figure 8 (d) is the lo ss change;

[0046] Figure 8 This is the backbone network structure diagram based on MobileNetv2 in the present invention;

[0047] Figure 9 It is a structural diagram of the CA module in the present invention;

[0048] Figure 10 This is a diagram of the fusion process of the CA module and the depth-wise separable convolution in the present invention;

[0049] Figure 11 It is the structural diagram of FPPM in the present invention;

[0050] Figure 12 It is a visual comparison diagram of each lightweight classifier in the present invention;

[0051] Figure 13 This is a diagram showing the recognition and segmentation effects of different networks on the model of the present invention; Figure 13 (a1) to (a5) are the seedlings of Guotai 181 in different models, Figure 13 (b1) to (b5) are Zengcheng Jianye 5-6-2-1 in different growth stages. Figure 13 (c1) to (c5) are Longtian No. 1 in different bolting stages. Figure 13 (d1) to (a5) are the 702 tip leaves at different model flowering stages;

[0052] Figure 14 This is a flow chart of a leafy vegetable image segmentation method proposed in the invention. DETAILED DESCRIPTION

[0053] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0054] like Figure 14 As shown, in this embodiment, a leafy vegetable image segmentation method is adopted, comprising the steps of:

[0055] Step S1: Collecting leafy vegetable image data.

[0056] The information of current public and private leafy vegetable segmentation datasets is shown in Table 1. The current private leafy vegetable segmentation datasets cover a relatively rich variety of vegetable varieties, but the segmentation datasets of leafy vegetables with subcategories and small varieties are still relatively few.

[0057] Table 1 Survey information of leafy vegetable segmentation dataset

[0058]

[0059] Because the shooting and manual segmentation and annotation of leafy vegetable datasets are time-consuming and labor-intensive, the phenotypic similarity of small varieties of leafy vegetables is too high, and the transition period of vegetables at different growth stages is difficult to distinguish, the field of leafy vegetable segmentation currently lacks authoritative public datasets covering a rich variety of vegetables with diverse backgrounds. As a result, it is difficult for current models to identify similar and different leafy vegetables with highly similar phenotypes, and they cannot be used in scenarios such as agricultural automated classification of vegetable leaves, plant phenotypic analysis, real-time growth monitoring, and provenance identification.

[0060] Therefore, we constructed a multi-variety leafy vegetable segmentation dataset, MaskVeg, that includes both single- and natural-background images. This dataset covers two categories, 30 varieties, and four stages of leafy vegetables, totaling 30,600 images. The following details the dataset construction process.

[0061] The various varieties of vegetables photographed in this invention were sourced from the experimental demonstration field of the Shenzhen Agricultural Product Quality and Safety Inspection and Testing Center in Guangdong Province. After vegetable variety identification and cultivation by professionals at the base, 28 small varieties of Chinese cabbage and 2 small varieties of Chinese kale were selected as experimental data. Each small variety has four growth stages: seedling stage, growth stage, bolting stage, and flowering stage. The varieties and corresponding growth stages are shown in Tables 2-1 and 2-2.

[0062] Table 2-1 Leafy vegetable variety information

[0063]

[0064] Table 2-2 Appendix on leafy vegetable varieties

[0065]

[0066] To ensure that the training model has strong scene applicability and wide applicability, the acquisition method of the present invention combines rotating platform shooting with manual handheld camera to collect multi-angle leafy vegetable videos in single and field scenes and extract them into RGB images to enhance the robustness of the model. Finally, the collected JPG format images are uniformly compressed to 960*540 pixels to ensure clarity while accelerating model training. The camera parameters are shown in Table 3.

[0067] In a single scene, a rotating platform is used to capture images. The steps are as follows: (1) Place the target object at the center of the rotating platform to ensure uniform illumination on the surface of the object and avoid overexposure or shadows; (2) Adjust the position and angle of the camera to ensure that the entire object is within the camera's field of view; (3) Set the rotation angle interval of the rotating platform and capture an image every 20° until 360° image acquisition of the object is completed; (4) Capture a single sample from the front and 45° above. 36 images from different angles can be obtained under a single background, allowing the model to fully learn the phenotypic characteristics of leafy vegetables and improve the accuracy and robustness of the model classification and segmentation. In natural field scenes, leafy vegetable image data is collected by extracting images by frame from multi-angle video and by multi-angle shooting. Samples are placed in different indoor and outdoor scenes with random postures and random occlusion levels, and are captured by the camera at random angles and random distances to obtain sample videos or images under different backgrounds, thereby improving the integrity and richness of the data and the generalization of the training model. The Python programming language is then combined with the OpenCV library to extract the video file frame by frame. To avoid data redundancy and improve processing efficiency, the present invention adopts an equal-interval sampling strategy, extracting one frame per second, and using the cv2.imwrite function to save each frame as an independent image file.

[0068] Table 3 Camera parameters

[0069] parameter Numerical sensor OmniVision OV68B 1 / 2 inch Pixel 64 million aperture f / 2.57 Ultra wide-angle lens 120 degrees

[0070] Through single scene and field natural scene shooting, we collected 30,600 images of 2 major categories and 30 minor categories of leafy vegetables. Some of the data sets are shown below. Figure 1 To improve the generalization ability of the model and avoid overfitting caused by partial scene similarity and sample size imbalance, the present invention performs data augmentation on a self-made leafy vegetable dataset. Data augmentation methods include geometric transformation (flipping, rotation, cropping, scaling, translation, color dithering, scale transformation, etc.) and pixel transformation (salt and pepper noise, Gaussian noise, adjusting HSV contrast, brightness, saturation, etc.). Specific descriptions and usage functions are shown in Table 4.

[0071] Table 4 Data augmentation methods

[0072]

[0073] Data labeling involves manual or semi-automatic tagging and annotation of raw data (such as images, videos, text, audio, etc.) to provide high-quality labeled data for machine learning model training.

[0074] The quality of data annotation directly affects the performance of the model. This paper completes data annotation with high-quality standards by assigning category labels to images, providing pixel-level segmentation masks for each target instance, and distinguishing different instances.

[0075] Furthermore, we used the EISeg semi-automatic annotation tool to segment and annotate leafy vegetable images. EISeg is an interactive image segmentation and annotation tool developed based on PaddlePaddle. It combines deep learning models with user interaction to efficiently and accurately generate pixel-level segmentation annotations, significantly improving annotation efficiency. The overall annotation process is as follows:

[0076] (1) Compile the file labels.txt containing the names of the various varieties of leafy vegetables collected. The order of the varieties in the file will be consistent with the label numbers of the marked files.

[0077] (2) Install the EISeg environment, download the high-precision pre-trained model parameters it provides, load the image folder to be labeled and the species name file labels.txt, and automatically assign different colors to each category to distinguish them. In the interface, you can set the segmentation threshold, label color, transparency, and other information.

[0078] (3) Semi-automatic segmentation and annotation of the image. Click the left mouse button in the target area, and the model will generate a mask in real time based on the pre-training results and the clicked area. Right-click the non-target area to mark it as background, fine-tune the contour boundary, and save it to generate a PNG format image of the segmented area and a JSON annotation file in Labelme format, which contains the image path, image size, annotation category, area contour information, color information, etc.

[0079] In the instance segmentation task, data set partitioning is a key step in model training and evaluation. Typically, the data set is divided into a training set, a validation set, and a test set. When partitioning the data, it is important to ensure that the samples of each class are evenly distributed, randomly divided, and non-overlapping across the training, validation, and test sets. The specific process for data set partitioning in this invention is as follows:

[0080] (1) All leafy vegetable image data were placed in a general image folder called images. The random function in the Python script was used to randomly divide the images into training, validation, and test sets in a ratio of 7:2:1. The images were saved in the train, val, and test directories, respectively. The details of the dataset are shown in Table 5.

[0081] Table 5 Dataset division information

[0082]

[0083] (2) According to the Mask R-CNN label data style requirements, the Labelme format is changed to a coco format annotation file. Rewrite the official labelme2coco.py file to convert each sample's JSON file into a coco format file with category labels, boundary coordinates, and pixel-level segmentation masks based on the input category labels.

[0084] (3) The annotation files of all images are integrated into the instances_train.json and instances_val.json files according to the training set and validation set, respectively, and placed in the annotations folder. Finally, the leafy vegetable segmentation dataset MaskVeg is constructed.

[0085] Step S2: Extracting multi-scale initial feature maps of leafy vegetable images.

[0086] Step S3: The multi-scale initial feature map is convolved several times to obtain a global information matrix, the global information matrix is ​​downsampled, the downsampled vector is multiplied by the initial feature map, and the information-enhanced multi-scale feature map is obtained by convolution. A new feature map is generated for the multi-scale feature map in a bottom-up path, and the new feature maps are fused to obtain the final feature map.

[0087] Step S4: Based on the final feature map, an initial prediction mask is generated. After the initial prediction mask is divided into multiple image block patches, the image block patches are subjected to block discrete cosine transform (DCT) transformation. The foreground and background image block patches that are incorrectly predicted after the DCT transformation are corrected. The boundaries of the image block patches where both foreground and background exist are optimized. A multidimensional DCT vector containing mask information is output. The multidimensional DCT vector is spliced ​​with the features of the multi-scale feature map, and the spliced ​​features are subjected to DCT transformation. The multidimensional DCT vector is then obtained again multiple times, and the final multidimensional DCT vector is used as the final prediction mask.

[0088] Among them, step S2, step S3 and step S4 are used to construct the leafy vegetable classification and segmentation model as follows:

[0089] Global information enhancement and path aggregation methods are introduced into Mask R-CNN to construct a feature fusion network EPA-FPN, block discrete cosine transform and boundary loss function are introduced, as well as a lightweight classifier, CA attention mechanism and FPPM module are introduced. A leafy vegetable classification and segmentation model is constructed using EPA-FPN, block discrete cosine transform, boundary loss function and FPPM module.

[0090] Since instance segmentation algorithms face the difficulties of inaccurate results and difficulty in deployment on mobile facilities when applied to the classification, identification and real-time segmentation of leafy vegetables in natural environments, it is necessary to design a high-precision and lightweight leafy vegetable classification and segmentation algorithm to adapt to actual agricultural operations. A network model LeafVegMask based on the improved Mask R-CNN is proposed, which is optimized from two aspects: accuracy improvement and lightweight improvement, aiming to reduce the complexity of the model while improving the classification and segmentation performance. First, the global information perception module and path aggregation method are introduced to construct the feature fusion network EPA-FPN, and the PatchDCT and Boundary Loss loss functions are introduced to improve the ability of FPN to fuse richer vegetable leaf features; secondly, a lightweight classifier and CA attention mechanism are introduced to construct the FPPM module, which strengthens the vegetable leaf area and greatly reduces the complexity of the model, improves the model running speed, and facilitates application on embedded devices. The optimized overall network architecture is shown below. Figure 2 As shown in the figure, the final feature map is generated into a candidate region through the region proposal network RPN, which is mapped to the corresponding area of ​​the original image through the RoIAlign layer to eliminate redundant target bounding boxes.

[0091] Feature extraction networks are typically used to extract rich visual features from input images, capture details and contextual information in the image, and provide a basis for RPN and subsequent ROI (Region of Interest) operations (Wang et al. 2022). The backbone network of Mask R-CNN is the ResNet series network. The deep network structure can learn rich feature representations, but it also makes the model relatively clumsy, which is not conducive to deployment on mobile devices or edge devices. It is difficult to adapt to high-precision and lightweight classification and segmentation of leafy vegetables in different environments. Therefore, the present invention introduces the inverted residual block structure of MobileNetv2, which reduces the number of model parameters and computational complexity while ensuring that the classification and segmentation accuracy is not reduced.

[0092] A lightweight feature extraction network is constructed, which includes a lightweight classifier backbone network of multiple inverted residual blocks, a CA attention mechanism and a fast pyramid pooling module FPPM; each inverted residual block corresponds to a CA attention mechanism; and the lightweight feature extraction network is used to extract multi-scale initial feature maps of leafy vegetable images.

[0093] Among them, the construction process of the lightweight feature extraction network is specifically as follows: the last convolutional layer of the feature extraction network is usually used for the final compression of features, and the fully connected layer is used to generate the final classification results in tasks with only image classification, but a segmentation branch is added to Mask R-CNN, and more vegetable feature information needs to be retained to facilitate the subsequent construction of FPN and RPN. The present invention first deletes the last average pooling layer, fully connected layer, and softmax layer of the MobileNetv2 network, and then divides its structure into five stages, C1 to C5. C1 contains 1 standard convolution and 1 depth-separable convolution, C2 and C3 contain 2 depth-separable convolutions respectively, C4 contains 6 depth-separable convolutions, and C5 contains 2 depth-separable convolutions. S1 to S5 represent the outputs of stages C1 to C5, respectively, and correspond to the subsequent FPN feature fusion, which greatly reduces the depth of the network. Figure 8 As shown. Secondly, based on the transfer learning of the MobileNetv2 pre-trained model, the parameters specified in the configuration file (.yaml file) are used in the feature extraction network to freeze some layers in the network, that is, to stop the parameter update of these layers during the training process. Finally, in order to connect the backbone network with the FPN to transmit the extracted leafy vegetable image features, the present invention sets the number of feature output channels to [24, 32, 96, 320] and the downsampling factor of the spatial resolution of each output feature layer relative to the input image to [4, 8, 16, 32], which is crucial for designing the size and position of the RPN and ROI head anchors.

[0094] The attention mechanism has been proven to help deep learning models focus on key areas in images more effectively. In the classification and segmentation tasks of leafy vegetables, channel attention (such as SE attention) is usually encoded using a global pooling method, but because it compresses global spatial information into the channel descriptor, it results in the loss of edge position information of some vegetable leaves. The CA module decomposes channel attention into two one-dimensional feature encoding processes, and accurately encodes the position information of channel relationships and long-range dependencies through the two steps of coordinate information embedding and coordinate attention generation. By integrating the CA module into the feature extraction network, the model strengthens the vegetable leaf area, extracts richer vegetable leaf feature information, increases the robustness of vegetable classification and segmentation in natural environments, accurately obtains vegetable categories and segments small structures such as vegetable stems and leaves, such as structures. Figure 9As shown. The depth-wise separable convolution of each inverted residual block includes sequentially connected point-by-point convolution PW, depth-wise convolution DW, and point-by-point convolution PW. The depth-wise convolution DW includes 3×3 convolution, normalization BN, and ReLU activation function. After the ReLU activation function operation, the CA attention mechanism is introduced. According to the output size and number of channels of the feature extraction network in Mask R-CNN, the parameters in the CA module are adjusted to ensure dimension matching, as shown in Figure 10 As shown in Figure 2, the fast pyramid pooling module FPPM is introduced after the last inverted residual block.

[0095] A lightweight feature extraction network is used to extract multi-scale initial feature maps of leafy vegetable images, including:

[0096] In each inverted residual block, features of leafy vegetable images are extracted Figure X , and use pooling to check features Figure X The horizontal and vertical coordinates of are encoded to obtain the channel output. In one embodiment, the pooling kernels (H, 1) and (1, W) are used to encode the input features respectively. Figure X The horizontal and vertical coordinates of are encoded, and the output of the cth channel with a height of h is expressed as formula (1):

[0097]

[0098] The same applies to the output in width. Based on the channel output, features are aggregated along the two spatial directions of feature dependency and feature position information to obtain a pair of direction-aware feature maps. The pair of direction-aware feature maps are spliced ​​together, and a 1×1 convolution operation is performed on the spliced ​​direction-aware feature maps using the convolution transformation function F1 to obtain the intermediate feature maps of the two directional spatial information, as shown in formula (2):

[0099] f=δ(F1([z h ,z w ])) (2);

[0100] Among them, [] represents the splicing operation along the spatial dimension, δ is the nonlinear activation function, f∈R C / r×(H+W) , r is the reduction ratio that controls the block size in the module. The intermediate feature map is decomposed into high f along the spatial dimension h and width f w Two independent tensors, and convert the two independent tensors into two with features Figure X For tensors with the same channel number, we get formulas (3) and (4):

[0101] g h =σ(F h (f h )) (3);

[0102] g w =σ(F w (f w )) (4);

[0103] The two transformed tensors g h and g w They are expanded and used as attention weights respectively, and multiplied with the input feature map to obtain the feature representation of channel attention to emphasize the feature representation of interest. The final output of CA is formula (5):

[0104]

[0105] The feature representation of the channel attention is passed through a 1×1 convolution operation and normalization operation to output the initial feature maps of branches of different scales; the initial feature maps of branches of different scales are upsampled respectively, and processed by 1×1 convolution and 3×3 convolution in turn, and the convolution processing results of multiple branches of different scales are mapped to feature maps and weighted fused through 1×1 convolution to obtain multi-scale initial feature maps.

[0106] Traditional feature aggregation methods often rely on contextual information at a single scale. The Pyramid Pooling Module (PPM) extracts multi-level image features through multi-scale pooling operations. Combining this with the FPN in Mask R-CNN enhances the model's global perception by introducing contextual information at different scales in leafy vegetable images, helping the model capture global information and improve the representation of fine-grained leaf features. The proposed Deep Pooling PPM (DAPPM) further improves PPM's contextual embedding capabilities, but it contains too many channels at each scale, resulting in a long runtime and unsuitable for real-time segmentation on mobile devices.

[0107] Therefore, the present invention proposes an FPPM module, which introduces PAPPM after the last inverted residual module of the backbone, and first upsamples the outputs of branches of different scales and then combines them with 1×1 convolution, and then passes through 3×3 convolution in parallel. It aggregates features from different scales of leafy vegetable images through multiple paths and performs convolution operations of groups=4. On this basis, the present invention reduces the number of channels of the vegetable feature map to 1 / 2 of the original, and only upsamples the feature maps of smaller sizes to the maximum size, reducing the frequency of upsampling and the amount of computation for each layer of convolution. Finally, 1×1 convolution is used to map and weightedly fuse all feature maps, so that high-level features capture the global information of leafy vegetables, and low-level features capture local leaf detail information, effectively reducing information loss in the leaf area of ​​the image, and achieving faster calculation and running speeds. The structure is as follows: Figure 11 As shown. For each input feature map F, the output after equations (6) and (7) is Ffp :

[0108]

[0109] F fp =Concat(F1,F2,…,F s ) (7);

[0110] Among them, H and W are the height and width of the feature map respectively, s represents different scales, and C 1×1 is a 1×1 convolution, C 3×3 is a 3×3 convolution, U represents upsampling operation, P j,k represents a pooling layer with a kernel size of j and a step size of k, P global Represents global average pooling.

[0111] In Mask R-CNN, FPN uses the pyramid hierarchy of convolutional neural networks to construct a feature pyramid with high-level semantics, and accurately locates the image through high-resolution feature maps and high-level semantic information (Lin et al. 2017). However, due to the lack of useful channel information, global information and underlying texture information in the FPN structure, it is impossible to obtain high-quality vegetable masks and high-precision vegetable categories in actual field scenes where multi-scale targets need to be accurately located and processed. Therefore, the present invention performs global information enhancement and path aggregation on the basic FPN, and proposes EPA-FPN based on GEM (Global-enhancementModule). The overall structure is as follows: Figure 3 shown.

[0112] (1) Introduction of GEM module. First, the data transmitted into FPN is convolved twice with 5×5 to obtain the feature information of the global image and obtain the global information matrix. Then, global average pooling (GAP) is used to reduce the size of the input vegetable image from H×W×C to 1×1×C. Finally, the channel score matrix is ​​multiplied with the original feature map and passed through 1×1 convolution to obtain the final feature map after information enhancement. The GEM module solves the problem of losing useful channel information in high-level feature mapping of vegetable datasets in FPN by making full use of global information in the process of feature fusion. The GEM structure is as follows: Figure 4 .

[0113] (2) Introduction of path aggregation method. With reference to the improvement of PANet, the adaptive feature pooling mechanism is introduced and the bottom-up path is added, such as Figure 3As shown in (b). Since the FPN feature map in the deep learning model Mask R-CNN uses [P2, P3, P4, P5], where [P2, P3, P4, P5] is a hierarchy of multi-scale feature maps, representing feature maps of different spatial resolutions; the PA module first downsamples the spatial size with a stride of 2 starting from P2 through a 3×3 convolution layer, and then subsamples the i+1th feature map P i+1 Each element of is horizontally connected to the down-sampled map, and the fused feature map is processed by another 3×3 convolution layer to generate the i+1th new feature map N i+1 , looping until P5, generating all new feature maps [N2, N3, N4, N5], which can more effectively transfer the precise positioning signals of the low layer to the entire feature pyramid, and enhance the feature pyramid's ability to locate the shape and structure of vegetables.

[0114] Step S4: The spatial domain information of the final feature map is converted into frequency domain information through block discrete cosine transform, each mixed patch is refined, and an n-dimensional DCT vector containing mask information is output.

[0115] The core of Mask R-CNN relies on convolutional neural networks (CNNs) to extract features. However, CNNs typically perform numerous convolution operations on high-resolution images, resulting in high computational overhead. Directly inputting leafy vegetable image data into the feature extraction network, the convolutional layers process the images, resulting in significant loss of image information. This results in significant feature loss in areas such as leaves and petals with irregular contours and highly similar phenotypic features, making accurate classification and segmentation of leafy vegetables difficult.

[0116] To improve the computational efficiency and representation capability of feature extraction of leafy vegetable images in the Mask R-CNN framework, the present invention introduces the Discrete Cosine Transform (DCT), which can effectively convert the spatial domain information of leafy vegetable images into frequency domain information, retaining the core information of leafy vegetable images, especially important details such as leaf edges and contours, thereby reducing data redundancy and achieving the effect of compressing images and improving segmentation accuracy.

[0117] Given a K×K mask M K×K , which is encoded into the frequency domain using DCT As shown in formula (8):

[0118]

[0119] When w=0, Otherwise C(w)=1. At this time, the non-zero values ​​are concentrated in The upper left corner of the image contains the most low-frequency elements of the vegetable mask information. Select n-dimensional DCT vector from , which represents the mask of the leafy vegetable. In the model inference stage, fill the remaining elements with zero to obtain Then use DCT inverse transform to get M K×K , the calculation of each pixel is as follows:

[0120]

[0121] M K×K Each pixel in All elements of M are calculated. When one element of the n-dimensional DCT vector is changed, M K×K All pixels in the DCT vector will be affected, so when DCT repairs the wrong segmented area, the correctly segmented area may also be affected unless all elements of the DCT vector are correctly refined, which is almost impossible to predict all elements correctly.

[0122] When refining the global DCT vector, the present invention introduces the PatchDCT method, extracts features from the P2 layer of the optimized EPA-FPN above, and then K×K Divide into m×m patches, and refine each patch separately. The overall structure of PatchDCT is as follows Figure 5 As shown in the figure, it mainly consists of a three-class classifier and a DCT vector regressor. Patches with only foreground leafy vegetable pixels and only background pixels are defined as foreground patches (green squares in the figure) and background patches (white squares in the figure), respectively, while the others are mixed patches (gray squares in the figure). After DCT, the foreground, background and mixed patches are easily distinguished by the classifier. At the same time, the classifier can also correct the incorrectly predicted foreground and background initial blocks. The DCT vector regressor is only applied to mixed patches, and each mixed patch is refined to output an n-dimensional DCT vector containing sufficient mask information. It is essentially a boundary attention module because the boundary part of the leafy vegetable mask exists in the mixed patch.

[0123] The feature sizes of PatchDCT input and output are exactly the same, so the data generated by one PatchDCT can be fed into another PatchDCT for further refinement, thereby using multiple PatchDCTs for multi-stage segmentation and refinement. The specific process is as follows: Figure 6 shown.

[0124] Step S5: Based on the final prediction mask, the segmentation result of the leafy vegetable image is obtained.

[0125] After taking the final multi-dimensional DCT vector as the final prediction mask, the final prediction mask is optimized by the boundary loss function, specifically:

[0126] After the model is built, it is optimized through the loss function, which is a key component used to measure the difference between the model prediction and the actual label. There are three types of loss functions in Mask R-CNN, namely classification loss L cls , bounding box regression loss L box and mask loss L mask , the above loss functions are combined to obtain the total loss L as shown in formula (10):

[0127] L=L cls +L box +L mask (10);

[0128] Since the mask branch needs to make predictions at the pixel level, which is much more detailed than the detection branch, the object contours obtained by instance segmentation are relatively rough. mask On this basis, by fusing Dice Loss and Binary Cross-Entropy Loss to optimize the boundary learning task, Boundary Loss is proposed to enhance the feature representation of boundary and mask prediction.

[0129] First, Binary Cross-Entropy Loss is used to measure the difference between the vegetable variety probability distribution predicted by the model and the actual vegetable label. The loss formula (11) for calculating a single vegetable sample is as follows:

[0130]

[0131] Among them, y is the true label, which takes the value of 0 or 1. is the probability that the model predicts that the sample is a positive class. For the entire vegetable dataset, the value of Binary Cross-Entropy Loss is the average loss of all vegetable samples, as shown in formula (12):

[0132]

[0133] Secondly, Dice Loss is used to measure the degree of overlap between the predicted vegetable segmentation mask and the true vegetable segmentation mask, that is, to calculate the IoU between the predicted value and the true value. It is insensitive to the number of pixels and can alleviate the problem of imbalanced categories in the vegetable dataset to a certain extent. Its formula is (13):

[0134]

[0135] Where H and W are the height and width of the predicted boundary map, respectively, i represents the i-th pixel, and ε is a smoothing term to avoid the denominator being 0 (set ε = 1).

[0136] Finally, the above two losses are weighted to obtain Boundary Loss L b The specific expression of formula (14) is as follows:

[0137] L b (p b ,y b )=L Dice (p b ,y b )+λL BCE (p b ,y b ) (14);

[0138] Among them, p b ∈R H×W and y b ∈R H×W They represent the predicted boundary of a specific category and the corresponding boundary truth value, respectively, and λ is a hyperparameter used to adjust the loss weight (set λ = 1).

[0139] Therefore, the loss function finally used in the present invention is as shown in formula (15), that is, the boundary loss function is obtained by weighted fusion of binary cross entropy loss and Dice loss, and combined with classification loss, bounding box regression loss and mask loss. The final predicted mask is optimized by the boundary loss function:

[0140] L MM =L cls +L box +L mask +L b (15);

[0141] The following experiments further illustrate the above scheme, specifically:

[0142] Experimental environment and settings:

[0143] The experiments used the PyTorch framework version of Mask R-CNN. The versions of the related libraries were cuda 11.6, torch 1.13.0, torchvision 0.14.0, torchaudio 0.13.0, and CUDNN 8.4.0. The hardware environment is shown in Table 6. The network was trained on four images in a batch, with the number of shorter and longer edges being 400 and 667, respectively. The number of iterations was 180,000, with momentum set to 0.9, weight decay to 0.0001, and the model learning rate to 0.001. All other parameters were set according to those of Mask R-CNN.

[0144] Table 6 Experimental hardware environment

[0145] name parameter Operation Learning Pass Windows 10 Pro GPU Nvidia GeForce RTX 3060 12G CPU AMD Ryzen 5 5500 @ 3.60GHz RAM 16G frame PyTorch 1.13.0 Development language Python 3.8

[0146] In terms of model classification and segmentation accuracy, the present invention uses Average Precision (AP), Average Recall (AR), AP 50 、AP 75 、AP s To evaluate. Precision refers to the proportion of samples that are actually positive among all samples predicted to be positive; Recall refers to the proportion of samples that are correctly predicted to be positive among all samples that are actually positive. AP is the area under the precision and recall curve (i.e., PR curve) at different IoU thresholds. 50 and AP 75 are the average precision when the IoU threshold is 0.5 and 0.75, respectively. AR is the average recall calculated when the IoU threshold varies from 0.50 to 0.95. It is an indicator to measure whether the detection box positioning of a model is accurate. Formulas (16) to (19) are the calculation formulas for precision, recall, average precision, and average recall.

[0147]

[0148] Among them, TP is the true classification result of correctly detecting vegetables when the IoU area exceeds a certain threshold, FP is the incorrectly detected vegetables, and FN is the undetected vegetables. In addition, the amount of floating-point calculations, the number of model parameters, and the inference speed are also key indicators for measuring model efficiency and practicality. The amount of calculation usually refers to the number of floating-point operations (Floating Point Operations, FLOPs) required for the model to perform a forward propagation, which is used to evaluate the computational complexity of the model. The number of parameters (Params) is the total number of trainable weights in the model, which is usually used to measure the size and complexity of the model. The inference speed refers to the time required for the model to complete a forward propagation, which is usually measured in frames per second (FPS) of the image.

[0149] Table 7 Segmentation performance of each improved module of FPN

[0150]

[0151] 1. Ablation experiment results and analysis of high-precision instance segmentation algorithm:

[0152] In order to strictly verify the effectiveness of the high-precision instance segmentation algorithm designed in this paper, Mask R-CNN-ResNet50 is used as the benchmark network. The proposed method and module are compared with the benchmark network in an ablation experiment on the self-made dataset MaskVeg.

[0153] The segmentation performance of the GEM module of the present invention, PA and the EPA-FPN constructed together are shown in Table 7. The segmentation performance after introducing DCT and PatchDCT on the benchmark network is shown in Table 8. Figure 7 The segmentation accuracy and segmentation loss of the validation set obtained by each improved model during the training process vary with the number of iterations.

[0154] Table 8 Segmentation performance of each DCT module

[0155] Baseline DCT PatchDCT AP (%) AP50(%) AP75 (%) AR (%) √ 88.558 97.911 97.558 88.7 √ 89.577(+1.019) 98.339(+0.428) 98.754(+1.196) 90.3(+1.6) √ 91.411(+2.853) 98.924(+1.013) 99.014(+1.456) 92.0(+3.3)

[0156] Table 9 Segmentation performance of each improved model

[0157]

[0158] Table 9 shows the segmentation performance of each improved module, both individually and in combination. Adding Boundary Loss to the baseline model's loss function slightly improves AP, AP50, AP75, and AR by 0.875%, 0.475%, 0.389%, and 0.6%, respectively. This demonstrates that Boundary Loss can enhance the optimization of mask contours for leafy vegetable segmentation in natural environments. The combination of EPA-FPN and PatchDCT performs better than either module alone, significantly improving overall segmentation performance. This is particularly evident in AP and AP50, which saw increases of 3.003% and 1.526%, respectively. The results show that after introducing global information enhancement and path aggregation to the FPN of the baseline model, introducing PatchDCT at the head of the mask branch, and adding BoundaryLoss to construct a multi-task joint loss function, the segmentation performance of the model is significantly improved. The AP, AP 50, AP75 and AR are improved by 3.185%, 1.23%, 1.196% and 4.0% respectively, reaching 91.743%, 99.141%, 98.754% and 92.7%.

[0159] 2. Ablation experiment analysis of the backbone network using a lightweight classifier:

[0160] In order to replace ResNet with lightweight classifiers in backbone networks, this paper investigates mainstream lightweight networks and conducts experimental verification on the self-made dataset MaskVeg. The average accuracy and complexity of each lightweight classifier are shown in Table 10, and the visual comparison effect is shown in Figure 12The results show that the MobileNetv2 architecture is well-suited to Mask R-CNN, reducing model parameters and computational complexity while maintaining a slight improvement in average precision. Compared to the baseline model, the model's AP value increased by 0.354%, while the number of parameters and computational complexity decreased by 44.9% and 32.8%, respectively, reaching 33.8M and 91.6G FLOPs, achieving a good balance.

[0161] Table 10 Comparison of average accuracy and complexity of each lightweight classifier

[0162] Backbone AP (%) Params(M) GFLOPs ResNet50 91.743 61.3 136.3 EfficientNet-b0 91.994(+0.251) 34.5(-26.8) 93.1(-43.2) EfficientNet-b1 91.587(-0.156) 34.9(-26.4) 93.4(-42.9) EfficientNet-b2 92.043(+0.300) 35.8(-25.5) 94.9(-41.4) ShuffleNet 91.238(-0.505) 32.3(-29.9) 88.1(-48.2) MobileNetv3-large 90.268(-1.475) 32.2(-29.1) 87.8(-48.5) MobileNeXt 92.147(+0.404) 35.7(-25.6) 94.7(-41.6) MobileNetv2 92.097(+0.354) 33.8(-27.5) 91.6(-44.7)

[0163] Table 11 shows the experimental results for the average accuracy, complexity, and speed of the model after adding the CA attention module and the FPPM module to the MobileNetv2 backbone as the baseline model. The simultaneous introduction of the CA and FPPM modules significantly improves the average accuracy of the model compared to the sum of the individual modules, indicating a good integration of the two modules with only a slight increase in computational complexity. This achieves an optimal balance between accuracy, complexity, and speed.

[0164] Table 11 Performance comparison of each improved module

[0165] Model AP (%) Params(M) GFLOPs FPS Baseline 92.097 33.8 91.6 28.1 +CA 92.997(+0.900) 34.1(+0.3) 91.8(+0.2) 28.0(-0.1) +FPPM 92.258(+0.161) 34.7(+0.9) 92.6(+1.0) 36.4(+8.3) +CA,FPPM 93.303(+1.206) 34.8(+1.0) 92.9(+1.3) 35.6(+7.5)

[0166] The performance comparison of the final improved model of the present invention and the original Mask R-CNN model is shown in Table 12. The results show that by adopting EPA-FPN in the FPN part, introducing PatchDCT in the mask branch, adopting the MobileNetv2 lightweight classifier in the backbone one part and introducing the CA attention mechanism and FPPM module, and using Boundary Loss at the same time, the model achieves an optimal balance between accuracy, complexity and running speed. The AP, AP50, AP75 and AR reach 93.303%, 99.437%, 98.476% and 94.8%, respectively, which are improved by 4.745%, 1.526%, 0.918% and 5.9%, respectively. The number of parameters and computational complexity decreases by 15.7% and 20.1%, respectively. The inference speed reaches 35.6FPS, which is an improvement of 71.1%.

[0167] Table 12 Performance comparison between the improved model and the original model

[0168]

[0169] The classification and segmentation accuracy of leafy vegetables for various varieties and growth stages is shown in Tables 13 and 14. To verify the reliability of the proposed model, the model was used to classify and segment images from a test set. The collected cabbage stalk data was divided into three categories: early maturing (such as 702 pointed leaves and Hybrid 91 Chinese cabbage heart), mid-maturing (such as Guotai 191 and 70 round leaves), and late maturing (such as Jiangmen White and Yutian 3). The results show that the proposed model can distinguish leafy vegetable varieties and growth stages in images, locate them, and determine leaf outlines, thus achieving the classification and segmentation of leafy vegetables in a single, natural field environment.

[0170] In order to objectively evaluate the proposed model, the MaskRCNN model was compared with other mainstream instance segmentation algorithms (MS R-CNN, BlendMask, Spar seInst, ours, etc.) on the self-made leafy vegetable dataset MaskVeg. As shown in Table 14, the visual comparison is as follows: Figure 13 Experiments demonstrate that the improved strategy of the present invention achieves an optimal balance between classification and segmentation accuracy and model complexity. The AP, AP50, and AP75 of the proposed model are all higher than those of mainstream segmentation models, while using only 34.8M parameters, far fewer than other single-stage or two-stage instance segmentation algorithms. This demonstrates that the proposed model achieves superior classification and segmentation performance with smaller network parameters and model size, making it suitable for applications with limited computing power and storage space but demanding speed and accuracy.

[0171] Table 13-1 Classification and segmentation accuracy of each variety

[0172]

[0173] Table 13-2 Appendix of classification and segmentation accuracy of various varieties

[0174]

[0175] Table 14 Classification and segmentation results of different networks on MaskVeg

[0176]

[0177] The present invention proposes a leafy vegetable image segmentation system, which comprises a data acquisition module, a feature map acquisition module, a mask prediction module and an output module.

[0178] Among them, the data acquisition module is used to collect leafy vegetable images and extract the multi-scale initial feature maps of the leafy vegetable images; the feature map acquisition module is used to perform several convolutions on the multi-scale initial feature maps to obtain the global information matrix, downsample the global information matrix, multiply the downsampled vector with the initial feature map, and obtain the information enhanced multi-scale feature map through convolution, and generate a new feature map for the multi-scale feature map in a bottom-up path, and fuse the new feature map to obtain the final feature map; the mask prediction module is used to generate an initial prediction mask based on the final feature map, and divide the initial prediction mask into multiple image blocks p After patch, the image block patch is subjected to block discrete cosine transform (DCT), the foreground and background image block patches that are incorrectly predicted after DCT transform are corrected, and the boundary of the image block patch with both foreground and background is optimized, and a multi-dimensional DCT vector containing mask information is output. The multi-dimensional DCT vector is spliced ​​with the features of the multi-scale feature map, and the spliced ​​features are subjected to DCT transform and then the multi-dimensional DCT vector is obtained again multiple times, with the final multi-dimensional DCT vector as the final prediction mask; the output module is used to obtain the segmentation result of the leafy vegetable image based on the final prediction mask.

[0179] The present invention also provides a computer device, comprising a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a leafy vegetable image segmentation method.

[0180] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., a keyboard, a pointing device, Bluetooth communications, etc.), or with any device that enables a computing device to communicate with one or more other computing devices (e.g., a router, a modem, etc.).

[0181] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned leafy vegetable image segmentation method are implemented.

[0182] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. For those skilled in the art to which the present invention belongs, several simple deductions or replacements can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A leafy vegetable image segmentation method, characterized in that: include: Collect leafy vegetable images and extract multi-scale initial feature maps of the leafy vegetable images; Perform several convolutions on the multi-scale initial feature map to obtain a global information matrix, downsample the global information matrix, multiply the downsampled vector with the initial feature map, and obtain an information-enhanced multi-scale feature map through convolution. Generate a new feature map for the multi-scale feature map in a bottom-up path, and fuse the new feature maps to obtain the final feature map; Based on the final feature map, an initial prediction mask is generated. After dividing the initial prediction mask into multiple image patches, the image patches are subjected to block discrete cosine transform (DCT) transformation. Foreground and background image patches that are incorrectly predicted after the DCT transformation are corrected, and the boundaries of image patches where both foreground and background exist are optimized. A multidimensional DCT vector containing mask information is output, and the multidimensional DCT vector is concatenated with features of the multi-scale feature map. The concatenated features are subjected to DCT transformation, and then the multidimensional DCT vector is obtained multiple times again, with the final multidimensional DCT vector being used as the final prediction mask. Based on the final prediction mask, a segmentation result of the leafy vegetable image is obtained.

2. The leafy vegetable image segmentation method according to claim 1, wherein: The bottom-up approach to generating a new feature map from a multi-scale feature map includes: In the deep learning model Mask R-CNN, the FPN feature map [P2, P3, P4, P5] is used. The spatial size is downsampled with a stride of 2 starting from P2 through a 3×3 convolutional layer to obtain the downsampled map. [P2, P3, P4, P5] is the level of the multi-scale feature map, representing feature maps of different spatial resolutions. The i+1th feature map P i+1 Each element of is horizontally connected to the down-sampled image to obtain the fused feature map, which is then processed through a 3×3 convolution layer to generate the i+1th new feature map N i+1 , and loop until P5 to generate all new feature maps [N2, N3, N4, N5].

3. The leafy vegetable image segmentation method according to claim 1, wherein: After the final multi-dimensional DCT vector is used as the final prediction mask, the final prediction mask is optimized by the boundary loss function, specifically: The boundary loss function is obtained by weighted fusion of binary cross entropy loss and Dice loss, and combined with classification loss, bounding box regression loss and mask loss. The final predicted mask is optimized through the boundary loss function. Among them, the binary cross entropy loss is used to obtain the average loss of all leafy vegetable samples, and the Dice loss is used to measure the degree of overlap between the predicted vegetable segmentation mask and the actual vegetable segmentation mask.

4. The leafy vegetable image segmentation method according to claim 1, wherein: The method for extracting the multi-scale initial feature map of the leafy vegetable image is as follows: Construct a lightweight feature extraction network consisting of a lightweight classifier backbone network with multiple inverted residual blocks, a CA attention mechanism, and a fast pyramid pooling module (FPPM); each inverted residual block corresponds to a CA attention mechanism; Multi-scale initial feature maps of leafy vegetable images are extracted through a lightweight feature extraction network.

5. The leafy vegetable image segmentation method according to claim 4, wherein: The construction includes a lightweight classifier backbone network of multiple inverted residual blocks, a CA attention mechanism, and a lightweight feature extraction network of a fast pyramid pooling module FPPM, specifically: The inverted residual block structure of MobileNetv2 is introduced into Mask R-CNN as the backbone network of the lightweight classifier. Specifically, the inverted residual block structure of MobileNetv2 is as follows: the last average pooling layer, fully connected layer, and softmax layer of the MobileNetv2 network are deleted, and the MobileNetv2 network structure is divided into five inverted residual blocks C1 to C5. C1 contains one standard convolution and one depthwise separable convolution, C2 and C3 contain two depthwise separable convolutions respectively, C4 contains six depthwise separable convolutions, and C5 contains two depthwise separable convolutions. The depth-wise separable convolution of each inverted residual block includes sequentially connected point-by-point convolution PW, depth-wise convolution DW and point-by-point convolution PW. The depth-wise convolution DW includes 3×3 convolution, normalization BN and ReLU activation function, and the CA attention mechanism is introduced after the ReLU activation function; the fast pyramid pooling module FPPM is introduced after the last inverted residual block.

6. The leafy vegetable image segmentation method according to claim 5, characterized in that: The method of extracting a multi-scale initial feature map of a leafy vegetable image by a lightweight feature extraction network includes: In each inverted residual block, the feature map X of the leafy vegetable image is extracted, and the horizontal and vertical coordinates of the feature map X are encoded using the pooling kernel to obtain the channel output; Based on the channel output, features are aggregated along two spatial directions, namely, feature dependency and feature position information, to obtain a pair of direction-aware feature maps, the pair of direction-aware feature maps are spliced, and a 1×1 convolution operation is performed on the spliced ​​direction-aware feature maps using a convolution transformation function F1 to obtain an intermediate feature map of two directional spatial information; Decompose the intermediate feature map into two independent tensors in the height and width directions along the spatial dimension, and convert the two independent tensors into two tensors with the same channel number as the feature map X; The two converted tensors are used as attention weights and multiplied with the input feature map to obtain the feature representation of channel attention. The feature representation of channel attention is then passed through a 1×1 convolution operation and normalization operation to output the initial feature maps of branches of different scales. The initial feature maps of branches of different scales are upsampled separately and processed by 1×1 convolution and 3×3 convolution in sequence. The convolution processing results of multiple branches of different scales are mapped to feature maps and weighted fused by 1×1 convolution to obtain multi-scale initial feature maps.

7. A leafy vegetable image segmentation system, characterized in that: include: A data acquisition module is used to collect leafy vegetable images and extract multi-scale initial feature maps of the leafy vegetable images; The feature map acquisition module is used to perform several convolutions on the multi-scale initial feature map to obtain a global information matrix, downsample the global information matrix, multiply the downsampled vector with the initial feature map, and obtain an information-enhanced multi-scale feature map through convolution. A new feature map is generated for the multi-scale feature map in a bottom-up path, and the new feature maps are fused to obtain the final feature map; A mask prediction module is configured to generate an initial prediction mask based on the final feature map, divide the initial prediction mask into multiple image patches, perform block discrete cosine transform (DCT) on the image patches, correct foreground and background image patches that are incorrectly predicted after DCT, optimize the boundaries of image patches where both foreground and background exist, output a multidimensional DCT vector containing mask information, concatenate the multidimensional DCT vector with features of the multiscale feature map, perform DCT on the concatenated features, and then re-acquire multiple multidimensional DCT vectors, using the final multidimensional DCT vector as the final prediction mask; The output module is used to obtain the segmentation result of the leafy vegetable image based on the final prediction mask.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the leafy vegetable image segmentation method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the leafy vegetable image segmentation method according to any one of claims 1 to 6 are implemented.

Images