Image segmentation method, system and equipment based on deep learning and differential expert model

Abstract

The invention discloses an image segmentation method, system and device based on deep learning and a difference expert model, and relates to the field of image segmentation, and the method comprises the steps: obtaining a target corn image; constructing an image segmentation model, and training the image segmentation model by using the corn sample image training set; the image segmentation model comprises an encoder and a decoder, the encoder comprises a plurality of encoding sub-modules, each encoding sub-module comprises two KVFormer units, each KVFormer unit comprises a key value pair matching attention mechanism sub-unit, the decoder comprises a plurality of decoding sub-modules, each decoding sub-module comprises a differential hybrid expert model unit, and the differential hybrid expert model unit comprises a differential hybrid expert model unit. The differential hybrid expert model unit comprises a plurality of expert subunits with the same structure and subtraction operation subunits; inputting the target corn image into the trained image segmentation model for target segmentation; targets include corn kernels, corncobs and corn bracts. According to the invention, the precision and efficiency of image segmentation are improved.

Description

Technical Field

[0001] The present application relates to the field of image segmentation, and in particular, to an image segmentation method, system and device based on deep learning and differential expert model. Background Art

[0002] Image segmentation technology has been widely used in various links of agricultural production such as plowing, sowing, managing and harvesting, such as the detection of the impurity content of corn. However, corn images often contain corn cobs and corn husks, etc. When detecting the impurity content of corn, they often show significant similarities, which are prone to cause misidentification of the image segmentation model. Secondly, corn images contain various shapes such as corn kernels, corn cobs and corn husks, which requires the image segmentation model to have extremely high generalization ability. In addition, the trained image segmentation model needs to be deployed to the edge computing device in agricultural machinery and equipment for the inference process, while the computing power of devices commonly used in the agricultural field such as NVIDIA Jetson TX2, Raspberry Pi, Jetson Nano, etc. is relatively low. Therefore, the detection frame rate of the image segmentation model is low, and it is difficult to meet the real-time requirements in the agricultural production process.

[0003] Generally speaking, the deep learning models in the current image segmentation field still face problems such as serious misidentification, weak generalization ability and poor real-time performance. Summary of the Invention

[0004] The purpose of the present application is to provide an image segmentation method, system and device based on deep learning and differential expert model, which can improve the accuracy and efficiency of image segmentation.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In the first aspect, the present application provides an image segmentation method based on deep learning and differential expert model, including:

[0007] Obtain a target corn image;

[0008] Construct an image segmentation model, and train the image segmentation model with a corn sample image training set; the image segmentation model includes an encoder and a decoder, wherein the encoder includes a plurality of encoding sub-modules, each encoding sub-module includes two KVFormer units, the KVFormer unit includes a key-value pair matching attention mechanism sub-unit, the decoder includes a plurality of decoding sub-modules, each decoding sub-module includes a differential mixture of experts model unit, and the differential mixture of experts model unit includes a plurality of expert sub-units with the same structure and a subtraction operation sub-unit;

[0009] Input the target corn image into the trained image segmentation model for target segmentation; the targets include corn kernels, corn cobs and corn husks.

[0010] In a second aspect, the present application provides an image segmentation system based on deep learning and a differential expert model, including:

[0011] An acquisition module, configured to acquire a target corn image;

[0012] A construction and training module, configured to construct an image segmentation model and train the image segmentation model using a training set of corn sample images; the image segmentation model includes an encoder and a decoder, wherein the encoder includes a plurality of encoding sub-modules, each encoding sub-module includes a plurality of KVFormer units, and a KVFormer unit includes a key-value pair matching attention mechanism sub-unit, the decoder includes a plurality of decoding sub-modules, each decoding sub-module includes a differential mixture of experts model unit, and the differential mixture of experts model unit includes a plurality of expert sub-units with the same structure and a subtraction operation sub-unit;

[0013] A segmentation module, configured to input the target corn image into the trained image segmentation model for target segmentation; the targets include corn kernels, corn cob cores, and corn husks.

[0014] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the above-mentioned image segmentation method based on deep learning and a differential expert model.

[0015] According to the specific embodiments provided by the present application, the following technical effects are disclosed in the present application:

[0016] The present application provides an image segmentation method, system, and device based on deep learning and a differential expert model, acquires a target corn image, and inputs the target corn image into the trained image segmentation model for target segmentation. The encoder in the image segmentation model includes a plurality of encoding sub-modules, each encoding sub-module includes two KVFormer units, and the key-value pair matching attention mechanism sub-unit in the KVFormer unit can establish the mutual dependence between each segmentation window, reducing the computational complexity. Based on this, the encoder can effectively reduce the computational complexity of the image segmentation model and improve the recognition efficiency; the decoder in the image segmentation model includes a plurality of decoding sub-modules, each decoding sub-module includes a differential mixture of experts model unit, and the differential mixture of experts model unit uses a plurality of expert sub-units with the same structure. The feature extraction ability of each expert sub-unit is uniform and balanced, effectively avoiding the phenomenon of uneven training effects between different expert models. At the same time, subtraction operations are used without introducing additional interference. Based on this, the decoder can more accurately distinguish different visual features with high similarity, thereby improving the segmentation accuracy. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0018] Figure 1 It is an application environment diagram of an image segmentation method based on deep learning and differential expert model in an embodiment of the present application;

[0019] Figure 2 It is a schematic flowchart of an image segmentation method based on deep learning and differential expert model provided in an embodiment of the present application;

[0020] Figure 3 It is a structural diagram of an image segmentation model provided in an embodiment of the present application;

[0021] Figure 4 It is a structural diagram of an encoding sub-module provided in an embodiment of the present application;

[0022] Figure 5 It is a structural diagram of the KVFormer unit in the encoding sub-module provided in an embodiment of the present application;

[0023] Figure 6 It is a structural diagram of the underlying decoding sub-module provided in an embodiment of the present application;

[0024] Figure 7 It is a structural diagram of the DMoE model unit provided in an embodiment of the present application;

[0025] Figure 8 It is a structural diagram of the second expert sub-unit provided in an embodiment of the present application;

[0026] Figure 9 It is the segmentation result of the target corn image;

[0027] Figure 10 It is the segmentation result of the corn plant;

[0028] Figure 11 It is the segmentation result of the leaf disease area;

[0029] Figure 12 It is a schematic structural diagram of a computer device provided in an embodiment of the present application. Specific Embodiments

[0030] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0031] To make the objectives, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0032] The image segmentation method based on deep learning and differential expert model provided by the embodiments of the present application can be applied to, for example, Figure 1 the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the target corn image to the server 104. After receiving the target corn image, for the target corn image, the server 104 inputs the target corn image into the trained image segmentation model for target segmentation. The server 104 can feedback the obtained target segmentation result to the terminal 102. In addition, in some embodiments, the image segmentation method based on deep learning and differential expert model can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform segmentation processing on the target corn image, or the server 104 can obtain the target corn image from the data storage system for segmentation processing.

[0033] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0034] In an exemplary embodiment, as Figure 2 shown, an image segmentation method based on deep learning and differential expert model is provided. This method is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or jointly executed by a terminal and a server. In the embodiments of the present application, taking this method applied to Figure 1 the server 104 in as an example for description, it includes the following steps 201 to step 203. Among them:

[0035] Step 201, obtain the target corn image.

[0036] Step 202, construct an image segmentation model and train the image segmentation model using a corn sample image training set; the image segmentation model includes an encoder and a decoder. Among them, the encoder includes multiple encoding sub-modules, each encoding sub-module includes two KVFormer units, and the KVFormer unit includes a key-value pair matching attention mechanism sub-unit. The decoder includes multiple decoding sub-modules, each decoding sub-module includes a differential mixture of experts model unit, and the differential mixture of experts model unit includes multiple expert sub-units with the same structure and a subtraction operation sub-unit.

[0037] Step 203, input the target corn image into the trained image segmentation model for target segmentation; the targets include corn kernels, corn cobs, and corn husks.

[0038] Implementing the above steps 201 to 203 can improve the accuracy and efficiency of the corn image segmentation result.

[0039] In an exemplary embodiment, step 202 specifically includes steps 301 - 302:

[0040] Step 301, perform downsampling operations on the target corn image through the encoder to obtain feature maps of different scales.

[0041] The encoder contains multiple encoding sub-modules, each encoding sub-module includes two KVFormer units, and the two KVFormer units are the first KVFormer unit and the second KVFormer unit respectively; each encoding sub-module includes a first convolutional layer, a first BatchNorm layer, a first KVFormer unit, and a second KVFormer unit. In a single encoding sub-module, the two KVForme units are cascaded together, the output of the first KVFormer unit is used as the input of the second KVFormer unit, and the output of the second KVFormer unit is passed to the next encoding sub-module.

[0042] For the i-th encoding sub-module, the input is F i-1 , and the output is F i , F i-1 , F i are the feature maps output by the (i - 1)-th encoding sub-module and the i-th encoding sub-module respectively. When i is equal to 1, F0 is the target corn image, and i takes positive integers greater than or equal to zero.

[0043] The processing process of the i-th encoding sub-module specifically includes steps 11 - 14:

[0044] Step 11, the feature map F iInput into the first convolutional layer to extract detailed feature information, obtaining the feature map F i1 。

[0045] Step 12: Input the feature map F i1 into the first BatchNorm layer for regularization processing, obtaining the feature map F i2 。

[0046] Step 13: Input the feature map F i2 into the first KVFormer unit to extract high-level semantic features, obtaining the feature map F i3 。

[0047] Step 14: Input the feature map F i3 into the second KVFormer unit to extract high-level semantic features, obtaining the feature map F i+1 。

[0048] In this embodiment, the encoder includes four encoding sub-modules, i takes the value of 4. As Figure 3 shown, the first convolutional layer in the first encoding sub-module 1 is a convolutional layer with a stride of 4 and a convolution kernel of 4×4; the first convolutional layers in the other three encoding sub-modules are convolutional layers with a stride of 2 and a convolution kernel of 2×2. The input feature map F0 is processed by the encoder to obtain the feature maps F1, F2, F3, and F4 respectively. The size of the feature map F1 is [64, H / 4, W / 4], the size of the feature map F2 is [128, H / 8, W / 8], the size of the feature map F3 is [256, H / 16, W / 16], and the size of the feature map F4 is [512, H / 32, W / 32], where H represents the length of the input image and W represents the width of the input image.

[0049] As Figure 4 shown, taking the processing process of the first encoding sub-module as an example, the input feature map F0 of the first encoding sub-module is the target corn image. After being processed by the first convolutional layer, the first BatchNorm layer, and two KVFormer units, the feature maps F 01 、feature map F 02 、feature map F 03 and feature map F1 are obtained respectively.

[0050] The KVFormer unit consists of a cascaded Key-Value Match Attention (KVA) sub-unit and a Feedforward Network (FFN). The KVA sub-unit includes a multi-scale depth feature extraction layer, a feature concatenation layer, a second convolutional layer, a third convolutional layer, a matrix dot product layer, and a second BatchNorm layer. Among them, the multi-scale depth feature extraction layer includes 6 depthwise separable convolutional layers with different dilation rates (dilation rates are 1, 2... 6 respectively) and a convolutional kernel of 3×3. Both the second convolutional layer and the third convolutional layer are convolutional layers with a convolutional kernel of 1×1.

[0051] Each Feedforward Neural Network layer in the KVFormer unit includes three convolutional layers with a convolutional kernel of 1×1, a depthwise separable convolutional layer with a convolutional kernel of 3×3, a ReLU activation layer, and a BatchNorm layer.

[0052] The processing procedures of the first KVFormer unit and the second KVFormer unit in each encoding sub-module are the same. As Figure 5 shown, for the first KVFormer unit in the i-th encoding sub-module, it includes a KVA sub-unit 410 and a Feedforward Neural Network layer 420. Step 13 specifically includes steps 131 - 137:

[0053] Step 131, input the feature map F i2 into the second convolutional layer for feature extraction to obtain the V value. The input feature map F i2 has a size of [C, H, W], where C represents the number of channels of the input image.

[0054] Step 132, input the feature map F i2 into the multi-scale depth feature extraction layer 180 to extract high-level semantic features and obtain a multi-scale depth feature map. Figure 5 The multi-scale depth feature maps in

[0055] are respectively feature map 411, feature map 412... feature map 416, and a total of six different-scale feature maps are obtained.

[0056] Step 133, input the multi-scale depth feature map into the feature concatenation layer 170 for feature concatenation to obtain a multi-scale fusion feature map 450.

[0057] Step 134, input the multi-scale fusion feature map 450 into the third convolutional layer for feature extraction to obtain the attention weight 460.

[0058] Step 136: Input the weighted fusion feature map into the second BatchNorm layer for regularization processing to obtain the regularized fusion feature map 417.

[0059] Step 137: Input the regularized fusion feature map 417 into the feedforward neural network layer for feature extraction to obtain the feature map F. i3 。

[0060] During the training process of the image segmentation model, the multi-scale fusion feature map extracts features of different scales in an adaptive manner for the feature map F. i2 In traditional Transformers, Self-Attention (SA) establishes the interdependence between each pixel point. In contrast, KVA establishes the interdependence between each window, where the size of the window is the size of the convolutional kernel in the multi-scale fusion feature map.

[0061] The computational complexity Ω of SA SA is defined as:

[0062] Ω SA = 4C 2 HW + 2C(HW) 2 (1)

[0063] The computational complexity Ω of KVA KVA is defined as:

[0064] Ω KVA = HWC′(7C + 456) (2)

[0065] Comparing formula (1) and formula (2), it can be clearly found that KVA effectively reduces the computational complexity of SA, solves the problem that the computational complexity of SA is quadratically correlated with the input image size, and thus reduces the demand for computing resources.

[0066] Step 302: Upsample the feature maps of different scales through the decoder to obtain the high-resolution feature map. The number of decoding sub-modules in the decoder is the same as the number of encoding sub-modules in the encoder.

[0067] For the decoding sub-module in the middle layer, the inputs are the feature map F i-1 and the feature map H j+1 , and the output is the feature map H j ; for the decoding sub-module at the top layer, the input is the feature map H j+1 , and the output is the feature map H j ; for the decoding sub-module at the bottom layer, the inputs are the feature map F i and the feature map F i-1 , and the output is the feature map H j; where j is the serial number of the decoding sub-module; there is 1 decoding sub-module at the top layer, 1 decoding sub-module at the bottom layer, and multiple decoding sub-modules in the middle layer. For the decoding sub-module at the top layer, the serial number is 1, and from the top layer to the bottom layer, the serial numbers of each encoding sub-module increase one by one.

[0068] The decoding sub-module at the top layer includes an upper first sampling layer, a fourth convolutional layer, and a Difference Mixture-of-Experts (DMoE) model unit; the first sampling layer is a 4-fold bilinear interpolation upsampling layer; the decoding sub-modules in the middle layer and at the bottom layer both include a second sampling layer, a matrix splicing operation layer, a fourth convolutional layer, and a DMoE model unit, and the second sampling layer is a 2-fold bilinear interpolation upsampling layer.

[0069] As Figure 6 shown, the processing process of the decoding sub-module at the bottom layer specifically includes steps 21 - step 24:

[0070] Step 21, input the feature map F i into the second sampling layer for sampling to obtain the first feature map 310.

[0071] Step 22, input the first feature map 310 and the feature map F i-1 into the matrix splicing operation layer 170 for feature splicing to obtain the second feature map 121.

[0072] Step 23, input the second feature map 121 into the fourth convolutional layer for feature extraction to obtain the third feature map 320.

[0073] Step 24, input the third feature map 320 into the DMoE model unit for feature extraction to obtain the feature map H j .

[0074] As Figure 3As shown in the figure, taking the decoder including four decoding sub-modules as an example to illustrate the processing process of the underlying decoding sub-module. Among them, there is 1 top-level decoding sub-module, 1 bottom-level decoding sub-module, and 2 middle-level decoding sub-modules, namely the decoding sub-module of the first middle layer and the decoding sub-module of the second middle layer. The input of the bottom-level decoding sub-module is the feature map F3 and the feature map F4, and the output is the feature map H4; the input feature maps of the decoding sub-module of the first middle layer are F2 and H4, and the output is the feature map H3; the input of the decoding sub-module of the second middle layer is the feature map F1 and the feature map H3, and the output is the feature map H2; the input of the top-level decoding sub-module is the feature map H2, and the output is the feature map H1; among them, the size of the feature map F4 is [512, H / 32, W / 32], the size of the first feature map is [256, H / 16, W / 16], the size of the second feature map is [768, H / 16, W / 16], the size of the third feature map is [256, H / 16, W / 16], and the size of the feature map H1 is [128, H / 16, W / 16]. After passing through the middle-level decoding sub-module and the top-level decoding sub-module, the final output feature map size is [N, H, W], where N represents the number of targets in the target segmentation task. In this embodiment, N takes the value of 4, and the number of targets here includes the image background.

[0075] The processing process of the middle-level decoding sub-module specifically includes steps 31 - step 34:

[0076] Step 31, input the feature map H j+1 into the second sampling layer for sampling to obtain the fourth feature map.

[0077] Step 32, input the fourth feature map and the feature map F i-1 into the matrix splicing operation layer 170 for feature splicing to obtain the fifth feature map.

[0078] Step 33, input the fifth feature map into the fourth convolutional layer for feature extraction to obtain the sixth feature map.

[0079] Step 34, input the sixth feature map into the DMoE model unit for feature extraction to obtain the feature map H j .

[0080] The processing process of the top-level decoding sub-module specifically includes steps 41 - step 43:

[0081] Step 41, input the feature map H j+1 into the first sampling layer for sampling to obtain the seventh feature map.

[0082] Step 42, input the seventh feature map into the fourth convolutional layer for feature extraction to obtain the eighth feature map.

[0083] Step 43: Input the eighth feature map into the DMoE model unit for feature extraction to obtain a feature map H j .

[0084] In this embodiment, there are three expert subunits in the DMoE model unit, namely the first expert subunit E1, the second expert subunit E2 and the third expert subunit E3; the processing process of the DMoE model unit in the top-level decoding submodule, the middle-level decoding submodule and the bottom-level decoding submodule is the same.

[0085] like Figure 7 As shown, for the DMoE model unit in the top-level decoding submodule, the processing process is steps 51 to 55:

[0086] Step 51 , input the eighth feature map 320 into E1 for feature extraction to obtain a first expert feature map 511 .

[0087] Step 52 , input the eighth feature map 320 into E2 for feature extraction to obtain a second expert feature map 521 .

[0088] Step 53: weighting the second expert feature map 521 to obtain a second expert feature map with a preset multiple. The preset multiple in this embodiment is α.

[0089] Step 54 , inputting the second expert feature map 521 and the first expert feature map 511 of the preset multiple into the subtraction operation subunit 540 for subtraction operation to obtain a fused expert feature map 531 .

[0090] Step 55: Input the fusion expert feature map 531 to E3 for feature extraction to obtain a feature map H j ,like Figure 7 Feature map 330 in .

[0091] When the number of decoding submodules in the decoder and the number of encoding submodules in the encoder are both 4, the size of the eighth feature map is [256, H / 16, W / 16], the size of the first expert feature map is [256, H / 16, W / 16], the size of the second expert feature map is [256, H / 16, W / 16], the size of the fused expert feature map is [256, H / 16, W / 16], and the size of the feature map H1 is [256, H / 16, W / 16].

[0092] The three expert sub-units process the input feature map in the same way, such as Figure 8As shown, taking the second expert sub-unit E2 as an example, the size of the eighth feature map 320 is [256, H / 16, W / 16]. After the input eighth feature map 320 continuously passes through a convolutional layer with a convolution kernel of 3×3, a BatchNorm layer, a ReLU activation layer, and a convolutional layer with a convolution kernel of 3×3 for feature extraction, the second expert feature map 521 is obtained, and the size of the second expert feature map 521 is [256, H / 16, W / 16].

[0093] Step 303: Obtain the target segmentation result based on the high-resolution feature map. Specifically, calculate the class probability of each pixel through Softmax and output the final target segmentation result.

[0094] The application of the image segmentation model in this application is Scenario 1, and it can also be applied to other scenarios. For example, Scenario 2: Deploy the trained image segmentation model to the edge computing device of a field weeding robot to identify and classify weeds, and the target is the corn plant at this time; Scenario 3: Deploy the trained image segmentation model to the edge computing device of a sprayer for pest and disease detection, and the target is the leaf disease area at this time.

[0095] The training process of the image segmentation model in this application: The training framework of the image segmentation model is Pytorch. In the forward propagation stage of the image segmentation model, the input corn sample image extracts high-level semantic features through the downsampling operations of multiple consecutive encoding sub-modules in the encoder and generates low-resolution feature maps of different sizes. Then, the decoder restores the spatial information of the features through the upsampling operations of multiple consecutive decoding sub-modules and skip connections and generates a high-resolution feature map. Finally, calculate the class probability of each pixel through Softmax and output the final target segmentation result.

[0096] In the backpropagation stage, use the error backpropagation algorithm (Backpropagation) to calculate the gradient of each layer's parameters. The optimizer Adam updates the network weights according to the gradient information, with an initial learning rate of 10e-3, a learning rate strategy of linear, a batch size of 8, a training epoch of 120, and a dropout rate of 0.1. The image segmentation model uses the cross-entropy loss function L (CrossEntropy Loss) as the loss value for calculating the prediction result and the label, and L is defined as formula (3). In addition, the image segmentation model uses the BatchNorm layer to accelerate training and improve generalization ability.

[0097]

[0098] Among them, i is the i-th corn sample image, N is the total number of corn sample images in the corn sample image training set, L i The loss value of the i-th corn sample image, y i is the true value, pi is the predicted value.

[0099] During the training process, to evaluate the model performance and prevent overfitting, Pixel Accuracy (PA) and Mean Intersection over Union (MIoU) are used as evaluation metrics. Among them, PA is used to measure the proportion of all pixels classified correctly and is defined by Equation (4). MIoU is used to calculate the mean IoU of each category and measure the overlap degree between the predicted value and the ground truth, and is defined by Equation (5).

[0100]

[0101] Finally, when the image segmentation model is completed, or during the training process of the image segmentation model, the training is terminated in advance after the loss value stops decreasing. The image segmentation model loads the optimal training weights and performs the final evaluation on the test set to ensure the generalization ability.

[0102] In Scenario 2, a corn plant graphic sample set is adopted, and in Scenario 3, a pest and disease image sample set is used to train the image segmentation model. The loss values of the trained image segmentation model on the three scenario training sets are respectively stabilized around 0.07, 0.03, and 0.04. During the test process, the PA and MIoU of the image segmentation model in the Scenario 1 test set reach 95.2% and 91.6% respectively, the PA and MIoU in the Scenario 2 test set reach 97.3% and 94.7% respectively, and the loss values in the Scenario 3 test set reach 96.8% and 93.8% respectively. In addition, Figure 9 、 Figure 10 and Figure 11 respectively show the segmentation results in the three scenarios. The image segmentation model has a good segmentation effect on different targets and demonstrates excellent generalization ability. In summary, the image segmentation model proposed in this application shows strong segmentation performance and application potential in the agricultural scene segmentation task.

[0103] Based on the same inventive concept, the embodiment of the present application also provides an image segmentation system based on deep learning and differential expert model for implementing the above-mentioned image segmentation method based on deep learning and differential expert model. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the following image segmentation system based on deep learning and differential expert model can refer to the limitations on the image segmentation method based on deep learning and differential expert model in the above text, and will not be repeated here.

[0104] In an exemplary embodiment, an image segmentation system based on deep learning and differential expert model is provided, including:

[0105] An acquisition module for acquiring a target corn image.

[0106] A construction and training module for constructing and training an image segmentation model; the image segmentation model includes an encoder and a decoder. Among them, the encoder includes multiple encoding sub-modules, each encoding sub-module includes two KVFormer units, and the KVFormer unit includes a key-value pair matching attention mechanism sub-unit. The decoder includes multiple decoding sub-modules, each decoding sub-module includes a differential mixture of experts model unit, and the differential mixture of experts model unit includes multiple expert sub-units with the same structure, a subtraction operation sub-unit, and weight coefficients; the image segmentation model is trained using a corn sample image training set.

[0107] A segmentation module for inputting the target corn image into the trained image segmentation model for target segmentation; the targets include corn kernels, corn cobs, and corn husks.

[0108] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 12 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the target corn image. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements an image segmentation method based on deep learning and a differential expert model.

[0109] Those skilled in the art can understand that Figure 12 the structure shown in

[0110] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0111] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0112] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0113] The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0114] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0115] In this article, specific examples are used to elaborate on the principles and implementation manners of this application. The descriptions of the above embodiments are only used to help understand the method and its core idea of this application. At the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. An image segmentation method based on deep learning and differential expert models, characterized in that, Including: Obtain a target corn image; Construct an image segmentation model and train the image segmentation model using a corn sample image training set; the image segmentation model includes an encoder and a decoder. Among them, the encoder includes multiple encoding sub-modules, and each encoding sub-module includes two KVFormer units. The KVFormer unit includes a key-value pair matching attention mechanism sub-unit. The decoder includes multiple decoding sub-modules, and each decoding sub-module includes a differential mixture of experts model unit. The differential mixture of experts model unit includes multiple expert sub-units with the same structure and a subtraction operation sub-unit; Input the target corn image into the trained image segmentation model for target segmentation; the targets include corn kernels, corn cobs, and corn husks.

2. The image segmentation method based on deep learning and differential expert model according to claim 1, characterized in that, Inputting the target corn image into the trained image segmentation model for target segmentation includes: Performing downsampling operations on the target corn image through the encoder to obtain feature maps of different scales; Performing upsampling operations on the feature maps of different scales through the decoder to obtain high-resolution feature maps; Obtain the target segmentation result based on the high-resolution feature map.

3. The image segmentation method based on deep learning and differential expert model according to claim 2, wherein For the i-th encoding sub-module, the input is F i-1 , and the output is F i , F i-1 and F i are the feature maps output by the (i-1)-th and i-th encoding sub-modules respectively. When i equals 1, F0 is the target corn image.

4. The image segmentation method based on deep learning and differential expert model according to claim 3, characterized in that The two KVFormer units are the first KVFormer unit and the second KVFormer unit respectively; each encoding sub-module includes a first convolutional layer, a first BatchNorm layer, a first KVFormer unit, and a second KVFormer unit; The processing process of the i-th encoding sub-module specifically includes: Input the feature map F i into the first convolutional layer to extract detailed feature information, obtaining the feature map F i1 ; Input the feature map F i1 into the first BatchNorm layer for regularization processing to obtain the feature map F i2 ; Input the feature map F i2 into the first KVFormer unit to extract high-level semantic features, obtaining the feature map F i3 ; Input the feature map F i3 into the second KVFormer unit to extract high-level semantic features, obtaining the feature map F i+1 .

5. The image segmentation method based on deep learning and differential expert model according to claim 4, characterized in that, The KVFormer unit also includes a feed-forward neural network layer; The key-value pair matching attention mechanism sub-unit includes a multi-scale depth feature extraction layer, a feature splicing layer, a second convolutional layer, a third convolutional layer, a matrix dot product layer, and a second BatchNorm layer; The processing procedures of the first KVFormer unit and the second KVFormer unit are the same. For the first KVFormer unit, the feature map F i2 is input into the first KVFormer unit to extract high-level semantic features, resulting in the feature map F i3 , which specifically includes: Input the feature map F i2 into the second convolutional layer for feature extraction to obtain the V value; Input the feature map F i2 into the multi-scale depth feature extraction layer to extract high-level semantic features, obtaining a multi-scale depth feature map; Input the multi-scale depth feature map into the feature splicing layer for feature splicing to obtain a multi-scale fusion feature map; Input the multi-scale fusion feature map into the third convolutional layer for feature extraction to obtain attention weights; Input the V value and the attention weights into the matrix dot product layer for weighted fusion processing to obtain a weighted fusion feature map; Input the weighted fusion feature map into the second BatchNorm layer for regularization processing to obtain a regular fusion feature map; Input the regular fusion feature map into the feedforward neural network layer for feature extraction to obtain the feature map F i3 .

6. The image segmentation method based on deep learning and differential expert model according to claim 5, characterized in that, The number of decoding sub-modules is the same as the number of encoding sub-modules; For the decoding sub-module in the middle layer, the inputs are feature map F i-1 and feature map H j+1 , and the output is feature map H j ; for the decoding sub-module at the top layer, the input is feature map H j+1 , and the output is feature map H j ; for the decoding sub-module at the bottom layer, the inputs are feature map F i and feature map F i-1 , and the output is feature map H j ; where j is the serial number of the decoding sub-module; there is 1 decoding sub-module at the top layer, 1 decoding sub-module at the bottom layer, and multiple decoding sub-modules in the middle layer.

7. The image segmentation method based on deep learning and differential expert model according to claim 6, characterized in that, The top decoding sub-module includes an up-first sampling layer, a fourth convolutional layer, and a differential mixture of experts model unit; the middle decoding sub-module and the bottom decoding sub-module both include a second sampling layer, a matrix splicing layer, a fourth convolutional layer, and a differential mixture of experts model unit; The processing process of the bottom decoding sub-module specifically includes: Input the feature map F i into the second sampling layer for sampling to obtain the first feature map; Input the first feature map and the feature map F i-1 into the matrix splicing layer for feature splicing to obtain the second feature map; Input the second feature map into the fourth convolutional layer for feature extraction to obtain a third feature map; Input the third feature map into the differential mixture of experts model unit for feature extraction to obtain a feature map Hj; The processing process of the middle decoding sub-module specifically includes: Input the feature map H j+1 into the second sampling layer for sampling to obtain the fourth feature map; Input the fourth feature map and the feature map F i-1 into the matrix splicing operation layer for feature splicing to obtain the fifth feature map; Input the fifth feature map into the fourth convolutional layer for feature extraction to obtain a sixth feature map; Input the sixth feature map into the differential mixture of experts model unit for feature extraction to obtain a feature map Hj; The processing process of the top decoding sub-module specifically includes: Input the feature map H j+1 into the first sampling layer for sampling to obtain the seventh feature map; Input the seventh feature map into the fourth convolutional layer for feature extraction to obtain the eighth feature map; Input the eighth feature map into the mixture-of-experts model unit for feature extraction to obtain the feature map Hj.

8. The image segmentation method based on deep learning and differential expert model according to claim 7, characterized in that, There are three expert sub-units in the mixture-of-experts model unit; the processing processes of the mixture-of-experts model units in the top decoding sub-module, the middle decoding sub-module, and the bottom decoding sub-module are the same; For the mixture-of-experts model unit in the top decoding sub-module, the processing process is as follows: Input the eighth feature map into the first expert sub-unit for feature extraction to obtain the first expert feature map; Input the eighth feature map into the second expert sub-unit for feature extraction to obtain the second expert feature map; Perform a weighting process on the second expert feature map to obtain the second expert feature map with a preset multiple; Input the second expert feature map with the preset multiple and the first expert feature map into the subtraction operation sub-unit for subtraction operation to obtain the fused expert feature map; Input the fused expert feature map into the third expert subunit for feature extraction to obtain the feature map H j .

9. An image segmentation system based on deep learning and differential expert models, characterized in that, Including: An acquisition module for acquiring a target corn image; A construction and training module for constructing and training an image segmentation model, and training the image segmentation model with a corn sample image training set; the image segmentation model includes an encoder and a decoder, where the encoder includes multiple encoding sub-modules, each encoding sub-module includes two KVFormer units, the KVFormer unit includes a key-value pair matching attention mechanism sub-unit, the decoder includes multiple decoding sub-modules, each decoding sub-module includes a mixture-of-experts model unit, and the mixture-of-experts model unit includes multiple expert sub-units with the same structure and a subtraction operation sub-unit; A segmentation module for inputting the target corn image into the trained image segmentation model for target segmentation; the targets include corn kernels, corn cobs, and corn husks.

10. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the image segmentation method based on deep learning and the differential expert model according to any one of claims 1-8.

Images