A key point-based domain adaptive plant organ instance segmentation method

Abstract

The application relates to a key point-based domain adaptive plant organ instance segmentation method, and aims to solve the problem that an existing deep learning-based instance segmentation algorithm cannot match a target domain and a source domain due to sample diversity when facing different data sets.The process is as follows: 1, constructing a key point-based instance segmentation network model: 11, performing deep feature extraction on a plant image of the source domain to obtain feature maps of different scales; 12, predicting a group of key points for each element on the feature maps; 13, converting the key points into a boundary box to extract local features of a region of interest corresponding to the boundary box on the feature maps; 14, inputting the local features into a semantic segmentation network; 2, obtaining a trained instance segmentation network model; 3, inputting an image of the target domain into the trained instance segmentation network model to output a domain adaptive instance segmentation result.The application is used in the field of plant organ instance segmentation.

Description

Technical Field

[0001] This invention relates to a domain-adaptive plant organ instance segmentation method based on key points. Background Technology

[0002] In recent years, plant phenotyping techniques have become a research hotspot in the biosciences. Cultivating plants in biological laboratories and monitoring and measuring the effects of various growth factors, such as humidity, temperature, oxygen content, light, and nutrient conditions, on plant growth and development can provide valuable information for assessing plant growth status in phenotyping, thereby improving plant seed production and breeding processes. Previously, phenotyping research was conducted manually, which was time-consuming and labor-intensive. With the rapid development of information technology, automated methods based on computer vision are increasingly used for non-invasive plant phenotyping research, such as instance segmentation. This aims to separate different plant organs, providing a detailed description of plant growth status and trends. These methods can significantly reduce labor costs, but require more robust, accurate, and reliable analysis.

[0003] Because deep learning algorithms have achieved top-level results in various visual tasks (such as classification, detection, and segmentation), most automated phenotyping techniques are currently based on deep neural networks. However, deep learning is a data-driven task, requiring large-scale, wide-ranging data for training to ensure the model's inference performance and generalization ability. This results in existing deep learning-based instance segmentation algorithms performing significantly worse on different types of datasets than on one type of dataset. Summary of the Invention

[0004] The purpose of this invention is to address the problem of target domain and source domain mismatch caused by sample diversity in different datasets when existing deep learning-based instance segmentation algorithms face this issue, and to propose a domain-adaptive plant organ instance segmentation method based on key points.

[0005] The specific process of a domain-adaptive plant organ instance segmentation method based on key points is as follows:

[0006] Step 1: Construct a keypoint-based instance segmentation network model; the specific process is as follows:

[0007] Step 11: Use a feature extraction network to extract deep features from the plant images in the source region to obtain feature maps at different scales;

[0008] Step 12: Use the keypoint prediction module to predict a set of keypoints for each element on the feature map;

[0009] Step 13: Convert key points into bounding boxes, and extract local features of the region of interest corresponding to the bounding boxes on the feature map;

[0010] Step 14: Input local features into the semantic segmentation network;

[0011] Step 2: Calculate the loss function of the instance segmentation network model, train the instance segmentation network model, and obtain the trained instance segmentation network model;

[0012] Step 3: Input the target domain image into the trained instance segmentation network model and output the domain-adaptive instance segmentation result.

[0013] The beneficial effects of this invention are as follows:

[0014] This invention proposes a domain-adaptive plant organ instance segmentation method based on key points, which unifies the edge shape characteristics of the same organ in different plants, and can effectively improve the generalization ability of the instance segmentation network.

[0015] The purpose of this invention is to alleviate the problem of target domain and source domain mismatch caused by sample diversity in existing instance segmentation methods across different domains. It proposes a domain-adaptive plant organ instance segmentation method based on key points. By utilizing the characteristic that plant organs of the same type have similar edge shapes, the problem is transformed into a fixed shape detection problem, which effectively improves the generalization ability of the instance segmentation model.

[0016] Since this method uses the characteristics of the edge shape of the same organ in a plant as a prior to learn and train the instance segmentation network, it can significantly improve the instance segmentation effect of the trained model on the target domain without the need for bounding boxes and mask annotations of the target domain, or even without the need for training using image samples.

[0017] To verify the performance of the algorithm proposed in this invention, simulation experiments were conducted on the publicly available CVPPP and KOMATSUNA datasets. The experimental results verified the effectiveness of the key-point-based domain-adaptive plant organ instance segmentation method proposed in this invention. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the implementation process of the present invention;

[0019] Figure 2 This is a structural diagram of the feature extraction network;

[0020] Figure 3 This is the network structure diagram of the key point prediction module;

[0021] Figure 4 This is a structural diagram of a semantic segmentation network;

[0022] Figure 5 Image (a) shows the domain-adaptive instance segmentation result of the original Mask RCNN method;

[0023] Figure 5 Image (b) is a diagram showing the domain adaptive instance segmentation result of the method of the present invention;

[0024] Figure 6 This is a visualization comparison of the method of this invention with other methods (DAF, DDF, and TIA) on the unsupervised adaptive target detection task. Detailed Implementation

[0025] Specific Implementation Method 1: The specific process of this implementation method for domain-adaptive plant organ instance segmentation based on key points is as follows:

[0026] Step 1: Construct a keypoint-based instance segmentation network model; the specific process is as follows:

[0027] Step 11: Use a feature extraction network to extract deep features from the plant images in the source region to obtain feature maps at different scales;

[0028] Step 12: Use the keypoint prediction module to predict a set of keypoints for each element on the feature map;

[0029] Step 13: Convert key points into bounding boxes, and extract local features of the region of interest corresponding to the bounding boxes on the feature map;

[0030] Step 14: Input local features into the semantic segmentation network;

[0031] Step 2: Calculate the loss function of the instance segmentation network model, train the instance segmentation network model, and obtain the trained instance segmentation network model;

[0032] Step 3: Input the target domain image into the trained instance segmentation network model, and output the domain-adaptive instance segmentation result;

[0033] The source domain and the target domain refer to different data samples. The source domain refers to the labeled sample set used for training, while the target domain is an unlabeled dataset of another plant type to be tested.

[0034] Domain adaptation refers to transferring a model trained in the source domain to the target domain, outputting domain-adaptive instance segmentation results of the target domain image, including segmentation prediction results, classification prediction results, and object detection prediction results.

[0035] An instance refers to each individual organ of a plant, and also to each target in object detection and instance segmentation.

[0036] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that, in step 11, a feature extraction network is used to extract deep features from the plant image of the source region to obtain feature maps at different scales; the specific process is as follows:

[0037] The feature extraction network adopts a structure combining a deep residual network (ResNet-101) and a feature pyramid network (FPN), such as... Figure 2 As shown;

[0038] The deep residual network (ResNet-101) includes, in sequence, an input layer, a first-level convolutional layer, a second-level convolutional layer, a third-level convolutional layer, a fourth-level convolutional layer, and a fifth-level convolutional layer;

[0039] The first-level convolutional layer consists of a 7×7 standard convolutional layer, a batch normalization layer, and a ReLU activation function.

[0040] The second-level convolutional layer consists of three residual block structures, each of which includes a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution. The three residual block structures are cascaded (the output of the first residual block is used as the input of the second residual block, the output of the second residual block is used as the input of the third residual block, and the output of the third residual block is used as the output of the second-level convolutional layer).

[0041] The third-level convolutional layer includes four residual block structures in sequence. Each residual block structure includes a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution in sequence. The four residual block structures are cascaded.

[0042] The fourth-level convolutional layer consists of 23 residual block structures, each of which consists of a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution, and the 23 residual block structures are cascaded.

[0043] The fifth-level convolutional layer includes three residual block structures in sequence. Each residual block structure includes a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution in sequence. The three residual block structures are cascaded.

[0044] The Feature Pyramid Network (FPN) consists of a fifth layer, a fourth layer, a third layer, and a second layer, in sequence.

[0045] The feature extraction network is used to extract depth features from plant image samples in the source domain; the specific process is as follows:

[0046] A. Plant image samples from the source domain are input into the first-level convolutional layer through the input layer of the deep residual network. The output feature map of the first-level convolutional layer is input into the second-level convolutional layer. The output feature map of the second-level convolutional layer is input into the third-level convolutional layer. The output feature map of the third-level convolutional layer is input into the fourth-level convolutional layer. The output feature map of the fourth-level convolutional layer is input into the fifth-level convolutional layer. The fifth-level convolutional layer outputs a feature map.

[0047] B. The output feature map of the fifth level convolutional layer of the deep residual network is used as the output feature map M5 of the fifth layer of the feature pyramid network.

[0048] Perform a 3×3 standard convolution operation on the output feature map M5 of the fifth layer of the feature pyramid network, and output features of the corresponding scale at the end of this layer. Figure 1 ;

[0049] C. Perform a 1×1 standard convolution operation on the output feature map of the fourth level convolutional layer of the deep residual network to make the number of channels of different feature maps consistent, and obtain the convolutional feature map.

[0050] Upsample the output feature map M5 of the fifth layer of the feature pyramid network to make the feature map scale consistent, and obtain the sampled feature map.

[0051] The convolutional feature map and the sampled feature map are fused together by addition to obtain the fused feature map M4;

[0052] The fused feature map M4 is subjected to a standard 3×3 convolution operation, and features of the corresponding scale are output at the end of the layer. Figure 2 ;

[0053] D. Perform a 1×1 standard convolution operation on the output feature map of the third convolutional layer of the deep residual network to make the number of channels of different feature maps consistent, and obtain the convolutional feature map.

[0054] Upsample the output feature map M4 of the fourth layer of the feature pyramid network to make the feature map scale consistent, and obtain the sampled feature map.

[0055] The convolutional feature map and the sampled feature map are fused together by addition to obtain the fused feature map M3;

[0056] The fused feature map M3 is subjected to a standard 3×3 convolution operation, and features of the corresponding scale are output at the end of the layer. Figure 3 ;

[0057] E. Perform a 1×1 standard convolution operation on the output feature map of the second-level convolutional layer of the deep residual network to make the number of channels of different feature maps consistent, and obtain the convolutional feature map.

[0058] Upsample the output feature map M3 of the third layer of the feature pyramid network to make the feature map scale consistent, and obtain the sampled feature map.

[0059] The convolutional feature map and the sampled feature map are fused together by addition to obtain the fused feature map M2;

[0060] The fused feature map M2 is subjected to a standard 3×3 convolution operation, and features of the corresponding scale are output at the end of this layer. Figure 4 .

[0061] like Figure 2 In the diagram, the first layer represents the first convolutional layer in the ResNet-101 network structure, which contains a 7×7 standard convolutional layer, a batch normalization, and a ReLU activation function. The second layer (C2), the third layer (C3), the fourth layer (C4), and the fifth layer (C5) represent the second, third, fourth, and fifth convolutional layers in the ResNet-101 network structure, respectively. The second layer contains 3 residual block structures, the third layer contains 4 residual block structures, the fourth layer contains 23 residual block structures, and the fifth layer contains 3 residual block structures. Each residual block structure contains a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution concatenated in sequence.

[0062] In contrast to the bottom-up structure of ResNet-101, FPN is a top-down structure. It directly uses the high-level features of C5 as M5, which serves as the highest layer in FPN. Then, in each layer downwards, there is a lateral connection fusion with the corresponding layer of ResNet-101. The specific fusion process is as follows: at each layer, the feature maps in ResNet-101 are first subjected to a 1×1 standard convolution operation to make the number of channels in different feature maps consistent. The feature maps in FPN are then upsampled to make the feature map scale consistent. Then, the two feature maps are fused by addition. Finally, the fused feature map is subjected to a 3×3 standard convolution operation, and a feature map of the corresponding scale of the layer is output at the end of each layer.

[0063] The other steps and parameters are the same as in Specific Implementation Method 1.

[0064] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that, in step 12, the key point prediction module predicts a set of key points for each element on the feature map; the specific process is as follows:

[0065] The keypoint prediction module includes a 3×3 convolutional layer, a 3×3 deformable convolutional layer, and a 1×1 convolutional layer;

[0066] The features extracted by the feature map extraction network in step 1 Figure 1The 3×3 convolutional layer of the input keypoint prediction module is subjected to a nonlinear transformation, and the new feature map output is fed into two branches respectively.

[0067] The features extracted by the feature map extraction network in step 1 Figure 2 The 3×3 convolutional layer of the input keypoint prediction module is subjected to a nonlinear transformation, and the new feature map output is fed into two branches respectively.

[0068] The features extracted by the feature map extraction network in step 1 Figure 3 The 3×3 convolutional layer of the input keypoint prediction module is subjected to a nonlinear transformation, and the new feature map output is fed into two branches respectively.

[0069] The features extracted by the feature map extraction network in step 1 Figure 4 The 3×3 convolutional layer of the input keypoint prediction module is subjected to a nonlinear transformation, and the new feature map output is fed into two branches respectively.

[0070] One branch is responsible for outputting the classification prediction score of each element of the new feature map. The new feature map is first passed through a 3×3 deformable convolution to adaptively adjust the convolution sampling position, and then through a 1×1 convolution to reduce the dimensionality. The output result has 1 channel. Each element of the output result (a 1×1 vector) represents the classification probability of the pixel at the corresponding position in the new feature map (the pixel in the new feature map) as a foreground instance.

[0071] Another branch is responsible for outputting the coordinates of the 9 key points of the instance corresponding to each element on the new feature map. The new feature map is first reduced in dimensionality by a 3×3 deformable convolution and then by a 1×1 convolution. The number of output channels is 18. Each element of the output result (a vector of size 1×18) represents the horizontal and vertical coordinate values ​​of the 9 key points of the predicted instance corresponding to the pixel point (pixel point of the new feature map) in the source domain of the plant image in step one.

[0072] Here, 3×3 and 1×1 represent the size of the convolution kernel.

[0073] Features Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 For each element of each new feature map, the process of the two branches mentioned above is performed to obtain the key points of all elements;

[0074] Each element predicts a set of 9 keypoints.

[0075] Each element outputs the coordinates of 9 points. Each point has two coordinates, one on the horizontal axis and one on the vertical axis, so there are a total of 18 values. Therefore, the number of output channels is 18.

[0076] For each level of features at different scales output by the feature extraction network, the keypoint prediction module is input to output the predicted keypoint coordinates. The network structure of the keypoint prediction module is as follows: Figure 3 As shown.

[0077] Other steps and parameters are the same as in specific implementation method one or two.

[0078] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step 13, key points are converted into bounding boxes, and local features of the region of interest corresponding to the bounding boxes are extracted from the feature map; the specific process is as follows:

[0079] Step 31: Transform the key points into bounding boxes. The process is as follows:

[0080] Suppose the keypoint set predicted from the source domain plant image in step one is:

[0081]

[0082] Where n = 9 represents the number of key points (for each element), (x i ,y i Let be the coordinates of the i-th key point. For key point set;

[0083] Calculate x in the key point set i and y i The mean and variance, assuming x i The mean and variance are respectively and x v y i The mean and variance are respectively and y v Then the coordinates of the center point of the bounding box are The width w and height h of the bounding box are respectively:

[0084]

[0085]

[0086] Where λ w and λ h These are the learnable modulation parameters;

[0087] Therefore, the coordinates of the top left and bottom right corners of the bounding box are respectively... and , as the coordinate values ​​of the candidate box;

[0088] Step 32: Using the RoIAlign module and a bilinear interpolation algorithm, obtain the local features of the region of interest (ROI) of the bounding box in the plant image corresponding to the source domain from Step 1 on the feature map; the specific process is as follows:

[0089] (1) First, set the coordinate values ​​of the candidate boxes obtained in step 31. and Mapping from plant images in the source domain to feature maps (feature maps) Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 );

[0090] The candidate box coordinates obtained in step 31 and Divide all by 32 to obtain the coordinate values ​​on the feature map in step 11, and the coordinate values ​​on the feature map in step 11 are floating-point numbers;

[0091] (2) Set the coordinates of the candidate boxes and Divide all values ​​by 32 to obtain the coordinates on the feature map obtained in step 11. Use these coordinates as the coordinates of the top left and bottom right corners of the bounding box to obtain the feature map candidate region. Divide the feature map candidate region evenly into 7×7 blocks, a total of 49 small blocks.

[0092] In the feature map, the region defined by the floating-point coordinates obtained in the previous step is the candidate region of the feature map. The expected output local feature size is 7×7. Therefore, the candidate region is evenly divided into 49 small blocks of 7×7.

[0093] (3) Finally, interpolation is performed on each small block, and max pooling is used to obtain the result. The results of each small block are combined to obtain the local features; the specific process is as follows:

[0094] None of the 49 small blocks obtained in the previous step are complete regions;

[0095] The first small block is then evenly divided into 7×7 regions. The center point of each region is then taken, and the pixel value of each center point is obtained by bilinear interpolation. The maximum value of these 49 points (i.e., max pooling) is then calculated to obtain the value of the first large region of the first small block.

[0096] The second small block is then evenly divided into 7×7 regions. The center point of each region is taken, and the pixel value of each center point is obtained by bilinear interpolation. The maximum value of these 49 points is then calculated to obtain the value of the first large region of the second small block.

[0097] Continue until the value of the first large region of the 49th small block is obtained;

[0098] This yields a local feature of size 7×7.

[0099] Each keypoint of an element is processed once (1), (2), and (3) to obtain all local features. The keypoint of an element obtains a local feature of size 7×7.

[0100] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0101] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that, in step 14, local features are input into the semantic segmentation network; the specific process is as follows:

[0102] After obtaining the local features, these features are input into the semantic segmentation network to obtain the final instance segmentation result. The specific structure of the semantic segmentation network is as follows: Figure 4 As shown:

[0103] The semantic segmentation network consists of a 3×3 convolutional layer, a 3×3 convolutional layer, a 3×3 convolutional layer, a 3×3 convolutional layer, and an upsampling layer.

[0104] The local features are input into the semantic segmentation network, and the process is as follows:

[0105] The local features are sequentially processed through four cascaded 3×3 convolutions, and then upsampling is achieved through transposed convolution to expand the feature scale to the size of the source domain plant image in step one, resulting in pixel-level segmentation results.

[0106] In addition to outputting instance segmentation predictions for each target in the source domain, the semantic segmentation network can also obtain target classification predictions and target detection predictions.

[0107] An instance refers to each individual organ of a plant, and also to each target in object detection and instance segmentation.

[0108] All local features are executed in step 14.

[0109] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0110] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that, in step 2, the loss function of the instance segmentation network model is calculated, and the instance segmentation network model is trained to obtain a trained instance segmentation network model; the specific process is as follows:

[0111] The prediction results obtained by inputting the source domain image into the instance segmentation network are compared with the ground truth values ​​of the source domain image.

[0112] The overall loss function L of the instance segmentation network model is:

[0113] L=λ1Lbox +λ2L cls +λ3L mask

[0114] Where λ1, λ2, and λ3 are coefficients; L mask L is the loss function for semantic segmentation. cls For the loss function of classification, L box Let L be the target detection loss function; cls and L box Corresponding to the keypoint prediction module in the instance segmentation model, L mask This corresponds to the semantic segmentation network in the instance segmentation model.

[0115] The classification and object detection loss functions correspond to the keypoint prediction module, while the semantic segmentation loss function corresponds to the semantic segmentation network.

[0116] Object detection loss function L box Corresponding to the smoothed L1 norm loss, we have:

[0117]

[0118] Among them, t i The bounding box coordinates are the true values. The coordinates of the predicted bounding box are given by x, y, w, and h, which are the x and y coordinates of the center point of the bounding box, as well as the width and height, respectively. The loss is the smoothed L1 norm.

[0119] The smoothed L1 norm loss is defined as:

[0120]

[0121] Classification loss function L cls The focus loss is expressed as follows:

[0122]

[0123]

[0124] Where y is the truth value. For classification prediction scores, α and γ are adjustable factors, p t To The intermediate results after binary classification are used to calculate L. cls ;

[0125] The loss function L for semantic segmentation maskThe loss is calculated by comparing the difference between the ground truth mask and the predicted mask using the cross-entropy loss function.

[0126] The network is trained by backpropagation of the loss function, and the network parameters are calculated and updated until the loss function stops decreasing. This indicates that the instance segmentation network model has been trained successfully. The trained instance segmentation network model is obtained.

[0127] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0128] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One to Six in that λ1, λ2, and λ3 are all set to 1.0.

[0129] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0130] Specific Implementation Method Eight: This implementation method differs from one of Specific Implementation Methods One to Seven in that the adjustable factor α is set to 0.25 and γ is set to 2.0.

[0131] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0132] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that, in step 3, the target domain image is input into the trained instance segmentation model, and the domain-adaptive instance segmentation result is output; the specific process is as follows:

[0133] The target domain image is input into the trained model for testing, and the domain adaptive instance segmentation result of the target domain image is output.

[0134] The domain-adaptive instance segmentation results include classification prediction results and object detection results output by the keypoint prediction module, and segmentation prediction results output by the semantic segmentation network.

[0135] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0136] Example 1:

[0137] The data used in the experiment were the publicly available CVPPP dataset and KOMATSUNA dataset. The source domain was set as the CVPPP dataset and the plant type as Arabidopsis thaliana. The target domain was set as the KOMATSUNA dataset and the plant type as Komatsuna.

[0138] The model trained in the source domain is applied to the target domain to perform instance segmentation and target detection accuracy evaluation, in order to measure and compare the model's generalization ability.

[0139] Figure 5The results show a visual comparison of the domain-adaptive instance segmentation of the Mask RCNN algorithm and the method of this invention;

[0140] Figure 6 The results show a visual comparison between the method of the present invention and other methods on the unsupervised adaptive target detection task;

[0141] Table 1 compares the accuracy of the proposed method with that of the Mask RCNN algorithm in in-domain adaptive object detection and in-domain adaptive instance segmentation.

[0142] Table 2 compares the accuracy of the method of this invention with other unsupervised adaptive target detection algorithms.

[0143] The comparison results show that the method proposed in this invention can significantly improve the generalization ability of the instance segmentation model, and outperforms other methods in the unsupervised adaptive object detection task, demonstrating the superiority of the method proposed in this invention.

[0144] Table 1 Comparison of the accuracy of the method of the present invention and the original Mask R-CNN algorithm in instance segmentation and object detection tasks.

[0145]

[0146] Table 2 compares the accuracy of the domain-adaptive target detection results of the method of the present invention with those of other unsupervised domain-adaptive target detection methods (DAF, SCL, SWDA, DDF, and TIA).

[0147]

[0148] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A domain-adaptive plant organ instance segmentation method based on key points, characterized in that: The specific process of the method is as follows: Step 1: Construct a keypoint-based instance segmentation network model; the specific process is as follows: Step 11: Use a feature extraction network to extract deep features from the plant images in the source region to obtain feature maps at different scales; Step 12: Use the keypoint prediction module to predict a set of keypoints for each element on the feature map; Step 13: Convert key points into bounding boxes, and extract local features of the region of interest corresponding to the bounding boxes from the feature map; the specific process is as follows: Step 31: Transform the key points into bounding boxes. The process is as follows: Suppose the keypoint set predicted from the source domain plant image in step one is: Where n = 9 represents the number of key points, (x i ,y i Let be the coordinates of the i-th key point. For key point set; Calculate x in the key point set i and y i The mean and variance, assuming x i The mean and variance are respectively and x v y i The mean and variance are respectively and y v Then the coordinates of the center point of the bounding box are The width w and height h of the bounding box are respectively: Where λ w and λ h These are the learnable modulation parameters; Therefore, the coordinates of the top left and bottom right corners of the bounding box are respectively... and As the coordinate values ​​of the candidate box; Step 32: Using the RoIAlign module and a bilinear interpolation algorithm, obtain the local features of the region of interest (ROI) of the bounding box in the plant image corresponding to the source domain from Step 1 on the feature map; the specific process is as follows: First, use the candidate box coordinates obtained in step 31. and Mapping from plant images in the source domain to feature maps; (1) The candidate box coordinates obtained in step 31 and Divide all by 32 to obtain the coordinate values ​​on the feature map in step 11, and the coordinate values ​​on the feature map in step 11 are floating-point numbers; (2) Set the coordinates of the candidate boxes and Divide all values ​​by 32 to obtain the coordinates on the feature map obtained in step 11. Use these coordinates as the coordinates of the top left and bottom right corners of the bounding box to obtain the feature map candidate region. Divide the feature map candidate region evenly into 7×7 blocks, a total of 49 small blocks. (3) Finally, interpolation is performed on each small block, and max pooling is used to obtain the result. The results of each small block are combined to obtain the local features; the specific process is as follows: The first small block is then divided into 7×7 regions. The center point of each region is then taken, and the pixel value of each center point is obtained by bilinear interpolation. The maximum value of these 49 points is then calculated to obtain the value of the first large region of the first small block. The second small block is then evenly divided into 7×7 regions. The center point of each region is taken, and the pixel value of each center point is obtained by bilinear interpolation. The maximum value of these 49 points is then calculated to obtain the value of the first large region of the second small block. Continue until the value of the first large region of the 49th small block is obtained; This yields a local feature of size 7×7; Step 14: Input local features into the semantic segmentation network; Step 2: Calculate the loss function of the instance segmentation network model, train the instance segmentation network model, and obtain the trained instance segmentation network model; Step 3: Input the target domain image into the trained instance segmentation network model and output the domain-adaptive instance segmentation result.

2. The domain-adaptive plant organ instance segmentation method based on key points according to claim 1, characterized in that: In step 11, a feature extraction network is used to extract deep features from the plant image in the source region to obtain feature maps at different scales; the specific process is as follows: The feature extraction network adopts a structure that combines a deep residual network with a feature pyramid. The deep residual network comprises, in sequence, an input layer, a first-level convolutional layer, a second-level convolutional layer, a third-level convolutional layer, a fourth-level convolutional layer, and a fifth-level convolutional layer; The first-level convolutional layer consists of a 7×7 convolutional layer, a batch normalization, and a ReLU activation function. The second-level convolutional layer consists of three residual block structures, each of which consists of a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution, and the three residual block structures are cascaded. The third-level convolutional layer includes four residual block structures in sequence. Each residual block structure includes a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution in sequence. The four residual block structures are cascaded. The fourth-level convolutional layer consists of 23 residual block structures, each of which consists of a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution, and the 23 residual block structures are cascaded. The fifth-level convolutional layer includes three residual block structures in sequence. Each residual block structure includes a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution in sequence. The three residual block structures are cascaded. The feature pyramid consists of a fifth layer, a fourth layer, a third layer, and a second layer, in sequence. The feature extraction network is used to extract depth features from plant image samples in the source region; the specific process is as follows: A. Plant image samples from the source domain are input into the first-level convolutional layer through the input layer of the deep residual network. The output feature map of the first-level convolutional layer is input into the second-level convolutional layer. The output feature map of the second-level convolutional layer is input into the third-level convolutional layer. The output feature map of the third-level convolutional layer is input into the fourth-level convolutional layer. The output feature map of the fourth-level convolutional layer is input into the fifth-level convolutional layer. The fifth-level convolutional layer outputs a feature map. B. The output feature map of the fifth level convolutional layer of the deep residual network is used as the output feature map M5 of the fifth layer of the feature pyramid network. Perform a 3×3 convolution operation on the output feature map M5 of the fifth layer of the feature pyramid network to output the feature map 1 of the corresponding scale; C. Perform a 1×1 convolution operation on the output feature map of the fourth level convolutional layer of the deep residual network to obtain the convolutional feature map; Upsample the output feature map M5 of the fifth layer of the feature pyramid network to obtain the sampled feature map; The convolutional feature map and the sampled feature map are fused together by addition to obtain the fused feature map M4; Perform a 3×3 convolution operation on the fused feature map M4 to output the feature map 2 of the corresponding scale. D. Perform a 1×1 convolution operation on the output feature map of the third-level convolutional layer of the deep residual network to obtain the convolutional feature map. The output feature map M4 of the fourth layer of the feature pyramid network is upsampled to obtain the sampled feature map. The convolutional feature map and the sampled feature map are fused together by addition to obtain the fused feature map M3; Perform a 3×3 convolution operation on the fused feature map M3 to output the feature map 3 of the corresponding scale. E. Perform a 1×1 convolution operation on the output feature map of the second-level convolutional layer of the deep residual network to obtain the convolutional feature map. The output feature map M3 of the third layer of the feature pyramid network is upsampled to obtain the sampled feature map; The convolutional feature map and the sampled feature map are fused together by addition to obtain the fused feature map M2; Perform a 3×3 convolution operation on the fused feature map M2 to output the feature map 4 of the corresponding scale.

3. The domain-adaptive plant organ instance segmentation method based on key points according to claim 2, characterized in that: In step 12, the key point prediction module is used to predict a set of key points for each element on the feature map. The specific process is as follows: The keypoint prediction module includes a 3×3 convolutional layer, a 3×3 deformable convolutional layer, and a 1×1 convolutional layer; The feature map 1 extracted by the feature map extraction network in step 1 is input into the 3×3 convolutional layer of the key point prediction module for nonlinear transformation, and the new feature map output is fed into two branches respectively. The feature map 2 extracted by the feature map extraction network in step 1 is input into the 3×3 convolutional layer of the key point prediction module for nonlinear transformation, and the output new feature map is fed into the two branches respectively. The feature map 3 extracted by the feature map extraction network in step 1 is input into the 3×3 convolutional layer of the key point prediction module for nonlinear transformation, and the new output feature map is fed into two branches respectively. The feature map 4 extracted by the feature map extraction network in step 1 is input into the 3×3 convolutional layer of the key point prediction module for nonlinear transformation, and the new output feature map is fed into two branches respectively. One of the new feature maps in one branch is first subjected to a 3×3 deformable convolution to adaptively adjust the convolution sampling position, and then subjected to a 1×1 convolution for dimensionality reduction. The output has 1 channel, and each element of the output represents the probability of classifying the pixel at the corresponding position in the new feature map as a foreground instance. Another branch of the new feature map is first reduced in dimensionality by a 3×3 deformable convolution and then by a 1×1 convolution. The number of output channels is 18. Each element of the output result represents the horizontal and vertical coordinate values ​​of the plant image in the source domain in step one, corresponding to the nine key points of the pixel prediction instance at the corresponding position in the new feature map. Here, 3×3 and 1×1 represent the size of the convolution kernel.

4. The domain-adaptive plant organ instance segmentation method based on key points according to claim 3, characterized in that: In step 14, local features are input into the semantic segmentation network; the specific process is as follows: The semantic segmentation network consists of a 3×3 convolutional layer, a 3×3 convolutional layer, a 3×3 convolutional layer, a 3×3 convolutional layer, and an upsampling layer. The local features are input into the semantic segmentation network, and the process is as follows: The local features are sequentially processed through four cascaded 3×3 convolutions, and then upsampling is achieved through transposed convolution to expand the feature scale to the size of the source domain plant image in step one, resulting in pixel-level segmentation results. In addition to outputting instance segmentation predictions for each target in the source domain, the semantic segmentation network can also obtain target classification predictions and target detection predictions.

5. The domain-adaptive plant organ instance segmentation method based on key points according to claim 4, characterized in that: In step 2, the loss function of the instance segmentation network model is calculated, and the network is trained to obtain a trained instance segmentation network model. The specific process is as follows: The overall loss function L of the instance segmentation network model is: L=λ1L box +λ2L cls +λ3L mask Where λ1, λ2, and λ3 are coefficients; L mask L is the loss function for semantic segmentation. cls For the loss function of classification, L box Let L be the target detection loss function; cls and L box Corresponding to the keypoint prediction module in the instance segmentation model, L mask This corresponds to the semantic segmentation network in the instance segmentation model; Object detection loss function L box Corresponding to the smoothed L1 norm loss, we have: Among them, t i The bounding box coordinates are the true values. The coordinates of the predicted bounding box are given by x, y, w, and h, which are the x and y coordinates of the center point of the bounding box, as well as the width and height, respectively. The loss is the smoothed L1 norm. The smoothed L1 norm loss is defined as: Classification loss function L cls Using focus loss, the expression is: Where y is the truth value. For classification prediction scores, α and γ are adjustable factors, p t To The intermediate results after binary classification are used to calculate L. cls ; The loss function L for semantic segmentation mask The loss is calculated by comparing the difference between the ground truth mask and the predicted mask using the cross-entropy loss function. The network is trained by backpropagation of the loss function, and the network parameters are calculated and updated until the loss function stops decreasing. This indicates that the instance segmentation network model has been trained successfully. The trained instance segmentation network model is obtained.

6. The domain-adaptive plant organ instance segmentation method based on key points according to claim 5, characterized in that: λ1, λ2 and λ3 are all set to 1.

0.

7. The domain-adaptive plant organ instance segmentation method based on key points according to claim 6, characterized in that: The adjustable factor α is set to 0.25, and γ is set to 2.

0.

8. The domain-adaptive plant organ instance segmentation method based on key points according to claim 7, characterized in that: In step 3, the target domain image is input into the trained instance segmentation model, and the domain adaptive instance segmentation result is output. The specific process is as follows: The target domain image is input into the trained model for testing, and the domain adaptive instance segmentation result of the target domain image is output. The domain-adaptive instance segmentation results include classification prediction results and object detection results output by the keypoint prediction module, and segmentation prediction results output by the semantic segmentation network.

Images