Method for identifying leaf straightness and maximum leaf spread of cabbage plug seedlings

Abstract

The application provides a cabbage plug seedling machine suitability identification method for leaf erectness and maximum leaf spread, which comprises the following steps: calling a Swin Transformer model based on RFP structure to perform feature extraction on binocular images of collected cabbage plug seedlings, to obtain feature maps of the binocular images; performing regression prediction on the feature maps of the binocular images, to obtain stem bounding boxes and leaf bounding boxes of the binocular images; performing key point matching on the stem bounding boxes and the leaf bounding boxes of the binocular images, and determining three-dimensional geometric position coordinates of the cabbage plug seedlings according to image coordinates of the matched key points, so as to determine the leaf erectness and the maximum leaf spread of the cabbage plug seedlings. Through the application, the defects that the existing seedling detection method mainly focuses on seedling condition detection and recognition in the plug cultivation process, and a plug seedling machine suitability identification standard has not been formed, the detection accuracy is limited, and the method is difficult to be applied to actual production are solved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method for identifying the leaf uprightness and maximum leaf spread of organically grown cabbage plug seedlings. Background Technology

[0002] The plant structure of cabbage has a significant impact on the quality of mechanized transplanting. Different varieties and cultivation strategies can lead to different plant structures. Cabbage seedlings with poor upright growth and large leaf spread are prone to getting stuck in the transplanting cups during transplanting, resulting in seedling damage or problems such as missing seedlings and replanting. Moreover, upright growth and leaf spread directly affect the plant's photosynthetic utilization rate. Plants with good upright growth can make full use of light and carry out better metabolism. Plants with well-clustered leaves grow more stably after transplanting, which is conducive to head formation, and the leaves are less damaged during transplanting, making them less susceptible to infectious diseases. Cabbage is a long-day plant, and its yield is also closely related to its plant shape. A good plant shape can better utilize space and sunlight, producing more dry matter. However, currently, there is a lack of standards and identification methods for judging upright growth and leaf spread in mechanized cabbage tray seedling cultivation, making it difficult for cabbage seedlings to adapt to mechanized transplanting. Problems such as seedling cup blockage, seedling damage, and seedling tilting during transplanting make it difficult to improve the efficiency of mechanized transplanting.

[0003] Furthermore, manually measuring the absolute and relative height of seedlings is labor-intensive, and the judgment criteria are highly subjective, making it unsuitable for large-scale plug seedling production. Existing seedling detection methods mainly focus on seedling condition detection and identification during plug cultivation, rarely considering the characteristics of mechanized transplanting agronomy. They have not developed a standard for judging the suitability of plug seedlings for mechanization, resulting in limited accuracy and making them difficult to apply in actual production. Summary of the Invention

[0004] This invention provides a method for identifying the leaf uprightness and maximum leaf expansion of organically adapted cabbage plug seedlings, which addresses the shortcomings of existing seedling detection methods that mainly focus on seedling condition detection during plug cultivation, lack a standard for judging the adaptability of plug seedlings to mechanization, have limited detection accuracy, and are difficult to apply in actual production.

[0005] This invention provides a method for identifying the leaf uprightness and maximum leaf spread of organically grown cabbage plug seedlings, characterized by the following steps:

[0006] Acquire binocular images of collected cabbage seedlings in plug trays;

[0007] The Swin Transformer model based on the RFP structure is invoked to extract features from the stereo image, thereby obtaining the feature map of the stereo image.

[0008] Regression prediction is performed on the feature map of the stereo image to obtain the stem bounding box and leaf bounding box of the stereo image;

[0009] The stem and leaf bounding boxes of the binocular images are matched for key points, and the three-dimensional geometric position coordinates of the cabbage seedlings in the plug trays are determined based on the image coordinates of the matched key points.

[0010] The leaf uprightness and maximum leaf spread of cabbage plug seedlings were determined based on the three-dimensional geometric position coordinates.

[0011] In some embodiments, the step of invoking the Swin Transformer model based on the RFP structure to extract features from the stereo image and obtain the feature map of the stereo image includes:

[0012] The image to be processed is divided into multiple patches, and the patches are input into the first composite stage module of the first Swin Transformer model with a 3-layer RFP structure. The first composite stage module extracts features from the patches to obtain the first feature map corresponding to the patches. The image to be processed is the left or right image in the stereo image.

[0013] The patch is input into the second composite stage module of the third layer based on the RFP structure in the second Swing Transformer model;

[0014] The first feature map is input into the second composite stage module with a 3-layer RFP structure in the second Swin Transformer model. The second composite stage module extracts features from the patch and the first feature map to obtain the second feature map.

[0015] The second feature map is used as the feature map of the stereo image.

[0016] In some embodiments, the step of extracting features from the patch using the first composite stage module to obtain a first feature map corresponding to the patch includes:

[0017] The patch is embedded to obtain the corresponding embedded patch sequence, and the embedded patch sequence is reshaped into a two-dimensional feature map through the W-MSA module in the first composite stage module.

[0018] The two-dimensional feature map is divided by non-overlapping detection windows, and the two-dimensional feature map divided within the detection window is grouped and self-attention is calculated by the W-MSA module and SW-MSA module in the first composite stage module to obtain the window representation corresponding to the detection window.

[0019] Global downsampling self-attention calculation is performed on the window representation corresponding to each detection window, and the calculated result is input into the Token fusion module for feature fusion to obtain the fused representation.

[0020] The fused representation is input into the MLP module to obtain the high-level feature map of the patch;

[0021] The high-level feature map is input into the feature pyramid network included in the first composite stage module for feature extraction to obtain the first feature map corresponding to the patch.

[0022] In some embodiments, the step of inputting the high-level feature map into the feature pyramid network included in the first composite stage module for feature extraction to obtain the first feature map corresponding to the patch includes:

[0023] The high-level feature map is subjected to edge adaptive upsampling to obtain multiple upsampled feature maps, and the upsampled feature maps are then fused together.

[0024] The fused feature map is input into the feature pyramid network for multiple iterative recursive calculations to obtain the first feature map corresponding to the patch.

[0025] In some embodiments, the step of performing edge adaptive upsampling on the high-level feature map to obtain multiple upsampled feature maps includes:

[0026] Iterate through each pixel in the high-level feature map and call the Sober operator to calculate the edge gradient value of each pixel along the four directions of up, down, left and right, and determine the maximum edge gradient value of each pixel;

[0027] The pixels whose maximum edge gradient value is less than or equal to the preset edge gradient threshold are determined as the first interpolation pixels in the flat region. The first interpolation pixels are then upsampled in both the horizontal and vertical directions using a bilinear interpolation algorithm to obtain the corresponding upsampled feature map.

[0028] The pixel with the maximum edge gradient value greater than the preset edge gradient threshold is determined as the second interpolation pixel in the edge region;

[0029] The direction in which the maximum edge gradient value is obtained in the second pixel to be interpolated is taken as the gradient direction, and the vertical direction of the gradient direction is taken as the edge line direction.

[0030] For the second pixel to be interpolated, which is located in a direction other than the gradient direction and the edge line direction, a bilinear interpolation algorithm is used for upsampling to obtain the corresponding upsampled feature map.

[0031] For the second pixel to be interpolated located in the gradient direction, an upsampling is performed using a gradient-weighted Lanczos window function to obtain the corresponding upsampled feature map;

[0032] For the second pixel to be interpolated located in the direction of the edge line, a higher-order Lanczos window function is used for upsampling to obtain the corresponding upsampled feature map.

[0033] In some embodiments, the step of performing regression prediction on the feature maps of the stereo image to obtain the stem bounding box and leaf bounding box of the stereo image includes:

[0034] A convolutional neural network is invoked to predict the feature map of the binocular image to obtain the preliminary position and preliminary offset of the representative points of the stem and leaf parts of the cabbage seedlings in the plug tray. The preliminary position and preliminary offset of the representative points of the stem and leaf parts are then corrected by a small regression network to obtain the corrected offset of the representative points of the stem and leaf parts.

[0035] The initial position of the representative point of the stem and leaf is corrected by the corrected offset to obtain the key points of the stem and leaf of the cabbage seedling in the plug tray;

[0036] The smallest bounding rectangle of the key points is determined as the stem bounding box and leaf bounding box of the binocular image.

[0037] In some embodiments, keypoint matching of the stem bounding box and leaf bounding box of the binocular image includes:

[0038] For the left and right images in the binocular images, candidate points are extracted from the stem bounding box and the leaf bounding box, respectively;

[0039] Based on binocular visual information, a coarse match is performed between candidate points in the left image and candidate points in the right image to obtain matching points;

[0040] Determine the Euclidean distance between the matching points of the left image and the matching points of the right image;

[0041] Matching points whose Euclidean distance is less than a distance threshold are identified as key points in the stereo image.

[0042] In some embodiments, determining the three-dimensional geometric position coordinates of the cabbage seedlings in the plug tray based on the image coordinates of the matched key points includes:

[0043] The disparity value of the key points in the binocular image is determined based on the image coordinates of the key points obtained through matching;

[0044] The three-dimensional geometric position coordinates of the cabbage seedlings in the plug trays are determined based on the image coordinates and the parallax value.

[0045] In some embodiments, after acquiring the binocular images of the collected cabbage seedlings in plug trays, the method further includes:

[0046] The target detection model is invoked to detect the stereo image, and the stem bounding box and leaf bounding box of the stereo image are obtained;

[0047] The leaf uprightness and maximum leaf spread of cabbage plug seedlings are determined based on the stem bounding box and leaf bounding box of the binocular image.

[0048] The target detection model includes a feature extraction network and a small multi-task network. It is trained using binocular image samples of cabbage seedlings in plug trays, each image carrying a true bounding box of the stem and leaves. The training process of the target detection model includes:

[0049] The binocular image samples of the cabbage seedlings in the plug trays are input into the feature extraction network for feature extraction processing to obtain the output feature map of the binocular image samples;

[0050] The output feature map is input into the small multi-task network for regression prediction processing to obtain the predicted stem and leaf bounding box and the classification information of the stem and leaf of cabbage plug seedlings.

[0051] Based on the actual stem and leaf bounding boxes and the predicted stem and leaf bounding boxes, a bounding box regression loss is constructed.

[0052] Extract real key points within the true stem and leaf bounding box in the binocular image samples, and construct stem and leaf information classification loss based on the real key points and the stem and leaf classification information of the cabbage plug seedlings.

[0053] The final loss of the object detection model is constructed based on the bounding box regression loss and the stem-and-leaf information classification loss, and the final loss is used for backpropagation in the feature extraction network and the small multi-task network to update the parameters of the object detection model.

[0054] The present invention also provides a device for identifying the leaf uprightness and maximum leaf spread of organically grown cabbage plug seedlings, the device comprising the following modules:

[0055] The acquisition module is used to acquire binocular images of the collected cabbage seedlings in plug trays;

[0056] The extraction module is used to call the Swin Transformer model based on the RFP structure to extract features from the stereo image and obtain the feature map of the stereo image.

[0057] The prediction module is used to perform regression prediction on the feature map of the stereo image to obtain the stem bounding box and leaf bounding box of the stereo image;

[0058] The matching module is used to perform key point matching on the stem bounding box and leaf bounding box of the binocular image, and determine the three-dimensional geometric position coordinates of the cabbage plug seedlings based on the image coordinates of the matched key points.

[0059] The determination module is used to determine the leaf uprightness and maximum leaf spread of cabbage plug seedlings based on the three-dimensional geometric position coordinates.

[0060] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for identifying leaf uprightness and maximum leaf spread of organic cabbage plug seedlings as described above.

[0061] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying leaf uprightness and maximum leaf spread of organic cabbage plug seedlings as described above.

[0062] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for identifying leaf uprightness and maximum leaf spread of organic cabbage plug seedlings as described above.

[0063] This invention provides a method for identifying leaf uprightness and maximum leaf spread in cabbage plug seedlings suitable for mechanization. Based on binocular images of cabbage plug seedlings, a Swing Transformer model based on an RFP structure is used to extract feature maps and perform regression prediction to obtain the stem and leaf bounding boxes of the binocular images. Next, keypoint matching is performed on the bounding boxes to determine the three-dimensional geometric coordinates of the cabbage plug seedlings based on the image coordinates of the matched keypoints. Finally, the leaf uprightness and maximum leaf spread of the cabbage plug seedlings are calculated. Therefore, on the one hand, the regression prediction of the model improves the accuracy of target detection of the stem and leaf parts of the cabbage plug seedlings; on the other hand, by simply capturing images of the plug seedlings, leaf uprightness and maximum leaf spread can be detected and determined in real time, forming a criterion for the mechanization of plug seedlings, which can be applied to the actual production of mechanized transplanting agriculture. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating the method for identifying leaf uprightness and maximum leaf spread in organically grown cabbage plug seedlings provided by the present invention.

[0066] Figure 2 This is a schematic diagram of the backbone network of the Swing Transformer model provided by the present invention.

[0067] Figure 3 This is a schematic diagram of feature extraction using the stage model provided by the present invention.

[0068] Figure 4 This is a schematic diagram of binocular camera imaging provided by the present invention.

[0069] Figure 5 This is a schematic diagram of determining the blade uprightness and maximum blade span provided by the present invention.

[0070] Figure 6 This is a schematic diagram of the structure of the organicated cabbage plug seedling leaf uprightness and maximum leaf spread recognition device provided by the present invention.

[0071] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0073] The following description, in conjunction with the accompanying drawings, describes the method for identifying the leaf uprightness and maximum leaf spread of organically grown cabbage plug seedlings according to the present invention. Figure 1 This is a flowchart illustrating the method for identifying leaf uprightness and maximum leaf spread in organically grown cabbage plug seedlings provided by this invention. Figure 1 As shown, the method includes the following steps 101 to 105.

[0074] Step 101: Obtain binocular images of the collected cabbage seedlings in plug trays.

[0075] In a real-world production environment involving the mechanized transplanting of cabbage plug seedlings, a portion of the cabbage plug seedlings are randomly selected from the same batch of cultivated seedlings. Then, this embodiment of the invention calculates the leaf uprightness and maximum leaf spread of the cabbage plug seedlings by real-time acquisition of binocular images.

[0076] Cabbage seedlings in plug trays were gripped in a light source box using a cabbage transplanting device. Once the seedlings were relatively stable (with the substrate parallel to the ground), binocular image data was acquired using a binocular camera and vernier calipers. The binocular camera, calibrated using the Zhang calibration method, was placed horizontally 50cm away from the seedlings, with the seedling gripped by the transplanting device as the image center. The images were then captured using a ZED 2i binocular camera connected to a computer and stored in JPG format. A typical binocular image includes a left image (the target image) and a right image (used for matching).

[0077] To alleviate the hardware burden on subsequent model processing, the acquired binocular images were proportionally scaled to 550 pixels × 550 pixels three-channel images using bilinear interpolation, and image augmentation was employed for data expansion. Furthermore, considering the irregular growth characteristics of plants such as cabbage and the varying intensity of sunlight in greenhouses, mirror rotation and adjustments to image brightness and saturation were used to adjust the binocular images, facilitating subsequent model processing.

[0078] Step 102: Call the Swin Transformer model based on RFP structure to extract features from the stereo image and obtain the feature map of the stereo image.

[0079] Traditional visual Transformer models have achieved good results in areas such as defect detection and face recognition. However, when applied to actual vegetable seedling production, they suffer from incompatibility with application scenarios due to complex backgrounds, uneven lighting, and small data scales. Therefore, this embodiment of the invention uses a Swin Transformer model based on a Feature Pyramid Network (RFP) structure to extract features from binocular images, obtaining feature maps for the binocular images. For example... Figure 2As shown, in this embodiment of the invention, the backbone network of the Swin Transformer model based on the RFP structure is composed of two Swin Transformer models (i.e., the first Swin Transformer model and the second Swin Transformer model). Each Swin Transformer model includes three top-down stage modules, corresponding to the three stages of feature extraction. These three stages extract features of different scales from the input image. Each stage module includes a linear embedding layer and a Swin Transformer block. The Swin Transformer block contains a W-MSA module, a SW-MSA module, and a multilayer perceptron (MLP). A token fusion (TM) module is also inserted between the SW-MSA module and the MLP. The reason why the Swin Transformer model based on the RFP structure is used here is because a feature pyramid network (RFP) is designed in the stage module. Therefore, the stage module is also called a composite stage module. The feature pyramid network included in the composite stage module can further extract features from the feature map processed by the stage module. Using the SwinTransformer model based on the RFP structure to extract feature maps from stereo images not only reduces the computational cost of feature maps and the model, making the network more efficient and faster, but also effectively improves the accuracy of feature extraction. The process of extracting feature maps is described in detail below.

[0080] First, the image to be processed is divided into multiple patches. The image to be processed here is either the left or right image in the stereo image. Both images need to be extracted separately. The following explanation uses the left image as an example.

[0081] like Figure 2 As shown, after the image to be processed (left image) is input into the Swin Transformer model based on the RFP structure, the input image to be processed (size H*W) is divided into non-overlapping patches by the Patch Partition module in the first Swin Transformer model, with width, height and channel size of 4*4*3 respectively.

[0082] Next, the patch is input into the first composite stage module of the three layers based on the RFP structure in the first Swin Transformer model. The first composite stage module extracts features from the patch to obtain the first feature map corresponding to the patch.

[0083] Specifically, such as Figure 3 As shown, after the patch is input into the first composite stage module, it is embedded through the Linear Embedding (LN) layer to obtain the corresponding embedded patch sequence. The embedding process directly converts the patch into a single-channel embedding feature in one go, thus forming the embedded patch sequence, i.e., the embedded patch. Here, the single-channel embedding features are all one-dimensional feature vectors.

[0084] After the embedding process is complete, the one-dimensional embedding patch sequence is then input into the W-MSA module and the SW-MSA module (corresponding to...). Figure 3 The improved SW-MSA is described. The W-MSA module in the first composite stage module reshapes the embedded patch sequence into a two-dimensional feature map of size H*W to facilitate feature extraction. The W-MSA and SW-MSA modules employ a window processing mechanism: first, the two-dimensional feature map is divided into non-overlapping detection windows; then, the W-MSA and SW-MSA modules in the first composite stage module perform grouped self-attention calculations on the divided two-dimensional feature maps within the detection windows to obtain the window representation corresponding to the detection window.

[0085] Within the detection window, the corresponding 2D feature map is further divided into sub-windows, each denoted as M*M. Then, grouped self-attention calculation is performed on the pixels of the 2D feature map within each sub-window to obtain the window representation corresponding to the detection window. This window representation only represents the local features within the detection window. The above process is repeated until the self-attention calculation for each sub-window is completed. Finally, the window representation of each sub-window is downsampled to obtain the feature information of the detection window.

[0086] like Figure 3 As shown, in the SW-MSA module, global downsampling self-attention is calculated for the window representation corresponding to each detection window, and the result is input into the Token merging module for feature fusion to obtain the fused representation. Finally, the fused representation is input into the MLP module to obtain the high-level feature map of the patch.

[0087] Here is as Figure 3As shown, the fused representation output by the Token fusion module is input into the Linear Embedding (LN) layer of the MLP module, and then input into the MLP layer for mapping processing, finally obtaining the high-level feature map of the patch. This high-level feature map is the output of the stage module in the first composite stage module, which contains the high-dimensional feature information of the image to be processed.

[0088] Considering the slender stems of cabbage seedlings in plug trays and the overlapping leaves of mature seedlings, conventional feature pyramid structures struggle to guarantee detection performance in complex scenarios. Therefore, this embodiment of the invention also incorporates feature pyramid networks (FPN) into the backbone network of the Swin Transformer model, enhancing the network's feature representation capabilities by introducing a recursive feature pyramid (RFP) structure.

[0089] The feature pyramid network is used to process the high-level feature maps output by the stage modules in the first composite stage module. Therefore, after the high-level feature maps of the patches are output in the MLP module, they are then input into the feature pyramid network included in the first composite stage module for feature extraction to obtain the first feature map corresponding to the patch. Since the backbone network of the SwinTransformer model is top-down, existing methods involve upsampling the high-level feature maps and then fusing them. Upsampling methods often use bilinear interpolation algorithms, which can only uniformly amplify the entire image, easily causing loss of high-frequency components and blurring of seedling edges to some extent. Therefore, this embodiment of the invention proposes an edge-adaptive upsampling method, which first performs edge-adaptive upsampling on the high-level feature maps to obtain multiple upsampled feature maps, and then fuses these upsampled feature maps, as described in detail below.

[0090] For edge adaptive upsampling, first, each pixel in the high-level feature map is traversed, and the Sober operator is called to calculate the edge gradient value of each pixel along the four directions of top, bottom, left, and right (i.e., 0 degrees, 45 degrees, 90 degrees, and 135 degrees), denoted as . , , , And determine the maximum edge gradient value of each pixel, denoted as . .

[0091] Next, an edge gradient threshold T can be preset. Pixels whose maximum edge gradient value is less than or equal to the preset edge gradient threshold T are determined as the first pixel to be interpolated in the flat region. Then, bilinear interpolation algorithm is used to upsample the first pixel to be interpolated in both the horizontal and vertical directions to obtain the corresponding upsampled feature map.

[0092] Here, when the maximum edge gradient value of a pixel is less than or equal to a preset edge gradient threshold T, that is... If the pixel is in a flat region, it can be determined that the pixel is to be interpolated. Then, the bilinear interpolation algorithm is directly used in both the horizontal and vertical directions to upsample the first pixel to be interpolated, and the corresponding upsampled feature map is obtained.

[0093] Pixels whose maximum edge gradient value is greater than a preset edge gradient threshold T are identified as the second interpolation pixels in the edge region. Here, when the maximum edge gradient value of a pixel is greater than the preset edge gradient threshold T, that is... If this is the case, then this pixel can be determined as the second pixel to be interpolated in the edge region. Next, before upsampling, it is necessary to determine the direction of the second pixel to be interpolated. Here, the direction in which the second pixel obtains the maximum edge gradient value is taken as the gradient direction, expressed as follows:

[0094]

[0095] Taking the direction perpendicular to the gradient direction as the edge line direction, it can be represented as follows:

[0096]

[0097] For the second pixel to be interpolated, which is located in a direction other than the gradient direction and the edge line direction, a bilinear interpolation algorithm is used for upsampling to obtain the corresponding upsampled feature map.

[0098] For the second pixel to be interpolated located in the gradient direction, an upsampling is performed using a gradient-weighted Lanczos window function to obtain the corresponding upsampled feature map. The upsampled pixel is represented as w(d), and the gradient-weighted Lanczos window function is expressed as follows (1):

[0099] (1)

[0100] In the above formula (1), d is the distance from the pixel to be interpolated to the edge pixel, and r is the radius of the interpolation window. This represents the edge gradient value corresponding to the pixel.

[0101] For the second pixel to be interpolated located in the edge line direction, a higher-order Lanczos window function is used for upsampling to obtain the corresponding upsampled feature map. The upsampled pixel is represented as w(d), and the higher-order Lanczos window function is represented as follows (2):

[0102] (2)

[0103] In the above formula (2), d is the distance from the pixel to be interpolated to the edge pixel, and r is the radius of the interpolation window.

[0104] In this embodiment of the invention, pixels are divided by calculating the edge gradient value of each pixel in the high-level feature map, and different upsampling methods are used for processing. This solves the problem that the existing upsampling methods often use bilinear interpolation algorithms, which can only uniformly amplify the whole image, easily causing loss of high-frequency components and blurring of seedling edges to a certain extent.

[0105] By performing edge adaptive upsampling on pixels in the high-level feature map, multiple upsampled feature maps are obtained. These upsampled feature maps are then fused, and the fused feature map is input into the feature pyramid network for multiple iterative recursive calculations to obtain the first feature map corresponding to the patch.

[0106] Here, the feature pyramid network output feature layer is denoted as... ,in, S represents the number of levels in the backbone network of the first SwinTransformer model (i.e., the number of layers in the first composite stage module, which is 3 layers in this case). This can be expressed as formulas (3) and (4) below:

[0107] (3)

[0108] (4)

[0109] In the above formula (3), This represents the computational function of the top-down feature pyramid network of the backbone network (i.e., the first Swing Transformer model, which will not be elaborated upon later). This represents the feature map input to the i-th layer in the backbone network, which is the feature map obtained by fusing multiple upsampled feature maps. This indicates the number of levels in the backbone network from top to bottom.

[0110] Therefore, the feature pyramid network outputs a set of features denoted as After adding feedback connections, the feature pyramid network outputs a feature layer denoted as... This can be expressed as formulas (5) and (6) below:

[0111] (5)

[0112] (6)

[0113] The parameters in formula (5) above can be explained by referring to formula (3) above, and will not be repeated here. In formula (5) This indicates the number of levels in the backbone network from top to bottom. This represents the feature transformation function before connecting the transformed feature layers back to the backbone network, so that the feature pyramid network can complete the recursive operation. Thus, the feature pyramid network is expanded into a sequential network, and the iterative calculation formulas for the sequential network are expressed as follows: Formula (7) and Formula (8):

[0114] (5)

[0115] (6)

[0116] In the above formula, T represents the number of iterations, which can be set to 2. The superscript t indicates the operation step of the feature pyramid network in the t-th iteration. The feature pyramid network is implemented using a structure containing a dilated spatial convolutional pooling pyramid (ASPP), as shown below. Figure 2 The ASPP in the algorithm is used to perform recursive computation together.

[0117] After recursive calculation of the feature pyramid, the first feature map corresponding to the patch is finally obtained. Thus, as... Figure 2 As shown, in the first Swin Transformer model, each first composite stage module based on the RFP structure outputs a corresponding first feature map, which is then input into the second Swin Transformer model.

[0118] In this embodiment of the invention, a feature pyramid network is added to the backbone network of the Swin Transformer model. By introducing a recursive feature pyramid structure, the feature representation capability of the network is enhanced. Even in complex detection scenarios where the stems of cabbage seedlings in plug trays are slender and the leaves of mature seedlings overlap, the image feature extraction effect can still be guaranteed.

[0119] like Figure 2As shown, since the image to be processed is also input into the second Swin Transformer model, the input image to be processed (size H*W) is similarly divided into non-overlapping patches with a width, height, and channel size of 4*4*3 by the Patch Partition module in the second Swin Transformer model. The patches are then input into the three-layer second composite stage module based on the RFP structure in the second Swin Transformer model. In specific implementation, the first feature map output by the first Swin Transformer model is also input into the three-layer second composite stage module based on the RFP structure in the second Swin Transformer model.

[0120] Thus, the second composite stage module extracts features from the patch and the first feature map to obtain the second feature map, which is then used as the feature map for the stereo image. This allows for the fusion of multi-layer features from the network, further enhancing the model's feature extraction capabilities.

[0121] In the second composite stage module based on the RFP structure, the input is the first feature map and the corresponding patch. Since the model structure of the first Swin Transformer model and the second Swin Transformer model is the same, the specific network of the second composite stage module is also similar to that of the first composite stage module. The feature extraction method can refer to the first composite stage module. The specific feature extraction process will not be described in detail here.

[0122] For the right image of the binocular image, similar to the left image, it is also treated as the image to be processed and the above step 102 is performed to finally extract the corresponding second feature map. In this way, the corresponding second feature maps are extracted from both the left and right images for subsequent target detection.

[0123] In this embodiment of the invention, the Swing Transformer model based on the RFP structure is used to extract features from binocular images, which can significantly improve the feature extraction capability of the model. Even in complex detection scenarios where the stems of cabbage seedlings in plug trays are thin and the leaves of mature seedlings are mutually occluded, the image feature processing effect can be guaranteed, the accuracy of feature extraction can be improved, and the foundation can be laid for accurate prediction of the stem and leaf bounding boxes in subsequent processes.

[0124] Step 103: Perform regression prediction on the feature map of the binocular image to obtain the stem bounding box and leaf bounding box of the binocular image.

[0125] After extracting features from the stereo images, the next step is target detection of the stems and leaves. Regression prediction is performed on the feature maps of the stereo images to obtain the stem and leaf bounding boxes. Here, the feature maps output by the Swin Transformer model based on the RFP structure are input into the detection head, which is used for target detection, specifically locating the stems and leaves of cabbage seedlings in the stereo images. The detection head specifically includes two modules: a convolutional neural network and a small regression network.

[0126] After the feature map of the binocular image is input into the detection head, a convolutional neural network is first invoked to predict the feature map of the binocular image, obtaining the preliminary position and preliminary offset of the representative point of the stem and leaf of the cabbage seedling. The role of the convolutional neural network is to calculate the representative point that can represent the stem and leaf of the cabbage seedling based on the feature value of the pixels in the feature map. The representative point of the stem and leaf may be a pixel of the stem and leaf of the cabbage seedling, and the preliminary offset is the positional offset value that may be generated by the predicted preliminary position. Since the preliminary position of the representative point of the stem and leaf may be offset and needs to be corrected, this embodiment of the invention constructs a small regression network to correct the preliminary position and preliminary offset of the representative point of the stem and leaf, obtaining the corrected offset of the representative point of the stem and leaf.

[0127] The specific structure of the small regression network is as follows: The first layer is a 3x3 depthwise separable convolutional layer with 64 output channels, followed by a ReLU activation function. The second layer is a 1x1 depthwise separable convolutional layer with 128 output channels, followed by a ReLU activation function. The third layer is a 1x1 convolutional layer with n output channels, representing the number of seedling representative points, followed by a fully connected layer that outputs the corrected offset of the representative points.

[0128] The role of the small regression network is to further determine the positions of representative points in the stem and leaf areas. Therefore, the initial positions of the representative points in the stem and leaf areas are corrected by adjusting the offset after correction, thus obtaining the key points of the stem and leaf areas of the cabbage seedlings in the plug tray. These key points are the key points of the stem and leaf areas of the cabbage seedlings in the binocular image. Finally, the minimum bounding rectangle of the key points is determined as the stem bounding box and leaf bounding box of the binocular image. The region of the minimum bounding rectangle can just encompass all the key points of the stem and leaf areas, thus forming the stem bounding box and leaf bounding box of the corresponding binocular image.

[0129] In this embodiment of the invention, when performing regression prediction on the left and right images of binocular images, a convolutional neural network and a small regression network are used to accurately locate key points on the stems and leaves of cabbage seedlings in plug trays, forming stem bounding boxes and leaf bounding boxes. This facilitates subsequent matching of binocular images to determine the image coordinates of key points, and then calculates leaf uprightness and maximum leaf spread.

[0130] Step 104: Perform key point matching on the stem bounding box and leaf bounding box of the binocular image, and determine the three-dimensional geometric position coordinates of the cabbage seedling in the plug tray based on the image coordinates of the matched key points.

[0131] After target detection in step 103, regression prediction is performed on the left and right images of the stereo image to obtain the corresponding stem bounding boxes and leaf bounding boxes. However, since the position coordinates of key points in the stereo image are two-dimensional, there may be positional differences between the key points in the stereo image. Therefore, it is necessary to perform stereo matching based on stereo information to filter out the real key points, and then obtain the three-dimensional geometric position information of the seedling based on the real key points.

[0132] First, candidate points are extracted from the stem and leaf bounding boxes in the left and right images of the binocular images, respectively. In other words, key points within the bounding boxes are identified from the binocular images as candidate points. Then, a coarse matching process is performed on the candidate points in the left and right images based on binocular visual information to obtain matching points. This coarse matching process utilizes binocular visual information to determine the different positional information of the candidate points in the two images, such as their horizontal and vertical coordinates. However, the detection information in the left and right images is rarely completely identical; therefore, coarse matching based solely on corresponding positional information cannot guarantee matching accuracy.

[0133] Next, fine matching of the matching points is performed. Fine matching calculates the Euclidean distance between two matching points in the left and right images. The smaller the Euclidean distance between the two matching points, that is, the closer it is to 0, the better the matching effect, and the more likely these two matching points are key points of the stem and leaves of cabbage seedlings. The process of calculating the Euclidean distance for fine matching is as follows: (7)

[0134] (7)

[0135] In the above formula (7), Represents the midpoint of the left image The pixel value at that location. Represented as the midpoint of the right image The pixel value at that location. The offset range is the size of the right image, i.e., x and y.

[0136] After calculating the Euclidean distance for fine-grained matching, the next step is to filter the matching points based on this distance. Matching points with an Euclidean distance less than a threshold are identified as keypoints in the stereo image. A smaller Euclidean distance, closer to 0, indicates a better matching result. Therefore, a preset distance threshold can be used to measure this matching effect. When the calculated Euclidean distance is less than this threshold, it means that the two matching points in the left and right images have matched successfully and can be identified as keypoints in the stems and leaves of cabbage seedlings. Conversely, when the calculated Euclidean distance is greater than this threshold, it means that the two matching points in the left and right images have not matched successfully, and these matching points are filtered out.

[0137] In this embodiment of the invention, keypoint matching is performed on the stem bounding box and leaf bounding box of the binocular image to calculate the keypoints in the binocular image, thereby achieving accurate prediction of the stem and leaf parts in the binocular image.

[0138] Finally, the three-dimensional geometric position coordinates of the cabbage seedlings in the plug trays are determined based on the image coordinates of the matched keypoints. Multiple keypoints can be identified through keypoint matching, and the image coordinates of each keypoint in the binocular image can also be determined. The disparity information of the binocular image can be calculated based on the image coordinates. Next, the three-dimensional position coordinates of the keypoints in the real space coordinate system, i.e., the three-dimensional geometric position coordinates, are calculated based on the disparity information of the binocular image.

[0139] Here, as Figure 4 As shown, the optical centers of the left and right lenses of the stereo camera are respectively and Images of cabbage seedlings in plug trays were taken along the left and right optical axes, respectively, with baseline B as... and The distance is given by f, and the focal length is f. On the left plane corresponding to the left image, the coordinates of the keypoints obtained through fine matching are... , recorded as Correspondingly, on the right plane corresponding to the right image, the coordinates of the keypoints obtained by fine matching are: , recorded as d represents the Euclidean distance calculated according to formula (7).

[0140] and These are the coordinates of the binocular camera image, that is, the coordinates of the key points on the binocular image. Therefore, it is also necessary to determine the three-dimensional position coordinates P of the key points in the real space coordinate system, denoted as P. ,in, express Figure 4 The coordinates of the midpoint P along the z-axis can be obtained from the geometric relations of similar triangles, as shown in formulas (8) to (10):

[0141] (8)

[0142] (9)

[0143] (10)

[0144] In formulas (8) to (10) above, B is the optical center of the left and right lenses of the binocular camera. and The distance is f, where f is the focal length of the binocular camera.

[0145] Based on the system of equations formed by formulas (8) and (10), the coordinates of point P are finally calculated as follows: formulas (11) to (13):

[0146] (11)

[0147] (12)

[0148] (13)

[0149] In formulas (11) to (13) above, D represents the disparity of the coordinates of point P, expressed as formula (14) below:

[0150] (14)

[0151] Substituting formula (14) into the system of equations (11) to (13) for calculation, we can obtain the coordinates of point P. This refers to the three-dimensional geometric coordinates of the cabbage seedlings in the plug trays. Each keypoint P determined by this fine-matching method can then have its corresponding three-dimensional geometric coordinates calculated.

[0152] In this embodiment of the invention, the three-dimensional geometric position coordinates of cabbage seedlings in plug trays are determined based on the image coordinates of the matched key points, realizing the conversion from two-dimensional image coordinates to three-dimensional spatial coordinates, which facilitates the final calculation of leaf uprightness and maximum leaf spread from a three-dimensional spatial perspective.

[0153] Step 105: Determine the leaf uprightness and maximum leaf spread of cabbage seedlings in plug trays based on the three-dimensional geometric position coordinates.

[0154] After determining the three-dimensional geometric coordinates of each key point of the cabbage plug seedling in step 104, the leaf uprightness and maximum leaf spread of the cabbage plug seedling are finally determined based on these three-dimensional geometric coordinates. By integrating the three-dimensional geometric coordinates of these key points, which are all in a real-world coordinate system, a polygonal region encompassing these three-dimensional geometric coordinates can be determined in the image using these real-world coordinates. This polygonal region serves as the detection bounding box for the stem and leaf of the cabbage plug seedling.

[0155] For example, such as Figure 5 As shown, after achieving the target detection of cabbage plug seedlings, the detection frames for plant stems were determined respectively. Figure 5 The coordinates of the top left corner of the blue rectangle and the coordinates of the bottom right corner Leaf detection frame ( Figure 5 The coordinates of the top left corner of the red rectangle in the middle and the coordinates of the bottom right corner Then calculate the leaf uprightness of cabbage plug seedlings. It is expressed as the following formula (15):

[0156] (15)

[0157] Calculate the maximum leaf spread of cabbage seedlings in plug trays It is expressed as the following formula (16):

[0158] (16)

[0159] like Figure 5 As shown, in a mechanized transplanting agronomic scenario, when the actuator at the seedling-grabbing end grips the substrate of the cabbage seedling tray, the leaf uprightness and maximum leaf spread of the cabbage seedling tray provided in this application embodiment can be calculated in real time. Then, the transplanting process can be adjusted based on the leaf uprightness and maximum leaf spread to avoid problems such as seedling cup blockage, seedling damage, and seedling tilting during transplanting, thereby improving the efficiency of mechanized transplanting.

[0160] In this embodiment of the invention, based on binocular images of cabbage plug seedlings, a SwinTransformer model based on an RFP structure is used to extract feature maps and perform regression prediction to obtain the stem and leaf bounding boxes of the binocular images. Next, keypoint matching is performed on the bounding boxes to determine the three-dimensional geometric coordinates of the cabbage plug seedlings based on the image coordinates of the matched keypoints. Finally, the leaf uprightness and maximum leaf spread of the cabbage plug seedlings are calculated. Thus, on the one hand, the regression prediction of the model improves the accuracy of target detection for the stem and leaf parts of the cabbage plug seedlings; on the other hand, by capturing images of the plug seedlings, leaf uprightness and maximum leaf spread can be detected and determined in real time, forming a criterion for the mechanization suitability of plug seedlings, which can be applied to the actual production of mechanized transplanting agriculture.

[0161] In other embodiments, the calculation of leaf uprightness and maximum leaf spread of cabbage plug seedlings mainly depends on the matching of key points in the binocular images. The selection of key points can have a great impact on the calculation results, and there are high requirements for the accuracy of key points, stem bounding boxes and leaf bounding boxes obtained by the regression prediction of the detection head in the model.

[0162] Therefore, to ensure the accuracy of calculating leaf uprightness and maximum leaf spread, this embodiment of the invention also trains an object detection model to more accurately predict key points, stem bounding boxes, and leaf bounding boxes of cabbage plug seedlings. After acquiring binocular images of the collected cabbage plug seedlings, the object detection model is called to detect the binocular images, obtaining the stem bounding boxes and leaf bounding boxes of the binocular images. Finally, the leaf uprightness and maximum leaf spread of the cabbage plug seedlings are determined based on the stem bounding boxes and leaf bounding boxes of the binocular images.

[0163] The process here involves keypoint matching between the stem and leaf bounding boxes of the binocular images. Based on the image coordinates of the matched keypoints, the three-dimensional geometric coordinates of the cabbage seedlings in the plug trays are determined. Finally, the leaf uprightness and maximum leaf spread of the cabbage seedlings are determined based on these three-dimensional geometric coordinates. The determination method is similar to steps 104 and 105 above and can be referred to accordingly; therefore, it will not be elaborated upon here.

[0164] Specifically, the object detection model includes a feature extraction network and a small multi-task network. The feature extraction network can be the Swin Transformer model based on the RFP structure used in step 102, or other improved Swin Transformer models. Its main function is to extract feature maps from the stereo image. The small multi-task network includes a bounding box regression network and a multi-task regression network. Its main function is to predict the stem and leaf keypoints, stem bounding boxes, and leaf bounding boxes based on the extracted feature maps. The training process of the object detection model is described below.

[0165] First, stereo images of cabbage seedlings in plug trays are acquired using a stereo camera. These images are then used to train an object detection model. During training, the stereo image samples are divided into five equal parts, with the training and test sets in a 4:1 ratio. Five-fold cross-validation is used for model training. Each stereo image sample contains corresponding ground truth keypoints and ground truth stem / leaf bounding boxes (i.e., ground truth stem bounding boxes and ground truth leaf bounding boxes), which serve as training labels used to construct loss values ​​for parameter optimization.

[0166] At the start of training, binocular image samples of cabbage seedlings in plug trays are input into the feature extraction network for feature extraction processing, resulting in the output feature map F of the binocular image samples. This is also the forward propagation process. The feature extraction process is similar to step 102 above, and will not be repeated here.

[0167] Next, the output feature map F obtained from the feature extraction network is input into a small multi-task network for regression prediction processing to obtain the predicted stem and leaf bounding boxes and the classification information of the stems and leaves of cabbage seedlings in plug trays. The small multi-task network is further divided into a bounding box regression network and a multi-task network. The bounding box regression network first predicts the corresponding preliminary stem and leaf bounding boxes based on the output feature map F obtained from the feature extraction network. The prediction process can refer to the regression prediction process for the feature map of the binocular image in step 103 above, which will not be elaborated here.

[0168] The multi-task network is then used to further predict the output feature map F, correcting the stem and leaf partial classification information of the initial stem and leaf bounding boxes. During the multi-task network processing, some initial representative points are defined in the output feature map F, denoted as... .

[0169] The multi-task network performs location correction and classification tasks on these initial representative points. The first three layers of the multi-task network structure are shared, and then it is divided into two branches: a location correction branch and a classification branch. The location correction branch performs the regression task, processes the location correction of the representative points, and outputs the predicted stem and leaf bounding boxes (i.e., the predicted stem bounding box and the predicted leaf bounding box). The classification branch performs the classification task and determines the classification information of the stem and leaf.

[0170] In the three shared network layers, the first layer is a depthwise separable convolutional layer, which outputs the corresponding feature information. It is expressed as the following formula (17):

[0171] (17)

[0172] In the above formula (17), This represents the processing function of the first depthwise separable convolutional layer. This represents the activation function of the first depthwise separable convolutional layer. The output feature map F obtained by the feature extraction network is represented as F. This represents the weight parameters of the first depthwise separable convolutional layer.

[0173] The second layer is also a depthwise separable convolutional layer, which takes the feature information output from the first depthwise separable convolutional layer as input. Output corresponding feature information It is expressed as the following formula (18):

[0174] (18)

[0175] In the above formula (18), This represents the processing function of the second depthwise separable convolutional layer. This represents the activation function of the second depthwise separable convolutional layer. This represents the feature information output by the first depthwise separable convolutional layer. This represents the weight parameters of the second depthwise separable convolutional layer.

[0176] The third layer of the shared network is also a depthwise separable convolutional layer, which takes the feature information from the output of the second depthwise separable convolutional layer as input. Output corresponding feature information It is expressed as the following formula (19):

[0177] (19)

[0178] In the above formula (19), This represents the processing function of the third depthwise separable convolutional layer. This represents the activation function of the third depthwise separable convolutional layer. This represents the feature information output by the second depthwise separable convolutional layer. This represents the weight parameters of the third depthwise separable convolutional layer.

[0179] The next step is to divide the process into two branches, using the feature information output from the third depthwise separable convolutional layer. The inputs are fed into two branches, with the position correction branch being a fully connected layer, represented as follows: It is used to perform regression tasks, process representative point position correction tasks, and output the correction values ​​of the initial key points. It is expressed as the following formula (20):

[0180] (20)

[0181] The classification branch is also a fully connected layer, represented as The feature information output by the third depthwise separable convolutional layer Mapping yields partial classification information of stems and leaves in cabbage seedling trays. It is expressed as the following formula (21):

[0182] (twenty one)

[0183] Next, we will use the correction value. For the initial representative point The correction is performed; specifically, each initial representative point is corrected using a corresponding correction value. The correction is expressed as follows: (22)

[0184] (twenty two)

[0185] After each initial representative point is corrected, the set of corrected representative points can be obtained. These are the predicted keypoints used as the final output of the small regression network. Then, the minimum bounding rectangle that determines these predicted keypoints is used. Let B be denoted as B, and expressed as the following formula (23):

[0186] (twenty three)

[0187] Next, standard bounding box regression techniques can be used to further refine the minimum bounding rectangle B, resulting in the predicted stem and leaf bounding box. It is expressed as the following formula (24):

[0188] (twenty four)

[0189] In the above formula (24), The bounding box regression network first predicts the corresponding preliminary stem-leaf bounding boxes based on the output feature map F obtained by the feature extraction network.

[0190] Through the above steps, the output feature map F is input into a small multi-task network for regression prediction processing to obtain the predicted stem and leaf bounding boxes. Information on the classification of stems and leaves of cabbage plug seedlings Next, based on the actual stem and leaf bounding boxes and the predicted stem and leaf bounding boxes... To construct the bounding box regression loss, the prediction results of the small multi-task network are compared with the true labels to construct the corresponding bounding box regression loss. .

[0191] The classification information for stems and leaves of cabbage seedlings in plug trays here specifically represents the feature information of key points on the stem and leaves. This means determining whether each pixel in the binocular image sample belongs to a key point on the stem or a key point on the leaf. This classification result is then compared with the actual results of the key points on the stem and leaves in the binocular image sample to construct the stem and leaf information classification loss. Specifically, the true key points within the true stem and leaf bounding boxes in the binocular image samples can be extracted first as the true results of the stem and leaf key points. Then, based on the true key points and the stem and leaf classification information of cabbage seedlings in plug trays, a stem and leaf information classification loss can be constructed. The true key points within the true stem and leaf bounding boxes are compared with the classification results of stem key points or leaf key points to construct the stem and leaf information classification loss. The construction method can be to construct cross-entropy loss or mean squared error loss.

[0192] Finally, the bounding box regression loss is used. and stem and leaf information classification loss The final loss L of the object detection model is expressed as formula (25):

[0193] (25)

[0194] In the above formula (25), Represents the bounding box regression loss and stem and leaf information classification loss The balancing weights are set according to actual needs, and are generally between 0 and 1.

[0195] In each training iteration, the final loss L is first calculated using the steps described above. Then, the final loss L is backpropagated in the feature extraction network and the small multi-task network to update the parameters of the object detection model. As mentioned earlier, the binocular image samples are divided into five equal parts, forming a training set and a test set in a 4:1 ratio. Five-fold cross-validation is used for model training. The batch size is set to 4. The number of training iterations (Epochs) is set to 1000. The model is optimized using the stochastic gradient descent (SGD) algorithm, with an initial learning rate of 0.01. Training stops when the loss function begins to converge or when the number of training iterations reaches 1000.

[0196] The trained object detection model can directly predict the results of binocular images of cabbage seedlings in plug trays, outputting stem and leaf bounding boxes, and thus determining the leaf uprightness and maximum leaf spread of the cabbage seedlings.

[0197] This invention trains an object detection model and uses the trained model to detect stem and leaf bounding boxes and classify key points of stems and leaves. This enables more accurate prediction of key points, stem bounding boxes, and leaf bounding boxes of cabbage plug seedlings, further ensuring the accuracy of prediction of leaf uprightness and maximum leaf expansion of cabbage plug seedlings.

[0198] The following describes the identification device for leaf uprightness and maximum leaf spread of organicated cabbage plug seedlings provided by the present invention. The identification device for leaf uprightness and maximum leaf spread of organicated cabbage plug seedlings described below can be referred to in correspondence with the identification method for leaf uprightness and maximum leaf spread of organicated cabbage plug seedlings described above.

[0199] like Figure 6 As shown, the device for recognizing leaf uprightness and maximum leaf spread of mechanized cabbage plug seedlings includes: an acquisition module 601, an extraction module 602, a prediction module 603, a matching module 604, and a determination module 605. Specifically, the acquisition module 601 is used to acquire binocular images of the collected cabbage plug seedlings; the extraction module 602 is used to call the SwinTransformer model based on the RFP structure to extract features from the binocular images, obtaining feature maps of the binocular images; the prediction module 603 is used to perform regression prediction on the feature maps of the binocular images, obtaining stem bounding boxes and leaf bounding boxes of the binocular images; the matching module 604 is used to perform keypoint matching on the stem bounding boxes and leaf bounding boxes of the binocular images, and determine the three-dimensional geometric position coordinates of the cabbage plug seedlings based on the image coordinates of the matched keypoints; the determination module 605 is used to determine the leaf uprightness and maximum leaf spread of the cabbage plug seedlings based on the three-dimensional geometric position coordinates.

[0200] It should be noted that the beneficial effects of the identification device for leaf uprightness and maximum leaf spread of organicated cabbage plug seedlings mentioned above correspond to each other. Therefore, the beneficial effects of the identification device for leaf uprightness and maximum leaf spread of organicated cabbage plug seedlings will not be elaborated here.

[0201] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a method for recognizing the leaf uprightness and maximum leaf spread of machine-friendly cabbage plug seedlings. This method includes: acquiring binocular images of the collected cabbage plug seedlings; calling a Swing Transformer model based on an RFP structure to extract features from the binocular images, obtaining feature maps of the binocular images; performing regression prediction on the feature maps of the binocular images to obtain stem bounding boxes and leaf bounding boxes of the binocular images; performing keypoint matching on the stem bounding boxes and leaf bounding boxes of the binocular images, and determining the three-dimensional geometric position coordinates of the cabbage plug seedlings based on the image coordinates of the matched keypoints; and determining the leaf uprightness and maximum leaf spread of the cabbage plug seedlings based on the three-dimensional geometric position coordinates.

[0202] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the methods provided above for identifying the leaf uprightness and maximum leaf spread of organic cabbage plug seedlings. The method includes: acquiring binocular images of collected cabbage plug seedlings; calling a SwinTransformer model based on an RFP structure to extract features from the binocular images to obtain feature maps of the binocular images; performing regression prediction on the feature maps of the binocular images to obtain stem bounding boxes and leaf bounding boxes of the binocular images; performing keypoint matching on the stem bounding boxes and leaf bounding boxes of the binocular images, and determining the three-dimensional geometric position coordinates of the cabbage plug seedlings based on the image coordinates of the matched keypoints; and determining the leaf uprightness and maximum leaf spread of the cabbage plug seedlings based on the three-dimensional geometric position coordinates.

[0204] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the method for recognizing leaf uprightness and maximum leaf spread of machine-friendly cabbage plug seedlings provided by the above methods. The method includes: acquiring binocular images of collected cabbage plug seedlings; calling a Swing Transformer model based on an RFP structure to extract features from the binocular images to obtain feature maps of the binocular images; performing regression prediction on the feature maps of the binocular images to obtain stem bounding boxes and leaf bounding boxes of the binocular images; performing keypoint matching on the stem bounding boxes and leaf bounding boxes of the binocular images, and determining the three-dimensional geometric position coordinates of the cabbage plug seedlings based on the image coordinates of the matched keypoints; and determining the leaf uprightness and maximum leaf spread of the cabbage plug seedlings based on the three-dimensional geometric position coordinates.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying the leaf stand and maximum leaf spread of cabbage seedlings in plug trays, characterized in that, The method comprises: acquiring a binocular image of the collected cabbage seedling plug; calling an RFP structure-based Swin Transformer model to perform feature extraction on the binocular image to obtain a feature map of the binocular image; performing regression prediction on the feature map of the binocular image to obtain a stem bounding box and a leaf bounding box of the binocular image; performing key point matching on the stem bounding box and the leaf bounding box of the binocular image, and determining a three-dimensional geometric position coordinate of the cabbage seedling plug according to image coordinates of the matched key points; determining a leaf straightness and a maximum leaf spread of the cabbage seedling plug according to the three-dimensional geometric position coordinate, respectively; the calling of the RFP structure-based Swin Transformer model to perform feature extraction on the binocular image to obtain a feature map of the binocular image comprises: dividing a to-be-processed image into a plurality of patches, inputting the patches into a 3-layer first composite stage module based on an RFP structure in a first Swin Transformer model, performing feature extraction on the patches through the first composite stage module to obtain a first feature map corresponding to the patches, wherein the to-be-processed image is a left image or a right image in the binocular image; wherein the backbone network of the Swin Transformer model is top-down, and the high-level feature map adopts edge adaptive upsampling; inputting the patches into a 3-layer second composite stage module based on an RFP structure in a second Swin Transformer model; inputting the first feature map into the 3-layer second composite stage module based on an RFP structure in the second Swin Transformer model, respectively, performing feature extraction on the patches and the first feature map through the second composite stage module to obtain a second feature map; taking the second feature map as the feature map of the binocular image.

2. The method of identifying the straightness of leaves and maximum leaf spread of cabbage plug seedling leaves for mechanization suitability according to claim 1, characterized in that, the feature extraction on the patches through the first composite stage module to obtain the first feature map corresponding to the patches comprises: performing embedding processing on the patches to obtain a corresponding embedded patch sequence, and remodeling the embedded patch sequence into a two-dimensional feature map through a W-MSA module in the first composite stage module; dividing the two-dimensional feature map through a non-overlapping detection window, and performing grouped self-attention calculation on the divided two-dimensional feature map in the detection window through a W-MSA module and a SW-MSA module in the first composite stage module to obtain a window representation corresponding to the detection window; performing global down-sampling self-attention calculation on the window representation corresponding to each detection window, and inputting the calculated result into a Token fusion module for feature fusion to obtain a fusion representation; inputting the fusion representation into an MLP module to obtain a high-level feature map of the patch; inputting the high-level feature map into a feature pyramid network included in the first composite stage module for feature extraction to obtain the first feature map corresponding to the patch.

3. The method of identifying the straightness of leaves and maximum leaf spread of cabbage plug seedling leaves for mechanization suitability according to claim 2, characterized in that, The inputting the high-level feature map into a feature pyramid network in the first composite stage module for feature extraction comprises: Edge adaptive up-sampling is performed on the high-level feature map to obtain a plurality of up-sampled feature maps, and the up-sampled feature maps are fused; The fused feature map is input into a feature pyramid network for multiple times of iterative recursive calculation to obtain the first feature map corresponding to the patch.

4. The method of identifying straightness and maximum leaf spread of cabbage plug seedling leaves for mechanization suitability as claimed in claim 3, wherein, The edge adaptive up-sampling on the high-level feature map to obtain a plurality of up-sampled feature maps comprises: Each pixel point in the high-level feature map is traversed, and the Sobel operator is called to calculate the edge gradient values of each pixel point in the up, down, left and right four directions, and the maximum edge gradient value of each pixel point is determined; The pixel point with a maximum edge gradient value less than or equal to a preset edge gradient threshold value is determined as a first to-be-interpolated pixel point of a flat area, and a bilinear interpolation algorithm is used to up-sample the first to-be-interpolated pixel point in the horizontal and vertical directions to obtain a corresponding up-sampled feature map; The pixel point with a maximum edge gradient value greater than the preset edge gradient threshold value is determined as a second to-be-interpolated pixel point of an edge area; The direction with the maximum edge gradient value in the second to-be-interpolated pixel point is taken as a gradient direction, and the vertical direction of the gradient direction is taken as an edge line direction; For the second to-be-interpolated pixel point located in the non-gradient direction and the non-edge line direction, a bilinear interpolation algorithm is used for up-sampling to obtain a corresponding up-sampled feature map; For the second to-be-interpolated pixel point located in the gradient direction, a gradient value weighted Lanczos window function is used for up-sampling to obtain a corresponding up-sampled feature map; For the second to-be-interpolated pixel point located in the edge line direction, a high-order Lanczos window function is used for up-sampling to obtain a corresponding up-sampled feature map.

5. The method of identifying the straightness of leaves and maximum leaf spread of cabbage plug seedling leaves for mechanization suitability as claimed in claim 1, wherein, The regression prediction on the feature map of the binocular image to obtain the stem and leaf boundary boxes of the binocular image comprises: A convolutional neural network is called to predict the feature map of the binocular image to obtain the preliminary position and preliminary offset of the cabbage plug seedling stem and leaf representative points, and a small regression network is used to correct the preliminary position and preliminary offset of the stem and leaf representative points to obtain the corrected offset of the stem and leaf representative points. The initial position of the stem and leaf representative points is corrected through the corrected offset to obtain the key points of the cabbage plug seedling stem and leaf; The minimum circumscribed rectangle of the key points is determined as the stem and leaf boundary boxes of the binocular image.

6. The method of identifying the straightness of leaves and maximum leaf spread of cabbage plug seedling leaves for mechanization suitability as claimed in claim 1, wherein, The key point matching on the stem and leaf boundary boxes of the binocular image comprises: For the left image and the right image in the binocular image, candidate points in the stem and leaf boundary boxes are extracted respectively; The candidate points of the left image and the candidate points of the right image are coarsely matched based on binocular vision information to obtain matching points; The Euclidean distance between the matching points of the left image and the matching points of the right image is determined; Determine the matching points with the Euclidean distance less than the distance threshold as the key points in the binocular image.

7. The method of identifying the straightness of leaves and maximum leaf spread of cabbage plug seedling leaves for mechanization suitability as claimed in claim 1, wherein, The method further comprises: According to the image coordinates of the key points obtained by matching, the parallax values of the key points in the binocular image are determined. According to the image coordinates and the parallax values, the three-dimensional geometric position coordinates of the cabbage plug seedlings are determined.

8. The method of identifying straightness of leaves and maximum leaf spread of cabbage plug seedling leaves for mechanization suitability as claimed in claim 1 wherein, After the binocular image of the collected cabbage plug seedlings is obtained, the method further comprises: Calling a target detection model to detect the binocular image to obtain the stem bounding box and the leaf bounding box of the binocular image; Based on the stem bounding box and the leaf bounding box of the binocular image, the leaf erectness and the maximum leaf spread of the cabbage plug seedlings are determined. The target detection model comprises a feature extraction network and a small multi-task network, and is trained by binocular image samples of cabbage plug seedlings, which carry real stem and leaf bounding boxes. The training process of the target detection model comprises: The binocular image samples of the cabbage plug seedlings are input into the feature extraction network for feature extraction processing to obtain the output feature map of the binocular image samples; The output feature map is input into the small multi-task network for regression prediction processing to obtain the predicted stem and leaf bounding boxes and the stem and leaf classification information of the cabbage plug seedlings; According to the real stem and leaf bounding boxes and the predicted stem and leaf bounding boxes, a bounding box regression loss is constructed. The real key points in the real stem and leaf bounding boxes in the binocular image samples are extracted, and based on the real key points and the stem and leaf classification information of the cabbage plug seedlings, a stem and leaf information classification loss is constructed. The final loss of the target detection model is constructed according to the bounding box regression loss and the stem and leaf information classification loss, and the parameters of the target detection model are updated through the final loss in the feature extraction network and the small multi-task network.

9. A device for identifying the straightness of leaves and the maximum leaf spread of cabbage seedling in plug tray, characterized in that, The device comprises: An acquisition module for acquiring a binocular image of collected cabbage plug seedlings; An extraction module for calling a Swin Transformer model based on RFP structure to perform feature extraction on the binocular image to obtain a feature map of the binocular image; A prediction module for performing regression prediction on the feature map of the binocular image to obtain the stem bounding box and the leaf bounding box of the binocular image; A matching module for matching the key points of the stem bounding box and the leaf bounding box of the binocular image, and determining the three-dimensional geometric position coordinates of the cabbage plug seedlings according to the image coordinates of the matching key points; A determination module for determining the leaf erectness and the maximum leaf spread of the cabbage plug seedlings according to the three-dimensional geometric position coordinates, respectively. The calling of the Swin Transformer model based on RFP structure to perform feature extraction on the binocular image to obtain the feature map of the binocular image comprises: The image to be processed is divided into a plurality of patches, the patches are input into a first composite stage module based on a 3-layer RFP structure in a first Swin Transformer model, feature extraction is performed on the patches through the first composite stage module, and a first feature map corresponding to the patches is obtained, wherein the image to be processed is a left image or a right image in the binocular image; Wherein the backbone network of the Swin Transformer model is top-down, and the edge adaptive up-sampling is adopted for the high-level feature map; The patch is input into a second composite stage module based on a 3-layer RFP structure in a second Swin Transformer model; The first feature map is input into a second composite stage module based on a 3-layer RFP structure in a second Swin Transformer model, feature extraction is performed on the patch and the first feature map through the second composite stage module, and a second feature map is obtained; The second feature map is taken as the feature map of the binocular image.

Images