Leafy seedling recognition and transplanting clamp position detection method and device

Abstract

The application provides a leafy seedling recognition and transplanting clamping position detection method and device, which comprises the following steps: obtaining a to-be-tested seedling image; performing seedling recognition and transplanting clamping position detection on the to-be-tested seedling image based on a detection model to obtain seedling recognition results and transplanting clamping position detection results; the detection model is obtained by training a convolutional neural network based on sample seedling images as training samples, disease seedling types and healthy seedling types as sample labels, and a complex sample mining loss function; the format and data length of the sample labels corresponding to the disease seedling types and the sample labels corresponding to the healthy seedling types are respectively aligned; and the complex sample mining loss function is determined based on a Focal loss function, a loss value sorting theory and a sample layering method. The method improves the transplanting efficiency and survival rate of leafy seedlings.

Description

Technical Field

[0001] The present invention relates to the fields of crop identification and image processing, and in particular to a method and device for identifying leafy vegetable seedlings and detecting their transplanting and clamping positions. Background Art

[0002] Vegetable transplanting is a key link in vegetable production, and the quality of transplanting will directly affect crop yield. Due to the large leaf area of ​​leafy vegetable seedlings (such as cabbage seedlings), the leaves and stems are easily damaged during the mechanized seedling transplanting process. According to statistics, the current damage rate of mechanized transplanting of seedlings (also known as seedlings) is about 7%-10%. In order to solve the problem of high damage rate in cabbage seedling transplanting, it is necessary to study the mechanized transplanting process of cabbage seedlings and other crops to reduce the seedling damage rate.

[0003] At present, the seedling removal device of the vegetable transplanter is mainly divided into insertion clamping type, ejection type and ejection clamping type, stem clamping type and seedling removal mechanism for non-standard seedling trays according to the different clamping methods; the stem clamping seedling removal mechanism can reduce the damage to the seedling body and its root system, and its seedling removal operation is simple, especially suitable for potted seedlings with thicker stems. The seedling removal effect is significant and the damage to the potted seedlings is small. It is widely used in actual production.

[0004] In related technologies, key image information of objects is usually extracted, and the movement of the robot is controlled according to the image information to study the transplanting process of seedlings of legumes, fruits, vegetables and flowers. When the target of seedling transplanting is leafy vegetable seedlings such as cabbage, it is necessary to select a suitable clamping position to protect the leaves or roots of the seedlings from damage during transplanting, otherwise it will affect the quality of the crop; moreover, the existing technology manually selects strong seedlings from the seedlings during transplanting to improve the survival rate of cabbage. The efficiency of manual screening is low, and it is easy to damage the seedlings due to improper operation, resulting in economic losses. Summary of the Invention

[0005] The present invention provides a method and device for identifying leafy vegetable seedlings and detecting the clamping position for transplanting, which are used to solve the defects of the prior art that leafy vegetable seedlings are easily damaged in leaves and stems and have low transplanting efficiency when transplanting, thereby improving the transplanting efficiency and survival rate of leafy vegetable seedlings.

[0006] The present invention provides a method for identifying leafy vegetable seedlings and detecting their transplanting clamping positions, comprising:

[0007] Acquire images of seedlings to be tested;

[0008] performing seedling recognition and transplanting clamping position detection on the seedling image to be tested based on the detection model to obtain a seedling recognition result and a transplanting clamping position detection result;

[0009] Among them, the detection model is based on sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels, and the convolutional neural network is trained with a complex sample mining loss function; the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type respectively; the complex sample mining loss function is determined based on the Focal loss function, loss value ranking theory and sample stratification method.

[0010] According to a method for detecting leafy vegetable seedling identification and transplanting clamping positions provided by the present invention, the detection model is obtained through the following steps:

[0011] extracting the phenotypic features from the sample seedling images based on the convolutional neural network;

[0012] Calculating a Focal loss value of each sample seedling image according to the phenotypic characteristics based on a Focal loss function; and updating a historical loss value of each sample seedling image according to the Focal loss value to determine historical difficulty data of each sample seedling image;

[0013] Calculating the comprehensive loss of each sample seedling image according to the Focal loss value and the historical difficulty data, and sorting the comprehensive losses of each sample seedling image by a non-maximum suppression sorting method to obtain a sorted comprehensive loss;

[0014] The sorted comprehensive loss is stratified according to a stratification threshold to obtain multiple stratification losses, and each stratification loss is sampled proportionally to obtain multiple sampling losses, where different stratification losses correspond to different sampling losses;

[0015] Calculate the average of each sample loss and accumulate the averages to obtain the target loss;

[0016] The convolutional network is optimized and trained according to the target loss, and the detection model is obtained under the condition that the network converges.

[0017] According to a leafy vegetable seedling identification and transplanting clamping position detection method provided by the present invention, the detection model is further obtained through the following steps:

[0018] Construct a plug tray dataset corresponding to the sample seedlings in the transplanting scenario;

[0019] Pre-training the 1st to Nth layers of the convolutional neural network using the ImageNet large dataset based on transfer learning to obtain a pre-trained convolutional neural network; N is a positive integer greater than 2;

[0020] The network parameters of the output layer of the pre-trained convolutional neural network are fine-tuned according to the hole tray data set to obtain the detection model.

[0021] According to a method for identifying leafy vegetable seedlings and detecting transplant clamping positions provided by the present invention, the convolutional neural network includes an improved inverted bottleneck block structure, and the convolutional neural network is provided with multiple channels to expand features; the convolutional neural network also includes a deformable convolutional network, and the deformable convolutional network is used to introduce a learnable offset and adopt a dedicated window mode to perform non-uniform sampling on the input feature map; the convolutional neural network is used to use a dedicated sliding window for each position through an unbounded range of aggregation weights; the shape of the sliding window is independent of the aggregation weight input.

[0022] According to a method for identifying leafy vegetable seedlings and detecting transplanting clamping positions provided by the present invention, the convolutional neural network includes an attention module based on scale change;

[0023] The scale-variation-based attention module is used to extract the phenotypic features from the sample seedling image using the following formula:

[0024] ;

[0025] in, A m (x, i) is the phenotypic feature; x is the network input tensor; i is the output size of the attention map corresponding to the sample seedling image, f(x, i) represents the average pooling function; P 1 (x, i) is the association probability, n Indicates the number of output pool tensors; P 1 (x, i) The following conditions are met:

[0026] and .

[0027] According to a method for identifying leafy vegetable seedlings and detecting the position of a transplanting clamp provided by the present invention, before extracting the phenotypic characteristics from the sample seedling image, the method further includes:

[0028] performing fine scaling processing on the sample seedling image by a bilinear interpolation algorithm to obtain a processed sample seedling image;

[0029] Image enhancement technology is used to perform data expansion on the processed sample seedling image to obtain an expanded sample seedling image.

[0030] The present invention also provides a leafy vegetable seedling identification and transplanting clamping position detection device, comprising:

[0031] An image acquisition module is used to acquire images of seedlings to be tested;

[0032] a detection module, configured to perform seedling recognition and transplant clamping position detection on the seedling image to be tested based on a detection model, and obtain a seedling recognition result and a transplant clamping position detection result;

[0033] Among them, the detection model is based on sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels, and the convolutional neural network is trained with a complex sample mining loss function; the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type respectively; the complex sample mining loss function is determined based on the Focal loss function, loss value ranking theory and sample stratification method.

[0034] The present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the above-described methods for identifying leafy vegetable seedlings and detecting transplanting clamping positions.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of the above is implemented.

[0036] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for identifying leafy vegetable seedlings and detecting transplanting clamping positions.

[0037] The method for identifying leafy vegetable seedlings and detecting the transplanting clamping position provided by the present invention uses diseased seedling types and healthy seedling types as sample labels, and uses a complex sample mining loss function to train a convolutional neural network to obtain a detection model, which performs seedling identification and transplanting clamping position detection on the seedling images to be tested, obtains seedling identification results and transplanting clamping position detection results, and improves the transplanting efficiency and seedling survival rate of leafy vegetable seedlings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a flow chart of the leafy vegetable seedling identification and transplanting clamping position detection method provided by the present invention.

[0040] Figure 2 This is a schematic diagram of the position of the plug tray seedlings, transplanting mechanism and camera provided by the present invention.

[0041] Figure 3 It is a structural diagram of the convolutional neural network provided by the present invention.

[0042] Figure 4 It is a structural schematic diagram of the leafy vegetable seedling identification and transplanting clamping position detection device provided by the present invention.

[0043] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] The following combination Figures 1-4 The present invention describes a method and device for identifying leafy vegetable seedlings and detecting the clamping position of the seedlings after transplantation.

[0046] Figure 1 FIG. 1 is a flow chart of the method for detecting leafy vegetable seedling identification and transplanting clamping position provided by the present invention, as shown in FIG. Figure 1 As shown, the leafy vegetable seedling identification and transplanting clamping position detection method includes the following steps:

[0047] Step 110: Acquire an image of the seedling to be tested.

[0048] In this step, the leafy vegetable seedlings include leafy vegetable seedlings such as cabbage, spinach, lettuce or garland chrysanthemum.

[0049] In this embodiment, the image of the seedling to be tested can be obtained from a public vegetable database or captured by a camera or other device.

[0050] Step 120: Based on the detection model, seedling recognition and transplant clamping position detection are performed on the seedling image to be tested to obtain seedling recognition results and transplant clamping position detection results; wherein, the detection model is based on the sample seedling image as the training sample, the diseased seedling type and the healthy seedling type as the sample labels, and the convolutional neural network is trained with the complex sample mining loss function; wherein, the sample label corresponding to the healthy seedling type is annotated with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type respectively; the complex sample mining loss function is determined based on the Focal loss function, the loss value ranking theory and the sample stratification method.

[0051] In this step, the detection model can automatically determine whether the growth status of the seedlings to be tested meets the transplanting standards, and detect the appropriate clamping position of the transplanting mechanism for the seedlings, thereby avoiding the stem-clamping transplanter from transplanting weak or injured seedlings during the transplanting process and improving the transplanting quality.

[0052] In this embodiment, the transplanting clamping position includes the optimal position for clamping the seedlings to be transplanted by a transplanting mechanism such as a robotic arm, which can further reduce physical damage to the seedlings during the clamping process.

[0053] In this embodiment, for diseased seedlings and seedlings suitable for transplanting, the method of marking samples includes: marking two key points on healthy seedlings by means of numbers, letters, or symbols.

[0054] In this embodiment, label format alignment means that the formats of the labels corresponding to the diseased seedlings and the labels corresponding to the healthy seedlings are the same, for example, the contents of the labels are both numerical values, symbols or letters.

[0055] In this embodiment, label length alignment means that the data dimensions of the labels corresponding to the diseased seedlings and the labels corresponding to the healthy seedlings are the same. For example, both labels are represented by multiple data points or coordinates, such as label A[a, b, c].

[0056] Specifically, if all the key points on a healthy seedling are marked as 1, the corresponding sample label is expressed as "(1, 1)", indicating that there are two key points "1, 1" on the label; while the key points on the diseased seedling are not marked, the corresponding sample label can be expressed as (null, null); in order to facilitate model calculation, these two types of labels are converted into a unified format; for healthy seedlings, the label conversion can be performed using the labeled json file, and the corresponding data label is expressed as (1, 1, 2). After the conversion, the corresponding sample labels of all diseased seedlings in the label set are (0, 0, 0); among them, the third value of the label indicates whether the label is visible, with 0 indicating that the key point does not exist, 1 indicating that the key point is invisible (occluded), and 2 indicating that the key point is visible.

[0057] In this embodiment, the file marking tool, format conversion type and conversion tool can all be set according to the actual needs of the user.

[0058] It should be noted that if the dimension of the label corresponding to each seedling sample is related to the number of clamping points, that is, when there are 4 clamping points marked on the seedling sample, the corresponding healthy seedling label is represented as (1,1,1,1,2), and the converted diseased seedling label is represented as (0,0,0,0,0), and so on.

[0059] Taking the seedlings to be tested as cabbage plug seedlings as an example, the growth status of cabbage plug seedlings is divided into strong seedlings suitable for transplanting and weak seedlings not suitable for transplanting; for example, strong seedlings suitable for transplanting have the following characteristics: seedlings suitable for transplanting have thick stems, short internode distance, high stem base toughness, compact overall shape, complete and stretched leaves, and dark green leaves. Seedlings suitable for transplanting have strong environmental resistance and adaptability, and they grow quickly after transplanting and have a high survival rate; weak seedlings not suitable for transplanting have the following characteristics: empty holes and weak seedlings, among which, an empty hole is a hole to be transplanted without a seedling, and the corresponding transplanting mechanism does not need to clamp the seedlings; weak seedlings have thin stems, long internode distance, and leaves that may be narrow and short, thin, and light green or dark green. For example, the leaves of the leggy seedlings are thin and light green, while the leaves of the aging seedlings are curled, the leaf area is small, and the leaf color is dark green.

[0060] In this embodiment, the complex sample mining loss function is used to solve the problem of sample imbalance during model training, in which the model over-focuses on simple samples and ignores challenging and difficult-to-distinguish samples. That is, the weight of easy-to-classify samples is reduced, thereby prompting the model to focus more on those difficult-to-distinguish samples during training, so as to improve the accuracy of leafy vegetable seedling identification and transplant survival rate.

[0061] In this embodiment, the detection model is obtained through iterative training or transfer learning. The acquisition method is related to the complexity of the convolutional neural network. For a more complex convolutional neural network, for example, when the convolutional neural network is a YOLOV8 network, the detection model can be obtained through transfer learning; when the convolutional neural network includes a small number of convolutional layers, the detection model can be obtained through iterative training.

[0062] In this embodiment, by designing a multi-task training strategy for cabbage seedling state recognition and key point detection, the detection model can simultaneously learn the feature representations of the two tasks in the same training process, improve the overall performance by sharing the underlying features, and enhance the applicability of the detection model in transplanting operations.

[0063] The leafy seedling recognition and transplanting clamping position detection method provided by the embodiment of the application improves the leafy seedling transplanting efficiency and seedling survival rate.

[0064] Further, the training samples include simple samples and complex samples; the detection model is obtained through the following steps: extracting phenotype features from the sample seedling images based on the convolutional neural network; calculating the Focal loss loss value of each sample seedling image according to the phenotype features based on the Focal loss function; updating the historical loss value of each sample seedling image according to the Focal loss loss value, and determining the historical difficulty data of each sample seedling image; calculating the comprehensive loss of each sample seedling image according to the Focal loss loss value and the historical difficulty data, and sorting the comprehensive loss of each sample seedling image through the non-maximum suppression sorting method to obtain the sorted comprehensive loss; stratifying the sorted comprehensive loss according to the stratification threshold to obtain a plurality of stratification losses; proportionally sampling each stratification loss to obtain a plurality of sampling losses, different stratification losses corresponding to different sampling losses; calculating the average value of each sampling loss, and accumulating the average value to obtain a target loss; and optimizing and training the convolutional network according to the target loss, and obtaining the detection model under the condition of meeting the network convergence.

[0065] In this embodiment, the convolutional neural network is constructed based on a YOLOV8-Pose network and a lightweight backbone network; the number of channels of the bottleneck layer of the lightweight backbone network is adjusted according to the input shape; the input channel number and the output channel number of the transition layer of the lightweight backbone network are the same; the lightweight backbone network adopts a post-activation mode in the training process, and the normalization layer and the convolution layer are merged in the inference stage.

[0066] In this embodiment, considering the limited computing power and memory resources of mobile agricultural equipment in agricultural production environment, a lightweight backbone network is designed based on the YOLOV8-Pose model to reduce the computing cost and model size; at the same time, in view of the current situation that seedlings are small in size and difficult to identify damage, a complex sample mining loss function is introduced in the improved model to solve the problem that the model is not suitable for complex background seedling recognition due to the imbalance of different types of samples in the model training process.

[0067] In this embodiment, the convolutional neural network adopts a bidirectional dense layer structure, and two layers of 3x3 convolution layers are stacked to realize in-depth learning of large target visual features, so as to realize perception field capture under multiple scales.

[0068] In some embodiments, the convolutional neural network includes an improved inverted bottleneck block structure, and the convolutional neural network is provided with multiple channels to expand features; the convolutional neural network also includes a deformable convolutional network, which is used to introduce a learnable offset and adopt a dedicated window mode to non-uniformly sample the input feature map; the convolutional neural network is used to use a dedicated sliding window for each position through an unbounded range of aggregation weights; the shape of the sliding window is independent of the aggregation weight input.

[0069] In this embodiment, in the above-mentioned lightweight backbone network, the number of bottleneck layer channels can be dynamically adjusted according to the input shape to avoid a fixed growth rate and ensure that the number of dense layer channels is reasonable; at the same time, in order to avoid the influence of the compression factor on the feature expression, the transition layer maintains the consistency of the number of input and output channels; the backbone network introduces the classic post-activation mode to improve computing efficiency, and merges the normalization layer and the convolution layer in the inference stage.

[0070] In this embodiment, in order to avoid the impact of model lightweighting on accuracy, the above-mentioned convolutional neural network adopts a shallow wide network architecture with a terminal 1x1 convolutional layer to enhance feature expression capabilities; at the same time, the above-mentioned convolutional neural network performs multi-scale feature extraction by selecting feature maps of at least five different scales.

[0071] It should be noted that in actual transplanting scenarios, cabbage seedlings after hardening have approximately 4-6 true leaves and a leaf span of approximately 6-12 cm. The mutual occlusion between leaves varies, and the background of seedling images in actual production processes is complex. Therefore, accurate detection and identification of transplanted seedlings is required. This is to better address the problem of sample imbalance during model training, which causes the model to over-focus on simple samples and ignore challenging, difficult-to-separate samples.

[0072] Focal Loss can reduce the weight of simple samples, but it does not completely solve the problem of too many simple samples. The traditional LRM method is fixedly layered according to the threshold, which may not be able to adapt to changes in the distribution of the data set, resulting in too many or too few samples in certain layers, affecting the training effect, and may over-focus on high-loss samples and ignore medium-loss samples, which may contain information that is helpful for model improvement. This paper combines Focal Loss and LRM theory to design a new difficult sample mining method based on historical difficulty, as shown below:

[0073] (1) During training, the model backbone first generates a feature map, and then calculates the current Focal Loss value for each sample 𝑖:

[0074] ;

[0075] Among them, FL i is the Focal Loss value of sample 𝑖; αt is a balancing factor for balancing the importance of positive and negative samples; is the model prediction value, and γ is a regulation factor.

[0076] (2) Let L i,history denote the historical loss array of sample i, which stores the loss values of the sample in the latest 10 epochs. According to the FL i Update the historical loss value record of each sample, and then calculate the historical difficulty Hi of sample i as:

[0077] .

[0078] (3) Combine the Focal loss and the historical difficulty to calculate the comprehensive loss of sample i:

[0079] ;

[0080] where λ is a regulation parameter.

[0081] (4) Sort the loss values by non-maximum suppression (NMS):

[0082] ;

[0083] where sort() represents the non-maximum suppression (NMS) sorting method.

[0084] (5) According to all the sorted loss values, use the quantile method to calculate the hierarchical threshold:

[0085] ;

[0086] where k is the total number of layers, qj is the threshold of the jth layer (1≤j≤k). The present application sets k=3; quantile represents the quantile function.

[0087] (6) According to the hierarchical threshold, the detection result loss value is layered:

[0088] ;

[0089] ;

[0090] ;

[0091] Sample from each layer in proportion to form the final sample set S final :

[0092] ;

[0093] where pj Indicates in j Sampling proportion of stratum;| S j | represents a set S j The cardinality is the number of samples in the jth level; the sample() function represents the number of samples from the set according to the sorting result. S j Extract elements; then, the average of the losses of each selected sample is calculated and these averages are added together to form the target loss for optimization training.

[0094] In this example, a five-fold cross-validation approach was used for training on the cabbage dataset to comprehensively evaluate model performance. The batch size was set to 4, the number of iterations was set to 1000, and the stochastic gradient descent (SGD) optimization algorithm was selected, with an initial learning rate of 0.01. These experimental parameters were set to optimize the training process and ensure that the model could stably and effectively learn image features.

[0095] The method for identifying leafy vegetable seedlings and detecting the clamping position of transplanted plants provided by an embodiment of the present invention extracts phenotypic features from sample seedling images, inputs the phenotypic features into a convolutional neural network, calculates the corresponding weights of each training sample according to the phenotypic features based on the Focal loss function, filters out simple samples in the training samples according to the corresponding weights of each sample according to the LRM loss function, calculates the cross entropy loss corresponding to each complex sample, and determines the complex sample mining loss value based on the cross entropy loss value corresponding to the target complex sample. When the complex sample mining loss value is less than the loss threshold or the number of training times of the convolutional neural network reaches a preset number of iterations, a detection model is obtained, thereby improving the efficiency and accuracy of the detection model in identifying and recognizing the clamping position of transplanted seedlings.

[0096] Furthermore, according to a method for identifying leafy vegetable seedlings and detecting the transplanting clamping position provided by the present invention, the detection model is also obtained through the following steps: constructing a plug tray dataset corresponding to the sample seedlings in the transplanting scenario; pre-training the 1st to Nth layers of the convolutional neural network using the ImageNet large dataset based on transfer learning to obtain a pre-trained convolutional neural network; N is a positive integer greater than 2; and fine-tuning the network parameters of the output layer of the pre-trained convolutional neural network according to the plug tray dataset to obtain a detection model.

[0097] In this embodiment, due to the current lack of high-quality cabbage seedling plug tray datasets, a cabbage seedling plug tray dataset during transplanting can be constructed by self-collection, and a transfer learning method can be used to obtain the corresponding detection model.

[0098] For example, the first few layers of the convolutional neural network were pre-trained using the ImageNet large dataset to acquire basic image classification capabilities. Subsequently, while retaining the parameters of these pre-trained layers, the final layer of the convolutional neural network was fine-tuned to better adapt to the characteristics of cabbage seedling transplanting images. This transfer learning method effectively utilizes the versatility of large datasets while improving the model's applicability to small-scale, specific datasets.

[0099] The method for identifying leafy vegetable seedlings and detecting the transplanting clamping position provided by an embodiment of the present invention constructs a plug tray dataset corresponding to sample seedlings in a transplanting scenario, pre-trains the 1st to Nth layers of the convolutional neural network using the ImageNet large dataset through transfer learning, and fine-tunes the network parameters of the output layer of the pre-trained convolutional neural network according to the plug tray dataset to obtain a detection model, thereby reducing the training difficulty of the detection model and further improving the detection efficiency of the seedlings to be transplanted.

[0100] Furthermore, the convolutional neural network includes an attention module based on scale variation;

[0101] The scale-variant attention module is used to extract phenotypic features from the sample seedling images using the following formula:

[0102] ;

[0103] in, A m (x, i) is a phenotypic characteristic; i is the output size of the attention map corresponding to the sample seedling image, f(x, i) represents the average pooling function; associated probability P 1 (x, i) is the association probability, n Indicates the number of output pool tensors; P 1 (x, i) The following conditions must be met:

[0104] and .

[0105] It should be noted that cabbage seedlings have slender stems during transplantation, and some leaves are small, accounting for a small portion of the overall image. Furthermore, leaves between seedlings may overlap, making feature extraction difficult. This invention introduces a novel scale-variation-based attention architecture to enable multi-scale seedling feature recognition. This attention architecture consists of four parts: Monte Carlo attention, cross-scale guidance, scale-transformation attention, and the construction of a Vision Transformer.

[0106] In this embodiment, a scale-change attention network is introduced into the above-mentioned lightweight YOLOV8-Pose model. By combining transformation Monte Carlo attention, scale-change attention and visual transformation, cross-scale features are integrated to enhance the recognition ability of small seedlings.

[0107] Specifically, the scale-varying attention network can be a Monte Carlo attention (MCAttn) module, which uses a pooling strategy based on random sampling to generate a scale-independent attention map, giving the network the ability to capture cross-scale information and significantly improving its accuracy in identifying subtle features of seedlings. The structure is shown in the figure; for example, the MCattn module dynamically constructs an attention map by randomly selecting a 1×1 attention map from three scale pooling tensors: 3×3, 2×2, and 1×1.

[0108] In this embodiment, although the global average pooling in the traditional SE method helps to calibrate the inter-channel dependency, it has limitations in utilizing cross-scale correlation. This embodiment uses the MCAtn module to calculate the attention map by fusing multi-scale features, thereby enhancing the interdependence between long-range semantics.

[0109] The method for identifying leafy vegetable seedlings and detecting the transplanting clamping position provided by an embodiment of the present invention improves the characterization capability of phenotypic characteristics and further improves the detection performance of the detection model by setting a convolutional neural network including an attention module based on scale change and using this module to extract phenotypic features from sample seedling images.

[0110] Furthermore, before extracting phenotypic features from the sample seedling image, the method also includes: performing fine scaling processing on the sample seedling image through a bilinear interpolation algorithm to obtain a processed sample seedling image; and performing data expansion on the processed sample seedling image using image enhancement technology to obtain an expanded sample seedling image.

[0111] Figure 2 This is a schematic diagram of the position of the seedling tray, transplanting mechanism and camera provided by the present invention. Figure 2 In the illustrated embodiment, a machine vision device (installed on the upper and obliquely opposite the conveyor belt transporting seedlings in the tray) consists of an industrial camera and a personal computer (PC); the seedlings in the tray are transported to the seedling removal operation position via the conveyor belt, and the machine vision system is triggered to collect image data. The collected image data is finely scaled using a bilinear interpolation algorithm and uniformly adjusted to a high-definition three-channel format of 550 pixels × 550 pixels, which can optimize the efficiency and accuracy of image analysis.

[0112] In this example, image enhancement technology was used to further enrich the dataset. Through diverse methods such as mirror flipping and brightness and saturation adjustment, the dataset was simulated to simulate the various lighting and posture variations that cabbage seedlings might encounter in actual production. The processed images were intelligently divided into five equal parts, and training and test sets were constructed using a 4:1 split. Finally, a 50-fold cross-validation strategy was used to train and optimize the deep learning model, ensuring high recognition accuracy and stability in complex and changing growing environments.

[0113] The method for identifying leafy vegetable seedlings and detecting the transplanting clamping position provided by an embodiment of the present invention performs fine scaling processing on the sample seedling image through a bilinear interpolation algorithm, and uses image enhancement technology to expand the data of the processed sample seedling image to obtain an expanded sample seedling image, thereby improving the sample quality and quantity, and thereby improving the detection accuracy of the detection model.

[0114] Figure 3 It is a structural diagram of the convolutional neural network provided by the present invention. Figure 3 In the illustrated embodiment, the convolutional neural network includes 6 Dynamic Bottleneck_Blocks (bottleneck layers) and 3 Dynamic FPAN layers. As shown in the figure, the first Dynamic Bottleneck_Block is connected to the input layer (Input) through a Dynamic StemBlock, and the second Dynamic Bottleneck_Block is connected to the first Dynamic FPAN layer. The input image is processed by Dynamic StemBlock to extract primary features. Dynamic Bottleneck_Block adjusts the number of primary feature channels, enhances the diversity and robustness of the features through SPPF, constructs an attention map through the MCAtn module, and further improves the feature representation capability. The attention map is upsampled through the Upsample layer, and then the Dynamic FPAN layer is used to further extract effective feature information. The input feature map is dynamically convolved through Dynamic Conv, and seedling type recognition and clamping position detection are performed based on the calculated feature information.

[0115] The leafy vegetable seedling identification and transplanting clamping position detection device provided by the present invention is described below. The leafy vegetable seedling identification and transplanting clamping position detection device described below and the leafy vegetable seedling identification and transplanting clamping position detection method described above can be referenced to each other.

[0116] Figure 4 Schematic diagram of the structure of the leafy vegetable seedling identification and transplanting clamping position detection device provided by the present invention, as shown in FIG. Figure 4As shown, the leafy vegetable seedling identification and transplant clamping position detection device includes: an image acquisition module 410 and a detection module 420.

[0117] An image acquisition module 410 is used to acquire an image of the seedling to be tested;

[0118] A detection module 420 is configured to perform seedling recognition and transplant clamping position detection on the seedling image to be tested based on a detection model, and obtain a seedling recognition result and a transplant clamping position detection result;

[0119] Among them, the detection model is based on sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels, and the convolutional neural network is trained with a complex sample mining loss function; the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type respectively; the complex sample mining loss function is determined based on the Focal loss function, the loss value ranking theory and the sample stratification method function.

[0120] The leafy vegetable seedling identification and transplanting clamping position detection device provided by an embodiment of the present invention uses diseased seedling types and healthy seedling types as sample labels, and a detection model obtained by training a convolutional neural network with a complex sample mining loss function to perform seedling identification and transplanting clamping position detection on the seedling images to be tested, thereby obtaining seedling identification results and transplanting clamping position detection results, thereby improving the transplanting efficiency and survival rate of leafy vegetable seedlings.

[0121] Figure 5 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a method for leafy vegetable seedling identification and transplant clamping position detection, which includes: obtaining an image of a seedling to be tested; performing seedling identification and transplant clamping position detection on the image of the seedling to be tested based on a detection model to obtain a seedling identification result and a transplant clamping position detection result; wherein the detection model is based on training a convolutional neural network with sample seedling images as training samples and diseased seedling types and healthy seedling types as sample labels using a complex sample mining loss function; wherein the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type; the complex sample mining loss function is determined based on the Focal loss function, the loss value sorting theory and the sample stratification method.

[0122] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the leafy vegetable seedling identification and transplant clamping position detection methods provided by the above methods, the method including: obtaining an image of a seedling to be tested; performing seedling identification and transplant clamping position detection on the image of the seedling to be tested based on a detection model to obtain seedling identification results and transplant clamping position detection results; wherein the detection model is based on training a convolutional neural network with sample seedling images as training samples and diseased seedling types and healthy seedling types as sample labels using a complex sample mining loss function; wherein the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type; the complex sample mining loss function is determined based on the Focal loss function, the loss value ranking theory and the sample stratification method.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the leafy vegetable seedling identification and transplanting clamping position detection method provided by the above-mentioned methods, the method comprising: obtaining an image of a seedling to be tested; performing seedling identification and transplanting clamping position detection on the image of the seedling to be tested based on a detection model to obtain seedling identification results and transplanting clamping position detection results; wherein the detection model is based on training a convolutional neural network using sample seedling images as training samples and diseased seedling types and healthy seedling types as sample labels using a complex sample mining loss function; wherein the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type; the complex sample mining loss function is determined based on the Focal loss function, the loss value ranking theory and the sample stratification method.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0126] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for identifying leafy vegetable seedlings and detecting their transplanting clamping positions, characterized in that: include: Acquire images of seedlings to be tested; performing seedling recognition and transplanting clamping position detection on the seedling image to be tested based on the detection model to obtain a seedling recognition result and a transplanting clamping position detection result; The detection model is based on sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels, and a convolutional neural network trained using a complex sample mining loss function. The sample labels corresponding to the healthy seedling type are annotated with two clamping position key points, and the formats and data lengths of the sample labels corresponding to the diseased seedling type and the sample labels corresponding to the healthy seedling type are aligned respectively. The complex sample mining loss function is determined based on the focal loss function, loss value ranking theory, and sample stratification method. Label format alignment means that the labels corresponding to diseased seedlings and healthy seedlings have the same format; label length alignment means that the data dimensions of the labels corresponding to diseased seedlings and healthy seedlings are the same; The convolutional neural network includes an improved inverted bottleneck block structure, and the convolutional neural network is provided with multiple channels to expand features; The convolutional neural network also includes a deformable convolutional network, which is used to introduce a learnable offset and adopt a dedicated window mode to non-uniformly sample the input feature map; The convolutional neural network is configured to use a dedicated sliding window for each position with an unbounded range of aggregation weights; the shape of the sliding window is independent of the aggregation weight input; The convolutional neural network is built based on the YOLOV8-Pose network and a lightweight backbone network. The number of channels in the bottleneck layer of the lightweight backbone network is adjusted according to the input shape. The number of input channels and output channels of the transition layer of the lightweight backbone network are the same. The lightweight backbone network adopts the post-activation mode during training and combines the normalization layer with the convolution layer during the inference phase. The detection model is obtained through the following steps: extracting phenotypic features from sample seedling images based on the convolutional neural network; Calculating a Focal loss value of each sample seedling image according to the phenotypic characteristics based on a Focal loss function; and updating a historical loss value of each sample seedling image according to the Focal loss value to determine historical difficulty data of each sample seedling image; The comprehensive loss of each sample seedling image is calculated according to the focal loss value and the historical difficulty data, and the comprehensive loss of each sample seedling image is sorted by a non-maximum suppression sorting method to obtain the sorted comprehensive loss.

2. The method for identifying and detecting the position of leafy vegetable seedlings for transplanting according to claim 1, characterized in that: The detection model is also obtained through the following steps: The sorted comprehensive losses are stratified according to a stratification threshold to obtain a plurality of stratified losses; each stratified loss is sampled proportionally to obtain a plurality of sampling losses, where different stratified losses correspond to different sampling losses; Calculate the average of each sample loss and accumulate the averages to obtain the target loss; The convolutional network is optimized and trained according to the target loss, and the detection model is obtained under the condition that the network converges.

3. The method for identifying and detecting the position of leafy vegetable seedlings for transplanting according to claim 1, characterized in that: The detection model is also obtained through the following steps: Construct a plug tray dataset corresponding to the sample seedlings in the transplanting scenario; Pre-training the 1st to Nth layers of the convolutional neural network using the ImageNet large dataset based on transfer learning to obtain a pre-trained convolutional neural network; N is a positive integer greater than 2; The network parameters of the output layer of the pre-trained convolutional neural network are fine-tuned according to the hole tray data set to obtain the detection model.

4. The method for identifying leafy vegetable seedlings and detecting the position of the transplanting clamp according to any one of claims 1 to 3, characterized in that: The convolutional neural network includes an attention module based on scale change; The scale-variation-based attention module is used to extract the phenotypic features from the sample seedling image using the following formula: Among them, A m (x,i) is the phenotypic feature; x is the network input tensor; i is the output size of the attention map corresponding to the sample seedling image, f(x,i) represents the average pooling function; P1(x,i) is the association probability, n represents the number of output pool tensors; P1(x,i) satisfies the following conditions: and 5. The method for identifying leafy vegetable seedlings and detecting the position of the transplanting clamp according to claim 2, characterized in that: Before extracting the phenotypic characteristics from the sample seedling image, the method further includes: performing fine scaling processing on the sample seedling image by a bilinear interpolation algorithm to obtain a processed sample seedling image; Image enhancement technology is used to perform data expansion on the processed sample seedling image to obtain an expanded sample seedling image.

6. A leafy vegetable seedling identification and transplanting clamping position detection device, using the leafy vegetable seedling identification and transplanting clamping position detection method according to claim 1, characterized in that: include: An image acquisition module is used to acquire images of seedlings to be tested; a detection module, configured to perform seedling recognition and transplant clamping position detection on the seedling image to be tested based on a detection model, and obtain a seedling recognition result and a transplant clamping position detection result; The detection model is based on sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels, and a convolutional neural network trained with a complex sample mining loss function; the sample labels corresponding to the healthy seedling type are marked with two clamping position key points, the format and data length of the sample labels corresponding to the diseased seedling type are aligned with those of the sample labels corresponding to the healthy seedling type, and the numerical value of the sample label corresponding to the diseased seedling type after label conversion is different from the numerical value of the sample label corresponding to the healthy seedling type; The complex sample mining loss function is determined based on the Focal loss function, loss value ranking theory and sample stratification method function.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of claims 1 to 5 is implemented.

Images