Leaf vegetable seedling identification and transplanting clamping position detection method and device

Abstract

The invention provides a leaf vegetable seedling identification and transplanting clamping position detection method and device. The method comprises the following steps: acquiring a to-be-detected seedling image; performing seedling identification and transplanting clamping position detection on the to-be-detected seedling image based on the detection model to obtain a seedling identification result and a transplanting clamping position detection result; the detection model is obtained by taking a sample seedling image as a training sample, taking a disease seedling type and a healthy seedling type as sample labels, and training a convolutional neural network by using a complex sample mining loss function; the formats and the data lengths of the sample labels corresponding to the disease seedling types and the sample labels corresponding to the healthy seedling types are respectively aligned; the complex sample mining loss function is determined based on a Focal loss function, a loss value sorting theory and a sample layering method. According to the method, the leaf vegetable seedling transplanting efficiency and the seedling survival rate are improved.

Description

Technical Field

[0001] The invention relates to the field of crop identification and image processing, and in particular to a method and a device for detecting leafy vegetable seedling identification and transplanting clamping positions. Background Art

[0002] Vegetable transplanting is a key link in vegetable production, and the quality of transplanting will directly affect crop yield. Due to the large leaf area of ​​leafy vegetable seedlings (such as cabbage seedlings), the leaves and stems are easily damaged during the mechanized seedling transplanting process. According to statistics, the current damage rate of mechanized transplanting of seedlings (also known as seedlings) is about 7%-10%. In order to solve the problem of high damage rate in cabbage seedling transplanting, it is necessary to study the mechanized transplanting process of cabbage seedlings and other crops to reduce the seedling damage rate.

[0003] At present, the seedling retrieval device of the vegetable transplanter is mainly divided into insertion clamping type, ejection type and ejection clamping type, stem clamping type and seedling retrieval mechanism for non-standard seedling trays according to the different clamping methods; the stem clamping seedling retrieval mechanism can reduce the damage to the seedling body and its root system, and its seedling retrieval operation is simple, especially suitable for potted seedlings with thicker stems, the seedling retrieval effect is significant and the damage to the potted seedlings is small, and it is widely used in actual production.

[0004] In the related art, key image information of an object is usually extracted, and the movement of a robot is controlled according to the image information to study the transplanting process of seedlings of legumes, fruits and vegetables, and flowers. When the target of seedling transplanting is leafy vegetable seedlings such as cabbage, it is necessary to select a suitable clamping position to protect the seedling leaves or rhizomes from damage during transplanting, otherwise it will affect the quality of the crop. Moreover, the existing technology manually selects strong seedlings from the seedlings during transplanting to improve the survival rate of cabbage. The efficiency of manual screening is low, and it is easy to damage the seedlings due to improper operation, resulting in economic losses. Summary of the invention

[0005] The present invention provides a leafy vegetable seedling identification and transplant clamping position detection method and device, which are used to solve the defects of the prior art that leafy vegetable seedlings are easily damaged when transplanting and the transplanting efficiency is low, thereby improving the transplanting efficiency and survival rate of leafy vegetable seedlings.

[0006] The present invention provides a leafy vegetable seedling identification and transplanting clamping position detection method, comprising: Acquire the image of the seedling to be tested; Based on the detection model, seedling recognition and transplanting clamping position detection are performed on the seedling image to be tested to obtain a seedling recognition result and a transplanting clamping position detection result; Among them, the detection model is based on the sample seedling images as training samples, the diseased seedling types and healthy seedling types as sample labels, and the convolutional neural network is trained with a complex sample mining loss function; the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type respectively; the complex sample mining loss function is determined based on the Focal loss function, the loss value ranking theory and the sample stratification method.

[0007] According to a leafy vegetable seedling identification and transplanting clamping position detection method provided by the present invention, the detection model is obtained through the following steps: Extracting the phenotypic features from the sample seedling image based on the convolutional neural network; Calculate the Focal loss value of each sample seedling image according to the phenotypic characteristics based on the Focal loss function; and update the historical loss value of each sample seedling image according to the Focal loss value to determine the historical difficulty data of each sample seedling image; Calculating the comprehensive loss of each sample seedling image according to the focal loss value and the historical difficulty data, and sorting the comprehensive losses of each sample seedling image by a non-maximum suppression sorting method to obtain a sorted comprehensive loss; The sorted comprehensive losses are stratified according to a stratification threshold to obtain a plurality of stratified losses, and each stratified loss is sampled in proportion to obtain a plurality of sampling losses, wherein different stratified losses correspond to different sampling losses; Calculate the average value of each sample loss, and add up the average values ​​to obtain the target loss; The convolutional network is optimized and trained according to the target loss, and the detection model is obtained under the condition that the network converges.

[0008] According to a leafy vegetable seedling identification and transplanting clamping position detection method provided by the present invention, the detection model is also obtained through the following steps: Construct a plug tray dataset corresponding to the sample seedlings in the transplanting scenario; Based on transfer learning, the 1st to Nth layers of the convolutional neural network are pre-trained using the ImageNet large data set to obtain a pre-trained convolutional neural network; N is a positive integer greater than 2; The network parameters of the output layer of the pre-trained convolutional neural network are fine-tuned according to the hole tray data set to obtain the detection model.

[0009] According to a method for identifying leafy vegetable seedlings and detecting transplant clamping positions, the convolutional neural network includes an improved inverted bottleneck block structure, and the convolutional neural network is provided with multiple channels to expand features; the convolutional neural network also includes a deformable convolutional network, and the deformable convolutional network is used to introduce a learnable offset and adopt a dedicated window mode to perform non-uniform sampling on the input feature map; the convolutional neural network is used to use a dedicated sliding window for each position through an unbounded range of aggregation weights; the shape of the sliding window is independent of the aggregation weight input.

[0010] According to a leafy vegetable seedling identification and transplant clamping position detection method provided by the present invention, the convolutional neural network includes an attention module based on scale change; The scale-variation-based attention module is used to extract the phenotypic features from the sample seedling image by the following formula: ; in, A m (x, i) is the phenotypic feature; x is the network input tensor; i is the output size of the attention map corresponding to the sample seedling image, f(x, i) represents the average pooling function; P 1 (x, i) is the association probability, n Indicates the number of output pool tensors; P 1 (x, i) The following conditions must be met: and .

[0011] According to a leafy vegetable seedling identification and transplant clamping position detection method provided by the present invention, before extracting the phenotypic characteristics from the sample seedling image, the method further includes: Performing fine scaling processing on the sample seedling image by using a bilinear interpolation algorithm to obtain a processed sample seedling image; The image enhancement technology is used to perform data expansion on the processed sample seedling image to obtain an expanded sample seedling image.

[0012] The present invention also provides a leafy vegetable seedling identification and transplanting clamping position detection device, comprising: An image acquisition module, used for acquiring images of seedlings to be tested; A detection module, used to perform seedling recognition and transplant clamping position detection on the seedling image to be detected based on the detection model, and obtain a seedling recognition result and a transplant clamping position detection result; Among them, the detection model is based on the sample seedling images as training samples, the diseased seedling types and healthy seedling types as sample labels, and the convolutional neural network is trained with a complex sample mining loss function; the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type respectively; the complex sample mining loss function is determined based on the Focal loss function, the loss value ranking theory and the sample stratification method.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of the above methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of the above methods is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the leafy vegetable seedling identification and transplant clamping position detection methods described above.

[0016] The method for leafy vegetable seedling identification and transplanting clamp position detection provided by the present invention uses diseased seedling types and healthy seedling types as sample labels, and uses a complex sample mining loss function to train a convolutional neural network to obtain a detection model, which performs seedling identification and transplanting clamp position detection on the seedling image to be tested, obtains seedling identification results and transplanting clamp position detection results, and improves the transplanting efficiency and survival rate of leafy vegetable seedlings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic flow chart of the leafy vegetable seedling identification and transplanting clamping position detection method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the position of the plug tray seedlings, transplanting mechanism and camera provided by the present invention.

[0020] Figure 3It is a schematic diagram of the structure of the convolutional neural network provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the leafy vegetable seedling identification and transplanting clamping position detection device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Combine the following Figure 1-Figure 4 The invention describes a method and a device for detecting leafy vegetable seedling identification and transplanting clamping positions.

[0025] Figure 1 : is a flow chart of the leafy vegetable seedling identification and transplanting clamping position detection method provided by the present invention, such as Figure 1 As shown, the leafy vegetable seedling identification and transplanting clamping position detection method comprises the following steps: Step 110: Acquire the image of the seedling to be tested.

[0026] In this step, the leafy vegetable seedlings include leafy vegetable seedlings such as cabbage, spinach, lettuce or chrysanthemum.

[0027] In this embodiment, the image of the seedling to be tested can be obtained from a public vegetable database, or can be captured by a camera or other device.

[0028] Step 120, based on the detection model, the seedling images to be tested are subjected to seedling recognition and transplant clamping position detection to obtain seedling recognition results and transplant clamping position detection results; wherein, the detection model is based on the sample seedling images as training samples, the diseased seedling types and the healthy seedling types as sample labels, and the convolutional neural network is trained using a complex sample mining loss function; wherein, the sample label corresponding to the healthy seedling type is annotated with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type; the complex sample mining loss function is determined based on the Focal loss function, the loss value sorting theory and the sample stratification method.

[0029] In this step, the detection model can automatically determine whether the growth status of the seedlings to be tested meets the transplanting standards, and detect the appropriate clamping position of the transplanting mechanism for the seedlings, thereby avoiding the clamping stem transplanter from transplanting weak or injured seedlings during the transplanting process and improving the transplanting quality.

[0030] In this embodiment, the transplanting clamping position includes the optimal position for clamping the seedlings to be transplanted by a transplanting mechanism such as a robotic arm, which can further reduce physical damage to the seedlings during the clamping process.

[0031] In this embodiment, for the diseased seedlings and the seedlings suitable for transplanting, the method of marking samples includes: marking two key points on the healthy seedlings by means of numbers, letters or symbols.

[0032] In this embodiment, label format alignment means that the formats of labels corresponding to diseased seedlings and labels corresponding to healthy seedlings are the same, for example, the contents of the labels are both numerical values, symbols or letters.

[0033] In this embodiment, label length alignment means that the data dimensions of the labels corresponding to the diseased seedlings and the labels corresponding to the healthy seedlings are the same. For example, both labels are represented by multiple data points or coordinates, such as label A[a, b, c].

[0034] Specifically, if all the key points on a healthy seedling are marked as 1, the corresponding sample label is expressed as "(1, 1)", indicating that there are two key points "1, 1" on the label; while the key points on the diseased seedling are not marked, the corresponding sample label can be expressed as (null, null); in order to facilitate model calculation, these two types of labels are converted in a unified format; for healthy seedlings, the label conversion can be performed using the annotation json file, and the corresponding data label is expressed as (1, 1, 2). After the conversion, the corresponding sample labels of all diseased seedlings in the label set are (0, 0, 0); among them, the third value of the label indicates whether the label is visible, with 0 indicating that the key point does not exist, 1 indicating that the key point is invisible (occluded), and 2 indicating that the key point is visible.

[0035] In this embodiment, the file marking tool, format conversion type and conversion tool can all be set according to actual needs of the user.

[0036] It should be noted that if the dimension of the label corresponding to each seedling sample is related to the number of clamping points, that is, when there are 4 clamping points marked on the seedling sample, the corresponding healthy seedling label is expressed as (1,1,1,1,2), and the converted diseased seedling label is expressed as (0,0,0,0,0), and so on.

[0037] Taking the seedlings to be tested as cabbage plug seedlings as an example, the growth status of cabbage plug seedlings is divided into strong seedlings suitable for transplanting and weak seedlings not suitable for transplanting; for example, strong seedlings suitable for transplanting have the following characteristics: seedlings suitable for transplanting have thick stems, short internode distance, high toughness of stem base, compact overall shape, complete and stretched leaves and dark green leaves, etc., seedlings suitable for transplanting have strong environmental resistance and adaptability, and the seedlings grow quickly after transplanting and have a high survival rate; while weak seedlings not suitable for transplanting have the following characteristics: empty holes and weak seedlings, among which, an empty hole is a hole to be transplanted without a seedling, and the corresponding transplanting mechanism does not need to clamp the seedlings; weak seedlings have thin stems and long internode distance, and the leaves may be narrow and short, thin, and light green or dark green, such as the thin leaves and light green leaves of the leggy seedlings, while the leaves of the aging seedlings are curled, the leaf area is small and the leaf color is dark green, etc.

[0038] In this embodiment, the complex sample mining loss function is used to solve the problem of excessive focus on simple samples and neglect of challenging hard-to-distinguish samples caused by sample imbalance during model training. That is, the weight of easy-to-classify samples is reduced, thereby prompting the model to focus more on those difficult-to-distinguish samples during training, so as to improve the accuracy of leafy vegetable seedling identification and transplant survival rate.

[0039] In this embodiment, the detection model is obtained by iterative training or transfer learning. The acquisition method is related to the complexity of the convolutional neural network. For a more complex convolutional neural network, for example, when the convolutional neural network is a YOLOV8 network, the detection model can be obtained by transfer learning; when the convolutional neural network includes a small number of convolutional layers, the detection model can be obtained by iterative training.

[0040] In this embodiment, by designing a multi-task training strategy for cabbage seedling state recognition and key point detection, the detection model can simultaneously learn the feature representations of the two tasks in the same training process, improve the overall performance by sharing the underlying features, and improve the applicability of the detection model in transplanting operations.

[0041] The method for leafy vegetable seedling identification and transplanting clamp position detection provided by an embodiment of the present invention uses diseased seedling types and healthy seedling types as sample labels, and uses a complex sample mining loss function to train a convolutional neural network to obtain a detection model, which performs seedling identification and transplanting clamp position detection on the seedling images to be tested, thereby obtaining seedling identification results and transplanting clamp position detection results, thereby improving the transplanting efficiency and survival rate of leafy vegetable seedlings.

[0042] Furthermore, the training samples include simple samples and complex samples; the detection model is obtained through the following steps: extracting phenotypic features from sample seedling images based on a convolutional neural network; calculating the Focal loss value of each sample seedling image according to the phenotypic features based on the Focal loss function; and updating the historical loss value of each sample seedling image according to the Focal loss value, and determining the historical difficulty data of each sample seedling image; calculating the comprehensive loss of each sample seedling image according to the Focal loss value and the historical difficulty data, and sorting the comprehensive loss of each sample seedling image by a non-maximum suppression sorting method to obtain the sorted comprehensive loss; stratifying the sorted comprehensive loss according to the stratification threshold to obtain multiple stratification losses; sampling each stratification loss proportionally to obtain multiple sampling losses, and different stratification losses correspond to different sampling losses; calculating the average value of each sampling loss, and accumulating the average value to obtain the target loss; optimizing the convolutional network according to the target loss, and obtaining the detection model under the condition that the network converges.

[0043] In this embodiment, a convolutional neural network is constructed based on a YOLOV8-Pose network and a lightweight backbone network; the number of channels of a bottleneck layer of the lightweight backbone network is adjusted according to an input shape; the number of input channels of a transition layer of the lightweight backbone network is connected to the number of output channels; the lightweight backbone network adopts a post-activation mode during training, and merges a normalization layer with a convolution layer in the inference stage.

[0044] In this embodiment, taking into account the limited computing power and memory resources of mobile agricultural machinery intelligent equipment in the agricultural production environment, this embodiment designs a lightweight backbone network based on the YOLOV8-Pose model to reduce computing costs and model size; at the same time, in view of the current situation that seedlings are small in size and damage is difficult to identify, a complex sample mining loss function is introduced into the improved model to solve the problem that the model is not suitable for complex background seedling recognition due to the imbalance in the number of different types of samples during model training.

[0045] In this embodiment, the convolutional neural network adopts a bidirectional dense layered structure, and by stacking two 3x3 convolutional layers, it can achieve in-depth learning of the visual features of larger targets, thereby achieving perceptual field capture at multiple scales.

[0046] In some embodiments, the convolutional neural network includes an improved inverted bottleneck block structure, and the convolutional neural network is provided with multiple channels to expand features; the convolutional neural network also includes a deformable convolutional network, and the deformable convolutional network is used to introduce a learnable offset and adopt a dedicated window mode to non-uniformly sample the input feature map; the convolutional neural network is used to use a dedicated sliding window for each position through an unbounded range of aggregation weights; the shape of the sliding window is independent of the aggregation weight input.

[0047] In this embodiment, in the above-mentioned lightweight backbone network, the number of bottleneck layer channels can be dynamically adjusted according to the input shape to avoid a fixed growth rate and ensure that the number of dense layer channels is reasonable; at the same time, in order to avoid the influence of the compression factor on the feature expression, the transition layer maintains the consistency of the number of input and output channels; the backbone network introduces the classic post-activation mode to improve computing efficiency, and merges the normalization layer and the convolution layer in the inference stage.

[0048] In this embodiment, in order to avoid the influence of model lightweighting on accuracy, the above-mentioned convolutional neural network adopts a shallow wide network architecture with a terminal 1x1 convolutional layer to enhance the feature expression capability; at the same time, the above-mentioned convolutional neural network performs multi-scale feature extraction by selecting feature maps of at least five different scales.

[0049] It should be noted that in the actual transplanting scenario, since the cabbage seedlings after hardening have about 4-6 true leaves and a leaf span of about 6-12 cm, the mutual occlusion between leaves is different, and the background in the seedling images in the actual production process is complex, it is necessary to accurately detect and identify the seedlings to be transplanted; in order to better solve the problem of sample imbalance during model training, which causes the model to over-focus on simple samples and ignore challenging samples.

[0050] Focal Loss can reduce the weight of simple samples, but it does not completely solve the problem of too many simple samples. The traditional LRM method is fixedly layered according to the threshold, which may not adapt to the changes in the distribution of the data set, resulting in too many or too few samples at certain levels, affecting the training effect, and may over-focus on high-loss samples and ignore medium-loss samples, which may contain information that is helpful for model improvement. The present invention combines Focal Loss and LRM theory to design a new difficult sample mining method based on historical difficulty, as shown below: (1) During training, the model backbone first generates a feature map, and then calculates the current Focal Loss value for each sample 𝑖: ; Among them, FL i is the Focal Loss value of sample 𝑖; α t is a balancing factor used to balance the importance of positive and negative samples; is the model prediction value, and 𝛾 is the control factor.

[0051] (2) Let L i,history Represents the historical loss array of sample i, storing the loss values ​​of samples in the latest 10 epochs. i Update the historical loss value record of each sample, and then calculate the historical difficulty Hi of sample i as: .

[0052] (3) Combining Focal loss and historical difficulty, calculate the comprehensive loss of sample i: ; Among them, λ is the control parameter.

[0053] (4) Sorting loss values ​​by non-maximum suppression (NMS): ; Among them, sort() represents the non-maximum suppression (NMS) sorting method.

[0054] (5) According to all the sorted loss values, the stratification threshold is calculated using the quantitative quantile method: ; Wherein, k is the total number of layers, qj is the threshold value of the jth layer (1≤j≤k). The present invention sets k=3; quantile represents the quantile function.

[0055] (6) According to the stratification threshold, the detection result loss value is stratified: ; ; ; Sampling from each stratum in proportion to form the final sample set S final : ; in, p j Indicated in j Sampling proportion of stratum; | S j | represents a collection S j The cardinality is the number of samples in the jth level; the sample() function represents the sorting result from the set S j Extract elements; then, the average of the loss of each selected sample is calculated and these averages are added together to form the target loss for optimization training.

[0056] In this embodiment, a five-fold cross validation method is used for training the cabbage dataset to comprehensively evaluate the model performance. In the experiment, the batch size is set to 4, the number of iterations is set to 1000, and the stochastic gradient descent (SGD) optimization algorithm is selected, and the initial learning rate is set to 0.01. The setting of these experimental parameters is intended to optimize the training process and ensure that the model can stably and effectively learn image features.

[0057] The leafy vegetable seedling identification and transplant clamping position detection method provided by the embodiment of the present invention extracts phenotypic features from the sample seedling image, inputs the phenotypic features into the convolutional neural network, calculates the corresponding weights of each training sample according to the phenotypic features based on the Focal loss function, filters out simple samples in the training samples according to the corresponding weights of each sample according to the LRM loss function, calculates the cross entropy loss corresponding to each complex sample, and determines the complex sample mining loss value according to the cross entropy loss value corresponding to the target complex sample. When the complex sample mining loss value is less than the loss threshold or the number of training times of the convolutional neural network reaches a preset number of iterations, a detection model is obtained, thereby improving the efficiency and accuracy of the detection model in identifying the transplanted seedlings and the clamping position.

[0058] Furthermore, according to a leafy vegetable seedling identification and transplant clamping position detection method provided by the present invention, the detection model is also obtained through the following steps: constructing a plug tray data set corresponding to the sample seedlings in the transplanting scenario; pre-training the 1st to Nth layers of the convolutional neural network using the ImageNet large data set based on transfer learning to obtain a pre-trained convolutional neural network; N is a positive integer greater than 2; and fine-tuning the network parameters of the output layer of the pre-trained convolutional neural network according to the plug tray data set to obtain a detection model.

[0059] In this embodiment, due to the lack of high-quality cabbage seedling plug tray datasets, a cabbage seedling plug tray dataset during transplanting can be constructed by self-collection, and a transfer learning method can be used to obtain a corresponding detection model.

[0060] For example, the first few layers of the convolutional neural network are pre-trained using the ImageNet large dataset to acquire basic image classification capabilities. Subsequently, while retaining the parameters of these pre-trained layers, the last layer of the convolutional neural network is fine-tuned to better adapt to the characteristics of the cabbage seedling transplanting images. This transfer learning method effectively utilizes the versatility of large datasets while improving the applicability of the model on small-scale specific datasets.

[0061] The method for identifying leafy vegetable seedlings and detecting the transplanting clamping position provided by an embodiment of the present invention constructs a plug tray dataset corresponding to sample seedlings in a transplanting scenario, pre-trains the 1st to Nth layers of the convolutional neural network using the ImageNet large dataset through transfer learning, and fine-tunes the network parameters of the output layer of the pre-trained convolutional neural network according to the plug tray dataset to obtain a detection model, thereby reducing the training difficulty of the detection model and further improving the detection efficiency of the seedlings to be transplanted.

[0062] Furthermore, the convolutional neural network includes a scale-variant based attention module; The scale-variant attention module is used to extract phenotypic features from the sample seedling images using the following formula: ; in, A m (x, i) For phenotypic characteristics; i is the output size of the attention map corresponding to the sample seedling image, f(x, i) represents the average pooling function; associated probability P 1 (x, i) is the association probability, n Indicates the number of output pool tensors; P 1 (x, i) The following conditions must be met: and .

[0063] It should be noted that the cabbage seedlings have thin stems when transplanted, and some leaves are small in area, accounting for a small area in the entire image, and there may be overlap between the leaves of the seedlings, making feature extraction difficult. This invention introduces a novel attention architecture based on scale changes to achieve the recognition of multi-scale seedling features. The attention architecture consists of 4 parts, namely Monte Carlo attention, cross-scale guidance, scale transformation attention, and Vision Transformer construction.

[0064] In this embodiment, a scale-change attention network is introduced into the above-mentioned lightweight YOLOV8-Pose model. By combining transformation Monte Carlo attention, scale-change attention and visual transformation components, cross-scale features are integrated to enhance the recognition ability of small seedlings.

[0065] Specifically, the scale-varying attention network can be a Monte Carlo attention (MCAttn) module, which uses a pooling strategy based on random sampling to generate a scale-independent attention map, giving the network the ability to capture cross-scale information, significantly improving its accuracy in identifying subtle features of seedlings. The structure is shown in the figure; for example, the MCattn module dynamically constructs an attention map by randomly selecting a 1×1 attention map from three scale pooling tensors: 3×3, 2×2, and 1×1.

[0066] In this embodiment, although the global average pooling in the traditional SE method helps to calibrate the inter-channel dependency, it has limitations in utilizing cross-scale correlations. This embodiment uses the MCAtn module to calculate the attention map by fusing multi-scale features, thereby enhancing the interdependence between long-distance semantics.

[0067] The method for identifying leafy vegetable seedlings and detecting the transplanting clamping position provided by an embodiment of the present invention improves the characterization capability of phenotypic characteristics and further improves the detection performance of the detection model by setting a convolutional neural network including an attention module based on scale changes and utilizing the module to extract phenotypic characteristics from sample seedling images.

[0068] Furthermore, before extracting phenotypic features from the sample seedling image, the method also includes: performing fine scaling processing on the sample seedling image through a bilinear interpolation algorithm to obtain a processed sample seedling image; and performing data expansion on the processed sample seedling image using image enhancement technology to obtain an expanded sample seedling image.

[0069] Figure 2 The present invention provides a schematic diagram of the location of the seedling tray, the transplanting mechanism and the camera. Figure 2 In the embodiment shown, a machine vision device (installed at the machine crossbeam obliquely above the conveyor belt for transporting seedlings in the tray, the machine vision device is composed of an industrial camera and a personal computer (PC); the seedlings in the tray are transported to the seedling removal operation position through the conveyor belt, and the machine vision system is triggered to collect image data; the collected image data is finely scaled by a bilinear interpolation algorithm and uniformly adjusted to a high-definition three-channel format of 550 pixels × 550 pixels, which can optimize the efficiency and accuracy of image analysis.

[0070] In this embodiment, image enhancement technology is used to further enrich the data set, and various lighting and posture changes that cabbage seedlings may encounter in actual production are simulated through various means such as mirror flipping, brightness and saturation adjustment. The processed images are intelligently divided into five equal parts, and the training set and test set are constructed according to the 4:1 distribution principle. Finally, a five-fold cross-validation strategy is used to train and optimize the deep learning model to ensure that it can still maintain a high degree of recognition accuracy and stability in a complex and changeable growth environment.

[0071] The leafy vegetable seedling identification and transplant clamping position detection method provided by the embodiment of the present invention performs fine scaling processing on the sample seedling image through a bilinear interpolation algorithm, and uses image enhancement technology to perform data expansion on the processed sample seedling image to obtain an expanded sample seedling image, thereby improving the sample quality and quantity, and thereby improving the detection accuracy of the detection model.

[0072] Figure 3 It is a schematic diagram of the structure of the convolutional neural network provided by the present invention. Figure 3In the illustrated embodiment, the convolutional neural network includes 6 Dynamic Bottleneck_Blocks (bottleneck layers) and 3 Dynamic FPAN layers. The first Dynamic Bottleneck_Block is connected to the input layer (Input) through a Dynamic StemBlock as shown in the figure, and the second Dynamic Bottleneck_Block is connected to the first Dynamic FPAN layer as shown in the figure. The input image is processed by Dynamic StemBlock to extract primary features. Dynamic Bottleneck_Block adjusts the number of primary feature channels, enhances the diversity and robustness of the features through SPPF, constructs an attention map through the MCattn module, further improves the feature representation capability, upsamples the attention map through the Upsample layer, and further extracts effective feature information through the Dynamic FPAN layer, and performs dynamic convolution calculation on the input feature map through Dynamic Conv, and performs seedling type recognition and clamping position detection based on the calculated feature information.

[0073] The leafy vegetable seedling identification and transplanting clamping position detection device provided by the present invention is described below. The leafy vegetable seedling identification and transplanting clamping position detection device described below and the leafy vegetable seedling identification and transplanting clamping position detection method described above can be referenced to each other.

[0074] Figure 4 : is a schematic diagram of the structure of the leafy vegetable seedling identification and transplanting clamping position detection device provided by the present invention, such as Figure 4 As shown, the leafy vegetable seedling identification and transplant clamping position detection device includes: an image acquisition module 410 and a detection module 420.

[0075] An image acquisition module 410 is used to acquire an image of a seedling to be tested; A detection module 420, configured to perform seedling recognition and transplant clamping position detection on the seedling image to be detected based on the detection model, and obtain a seedling recognition result and a transplant clamping position detection result; Among them, the detection model is based on the sample seedling images as training samples, the diseased seedling types and healthy seedling types as sample labels, and the convolutional neural network is trained with a complex sample mining loss function; the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type respectively; the complex sample mining loss function is determined based on the Focal loss function, the loss value ranking theory and the sample stratification method function.

[0076] The leafy vegetable seedling identification and transplanting clamping position detection device provided by the embodiment of the present invention uses diseased seedling types and healthy seedling types as sample labels, and uses a complex sample mining loss function to train a convolutional neural network to obtain a detection model, which performs seedling identification and transplanting clamping position detection on the seedling image to be tested, obtains seedling identification results and transplanting clamping position detection results, and improves the transplanting efficiency of leafy vegetable seedlings and the survival rate of seedlings.

[0077] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a method for leafy vegetable seedling identification and transplant clamping position detection, the method comprising: obtaining a seedling image to be tested; performing seedling identification and transplant clamping position detection on the seedling image to be tested based on a detection model to obtain seedling identification results and transplant clamping position detection results; wherein the detection model is based on training a convolutional neural network using sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels using a complex sample mining loss function; wherein the sample label corresponding to the healthy seedling type is annotated with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type; the complex sample mining loss function is determined based on the Focal loss function, the loss value sorting theory and the sample stratification method.

[0078] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the leafy vegetable seedling identification and transplant clamping position detection methods provided by the above methods, and the method includes: obtaining a seedling image to be tested; performing seedling identification and transplant clamping position detection on the seedling image to be tested based on a detection model to obtain seedling identification results and transplant clamping position detection results; wherein the detection model is based on training a convolutional neural network with sample seedling images as training samples and diseased seedling types and healthy seedling types as sample labels using a complex sample mining loss function; wherein the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type; the complex sample mining loss function is determined based on the Focal loss function, the loss value sorting theory and the sample stratification method.

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the leafy vegetable seedling identification and transplant clamping position detection methods provided by the above-mentioned methods, the method comprising: obtaining an image of a seedling to be tested; performing seedling identification and transplant clamping position detection on the image of the seedling to be tested based on a detection model, and obtaining seedling identification results and transplant clamping position detection results; wherein the detection model is based on training a convolutional neural network using sample seedling images as training samples and diseased seedling types and healthy seedling types as sample labels using a complex sample mining loss function; wherein the sample label corresponding to the healthy seedling type is marked with two clamping position key points, and the format and data length of the sample label corresponding to the diseased seedling type are aligned with the sample label corresponding to the healthy seedling type; the complex sample mining loss function is determined based on the Focal loss function, the loss value sorting theory and the sample stratification method.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying leafy vegetable seedlings and detecting the clamping position of transplanting, characterized in that: include: Acquire the image of the seedling to be tested; Based on the detection model, seedling recognition and transplanting clamping position detection are performed on the seedling image to be tested to obtain a seedling recognition result and a transplanting clamping position detection result; The detection model is obtained by training the convolutional neural network based on sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels, and complex sample mining loss function; the sample labels corresponding to the healthy seedling type are marked with two clamping position key points, and the formats and data lengths of the sample labels corresponding to the diseased seedling type and the sample labels corresponding to the healthy seedling type are aligned respectively; The complex sample mining loss function is determined based on the focal loss function, the loss value ranking theory and the sample stratification method.

2. The leafy vegetable seedling identification and transplanting clamping position detection method according to claim 1, characterized in that: The detection model is obtained through the following steps: Extracting the phenotypic features from the sample seedling image based on the convolutional neural network; Calculate the Focal loss value of each sample seedling image according to the phenotypic characteristics based on the Focal loss function; and update the historical loss value of each sample seedling image according to the Focal loss value to determine the historical difficulty data of each sample seedling image; Calculating the comprehensive loss of each sample seedling image according to the focal loss value and the historical difficulty data, and sorting the comprehensive losses of each sample seedling image by a non-maximum suppression sorting method to obtain a sorted comprehensive loss; The sorted comprehensive losses are stratified according to a stratification threshold to obtain a plurality of stratified losses; each stratified loss is sampled in proportion to obtain a plurality of sampling losses, and different stratified losses correspond to different sampling losses; Calculate the average value of each sample loss, and add up the average values ​​to obtain the target loss; The convolutional network is optimized and trained according to the target loss, and the detection model is obtained under the condition that the network converges.

3. The leafy vegetable seedling identification and transplanting clamping position detection method according to claim 1, characterized in that: The detection model is also obtained through the following steps: Construct a plug tray dataset corresponding to the sample seedlings in the transplanting scenario; Based on transfer learning, the 1st to Nth layers of the convolutional neural network are pre-trained using the ImageNet large data set to obtain a pre-trained convolutional neural network; N is a positive integer greater than 2; The network parameters of the output layer of the pre-trained convolutional neural network are fine-tuned according to the hole tray data set to obtain the detection model.

4. The leafy vegetable seedling identification and transplanting clamping position detection method according to any one of claims 1 to 3, characterized in that: The convolutional neural network includes an improved inverted bottleneck block structure, and the convolutional neural network is provided with multiple channels to expand features; The convolutional neural network also includes a deformable convolutional network, which is used to introduce a learnable offset and adopt a dedicated window mode to non-uniformly sample the input feature map; The convolutional neural network is used to use a dedicated sliding window for each position with an unbounded range of aggregation weights; the shape of the sliding window is independent of the aggregation weight input.

5. The method for identifying leafy vegetable seedlings and detecting the position of the transplanting clamp according to any one of claims 1 to 3, characterized in that: The convolutional neural network includes an attention module based on scale variation; The scale-variation-based attention module is used to extract the phenotypic features from the sample seedling image by the following formula: ； in, A m (x, i) is the phenotypic feature; x is the network input tensor; i is the output size of the attention map corresponding to the sample seedling image, f(x, i) represents the average pooling function; P 1 (x, i) is the association probability, n Indicates the number of output pool tensors; P 1 (x, i) The following conditions must be met: and .

6. The leafy vegetable seedling identification and transplanting clamping position detection method according to claim 2, characterized in that: Before extracting the phenotypic characteristics from the sample seedling image, the method further includes: Performing fine scaling processing on the sample seedling image by a bilinear interpolation algorithm to obtain a processed sample seedling image; The image enhancement technology is used to perform data expansion on the processed sample seedling image to obtain an expanded sample seedling image.

7. A leafy vegetable seedling identification and transplanting clamping position detection device, characterized in that: include: An image acquisition module, used for acquiring images of seedlings to be tested; A detection module, used to perform seedling recognition and transplant clamping position detection on the seedling image to be detected based on the detection model, and obtain a seedling recognition result and a transplant clamping position detection result; The detection model is based on sample seedling images as training samples, diseased seedling types and healthy seedling types as sample labels, and a convolutional neural network is trained using a complex sample mining loss function; the sample label corresponding to the healthy seedling type is annotated with two clamping position key points, the format and data length of the sample label corresponding to the diseased seedling type are aligned with those of the sample label corresponding to the healthy seedling type, and the value of the sample label corresponding to the diseased seedling type after label conversion is different from the value of the sample label corresponding to the healthy seedling type; The complex sample mining loss function is determined based on the focal loss function, the loss value ranking theory and the sample stratification method function.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying leafy vegetable seedlings and detecting the transplanting clamping position as described in any one of claims 1 to 6 is implemented.

Images