Seeding detection and reseeding method of air vibration type precision seeder based on visual language model

Abstract

The invention discloses a visual language model-based seeding detection and reseeding method for an air vibration type precision seeder, and belongs to the field of agricultural informatization and intelligent seeding control. The method comprises the following steps: establishing an image acquisition system to obtain a seedling raising tray image after seeding, and performing preprocessing operations such as filtering, distortion correction and enhancement; constructing a high-quality data set in combination with an open source labeling tool and the pre-training model; an improved CGG instance segmentation network is provided, a BLIP pseudo-tag generation module, a BERT semantic guidance module and an enhanced V2L feature mapping module are introduced, and the semantic understanding and segmentation precision of the model in dense overlapping rice seed target detection is improved. Experimental results show that the model obtains 92.2% of mAP50seg segmentation precision in a seeding detection task. The method effectively improves the sowing yield and the operation efficiency, and is suitable for an intelligent seedling raising assembly line scene.

Description

Technical Field

[0001] This invention belongs to the fields of agricultural precision seeding technology and multimodal deep learning. Specifically, it is a seeding detection and reseeding method for a pneumatic precision seeder based on a visual language model, which falls under the categories of agricultural informatization, intelligent seeding control, and agricultural machinery automation technology. Background Technology

[0002] In my country's rice cultivation system, factory-style seedling raising has become an important means to improve seedling quality and achieve refined seedling management. Pneumatic precision seeders offer advantages such as high sowing accuracy, minimal seed damage, and minimal impact from seed shape. They can also meet the technical requirements for precise seedling raising and sowing of different rice seeds per hill, making them widely used in precision sowing and a highly productive precision sowing device. However, due to the complexity of the equipment's mechanical structure and the operating environment, problems such as missed sowing, double sowing, and misalignment often occur during actual sowing, seriously affecting subsequent seedling quality and automated transplanting efficiency. Therefore, accurate detection and intelligent re-sowing of rice seeds in the seedling trays after sowing have become crucial for improving the rate of good planting hills and operational efficiency.

[0003] Traditional methods often employ image processing algorithms such as threshold segmentation and edge detection to analyze seedling tray images. However, in complex scenarios involving overlapping, occluded, or partially covered rice seeds, these methods suffer from low accuracy and poor robustness, failing to meet the real-time detection requirements of industrial production lines. In recent years, deep learning methods have been increasingly applied to agricultural image analysis tasks, particularly instance segmentation techniques, which can achieve pixel-level segmentation of target objects while preserving individual morphological information, making rice seed identification and counting possible. However, existing instance segmentation models still have limitations when handling dense, small, or overlapping targets, particularly in generalization ability and semantic understanding. Especially when encountering rice seed occlusion, blurred edges, or the presence of non-seed interference, traditional models are prone to missed or false detections. Furthermore, there is currently a lack of a complete closed-loop solution that can simultaneously achieve sowing status recognition and the generation of missing planting points.

[0004] Visual language models have demonstrated excellent semantic representation and generalization capabilities in tasks such as image understanding and multimodal fusion in recent years. Deeply fusing image information with language descriptions can effectively enhance the model's understanding of target semantics and improve recognition accuracy in complex environments. However, existing research has not systematically introduced visual language models into rice seed identification tasks in seedling trays, nor has it used them to support sowing quality analysis and automatic replanting point planning. Therefore, there is an urgent need to propose a method for rice seed detection and automatic replanting that integrates visual language models to improve recognition accuracy and replanting efficiency in complex environments, and to assist pneumatic precision seeders in achieving intelligent closed-loop control of sowing quality. Summary of the Invention

[0005] To address the shortcomings of existing technologies, such as insufficient accuracy in rice seed instance segmentation, poor recognition performance under dense shading conditions, and low automation in sowing quality detection, this invention proposes a sowing detection and reseeding method for a pneumatic precision seeder based on a visual language model.

[0006] A method for seed detection and reseeding of a pneumatic precision seeder based on a visual language model includes the following steps: Step S1, Image Acquisition: On the air-vibration precision seeding production line, an industrial camera with a light source is used to acquire images of the seedling trays after seeding, and a set of images of the seedling trays after seeding is constructed. Step S2, Image Preprocessing: The acquired seedling tray images are sequentially subjected to denoising and smoothing, distortion correction, and image enhancement operations to improve image quality and enhance the target features of rice seeds.

[0007] Step S3, Dataset Creation: Using the open-source annotation tool Labelme (combined with EfficientSAM (EfficientSegment Anything Model) for localization assistance), the outer contours of all rice seeds in the preprocessed images are accurately annotated and uniformly labeled as "seed", thus constructing a high-quality dataset of post-sowing seedling tray images. Step S4, Network Model Construction and Training: Construct a rice seed instance segmentation network model in the seedling tray. Input the seedling tray image dataset after sowing prepared in step S3 into the constructed network model for training and testing to obtain the trained rice seed instance segmentation network model in the seedling tray. Step S5, Online Image Detection: Deploy the trained seedling tray sowing detection model to the industrial control computer. Real-time detection of the seedling tray image after sowing; the model outputs a rice seed instance segmentation mask. This mask is spatially mapped to a standard seedling tray hole template, and the number of rice seeds in each hole is counted, outputting a sowing quality distribution matrix. If empty holes or holes requiring replanting are detected, the industrial control computer generates a replanting command.

[0008] Step S6, Replanting: When the production line carrying the seedling trays moves to the replanting stage, the replanting robotic arm located downstream of the detection stage receives instructions from the industrial control computer, moves precisely above the planting hole to be replanted, and releases the required number of seeds according to the instructions to complete the replanting operation.

[0009] In the above scheme, the rice seed instance segmentation network model in the seedling tray in step S4 is an improved CGG instance segmentation network model, and the network structure design is as follows: This invention makes targeted improvements to the original CGG (Caption Grounding and Generation) network architecture, constructing a multimodal visual-linguistic instance segmentation model for rice seed identification tasks in seedling trays. This model consists of two parts: an image module and a language module, fusing visual and linguistic information to improve the segmentation and recognition accuracy of rice seed instances in densely occluded scenes.

[0010] Regarding the image module: the original input is an RGB color image, which is first scaled to a standard size of 640×640×3 before being input into the Mask2Former-based backbone network for extracting multi-scale image features. The extracted features are processed by the Transformer decoder to generate image embedding vectors, which are then input into the visual-to-language conversion module (V2LTransformHead) designed in this invention. The features output by the V2L module are fused with the output of the pixel-level decoder to generate mask prediction results for N rice seed instances. Simultaneously, the pre-defined query vector in the model is mapped to N multimodal embedding vectors, and similarity calculations are performed with the semantic category embedding vectors to achieve category discrimination and assignment.

[0011] In terms of the language module, it comprises two core parts: image-text alignment loss calculation and semantic category guidance. Specifically, a pre-trained visual language model, BLIP, is used to generate multiple image captions from the input image. The target semantic keyword "rice seed" is extracted from these captions, and its semantic features are extracted using a word encoder. These features are then aligned with the image semantic features to calculate the image-text anchoring loss. To further enhance the expressive power of semantic embedding, a pre-trained language model, BERT, is introduced to perform contextual semantic modeling on the keyword "rice seed," generating a context-aware semantic vector that improves the discriminative performance of the semantic classification branch.

[0012] During model training, a joint multi-task strategy is employed, designing three types of loss functions: classification loss for category discrimination, masking loss for instance masking supervision, and image-text matching loss for visual and linguistic semantic alignment. These three losses are weighted and fused to form the total loss function, which is then optimized through end-to-end backpropagation training to achieve a synergistic improvement in the model's semantic understanding and instance segmentation capabilities.

[0013] The process of BLIP generating language descriptions is as follows: The input image first extracts visual features through a multi-layer Transformer network and then feeds them into a cross-attention module; the text decoder is constructed using a causal self-attention mechanism and shares the cross-attention layer and feedforward neural network with the encoder. The entire model is trained using language modeling loss to achieve automatic generation of image semantic descriptions.

[0014] The BERT word embedding process is as follows: Semantic words are input into the BERT embedding layer to obtain initial word vector representations. Subsequently, these vectors are passed through a 12-layer Transformer network, each layer containing self-attention, multi-head attention, and feedforward network modules to model contextual semantic relationships. Unlike traditional static word vectors, BERT employs a bidirectional encoding mechanism, which can simultaneously capture the semantic dependencies before and after words, generating high-dimensional word embeddings with context-aware capabilities. The resulting word vectors not only contain the semantic information of the words themselves but also incorporate their semantic roles within the image description context. The BERT-encoded semantic vectors are introduced as category-guided features into the model's classification branch, calculating similarity with multimodal query embeddings, effectively enhancing the model's ability to distinguish the "rice seed" category and significantly improving classification accuracy and stability in complex scenarios.

[0015] To enhance the mapping ability of visual features to the language embedding space, this invention structurally enhances the original V2L module. The original module employs only a single-layer linear transformation, lacking nonlinear activation, normalization, and residual mechanisms, making it difficult to capture complex semantic relationships in image features. Therefore, a multi-level V2L module is proposed, with each layer including LayerNorm normalization, GELU activation function, Dropout regularization, and residual connections, thereby enhancing the network's nonlinear modeling ability and training stability. Furthermore, a learnable Layer Scale scaling factor is introduced into each non-output layer to adjust the feature transformation amplitude, improving the trainability and robustness of deep networks. This improvement, while maintaining a lightweight model, significantly enhances the semantic alignment between visual and linguistic features, providing solid support for multimodal instance segmentation tasks.

[0016] A vision-language driven seeding detection and reseeding device for pneumatic precision seeders includes an image acquisition module, an image preprocessing module, a dataset creation module, a network model construction and training module, an online seeding detection module for seedling trays, and a reseeding module.

[0017] The image acquisition module is used to acquire images of the seedling trays after sowing and obtain a set of images of the seedling trays after sowing. It includes an industrial camera and a matching light source to cooperate in completing the image acquisition task. The industrial camera is connected to a photoelectric sensor to trigger the photo-taking operation when the seedling tray is detected passing by.

[0018] The image preprocessing module is used to perform noise reduction and smoothing, distortion correction and image enhancement operations on the acquired seedling tray images in sequence, so as to improve image quality and enhance the target features of rice seeds.

[0019] The dataset creation module is used to use the open-source image annotation tool Labelme to annotate preprocessed images and construct a dataset of seedling tray images after sowing, which can be used for subsequent model training.

[0020] The network model construction and training module is used to construct a seedling tray sowing detection model based on visual-language fusion, and input the completed dataset into the model for training and testing to obtain a sowing detection network model with high segmentation and recognition accuracy.

[0021] The online detection module for seedling tray sowing is used to deploy the trained sowing detection model to an industrial control computer, enabling real-time processing and analysis of images after sowing. This module uses the model to segment images into instances, extracts the location information of each rice seed, and counts the number of seeds per hole to determine if any seeds have been missed. If no seeds are detected in a hole, a reseeding command is generated, and the industrial control computer controls a robotic arm to perform subsequent operations.

[0022] The reseeding module is used to accurately reseed missed planting holes. Located in the downstream stage of the sowing production line, this module controls the reseeding robotic arm to move above the target hole and perform a fixed-point seed placement operation to achieve automatic reseeding, thereby improving sowing quality and the level of intelligent operation.

[0023] The beneficial effects of this invention are: 1. Enhanced Rice Seed Segmentation Accuracy and Occlusion Robustness: This invention constructs a multimodal visual-language instance segmentation model. Based on an improved CGG network structure, it introduces a visual-language interaction mechanism and contextual modeling strategy, significantly enhancing the model's semantic understanding and spatial perception capabilities. Even in complex scenes with densely overlapping or severely occluded rice seeds, it can still achieve accurate instance differentiation and segmentation, with a segmentation accuracy (mAP50). seg The accuracy rate reached 92.2%, outperforming existing mainstream instance segmentation models. 2. A BLIP-based pseudo-label-assisted supervision mechanism: Addressing the issues of high cost and limited data volume in rice seed image annotation, this invention introduces a pre-trained visual language model, BLIP, to generate semantic descriptions of images and automatically extract the keyword "rice seed," constructing pseudo-labels to expand training data and form auxiliary supervision signals. This mechanism effectively alleviates the reliance on manual annotation, improves the model's learning ability and generalization performance under weak supervision, and provides stronger data support for multimodal segmentation tasks. 3. Semantic guidance for accurate classification: Combining the context-aware semantic vectors extracted by BERT, the model is guided to make category judgments, which significantly improves the stability of the semantic branch in recognizing "rice seed" type targets and enhances the model's ability to semantically distinguish between similar targets.

[0024] 4. Enhanced visual-language mapping capability: The improved multi-level V2L module of this invention adopts LayerNorm, GELU activation, residual connections and learnable Layer Scale scaling factors to enhance the efficient mapping of visual features to the language semantic space and improve cross-modal semantic consistency and training stability. 5. High degree of system automation: It realizes an end-to-end automated process from seeding quality detection to reseeding missing holes. Combining vision detection and robotic arm execution system, the reseeding operation can be completed automatically without human intervention, which significantly improves the intelligence level and work efficiency of the seeding process. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a general design block diagram of one embodiment of the present invention; Figure 2 This is a schematic diagram of an improved CGG network structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the BLIP module structure according to one embodiment of the present invention; Figure 4 This is a schematic diagram of the BERT module structure according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the V2L module structure according to one embodiment of the present invention; Figure 6 This is a detection effect diagram of an embodiment of the present invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] Figure 1 The above illustrates a preferred embodiment of the pneumatic precision seeder seeding detection and reseeding method based on a visual language model. The method includes the following steps: Step S1, Image Acquisition: On the air-vibration precision seeding production line, an industrial camera with a light source is used to acquire images of the seedling trays after seeding, and a set of images of the seedling trays after seeding is constructed. Step S2, Image Preprocessing: The acquired seedling tray images are sequentially subjected to denoising and smoothing, distortion correction and image enhancement operations to improve image quality and enhance the target features of rice seeds; Preferably, the denoising and smoothing process employs a Gaussian filtering method to suppress image noise and preserve edge features; distortion correction utilizes a preset checkerboard calibration plate to obtain the in-camera distortion coefficients, and then performs geometric transformation on the image to achieve distortion correction of the seedling tray image, ensuring that the hole area presents a standard geometric shape; image enhancement processing preferably employs an adaptive histogram equalization method to improve local image contrast and the distinguishability of rice seed targets.

[0029] Step S3, Dataset Creation: Using the open-source annotation tool Labelme, combined with the pre-trained EfficientSAM model to assist in localization, the outer contours of all rice seeds in the preprocessed images are accurately annotated and uniformly labeled as "seed", thus constructing a high-quality dataset of post-sowing seedling tray images. Step S4, Network Model Construction and Training: An improved CGG instance segmentation model is constructed. The image dataset of rice seedling trays after sowing, prepared in Step S3, is input into the constructed network model for training and testing. During model training, this invention adopts a joint multi-task strategy, designing three types of loss functions: classification loss, masking loss, and image-text matching loss. These functions are then weighted and fused to form the total loss, thereby obtaining the trained rice seed instance segmentation network model in the seedling tray. The calculation formula is shown below: ; ; in, L cls The cross-entropy loss function is used for category discrimination; L mask Used for instance mask supervision of Dice loss or Focal loss; L vl The image-text matching loss is for visual and linguistic semantic alignment; λ i These are the loss weighting coefficients; sim () represents the cosine similarity function; Indicates the first i Image feature vectors; This represents the corresponding text semantic vector; τ This is the temperature coefficient.

[0030] Preferably, the improvement of the CGG detection network in step S4 includes: The addition of the BLIP module automatically outputs 5 title descriptions for each image, reducing the burden of manual annotation while improving the model's learning ability and generalization performance under weak supervision, providing stronger data support for multimodal segmentation tasks.

[0031] Adding the BERT module to embed word vectors for the word "seed" significantly improves the stability of semantic branch recognition of "rice seed" type targets and enhances the model's ability to semantically distinguish between similar targets.

[0032] The V2L module is enhanced to improve the efficient mapping of visual features to the linguistic semantic space, thereby improving cross-modal semantic consistency and training stability.

[0033] The BLIP module specifically comprises: the input image first extracts visual features through a 6-layer Transformer network and then feeds them into the cross-attention module; the text decoder is constructed using a causal self-attention mechanism and shares the cross-attention layer and feedforward neural network with the encoder. In this embodiment, the training objective of the BLIP module is the language modeling loss, which takes the following form: ; in, T The total length of the text sequence; y t For the first t The true label of each word; y ＜t Position in the text t All the previous words; I Input image; θ These are model parameters; P () represents the probability. The entire model is trained using language modeling loss to achieve automatic generation of semantic descriptions of images.

[0034] The BERT module specifically works by inputting the semantic word "seed" into the BERT embedding layer to obtain an initial word vector representation. This vector is then passed through 12 layers of a Transformer network, each layer containing self-attention, multi-head attention, and feedforward network modules to model contextual semantic relationships. Unlike traditional static word vectors, BERT employs a bidirectional encoding mechanism, which can simultaneously capture the semantic dependencies before and after words, generating high-dimensional word embeddings with context-aware capabilities. Its embedding output is represented as follows: ; ; in, x i It is the first i The index of each input word; We It is a word embedding matrix; p i Indicates the first i The position of each word in the sentence. e i It is the first i The input vectors are words. The resulting word vectors not only contain the semantic information of the words themselves, but also incorporate the semantic role they play in the context of image description.

[0035] The enhanced V2L module replaces the original single-layer linear transformation with a multi-level structure to improve the mapping ability of visual features to the linguistic semantic space. Specifically, the original input first undergoes a linear transformation, then sequentially passes through a learnable Layer Scale scaling factor, a LayerNorm normalization layer, and a GELU activation function, before being residually connected with the input features and then entering a Dropout regularization layer. This process is repeated twice, and finally a linear transformation layer is added to generate the output features. This structure integrates LayerNorm, GELU, Dropout, and residual connection mechanisms, significantly enhancing the network's non-linear expressive power and training stability. The introduced Layer Scale factor can adaptively adjust the feature transformation amplitude, further improving the model's trainability and robustness. The layer-by-layer operation process of the V2L module can be represented as follows: ; in, z (l+1) It is the output feature; z (l) It is the first l +1 layer input feature vector; α (l) A learnable Layer Scale scaling factor; W (l) and b (l) They are the first l Layer linear transformation weight matrix and bias vector. This module effectively improves the semantic alignment between visual and linguistic features while maintaining a lightweight model, providing strong support for multimodal instance segmentation tasks.

[0036] Step S5, Online Image Detection: Deploy the trained seedling tray sowing detection model to the industrial control computer. Real-time detection of the seedling tray image after sowing; the model outputs a rice seed instance segmentation mask. This mask is spatially mapped to a standard seedling tray hole template, the number of rice seeds in each hole is counted, and a sowing quality distribution matrix is ​​output, thus realizing seedling tray sowing detection. If empty holes or holes requiring replanting are detected, the industrial control computer generates a replanting command.

[0037] Step S6, Replanting: When the production line carrying the seedling trays moves to the replanting stage, the replanting robotic arm located downstream of the detection stage receives instructions from the industrial control computer, moves precisely above the planting hole to be replanted, and releases the required number of seeds according to the instructions to complete the replanting operation.

[0038] A visual language-driven seeding detection and reseeding method for pneumatic precision seeders is proposed, comprising an image acquisition module, an image preprocessing module, a dataset creation module, a network model construction and training module, an online seeding detection module for seedling trays, and a reseeding module.

[0039] The image acquisition module is used to acquire images of the seedling trays after sowing and obtain a set of images of the seedling trays after sowing. It includes an industrial camera and a matching light source to cooperate in completing the image acquisition task. The industrial camera is connected to a photoelectric sensor to trigger the photo-taking operation when the seedling tray is detected passing by.

[0040] The image preprocessing module is used to perform noise reduction and smoothing, distortion correction and image enhancement operations on the acquired seedling tray images in sequence, so as to improve image quality and enhance the target features of rice seeds.

[0041] The dataset creation module is used to use the open-source image annotation tool Labelme to annotate preprocessed images and construct a dataset of seedling tray images after sowing, which can be used for subsequent model training.

[0042] The network model construction and training module is used to construct a seedling tray sowing detection model based on visual-language fusion, and input the completed dataset into the model for training and testing to obtain a sowing detection network model with high segmentation and recognition accuracy.

[0043] The online detection module for seedling tray sowing is used to deploy the trained sowing detection model to an industrial control computer, enabling real-time processing and analysis of images after sowing. This module uses the model to segment images into instances, extracts the location information of each rice seed, and counts the number of seeds per hole to determine if any seeds have been missed. If no seeds are detected in a hole, a reseeding command is generated, and the industrial control computer controls a robotic arm to perform subsequent operations.

[0044] The reseeding module is used to accurately reseed missed planting holes. Located in the downstream stage of the sowing production line, this module controls the reseeding robotic arm to move above the target hole and perform a fixed-point seed placement operation to achieve automatic reseeding, thereby improving sowing quality and the level of intelligent operation.

[0045] Preferably, the image acquisition module includes an industrial camera and a light source; the industrial camera, in conjunction with the light source, acquires images of the seedling tray after sowing on the pneumatic precision seeder; the industrial camera is connected to a photoelectric sensor for controlling the industrial camera to take pictures.

[0046] According to this embodiment, preferably, the model training parameters are set as follows: Batch size is set to 4, base learning rate is set to 0.0001, optimizer is selected as ADAMW, momentum parameter is set to 0.937, input image size is 640×640 pixels, and number of training rounds is 200.

[0047] The overall flowchart of the sowing detection and reseeding method for a pneumatic precision seeder based on a visual language model described in this invention is as follows: Figure 1 As shown, it includes the following operation steps: Step S101: Image Acquisition: An image acquisition device is built using an industrial camera, a light source, a camera controller, and a light source controller. Preferably, the image acquisition device is located 500mm above the seedling tray to acquire images of the seedling tray after sowing by the pneumatic precision seeder, thus obtaining a set of images of the seedling tray after sowing. Step S102: Image preprocessing: The seedling tray image obtained in step S101 is subjected to denoising and smoothing, distortion correction and image enhancement operations in sequence to improve image quality and enhance the target features of rice seeds. Step S103: Dataset creation: Using the open-source annotation tool Labelme, combined with the pre-trained EfficientSAM model to assist in localization, the outer contours of all rice seeds in the preprocessed images are accurately annotated and uniformly labeled as "seed", thus constructing a high-quality dataset of post-sowing seedling tray images. Step S104: Network Model Construction and Training: Construct an improved CGG instance segmentation model. Input the image dataset of seedling trays after sowing prepared in step S103 into the constructed network model for training and testing, and obtain a trained rice seed instance segmentation network model in the seedling tray. In this embodiment, to improve the segmentation accuracy of small rice grain targets in densely overlapping scenes, the instance segmentation network is specifically improved in three aspects: semantic understanding, semantic guidance, and visual language feature alignment. First, a pre-trained visual language model, BLIP, is introduced to generate image pseudo-labels, enhance the supervision signal, and improve the model's ability to perceive semantic regions. Second, context-aware semantic embeddings generated by the BERT model are fused as category-guided features to improve the model's stability and accuracy in category discrimination. Third, an enhanced V2L module is designed to optimize the mapping effect of visual features to the semantic space, further enhancing the alignment ability between multimodal features, thereby achieving a more refined instance segmentation effect. The improved CGG network architecture is as follows: Figure 2 As shown.

[0048] The original image is input into the network and passes through the BLIP module to generate the text data needed for training. Its structure is as follows: Figure 3As shown, the input image first extracts visual features through a 6-layer Transformer network and then feeds them into a cross-attention module; the text decoder is constructed using a causal self-attention mechanism and shares the cross-attention layer and feedforward neural network with the encoder. The entire model is trained using language modeling loss to achieve automatic generation of image semantic descriptions.

[0049] To improve the model's ability to connect semantic information about words, a BERT module was added during word embedding, with the following structure: Figure 4 As shown, the semantic word "seed" is input into the BERT embedding layer to obtain an initial word vector representation. This vector is then passed through a 12-layer Transformer network, each layer containing self-attention, multi-head attention, and feedforward network modules to model contextual semantic relationships. Unlike traditional static word vectors, BERT employs a bidirectional encoding mechanism, which can simultaneously capture the semantic dependencies before and after the word, generating high-dimensional word embeddings with context-aware capabilities. The resulting word vectors not only contain the semantic information of the word itself but also incorporate its semantic role within the image description context.

[0050] To enhance the mapping capability of visual features to the language embedding space, this invention structurally enhances the original V2L module, as shown in the following structure. Figure 5 As shown, the original module only employs a single-layer linear transformation, lacking nonlinear activation, normalization, and residual mechanisms, making it difficult to capture complex semantic relationships in image features. To address this, a multi-layer V2L module is proposed, with each layer including LayerNorm normalization, GELU activation function, Dropout regularization, and residual connections, thereby enhancing the network's nonlinear modeling ability and training stability. Furthermore, a learnable Layer Scale scaling factor is introduced into each non-output layer to adjust the feature transformation amplitude, improving the trainability and robustness of deep networks. This improvement, while maintaining a lightweight model, significantly enhances the semantic alignment between visual and linguistic features, providing solid support for multimodal instance segmentation tasks.

[0051] Step S105: Online Image Detection: Deploy the trained seedling tray sowing detection model to the industrial control computer. Real-time detection of the seedling tray image after sowing; the model outputs a rice seed instance segmentation mask image, the segmentation result as shown below. Figure 6 As shown, the mask image is spatially mapped to the standard seedling tray hole template. The number of rice seeds in each hole is counted, and the sowing quality distribution matrix is ​​output to realize seedling tray sowing detection. If an empty hole or a hole that needs to be replanted is detected, the industrial control computer generates a replanting command.

[0052] Step S106: Replanting: When the production line carrying the seedling trays moves to the replanting stage, the replanting robotic arm located downstream of the detection stage receives instructions from the industrial control computer, moves precisely above the hole to be replanted, and releases the required number of seeds according to the instructions to complete the replanting operation.

[0053] Therefore, through the above steps, the average accuracy mAP50 for detecting rice seeds in the seedling tray of a pneumatic precision seeder using the improved CGG instance segmentation model is achieved. seg The optimal hardware and software platform settings are shown in Table 1, with a success rate of 92.2%.

[0054] Table 1: Configuration Name Version parameters operating system Ubuntu 20.04 GPU NVIDIA GeForce RTX 3060 CPU 12th Gen Intel(R) Core(TM) i5-12400F Deep learning framework PyTorch 1.12.1 Python 3.8.0 CUDA 11.6 The sowing detection and reseeding method for pneumatic precision seeders based on visual language models described in this invention can improve the sowing detection and reseeding capabilities of pneumatic precision seeders, meeting the requirements for detection speed and accuracy in factory production.

[0055] This invention presents an improved CGG instance segmentation network, incorporating a BLIP pseudo-label generation module, a BERT semantic guidance module, and an enhanced V2L feature mapping module to enhance semantic understanding and segmentation accuracy in densely overlapping rice seed target detection. Experimental results show that the model achieves a 92.2% mAP50seg segmentation accuracy in sowing detection tasks. The system can be deployed on an industrial control computer to achieve online identification of sowing status and generation of empty seeding points, combined with a replanting robotic arm to complete end-to-end automated operations. This method effectively improves the rate of good seeding holes and operational efficiency, and is suitable for intelligent seedling raising production lines.

[0056] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for seeding detection and reseeding in a pneumatic precision seeder based on a visual language model, characterized in that, Includes the following steps: S1. Image Acquisition: Acquire images of seedling trays after sowing using industrial cameras and light sources, and construct a sowing image set; S2. Image preprocessing: Denoising and smoothing, distortion correction and image enhancement operations are performed on the image; S3. Image Dataset Creation: Using annotation tools and pre-trained models, rice seed outlines are annotated to construct a sowing image dataset; S4. Network Model Construction and Training: An improved instance segmentation model is constructed and trained using the dataset to obtain a sowing detection model. The instance segmentation model is a multimodal visual language model based on an improved image description localization and generation (CGG) architecture, which integrates image features, language features, and semantic guidance mechanisms to achieve high-precision rice seed segmentation and intelligent replanting point generation. S5. Online Image Detection: Deploy the trained model to the industrial control computer to segment the sowing image in real time and count the number of rice seeds in each hole to generate replanting points; S6. Replanting Execution: The replanting robotic arm moves precisely above the empty hole according to the replanting command to perform replanting.

2. The method according to claim 1, characterized in that, The image preprocessing uses Gaussian filtering for denoising and smoothing, geometric transformation for distortion correction based on checkerboard calibration parameters, and adaptive histogram equalization for image enhancement.

3. The method according to claim 1, characterized in that, The image dataset was created using the open-source image annotation tool Labelme to assist in localization, and all rice seed outlines were annotated at the instance level, with the unified labeling category being "seed".

4. The method according to claim 1, characterized in that, The specific structure of the instance segmentation model in step S4 is as follows: Based on the original CGG network architecture, an optimized and improved multimodal visual-language instance segmentation model was constructed for the rice seed identification task in seedling trays. This model is used to fuse image and text information to improve segmentation accuracy and semantic understanding under dense occlusion conditions. The model consists of two parts: an image module and a language module. It achieves accurate identification of rice seed instances through visual-language fusion. In the image module, the input is an RGB color image, which is first scaled to a standard size of 640×640×3 and then input into a backbone network based on the Mask2Former general image segmentation model based on the Mask Transformer. Multi-scale image features are extracted, and the extracted features are used to generate image embedding vectors through the Transformer decoder. These vectors are then fed into the vision-to-language V2L module designed in this invention. The features output by the V2L module are fused with the pixel-level decoder output to generate N mask prediction results for rice seed instances. At the same time, the query vector set by the model is mapped to N multimodal embedding vectors and similarity calculation is performed with the semantic category embedding vectors to achieve category discrimination and assignment. The language module comprises two core functions: image-text alignment loss calculation and semantic guidance. Specifically, it uses a pre-trained visual language model, BLIP (Bootstrapping Language-Image Pre-training), to generate multiple semantic descriptions for the input image, extracts the target keyword "seed," obtains its semantic features through a word encoder, and constructs an image-text anchoring loss by aligning it with the image semantics. Simultaneously, it introduces a pre-trained language model, BERT (Bidirectional Encoder Representations from Transformers), to perform contextual modeling on the keywords, generating context-aware semantic vectors as category guidance features for semantic classification branches, thereby improving the model's semantic discrimination ability in complex contexts. The BLIP module extracts visual features from images through a multi-layer Transformer network, processes them through a cross-attention module, and then generates image descriptions using a causal self-attention text decoder. It is trained using language modeling loss to achieve automatic pseudo-label generation. The BERT module inputs semantic words into the embedding layer to obtain initial word vectors, and performs context modeling through a 12-layer Transformer network. Each layer contains self-attention, multi-head attention, and feedforward network modules to generate high-dimensional word embeddings with context awareness. These embeddings are used for category-guided feature calculation and are matched with multimodal query embedding vectors for similarity matching, significantly improving the stability of the "rice seed" target recognition.

5. The method according to claim 4, characterized in that, The V2L module was structurally improved by replacing the single-layer linear transformation with a multi-level structure. Each layer includes linear transformation, Layer Scale scaling, LayerNorm normalization, Gaussian error linear unit (GELU) activation function, residual connection, and Dropout regularization, which are repeated twice to output the final features. This structure enhances the network's nonlinear expressive power and training stability, and the Layer Scale factor can adjust the feature transformation amplitude.

6. A sowing detection and reseeding device for a pneumatic precision seeder based on a visual language model, characterized in that, The device deploys the sowing detection model trained in step S4 as described in any one of claims 1-5. The device's industrial control computer acquires images of the seedling trays in real time and performs real-time sowing detection using the trained sowing detection network model. Specifically, it includes an image acquisition module, an image preprocessing module, a dataset creation module, a network model construction and training module, an online sowing detection module for seedling trays, and a reseeding module. The image acquisition module is used to acquire images of the seedling tray after sowing and obtain a set of seedling tray sowing images; The image preprocessing module is used to perform noise reduction and smoothing, distortion correction and image enhancement operations on the acquired seedling tray images in sequence, so as to improve image quality and enhance the target features of rice seeds. The dataset creation module is used to use the open-source image annotation tool Labelme to perform instance annotation on the preprocessed images and construct a dataset of seedling tray images after sowing, which can be used for subsequent model training. The network model construction and training module is used to construct a seedling tray sowing detection model based on visual language fusion, and input the completed dataset into the model for training and testing to obtain a sowing detection network model with high segmentation and recognition accuracy. The online detection module for seedling tray sowing is used to spatially map the segmented rice seed mask image with the standard hole template to generate a hole rice seed distribution matrix and determine the location of empty or abnormal holes. In the replanting module, after receiving instructions from the industrial control computer, the replanting robotic arm automatically moves to the designated hole and completes the fixed-point placement of single or multiple seeds, thus realizing the replanting of missing holes.

7. The apparatus according to claim 6, characterized in that: The trained seedling tray sowing detection model is deployed to an industrial control computer for real-time detection of seedling tray images after sowing. The model processes the input image and outputs an instance segmentation mask of rice seeds. This mask is then spatially mapped to a standard seedling tray hole template to accurately count the number of rice seeds in each hole and generate a sowing quality distribution matrix. If the detection result indicates that a hole is empty or has insufficient seeds, the industrial control computer automatically generates a reseeding instruction based on the analysis results.

8. The apparatus according to claim 7, characterized in that, The trained sowing detection model is deployed on an industrial control computer to realize real-time processing and analysis of images after sowing. This module performs instance segmentation on the image using the model, extracts the location information of each rice seed, and counts the number of rice seeds in each hole according to the planting hole to determine whether there is any missed sowing. If no rice seed is detected in the hole, a replanting command is generated, and the industrial control computer controls the replanting robotic arm to perform subsequent operations.

9. The apparatus according to claim 6, characterized in that, The replanting module is used to accurately replant the missed planting holes. This module is located in the subsequent stage of the sowing production line. It controls the replanting robotic arm to move above the target hole and perform a fixed-point seeding operation to achieve automatic replanting, improve sowing quality and the level of intelligent operation.

10. The apparatus according to claim 6, characterized in that, The replanting robotic arm is a three-axis gantry structure, installed on both sides of the assembly line support. Its working end is initially positioned 100mm above the seedling tray, and is used to perform precise replanting operations on detected empty or missing holes.

Images