Farmland obstacle detection method for low computing resource equipment

Abstract

The present invention discloses a farmland obstacle detection method for low computing resource equipment, comprising the following steps: step 1: collecting farmland image data; step 2: using an image synthesis method, synthesizing obstacles into farmland image data to construct a farmland image target detection data set with obstacles; step 3: pre-training knowledge distillation on the Imagenet data set to compress a large model into a lightweight model; step 4: using the lightweight model as a backbone to adapt the FPN component in the Mask R-CNN framework to become a detection model; step 5: reading the farmland image target detection data set, and using data enhancement to fine-tune the detection model on the constructed farmland data set with obstacles; step 6: deploying the trained model to agricultural equipment with low computing resources. In this way, a target detection data set with obstacles and a target detection model suitable for low computing resource equipment can be obtained, and the intelligent design of automatic obstacle detection of agricultural equipment can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of agricultural automation, and in particular relates to a farmland obstacle detection method for low computing resource equipment. Background Art

[0002] Today, the agricultural sector is increasingly benefiting from the vigorous development of modern science and technology. Various electronic monitoring and control devices have been widely used in agricultural machinery, seamlessly integrating modern microelectronics technology, sensor technology, instrumentation and control technology, and promoting agricultural equipment to move rapidly towards intelligence and mechatronics. Behind this trend is the rapid popularization of precision agriculture technology, in which sensors and monitoring devices, as well as automated control technology, are widely used in product research and development and agricultural production. With the development of this technological revolution, visual information processing of agricultural edge devices has become a key issue, because accurate visual information processing can provide more reliable support for agricultural decision-making.

[0003] Deep learning methods in image detection have been widely used in the agricultural field, but the problem of lack of obstacle datasets in the agricultural field is widespread. At present, there is no perfect obstacle dataset. Lack of datasets is a common problem, especially when conducting computer vision and deep learning research. Therefore, we can alleviate the problem of lack of datasets through image synthesis. Existing image synthesis methods can be divided into four categories. The first category is the generative adversarial network (GAN). The idea of ​​GAN is to use the game to continuously optimize the generator and discriminator so that the generated image is closer and closer to the real image in distribution. However, the images generated by GAN may be blurred, unclear or inconsistent with the actual scene, because the model may have difficulty capturing the details of special structures and obstacles in agricultural scenes. The second category is to generate images using large models. Large models usually refer to neural networks with a large number of parameters in deep learning. They have strong learning and representation capabilities and are suitable for complex tasks such as image generation. However, it is difficult to use large models to generate a large number of obstacle images. The third category is to generate images using diffusion models. Diffusion models are a generation model based on probability distribution. Images are generated by iteratively diffusing noise signals. However, diffusion models may not be accurate enough for the generation of obstacles in specific agricultural scenes. The fourth category is image collage, which creates a new synthetic image by copying and pasting a specific area (or multiple areas) of one image into another image. The advantage of this method is its simplicity. At the same time, since it uses real images as the basis, it can effectively synthesize a large number of diverse images.

[0004] The application of computer vision technology in the agricultural field has attracted widespread attention. Computer vision models deployed on agricultural edge devices have the potential to help farmers monitor information such as farmland conditions, crop growth status, pest infestation, etc., providing important basis for decision-making. However, to achieve this goal, a powerful visual processing model is needed. Vision Transformer (ViT). As a state-of-the-art deep learning architecture, it has achieved remarkable success in computer vision tasks. The agricultural field is often challenged by data scarcity, because agricultural data is usually difficult to obtain and the annotation cost is high; the traditional ViT architecture has difficulty in showing good performance on small data sets. The root cause of this problem lies in the global nature and lack of local priors of ViT, which makes it difficult for the model to learn local features and knowledge under limited data. Summary of the invention

[0005] The purpose of an embodiment of the present invention is to provide a farmland obstacle detection method for low computing resource equipment, so as to solve the data scarcity problem of the prior art and improve the visual information processing performance on agricultural edge devices by establishing a knowledge distillation method combined with curriculum learning.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is a farmland obstacle detection method for low computing resource equipment, comprising the following steps:

[0007] S1, collect farmland image data and preprocess them;

[0008] S2, select an image dataset and perform foreground segmentation on it, and then synthesize it with the farmland image data to obtain a farmland obstacle detection dataset;

[0009] S3, establish a detection model and perform pre-training of knowledge distillation on the image dataset selected in S2 to obtain a lightweight detection model;

[0010] S4. Use the lightweight model as the backbone network to build a detection model, and use the farmland obstacle detection dataset to fine-tune the detection model;

[0011] S5. Deploy the configured model to agricultural equipment with low computing resources.

[0012] Furthermore, the specific steps of S1 are:

[0013] S101, batch-acquire farmland front image data photos, and perform uniform preprocessing on the photos;

[0014] S102, classifying the acquired images, including assigning a scene label to each image;

[0015] S103, based on the scene classified in S102, using the question-answering language model to conduct interactive question-answering; using the generated language description combined with the original image to generate a new image sample, and adjusting the generation parameters;

[0016] S104, performing data cleaning to remove identical images and images of lower quality;

[0017] S105, cropping the image samples after data cleaning, and dividing the images into a training set, a validation set, and a test set according to a quantity ratio of 7:1:2.

[0018] Furthermore, the specific steps of S2 are:

[0019] S201, selecting a public image dataset, wherein the public image dataset includes image samples related to farmland scenes; selecting categories related to farmland scenes in the public image dataset, extracting some images in each category, extracting foreground objects in these images through a foreground segmentation algorithm, and obtaining a foreground mask M;

[0020] S202, synthesizing the image obtained in S1 with the image segmented in S201 to obtain a farmland obstacle detection data set; the synthesis method includes an object placement algorithm, an image fusion algorithm, an image harmonization algorithm and a noise generation algorithm.

[0021] Furthermore, the image synthesis process of S202 is specifically as follows:

[0022] S2011, obtaining farmland background image data from another public data set, performing background segmentation processing on the farmland background image, and then obtaining farmland background image segmentation area data through SAM segmentation, thereby obtaining the location where the foreground obstacle image collage is placed; collage the obstacle foreground data obtained in S201 on the farmland background image segmentation area data, and adjust the size of the obstacle through perspective perception;

[0023] S2012, adjust the color and lighting of the synthesized image to ensure that the synthesized obstacles are consistent with the surrounding environment; then use the UNet generator G to convert the synthesized image and the foreground mask M into a harmonious image, use the global discriminator to distinguish the real image from the generated image, guide the generator to generate realistic images, use the domain verification discriminator to verify the domain matching degree of the foreground and background in the generated image, and optimize the generator according to the domain verification loss;

[0024] S2013, simulating rainy day, foggy day and night scenes on the adjusted image; and adding image noise to the image, wherein the image noise includes light spots and lens vignetting;

[0025] S2014 outputs a farmland obstacle detection dataset with detailed image labels, where the image labels include the category, location and size of the obstacle image.

[0026] Furthermore, the specific steps of S3 are:

[0027] S301, construct teacher model and student model;

[0028] S302, the first stage of knowledge distillation, pre-training the constructed model on the public dataset selected in S201, using unlabeled images as input and using data augmentation;

[0029] S303, performing knowledge distillation on the model again to obtain a student model after knowledge distillation.

[0030] Furthermore, the teacher model and the student model in S301 each include an input embedding layer, a position encoding layer, and an encoder layer; the specific working method is as follows:

[0031] The embedding layer will be used to divide the image into patches and project each patch into a vector token, including a vector class token for image classification.

[0032] The position encoding layer is used to have an absolute position digital representation for each token, and the position value is added to the token;

[0033] The encoder layer includes:

[0034] A normalization module, used to calculate the mean and variance of all feature maps of a sample, and then normalize the sample;

[0035] The multi-head attention mechanism divides the input feature map into multiple heads, each head learns different attention weights, so that the model can focus on different local information of the input features at the same time; each head produces an independent attention map, and then these maps are fused together by weighted summation to form the final attention representation;

[0036] Regularization module, which is used to randomly discard the output of some neurons during training to prevent the model from over-relying on specific neurons and force the network to learn more robust feature representations;

[0037] The multi-layer perceptron module includes two fully connected layers, which introduces richer feature representation by performing nonlinear transformation on the input features. Specifically, the first fully connected layer maps the input features to a high-dimensional space, then introduces nonlinear transformation through the activation function, and the second fully connected layer maps the features back to the original dimension.

[0038] The last dimension of the token of the teacher model after the input embedding layer is 768, and each encoder layer has a multi-head attention mechanism with 12 heads;

[0039] The last dimension of the token of the student module after the input embedding layer is 192. The first and tenth layers of the encoder layer are multi-head attention mechanisms with 12 heads, and the remaining layers are multi-head attention mechanisms with 6 heads.

[0040] Furthermore, in S302, the first knowledge distillation process is to calculate the KL divergence of the self-attention behavior between the teacher model and the student model, and then use it as a loss function to train the model.

[0041] Furthermore, in S302, the process of the second knowledge distillation is to first obtain the cross entropy loss based on the classification probability and label information output by the student model; then calculate the KL discreteness of the logit values ​​of the student model and the teacher model; then add the cross entropy loss to the KL discreteness loss of the logit values ​​of the student model and the teacher model; finally, the loss obtained by adding the two losses is used as the loss function of the second distillation to train the student model; and output the student model after knowledge distillation.

[0042] Furthermore, the specific steps of S4 are:

[0043] S401. Integrate the processed lightweight student model into the Mask R-CNN framework, replacing the original ResNet, and ensure that the intermediate layers of the lightweight model are properly connected for subsequent use.

[0044] S402, adding an FPN component on the output of the lightweight student model, wherein the FPN component is a neural network architecture component for target detection and semantic segmentation tasks;

[0045] S403. According to the characteristics of the lightweight model and FPN components, adjust the network architecture to ensure that the input size and number of channels can work together;

[0046] S404, using the farmland image object detection dataset with obstacles to train the modified Mask R-CNN;

[0047] S405. Use the test set to evaluate the trained modified Mask R-CNN model, check the accuracy and efficiency of the model in object detection and instance segmentation tasks, and ensure that the lightweight model meets the needs of practical applications while maintaining performance.

[0048] Furthermore, the specific steps of S5 are:

[0049] S501, selecting a deep learning inference framework suitable for low computing resource devices;

[0050] S502, converting the student model into a format suitable for the target reasoning framework;

[0051] S503. For agricultural equipment with low computing resources, use the inference hardware of edge computing devices;

[0052] S504, embed the model into the embedded system of the equipment, use a single-board computer with ARM architecture, and use a ranging camera to determine the distance, and determine the objects within the target range as obstacles.

[0053] The beneficial effects of the present invention are as follows: the present invention adopts a comprehensive image generation framework to achieve highly realistic and diverse farmland obstacle image generation through the organic combination of an image collage module, a foreground and background image pixel harmony module, and an image noise generation module. The present invention collages the obstacle foreground image on the background, and introduces precise perspective perception adjustment, simulating the natural effect of relatively small objects in the distance, so that the generated image is closer to the real farmland scene, and improves the realism and fidelity of the image. Through the introduction of the global discriminator and the domain verification discriminator, the module better optimizes the generated image and ensures the performance of the generated image in terms of domain matching and realism. The image noise generation module comprehensively considers different environmental factors and enriches the diversity of the generated image. This diversity makes the generated image more challenging, helps to improve the generalization ability of the model, and enables it to adapt to various complex farmland environments. The present invention adopts a comprehensive image generation framework, which includes an organic combination of multiple modules such as image collage, foreground and background image pixel harmony, and image noise generation. This comprehensive design enables the various modules to work together to optimize the generated image. By introducing mechanisms such as adversarial training and domain verification, the generated images are made to achieve better results in terms of realism, diversity, domain matching, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 It is the overall flow chart of the present invention;

[0056] Figure 2 It is the image synthesis flow chart. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] This implementation provides a farmland obstacle detection method for low-computing resource equipment, creates a farmland image detection dataset with obstacles, and develops a new knowledge distillation method combined with curriculum learning to improve the visual information processing performance on agricultural edge devices. Figure 1 , specifically including the following steps:

[0059] Step S1: Collect farmland image data;

[0060] Step S11: Use Python program to automatically crawl 1,000 farmland image data photos from public websites to ensure that natural images are obtained. In this embodiment, 1,000 farmland image data photos are selected. The specific image data can be adjusted according to needs. The specific process of selecting pictures in this embodiment includes determining the target website, using web crawler technology to batch download images, and using image processing library for preprocessing to ensure consistency;

[0061] Step S12: Classify the farmland images and divide the images into 100 categories of different scenes. In this embodiment, the images are divided into 100 categories. In other possible embodiments, the number of scene categories can be adjusted according to specific needs. The specific classification process of this embodiment includes selecting appropriate annotation tools, manually annotating images, and assigning scene labels to each image. In some possible embodiments, the classification process can be performed by an intelligent algorithm.

[0062] Step S13: Based on the 100 types of scenes classified in S12, the GPT-4 model is used to generate language question-answering images to generate 1,000 images. This embodiment uses an integrated GPT-4 model for interactive question-answering, uses the generated language description combined with the original image to generate new image samples, and adjusts the generation parameters to ensure diversity and realism; in some possible implementations, other generative language models with stronger image-text interaction performance can be used.

[0063] Step S14: performing data cleaning to remove identical images and images of lower quality; including using an image similarity algorithm to identify and remove duplicate images, and using a quality assessment tool to exclude images of lower quality, thereby improving overall data quality;

[0064] Step S15: Write a Python program to cut each image to the same size (1000*1000), and divide the images into training set, validation set and test set according to the ratio of 7:1:2. This includes using the image processing library to adjust the image size, randomly divide the data set to ensure a reasonable sample distribution, and write a Python script to automatically divide the data set.

[0065] Step S2: Obtain objects by using the foreground segmentation algorithm on the imagenet dataset, and then use the image synthesis method to synthesize them into the farmland image data to construct a farmland image target detection dataset with obstacles; Figure 2 , the specific steps are as follows:

[0066] Step S21: Using the foreground segmentation algorithm, 50 categories related to farmland scenes are selected from the Imagenet dataset, and 10 images of each category are extracted. This embodiment uses the Imagenet dataset, and other relevant datasets can be selected according to specific needs. The foreground objects in these images are extracted by the foreground segmentation algorithm to obtain various foreground object images that may appear in the farmland;

[0067] Step S22: Using image synthesis technology, create a comprehensive farmland obstacle detection dataset. The specific description is as follows:

[0068] The image fusion algorithm described in this embodiment obtains common obstacle image data in farmland from the public data set ImageNet, performs foreground segmentation processing on the obstacle data image, and obtains obstacle foreground image data through SAM segmentation model (SegmentAnything, SAM); obtains farmland background image data from the public data set FieldSafe, performs background segmentation processing on the farmland background image, and then obtains farmland background image segmentation area data through SAM segmentation, thereby obtaining the position for placing the foreground obstacle image collage; uses the object placement algorithm to collage the obstacle foreground data on the farmland background data, and adjusts the size of the obstacle through perspective perception. Perspective perception refers to simulating the human eye's perception of three-dimensional scenes, making distant objects appear smaller than nearby objects, so that they can be more naturally integrated with the background.

[0069] Adjust the synthetic image in terms of color, lighting, etc. to ensure that the synthetic obstacles are consistent with the surrounding environment. Given a synthetic image and a foreground mask M, the model will first pattern the image. The input synthetic image and foreground mask M are converted into a harmonic image through the UNet generator G. The synthetic image contains an image of the obstacle and its surroundings, while the foreground mask indicates the location of the obstacle in the image. The task of the model is to harmonize the synthetic image so that the obstacle is consistent with the surrounding environment to achieve a more natural synthesis effect. A global discriminator is used to distinguish between real images and generated images, guiding the generator to generate more realistic images. The domain verification discriminator is used to verify the domain matching degree of the foreground and background in the generated image to ensure the domain similarity of the foreground and background. The generator is optimized according to the domain verification loss to improve the domain matching of the generated image.

[0070] Simulate rainy scenes by introducing raindrop effects into the image. This process involves generating raindrops of random positions, lengths, and transparency on the image, and adjusting the image brightness at these positions to simulate the visual effect of a rainy environment. Implement foggy simulation by introducing a foggy effect into the image to reduce visual clarity. This process involves adding randomly distributed fog particles to the image and adjusting their transparency and concentration to simulate the visual effect of a foggy day. Simulate night scenes by adjusting the brightness and hue of the image. This process involves reducing the image brightness, increasing the contrast of dark areas, and simulating night lighting conditions by introducing cool tones to produce a realistic night effect. Other noise generation, this step may include other image noise generation processes, such as adding light spots, lens vignetting and other effects to enrich the diversity and realism of the image.

[0071] The resulting farmland obstacle detection dataset has detailed image labels, including obstacle categories, locations, and obstacle image sizes. This dataset provides highly realistic and diverse image samples for the training of farmland target detection models.

[0072] Step S3: Perform knowledge distillation pre-training on the Imagenet dataset to compress the large model into a lightweight model;

[0073] Step S31: Construct the teacher model ViT-Base and the student model ViT-Tiny.

[0074] The ViT model consists of an input embedding layer, a position encoding layer, and an encoder layer.

[0075] The input embedding layer divides the image into patches and projects each patch into a vector, including a vector class token for image classification.

[0076] The position encoding layer includes an absolute position numerical representation for each token, and the position value is added to the token.

[0077] The encoder layer contains four important components, namely Layer Normalization, Multi-HeadAttention, Dropout, and MLP Block.

[0078] Layer Normalization is performed on the sample, calculating the mean and variance of all feature maps of the sample, and then normalizing the sample.

[0079] The Multi-Head Attention mechanism divides the input feature map into multiple heads (sub-heads), and each head learns different attention weights, so that the model can simultaneously focus on different local information of the input features. Each head will produce an independent attention map, and then these maps are fused together by weighted summation to form the final attention representation.

[0080] Dropout randomly discards the output of some neurons during training, which prevents the model from over-relying on specific neurons and forces the network to learn more robust feature representations. This helps reduce the risk of overfitting, enables the model to better adapt to different input data, and improves its performance on unseen data.

[0081] The MLP Block is a structure consisting of two fully connected layers, which is inserted before and after each Transformer block. This block introduces a richer feature representation by performing nonlinear transformations on the input features. The first MLP layer maps the input features to a high-dimensional space, then introduces nonlinear transformations through activation functions, and finally maps the features back to the original dimensions through the second MLP layer.

[0082] ViT-Base has a token with a final dimension of 768 after the input embedding layer, and has a 12-head Multi-Head Attention. ViT-Tiny has a token with a final dimension of 192 after the input embedding layer, and the first and tenth layers of Blocks have a 12-head Multi-Head Attention. The remaining layers only have a 6-head Multi-Head Attention.

[0083] Step S32: Pre-training is performed on the imagenet dataset. First, unlabeled images are used as input and data enhancement is used. The specific process is as follows:

[0084] This implementation assumes that the Tensor input to the i-th Transformer block is X i . If there are N Transformer Blocks, then i∈[1,N]. Each Transformer Block consists of a Multi-Head Self-Attention Layer, a Fully Connected Feed-Forward network, and a Layer Normalization. The i-th Transformer Block can be expressed as:

[0085] T i =MHSA(LN(X i-1 ),

[0086]

[0087]

[0088]

[0089] Among them, MHSA() stands for Multi-Head Self-Attention layer, LN() stands for LayerNormalization, and FFN() stands for Fully Connected Feed-Forward Network. represents the output of the i-th Transformer Block, It represents the value of residual addition after Multi-Head Self-Attention Layer. Represents the value after the fully connected layer.

[0090] Each Multi-Head Self-Attention Layer consists of K heads. Each head takes the input TensorX i Mapped into Query(Q),Key(K)and Value(V):

[0091]

[0092]

[0093]

[0094] in They represent Q, K, and V of the kth Head, respectively, and i represents the number of Transformer Blocks, where N = H × W / P2 Represents the number of Paths, where H, W represent the height and width of the image respectively, P represents Patch Stride, C represents the number of channels, and K represents the number of heads in Multi-head Self-Attetion.

[0095] Will It is defined as the Self-AttentionBehavior of the query-key in the i-th Transformer Block. The specific calculation method is: in Represents Q, K, V of the kth Head respectively. i represents the number of Transformer Blocks, where N = H × W / P 2 Represents the number of Paths, where H and W represent the height and width of the image respectively, P represents Patch Stride, C represents the number of channels, and K represents the number of heads in Multi-headSelf-Attetion. It is defined as the Self-Attention Behavior of value-value in the i-th TransformerBlock. The specific calculation formula is

[0096] Define the teacher model including K attention heads and calculate the difference between the teacher model and the student model. and The KL-divergence between is used as the loss function to optimize the model parameters so that the student model can imitate the self-attention mechanism of the teacher model. represents the self-attention mechanism of Query-KeyT of the k-th head of the teacher model, Represents the self-attention mechanism of Query-KeyT of the k-th head of the student model. The same method definition.

[0097] The loss formula is defined as:

[0098]

[0099]

[0100]

[0101] in The 1-1 represents the Self-AttentionBehavior loss of the first layer of the teacher model and the first layer of the student model. Represents the learning objective block of the teacher model and the last layer of the student model Self-Attention Behavior loss, L ALI It represents the loss calculation method of ALI in the first stage.

[0102] At this stage, the focus of knowledge distillation is to imitate the self-attention behavior between the teacher model (ViT-Base) and the student model (ViT-Tiny). This is achieved by calculating the Kullback-Leibler (KL) divergence between the self-attention distributions between the two. KL divergence can measure the degree of difference between two probability distributions, and here it is used to quantify how close the student model's behavior on the self-attention mechanism is to the teacher model.

[0103] Step S33: For C-type training samples, the classification probability p of the model can be expressed as

[0104]

[0105] Among them, p i is the probability of the ith category, The representative dimension is 1×C. Each element in p can be obtained by the softmax function: where z i ,z j They represent the logit of the i-th class and the logit of the j-th class respectively. The logit value represents the "confidence" of the model for each class. The larger the value, the higher the possibility that the model believes that it belongs to the class, and the smaller the value, the lower the possibility that the model believes that it belongs to the class. exp represents the exponential function. For generality, in this embodiment, the softmax function with temperature T is used. Defined as:

[0106]

[0107] For the target class t, the softmax function with temperature T for all other non-target classes It can be expressed as:

[0108]

[0109]

[0110] where z t represents the logit of the target class, z k represents the logit of the kth class. Meanwhile, in this embodiment, the probability of the non-target class is

[0111]

[0112] It can be calculated by the softmax function with temperature T:

[0113]

[0114] In this implementation, Te, St are used to represent the teacher model ViT-Base and the student model ViT-Tiny. The knowledge distillation loss function of the second distillation stage Logit Mimicking stage is:

[0115]

[0116] in Represents the teacher model ViT-Base Representative student model ViT-Tiny α represents a hyperparameter and is usually set to 3. We define the output probability of the student model as Ρ, the label information as Y, and the logit output of the student model as z St , the logit output of the teacher model is z Te , the logit value represents the model's "confidence" for each category. The larger the value, the higher the possibility that the model believes it belongs to that category, and the smaller the value, the lower the possibility that the model believes it belongs to that category. The temperature is T, and the loss function is: L = L CE (Ρ,Y)+L LM (z St ,z Te ,T), where Y represents the image label, L LM Represents the knowledge distillation loss function of the second distillation stage Logit Mimicking stage, L CE represents the cross entropy loss function and T represents the temperature.

[0117] In this stage, knowledge distillation not only focuses on the matching of self-attention behavior, but also introduces the KL divergence of the model prediction results (logit) as part of the loss function to encourage the student model to learn to imitate the final classification or regression decision of the teacher model. At the same time, in order to ensure the learning effect of the student model on the original data set, the cross-entropy loss between the student model output and the real data set label is also included in the overall loss function. Therefore, the total loss of the second stage is obtained by adding these two parts (the KL divergence loss and the cross-entropy loss of logit), aiming to enable the student model to inherit the generalization ability and decision-making wisdom of the teacher model, and to directly optimize for actual tasks and improve performance on image data sets.

[0118] Step S4: Use the lightweight model as the backbone network to adapt to the FPN component in the Mask R-CNN framework to build a detection model. Read the farmland image target detection dataset and use data enhancement to fine-tune the detection model on the constructed farmland dataset with obstacles;

[0119] Step S41: Integrate the selected lightweight model ViT-Tiny into the MaskR-CNN framework, replacing the original ResNet or other standard Backbone. Ensure that the intermediate layers of the lightweight model are properly connected for subsequent use;

[0120] Step S42: Add a Feature Pyramid Network (FPN) component on top of the output of the lightweight model ViT-Tiny. This includes selecting appropriate feature layers from the lightweight model and creating a feature pyramid through upsampling and downsampling operations;

[0121] Step S43: According to the characteristics of the lightweight model and FPN components, the network architecture is adjusted to ensure that the input size, number of channels, etc. can work together.

[0122] Feature Pyramid Network (FPN) is a neural network architecture used for tasks such as object detection and semantic segmentation. Its main purpose is to solve the problem of object detection at different scales. Typically, smaller objects may be easier to detect in high-level features of an image, while larger objects may be easier to detect in low-level features. The core idea of ​​FPN is to solve this problem by building a multi-scale feature pyramid in the network. Specifically, FPN builds a feature pyramid by adding additional connections to a convolutional neural network. These connections allow information to be obtained from feature maps at different levels, allowing the network to detect at different scales. FPN generates feature maps with different resolutions through top-down and bottom-up pathways. The top-down pathway generates lower resolution but richer semantic feature maps from higher-level feature maps through upsampling operations, while the bottom-up pathway obtains higher resolution but less semantic feature maps from the bottom layer. Finally, these feature maps are cascaded together to form a multi-scale feature pyramid that can be used for tasks such as object detection. In Mask R-CNN, FPN is used as part of the backbone network to extract image features and generate multi-scale feature pyramids. This enables the model to effectively detect objects at different scales and generate accurate object bounding boxes and segmentation masks. Mask R-CNN is a neural network model for object detection and segmentation, which is an extension of Faster R-CNN. Mask R-CNN can detect objects at the same time, so it can be used to detect obstacles. Mask R-CNN implements obstacle detection mainly in the following steps:

[0123] Region Proposal Network (RPN): First, use the RPN network to extract candidate object regions in the image. These candidate regions may contain obstacles to be detected.

[0124] Region Classification and Bounding Box Regression: For each candidate region, a classification network is used to classify it to determine whether it contains an object, and a bounding box is regressed to obtain the precise location of the object.

[0125] Step S44: Use the farmland image object detection dataset with obstacles to train the modified Mask R-CNN. Set an appropriate loss function, optimize the object detection and instance segmentation tasks at the same time, and perform model tuning, including learning rate adjustment, data enhancement, etc. Ensure that the performance and lightweight characteristics of the model are balanced during the training process. In this embodiment, in order to ensure that the model can effectively learn object detection and instance segmentation tasks during the training process, while maintaining the lightweight characteristics of the model, that is, to reduce the number of parameters and the amount of calculation of the model as much as possible while ensuring the accuracy of the model. By monitoring the performance indicators of the model on the validation set, such as object detection accuracy, segmentation accuracy, etc., the training effect of the model can be evaluated and adjusted as needed.

[0126] Step S45: Use the test set to evaluate the performance of the trained model. Check the accuracy and efficiency of the model on the object detection and instance segmentation tasks to ensure that the lightweight model meets the needs of practical applications while maintaining performance.

[0127] Step S5: Deploy the trained model to agricultural equipment with low computing resources.

[0128] Step S51: Select a deep learning inference framework suitable for low computing resource devices, which is PyTorch Mobile in this implementation;

[0129] Step S52: converting the ViT-Tiny model into a format suitable for the target reasoning framework. In this embodiment, the PyTorch model is converted into TorchScript.

[0130] Step S53: For agricultural equipment with low computing resources, use the inference hardware of the edge computing device;

[0131] Step S54: Deploy the converted model and reasoning framework to the agricultural equipment. The model is embedded in the embedded system of the equipment, using a single-board computer with ARM architecture and a range-finding camera to determine the distance, and objects within the target range are determined as obstacles.

[0132] Step S55: Integrate sensors and data streams into the reasoning system for agricultural tasks, ensuring that the model can receive and process data from agricultural equipment sensors for real-time reasoning.

[0133] Step S56: Conduct real-time reasoning tests in an actual farm environment to check the performance and accuracy of the model on agricultural equipment, and make adjustments and optimizations as needed.

[0134] Step S57: Deploy a remote monitoring and updating mechanism to remotely monitor the model performance and make necessary updates. This can be achieved through Internet of Things (IoT) technology to ensure that the model is always up to date.

[0135] The key innovation of the synthesis method of the present invention is that it adopts a comprehensive image generation framework. Through the organic combination of the image collage module, the foreground and background image pixel harmony module, and the image noise generation module, highly realistic and diverse farmland obstacle image generation is achieved. The image collage module not only collages the obstacle foreground image on the background, but also introduces precise perspective perception adjustment to simulate the natural effect of relatively small distant objects. This innovation makes the generated image closer to the real farmland scene and improves the realism and fidelity of the image. The foreground and background image pixel harmony module adopts the UNet generator and introduces the attention mechanism to enhance the attention of the generated model to the complex obstacles in the farmland scene. Through the introduction of the global discriminator and the domain verification discriminator, the module better optimizes the generated image and ensures the performance of the generated image in terms of domain matching and realism. The image noise generation module comprehensively considers different environmental factors, such as rainy days, foggy days, and nights, and enriches the diversity of the generated images by introducing a variety of noises and effects. This diversity makes the generated images more challenging and helps to improve the generalization ability of the model so that it can adapt to various complex farmland environments. The present invention adopts a comprehensive image generation framework, which includes an organic combination of multiple modules such as image collage, foreground and background image pixel harmony, and image noise generation. This comprehensive design enables each module to work together to optimize the generated image. By introducing mechanisms such as adversarial training and domain verification, the generated image is comprehensively promoted to achieve better results in terms of realism, diversity, domain matching, etc.

[0136] Second, we study an effective and efficient method to significantly improve the performance of ViT on small datasets, and reveal a new layer selection strategy for the teacher model on small datasets through extensive experiments. When trained from scratch with limited datasets, ViT has difficulty learning locality in images. To this end, we propose a knowledge distillation method combined with curriculum learning, so that the student model can learn from easy to difficult, first learning locality through low-level semantic features in the first stage, and then learning high-level semantic logit and label information in the second stage.

[0137] Finally, we propose a set of methods for model deployment, so that the model can be deployed on edge computing devices and updated according to different tasks.

[0138] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A farmland obstacle detection method for low computing resource equipment, characterized in that: The following steps are involved: S1, collect farmland image data and preprocess them; S2, select an image dataset and perform foreground segmentation on it, and then synthesize it with the farmland image data to obtain a farmland obstacle detection dataset; S3, build a detection model and perform pre-training of knowledge distillation on the image dataset selected in S2 to obtain a lightweight student model; S4, using the lightweight student model as the backbone network to build a detection model, and using the farmland obstacle detection dataset to fine-tune the detection model; S5. Deploy the configured model to agricultural equipment with low computing resources; The specific steps of S1 are: S101, batch-acquire farmland front image data photos, and perform uniform preprocessing on the photos; S102, classifying the acquired images, including assigning a scene label to each image; S103, based on the scene classified in S102, using the question-answering language model to conduct interactive question-answering; using the generated language description combined with the original image to generate a new image sample, and adjusting the generation parameters; S104, performing data cleaning to remove identical images and images of lower quality; S105, trimming the image samples after data cleaning, and dividing the images into a training set, a validation set, and a test set according to a quantity ratio of 7:1:2; The specific steps of S3 are: S301, build a teacher model and a lightweight student model; S302, the first stage of knowledge distillation, pre-training the constructed model on the public dataset selected in S201, using unlabeled images as input and using data augmentation; S303, performing knowledge distillation on the model again to obtain a lightweight student model after knowledge distillation; The specific steps of S4 are: S401. Integrate the processed lightweight student model into the Mask R-CNN framework, replacing the original ResNet, and ensure that the intermediate layers of the lightweight student model are properly connected for subsequent use; S402, adding an FPN component on the output of the lightweight student model, wherein the FPN component is a neural network architecture component for target detection and semantic segmentation tasks; S403. According to the characteristics of the lightweight student model and FPN components, adjust the network architecture to ensure that the input size and number of channels can work together; S404, using the farmland image object detection dataset with obstacles to train the modified Mask R-CNN; S405. Use the test set to evaluate the trained modified Mask R-CNN model, check the accuracy and efficiency of the model in object detection and instance segmentation tasks, and ensure that the lightweight student model meets the needs of practical applications while maintaining performance; The specific steps of S5 are: S501, selecting a deep learning inference framework suitable for low computing resource devices; S502, converting the lightweight student model into a format suitable for the target reasoning framework; S503. For agricultural equipment with low computing resources, use the inference hardware of edge computing devices; S504, embed the model into the embedded system of the equipment, use a single-board computer with ARM architecture, and use a ranging camera to determine the distance, and determine the objects within the target range as obstacles.

2. The farmland obstacle detection method for low computing resource equipment according to claim 1, characterized in that: The specific steps of S2 are: S201, selecting a public image dataset, wherein the public image dataset includes image samples related to farmland scenes; selecting categories related to farmland scenes in the public image dataset, extracting some images in each category, extracting foreground objects in these images through a foreground segmentation algorithm, and obtaining a foreground mask M; S202, synthesizing the image obtained in S1 with the image segmented in S201 to obtain a farmland obstacle detection data set; the synthesis method includes an object placement algorithm, an image fusion algorithm, an image harmonization algorithm and a noise generation algorithm.

3. The farmland obstacle detection method for low computing resource equipment according to claim 2, characterized in that: The image synthesis process of S202 is specifically as follows: S2011, obtaining farmland background image data from another public data set, performing background segmentation processing on the farmland background image, and then obtaining farmland background image segmentation area data through SAM segmentation, thereby obtaining the location where the foreground obstacle image collage is placed; collage the obstacle foreground data obtained in S201 on the farmland background image segmentation area data, and adjust the size of the obstacle through perspective perception; S2012, adjust the color and lighting of the synthesized image to ensure that the synthesized obstacles are consistent with the surrounding environment; then use the UNet generator G to convert the synthesized image and the foreground mask M into a harmonious image, use the global discriminator to distinguish the real image from the generated image, guide the generator to generate realistic images, use the domain verification discriminator to verify the domain matching degree of the foreground and background in the generated image, and optimize the generator according to the domain verification loss; S2013, simulating rainy day, foggy day and night scenes on the adjusted image; and adding image noise to the image, wherein the image noise includes light spots and lens vignetting; S2014 outputs a farmland obstacle detection dataset with detailed image labels, where the image labels include the category, location and size of the obstacle image.

4. The farmland obstacle detection method for low computing resource equipment according to claim 1, characterized in that: The teacher model and the lightweight student model in S301 each include an input embedding layer, a position encoding layer, and an encoder layer; the specific working method is as follows: The embedding layer will be used to divide the image into patches and project each patch into a vector token, including a vector class token for image classification. The position encoding layer is used to have an absolute position digital representation for each token, and the position value is added to the token; The encoder layer includes: A normalization module, used to calculate the mean and variance of all feature maps of a sample, and then normalize the sample; The multi-head attention mechanism divides the input feature map into multiple heads, each head learns different attention weights, so that the model can focus on different local information of the input features at the same time; each head will produce an independent attention map, and then these maps are fused together by weighted summation to form the final attention representation; Regularization module, which is used to randomly discard the output of some neurons during training to prevent the model from over-relying on specific neurons and force the network to learn more robust feature representations; The multi-layer perceptron module includes two fully connected layers, which introduces richer feature representation by performing nonlinear transformation on the input features. Specifically, the first fully connected layer maps the input features to a high-dimensional space, then introduces nonlinear transformation through the activation function, and the second fully connected layer maps the features back to the original dimension. The last dimension of the token of the teacher model after the input embedding layer is 768, and each encoder layer has a multi-head attention mechanism with 12 heads; The last dimension of the token of the lightweight student module after the input embedding layer is 192. The first and tenth layers of the encoder layer are multi-head attention mechanisms with 12 heads, and the remaining layers are multi-head attention mechanisms with 6 heads.

5. The farmland obstacle detection method for low computing resource equipment according to claim 1, characterized in that: In S302, the first knowledge distillation process is to calculate the KL divergence of the self-attention behavior between the teacher model and the lightweight student model, and then use it as the loss function to train the model.

6. The farmland obstacle detection method for low computing resource equipment according to claim 1, characterized in that: In the S302, the process of the second knowledge distillation is to first obtain the cross entropy loss based on the classification probability and label information output by the lightweight student model; then calculate the KL discreteness of the logit values ​​of the lightweight student model and the teacher model; then add the cross entropy loss to the KL discreteness loss of the logit values ​​of the lightweight student model and the teacher model; finally, the loss obtained by adding the two losses is used as the loss function of the second distillation to train the lightweight student model; and output the lightweight student model after knowledge distillation.

Images