A dual-mode weeding method and weeding robot based on machine vision

Abstract

The present application relates to the field of machine vision and weeding technology, a kind of double mode weeding method based on machine vision, including obtaining image data, the image data obtained is uploaded to cloud;In cloud, whether there is an obstacle in image data is judged, if there is an obstacle, the position information and height information of the obstacle are judged;If there is no obstacle, whether there is weed in image data is judged;When judging that there is weed in image data, the category number of weed and the terrain where it is located are judged;When judging that there is no weed in image data, the angle of the device for obtaining image data is adjusted, and image data is reacquired;When judging the category number of weed and the terrain where it is located are a large number of weeds or flat terrain, a first weeding strategy is used;When judging the category number of weed and the terrain where it is located are a small number of weeds or irregular terrain, a second weeding strategy is used.The present application also proposes a kind of double mode weeding robot based on machine vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision and weeding technology, in particular to a dual-mode weeding method and robot based on machine vision. BACKGROUND

[0002] With the increasing importance of forest management, forest weeding, as a key link in maintaining forest ecological balance and promoting tree growth, is crucial to improving forest economic benefits and maintaining ecological system balance. Weeds compete for sunlight, water, nutrients and other resources, and also occupy the growth space of vegetation, which may become a breeding ground for pests and diseases, threatening the stability of the forest ecosystem. Every year, a large amount of manpower and material resources need to be invested in the prevention and cleaning of forest weeds. Traditional manual weeding is not suitable due to its long time-consuming, low efficiency and high labor cost. Chemical weeding is fast and efficient, but long-term use will damage the forest environment, pollute the soil and lead to weed resistance; biological weeding is also limited. There are some intelligent weeding robots on the market that focus on lawns and farmland, but these robots do not fully consider the unique environment and vegetation characteristics of forest land. The complex terrain, rich vegetation types and different soil characteristics from farmland in forest land make the existing technology unsatisfactory in forest weeding.

[0003] The main problems of current technology include poor adaptability, existing robots cannot effectively adapt to the diverse environment of forest land, and it is difficult to accurately identify seedlings and weeds in dense vegetation in the forest. Traditional robots may not be able to perform real-time image data analysis and processing, limiting the weeding effect. There are many different types of weeds in forest land, which have different growth habits, heights and structures. The mechanical weeding method used by traditional weeding robots is too single, lacking the ability to handle different weeds. SUMMARY

[0004] To solve the above problems, the present application proposes a dual-mode weeding method based on machine vision, which can be applied to a weeding robot. For complex forest environments, the method uses machine vision to identify weeds, distinguish between seedlings and weeds, and uses a mechanical arm weeding method or a grass cutting blade weeding method to weed according to the different types, heights and structures of weeds. The present application also proposes a dual-mode weeding robot based on machine vision.

[0005] To achieve the above purpose, the technical solution adopted by the present application is:

[0006] In the first technical solution, a dual-mode weeding method based on machine vision includes the following steps,

[0007] Step 1, acquire image data and upload the acquired image data to the cloud;

[0008] Step 2, judge whether there is an obstacle in the image data in the cloud, if there is an obstacle, judge the position information and height information of the obstacle, and return the position information and height information of the obstacle; if there is no obstacle, judge whether there is grass in the image data;

[0009] Step 3, when judging that there is grass in the image data, judge the number of categories and the terrain where the grass is located; when judging that there is no grass in the image data, adjust the angle of the image data acquisition device and reacquire the image data;

[0010] Step 4, when the number of categories and the terrain where the grass is located are large amount of grass or flat terrain, use the first weeding strategy; when the number of categories and the terrain where the grass is located are small amount of grass or irregular terrain, use the second weeding strategy.

[0011] In the first technical solution, as a preferred, the first weeding strategy is motor mowing, and the second weeding strategy is mechanical arm weeding.

[0012] In the first technical solution, as a preferred, the analysis of the acquired image data in the cloud includes: using a deformable convolution to correct the original feature map to improve the adaptability of the algorithm in weed detection, including:

[0013] Step 11, offset learning: deformable convolution learns a set of offsets including learnable parameters; the number of offsets is associated with the size and channel number of the convolution kernel, and each position and channel has a corresponding offset;

[0014] Step 12, offset application: deformable convolution uses the learned offset to adjust the sampling position of the convolution kernel before performing the convolution operation;

[0015] Step 13, sampling point interpolation: deformable convolution uses interpolation technology to obtain the feature value of the corresponding position for the adjusted sampling position;

[0016] Step 14, convolution operation: perform standard convolution operation on the adjusted feature map, in which the feature value of the sampling point is multiplied by the weight of the corresponding position in the convolution kernel, and the sum is obtained. The final output feature value.

[0017] In the first technical solution, as a preferred, in step 12, for each sampling point in the convolution kernel, its position is adjusted according to the corresponding offset, so that the convolution kernel can more accurately capture features in a specific area.

[0018] In the first technical solution, as preferred, in step 12, bilinear interpolation or trilinear interpolation is used for interpolation operation to obtain more accurate feature values.

[0019] In the first technical solution, as preferred, the analysis of the obtained image data in the cloud includes:

[0020] According to the distribution of the target features learned by the network, the fusion weights between different scale feature maps are dynamically adjusted, so that the network can better adapt to different data sets and scenes, including

[0021] Step 21, feature pyramid network construction: first, a feature pyramid network is constructed, a basic network is used as the backbone network, lateral connections are added to the network to combine high-resolution feature maps of the bottom layer network with semantic information of the top layer network, and a multi-scale feature pyramid is generated;

[0022] Step 22, target feature distribution learning: in the training phase, the back propagation algorithm and the loss function are used to optimize the network parameters, and the network is supervised to learn the distribution of the target features, the network learns the distribution of the target features in each feature map, the importance and distribution density of the target under different scales;

[0023] Step 23, adaptive feature fusion: in each stage of generating the feature pyramid, the learned target feature distribution is used to dynamically adjust the fusion weights between different scale feature maps, the weight is a scalar value or a vector, which is used to adjust the weight of each channel; The adjustment process introduces learnable parameters such as attention mechanism; In the process of feature fusion, the contribution of different scale feature maps will be dynamically adjusted according to the distribution of the target features, so that the network can better capture the features of the target;

[0024] Step 24, training and optimization: the network structure after adaptive feature fusion is trained and optimized, in the training process, the back propagation algorithm and the optimizer are used to update the network parameters, so that the prediction results of the network are closer to the real labels;

[0025] Step 25, inference phase: through the trained and optimized model, the input image is subjected to target detection or semantic segmentation, and the adaptive feature fusion weight learned in the training process is used to dynamically adjust the fusion of the feature map, so as to obtain the final detection result.

[0026] In the first technical solution, as preferred, the analysis of the obtained image data in the cloud includes:

[0027] Using the target detection algorithm YOLOv5, including:

[0028] Model parallel computing: the target detection task in YOLOv5 can be divided into multiple small tasks and executed in parallel on different processing units; through model parallel computing, different parts of the model are allocated to different processing units, and target detection is performed in parallel;

[0029] Data parallel computing: divide the data into multiple batches, and process each batch in parallel on different processing units;

[0030] Pipeline parallel computing: divide the image processing task into multiple stages, and use pipeline technology to execute these stages in parallel; in YOLOv5, image preprocessing, feature extraction, target detection, etc. can be processed respectively, and executed in sequence on the pipeline to improve the overall processing speed.

[0031] In the second technical solution, a dual-mode weeding robot based on machine vision comprises

[0032] A vehicle;

[0033] A motorized mowing assembly arranged at the front of the vehicle in the direction of travel;

[0034] A weed pulling robotic arm arranged at the rear of the vehicle.

[0035] In the second technical solution, as a preferred embodiment, the vehicle comprises a lower bottom plate and an upper bottom plate, the lower bottom plate and the upper bottom plate are arranged in a spaced manner, and the motorized mowing assembly is installed at the front of the vehicle and located between the upper bottom plate and the lower bottom plate.

[0036] The swing arm suspension is arranged between the four corners of the upper bottom plate and the lower bottom plate and located at the corners, each of the suspensions is installed with a Mecanum wheel, and each of the Mecanum wheels is installed with a driving motor.

[0037] In the second technical solution, as a preferred embodiment, the weed pulling robotic arm comprises a rotatable base, and four rotatable joints are arranged on the rotatable base to enable the main body of the weed pulling robotic arm to be bent, the end of the main body of the weed pulling robotic arm is a rotary joint, and the action end of the rotary joint is an execution end with a clamping function.

[0038] The beneficial effects of the present application are:

[0039] The dual-mode weeding method based on machine vision proposed in the present application is applied to a weeding robot, and for complex forest environments, the method uses machine vision to identify weeds, distinguishes between saplings and weeds, and uses a mechanical arm to pull weeds or a mowing blade to mow weeds according to different types, heights and structures of weeds.

[0040] The machine vision-based dual-mode weeding robot has all-terrain adaptability, and has two weeding modules, so that the appropriate weeding module can be used according to actual needs, and the effect of rapid weeding is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of the machine vision-based dual-mode weeding method.

[0042] Figure 2 The deformable convolution is introduced to correct the original feature map.

[0043] Figure 3 A schematic diagram of the original feature pyramid network structure of the detection algorithm.

[0044] Figure 4 A schematic diagram of using the AF-FPN structure to enhance the feature pyramid network.

[0045] Figure 5 A comparison chart of the results of traditional YOLOv5, adding deformable convolution, replacing AF-FPN, and comprehensive results.

[0046] Figure 6 A flowchart of Cartesian space interpolation.

[0047] Figure 7 A schematic diagram of the machine vision-based dual-mode weeding robot.

[0048] Figure 8 A schematic diagram of the weeding mechanical arm.

[0049] The reference signs include:

[0050] 10-vehicle, 11-lower floor, 12-upper floor, 13-swing arm suspension, 14-Mecanum wheel;

[0051] 20-motor weeding assembly, 21-cutting motor, 22-blade device,

[0052] 30-weeding mechanical arm, 31-first joint, 32-second joint, 33-third joint, 34-fourth joint, 35-fifth joint, 36-sixth joint, 37-execution end. DETAILED DESCRIPTION

[0053] To make the purpose, technical scheme and advantages of the technical solution clearer and more intelligible, the technical solution will be further described in detail below in combination with specific embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the technical solution.

[0054] Example 1

[0055] As Figure 1As shown, the embodiment proposes a dual-mode weeding method based on machine vision, including the following steps,

[0056] Step 1, acquire image data, upload the acquired image data to the cloud; Step 2, judge whether there is an obstacle in the image data in the cloud, if there is an obstacle, judge the position information and height information of the obstacle, and return the position information and height information of the obstacle; if there is no obstacle, judge whether there is weed in the image data; Step 3, when it is judged that there is weed in the image data, judge the category number and the terrain where the weed is located; when it is judged that there is no weed in the image data, adjust the angle of the image data acquisition device, and reacquire image data; Step 4, when the category number and the terrain where the weed is located are a large amount of weeds or flat terrain, use the first weeding strategy; when the category number and the terrain where the weed is located are a small amount of weeds or irregular terrain, use the second weeding strategy.

[0057] The method is applied to a weeding robot, and for complex forest environments, machine vision is used for weed identification, distinguishing between saplings and weeds, and using a mechanical arm weeding method or a grass cutting blade weeding method for weeding according to different types, heights and structures of weeds.

[0058] The key of the method is to reasonably plan the decision through the machine vision recognition mode, and use the corresponding strategy. The key technologies include:

[0059] 1. Introducing deformable convolution to correct the original feature map, improving the adaptability of the algorithm in plant detection.

[0060] 2. Using AF-FPN structure to enhance the feature pyramid network, improving the adaptability and robustness of the improved algorithm to different scale targets;

[0061] 3. Using advanced deep learning technology can improve the accuracy of target detection and classification;

[0062] 4. Through training the deep neural network, the system can learn and adapt to complex forest scenes. At the same time, through the algorithm processing the image data collected by the camera, factual image processing can be realized, through efficient algorithm and parallel computing, the image processing speed is improved, so as to be suitable for dynamic and fast changing environment. The contents of each part are described in detail as follows.

[0063] For example, Figure 3As shown in the figure, in the part of introducing deformable convolution to correct the original feature map, in the target detection process, anchor boxes are usually used to model the target to adapt to different scales and aspect ratios. However, when the width and height of the anchor box change, the relationship between the original anchor box feature mapping and the corresponding grid position may lose correspondence. In order to solve this problem, this paper adopts deformable convolution to correct the original feature map.

[0064] Deformable Convolution is a special type of convolution operation that introduces a mechanism of deformable sampling based on traditional convolution operation, so that the model can adaptively adjust the position of the sampling point, so as to better capture the local features of the target object. The specific workflow of deformable convolution is as follows:

[0065] Offset learning: In the training stage, deformable convolution learns a set of offsets, which are learnable parameters. The number of offsets is usually related to the size and channel number of the convolution kernel, and each position and channel has a corresponding offset.

[0066] Offset application: Before performing convolution operation, deformable convolution uses the learned offsets to adjust the sampling position of the convolution kernel. Specifically, for each sampling point in the convolution kernel, it is adjusted according to the offset of the corresponding position, so that the convolution kernel can more accurately capture the features in a specific area.

[0067] Sampling point interpolation: For the adjusted sampling position, deformable convolution uses interpolation techniques to obtain the feature value of the corresponding position. Usually bilinear interpolation or trilinear interpolation is used for interpolation operation to obtain more accurate feature values.

[0068] Convolution operation: Finally, the adjusted feature map is subjected to standard convolution operation. In the convolution operation, the feature values of the sampling points are multiplied by the weights of the corresponding positions in the convolution kernel, and the sum is obtained to get the final output feature value.

[0069] Through the above steps, deformable convolution introduces two-dimensional displacement at each sample point, so that the convolution operation can deform the feature map in any shape. This method can improve the modeling ability of the model for the deformation of the target object, so that it can better adapt to targets of various scales, shapes and angles. By using deformable convolution, the model can more accurately capture the detailed information and shape changes of the target, thereby improving the accuracy and robustness of target detection.

[0070] As Figure 4As shown, FPN-PAN is the original feature pyramid network structure of the detection algorithm. It is composed of two parts: Feature Pyramid Network (FPN) and Path Aggregation Network (PAN). The role of FPN is to generate multi-scale feature maps, so that the network can perform target detection at different scales. FPN builds a feature pyramid by adding lateral connections in the network, combining high-resolution feature maps of the bottom network with semantic information of the top network, forming a multi-scale feature pyramid. The role of PAN is to fuse the multi-scale feature maps generated by FPN to better capture the multi-scale information of the target. PAN aggregates feature maps of different scales by introducing cross-stage connections and local connections, and generates the final detection results by step-by-step aggregation and upsampling.

[0071] Although FPN-PAN structure performs well in target detection tasks, it also has some shortcomings, one of which is the fixed feature fusion weight. The feature fusion weight in FPN-PAN is usually fixed, that is, it will not be adjusted during network training. This may cause the weight to be less adaptive in different datasets or scenarios, affecting the performance of the model.

[0072] AF-FPN (Adaptive Feature Pyramid Network) is an improved method based on traditional feature pyramid network, which introduces adaptive attention module and feature enhancement module to improve feature representation and performance. It solves the problem of fixed feature fusion weight in traditional FPN-PAN structure. Specifically, AF-FPN dynamically adjusts the fusion weight between different scale feature maps according to the distribution of target features learned by the network, so that the network can better adapt to different datasets and scenarios. This adaptability can improve the robustness and generalization ability of the network, so that the network can achieve better results in handling various complex scenarios. The main advantage of AF-FPN is to enhance the performance of the feature pyramid network, so that the performance of tasks such as target detection and semantic segmentation is further improved. The following is the detailed workflow:

[0073] Feature Pyramid Network (FPN) construction: First, construct the feature pyramid network, usually use a basic network as the backbone network, then add lateral connections in the network, combine the high-resolution feature maps of the bottom network with the semantic information of the top network, generate a multi-scale feature pyramid. This process produces a series of feature maps with different resolutions and semantic information.

[0074] Target Feature Distribution Learning: During the training phase, a backpropagation algorithm and a loss function are used to optimize the network parameters while supervising the network to learn the distribution of target features. In this process, the network learns the distribution of target features in each feature map, i.e., the importance and distribution density of the target at different scales.

[0075] Adaptive Feature Fusion: At each stage of generating the feature pyramid, the learned distribution of target features is used to dynamically adjust the fusion weights between different scale feature maps. This weight can be a scalar value or a vector, adjusting the weight of each channel. Usually, this adjustment process can be achieved by introducing some learnable parameters such as attention mechanisms. In this way, during the feature fusion process, the contribution of different scale feature maps will be dynamically adjusted according to the distribution of target features, so that the network can better capture the features of the target.

[0076] Training and Optimization: After the adaptive feature fusion of the network structure, training and optimization are performed to make it better adapt to the target detection or semantic segmentation task. During the training process, the backpropagation algorithm and optimizers (such as SGD, Adam, etc.) are usually used to update the network parameters, so that the network's prediction results are closer to the true labels.

[0077] Inference Phase: In the inference phase, the trained and optimized model is used to perform target detection or semantic segmentation on input images. In this process, the adaptive feature fusion weights learned during training are used to dynamically adjust the fusion of feature maps, resulting in the final detection results.

[0078] Through these improvements, the detection performance and robustness of the target detection algorithm in forest scenes are improved.

[0079] As shown in Figure 5 , the image processing speed is improved through efficient algorithms and parallel computing.

[0080] Efficient Algorithm:

[0081] Optimized Algorithm Implementation: Some optimization strategies are adopted in the improved YOLOv5, such as the optimization of network structure design, to improve the execution efficiency of the model. This helps to improve the processing speed of the model;

[0082] Real-time Processing Optimization: One of the design goals of YOLOv5 is to achieve real-time target detection, so during the model design and training process, special attention is paid to the optimization of processing speed. Through optimization of model structure, parameter settings, etc., high real-time processing performance can be achieved.

[0083] Parallel Computing:

[0084] Model Parallelism: The object detection task in YOLOv5 can be broken down into multiple small tasks that can be executed in parallel on different processing units. By model parallelism, different parts of the model are assigned to different processing units, such as GPUs or multiple CPU cores, and the object detection is performed in parallel. For example, the image is divided into multiple small blocks, and the object detection task is executed on multiple GPUs in parallel, and then the results are merged.

[0085] Data Parallelism: The data is divided into multiple batches, and each batch is processed in parallel on different processing units. In YOLOv5, the image data is divided into multiple batches, and the object detection task is performed simultaneously on multiple GPUs. This way, the parallel computing capabilities of the GPU can be fully utilized to speed up the entire image processing process.

[0086] Pipeline Parallelism: The image processing tasks are divided into multiple stages, and the pipeline technology is used to execute these stages in parallel. In YOLOv5, image preprocessing, feature extraction, and object detection tasks can be handled separately and executed sequentially on the pipeline to improve overall processing speed. For example, image preprocessing can be performed on one GPU, and then the preprocessed image data is passed to another GPU for feature extraction and object detection.

[0087] As shown in Figure 6 , the effect comparison after improving the algorithm.

[0088] 1、Effect evaluation index

[0089] mAP (Mean Average Precision) and F1 Score are two important indicators for evaluating the performance of object detection tasks.

[0090] mAP (Mean Average Precision) is an indicator that comprehensively evaluates the precision of the target detection model under different categories. It first calculates the Average Precision (AP) of each category, and then averages the AP of all categories to obtain mAP. When calculating AP, the area under the Precision-Recall curve is usually used to measure the accuracy of the detection results. The range of mAP is usually between 0 and 1, and the higher the value, the better the model performance.

[0091] F1 Score is an index that takes into account the precision and recall of the model, especially suitable for unbalanced classes. F1 Score is the harmonic mean of Precision and Recall, and the formula is F1=2*(Precision*Recall) / (Precision+Recall). The value of F1 Score ranges from 0 to 1, and the higher the value, the better the performance of the model.

[0092] These two indicators are often used together in the target detection task. mAP can evaluate the overall performance of the model in different categories, while F1 Score can provide more detailed evaluation to help identify the accuracy and recall of the model when detecting targets.

[0093] The results of the above are shown in Table 1 below.

[0094]

[0095] Table 1 is a comparison table of the results of traditional YOLOv5, adding deformable convolution, replacing AF-FPN, and comprehensive.

[0096] As shown in Figure 6 , wherein yolov5 is the original target detection algorithm model, Deformable Convolution is the model after adding deformable convolution, AF-FPN is the model after replacing the neck network, and sum is the comprehensive model, that is, the model after adding deformable convolution and replacing the backbone network with AF-FPN.

[0097] The dual-mode weeding method based on machine vision proposed by the application is applied to a weeding robot, and for complex forest environments, machine vision is used for weed identification, distinguishing between saplings and weeds, and weeds are removed by using a mechanical arm weeding method or a grass cutting blade weeding method according to different types, heights and structures of weeds.

[0098] Example 2

[0099] The embodiment proposes a dual-mode weeding robot based on machine vision, which includes a vehicle 10, a motor grass cutting assembly 20 arranged at the front of the vehicle 10 in the walking direction, and a weeding mechanical arm 30 arranged at the rear of the vehicle 10.

[0100] As preferred, the vehicle 10 comprises a lower floor 11 and an upper floor 12, the lower floor 11 and the upper floor 12 are arranged in a spaced manner, the electric mowing assembly is installed at the front of the vehicle 10 and located between the upper floor 12 and the lower floor 11; the swing arm suspensions 13 are arranged between the four corners of the upper floor 12 and the lower floor 11 and located at the corners, each suspension is provided with a Mecanum wheel 14, and each Mecanum wheel 14 is provided with a driving motor.

[0101] The grass pulling mechanical arm 30 comprises a rotatable base, and four rotatable joints are arranged on the rotatable base to enable the main body of the grass pulling mechanical arm 30 to be bent, the end of the main body of the grass pulling mechanical arm 30 is a rotating joint, and the action end of the rotating joint is an execution end 37 with a clamping function.

[0102] The advantage of the mechanical arm for pulling grass is that it can remove weeds at a fixed point to ensure that the surrounding vegetation is not damaged. At the same time, the forest land usually has an irregular terrain, and the mechanical arm can adapt to and work in different terrain conditions, including slopes, uneven ground, etc. However, the efficiency of the mechanical arm for pulling grass is relatively low.

[0103] The advantage of the cutting motor 21 cooperating with the sawtooth operating disc for mowing grass is that it can quickly cut off the grass clumps, especially the shrubs and low vegetation that may exist in the forest floor. At the same time, the motor mowing system can be used for trimming the edges of the forest land, maintaining the vegetation beside the forest land roads, and mowing the lawn when needed. However, the motor mowing system has insufficient processing capacity for specific terrains and different structures of weeds.

[0104] When the two mowing methods are combined together, the mechanical arm can remove weeds on slopes and uneven ground at a fixed point, and the motor mowing system can remove weeds in a large area to improve efficiency. The two mowing methods are selected according to the types and quantities of weeds and the terrain, which improves the working efficiency and reduces the disturbance to the ecological system, and ensures the natural balance of the forest land.

[0105] The mechanical arm for pulling grass is a key technology in the dual-mode weeding. The path planning algorithm adopts Cartesian space interpolation. Cartesian space interpolation is a method of planning the path of a mechanical arm in a Cartesian coordinate system. In the scenario of pulling weeds, Cartesian space interpolation can be used to plan the motion path of the execution end 37 of the mechanical arm to ensure that the mechanical arm can smoothly and accurately move to the target position to pull out the weeds.

[0106] As Figure 6As shown, the above-mentioned Cartesian space interpolation process is to first determine the starting point of the robot arm, i.e. the initial position; determine the target point, i.e. the position where the weeds need to be removed. A series of interpolation points are set between the starting point and the target point, which will constitute the path of the robot arm. The number and density of interpolation points are determined according to the degree of smooth path required. Then use interpolation methods such as linear interpolation or spline interpolation to generate a smooth path between the set interpolation points. According to the generated interpolation path, a motion control algorithm is designed to control the movement of the end 37 of the robot arm. The control algorithm usually includes speed planning, acceleration planning, etc. to ensure the smooth movement of the robot arm on the path. During the generation of the interpolation path, collision detection and obstacle avoidance problems in the environment need to be considered. Collision detection is performed on each point on the interpolation path to ensure that the robot arm does not collide with obstacles during movement. During the movement of the robot arm, the interpolation path needs to be adjusted in real time to adapt to the changing environment or work requirements. Real-time adjustment can be achieved through sensor feedback and control algorithm. During the weeding process, the motion state of the robot arm and the environmental conditions are monitored in real time. Through this process, the robot arm can plan and execute the task of removing weeds in the Cartesian space, while avoiding collisions and ensuring efficient motion trajectories.

[0107] As shown in Figure 7 and Figure 8 The weeding robot includes a vehicle 10, a weeding device mounted on the vehicle 10, a positioning and obstacle avoidance mechanism, and a camera for collecting image data. The weeding device includes a robot arm and a blade device 22, the positioning and obstacle avoidance mechanism includes an ultrasonic device and a GPS positioning device, and the visual perception device includes a camera and a gateway for uploading image data to the cloud. The motion mechanism is provided with a control module, an information receiving module and an information processing module. The control module includes a motion module, a weeding module, a positioning and obstacle avoidance module, and a visual control module; the motion module, the weeding module, the positioning and obstacle avoidance module, and the visual control module are respectively used to control the motion mechanism, the weeding device, the positioning and obstacle avoidance device, and the visual perception module. The information receiving module is used to receive information of each module; the information processing module is used to process the received information data.

[0108] The weeding device includes a six-degree-of-freedom weeding robot arm 30 and a motor mowing assembly 20.

[0109] The weeding robot arm 30 includes a base, a first joint 31, a second joint 32, a third joint 33, a fourth joint 34, a fifth joint 35, a sixth joint 36, and an execution end 37. The base is an aluminum alloy base, the first joint 31 to the sixth joint 36 are rudders, and the execution end 37 is a mechanical claw. The blade device 22 controlled by the cutting motor 21 includes a two-degree-of-freedom rudder holder and a mowing blade controlled by the cutting motor 21.

[0110] The mechanical arm 30 for pulling out weeds and the motor mower assembly 20 are respectively fixed on two sides of the motion mechanism, and different devices are selected for weeding according to the identification of the types and quantities of weeds. The positioning and obstacle avoidance mechanism includes an ultrasonic device and a GPS positioning device, the ultrasonic device is fixed at the front end of the motion mechanism, and the GPS positioning device is fixed in the motion mechanism. The visual perception device includes a camera and a gateway, there are two cameras, camera 1 is fixed on the third joint 33 of the mechanical arm, camera 2 is fixed on the two-dimensional holder of the motor blade, and the gateway is fixed in the motion mechanism. The motion mechanism includes a Mecanum wheel 14 base, four Mecanum wheels 14 with motors and a motor drive module.

[0111] The robot motion structure adopts a Mecanum wheel 14 car chassis, which is composed of four groups of motor supports, four groups of 24V reduction motors, four Mecanum wheels 14, an aluminum alloy upper bottom plate 12 and a lower bottom plate 11, a double-pass aluminum column and a swing arm suspension 13. The load capacity of the reduction motor is 20kg, and the encoder can realize closed-loop PID control, realize more accurate position or speed control, and improve the stability and responsiveness of the robot system; the Mecanum wheel 14 has omnidirectional movement, high mobility, simple control, no need for a steering mechanism, stable movement, multi-point contact and strong adaptability; the swing suspension system improves the adhesion of the vehicle 10, adapts to different road conditions, enhances the off-road performance, and improves the controllability and stability.

[0112] During the forward, backward, left, right and rotation processes of the car, the motor speed of each Mecanum wheel 14 is adjusted, the control system calculates and transmits the corresponding speed vector to each motor in real time. The motor rotates according to these vectors to drive the wheels to perform corresponding movements, realizing various flexible movements of the car on the plane. The sensor provides real-time feedback, and the embedded computing unit is responsible for coordinating motion control to ensure that the car maintains the expected trajectory and stability during movement.

[0113] The dual-mode weeding robot based on machine vision has full-terrain adaptability, and has two weeding modules, so that the appropriate weeding module can be used according to actual needs, and the effect of rapid weeding is achieved.

[0114] The above is only a preferred embodiment of the present application, and those skilled in the art can make many changes in specific implementation and application range according to the technical content of the present application, as long as these changes do not deviate from the concept of the present application, and belong to the protection scope of the present patent.

Claims

1. A dual mode weeding method based on machine vision, characterized in that: The method comprises the following steps: Step 1, acquiring image data, uploading the acquired image data to the cloud; Step 2, judging whether there is an obstacle in the image data in the cloud, if there is an obstacle, judging the position information and height information of the obstacle, and returning the position information and height information of the obstacle; if there is no obstacle, judging whether there is grass in the image data; Step 3, when it is judged that there is grass in the image data, judging the number of categories of the grass and the terrain where the grass is located; When it is judged that there is no grass in the image data, adjusting the angle of the image data acquisition device and reacquiring image data; Step 4, when the number of categories of the grass and the terrain where the grass is located are a large amount of grass or a flat terrain, using a first weeding strategy; when the number of categories of the grass and the terrain where the grass is located are a small amount of grass or an irregular terrain, using a second weeding strategy; The first weeding strategy is motor mowing, and the second weeding strategy is mechanical arm weeding; The analysis of the acquired image data in the cloud comprises: using deformable convolution to correct the original feature map to improve the adaptability of the algorithm in grass detection, comprising: Step 11, offset learning: deformable convolution learns a set of offsets including learnable parameters; the number of offsets is associated with the size and channel number of the convolution kernel, and each position and channel has a corresponding offset; Step 12, offset application: deformable convolution uses the learned offset to adjust the sampling position of the convolution kernel before performing the convolution operation; Step 13, sampling point interpolation: deformable convolution uses interpolation technology to obtain the feature value of the corresponding position for the adjusted sampling position; Step 14, convolution operation: performing standard convolution operation on the adjusted feature map, in which the feature value of the sampling point is multiplied by the weight of the corresponding position in the convolution kernel, and the sum is obtained to obtain the final output feature value; The analysis of the acquired image data in the cloud comprises: According to the distribution of the target features learned by the network, the fusion weights between different scale feature maps are dynamically adjusted, so that the network can better adapt to different data sets and scenes, comprising Step 21, feature pyramid network construction: first, a feature pyramid network is constructed, a basic network is used as a backbone network, horizontal connections are added to the network to combine the high-resolution feature map of the bottom layer network with the semantic information of the top layer network to generate a multi-scale feature pyramid; Step 22, target feature distribution learning: in the training stage, the back propagation algorithm and the loss function are used to optimize the network parameters, and the network is supervised to learn the distribution of the target features, and the network learns the distribution of the target features in each feature map, the importance and distribution density of the target under different scales; Step 23, adaptive feature fusion: at each stage of generating the feature pyramid, the fusion weights between different scale feature maps are dynamically adjusted based on the learned target feature distribution. The weight is a scalar value or a vector, which is used to adjust the weight of each channel. The adjustment process introduces learnable parameters such as attention mechanisms. During the feature fusion process, the contribution of different scale feature maps will be dynamically adjusted according to the distribution of target features, so that the network can better capture the features of the target. Step 24, training and optimization: the network structure after adaptive feature fusion is trained and optimized. During the training process, the network parameters are updated using the backpropagation algorithm and the optimizer, so that the prediction results of the network are closer to the true labels. Step 25, inference stage: through the trained and optimized model, the input image is subjected to target detection or semantic segmentation, and the adaptive feature fusion weight learned during the training process is used to dynamically adjust the fusion of feature maps, so as to obtain the final detection result. Analyzing the acquired image data in the cloud includes: Using the target detection algorithm YOLOv5, including: Model parallel computing: the target detection task in YOLOv5 can be decomposed into multiple small tasks and executed in parallel on different processing units. Through model parallel computing, different parts of the model are allocated to different processing units, and target detection is performed in parallel. Data parallel computing: divide the data into multiple batches, and process each batch in parallel on different processing units. Pipeline parallel computing: divide the image processing task into multiple stages and use pipeline technology to execute these stages in parallel. In YOLOv5, image preprocessing, feature extraction, target detection, etc. can be processed respectively and executed in sequence on the pipeline to improve the overall processing speed.

2. The dual mode machine vision based weeding method as claimed in claim 1, wherein: In step 12, for each sampling point in the convolution kernel, the offset at the corresponding position is adjusted to make the convolution kernel more accurately capture features in a specific area.

3. The dual mode machine vision based weeding method as claimed in claim 1, wherein: In step 12, bilinear interpolation or trilinear interpolation is used for interpolation to obtain more accurate feature values.

4. A dual mode weeding robot based on machine vision, applying the dual mode weeding method based on machine vision of any one of claims 1-3, characterized in that: Including A vehicle, and a weeding device, a positioning and obstacle avoidance mechanism, and a camera for collecting image data mounted on the vehicle, the weeding device comprising a motor mowing assembly and a grass pulling mechanical arm, the positioning and obstacle avoidance mechanism comprising an ultrasonic device and a GPS positioning device, the visual perception device comprising a camera and a gateway for uploading image data to the cloud; the motion mechanism is provided with a control module, an information receiving module and an information processing module; the control module includes a motion module, a weeding module, a positioning and obstacle avoidance module and a visual control module; the motion module, the weeding module, the positioning and obstacle avoidance module and the visual control module are respectively used to control the motion mechanism, the weeding device, the positioning and obstacle avoidance device and the visual perception module; the information receiving module is used to receive information of each module; the information processing module is used to process the received information data; The motor mowing assembly is arranged at the front of the vehicle in the walking direction. The grass pulling mechanical arm is arranged at the rear of the vehicle in the walking direction.

5. The dual mode weeding robot based on machine vision as claimed in claim 4, wherein: The vehicle comprises a lower bottom plate and an upper bottom plate, which are arranged in a spaced manner, and the electric mowing assembly is installed at the front of the vehicle and between the upper bottom plate and the lower bottom plate; The swing arm suspensions are arranged between the four corners of the upper bottom plate and the lower bottom plate and at the corners, each of the suspensions is provided with a Mecanum wheel, and each of the Mecanum wheels is provided with a driving motor.

6. The dual mode weeding robot based on machine vision as claimed in claim 4 wherein: The weeding mechanical arm comprises a rotatable base, four rotatable joints are arranged on the rotatable base, so that the main body of the weeding mechanical arm is bent, the end of the main body of the weeding mechanical arm is a rotating joint, and the action end of the rotating joint is an execution end with a clamping function.

Images