Method for applying pesticide by visual spraying robot based on augmented reality remote control

Abstract

The application provides a pesticide application method of a visual spraying robot based on augmented reality remote control, relates to the field of agricultural intelligent robots, and the spraying robot comprises a pesticide application robot and a virtual robot, the pesticide application robot and the virtual robot correspond to each other, the pesticide application robot comprises a mobile chassis, a mechanical arm, a binocular vision system, a spraying system, a sensing system and a control module, and the virtual robot comprises a display, a gesture virtual module and an industrial computer; the specific method is that: field pest information is collected in real time through the binocular vision system, pest target identification, pest coordinate calculation, motion trajectory planning and spraying decision control are performed, the planned motion trajectory is verified by using the gesture virtual module, and finally output to the pesticide application robot. The method can perform accurate operation on the field pest area, has good stability, high safety factor, and can solve the sudden conditions in the pesticide application process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural intelligent robot technology, in particular to a method for applying pesticide by a visual spraying robot based on remote control of augmented reality. BACKGROUND

[0002] China is a large agricultural country, and the overall development and existing problems of agriculture will directly affect the national economic development and social stability. In the process of crop planting, the serious and frequent occurrence of pests and diseases will directly affect the quality and yield of crops. At present, with the development of science and technology, the prevention and control of field pests and diseases mainly relies on plant protection unmanned aerial vehicle spraying pesticide and large-scale pesticide spraying machine: plant protection unmanned aerial vehicle can spray large area in the field, effectively save labor, improve the efficiency of pesticide application, so as to realize rapid, efficient, large-scale, no dead angle of pest and disease control, however, plant protection unmanned aerial vehicle adopts the way of uniform spraying pesticide in each area without difference, which cannot identify pest-free area and pest area, thus causing pesticide in pest-free area, resulting in waste of pesticide, and the excess pesticide will be scattered in the air and soil, thus causing damage to the balance of air environment and soil. At the same time, the large area and non-discriminatory plant protection unmanned aerial vehicle spraying has low precision, that is, it cannot spray corresponding pesticide according to the severity and mildness of pests and diseases, which also causes waste of pesticide and damage to ecological environment. Large-scale pesticide spraying machine for field operation can spray pesticide in different areas, thus distinguishing pest-free area and pest area, but it still sprays in pest area without difference, which still has problems such as low precision, waste of pesticide and easy damage to ecology. In addition, the existing plant protection unmanned aerial vehicle and large-scale pesticide spraying machine are sprayed through the established program, which cannot deal with the time without prior knowledge and cannot respond to emergencies, and the overall adaptability of the machine is low. SUMMARY

[0003] In view of the above problems existing in the prior art, the present application aims to provide a method for applying pesticide by a visual spraying robot based on remote control of augmented reality, which can identify field pests and diseases through visual detection of the spraying robot on site and send them to the remote server decision system, and the artificial can control the spraying robot on site to work on the field pest and disease area through gesture operation in remote, so as to realize precise pesticide spraying, avoid waste of pesticide and the problem of excess pesticide scattered in the air, and affect the health of the operator, etc. At the same time, through artificial remote operation, the instability of spraying robot or plant protection unmanned aerial vehicle can be effectively avoided, and special events, crisis events and other emergencies encountered can be effectively handled.

[0004] The application achieves the objective through the following technical solutions.

[0005] A pesticide application method of a visual spraying robot based on augmented reality remote control, the spraying robot comprising two pesticide application robots arranged in a field and two virtual robots arranged remotely, the pesticide application robots and the virtual robots corresponding to each other, the pesticide application robot comprising a mobile chassis, a mechanical arm, a binocular vision system, a spraying system, a sensing system and a control module, the mechanical arm being two, which are arranged at the end face of the mobile chassis (to ensure work on both sides of the mobile chassis), the binocular vision system being arranged at the end of the mechanical arm away from the mobile chassis, the spraying system, the sensing system and the control module being arranged at the end face of the mobile chassis, and the control module being electrically connected with the mobile chassis, the mechanical arm, the binocular vision system, the spraying system and the sensing system respectively; the virtual robot comprising a display, a gesture virtual module and an industrial computer, the industrial computer being connected with the control module through a remote communication module, and the industrial computer being electrically connected with the display and the gesture virtual module respectively.

[0006] The specific method is: the binocular vision system is used to realize real-time collection of field pest information, and the collected information is transmitted to the industrial computer of the virtual robot through the remote communication module; after receiving the visual information of the pesticide application robot, the industrial computer performs pest target identification, pest coordinate calculation, motion trajectory planning and spraying decision control, and verifies the planned motion trajectory through the gesture virtual module, and finally outputs to the pesticide application robot to complete the work task.

[0007] Based on the further optimization of the above scheme, the mobile chassis adopts a tracked chassis, and the mechanical arm adopts a six-degree-of-freedom mechanical arm; the binocular vision system comprises a binocular camera and a camera support; the spraying system comprises a pesticide tank, a pump body, a pesticide application pipeline and a spray head, the pesticide tank and the pump body are fixedly arranged at the end face of the mobile chassis, the spray head is arranged at the end of the mechanical arm away from the mobile chassis, and the pump body is connected with the pesticide tank through the pesticide application pipeline.

[0008] Based on the further optimization of the above scheme, the sensing system comprises a laser radar, a Beidou positioning device and a perception sensor.

[0009] In order to reduce the influence of the complex field environment on pest identification and improve the accuracy of model identification, based on the further optimization of the above scheme, the industrial computer adopts an improved YOLO v9 module for pest target identification, the module (i.e. the improved YOLO v9 module) measures the linear separability between the target neuron and other neurons through the design of an energy function, controls the weight of each neuron to calculate the attention weight without increasing additional parameters, and specifically, a SimAM attention mechanism is added to each detection head of the conventional YOLOv9 network, thereby improving the detection accuracy of small targets and improving the feature extraction capability of the model for larvae.

[0010] The specific steps of adopting the improved YOLO v9 module for pest target recognition are as follows:

[0011] Step A1, first, collect the pest image on the crop, and mark the larvae and winged adults respectively through the marking tool to make a data set; then, divide the data set into a training set, a validation set and a test set, and store them classifiedly;

[0012] Step A2, adopt the improved YOLO v9 module to identify the characteristics of the pest, and add an adaptive penalty coefficient in the identification process to complete the identification of the pest;

[0013] Step A3, judge whether the preset number of times is reached, if not, return to step A1 for image marking; if yes, use the validation set and the test set to verify the accuracy of the model training, and output the improved YOLO v9 model for pest identification.

[0014] Based on the further optimization of the above scheme, the initial value range of the adaptive penalty coefficient is [0.5, 1] and the adaptive penalty coefficient is adjusted according to the pest image rule.

[0015] Based on the further optimization of the above scheme, the specific steps of the industrial computer for pest coordinate calculation are as follows:

[0016] First, after obtaining the pest target by using the improved YOLO v9 module, the two-dimensional coordinates of the center point of a single pest detection frame are obtained :

[0017] ;

[0018] In the formula, , are the upper left corner coordinates and the lower right corner coordinates of a single pest detection frame, respectively;

[0019] Then, through the space mapping relationship between the pixel coordinate system and the camera coordinate system, the three-dimensional coordinates of a single pest are obtained :

[0020]

[0021] In the formula, z depth represents the current point depth value obtained by the camera; f x 、f y fL and fR represent the camera focal lengths of the binocular vision system, respectively; （c x ,cy ） represents the camera principal point coordinate of the binocular vision system.

[0022] Further optimization based on the above scheme, the specific steps of the industrial computer for motion trajectory planning are:

[0023] Firstly, the point cloud information of the pest is obtained through the binocular vision system and the above pest coordinate calculation, and the camera point and the target pest position , Construct a plane equation:

[0024] ;

[0025] In the formula: represents any point in the plane equation (that is, any point in , , );

[0026] After that, a preset point cloud thickness threshold , traverse the point cloud information of the pest and filter out the point cloud that belongs to the above plane range and meets the point cloud thickness threshold ; Based on the projection method, the filtered point cloud is rendered into a bmp format image, realizing the projection of the field model point cloud to a two-dimensional image.

[0027] Then, a feasible path is found in the two-dimensional image by random point sampling, and the cost function of the feasible path is calculated , and the cost function is used as the parameter constraint of the ellipse.

[0028] ;

[0029] In the formula: represents the target point (that is, the first target point on the feasible path), represents the secondary target point (that is, the second target point on the feasible path), Dis represents the Euclidean distance between the parent node and the starting point.

[0030] After that, a search tree is constructed in the ellipse range with random sampling points as root nodes, a new path is repeatedly iterated and searched, and the cost function of the new path is repeatedly calculated. If the cost function of the new path is smaller than the cost function of the old path and meets the collision detection, the ellipse constraint information is updated, that is:

[0031] ;

[0032] In the formula: represents the starting point; A, B, and C represent the parameters in the ellipse function, respectively.

[0033] If the cost function of the new path is less than a set threshold, or the iteration number of the new path is equal to a preset iteration number, the corresponding new path is the optimal path for spraying.

[0034] Based on further optimization of the above scheme, the specific steps for the industrial computer to make spraying decision regulation are:

[0035] Suppose two pesticide application robots are A* and B*, and their corresponding virtual robots are A# and B#. When both A* and B* find pests in their respective areas, the remote communication module adopts a token holding queuing mechanism: when A* finds pests first, A# takes the token and B# waits in line with the token, that is, A# prioritizes the pose behavior planning decision of A* and plans the spraying trajectory of A*, and then transmits the optimized simulation data to A* for operation. After completion, A# exits the decision and B# takes the token to make the calculation and behavior planning decision.

[0036] When only one of the areas of A* and B* finds pests, spraying decision is made only for the area with pests.

[0037] When neither A* nor B* finds pests, there is no need to make a pose behavior decision for the robots, and A* and B* continue to move forward and detect pests.

[0038] Based on further optimization of the above scheme, the gesture virtual module is developed using open-source virtual reality Unity3D software and its SDK package, the LeapMotion function and its vision system of the SDK package are used as gesture data acquisition devices, the sensor is used to capture hand information in the real world and sample gesture data frame by frame, and a processing signal is sent to the server after sampling is completed.

[0039] Data acquisition and processing: six common gestures are selected as virtual gesture models, each gesture includes training set, validation set and test set images, and a training model data set is constructed. Then, the R component of the original image is extracted to generate a grayscale image using the OpenCV library, and the grayscale image is histogram equalized and threshold segmented to obtain a binary image with stronger contrast and clearer gesture contour. Specifically:

[0040] ;

[0041] In the formula: represents the pixel value of the current pixel point in the original convex shape; represents a threshold value selected according to actual test; represents an optimization coefficient;

[0042] The gesture virtual module adopts the gesture perception AI recognition decision model of the light network CNN-L5 as the target network of the static gesture recognition model, responds to the processing signal around the static gesture recognition model, adopts the binary image as the training set, and completes the establishment of the static gesture model;

[0043] Finally, based on the real-time operation of the TCP / IP protocol, the real-time gesture feedback of the industrial computer calculation end is carried out around the operation demand of the user end virtual robot, and stable and intelligent human-computer interaction is realized, that is, the remote operator can use the hand type to simulate the virtual robot to complete the trajectory planning, obstacle avoidance and spraying operation, and the emergency events can also be handled through manual processing.

[0044] The following are the technical effects possessed by the scheme of the application:

[0045] Firstly, the application realizes the identification, classification, counting and position acquisition of the field insect pests by image acquisition of the field site by the pesticide application robot, remote communication between the pesticide application robot and the virtual robot, and the deep learning configured in the virtual robot, so as to correspondingly generate a pesticide spraying prescription. Meanwhile, the motion trajectory of the spray head for spraying pesticide is generated in real time by using the acquired insect pest position, so as to realize accurate pesticide spraying, effectively avoid the problems of pesticide waste, damage to the surrounding ecological environment, poor development of plants stimulated by high-concentration pesticide, and ineffective killing of insect pests in the insect pest flooding area caused by large-area and indiscriminate spraying. In addition, the motion trajectory of the virtual robot for spraying insect pests is simulated by manual gesture control, so as to effectively verify the accuracy of the spraying trajectory, avoid the problems of affecting surrounding crops or failing to reach the specified spraying point and poor spraying effect, and effectively avoid the safety hazards of human site operation and inhalation of pesticides to damage the human body by remote control, so as to achieve the purposes of high safety factor and high spraying precision. Moreover, the remote access of the operator can optimize the operation decision and operation scheme, effectively solve the sudden conditions such as special events and crisis events encountered in the spraying process, and ensure the stability of the spraying operation, so as to effectively improve the spraying operation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 It is a structural block diagram of the improved YOLO v9 module for insect pest target recognition in embodiment 1 of the application.

[0047] Figure 2 It is a structural diagram of the improved YOLO v9 model in embodiment 1 of the application.

[0048] Figure 3 It is a framework diagram of the gesture virtual module in embodiment 1 of the application for verifying the planned motion trajectory and controlling the operation of the pesticide application robot.

[0049] Figure 4 The network structure diagram of VGG16-Small in the embodiment 1 of the present application.

[0050] Figure 5 The network structure diagram of CNN-L5 in the embodiment 1 of the present application.

[0051] Figure 6 The schematic diagram of acquiring images by the binocular vision system in the embodiment 2 of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application.

[0053] Embodiment 1:

[0054] The application discloses a pesticide application method of a visual spraying robot based on augmented reality remote control. The spraying robot comprises two pesticide application robots arranged in a field and two virtual robots remotely arranged (i.e. arranged indoors outside the field). The pesticide application robot and the virtual robot correspond to each other. The pesticide application robot comprises a mobile chassis, a mechanical arm, a binocular vision system, a spraying system, a sensing system and a control module. The mobile chassis adopts a tracked chassis, facilitating the pesticide application robot to perform slope climbing during identification and pesticide spraying. The mechanical arm is provided on the end face of the mobile chassis (ensuring work on both sides of the mobile chassis), and adopts a six-degree-of-freedom mechanical arm (a six-degree-of-freedom mechanical arm for picking in the prior art can be adopted). The binocular vision system is arranged at the end of the mechanical arm away from the mobile chassis. The binocular vision system comprises a binocular camera and a camera support (a common model of the binocular camera can be adopted, and the application does not make specific limitation). The spraying system, the sensing system and the control module are arranged on the end face of the mobile chassis, and the control module is electrically connected with the mobile chassis, the mechanical arm, the binocular vision system, the spraying system and the sensing system (the control module is loaded with software such as identification, edge detection algorithm, database and robot motion control). The spraying system comprises a pesticide box, a pump body, a pesticide application pipeline and a spray head. The pesticide box and the pump body are fixedly arranged on the end face of the mobile chassis. The spray head is fixedly arranged at the end of the end of the mechanical arm away from the mobile chassis. The pump body is connected with the spray head and the pesticide box through the pesticide application pipeline. The sensing system comprises a laser radar, a Beidou positioning device and a perception sensor, and is used for sensing the external environment of the pesticide application robot and obstacle avoidance. The virtual robot comprises a display, a gesture virtual module and an industrial computer. The industrial computer is connected with the control module through a remote communication module (specifically, a network protocol of a wireless network hole is used), and is electrically connected with the display and the gesture virtual module. The gesture virtual module comprises a gesture module based on augmented reality and a binocular stereo vision system, a stereo camera support, a gesture posture model acquisition and a gesture database. The industrial computer comprises a simulation platform, a decision platform and a communication platform. The display is used for displaying simulation results and gesture recognition results.

[0055] The specific method is that the binocular vision system is used to realize real-time collection of field pest information, and the remote communication module is used to transmit the collected information to the industrial computer of the virtual robot. After the industrial computer receives the visual information of the pesticide application robot, pest target identification, pest coordinate calculation, motion trajectory planning and spraying decision control are performed. Specifically,

[0056] Firstly, an improved YOLO v9 module is used for pest target identification. The module (i.e. the improved YOLO v9 module) measures the linear separability between target neurons and other neurons through a designed energy function, controls the weight of each neuron to calculate attention weight without increasing additional parameters, and the like. Figure 2As shown, specifically: in each detection head of the conventional YOLOv9 network, the SimAM attention mechanism is added respectively, so as to improve the detection accuracy of small targets and improve the feature extraction capability of the model for larvae;

[0057] As shown in the figure, the specific steps of identifying the pest target by using the improved YOLO v9 module are as follows: Figure 1

[0058] Step A1, first, collect the images of pests and diseases on crops, and mark the larvae and winged adults respectively through a labeling tool (a conventional labeling tool can be used, such as the labelling labeling tool), and make a data set; then, divide the data set into a training set, a validation set and a test set (in this embodiment, the ratio of the training set, the validation set and the test set is 8:1:1), and store them classifiedly;

[0059] Step A2, using the improved YOLO v9 module to identify the characteristics of pests, and adding an adaptive penalty coefficient in the identification process , to complete the identification of pests; the initial value range of the adaptive penalty coefficient is [0.5, 1] and the adaptive penalty coefficient is adjusted according to the rules of the pest image, for example: first set the adaptive penalty coefficient to 0.5, if the intersection and union ratio IoU of the pest characteristics of the image detected by the improved YOLO v9 module is greater than 0.5, then the adaptive penalty coefficient is doubled until the ratio IoU is less than the corresponding adaptive penalty coefficient ; if the intersection and union ratio IoU of the pest characteristics of the image detected by the improved YOLO v9 module is not greater than 0.5, then take 0.5 as the corresponding adaptive penalty coefficient, and add the adaptive penalty coefficient to the improved YOLO v9 module.

[0060] Step A3, determine whether the preset number of times is reached (the preset number of times is set according to a large amount of experimental data), if not, return to step A1 to mark the image; if yes, use the validation set and the test set to verify the accuracy of the model training, and output the improved YOLO v9 model for pest identification.

[0061] Then, the pest coordinates are calculated, and the specific steps are as follows:

[0062] Step B1, after obtaining the pest target by using the improved YOLO v9 module, obtain the two-dimensional coordinates of the center point of a single pest detection frame :

[0063] ;

[0064] In the formula:​ , These are the coordinates of the top left and bottom right corners of a single pest detection frame, respectively.

[0065] Step B2: Through the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, realize the three-dimensional coordinates of a single pest. Acquisition:

[0066]

[0067] In the formula: z depth This represents the current depth value acquired by the camera. f x 、f y These represent the camera focal lengths of the binocular vision system; （c x ,c y ） This represents the principal point coordinates of the camera in a binocular vision system.

[0068] Next, the motion trajectory planning (of the nozzle at the end of the robotic arm) is performed, and the specific steps are as follows:

[0069] Step C1: Obtain point cloud information of the pests using a binocular vision system and the aforementioned pest coordinates, and then use a camera to view the point cloud information. Location of the target pest , Construct the plane equation:

[0070] ;

[0071] In the formula: Represent any point in the plane equation (i.e., is) , , (any point in the middle)

[0072] Step C2: Preset point cloud thickness threshold Traverse the point cloud information of pests and filter out those belonging to the above-mentioned plane range and meeting the point cloud thickness threshold. The point cloud is generated; based on the projection method, the selected point cloud is rendered into a BMP format image, realizing the projection of the field model point cloud to a two-dimensional image;

[0073] Step C3: Sampling at random points, finding a feasible path in the two-dimensional image, and calculating the cost function of the feasible path. With cost function As a parameter constraint of the ellipse;

[0074] ;

[0075] wherein: represents the target point (i.e. the first target point on the feasible path), represents the sub-target point (i.e. the second target point on the feasible path), Dis represents the Euclidean distance between the parent node and the start point;

[0076] Step C4, constructing a search tree with random sampling points as root nodes within the elliptical range, repeatedly iterating to find a new path, and repeatedly calculating the cost function of the new path, if the cost function of the new path is smaller than the cost function of the old path and meets the collision detection, updating the elliptical constraint information, i.e.

[0077] ;

[0078] wherein: represents the start point; A, B, C represent parameters in the elliptical function, respectively;

[0079] If the cost function of the new path is smaller than a set threshold (the set threshold is obtained through a large amount of experimental data) or the iteration number of the new path is equal to a preset iteration number (the preset iteration number is set according to experimental data), the corresponding new path is the optimal path for spraying.

[0080] Finally, the industrial computer makes a spraying decision and regulation, and the specific steps are as follows:

[0081] Suppose that the two spraying robots are A* and B*, and the corresponding virtual robots are A# and B#, when both the areas of the two spraying robots are found to have pests (i.e. when the areas of the A* and B* robots are found to have pests), the remote communication module adopts a token holding queuing mechanism: when the first discovered pest area is the A* range, the virtual robot A# takes the token first, and the B# virtual robot holds the token and waits in line, i.e. the virtual robot A# prioritizes the pose behavior planning decision of the corresponding spraying robot A* and plans its spraying motion trajectory, and then transmits the optimized simulation data to the spraying robot A* for operation, after completion, the virtual robot A# exits the decision, and the virtual robot B# holds the token to make a calculation and behavior planning decision.

[0082] When only one of the areas of the two spraying robots is found to have pests: only the area with pests is subjected to spraying decision;

[0083] When neither of the areas of the two spraying robots is found to have pests: no pose behavior decision is needed for the spraying robots, and the two spraying robots A* and B* continue to move forward and detect pests.

[0084] Finally, the planned motion trajectory is verified by the gesture virtual module and output to the spraying robot to complete the task, specifically:

[0085] The gesture virtual module is developed by using open-source virtual reality Unity3D software and an SDK package thereof, and uses a LeapMotion function and a visual system of the SDK package as a gesture data collection device. Hand information in a real world is captured by using a sensor, and gesture data is sampled frame by frame. After sampling is completed, a processing signal is sent to a server;

[0086] Data collection and processing: six common gestures are selected as virtual gesture models (six gestures are selected according to actual conditions, for example: forward - fingers up, backward - fingers down, extend the mechanical arm - cloth, retract the mechanical arm - stone, spray on - open hands, and spray off - close hands), each gesture includes a training set, a verification set and a test set image (in this embodiment, each gesture includes 1500 training set images, 250 verification set images and 100 test set images), and a training model data set is constructed (the original image size in the data set is 300*300). In the training process, the total number of each round is M = nt, and the time consumption of each round is T = hn + tk (in the formula: n represents the number of times, t represents the number of times, h represents the sampling frequency, and k represents the time interval between each time, in this embodiment, two photos are obtained within 1 second, and the time interval for obtaining each picture is 0.4 seconds); then, the R component of the original image is extracted to generate a grayscale image, and the grayscale image is subjected to histogram equalization and threshold segmentation to obtain a binary image with stronger contrast and clearer gesture contour, specifically as follows:

[0087] ;

[0088] In the formula: represents the pixel value of the current pixel point in the original convex shape; represents a threshold value selected according to actual testing; represents an optimization coefficient;

[0089] The gesture virtual module uses a gesture perception AI recognition decision model of a lightweight network CNN-L5 as a target network of a static gesture recognition model (the lightweight network CNN-L5 is a VGG16 network structure modification, the VGG16 network structure is simple, but the parameter quantity is large, resulting in a large space occupied by the model and slow model loading, in this application, first, the total number of layers of the VGG16 network is reduced by 1 / 2, then a BathNormalization layer is added to accelerate the training curve convergence, then the convolution layer number is adjusted according to the image size, and the parameter is optimized, the original three-layer full connection layer is changed to one layer, and the unit number is reduced, to obtain a VGG16-small network as shown in Figure 4 Then, the convolution layer is reduced, the BathNormalization layer is removed, and the full connection layer unit number is further reduced, to obtain a VGG16-small network as shown in Figure 5The static gesture recognition model is established by using the binary image as a training set and processing the signal around the static gesture recognition model.

[0090] Finally, based on the TCP / IP protocol, the industrial computer calculates the real-time gesture feedback around the user's virtual robot job requirements, realizes stable and intelligent human-computer interaction, realizes remote operators using hands to simulate virtual robots to complete trajectory planning, obstacle avoidance, spraying operations, and can also handle emergency events through manual processing. Figure 3

[0091] Embodiment 2

[0092] As a preferred embodiment of the present application, on the basis of the scheme of embodiment 1, the spraying decision control of the industrial computer further includes the acquisition of the application amount, specifically:

[0093] First, fix the shooting time interval of the binocular vision system to avoid overlapping of single sample images:

[0094] ;

[0095] In the formula, W represents the width of a single sample, V represents the walking speed of the robot, as shown in Figure 6 ;

[0096] The length of a single row of field crops is L , and the number of samples taken in a single row is N , that is L = NW ;

[0097] Then, the improved YOLO v9 module is used to identify and obtain the pest species and the target detection frame of the corresponding pest species, and the length of each target detection frame is obtained L jc :

[0098] ;

[0099] In the formula: , are the upper left corner coordinates and the lower right corner coordinates of the target detection frame, respectively;

[0100] A pest length threshold L cd is preset, and if the length of the target detection frame L jc is greater than the pest length threshold L cd , it is determined that the length of the target detection frame L ​jc The corresponding insect pest is adult, if the target detection box length L jc is not greater than the insect pest length threshold L cd , then it is judged that the target detection box length L jc The corresponding insect pest is larva, and the number of adult and larva in a single sample is counted respectively m 1 、m 2;

[0101] Then, the single sample pesticide amount is obtained according to the proportion of adult and larva in a single sample, that is: Q

[0102]

[0103] In the formula: It represents the pesticide amount required for a single adult, It represents the proportion adjustment coefficient of adult, which is obtained through experimental data; It represents the pesticide amount required for a single larva, It represents the proportion adjustment coefficient of larva, which is obtained through experimental data;

[0104] Finally, the pesticide amount of a single row with a length of L is obtained Q L :

[0105] .​

Claims

1. A method for applying pesticide by a visual spraying robot based on remote control of augmented reality, characterized in that: The spraying robot comprises two pesticide application robots arranged in a field and two virtual robots arranged remotely, the pesticide application robots and the virtual robots correspond to each other, the pesticide application robot comprises a mobile chassis, a mechanical arm, a binocular vision system, a spraying system, a sensing system and a control module, the mechanical arm is two, which are arranged at the end face of the mobile chassis, the binocular vision system is arranged at the end of the mechanical arm away from the mobile chassis, the spraying system, the sensing system and the control module are arranged at the end face of the mobile chassis, and the control module is electrically connected with the mobile chassis, the mechanical arm, the binocular vision system, the spraying system and the sensing system; the virtual robot comprises a display, a gesture virtual module and an industrial computer, the industrial computer is connected with the control module through a remote communication module, and the industrial computer is electrically connected with the display and the gesture virtual module; Specific methods are as follows: the binocular vision system is used to realize real-time collection of field pest information, and the collected information is transmitted to the industrial computer of the virtual robot through the remote communication module; after the industrial computer receives the visual information of the pesticide application robot, pest target recognition, pest coordinate calculation, motion trajectory planning and spraying decision control are performed, the planned motion trajectory is verified through the gesture virtual module, and finally output to the pesticide application robot to complete the work task; The industrial computer uses an improved YOLO v9 module to perform pest target recognition, the module measures the linear separability between the target neuron and other neurons through the design of an energy function, controls the weight of each neuron to calculate the attention weight without increasing additional parameters, and specifically, a SimAM attention mechanism is added to each detection head of the conventional YOLO v9 network to improve the detection accuracy of small targets and improve the feature extraction capability of the model for larvae; The specific steps of the industrial computer for motion trajectory planning are as follows: Firstly, the point cloud information of the insect pests is obtained through the binocular vision system and the above-mentioned insect pest coordinate calculation, and the camera point With the target pest position , The plane equation is constructed: ； In the formula: represents any point in the plane equation, that is, , , any point in Afterwards, preset point cloud thickness threshold , traverse the point cloud information of the pest and screen out the point cloud belonging to the above plane range and satisfying the point cloud thickness threshold . The selected point cloud is rendered into a bmp format image based on the projection method, and the projection of the field model point cloud to a two-dimensional image is realized; Then, a feasible path is found in the two-dimensional image by random point sampling, and a cost function of the feasible path is calculated The cost function As a parameter constraint of the ellipse ； In the formulae: represents a target point, i.e. a first target point on the feasible path, represents a sub-target point, i.e. a second target point on the feasible path, Then, a search tree is constructed with the random sampling points as the root nodes within the elliptical range, a new path is repeatedly iterated and searched, and the cost function of the new path is repeatedly calculated, if the cost function of the new path is smaller than that of the old path and the collision detection is met, the elliptical constraint information is updated, namely: represents the Euclidean distance between the parent node and the start point; If the cost function of the new path is smaller than a set threshold value, or the iteration number of the new path is equal to a preset iteration number, the corresponding new path is the optimal path for spraying. ； wherein: denotes the starting point; A, B, C denote parameters in the elliptic function, respectively; The mobile chassis adopts a tracked chassis, and the mechanical arm adopts a six-degree-of-freedom mechanical arm; the binocular vision system comprises a binocular camera and a camera support; the spraying system comprises a pesticide box, a pump body, a pesticide application pipeline and a spray head, the pesticide box and the pump body are fixedly arranged at the end face of the mobile chassis, the spray head is arranged at the end of the mechanical arm away from the mobile chassis, and the pump body is communicated with the pesticide box through the pesticide application pipeline; the sensing system comprises a laser radar, a Beidou positioning device and a perception sensor. 2.The method according to claim 1, wherein: The specific steps of using the improved YOLO v9 module for pest target recognition are as follows:

3. The method according to claim 1 or 2, wherein the method is a method for applying pesticide by a visual spraying robot based on remote control of augmented reality. ​ Step A1, first, the image of the crop disease and insect pests is collected, and the larvae and winged adults are marked respectively through the marking tool to make a data set; then, the data set is divided into a training set, a validation set and a test set, and is stored separately; Step A2, using a modified YOLO v9 module to identify the characteristics of the insect pests, adding an adaptive penalty coefficient in the identification process , completing the identification of insect pests; Step A3, whether the preset number of times is reached is judged, if not, the image marking in step A1 is returned to; if yes, the validation set and the test set are used to verify the accuracy of the model training, and the improved YOLO v9 model is output for pest identification.

4. The method of claim 3, wherein the method further comprises: receiving a user input from the remote control device; and controlling the visual spraying robot based on the user input. The specific steps for the industrial computer to calculate the pest coordinates are: First, the two-dimensional coordinates of the center point of the single pest detection box are obtained after the pest target is obtained by using the improved YOLO v9 module : ； In the formula: , are respectively the upper left corner coordinates and the lower right corner coordinates of a single pest detection frame. Then, through the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, the three-dimensional coordinates of a single pest are obtained : In the formula: z depth represents the current point depth value acquired by the camera; f x 、f y respectively represent the focal length of the camera of the binocular vision system; （c x ,c y ） represents the principal point coordinate of the camera of the binocular vision system.

5. The method of claim 3, wherein the method further comprises: determining a distance between the user and the visual spraying robot; and controlling the visual spraying robot based on the distance. The specific steps for the industrial computer to make spraying decision control are: When two pesticide application robots are found to have pests in their respective areas, the remote communication module adopts a token queuing mechanism: when the first pest area A* is found, the virtual robot A# takes the token first, and the virtual robot B# waits in line with the token, that is, the virtual robot A# has priority to make posture behavior planning decisions for the corresponding pesticide application robot A* and plan its spraying trajectory, and then transmits the optimized simulation data to the pesticide application robot A* for operation, after completion, the virtual robot A# exits the decision, and the virtual robot B# takes the token to make calculation and behavior planning decisions; When only one of the two pesticide application robots has pests in its area: only the area with pests is sprayed; When neither of the two pesticide application robots has pests in their areas: no posture behavior decision needs to be made for the pesticide application robots, and the two pesticide application robots A* and B* continue to move forward and detect pests.

6. The method of claim 5, wherein the method further comprises: receiving a user input from the remote control device; and controlling the visual spraying robot based on the user input. The gesture virtual module is developed by using the open-source virtual reality Unity3D software and its SDK package, uses the LeapMotion function and its visual system of the SDK package as a gesture data acquisition device, captures hand information in the real world by using a sensor, and samples gesture data frame by frame, and sends a processing signal to the server after sampling is completed; Data acquisition and processing: six common gestures are selected as virtual gesture models, each gesture includes a training set, a validation set and a test set image, and a training model data set is constructed; then, the R component of the original image is extracted to generate a grayscale image by using the OpenCV library, and the grayscale image is histogram equalized and threshold segmented to obtain a binary image with stronger contrast and clearer gesture contour, specifically: ； In the formula: represents the pixel value of the current pixel point in the original convex shape; represents the threshold value selected according to actual test; represents the optimization coefficient; The gesture virtual module uses the gesture perception AI recognition decision model of the lightweight network CNN-L5 as the target network of the static gesture recognition model, responds to the processing signal around the static gesture recognition model, uses the above binary image as the training set, and completes the establishment of the static gesture model; Finally, based on the TCP / IP protocol, real-time operation is realized, the industrial computer calculation end makes real-time gesture feedback around the work demand of the user end virtual robot, realizes stable and intelligent human-computer interaction, realizes that the remote operator uses hands to simulate the virtual robot to complete trajectory planning, obstacle avoidance and spraying operation, and also handles emergency events through manual processing.

Images