Intelligent laser weeding dynamic fine positioning method and system based on machine vision

Abstract

The invention discloses an intelligent laser weeding dynamic fine positioning method and system based on machine vision, and the method comprises the steps: carrying out the target detection based on an image collected by a coarse positioning camera, and carrying out the coarse positioning of a detection frame of weeds; carrying out optical flow field calculation based on an image acquired by the coarse positioning camera to obtain a displacement offset; correcting the position of a coarse positioning detection frame in real time based on the displacement offset; controlling a fine positioning camera to carry out image acquisition according to the corrected position; and calculating the center position of the weed rhizome based on the acquired image. The accurate weed center pixel position is obtained through combination of coarse positioning and fine positioning, the motion direction and distance of a machine are predicted through an optical flow method, dynamic and accurate positioning is achieved, and the problem that in existing laser weeding, the weed visual recognition precision is not high is solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and agricultural laser weeding application, and in particular relates to a dynamic precise positioning method and system for intelligent laser weeding based on machine vision. Background Art

[0002] Currently, laser weed control is being widely tested and promoted due to its advantage of less harm to the environment. Among them, the visual positioning function of weeds is an important part of laser weed control.

[0003] Most visual positioning methods use a camera for positioning, but in actual application scenarios, the field of view of a camera is limited or the crops obtained are relatively blurred, and it is impossible to identify all weeds or the recognition accuracy is limited. Chinese patent application CN117274566A, published on April 26, 2024, discloses a real-time weeding method based on deep learning and the distribution of weeds between plants. It uses a neural network to identify the smallest detection frame and uses the center position of the detection frame as the target position of the weeds, resulting in limited recognition accuracy. Chinese patent CN117882697A, published on April 16, 2024, discloses a precise and fast intelligent laser weeding method, which uses a visual system and an optical ranging sensor to accurately control the movement posture of the galvanometer to ensure that the focal point of the laser spot can cover the weed roots and stems to achieve effective elimination, but this method does not take into account the impact of the dynamic movement of the weeding robot on weeding positioning. Summary of the invention

[0004] In order to solve the problem of low accuracy in visual identification of weeds during existing laser weeding, the present invention provides a dynamic precise positioning method for intelligent laser weeding based on machine vision, which obtains the precise pixel position of the weed center by combining coarse positioning and fine positioning, and predicts the movement direction and distance of the machine by optical flow method, thereby achieving dynamic precise positioning.

[0005] According to one aspect of the present invention, a method for dynamic precise positioning of intelligent laser weeding based on machine vision is provided, comprising: Perform target detection based on the images collected by the coarse positioning camera and roughly locate the detection frame of the weeds; The optical flow field is calculated based on the images collected by the coarse positioning camera to obtain the displacement offset; Based on the displacement offset, correct the position of the coarse positioning detection frame in real time; Controlling the precision positioning camera to collect images according to the corrected position; The center position of the weed rhizome is calculated based on the collected images.

[0006] As a further technical solution, target detection is performed based on images collected by the coarse positioning camera, including: The image captured by the coarse positioning camera is input into the target detection model constructed based on the RT-DETR network. The target detection model performs feature extraction and target detection on the input image, combines the image features with the target query through the self-attention mechanism and the cross-attention mechanism to perform target positioning and classification, and outputs the position information of the weed detection frame. The position information of the detection frame includes the pixel coordinates of the upper left corner vertex of the detection frame and the width and height of the detection frame.

[0007] As a further technical solution, optical flow field calculation is performed based on the images collected by the coarse positioning camera, including: Based on the optical flow prediction network, the displacement field information based on the pixel level is separated for two consecutive input frames of images.

[0008] As a further technical solution, the displacement field information based on the pixel level is separated, including: The collected image is evenly divided into blocks to form several areas of set sizes; Calculate the displacement of the center point of each region as the displacement of the current region; The average displacement of all areas is calculated as the real-time motion displacement of the laser weeding robot.

[0009] As a further technical solution, the method further includes: The movement speed of the laser weeding robot is obtained according to the movement displacement and the time interval between two consecutive input frames of images.

[0010] As a further technical solution, the method also includes: according to the motion displacement, correcting the position of the coarse positioning detection frame (m+dx, n+dy, w, h), where (m, n) represents the pixel coordinates of the upper left corner vertex of the detection frame, w represents the width of the detection frame, h represents the height of the detection frame, and (dx, dy) represents the motion displacement.

[0011] As a further technical solution, after obtaining the image captured by the precision positioning camera, the method further includes: calculating the center position of the weed rhizome in the image based on a skeleton line extraction algorithm.

[0012] As a further technical solution, the center position of the weed rhizome in the image is calculated, including: Based on the collected image, background segmentation is performed by converting the RGB channel into the HSV channel; Use binarization and erosion algorithms to obtain weed images; The Zhang-Suen skeleton extraction algorithm is used to extract the central skeleton line; Use intersection detection and calculate the center point.

[0013] According to one aspect of the present invention, there is provided an intelligent laser weeding dynamic precision positioning system based on machine vision, comprising: A coarse positioning module is used to detect targets based on images collected by a coarse positioning camera and to coarsely locate the detection frame of weeds; A displacement correction module is used to calculate the optical flow field based on the image collected by the coarse positioning camera to obtain a displacement offset, and based on the displacement offset, to correct the position of the coarse positioning detection frame in real time; The precise positioning module is used to control the precise positioning camera to collect images according to the corrected position, and calculate the center position of the weed rhizome based on the collected image.

[0014] According to one aspect of the present invention, there is provided an intelligent laser weeding robot, which is equipped with a coarse positioning camera, a fine positioning camera and a processor. The coarse positioning camera is arranged at the front end of the laser weeding robot and is used to collect weed images with a wide field of view. The fine positioning camera is arranged at the bottom end of the laser weeding robot and is used to collect weed images after position correction. The processor is used to receive images collected by the coarse positioning camera and the fine positioning camera to execute the steps of the intelligent laser weeding dynamic fine positioning method based on machine vision.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention realizes weed recognition with a wide viewing angle and meets the recognition accuracy through a combined camera solution of coarse positioning and fine positioning; 2. The present invention uses an optical flow prediction network to calculate the displacement field without using other sensors. It corrects the rough positioning detection area in real time according to the motion parameters of the weeding robot to control the precise positioning camera to move in advance, ensuring that the precise positioning can capture the weeds and identify the precise weed root position; 3. The coarse positioning of the present invention adopts the RT-DETR network structure, which optimizes the accuracy and ensures real-time deployment; 4. In the precise positioning process of the present invention, the central skeleton method is used to extract the outline of the weeds, and then the precise root and stem position is calculated according to the geometric characteristics of the weeds. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A schematic flow chart of a method for dynamic precise positioning of intelligent laser weeding based on machine vision provided in an embodiment of the present invention.

[0018] Figure 2 A schematic diagram of the principle of a positioning method provided by an embodiment of the present invention.

[0019] Figure 3 A schematic diagram of the installation of a coarse positioning camera and a fine positioning camera provided in an embodiment of the present invention.

[0020] Figure 4 A schematic diagram of optical flow information predicted by an optical flow prediction network provided in an embodiment of the present invention.

[0021] Figure 5 A schematic diagram of predicting the positions of future crops (blue dotted box) and weeds (red dotted box) based on optical flow prediction and RT-DETR provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] It should be noted that: The present invention is based on deep learning target detection. Through the color pictures collected by the common RGB optical camera, the rough positioning of the weeds and the precise positioning of the feature points of the weed stems targeted by the final laser are calculated respectively. By deploying the deep learning network on the engineering board through the optimized acceleration technology, the hardware equipment requirements are reduced and the requirements for real-time identification and weeding are met. The method of the present invention can quickly locate the positions of weeds and crops by collecting data and weed data of various crops such as soybeans, corn seedlings, etc., and can correct and estimate the positions of the monitored crops and weed targets in the machine motion state in real time according to the calculation of the optical flow field, and finally realize the precise positioning of weed identification with a positioning error of less than 1mm.

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0024] The embodiment of the present invention provides a dynamic precise positioning method for intelligent laser weeding based on machine vision, which is used to accurately locate the center pixel position of weeds through the combination of coarse positioning and fine positioning, and solves the problem of limited visual recognition accuracy in existing laser weeding. Figure 1 As shown, the method provided by the embodiment of the present invention first obtains the coarse positioning position of the weed detection frame according to the image captured by the coarse positioning camera; then calculates the real-time displacement offset through the optical flow prediction network, and uses the offset to correct the real-time position of the laser weeding robot; finally, controls the fine positioning camera to capture the weed image after the position correction, and calculates the center position of the weed rhizome.

[0025] It should be noted that, in order to improve work efficiency, the laser weeding robot referred to in the embodiment of the present invention controls the direction of the galvanometer to quickly process the weed target during the forward process when working in the field, that is, laser weeding is also performed during the movement process. Therefore, it is necessary to correct and predict the position of the weeds according to the real-time position of the weeding robot, and control the galvanometer to move in the direction of the target point in advance. The present invention predicts the movement direction and distance of the machine through the method of optical flow calculation, and precisely locates the camera position so that the weed target is located under the camera field of view, identifies the weed outline and calculates the position of the weed rhizome, thereby realizing the function of dynamic and precise positioning.

[0026] As a preferred embodiment, please refer to Figure 2 The dynamic precise positioning method provided by the embodiment of the present invention specifically includes the following steps: Step 1: Use a coarse positioning camera to collect images. The main function of the coarse positioning camera is a wide field of view to meet the needs of weed identification with a wide viewing angle. The present invention uses a wide field of view 4 million pixel camera, and the camera is installed at the front end of the unmanned robot to roughly locate the weed position in advance.

[0027] The present invention also installs multiple rows of galvanometer lasers at the bottom of the robot, and each galvanometer laser is equipped with a coaxial precision positioning 1 million pixel precision positioning camera. The installation diagram of the rough positioning camera and the precision positioning camera can be referred to Figure 3 The embodiment of the present invention realizes weed recognition with a wide viewing angle and meets the recognition accuracy through the joint camera setting of coarse positioning and fine positioning.

[0028] Step 2: Get the coarse positioning image and identify the detection frame. The coarse positioning camera collects color RGB channel images, performs target detection through the neural network RT-DETR, and preliminarily locates the detection frame of the weeds. Compared with the commonly used YOLO series network, the RT-DETR network has higher positioning accuracy and meets the needs of real-time computing. In the test environment of Intel i7-12700KF processor and NVIDIARTX3060 graphics card, it takes 6ms to process an image.

[0029] The overall framework of RT-DETR consists of a feature extraction network and a Transformer-based detection network. The image features are combined with object queries through self-attention mechanism and cross-attention mechanism to achieve object localization and classification.

[0030] The output of the RT-DETR network is the position information of the detected weed detection frame. In the embodiment of the present invention, the coarse positioning visual output data format is a detection frame of (m, n, w, h), where (m, n) is the pixel coordinates of the upper left corner vertex of the detected weed detection frame, and (w, h) is the width and height of the detection frame.

[0031] Step 3: Calculate the optical flow field and obtain the average offset. Figure 4 As shown in the figure, while obtaining the coarse positioning position, the optical flow prediction network separates the displacement field information based on the pixel level by inputting two consecutive frames of pictures.

[0032] The optical flow prediction network adopts the principle of dual branches and feature pyramid multi-scale to achieve end-to-end optical flow field training, which has higher accuracy in detecting small-scale and long-distance moving objects. It should be noted that when estimating the movement direction and speed of the weeding robot, the optical flow information of each pixel is not required, only an average speed needs to be calculated to represent the speed of the robot.

[0033] The embodiment of the present invention adopts a method of calculating a uniform displacement field in blocks, and evenly divides the entire image into areas of 100x100 size. The displacement of the center point of the area is taken to represent the displacement of this area. The average displacement of all areas is calculated to be the real-time motion information of the robot, and this motion information is recorded as (dx, dy).

[0034] Furthermore, by inputting the time interval t between two frames of images, the real-time motion speed of the weeding robot can be roughly estimated. Based on the movement speed, the galvanometer control unit of the laser weeding can adjust the direction to burn the weeds according to the speed and time. Moreover, during the movement of the weeding robot, considering that the galvanometer control also takes time, the light output point position of the current laser device can be adjusted according to the calculated real-time movement speed.

[0035] Step 4: Based on the average offset obtained in step 3, the position of the rough positioning detection frame can be corrected in real time to (m+dx, n+dy, w, h). Based on the optical flow prediction and RT-DETR network, the positions of future crops (blue dotted frame) and weeds (red dotted frame) are predicted as follows: Figure 5 shown.

[0036] Step 5: Control the precision positioning camera to collect images and perform precision positioning calculations based on the collected images. The precision positioning camera coaxial with the laser galvanometer laser is moved to this corrected position through galvanometer control to obtain high-definition weed images, and the pixel-level position of the weed rhizomes is calculated based on the skeleton line extraction algorithm to achieve precise weeding.

[0037] Furthermore, the pixel-level position of the weed rhizomes is calculated based on the skeleton line extraction algorithm, which is mainly based on the conversion of RGB channels into HSV channels and the extraction of green part information to identify the rhizomes and leaves. The specific process is as follows: first, the background is segmented by converting the RGB channel into the HSV channel; then, the weed image is obtained using the binarization and corrosion algorithm; then, the central skeleton line is extracted using the Zhang-Suen skeleton extraction algorithm; finally, the intersection is detected and the center point is calculated.

[0038] Based on the same inventive concept as the above method embodiment, the embodiment of the present invention provides an intelligent laser weeding dynamic precision positioning system based on machine vision, comprising: A coarse positioning module is used to detect targets based on images collected by a coarse positioning camera and to coarsely locate the detection frame of weeds; A displacement correction module is used to calculate the optical flow field based on the image collected by the coarse positioning camera to obtain a displacement offset, and based on the displacement offset, to correct the position of the coarse positioning detection frame in real time; The precise positioning module is used to control the precise positioning camera to collect images according to the corrected position, and calculate the center position of the weed rhizome based on the collected image.

[0039] When the dynamic precise positioning system provided by the embodiment of the present invention is used to position the weeding robot, the position of the roughly located weeds is calculated through the color picture collected by the ordinary RGB optical camera, and the optical flow field is calculated to obtain the real-time motion information. The position of the roughly located weeds is corrected using the real-time motion information, and then the precise positioning camera is controlled to collect images of the weeds after the position correction, so as to calculate the position of the feature points of the weed stems targeted by the precise positioning laser. Through the coordination of the coarse positioning and the precise positioning, high-precision recognition is achieved under the premise of satisfying a wide viewing angle.

[0040] The present invention uses the RT-DETR network structure for coarse positioning to optimize accuracy and ensure real-time deployment, and uses the optical flow prediction network to calculate real-time motion information, realizing real-time correction of the detection area without relying on other sensors, so as to ensure that the precise positioning image acquisition based on the corrected position can capture weeds and accurately identify the position of weed rhizomes. The present invention deploys the deep learning network on the engineering board through optimized acceleration technology, reduces the hardware equipment requirements, and meets the needs of real-time identification and weeding.

[0041] Based on the same inventive concept as the above method embodiment, the embodiment of the present invention also provides an intelligent laser weeding robot, which is equipped with a coarse positioning camera, a fine positioning camera and a processor. The coarse positioning camera is arranged at the front end of the laser weeding robot and is used to collect weed images with a wide field of view. The fine positioning camera is arranged at the bottom end of the laser weeding robot and is used to collect weed images after position correction. The processor is used to receive images collected by the coarse positioning camera and the fine positioning camera to execute the steps of the intelligent laser weeding dynamic fine positioning method based on machine vision.

[0042] In the intelligent laser weeding robot provided in the embodiment of the present invention, the installation diagram of the coarse positioning camera and the fine positioning camera can be referred to in Figure 3 The coarse positioning camera is installed at the front end of the unmanned robot to roughly locate the weeds in advance. The fine positioning camera is installed at the bottom of the laser weeding robot, and multiple rows of galvanometer lasers are installed at the bottom of the laser weeding robot, and each galvanometer laser is equipped with a coaxial fine positioning 1 million pixel fine positioning camera.

[0043] In the embodiments of the present invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0044] Compared with traditional mechanical weeding and chemical weeding, laser weeding can achieve precise targeting, high speed and less damage to the environment. However, when locating the target weeds, the existing laser weeding robots do not consider the impact of the dynamic movement of the weeding robots on the weeding positioning, and the recognition accuracy is not high. Therefore, the present invention provides an intelligent laser weeding dynamic precise positioning method based on machine vision, which takes into account the positioning problem during the dynamic movement of the weeding robot, and improves the recognition accuracy by combining coarse positioning and precise positioning.

[0045] In summary, the present invention provides a method for accurately locating the rhizomes of weeds, through a combined camera installation structure of coarse positioning (large field of view) and fine positioning (precision), and through RT-DETR and optical flow prediction network, the real-time dynamic position correction of the coarse positioning detection frame during the movement of the robot is realized, and the fine positioning camera is controlled according to the corrected position to obtain a clearer picture, the weed outline is extracted through the central skeleton line method, and then the precise rhizome position center is calculated according to the geometric characteristics of the weed, so as to improve the recognition accuracy within 1 mm.

[0046] In the coarse positioning and recognition neural network test, the present invention tested YOLO5, YOLO8, mamba-YOLO8 and RT-DETR networks based on the same batch of weed soybean seedling data collected, and the test results are shown in the following table (Table 1). Among them, RT-DETR has the best performance with a recognition accuracy of 92%, while meeting the real-time deployment speed requirements. Optical flow prediction can correct the position of the following detection frame according to the different working speeds of the weeding robot, which is convenient for dynamic planning of multi-point weeding tasks. In addition, the present invention improves the efficiency of weeding through the control scheme of independent multi-vibrator.

[0047] Table 1 Comparison of the accuracy of each model network

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic precise positioning method for intelligent laser weeding based on machine vision, characterized in that: include: Perform target detection based on the images collected by the coarse positioning camera and roughly locate the detection frame of the weeds; The optical flow field is calculated based on the images collected by the coarse positioning camera to obtain the displacement offset; Based on the displacement offset, correct the position of the coarse positioning detection frame in real time; Controlling the precision positioning camera to collect images according to the corrected position; The center position of the weed rhizome is calculated based on the collected images.

2. According to the method of intelligent laser weeding dynamic precise positioning based on machine vision in claim 1, it is characterized in that: Target detection based on images collected by the coarse positioning camera includes: The image captured by the coarse positioning camera is input into the target detection model constructed based on the RT-DETR network. The target detection model performs feature extraction and target detection on the input image, combines the image features with the target query through the self-attention mechanism and the cross-attention mechanism to perform target positioning and classification, and outputs the position information of the weed detection frame. The position information of the detection frame includes the pixel coordinates of the upper left corner vertex of the detection frame and the width and height of the detection frame.

3. According to the method of intelligent laser weeding dynamic precise positioning based on machine vision in claim 1, it is characterized in that: The optical flow field is calculated based on the images collected by the coarse positioning camera, including: Based on the optical flow prediction network, the displacement field information based on the pixel level is separated for two consecutive input frames of images.

4. According to the method of intelligent laser weeding dynamic precise positioning based on machine vision in claim 3, it is characterized in that: Separate pixel-level displacement field information, including: The collected image is evenly divided into blocks to form several areas of set sizes; Calculate the displacement of the center point of each region as the displacement of the current region; The average displacement of all areas is calculated as the real-time motion displacement of the laser weeding robot.

5. According to the method of intelligent laser weeding dynamic precise positioning based on machine vision in claim 3, it is characterized in that: The method further comprises: The movement speed of the laser weeding robot is obtained according to the movement displacement and the time interval between two consecutive input frames of images.

6. According to the method of intelligent laser weeding dynamic precise positioning based on machine vision in claim 3, it is characterized in that: The method also includes: according to the motion displacement, correcting the position of the coarse positioning detection frame (m+dx, n+dy, w, h), where (m, n) represents the pixel coordinates of the upper left corner vertex of the detection frame, w represents the width of the detection frame, h represents the height of the detection frame, and (dx, dy) represents the motion displacement.

7. The method for dynamic precise positioning of intelligent laser weeding based on machine vision according to claim 1, characterized in that: After obtaining the image captured by the precision positioning camera, the method further comprises: calculating the center position of the weed rhizome in the image based on a skeleton line extraction algorithm.

8. The method for dynamic precise positioning of intelligent laser weeding based on machine vision according to claim 7, characterized in that: Calculate the center position of the weed rhizome in the image, including: Based on the collected image, background segmentation is performed by converting the RGB channel into the HSV channel; Use binarization and erosion algorithms to obtain weed images; The Zhang-Suen skeleton extraction algorithm is used to extract the central skeleton line; Use intersection detection and calculate the center point.

9. An intelligent laser weeding dynamic precision positioning system based on machine vision, characterized in that: include: A coarse positioning module is used to detect targets based on images collected by a coarse positioning camera and to coarsely locate the detection frame of weeds; A displacement correction module is used to calculate the optical flow field based on the image collected by the coarse positioning camera to obtain a displacement offset, and based on the displacement offset, to correct the position of the coarse positioning detection frame in real time; The precise positioning module is used to control the precise positioning camera to collect images according to the corrected position, and calculate the center position of the weed rhizome based on the collected image.

10. An intelligent laser weeding robot, characterized in that: The robot is provided with a coarse positioning camera, a fine positioning camera and a processor. The coarse positioning camera is arranged at the front end of the laser weeding robot and is used to collect weed images with a wide field of view. The fine positioning camera is arranged at the bottom end of the laser weeding robot and is used to collect weed images after position correction. The processor is used to receive the images collected by the coarse positioning camera and the fine positioning camera to execute the steps of a dynamic fine positioning method for intelligent laser weeding based on machine vision as described in any one of claims 1 to 8.

Images