Method and control system for locating missed seeds in replanting swing arm operation

Abstract

The present invention discloses a replanting swing arm positioning method, comprising the following steps: (1) sugarcane seedling identification part; (2) binocular ranging part; (3) missed seed position positioning part; (4) sugarcane seedling dynamic tracking part. The method of the present invention is based on a two-stage camera missed seed positioning strategy, which can not only ensure the real-time operation of the machine, but also ensure the accuracy of replanting, and at the same time monitor the seeding quality. Based on the sugarcane seedling dynamic tracking strategy, the previous and next frames of the video can be associated to ensure that the sugarcane seedling position information is not lost, thereby improving the accuracy of missed seed positioning. The embedded device is used to integrate the sugarcane seedling feature network model, sugarcane seedling position information and depth information, and acts as a control device to control the robotic arm. The control system provided has a mechanism to reduce the swing arm callback time, which greatly improves the efficiency of replanting operations.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent agricultural machinery, and in particular to a replanting swing-arm operation positioning method and control system. Background Art

[0002] Research on sugarcane has shown that compared to newly planted sugarcane, perennial sugarcane cultivation reduces planting steps and time, saving costs while also providing favorable growth advantages, creating crucial conditions for stable and high yields. Managing perennial sugarcane effectively is crucial for increasing sugarcane production and revenue. However, repeated crushing of perennial roots during mechanical harvesting in the previous year often leads to an increased incidence of perennial root disease, resulting in uneven distribution of perennial cane stumps in the field and severe ridge breakage and missing stumps.

[0003] The sowing process of the reseeding machine: the reseeding machine moves to the position where reseeding is required, controls the mechanical arm to lift and swing above the sowing point, and the gripper under the mechanical arm dives to grab the soil. There is a seed dropping device above the end of the mechanical arm, which drops the sugarcane seeds into the dug pit, followed by fertilizer and watering, and then covers the sugarcane seeds with the soil in the gripper to complete the soil covering. The above steps are the reseeding process.

[0004] Patent application number 202210890095.3 discloses a perennial sugarcane replanting robot and method. The device comprises a walking mechanism, a hole-drilling and soil-covering mechanism, and a seed-dropping mechanism. The hole-drilling and soil-covering mechanism includes a drill barrel, a drill bit, a rotary drive mechanism for rotating the drill bit, and a vertical drive mechanism for driving the drill barrel and drill bit downward. The drill bit is mounted within the drill barrel and is equipped with spiral blades. The rotary drive mechanism is mounted on a vertical transmission plate of the vertical drive mechanism, and the drive end of the rotary drive mechanism is connected to the drill bit. A vertical sliding structure is provided between the drill barrel and the vertical transmission plate for relative vertical movement. The seed-dropping mechanism comprises a seed storage hopper, several seed scoops, and a seed-dropping mechanism for driving the seed scoops to remove sugarcane from the seed storage hopper. However, the aforementioned replanting robot device is small in size and the replanting method is inefficient. This makes actual replanting difficult to meet the needs of farmers and is not suitable for large-scale application.

[0005] The information disclosed in this background technology section is only intended to enhance understanding of the overall background of the invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a replanting swing arm operation positioning method, thereby overcoming the shortcomings of inaccurate replanting operation positioning, missed seeds in supplementary operations, and low sugarcane seedling identification accuracy.

[0007] Another object of the present invention is to provide a control system for replanting swing-arm operations to improve the intelligence of replanting equipment.

[0008] To achieve the above object, the present invention provides a replanting swing arm operation positioning method, comprising the following steps:

[0009] (1) Sugarcane seedling recognition part: collect images of sugarcane seedlings, create training sets, validation sets and test sets, then establish a convolutional neural network model, iteratively train the collected image data within the convolutional neural network model, and obtain the sugarcane seedling feature recognition network model after the training is completed;

[0010] (2) Binocular ranging part: A binocular camera is used to obtain the depth information of the sugarcane seedlings. The binocular camera is first calibrated, and then the parameters are derived. The pixel coordinates of the sugarcane seedlings are converted into world coordinates through the internal and external parameter coefficients of the binocular camera and the camera coordinate conversion method. The world coordinate information of the sugarcane seedlings is obtained and input into the sugarcane seedling feature recognition network model.

[0011] (3) Missing seed location positioning part: A two-stage binocular camera system is used to locate the missing seed location, and the coordinate information of the replanting location is determined based on the coordinate information of the sugarcane seedlings. The coordinate information of the replanting location is input into the sugarcane seedling feature recognition network model;

[0012] (4) Sugarcane seedling dynamic tracking part: obtain real-time video stream, segment the video screen according to pixel areas, match the sugarcane seedling ID and coordinate information, and add a feature sequence module to the back end of the sugarcane seedling feature recognition network model to realize the dynamic tracking of the sugarcane seedling.

[0013] Preferably, in the above technical solution, the method of identifying the sugarcane seedlings in step (1) comprises:

[0014] 1) Using a camera device of an image recognition device to collect images of sugarcane seedlings, the sugarcane seedlings are preprocessed to obtain the marker features required for training. Then, the camera device records videos of the sugarcane seedlings under different light and light intensities. The recorded videos are segmented into frames and then divided into images to obtain sugarcane seedling image samples.

[0015] 2) Use annotation tools to annotate key features of the area where the roots of the sugarcane seedlings touch the soil and the center of the sugarcane seedlings in the image. Different types of objects are labeled with different labels, while similar types of objects are labeled with the same category. A label file is generated, which contains the coordinates of the center pixel of the marked feature, the width and height of the marking box, and the category to which the marked feature belongs.

[0016] 3) Use the image data of sugarcane seedlings with multiple features as the test set samples, and the image data of ordinary sugarcane seedlings as the training set samples and the validation set samples; the number of test set samples, the number of training set samples, and the number of validation set samples are distributed according to a certain ratio;

[0017] 4) Establish a convolutional neural network model, load the configuration file and set the network parameters. Input the label file and the prepared 3-channel color sugarcane seedling image into the established convolutional neural network model for iterative training. After a certain number of training rounds, the optimal weight file is obtained. After the training is completed, the sugarcane seedling feature recognition network model is obtained.

[0018] 5) Use the convolutional neural network and the trained sugarcane seedling feature recognition network model to predict the test set to evaluate the accuracy of the model. The evaluation model accuracy is required to be ≥ 90% to complete the sugarcane seedling recognition part.

[0019] Preferably, in the above technical solution, in step 1), the number of sample images of sugarcane seedlings obtained is at least 10,000; and / or in step 3), the number of test set samples, the number of training set samples, and the number of validation set samples are distributed proportionally, and the data volume distribution is: (number of training set samples + number of validation set samples): number of test set samples = 9:1, number of training set samples: number of validation set samples = 9:1.

[0020] Preferably, in the above technical solution, step (2) binocular ranging calibration includes the following steps:

[0021] 1) First, create a black and white checkerboard pattern. After the checkerboard is generated, use a binocular camera to take multiple photos of the checkerboard at different positions to obtain multiple images;

[0022] 2) Split the binocular image into left and right camera images, placing them in separate folders. Then, use the calibration toolbox in Matlab to open the folders containing the left and right camera images. The program automatically detects which images are available and then begins calibration. It obtains the average calibration error and deletes images with excessively large calibration errors.

[0023] 3) Export the parameters to obtain the intrinsic parameter matrix of the binocular camera and the translation parameters and rotation parameters of the binocular camera.

[0024] Preferably, in the above technical solution, the two-stage binocular camera system is used in the seed leakage position positioning part of step (3) to locate the seed leakage position, including: at least one binocular camera is installed at the front end of the driving position of the reseeder equipment, and at least one binocular camera is installed at the end of the mechanical arm;

[0025] The robot identifies distant sugarcane seedlings and returns depth information. The binocular camera at the end of the robot arm observes the sugarcane seedlings near the swing arm position. The binocular camera at the driving position feeds back information about the approximate direction of the missed seed and the ridge in which it is located to the binocular camera at the end of the robot arm. The binocular camera at the end of the robot arm recognizes the sugarcane seedlings near the robot arm in real time, measures the distance between the sugarcane seedling and the current position using the SGBM ranging algorithm, and combines the coordinate information of the sugarcane seedling to obtain the coordinates of the missed seed position.

[0026] The DH method is used to model the robotic arm and perform forward and inverse kinematics analysis. The forward kinematics algorithm is first used to calculate the position and posture of the robotic arm's end effector, and then the inverse kinematics algorithm is used to calculate the angle values ​​of each joint of the robotic arm.

[0027] Preferably, in the above technical solution, step (3) of determining the coordinate information of the replanting position includes:

[0028] 1) Determine the planting distance between two sugarcane seedlings and define an interval threshold. If the distance between two adjacent sugarcane seedlings is greater than the interval threshold, it is determined that there is a missed planting between the two sugarcane seedlings. If the distance between two adjacent sugarcane seedlings is less than the interval threshold, no replanting is required.

[0029] 2) Use a binocular camera to measure the depth of the identified sugarcane seedlings, record their world coordinate information, obtain the world coordinate information of multiple planting ridges, obtain the world coordinate information of the replanting machine through a positioning system, and calculate the distance between adjacent sugarcane seedlings;

[0030] 3) Determine whether the interval threshold is exceeded. If it exceeds one interval threshold but is less than two interval thresholds, the replanting position is located at the center of the two sugarcane seedlings. If it exceeds two interval thresholds, the distance between the two adjacent sugarcane seedlings is averaged into multiple interval thresholds, and a replanting position is recorded every other interval.

[0031] Preferably, in the above technical solution, the reseeding machine performs a three-ridge operation, and the coordinate information for determining the reseeding position includes:

[0032] A) Ensure that the planting distance between two sugarcane seedlings is no greater than 80 cm. 80 cm is defined as a spacing threshold. If the spacing is greater than the spacing threshold, it is determined that there is a missed planting between the two sugarcane seedlings. If the distance between two adjacent sugarcane seedlings is less than the spacing threshold, no replanting is required.

[0033] B) The binocular camera can measure the depth information of the identified sugarcane seedlings and record the world coordinate information of the sugarcane seedlings. The world coordinate information of the three ridges are (Xl, Yl, Zl), (Xm, Ym, Zm), (Xr, Yr, Zr), which represent the world coordinate information of the sugarcane seedlings on the left, middle and right ridges respectively. The coordinate information of the sugarcane seedlings on the left, middle and right ridges are stored in three lists respectively, namely [(Xl1, Yl1, Zl1), (Xl2, Yl2, Zl2), ..., (Xln, Yln, Zln)], [(Xm1, Ym1, Zm1), (Xm2, Ym2, Zm2), ..., (Xmn, Ymn, Zmn)], [(Xr1, Yr1, Zr1), (Xr2, Yr2, Zr2), ..., (Xrn, Yrn, Zrn)], the world coordinate information of the identified sugarcane seedlings can be obtained through the binocular camera. Each coordinate information represents the detection of a sugarcane seedling. Each sugarcane seedling is accompanied by a depth information d. The world coordinate information of the reseeder is obtained through the positioning system. The distance between the two adjacent sugarcane seedlings can be directly obtained by subtracting the distance of the middle ridge. The sugarcane seedlings on the left and right ridges and the binocular camera can form a right triangle. According to the Pythagorean theorem, the distance between the two adjacent sugarcane seedlings can be obtained by subtracting the right triangle. 、 , where m represents the distance between two adjacent ridges. So far, the distance between two adjacent sugarcane seedlings can be calculated according to the formula;

[0034] C) If there is only one sugarcane seedling on a single ridge, a replanting position is recorded when it exceeds 80 cm; if there are multiple sugarcane seedlings on a single ridge, the distance between two adjacent sugarcane seedlings is calculated and judged whether it exceeds the interval threshold. If it exceeds one interval threshold and is less than two interval thresholds, the replanting position is , , , the planting point is located at the center of the two sugarcane seedlings; if the distance between the two adjacent sugarcane seedlings is n( ), where p represents the number of interval thresholds, then n is divided into p parts on average, and a replanting position is recorded every other section.

[0035] Preferably, in the above technical solution, step (4) of dynamically tracking sugarcane seedlings includes:

[0036] 1) The reseeder operates on multiple planting ridges simultaneously, tracking targets on each planting ridge separately. The video screen is divided into multiple areas based on pixel positions, and target tracking is performed simultaneously in multiple areas.

[0037] 2) Every time a sugarcane seedling is identified in the image, an ID is marked on its detection frame, and the coordinate information at that time is recorded, and the ID and coordinate information are matched;

[0038] 3) A feature sequence module is added to the back end of the sugarcane seedling feature recognition network model trained in step (1). The feature sequence is used to store some features in the past time. The feature sequence is driven by time and is updated every 30ms. For example, if the features of the sugarcane seedlings are temporarily blocked during the movement of the reseeding machine, the cache information recorded in the time sequence will feedback that the sugarcane seedlings are not recognized now. In the previous frame, there were sugarcane seedlings at this position, and the sugarcane seedling features were temporarily blocked. The feature sequence generates a detection frame at the relevant position based on the coordinate information and maintains the original ID.

[0039] A control system for replanting swing-arm operation, comprising:

[0040] An embedded device includes the above-mentioned sugarcane seedling feature recognition network model, the embedded device is installed on the replanting machine to form a control unit; a display screen is data-connected to the embedded device, the display screen is used to mark the location that needs to be replanted; a binocular camera includes at least two binocular cameras, at least one binocular camera is installed at the front end of the replanting machine cab, and at least one binocular camera is installed at the front end of the replanting machine mechanical arm, the binocular camera is data-connected to the embedded device, and performs real-time identification and distance measurement of the sugarcane seedlings; and an execution unit includes a mechanical arm, the mechanical arm is data-connected to the embedded device, and the extension and swing of the mechanical arm is controlled by the embedded device to complete the replanting operation.

[0041] A control method for the control system of the above-mentioned replanting swing arm operation includes the following steps:

[0042] (1) Binocular cameras perform real-time identification and distance measurement of sugarcane seedlings;

[0043] (2) The embedded device obtains the coordinate information and depth information of the sugarcane seedlings and performs internal calculations to obtain the location where replanting is required;

[0044] (3) When reseeding multiple planting ridges, the coordinate information of the reseeding position is stored in multiple lists according to the different ridges. The reseeding point closest to the reseeding machine is found and the ridge where it is located is marked as ridge 1. If ridge 1 is the middle ridge, after the reseeding point on the current ridge is completed, the point closest to the reseeding machine is found in the remaining multiple lists and the ridge where it is located is marked as ridge 2. The same marking is repeated for ridges 3...n. The reseeding operation of ridge 2 is processed first and then the reseeding operation of the ridges marked in sequence is processed. If ridge 1 is the left or right ridge, the middle ridge is directly marked as ridge 2, and the ridge on the farthest side is marked as ridge 3...n. Then the reseeding operation is processed in sequence according to the marked sequence number.

[0045] (4) The embedded device controls the hydraulic solenoid valve to drive the robotic arm to rotate, and rotates a certain angle to reach above the replanting position, thereby completing the replanting work.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The present invention's replanting swing arm positioning method can roughly determine the location of missed sugarcane seedlings based on drone aerial photography. The vehicle is equipped with a positioning system. According to the clustering algorithm, the replanting machine's operating path is planned according to the missed seed density to ensure that as many replanting tasks as possible are completed within a limited time. Then, the sugarcane seedling feature recognition network model is deployed in the embedded device. The binocular camera is connected via USB. The binocular camera obtains real-time video streams. The sugarcane seedling feature recognition network model in the embedded device is used to perform real-time target detection, identify sugarcane seedlings, and mark the distance of the sugarcane seedlings in the detection frame. At the same time, the world coordinate information of the sugarcane seedlings is obtained through coordinate transformation. The coordinate position of the missed seed is converted through the relevant algorithm. At the same time, during the operation of the replanting machine, a sugarcane seedling tracking system is added to ensure the continuity of the sugarcane seedling features in the previous and next frames of the picture, reduce the missed detection of sugarcane seedlings, and improve the quality of the replanting operation.

[0048] (2) The method of the present invention, based on a two-stage camera missed seed location strategy, can ensure both the real-time operation of the machine and the accuracy of replanting, while also monitoring the seeding quality. Based on the sugarcane seedling dynamic tracking strategy, it can associate the previous and next frames of the video to ensure that the sugarcane seedling position information is not lost, thereby improving the accuracy of missed seed location. An embedded device is used to integrate the sugarcane seedling feature network model, sugarcane seedling position information, and depth information, and acts as a control device to control the robotic arm.

[0049] (3) In the method of the present invention, the sugarcane seedling feature recognition network model adopts a convolutional neural network and is deployed in an embedded device. It can be accelerated by a physical engine, so that the frame rate of the picture is higher and more stable during the recognition process, thereby improving the accuracy of sugarcane seedling recognition.

[0050] (4) The control system of the present invention is based on a mechanism that allows the swing arm to simultaneously operate three ridges and reduces the swing arm's callback time, which greatly improves the efficiency of replanting operations. The embedded device is connected to a display screen to display the location of missed seeds, improving the human-computer interaction experience. The replanting machine adopts a horizontal planting method, which consumes less sugarcane seeds, has a uniform planting density, a high seedling emergence rate per unit area, and strong resistance to lodging. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the replanting swing arm operation positioning method according to the present invention;

[0052] Figure 2 is an original image of sugarcane seedlings collected according to the method of the present invention;

[0053] Figure 3 is a sugarcane seedling identification diagram in the method according to the present invention;

[0054] Figure 4 2. It is a schematic diagram of the structure of the installation of two-stage binocular cameras on the reseeding machine according to the method of the present invention;

[0055] Figure 5 is a schematic diagram of the operation of the reseeding machine in the method according to the present invention;

[0056] Figure 6 is a flow chart of a control system for replanting swing-arm operation according to the present invention;

[0057] Figure 7 It is an identification ranging graph that integrates the ranging algorithm into the target detection network model according to the method of the present invention.

[0058] Description of main reference numerals:

[0059] 1-First binocular camera, 2-Second binocular camera, 3-Replanter cab, 4-Robot arm, 5-Sugarcane seedling position, 6-Replanter, 7-Missed seed position, 8-Planting ridge. DETAILED DESCRIPTION

[0060] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific embodiments.

[0061] Unless expressly stated otherwise, throughout the specification and claims, the term "comprise" or variations such as "include" or "comprising", etc., will be understood to include the stated elements or components but not to exclude other elements or other components.

[0062] like Figure 1-7 As shown, a replanting swing arm operation positioning method includes the following steps:

[0063] 1. Preprocessing

[0064] First, a drone-based aerial image is generated, which roughly reflects the location of missed seeds. A clustering algorithm is then used to identify clusters by density, indicating that high density indicates more missed seeds. The reseeding machine is equipped with GPS and / or Beidou navigation, and a path planning algorithm is used to calculate the most efficient reseeding path. The reseeding machine's walking mode in the field is that its two wheels walk in the ditch, with three ridges on both sides and in the middle of the wheels. The design of the present invention is that the reseeding machine can simultaneously reseed a total of three ridges.

[0065] 2. Target Recognition

[0066] This part uses a deep learning-based convolutional neural network for target recognition. First, the camera device of the image recognition device collects images of sugarcane seedlings. The seedlings are preprocessed to obtain the marker features required for training. Then, the camera device records videos of the sugarcane seedlings under different light and light intensities to increase the diversity of the dataset. The recorded videos are then segmented into images by frame, obtaining at least 10,000 sugarcane seedling image samples. Annotation tools are used to annotate key features in the images, such as the part where the roots of the sugarcane seedlings contact the soil and the center of the sugarcane seedlings. Different class labels are assigned to different classes of targets, and the same class labels are assigned to similar classes. A label file is generated. The label file contains the coordinates of the center pixel of the marked feature location, the width and height of the marking box, and the category to which the marked feature belongs. The sugarcane seedling images with multiple features are then used as test set samples, and the images of ordinary sugarcane seedlings are used as training and validation set samples. The data volume distribution is roughly (number of training set samples + number of validation set samples): number of test set samples = 9:1, and the number of training set samples: number of validation set samples = 9:1. Next, a convolutional neural network model was built. The configuration file was loaded and the network parameters were set. The label file and the prepared three-channel color sugarcane seedling image were input into the constructed convolutional neural network model for iterative training. After a certain number of training rounds, the optimal weight file was obtained. After training, the sugarcane seedling feature recognition network model was obtained. The convolutional neural network and the previously trained sugarcane seedling feature recognition network model were used to predict the test set to evaluate the model accuracy. The evaluation model accuracy was required to be ≥ 90%. Sugarcane seedling recognition was completed.

[0067] 3. Binocular ranging part

[0068] After the above steps, a convolutional neural network that can identify sugarcane seedlings can be obtained. However, it is not enough to just identify the sugarcane seedlings. To determine the location of the missed seeds, a binocular depth camera is used in the patent of this invention to obtain the depth information of the sugarcane seedlings (i.e., the distance from the sugarcane seedlings to the camera). Before measuring the distance, the binocular camera must be preprocessed and calibrated. The binocular camera calibration steps are as follows: first, a black and white checkerboard pattern is made. After the checkerboard is generated, the binocular camera is used to take multiple photos of the checkerboard in different poses, about 30 photos. Then, the binocular image is divided into left and right camera images, which are placed in different folders one by one. Then, the calibration toolbox in Matlab is used to open the folder where the left and right camera images were just saved. The program will automatically detect how many collected images are available and then start calibration. At this time, the average calibration error can be seen. The images with too large calibration errors are deleted, and then the parameters are exported to obtain the intrinsic parameter matrix of the binocular camera and the translation and rotation parameters of the binocular camera. The pixel coordinates of the sugarcane seedlings are converted into world coordinates through the internal and external parameter coefficients of the binocular camera and the camera coordinate conversion method. The center pixel coordinates (u, v) of the sugarcane seedling detection frame are converted into world coordinates (Xw, Yw, Zw). Thus, the world coordinate information of the sugarcane seedlings is obtained.

[0069] 4. Positioning of leaking seeds

[0070] In the present invention, based on the fact that the reseeding machine has a rocker-type operation mode, a two-stage binocular camera system is used to locate the missed seed position. The so-called two-stage binocular camera system means installing two binocular cameras. The first binocular camera 1 is installed at the front end of the reseeding machine cab 3, and the second binocular camera is installed at the end of the robotic arm 4. Both binocular cameras are connected to the embedded device through the serial port. The binocular camera at the driving end can see information at a farther distance, identify distant sugarcane seedlings and return depth information. The binocular camera at the end of the robotic arm observes the sugarcane seedlings near the swing arm position. The binocular camera at the driving end feeds back the information of the approximate missed seed direction and which ridge it is located in to the binocular camera at the end of the robotic arm. The binocular camera at the end of the robotic arm recognizes the sugarcane seedlings near the robotic arm in real time, measures the distance between the sugarcane seedlings and the current position through the SGBM ranging algorithm, and obtains the coordinates of the missed seed position in combination with the coordinate information of the sugarcane seedlings.

[0071] The DH method was used to model and analyze the forward and inverse kinematics of a reseeding machine's robotic arm. First, the forward kinematics algorithm was used to calculate the position and posture of the robotic arm's end effector. Then, the inverse kinematics algorithm was used to calculate the angles of each joint in the robotic arm. This method enables high-precision control of the robotic arm, ensuring that the end effector reaches the reseeding position accurately.

[0072] Take the three-ridge operation of the reseeder as an example. Figure 4As shown, determining the coordinate information of the replanting position includes the following steps:

[0073] 1) The base of the robotic arm can rotate and the robotic arm can be freely extended and retracted. Based on the ductility of the robotic arm and the rotatability of the base, the replanting machine can simultaneously operate three ridges, greatly improving work efficiency. Based on actual planting experience and the structure of the replanting machine, it is known that the planting distance between two sugarcane seedlings should not exceed 80 cm. I = 80 (cm) is now defined as an interval threshold. If it is greater than 80 cm, it is determined that there is a missed seed between the two sugarcane seedlings. If the distance between two adjacent sugarcane seedlings is less than 80 cm, no replanting is required.

[0074] 2) The binocular camera can measure the depth information of the identified sugarcane seedlings (i.e., the distance from the sugarcane seedlings to the binocular camera) and record the world coordinate information of the sugarcane seedlings. The world coordinate information of the three ridges are (Xl, Yl, Zl), (Xm, Ym, Zm), (Xr, Yr, Zr), representing the world coordinate information of the sugarcane seedlings on the left, middle, and right ridges respectively. The coordinate information of the sugarcane seedlings on the left, middle, and right ridges are stored in three lists respectively, namely [(Xl1, Yl1, Zl1), (Xl2, Yl2, Zl2), ..., (Xln, Yln, Zln)], [(Xm1, Ym1, Zm1), (Xm2, Ym2, Zm2), ..., (Xmn, Ymn, Zmn)], [(Xr1, Yr1, Zr1), (Xr2, Yr2, Zr2), ..., (Xrn, Yrn, Zrn)], the binocular camera can obtain the world coordinate information of the identified sugarcane seedlings. Each coordinate information represents the detection of a sugarcane seedling, which means that each sugarcane seedling is accompanied by a depth information d. The replanting machine is also equipped with GPS, which can obtain the world coordinate information of the replanting machine and the distance information from the sugarcane seedling to the binocular camera. The distance between two adjacent sugarcane seedlings can be directly subtracted from the distance in the middle ridge to obtain the distance. The sugarcane seedlings on the left and right ridges and the binocular camera can form a right triangle. According to the Pythagorean theorem, the distance between the two adjacent sugarcane seedlings can be obtained by subtracting the right triangle. 、 , where m represents the distance between two adjacent ridges. So far, the distance between two adjacent sugarcane seedlings can be calculated according to the formula.

[0075] 3) If there is only one sugarcane seedling on a single ridge in the image, a replanting position is recorded when it exceeds 80 cm; if there are multiple sugarcane seedlings on a single ridge in the image, the distance between two adjacent sugarcane seedlings is calculated and whether it exceeds the interval threshold is determined. If it exceeds one interval threshold and is less than two interval thresholds, the replanting position is , , , the planting point is located at the center of the two sugarcane seedlings; if the distance between the two adjacent sugarcane seedlings is n( ), where p represents the number of interval thresholds, then n is divided into p equal parts, and a replanting position is recorded every other period. Thus, the coordinate information of the replanting position is obtained.

[0076] 5. Sugarcane seedling dynamic tracking part

[0077] During the movement of the replanting machine, the characteristics and appearance models of the identified sugarcane seedlings will change. At the same time, the characteristics of the sugarcane seedlings may be blocked or disappear temporarily during the movement, which may easily lead to tracking failure. Therefore, this patent proposes a target tracking algorithm that can track the sugarcane seedlings without affecting the determination of the replanting position. Because the reseeding machine operates three ridges simultaneously, it is necessary to track the targets of the three ridges separately. The video screen is divided into three areas according to the pixel position, that is, the ridges are divided into three areas, and target tracking is performed simultaneously in the three areas. Every time a sugarcane seedling is recognized in the screen, an ID is marked on its detection frame, and the coordinate information at this time is recorded. The ID and coordinate information are matched together. The sugarcane seedling feature recognition network model has been obtained in the previous training. A feature sequence module is added to the back end of the network model. The feature sequence is used to store some features in the past time. The feature sequence is driven by time and is updated every 30ms. The function is that if the features of the sugarcane seedlings are temporarily blocked during the movement of the reseeding machine, the cache information recorded in the time series will feedback that although the sugarcane seedlings are not recognized now, there were sugarcane seedlings at this position in the previous frame. Therefore, even if the sugarcane seedling features are temporarily blocked, a detection frame will be generated at the relevant position according to the coordinate information and the original ID will be maintained.

[0078] A control system for replanting swing-arm operations includes an embedded device, a display screen, a binocular camera, and an execution unit. The embedded device includes the aforementioned sugarcane seedling feature recognition network model, and the embedded device is installed on a replanting machine to form a control unit. The display screen is data-connected to the embedded device, and the display screen is used to mark the location that needs replanting. The binocular camera includes at least two. This embodiment has two binocular cameras, one installed at the front end of the replanting machine cab, and the other installed at the front end of the replanting machine's mechanical arm. The binocular camera is data-connected to the embedded device to perform real-time identification and distance measurement of the sugarcane seedlings; the execution unit includes a mechanical arm, which is data-connected to the embedded device. The extension and swinging of the mechanical arm are controlled by the embedded device to complete the replanting operation.

[0079] Specifically, the network model for identifying sugarcane seedling characteristics can be transplanted into an embedded device. This compact and convenient embedded device can be installed on the replanting machine and connected to a binocular camera for real-time identification and distance measurement of the sugarcane seedlings. The embedded device acts as a control unit, with the robotic arm as the execution unit. After wiring is complete, the robotic arm's extension and retraction, as well as its swinging motion, are controlled by a program. For a better human-machine interaction experience, a display screen is connected to the cab, and the location requiring replanting is marked on it. The embedded device then controls the robotic arm to swing to the location above the replanting position before replanting can begin. The embedded device controls the robotic arm's replanting process as follows: After obtaining the coordinates and depth information of the sugarcane seedlings, the embedded device performs internal calculations to determine the location requiring replanting. The embedded device then controls the hydraulic solenoid valve to rotate the robotic arm, rotating it a certain angle to reach the location above the replanting position, completing the replanting operation.

[0080] A control method for the control system of the above-mentioned replanting swing arm operation includes the following steps:

[0081] (1) Binocular cameras perform real-time identification and distance measurement of sugarcane seedlings;

[0082] (2) The embedded device obtains the coordinate information and depth information of the sugarcane seedlings and performs internal calculations to obtain the location where replanting is required;

[0083] (3) When reseeding multiple planting ridges, the coordinate information of the reseeding position is stored in multiple lists according to the different ridges. The reseeding point closest to the reseeding machine is found and the ridge where it is located is marked as ridge 1. If ridge 1 is the middle ridge, after the reseeding point on the current ridge is completed, the point closest to the reseeding machine is found in the remaining multiple lists and the ridge where it is located is marked as ridge 2. The same marking is repeated for ridges 3...n. The reseeding operation of ridge 2 is processed first and then the reseeding operation of the ridges marked in sequence is processed. If ridge 1 is the left or right ridge, the middle ridge is directly marked as ridge 2, and the ridge on the farthest side is marked as ridge 3...n. Then the reseeding operation is processed in sequence according to the marked sequence number.

[0084] (4) The embedded device controls the hydraulic solenoid valve to drive the robotic arm to rotate, and rotates a certain angle to reach above the replanting position, thereby completing the replanting work.

[0085] The three-ridge replanting operation is used as an example. During the replanting process, based on the fact that the replanting machine operates on three ridges simultaneously during its movement, this patent designs a method that can improve the replanting efficiency. The method is summarized as follows: because the replanting machine operates on three ridges simultaneously, the robotic arm also needs to constantly swing back and forth between the three ridges, which will reduce the replanting efficiency. In the present invention, after obtaining the replanting position information, the coordinate information of the replanting position is stored in three lists according to the different ridges where it is located. The replanting point closest to the replanting machine position is found and the ridge where it is located is marked as ridge 1. If ridge 1 is the middle ridge, after the replanting point on the current ridge is completed, the point closest to the replanting machine position is found in the remaining two lists. The ridge where it is located is marked as ridge 2, and the remaining ridge is marked as ridge 3. The replanting operation of ridge 2 is processed first and then the replanting operation of ridge 3 is processed. If ridge 1 is the left or right ridge, the middle ridge is directly marked as ridge 2, and the ridge on the farthest side is marked as ridge 3, and then the processes are carried out in sequence according to the marked sequence. This approach can prevent the robotic arm from swinging back and forth between the three ridges, and reducing the swing time can improve the replanting efficiency.

[0086] When the control system of the present invention is used for replanting operations, the sugarcane seedling feature recognition network model used in the control system has an average recognition accuracy of 90%, an average detection time of 7.6 ms, and a size of about 20 MB.

[0087] The foregoing descriptions of specific exemplary embodiments of the present invention are for purposes of illustration and description. These descriptions are not intended to limit the invention to the precise forms disclosed, and it is apparent that many variations and modifications are possible in light of the foregoing teachings. The exemplary embodiments have been selected and described for the purpose of explaining the specific principles of the invention and their practical application, thereby enabling those skilled in the art to realize and utilize a variety of exemplary embodiments of the invention and various options and modifications. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A replanting swing arm positioning method, characterized in that: The following steps are involved: (1) Sugarcane seedling recognition part: collect images of sugarcane seedlings, create training sets, validation sets and test sets, then establish a convolutional neural network model, iteratively train the collected image data within the convolutional neural network model, and obtain the sugarcane seedling feature recognition network model after the training is completed; (2) Binocular ranging part: A binocular camera is used to obtain the depth information of the sugarcane seedlings. The binocular camera is first calibrated, and then the parameters are derived. The pixel coordinates of the sugarcane seedlings are converted into world coordinates through the internal and external parameter coefficients of the binocular camera and the camera coordinate conversion method. The world coordinate information of the sugarcane seedlings is obtained and input into the sugarcane seedling feature recognition network model. (3) Missing seed location positioning part: A two-stage binocular camera system is used to locate the missing seed location, and the coordinate information of the replanting location is determined based on the coordinate information of the sugarcane seedlings. The coordinate information of the replanting location is input into the sugarcane seedling feature recognition network model; (4) Sugarcane seedling dynamic tracking part: obtain real-time video stream, segment the video screen according to pixel area, match the sugarcane seedling ID and coordinate information, and add a feature sequence module to the back end of the sugarcane seedling feature recognition network model to realize the dynamic tracking of the sugarcane seedling; In step (3), the two-stage binocular camera system is used to locate the seed leakage position, including: at least one binocular camera is installed at the front end of the driving position of the reseeding machine equipment, and at least one binocular camera is installed at the end of the mechanical arm; The robot identifies distant sugarcane seedlings and returns depth information. The binocular camera at the end of the robot arm observes the sugarcane seedlings near the swing arm position. The binocular camera at the driving position feeds back information about the approximate direction of the missed seed and the ridge in which it is located to the binocular camera at the end of the robot arm. The binocular camera at the end of the robot arm recognizes the sugarcane seedlings near the robot arm in real time, measures the distance between the sugarcane seedling and the current position using the SGBM ranging algorithm, and combines the coordinate information of the sugarcane seedling to obtain the coordinates of the missed seed position. The DH method is used to model the robot arm and perform forward and inverse kinematics analysis. The forward kinematics algorithm is first used to calculate the position and posture of the robot arm's end effector, and then the inverse kinematics algorithm is used to calculate the angle value of each joint of the robot arm. A feature sequence module is added to the back end of the sugarcane seedling feature recognition network model trained in step (1). The feature sequence is used to store some features in the past time. The feature sequence is driven by time and is updated every certain period of time. For example, when the reseeding machine is moving, the features of the sugarcane seedlings are temporarily blocked. The cache information recorded in the time sequence will feedback that the sugarcane seedlings are not recognized now, but there were sugarcane seedlings at this position in the previous frame. The sugarcane seedling features are temporarily blocked. The feature sequence generates a detection frame at the relevant position based on the coordinate information and maintains the original ID.

2. The replanting swing arm positioning method according to claim 1, characterized in that: The method of step (1) identifying the sugarcane seedlings comprises: 1) Using a camera device of an image recognition device to collect images of sugarcane seedlings, the sugarcane seedlings are preprocessed to obtain the marker features required for training. Then, the camera device records videos of the sugarcane seedlings under different light and light intensities. The recorded videos are segmented into frames and then divided into images to obtain sugarcane seedling image samples. 2) Use annotation tools to annotate key features of the area where the roots of the sugarcane seedlings touch the soil and the center of the sugarcane seedlings in the image. Different types of objects are labeled with different labels, while similar types of objects are labeled with the same category. A label file is generated, which contains the coordinates of the center pixel of the marked feature, the width and height of the marking box, and the category to which the marked feature belongs. 3) Use the image data of sugarcane seedlings with multiple features as the test set samples, and the image data of ordinary sugarcane seedlings as the training set samples and the validation set samples; the number of test set samples, the number of training set samples, and the number of validation set samples are distributed according to a certain ratio; 4) Establish a convolutional neural network model, load the configuration file and set the network parameters, input the label file and the prepared color sugarcane seedling image into the established convolutional neural network model for iterative training. After a certain number of training rounds, the optimal weight file is obtained. After the training is completed, the sugarcane seedling feature recognition network model is obtained; 5) Use the convolutional neural network and the trained sugarcane seedling feature recognition network model to predict the test set to evaluate the accuracy of the model. The evaluation model accuracy is required to be ≥ 90% to complete the sugarcane seedling recognition part.

3. The replanting swing arm positioning method according to claim 2, characterized in that: Step 1) Obtain at least 10,000 sample images of sugarcane seedlings; and / or Step 3) The number of test set samples, training set samples, and validation set samples are distributed proportionally. The data volume distribution is: number of training set samples + number of validation set samples: number of test set samples = 9:1, and number of training set samples: number of validation set samples = 9:

1.

4. The replanting swing arm positioning method according to claim 1, characterized in that: Step (2) Binocular ranging calibration includes the following steps: 1) First, create a black and white checkerboard pattern. After the checkerboard is generated, use a binocular camera to take multiple photos of the checkerboard at different positions to obtain multiple images; 2) Split the binocular image into left and right camera images, placing them in separate folders. Then, use the calibration toolbox in Matlab to open the folders containing the left and right camera images. The program automatically detects which images are available and then begins calibration. It obtains the average calibration error and deletes images with excessively large calibration errors. 3) Export the parameters to obtain the intrinsic parameter matrix of the binocular camera and the translation parameters and rotation parameters of the binocular camera.

5. The replanting swing arm positioning method according to claim 1, characterized in that: Step (3) determines the coordinate information of the replanting location, including: 1) Determine the planting distance between two sugarcane seedlings and define an interval threshold. If the distance between two adjacent sugarcane seedlings is greater than the interval threshold, it is determined that there is a missed planting between the two sugarcane seedlings. If the distance between two adjacent sugarcane seedlings is less than the interval threshold, no replanting is required. 2) Use a binocular camera to measure the depth of the identified sugarcane seedlings, record their world coordinate information, obtain the world coordinate information of multiple planting ridges, obtain the world coordinate information of the replanting machine through a positioning system, and calculate the distance between adjacent sugarcane seedlings; 3) Determine whether the interval threshold is exceeded. If it exceeds one interval threshold but is less than two interval thresholds, the replanting position is located at the center of the two sugarcane seedlings. If it exceeds two interval thresholds, the distance between the two adjacent sugarcane seedlings is averaged into multiple interval thresholds, and a replanting position is recorded every other interval.

6. The replanting swing arm positioning method according to claim 5, characterized in that: The reseeding machine performs three-ridge operation, and the coordinate information of the reseeding position is determined including: A) Ensure that the planting distance between two sugarcane seedlings is no greater than 80 cm. 80 cm is defined as a spacing threshold. If the spacing is greater than the spacing threshold, it is determined that there is a missed planting between the two sugarcane seedlings. If the distance between two adjacent sugarcane seedlings is less than the spacing threshold, no replanting is required. B) The binocular camera can measure the depth information of the identified sugarcane seedlings and record the world coordinate information of the sugarcane seedlings. The world coordinate information of the three ridges are (Xl, Yl, Zl), (Xm, Ym, Zm), (Xr, Yr, Zr), which represent the world coordinate information of the sugarcane seedlings on the left, middle and right ridges respectively. The coordinate information of the sugarcane seedlings on the left, middle and right ridges are stored in three lists respectively, namely [(Xl1, Yl1, Zl1), (Xl2, Yl2, Zl2), ..., (Xln, Yln, Zln)], [(Xm1, Ym1, Zm1), (Xm2, Ym2, Zm2), ..., (Xmn, Ymn, Zmn)], [(Xr1, Yr1, Zr1), (Xr2, Yr2, Zr2), ..., (Xrn, Yrn, Zrn)], the world coordinate information of the identified sugarcane seedlings can be obtained through the binocular camera. Each coordinate information represents the detection of a sugarcane seedling. Each sugarcane seedling is accompanied by a depth information d. The world coordinate information of the reseeder is obtained through the positioning system. The distance between the two adjacent sugarcane seedlings can be directly obtained by subtracting the distance of the middle ridge. The sugarcane seedlings on the left and right ridges and the binocular camera can form a right triangle. According to the Pythagorean theorem, the distance between the two adjacent sugarcane seedlings can be obtained by subtracting the right triangle. 、 , where m represents the distance between two adjacent ridges. So far, the distance between two adjacent sugarcane seedlings can be calculated according to the formula; C) If there is only one sugarcane seedling on a single ridge, a replanting position is recorded when it exceeds 80 cm; if there are multiple sugarcane seedlings on a single ridge, the distance between two adjacent sugarcane seedlings is calculated and judged whether it exceeds the interval threshold. If it exceeds one interval threshold and is less than two interval thresholds, the replanting position is , , , the replanting point is located at the center of the two sugarcane seedlings; if the distance between the two adjacent sugarcane seedlings exceeds two interval thresholds, ), where p represents the number of interval thresholds, then n is divided into p parts on average, and a replanting position is recorded every other section.

7. The replanting swing arm positioning method according to claim 1, characterized in that: Step (4) sugarcane seedling dynamic tracking includes: 1) The reseeder operates on multiple planting ridges simultaneously, tracking targets separately for each ridge. The video image is divided into multiple areas based on pixel positions, and target tracking is performed simultaneously in each area. 2) Every time a sugarcane seedling is recognized in the image, an ID is marked on its detection frame, and the coordinate information at that time is recorded, and the ID and coordinate information are matched.

8. A control system for a replanting swing-arm operation, which implements the replanting swing-arm operation positioning method according to any one of claims 1 to 7, characterized in that: include: An embedded device, comprising a sugarcane seedling feature recognition network model, the embedded device being installed on the replanting machine to form a control unit; a display screen, which is data-connected to the embedded device, and is used to mark locations that require replanting; A binocular camera, comprising at least two binocular cameras, at least one of which is mounted on the front end of the replanting machine cab, and at least one of which is mounted on the front end of the replanting machine's mechanical arm, wherein the binocular camera is data-connected to the embedded device to perform real-time identification and distance measurement of the sugarcane seedlings; as well as The execution unit includes a robotic arm, which is data-connected to the embedded device. The extension and swing of the robotic arm are controlled by the embedded device to complete the replanting operation.

9. A control method based on the control system of the replanting swing arm operation according to claim 8, characterized in that: The following steps are involved: (1) Binocular cameras perform real-time identification and distance measurement of sugarcane seedlings; (2) The embedded device obtains the coordinate information and depth information of the sugarcane seedlings and performs internal calculations to obtain the location where replanting is required; (3) When reseeding multiple planting ridges, the coordinate information of the reseeding position is stored in multiple lists according to the different ridges. The reseeding point closest to the reseeding machine is found and the ridge where it is located is marked as ridge 1. If ridge 1 is the middle ridge, after the reseeding point on the current ridge is completed, the point closest to the reseeding machine is found in the remaining multiple lists and the ridge where it is located is marked as ridge 2. The same marking is repeated for ridges 3...n. The reseeding operation of ridge 2 is processed first and then the reseeding operation of the ridges marked in sequence is processed. If ridge 1 is the left or right ridge, the middle ridge is directly marked as ridge 2, and the ridge on the farthest side is marked as ridge 3...n. Then the reseeding operation is processed in sequence according to the marked sequence number. (4) The embedded device controls the hydraulic solenoid valve to drive the robotic arm to rotate, and rotates a certain angle to reach above the replanting position, thereby completing the replanting work.

Images