Pineapple picking control method and system based on image recognition positioning and picking machine

Abstract

This invention discloses a pineapple harvesting control method, system, and harvesting machine based on image recognition and positioning. The invention relates to the field of agricultural equipment technology and includes the following steps: First, images of the pineapple planting area are acquired; a machine learning network is used to identify the pineapple target to be harvested and its location is marked as the target task point; then, the pineapple harvesting machine is controlled to move to the task point, acquire images of the pineapple plants to be harvested, and gradually slides downwards at a set distance to monitor height; the width and curvature changes of the detected images in adjacent height monitoring intervals are analyzed to determine the sliding distance, and then the initial shearing height is set; hyperspectral reflectance data below this height is acquired, and the machine slides to the detection interval at a preset step size, matching it with stem component data, and then stops sliding to determine the precise shearing interval; finally, the precise shearing position is determined for harvesting, achieving efficient and accurate pineapple harvesting.

Description

Technical Field

[0001] This invention relates to the field of agricultural equipment technology, specifically to a pineapple harvesting control method, system, and harvesting machine based on image recognition and positioning. Background Technology

[0002] With the development of modern agriculture towards intelligence and automation, traditional manual harvesting methods are gradually becoming unable to meet the needs of large-scale planting. As a tropical fruit with high economic value and a wide planting area, pineapple harvesting has always been an indispensable part of the production process. However, pineapple harvesting presents certain technical difficulties and unique characteristics, making manual harvesting inefficient and costly, while also making it difficult to guarantee the preservation of the fruit's integrity.

[0003] The growth characteristics of pineapples make their harvesting process relatively complex. Firstly, the rough scales covering the fruit and the dense growth of the fruit alongside the leaves make manual harvesting risky, as it increases the risk of injury from sharp leaves. Secondly, the growing environment typically requires harvesters to work for extended periods in high temperature and humidity, reducing efficiency and increasing labor intensity and costs. Thirdly, the fruit is connected to the stem by a hidden section, the width and angle of which vary depending on the fruit's ripeness. Accurate identification and precise cutting of this section are crucial during harvesting to avoid damaging the fruit and affecting its market value.

[0004] In the prior art, CN108575283A discloses a control method and control system for a pineapple harvesting robot. The specific scheme includes the pineapple harvesting robot autonomously planning its path, identifying the pineapple and obtaining its harvesting coordinates through image recognition, analyzing the harvesting coordinates, and adjusting the horizontal and vertical positions of the cutting module to move it to the stem of the pineapple to be harvested, facilitating the closing and cutting action until the harvesting target is met and the robot returns to its docking point. However, the pineapple stem is usually quite hidden, especially in dense plant environments, where intact fruit may be obscured or surrounded by leaves, limiting the visual information visible in the image. While image recognition technology can identify the appearance of the fruit, it may be difficult to accurately identify the pineapple stem in complex backgrounds, especially under conditions of changing light, shadows, and leaf obstruction. Therefore, relying solely on image recognition to determine the connection position may lead to misjudgments, resulting in fruit damage or the need for secondary processing, thus reducing the accuracy and effectiveness of harvesting.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a pineapple harvesting control method, system, and harvesting machine based on image recognition and positioning, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A pineapple harvesting control method based on image recognition and localization, comprising the following steps:

[0009] Collect regional images of the pineapple planting area, identify several pineapple targets to be picked in the images based on machine learning networks, and mark the picking location of each pineapple target as the target task point;

[0010] Control the pineapple harvester to go to the target task point, collect images of the pineapple plants to be harvested to extract the images of the pineapples to be harvested, record them as the detection images, and set the height monitoring range that gradually slides down from the top of the detection image according to the set sliding distance.

[0011] The changes in width and curvature of the detected image within each height monitoring interval are analyzed. When the slip termination condition is reached, the sliding distance is determined based on the minimum width of the detected image within the lowest height monitoring interval. The height at which the minimum width is located is moved down by the corresponding sliding distance and recorded as the initial shearing height.

[0012] The system acquires hyperspectral reflectance data of pineapple plants below the initial shearing height. It sets a hyperspectral detection range that gradually slides downwards from the initial shearing height according to a preset detection step size. The sliding stops when the hyperspectral reflectance data within the hyperspectral detection range matches the stem component content data of the pineapple plant. The lowest hyperspectral detection range is taken as the precise shearing range, so as to control the pineapple harvester to harvest the pineapple target within the height range of the precise shearing range.

[0013] Furthermore, the specific method for acquiring regional images within the pineapple planting area is as follows: determining the range of the pineapple planting area, establishing a spatial coordinate system within the pineapple planting area, acquiring depth images of the pineapple planting area at image acquisition points using a binocular camera, and preprocessing the depth images, including image denoising and image enhancement preprocessing, and using the preprocessed depth images as regional images.

[0014] The logic behind identifying several pineapple targets to be picked in an image based on a machine learning network is as follows: the machine learning network is specifically established through a convolutional neural network, which consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer.

[0015] The specific logic for training the convolutional neural network is as follows: several images of pineapple planting areas are collected, and pineapple targets within the area images are marked manually. Specifically, the pineapple targets are marked using the minimum bounding rectangle. The unmarked area images are used as the input to the convolutional neural network, and the marked area images are used as labels to train the convolutional neural network. The input of the trained convolutional neural network is the area image within the pineapple planting area, and the output is the area image with the marked pineapple targets.

[0016] The logic behind marking the picking location of each pineapple target as the target task point is as follows: Within the pineapple planting area, a picking route is planned and its boundaries are recorded. Based on the area image of the marked pineapple targets, the coordinates of the pineapple targets are determined using the image's depth data. The point within the picking route boundary closest to the pineapple target is taken as the picking location, which is the target task point.

[0017] Furthermore, the logic for controlling the pineapple harvester to go to the target task points is as follows: based on each target task point and the start and end points of the harvesting task, a harvesting path is generated through a path planning algorithm. The harvesting path specifically refers to the shortest path that starts from the start point of the harvesting task, covers all target task points, and finally returns to the task end point. The pineapple harvester goes to different target task points in sequence according to the harvesting path to harvest.

[0018] Furthermore, the logic behind acquiring images of pineapple plants to be harvested to extract images of pineapples to be harvested is as follows: acquire a visible light image of the pineapple plant to be harvested, convert the visible light image into a grayscale image, extract the contour edge information of the pineapple fruit to be harvested through an edge detection algorithm, separate the pineapple fruit to be harvested from the background, and obtain the image of the pineapple to be harvested; at the same time, determine the mapping relationship between the detected image position and the actual height.

[0019] Furthermore, the height of the height monitoring interval is measured in pixels, and is at least the height of two rows of pixels. The logic for obtaining the change in width and curvature within each height monitoring interval is as follows:

[0020] 1) Count the number of pixels in each row within the height monitoring interval and take the average value to use as the width of the height monitoring interval. Calculate the rate of change of the number of pixels in adjacent rows within the height monitoring interval and take the average value to use as the curvature of the height monitoring interval. The formula for calculating the curvature is as follows:

[0021] ;

[0022] In the formula, Let be the curvature of the k-th height monitoring interval. and , respectively, represent the number of pixels in the i-th row and the (i-1)-th row within the height monitoring interval, where i is the index of the pixel row within the height monitoring interval. is the total number of pixel rows within the height monitoring interval, and k is the index of the height monitoring interval;

[0023] 2) Based on the width and curvature of each height monitoring interval, calculate the rate of change of width and the rate of change of curvature. The specific mathematical expressions are as follows:

[0024] ;

[0025] ;

[0026] In the formula, Let be the width value of the k-th height monitoring interval. , These represent the changes in width and curvature of the k-th height monitoring interval, respectively.

[0027] The sliding height monitoring range is gradually lowered according to the set sliding distance, where the set sliding distance is in units of pixel height, and the minimum sliding distance is one row of pixels;

[0028] The slip termination condition is specifically set through the width value, the amount of width change, and the amount of curvature change, as follows:

[0029] ;

[0030] In the formula, 1, These are the minimum width threshold and the maximum width threshold, based on the width of the pineapple fruit. 1 represents the threshold for width variation. 2 represents the threshold for curvature change.

[0031] Furthermore, the descent distance is determined based on the minimum width of the detected image within the lowest height monitoring range. The specific formula used to calculate the descent distance is as follows:

[0032] ;

[0033] In the formula, This is the initial value of the descent distance. The minimum width threshold is determined based on the stem width. This represents the minimum number of pixels in each row within the lowest height monitoring interval. This represents the distance traveled downhill.

[0034] Furthermore, hyperspectral images within the hyperspectral detection range are acquired, and hyperspectral reflectance data are extracted based on the hyperspectral images. The component content data is determined based on the hyperspectral reflectance data, which includes moisture, sugar, cellulose, and lignin.

[0035] The sliding process stops when the hyperspectral reflectance data within the hyperspectral detection interval matches the stem component content data of the pineapple plant. The specific logic for determining whether the hyperspectral reflectance data matches the stem component content data is as follows: Samples of stem component content data from pineapple plants are collected; the average stem component content of different data samples is calculated as a reference value; the pineapple plant component content data within each hyperspectral detection interval is determined based on the hyperspectral reflectance data; a similarity coefficient is calculated between this similarity coefficient and the reference value; and the similarity coefficient is used to determine whether the hyperspectral reflectance data within the hyperspectral detection interval matches the stem component content data. The specific formula for calculating the similarity coefficient is as follows:

[0036] ;

[0037] In the formula, Let be the similarity coefficient within the p-th hyperspectral detection interval. and These represent the moisture and sugar content within the p-th hyperspectral detection interval, respectively. and These represent the cellulose and lignin contents within the p-th hyperspectral detection interval, respectively. and These are reference values ​​for moisture and sugar content, respectively. and These are the reference values ​​for cellulose and lignin content, respectively. and Here are the weighting coefficients, where and and All are greater than 0, where p is the index of the hyperspectral detection range;

[0038] The logic behind determining whether hyperspectral reflectance data matches the stem component content data of pineapple plants based on similarity coefficients is as follows:

[0039] like If the component content in the p-th hyperspectral detection interval does not match the stem component content data, then continue to gradually slide downwards.

[0040] like If the component content of the p-th hyperspectral detection interval matches the stem component content data, then the current hyperspectral detection interval is taken as the precise shearing interval; where This is the similarity matching threshold.

[0041] This invention also provides a pineapple harvesting control system based on image recognition and positioning. This system is used to execute the aforementioned pineapple harvesting control method based on image recognition and positioning, and includes:

[0042] The pineapple target recognition module is used to collect regional images within the pineapple planting area, identify several pineapple targets to be picked in the images based on a machine learning network, and mark the picking location of each pineapple target as a target task point;

[0043] The image optimization and detection module is used to control the pineapple harvester to move to the target task point, collect images of the pineapple plants to be harvested, extract the images of the pineapples to be harvested, record them as the detection images, and set the height monitoring range that gradually slides down from the top of the detection image according to the set sliding distance.

[0044] The initial detection and analysis module is used to analyze the changes in width and curvature of the detection image in each height monitoring interval. When the sliding termination condition is reached, the sliding distance is determined based on the minimum width of the detection image in the lowest height monitoring interval. The height where the minimum width is located is moved down by the corresponding sliding distance is recorded as the initial shearing height.

[0045] The precise shearing determination module is used to acquire hyperspectral reflectance data of the pineapple plant below the initial shearing height. It sets a hyperspectral detection range that gradually slides downwards from the initial shearing height according to a preset detection step size. The sliding stops when the hyperspectral reflectance data in the hyperspectral detection range matches the stem component content data of the pineapple plant. The lowest hyperspectral detection range is taken as the precise shearing range, so as to control the pineapple harvester to harvest the pineapple target within the height range of the precise shearing range.

[0046] The present invention also provides a pineapple harvesting machine based on image recognition and positioning, used to execute the above-mentioned pineapple harvesting control method based on image recognition and positioning to complete the pineapple harvesting.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] By using machine learning networks to accurately identify pineapples within the planting area, multiple pineapple targets to be harvested can be effectively marked, thereby simplifying the task planning of the harvesting robot, improving the accuracy of target recognition, and ensuring that the harvesting robot can reach the target task point quickly and accurately.

[0049] After the harvester is guided to the target task point, the acquired visible light images are used to extract the outline features of the pineapple plant. By identifying the outline features, the approximate range of the connection position between the bottom of the pineapple fruit and the stem is determined, which serves as the initial cutting height and provides a reliable basis for subsequent cutting operations.

[0050] By collecting hyperspectral reflectance data of the initial shearing height and combining it with the analysis of component content data, the introduction of hyperspectral data can effectively distinguish between fruit and stem parts, enhance the accuracy of positioning, overcome the identification error caused by environmental complexity, reduce the risk of damage during fruit picking, and improve the quality of harvested fruit. Finally, by setting detection convergence conditions, the system can automatically optimize the shearing position to ensure that the shearing operation is performed at the optimal time, thereby improving picking efficiency. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the overall method flow of the present invention;

[0052] Figure 2 A curve fitting the minimum number of pixels minus the downward sliding distance;

[0053] Figure 3 A line graph comparing the breakage rate of harvesting by ordinary machines with the breakage rate of harvesting in the precise cutting zone;

[0054] Figure 4 A bar chart comparing the length of stalks harvested by ordinary machines with the length of stalks harvested in the precise cutting range;

[0055] Figure 5 This is a schematic diagram of the overall system structure of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0057] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0058] Example:

[0059] Please see Figures 1-4 The present invention provides a technical solution:

[0060] A pineapple harvesting control method based on image recognition and localization, comprising the following steps:

[0061] Step 1: Collect regional images of the pineapple planting area, identify several pineapple targets to be picked in the images based on machine learning networks, and mark the picking location of each pineapple target as the target task point.

[0062] The specific method for collecting regional images within the pineapple planting area is as follows: determine the range of the pineapple planting area, establish a spatial coordinate system within the pineapple planting area, collect depth images of the pineapple planting area at the image acquisition point using a binocular camera, and preprocess the depth images, including image denoising and image enhancement preprocessing, and use the preprocessed depth images as regional images.

[0063] The purpose of image denoising is to remove unnecessary noise from an image, making it clearer and improving the accuracy of subsequent processing. Common image denoising methods include mean filtering, median filtering, Gaussian filtering, and bilateral filtering. The purpose of image enhancement is to improve the visual effect of an image, making it more suitable for subsequent processing or analysis. Common image enhancement methods include histogram equalization, gamma correction, and data augmentation.

[0064] Specific methods for determining the pineapple planting area include: using GPS equipment or geographic mapping tools to obtain the latitude and longitude range of the pineapple planting area, clearly delineating the boundaries of the planting area through manual marking or drone aerial photography, and converting it into an actual geographical area;

[0065] The logic behind identifying several pineapple targets to be picked in an image based on a machine learning network is as follows: the machine learning network is specifically built through a convolutional neural network, which consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer, wherein the activation function in the convolutional layer is the ReLU function;

[0066] The specific logic for training the convolutional neural network is as follows: several images of pineapple planting areas are collected, and the pineapple targets in the images are marked manually. Specifically, the pineapple targets are marked using the minimum bounding rectangle. The unmarked pineapple planting area images are used as inputs to the convolutional neural network, and the corresponding labeled images are used as labels to train the convolutional neural network. The input of the trained convolutional neural network is the regional images within the pineapple planting area, and the output is the image data of the marked pineapple targets.

[0067] A large number of images were collected from pineapple growing areas to ensure image diversity, including different angles, lighting, backgrounds, and plant densities. Basic image processing, including scaling, denoising, and enhancement, was performed on the collected images to improve the robustness and accuracy of the model. Image processing tools, such as LabelImg and VGG Image Annotator, were used to label the pineapple targets in the images. The minimum bounding rectangle method was used, and each pineapple target was labeled as a rectangle in the image, and its coordinates, such as the coordinates of the upper left and lower right corners, were recorded. Each label was assigned a corresponding label, usually "pineapple".

[0068] The labeled image dataset is divided into training and validation sets. Typically, 70%-80% of the data is used for training, and the remainder is used for validation to ensure the model's generalization ability. An appropriate loss function is selected to evaluate the model's prediction performance. For object detection tasks, commonly used loss functions include cross-entropy loss, regression loss, and non-maximum suppression. An appropriate optimizer, such as Adam or SGD, is selected to update the model weights.

[0069] The performance of the trained model is evaluated on the validation set using metrics such as precision, recall, F1-score, mean precision, etc. If the performance metrics meet the requirements, the training is completed; otherwise, the training is repeated.

[0070] The logic behind marking the picking location of each pineapple target as the target task point is as follows: Within the pineapple planting area, a picking route is planned and its boundaries are recorded. Based on the area image of the marked pineapple targets, the coordinates of the pineapple targets are determined using the image's depth data. The point closest to the pineapple target's location on the boundary of the picking route is taken as the picking location, which is the target task point.

[0071] Based on the size and shape of the planting area, the area is divided into multiple walking zones, usually planned in the form of a grid or path. The walking zones can be regular straight lines, curves, or complex paths that incorporate the terrain. The boundaries of each walking zone, usually a rectangular area or path range, are recorded and marked as the scope of the harvesting walking zone, and the boundary information is recorded using a spatial coordinate system.

[0072] The logic behind controlling the pineapple harvester to go to the target task points is as follows: based on each target task point and the start and end points of the harvesting task, a harvesting path is generated through a path planning algorithm. The harvesting path specifically refers to the shortest path that starts from the start point of the harvesting task, covers all target task points, and finally returns to the task end point. The pineapple harvester goes to different target task points in sequence according to the harvesting path to harvest.

[0073] The specific method for generating the picking route through path planning algorithms is as follows: use path planning algorithms, such as greedy algorithms, A* algorithms, or Dijkstra's algorithms, to optimize the picking route, determine the picking order of the task points based on the spatial distribution of the target task points, and ensure that the travel distance is shortest and the time is minimized.

[0074] Step 2: Control the pineapple harvester to go to the target task point, collect images of the pineapple plants to be harvested to extract the images of the pineapples to be harvested, record them as the detection images, and set the height monitoring range that gradually slides down from the top of the detection image according to the set sliding distance.

[0075] The specific logic for acquiring images of pineapple plants to be harvested in order to extract images of pineapples to be harvested is as follows: obtain a visible light image of the pineapple plant to be harvested, convert the visible light image into a grayscale image, extract the contour edge information of the pineapple fruit to be harvested through an edge detection algorithm, separate the pineapple fruit to be harvested from the background, and obtain the image of the pineapple to be harvested; at the same time, determine the mapping relationship between the detected image position and the actual height.

[0076] The specific methods for converting a visible light image to a grayscale image are as follows: First, ensure that a visible light image of the pineapple plant to be harvested has been obtained. Visible light images are usually in RGB format, with each pixel containing information from three channels: red, green, and blue. Common methods for converting visible light images to grayscale images include: Weighted average method: Iterate through the RGB values ​​of each pixel, calculate the grayscale value using a preset weighting coefficient, and replace the original pixel's RGB value with the grayscale value; Mean method: Iterate through each pixel, calculate the average of its three RGB channels, and replace the original pixel's RGB value with the calculated average value; Maximum value method: Iterate through each pixel, take the maximum value among the three RGB channels as the grayscale value, and replace the original pixel's RGB value; Minimum value method: Iterate through each pixel, take the minimum value among the three RGB channels as the grayscale value, and replace the original pixel's RGB value.

[0077] The contour edge information of the pineapple fruit to be harvested is extracted by edge detection algorithms to separate the pineapple fruit from the background. The specific methods used include Canny edge detection, Sobel operator, and Laplacian operator. From the edge detection results, contour detection algorithms, such as OpenCV's findContours method, are used to extract the contour edge of the pineapple fruit. Based on the extracted contour information, the pineapple fruit area is marked or filled in the original image. A mask image is used to separate the pineapple plant from the background to generate a detection image.

[0078] The height monitoring range is gradually lowered according to the set sliding distance, which is in units of pixel width, with the minimum sliding distance being one row of pixels.

[0079] The specific method for determining the mapping relationship between the detected image position and the actual height is as follows: Data containing position and height information needs to be acquired. This data can be obtained in the following ways: directly measuring the plant height using sensors such as laser rangefinders or ultrasonic sensors; analyzing plant images using image processing techniques and combining them with known reference scales for measurement; selecting specific points from the image, such as baselines, the bottom and top of the plant, and recording the corresponding height values ​​of these points in the actual height coordinate system to obtain image coordinates; using image processing tools, such as OpenCV, to extract the coordinates of these specific points in the image. For irregular height variations, interpolation methods, such as spline interpolation, can be used to establish the mapping, thus determining the mapping relationship between the detected image position and the actual height.

[0080] Step 3: Analyze the changes in width and curvature of the detection image in each height monitoring interval. When the slip termination condition is reached, determine the sliding distance based on the minimum width of the detection image in the lowest height monitoring interval. After moving the height of the minimum width downward by the corresponding sliding distance, the height where it is located is recorded as the initial shearing height.

[0081] The height of the height monitoring interval is measured in pixels, and is at least the height of two rows of pixels. The logic for obtaining the change in width and curvature within each height monitoring interval is as follows:

[0082] 1) Count the number of pixels in each row within the height monitoring interval and take the average value to use as the width of the height monitoring interval. Calculate the rate of change of the number of pixels in adjacent rows within the height monitoring interval and take the average value to use as the curvature of the height monitoring interval. The formula for calculating the curvature is as follows:

[0083] ;

[0084] In the formula, Let be the curvature of the k-th height monitoring interval. and , respectively, represent the number of pixels in the i-th row and the (i-1)-th row within the height monitoring interval, where i is the index of the pixel row within the height monitoring interval. is the total number of pixel rows within the height monitoring interval, and k is the index of the height monitoring interval;

[0085] In this method, curvature is used to measure the width variation of a pineapple to be harvested within different height monitoring intervals in an image, reflecting the pineapple's morphological characteristics. By analyzing curvature changes, the position and state of the pineapple can be located more effectively, providing a basis for harvesting control. This formula calculates the difference in the number of pixels in adjacent rows (…). ) and the number of pixels in the current row ( The ratio of curvature to width reflects the relative speed of width change. This setting allows curvature to reflect not only the amount of width change, but also the degree of change relative to the current width, thus more effectively capturing subtle differences in shape changes.

[0086] Calculate the mean To reduce the impact of random fluctuations on curvature calculations, averaging the pixel count changes across multiple rows yields a more stable and reliable curvature value. This processing method effectively improves the accuracy of the results and avoids noise interference caused by individual data points.

[0087] 2) Based on the width and curvature of each height monitoring interval, calculate the rate of change of width and the rate of change of curvature. The specific mathematical expressions are as follows:

[0088] ;

[0089] ;

[0090] In the formula, Let be the width value of the k-th height monitoring interval, specifically the average width value within the height monitoring interval. , These represent the changes in width and curvature of the k-th height monitoring interval, respectively.

[0091] The sliding height monitoring range is gradually lowered according to the set sliding distance, where the set sliding distance is in units of pixel height, and the minimum sliding distance is one row of pixels;

[0092] The slip termination condition is specifically set through the width value, the amount of width change, and the amount of curvature change, as follows:

[0093] ;

[0094] In the formula, 1, These are the minimum width threshold and the maximum width threshold, based on the width of the pineapple fruit. 1 represents the threshold for width variation. 2 represents the threshold for curvature change.

[0095] It should be noted that, it should be noted that, A condition is represented in the first... Within one height monitoring interval and the next height monitoring interval, the width of the pineapple changes drastically. This indicates that the height monitoring interval is close to the connection between the fruit and the stem, because at this part, the width of the fruit may increase drastically due to leaf shading, or the width value changes drastically as the pineapple transitions from the fruit to the stem.

[0096] The principle behind setting this condition is the same as that for the change in width. Leaf shading or the transition of the pineapple fruit to the stem will increase the magnitude of the change in curvature. 1, The specific settings are usually determined based on expert experience.

[0097] This condition indicates that if the width of the height monitoring interval is less than... At this time, it indicates that the height monitoring range may be located at the junction of the fruit and the stem, because the width of the stem is significantly reduced compared to the width of the fruit.

[0098] If the width of the height monitoring range is greater than When the height monitoring interval is located in the fruit area, it indicates that there may be leaves blocking the view. The area of ​​leaves causes the width of the height monitoring interval to increase, thus satisfying the slippage termination condition. If any one of the slippage termination conditions is met, the downward slippage will stop.

[0099] in 1. The specific settings should be based on the maximum width of the pineapple fruit combined with expert experience. The setting is based on the average width of the stalk combined with expert experience.

[0100] The descent distance is determined based on the minimum width of the detected image within the lowest height monitoring range. The specific formula used to calculate the descent distance is as follows:

[0101] ;

[0102] In the formula, This is the initial value of the descent distance. The minimum width threshold is determined based on the stem width. This represents the minimum number of pixels in each row within the lowest height monitoring interval. The sliding distance is defined by a minimum width threshold. The specific settings should be based on expert experience.

[0103] It should be noted that by dynamically correcting the initial value of the descent distance, the aim is to adaptively adjust the range of the detection zone based on the changes in width and curvature of the current height monitoring zone, so as to ensure effective capture of the connection between the pineapple fruit and the stem.

[0104] The minimum number of pixels in each row within the lowest height monitoring interval. This reflects the width characteristics of the current height monitoring range. A smaller value indicates proximity to the junction of the fruit and stem. In this case, the appropriate range of the detection interval should be reduced to ensure accurate positioning during cutting. A larger reading indicates that the area is far from the connection point between the fruit and the stem, requiring an increased detection range to ensure the connection point is covered and to avoid missing any cuts.

[0105] Step 4: Obtain hyperspectral reflectance data of the pineapple plant below the initial shearing height. Set a hyperspectral detection range starting from the initial shearing height and gradually sliding downwards according to a preset detection step size. Stop sliding when the hyperspectral reflectance data in the hyperspectral detection range matches the stem component content data of the pineapple plant. The lowest hyperspectral detection range is taken as the precise shearing range to control the pineapple harvester to harvest the pineapple target within the height range of the precise shearing range.

[0106] Hyperspectral images within the hyperspectral detection range are acquired, and hyperspectral reflectance data are extracted based on the hyperspectral images. The component content data, including moisture, sugar, cellulose, and lignin, is determined based on the hyperspectral reflectance data.

[0107] Hyperspectral imaging technology can simultaneously acquire reflectance spectral information of a target at multiple wavelengths, making it ideal for analyzing the composition of plants. Hyperspectral data allows for the identification and quantification of the content of different substances within the plant, such as water, sugars, cellulose, and lignin. Hyperspectral images are acquired at each set height location, and reflectance data at each wavelength is recorded. Selecting an appropriate wavelength range is crucial for component analysis, as different components exhibit specific absorption characteristics within the hyperspectral range. For example: water typically shows significant absorption characteristics around 1,450 nm and 1,940 nm; sugars have a significant absorption band around 1,600 nm; cellulose has a prominent absorption characteristic around 1,740 nm; and lignin typically absorbs light around 1,700 nm and 1,900 nm.

[0108] The sliding process stops when the hyperspectral reflectance data within the hyperspectral detection interval matches the stem component content data of the pineapple plant. The specific logic for determining whether the hyperspectral reflectance data matches the stem component content data is as follows: Samples of stem component content data from pineapple plants are collected; the average stem component content of different data samples is calculated as a reference value; the pineapple plant component content data within each hyperspectral detection interval is determined based on the hyperspectral reflectance data; a similarity coefficient is calculated between this similarity coefficient and the reference value; and the similarity coefficient is used to determine whether the hyperspectral reflectance data within the hyperspectral detection interval matches the stem component content data. The specific formula for calculating the similarity coefficient is as follows:

[0109] ;

[0110] In the formula, Let be the similarity coefficient within the p-th hyperspectral detection interval. and These represent the moisture and sugar content within the p-th hyperspectral detection interval, respectively. and These represent the cellulose and lignin contents within the p-th hyperspectral detection interval, respectively. and These are reference values ​​for moisture and sugar content, respectively. and These are the reference values ​​for cellulose and lignin content, respectively. and Here are the weighting coefficients, where and and All are greater than 0, where p is the index of the hyperspectral detection range;

[0111] It should be noted that, This is used to indicate the degree of matching between the component content data and the stem component content data within the p-th hyperspectral detection interval. The higher the value, the lower the degree of matching;

[0112] Fruits typically have a high water content, especially during ripening. Stems, on the other hand, have a relatively low water content, particularly near the base. This difference in composition results in a significant variation in water content between fruits and stems. Fruits have a relatively high sugar content, giving them a strong sweetness, while stems generally have a lower sugar content. Therefore, variations in sugar content also manifest differently between fruits and stems. Stems typically have higher cellulose and lignin content, which provide structural support. Compared to fruits, fruits have relatively lower cellulose and lignin content, leading to a significant difference in cellulose and lignin content between the two.

[0113] in Measure the first The relative difference between the moisture content and the reference moisture content within each hyperspectral detection range. Measure the first The relative difference between the sugar content and the reference sugar content within a hyperspectral detection range eliminates the influence of the absolute values ​​of different components, making the comparison more equitable. For example, the range of moisture content variation may be larger than that of sugar content; using absolute differences would lead to an excessive influence of moisture on the similarity coefficient, ensuring that different components maintain a consistent scale in similarity calculations.

[0114] and The reason for this setting is the same, and will not be elaborated here.

[0115] The stem is mainly composed of cellulose and lignin. While these components play important roles in plant structural support and stress resistance, the water and sugar content of the stem is usually lower than that of the fruit. Therefore, water and sugar content can be used to more accurately distinguish between the fruit and the stem. and and All are greater than 0.

[0116] The logic behind determining whether hyperspectral reflectance data matches the stem component content data of pineapple plants based on similarity coefficients is as follows:

[0117] like If the component content in the p-th hyperspectral detection interval does not match the stem component content data, then continue to gradually slide downwards.

[0118] like If the component content of the p-th hyperspectral detection interval matches the stem component content data, then the current hyperspectral detection interval is taken as the precise shearing interval; where The similarity matching threshold should be set based on expert experience.

[0119] Please see Figure 5 The present invention also provides a pineapple harvesting control system based on image recognition and positioning. This system is used to execute the aforementioned pineapple harvesting control method based on image recognition and positioning, and includes:

[0120] The pineapple target recognition module is used to collect regional images within the pineapple planting area, identify several pineapple targets to be picked in the images based on a machine learning network, and mark the picking location of each pineapple target as a target task point;

[0121] The image optimization and detection module is used to control the pineapple harvester to move to the target task point, collect images of the pineapple plants to be harvested, extract the images of the pineapples to be harvested, record them as the detection images, and set the height monitoring range that gradually slides down from the top of the detection image according to the set sliding distance.

[0122] The initial detection and analysis module is used to analyze the changes in width and curvature of the detection image in each height monitoring interval. When the sliding termination condition is reached, the sliding distance is determined based on the minimum width of the detection image in the lowest height monitoring interval. The height where the minimum width is located is moved down by the corresponding sliding distance is recorded as the initial shearing height.

[0123] The precise shearing determination module is used to acquire hyperspectral reflectance data of the pineapple plant below the initial shearing height. It sets a hyperspectral detection range that gradually slides downwards from the initial shearing height according to a preset detection step size. The sliding stops when the hyperspectral reflectance data in the hyperspectral detection range matches the stem component content data of the pineapple plant. The lowest hyperspectral detection range is taken as the precise shearing range, so as to control the pineapple harvester to harvest the pineapple target within the height range of the precise shearing range.

[0124] The present invention also provides a pineapple harvesting machine based on image recognition and positioning, used to execute the above-mentioned pineapple harvesting control method based on image recognition and positioning to complete the pineapple harvesting.

[0125] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0126] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A pineapple harvesting control method based on image recognition and localization, characterized in that, The specific steps include: Collect regional images of the pineapple planting area, identify several pineapple targets to be picked in the images based on machine learning networks, and mark the picking location of each pineapple target as the target task point; Control the pineapple harvester to go to the target task point, collect images of the pineapple plants to be harvested to extract the images of the pineapples to be harvested, record them as the detection images, and set the height monitoring range that gradually slides down from the top of the detection image according to the set sliding distance. The changes in width and curvature of the detected image within each height monitoring interval are analyzed. When the sliding termination condition is reached, the sliding distance is determined based on the minimum width of the detected image within the lowest height monitoring interval. The height at which the minimum width is located is moved down by the corresponding sliding distance and recorded as the initial shearing height. The system acquires hyperspectral reflectance data of pineapple plants below the initial shearing height. It sets a hyperspectral detection range that gradually slides downwards from the initial shearing height according to a preset detection step size. The sliding stops when the hyperspectral reflectance data within the hyperspectral detection range matches the stem component content data of the pineapple plant. The lowest hyperspectral detection range is taken as the precise shearing range, so as to control the pineapple harvester to harvest the pineapple target within the height range of the precise shearing range.

2. The pineapple harvesting control method based on image recognition and positioning according to claim 1, characterized in that: The specific method for collecting regional images within the pineapple planting area is as follows: determine the range of the pineapple planting area, establish a spatial coordinate system within the pineapple planting area, collect depth images of the pineapple planting area at the image acquisition point using a binocular camera, and preprocess the depth images, including image denoising and image enhancement preprocessing, and use the preprocessed depth images as regional images. The logic behind identifying several pineapple targets to be picked in an image based on a machine learning network is as follows: the machine learning network is specifically established through a convolutional neural network, which consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The specific logic for training the convolutional neural network is as follows: several images of pineapple planting areas are collected, and pineapple targets within the area images are marked manually. Specifically, the pineapple targets are marked using the minimum bounding rectangle. The unmarked area images are used as the input to the convolutional neural network, and the marked area images are used as labels to train the convolutional neural network. The input of the trained convolutional neural network is the area image within the pineapple planting area, and the output is the area image with the marked pineapple targets. The logic behind marking the picking location of each pineapple target as the target task point is as follows: Within the pineapple planting area, a picking route is planned and its boundaries are recorded. Based on the area image of the marked pineapple targets, the coordinates of the pineapple targets are determined using the image's depth data. The point on the boundary of the picking route that is closest to the pineapple target is taken as the picking location, which is the target task point.

3. The pineapple harvesting control method based on image recognition and positioning according to claim 2, characterized in that: The logic behind controlling the pineapple harvester to go to the target task points is as follows: based on each target task point and the start and end points of the harvesting task, a harvesting path is generated through a path planning algorithm. The harvesting path specifically refers to the shortest path that starts from the start point of the harvesting task, covers all target task points, and finally returns to the task end point. The pineapple harvester goes to different target task points in sequence according to the harvesting path to harvest.

4. The pineapple harvesting control method based on image recognition and positioning according to claim 1, characterized in that: The specific logic for acquiring images of pineapple plants to be harvested in order to extract images of pineapples to be harvested is as follows: obtain a visible light image of the pineapple plant to be harvested, convert the visible light image into a grayscale image, extract the contour edge information of the pineapple fruit to be harvested through an edge detection algorithm, separate the pineapple fruit to be harvested from the background, and obtain the image of the pineapple to be harvested; at the same time, determine the mapping relationship between the detected image position and the actual height.

5. The pineapple harvesting control method based on image recognition and positioning according to claim 4, characterized in that: The height of the height monitoring interval is measured in pixels, and must be at least the height of two rows of pixels. The logic for obtaining the changes in width and curvature within each height monitoring interval is as follows: 1) Count the number of pixels in each row within the height monitoring interval and take the average value to use as the width of the height monitoring interval. Calculate the rate of change of the number of pixels in adjacent rows within the height monitoring interval and take the average value to use as the curvature of the height monitoring interval. The formula for calculating the curvature is as follows: ； In the formula, Let be the curvature of the k-th height monitoring interval. and , respectively, represent the number of pixels in the i-th row and the (i-1)-th row within the height monitoring interval, where i is the index of the pixel row within the height monitoring interval. is the total number of pixel rows within the height monitoring interval, and k is the index of the height monitoring interval; 2) Based on the width and curvature of each height monitoring interval, calculate the rate of change of width and the rate of change of curvature. The specific mathematical expressions are as follows: ； ； In the formula, Let be the width value of the k-th height monitoring interval. , These represent the changes in width and curvature of the k-th height monitoring interval, respectively. The height monitoring range is gradually lowered according to the set sliding distance, where the set sliding distance is in units of pixel height, and the minimum sliding distance is one row of pixels; The slip termination condition is specifically set through the width value, the change in width value, and the change in curvature, as follows: ； In the formula, 1, These are the minimum width threshold and the maximum width threshold, based on the width of the pineapple fruit. 1 represents the threshold for width variation. 2 represents the threshold for curvature change.

6. The pineapple harvesting control method based on image recognition and positioning according to claim 5, characterized in that: The descent distance is determined based on the minimum width of the detected image within the lowest height monitoring range. The specific formula used to calculate the descent distance is as follows: ； In the formula, This is the initial value of the descent distance. The minimum width threshold is determined based on the stem width. This represents the minimum number of pixels in each row within the lowest height monitoring interval. This represents the distance traveled downhill.

7. The pineapple harvesting control method based on image recognition and positioning according to claim 6, characterized in that: Hyperspectral images within the hyperspectral detection range are acquired, and hyperspectral reflectance data are extracted based on the hyperspectral images. The component content data, including moisture, sugar, cellulose, and lignin, are determined based on the hyperspectral reflectance data. The sliding process stops when the hyperspectral reflectance data within the hyperspectral detection interval matches the stem component content data of the pineapple plant. The specific logic for determining whether the hyperspectral reflectance data matches the stem component content data is as follows: Samples of stem component content data from pineapple plants are collected; the average stem component content of different data samples is calculated as a reference value; the pineapple plant component content data within each hyperspectral detection interval is determined based on the hyperspectral reflectance data; a similarity coefficient is calculated between this similarity coefficient and the reference value; and the similarity coefficient is used to determine whether the hyperspectral reflectance data within the hyperspectral detection interval matches the stem component content data. The specific formula for calculating the similarity coefficient is as follows: ； In the formula, Let be the similarity coefficient within the p-th hyperspectral detection interval. and These represent the moisture and sugar content within the p-th hyperspectral detection interval, respectively. and These represent the cellulose and lignin contents within the p-th hyperspectral detection interval, respectively. and These are reference values ​​for moisture and sugar content, respectively. and These are the reference values ​​for cellulose and lignin content, respectively. and Here are the weighting coefficients, where and and All are greater than 0, where p is the index of the hyperspectral detection range; The logic behind determining whether hyperspectral reflectance data matches the stem component content data of pineapple plants based on similarity coefficients is as follows: like If the component content in the p-th hyperspectral detection interval does not match the stem component content data, then continue to gradually slide downwards. like If the component content of the p-th hyperspectral detection interval matches the stem component content data, then the current hyperspectral detection interval is taken as the precise shearing interval; where This is the similarity matching threshold.

8. A pineapple harvesting control system based on image recognition and positioning, characterized in that: The image recognition-based positioning pineapple harvesting control system is used to execute the image recognition-based positioning pineapple harvesting control method according to any one of claims 1-7, including: The pineapple target recognition module is used to collect regional images within the pineapple planting area, identify several pineapple targets to be picked in the images based on a machine learning network, and mark the picking location of each pineapple target as a target task point; The image optimization and detection module is used to control the pineapple harvester to move to the target task point, collect images of the pineapple plants to be harvested, extract the images of the pineapples to be harvested, record them as the detection images, and set the height monitoring range that gradually slides down from the top of the detection image according to the set sliding distance. The initial detection and analysis module is used to analyze the changes in width and curvature of the detection image in each height monitoring interval. When the sliding termination condition is reached, the sliding distance is determined based on the minimum width of the detection image in the lowest height monitoring interval. The height where the minimum width is located is moved down by the corresponding sliding distance is recorded as the initial shearing height. The precise shearing determination module is used to acquire hyperspectral reflectance data of the pineapple plant below the initial shearing height. It sets a hyperspectral detection range that gradually slides downwards from the initial shearing height according to a preset detection step size. The sliding stops when the hyperspectral reflectance data in the hyperspectral detection range matches the stem component content data of the pineapple plant. The lowest hyperspectral detection range is taken as the precise shearing range, so as to control the pineapple harvester to harvest the pineapple target within the height range of the precise shearing range.

9. A pineapple harvesting machine based on image recognition and positioning, characterized in that: The pineapple harvesting machine based on image recognition and positioning includes the pineapple harvesting control system based on image recognition and positioning as described in claim 8.

Images