Tomato yield prediction method considering maturity

Abstract

The application discloses a tomato yield prediction method considering maturity. The method comprises the following steps: first, collecting an RGB tomato video of a tomato plant and a corresponding depth map; then, performing target identification and target tracking on the RGB tomato video respectively to obtain tomato identification results corresponding to each frame of RGB tomato image and interframe corresponding relationship of the fruits; then, based on the tomato identification results of each frame, the maturity corresponding to all tomato fruits in the frame is calculated; then, combining the depth map and the maturity of each frame, the occlusion rate and the weight corresponding to all tomato fruits in the frame are calculated; finally, combining the interframe corresponding relationship of the fruits, the future yield of the tomatoes in the detection range is generated according to the maturity, the occlusion rate and the weight corresponding to different tomato fruits in all frames. The application estimates the yield of the fruits in the detection range based on the morphological characteristics of the tomato fruits, dynamically adjusts the estimation result in combination with the occlusion degree of the fruits, and can predict the maturity and weight information of the tomatoes in the future several days.

Description

Technical Field

[0001] This invention belongs to the field of non-destructive testing and intelligent sorting of agricultural products, and specifically relates to a method for predicting tomato yield that takes into account maturity. Background Technology

[0002] In mechanized tomato harvesting, there is a problem of waste of resources due to the presence of unripe fruits.

[0003] Furthermore, the tomato industry exhibits diverse requirements for tomato maturity. Depending on the degree of processing, storage time, season, climate, and transportation distance, the quality requirements and post-harvest preservation time vary, leading to differentiated demands for tomato maturity at harvest. However, current research on tomato maturity identification suffers from inconsistent grading standards and relatively coarse grading accuracy, making it difficult to achieve precise classification.

[0004] Patent CN120220139A discloses a method, system, and device for estimating and grading tomato maturity, which uses a deep learning model for tomato maturity estimation and grading. However, this patent uses manual grading and hue information from images. This method of estimating tomato maturity is affected by human annotation capabilities and requires a lot of annotation work, resulting in a relatively low accuracy rate for obtaining tomato maturity information.

[0005] Kim et al. from Korea University proposed a harvesting robot system based on a deep learning network using transform loss for classifying the maturity of lateral stem tomatoes. This system performs three-level maturity classification and 6D pose estimation for the target fruit. Zhang Li et al. from China Agricultural University designed a tomato harvesting robot classification system based on deep learning, classifying tomatoes into five levels according to different maturity stages. Besides obtaining maturity information from images, some researchers have used spectral information to determine the maturity stage of tomatoes. Researchers at the University of Lanchester in the UK used infrared spectral information to determine the maturity stage of tomatoes and proposed a support vector machine stoichiometric model to detect the maturity stage of tomatoes. However, the tomato maturity classifications in the above studies can only output coarse grading results within six levels, resulting in a rough classification.

[0006] Research on tomato yield prediction is limited and lacks unified standards. Existing studies mainly focus on crop growth and development simulation techniques, with common methods including effective accumulated temperature, physiological development time, and clock modeling. However, these methods aim to predict the phenological stage of the crop, only forecasting tomato maturity in terms of time, without addressing yield prediction by weight. Furthermore, they are all based on manual calculations, making correction and optimization difficult. Therefore, a method that is highly adaptable, easily optimized, and can automatically identify and predict target yields is lacking. Summary of the Invention

[0007] To address key issues in existing tomato maturity identification research, such as inconsistent grading standards, coarse grading accuracy, susceptibility to occlusion and background interference, and difficulty in refining maturity grading results, this invention proposes a tomato yield prediction method that considers maturity. This invention utilizes the morphological characteristics of tomato fruits to estimate the yield of fruits within a detection range and dynamically adjusts the estimation results based on the degree of fruit occlusion. It can predict the maturity and weight information of tomatoes in the coming days, as well as the number of harvestable fruits, to guide harvesting operations.

[0008] The technical solution adopted in this invention is as follows:

[0009] I. A method for predicting tomato yield considering maturity

[0010] Step 1: Collect RGB video of the tomato plant and the corresponding depth map;

[0011] Step 2: Perform target recognition and target tracking on each frame of the RGB tomato video to obtain the tomato recognition results corresponding to each frame of the RGB tomato video and the inter-frame correspondence of the fruit;

[0012] Step 3: Based on the tomato recognition results of each frame, calculate the maturity of all tomato fruits in that frame; then, combine the depth map and maturity of each frame to calculate the occlusion rate and weight of all tomato fruits in that frame; traverse all frames to obtain the occlusion rate and weight of different tomato fruits in all frames.

[0013] Step 4: Based on the inter-frame correspondence of the fruits, generate the future yield of tomatoes within the detection range according to the maturity, occlusion rate and weight of different tomato fruits in all frames.

[0014] In step 2, target recognition is performed on each frame of the RGB tomato image in the RGB tomato video, specifically including:

[0015] A tomato target recognition model based on deep learning instance segmentation is used to perform target recognition on each frame of RGB tomato image, and the model outputs tomato recognition results; in the tomato recognition results, the same tomato fruit is marked in the same recognition box, and the recognition box is a polygon.

[0016] In step 3, based on the tomato recognition results of each frame, the maturity of all tomato fruits in that frame is calculated, including:

[0017] The mask in the tomato recognition results is converted to HSV space. Then, the mode of the three channel values ​​of the mask corresponding to each tomato fruit in HSV space is determined. Then, the ripeness of the tomato fruit is calculated based on the mode of the three channels.

[0018] In step 3, combining the depth map and maturity level of each frame, the occlusion rate and weight of all tomato fruits in that frame are calculated, including:

[0019] The weight of each tomato fruit is calculated as follows:

[0020] First, an approximate circle is fitted to the recognition area of ​​each tomato fruit in the tomato recognition results to obtain the radius of the fitted circle for that tomato fruit. Next, the ratio of the area of ​​the recognition area to the area of ​​the fitted circle is calculated based on the fitted circle radius and recorded as the occlusion ratio Ratio. Then, the volume of the tomato fruit is calculated based on the camera intrinsic parameters, the depth map of the frame, and the fitted circle radius. Finally, the weight of the tomato fruit is calculated based on its volume and maturity.

[0021] The calculation of the tomato fruit volume based on camera intrinsic parameters, the depth map of the frame, and the radius of the fitted circle includes:

[0022] First, the depth value of the center position of each tomato fruit is determined based on the depth map; then, the radius of the fitted circle is converted into the true radius of the fruit by using camera intrinsic parameters and then corrected to obtain the corrected radius; finally, the volume of the tomato fruit is calculated based on the corrected radius.

[0023] The calculation of the tomato fruit's weight based on its volume and ripeness includes:

[0024] The density value is dynamically adjusted based on maturity level (MA) using the following formula:

[0025] ρ adjusted =ρ based ·(1-β·MA / 100)

[0026] Where, ρ based ρ is the base density of the tomato fruit; β is the correction factor; adjusted Corrected density for tomato fruit;

[0027] Then, calculate the weight of the tomato fruit based on its volume and corrected density.

[0028] In step 4, combining the inter-frame correspondence of the fruit, the future maturity and weight of the tomato fruit are generated based on the maturity, occlusion rate, and weight of the same tomato fruit in all frames. Specifically, this includes:

[0029] First, calculate the maturity weight of the tomato fruit in each frame based on its occlusion rate. Then, calculate the weighted maturity and weight of the tomato fruit in that frame based on the maturity weight. Next, sum the weighted maturity and weight of the tomato fruit in all frames to obtain the final weighted maturity and weight sum of the tomato fruit. Then, divide the final weighted maturity and weight sum of the tomato fruit by the maturity weight to obtain the final maturity and weight of the tomato fruit. Finally, generate the future maturity and weight of the tomato fruit based on its final maturity and weight.

[0030] II. A Tomato Yield Prediction System Considering Maturity

[0031] The data acquisition platform is used to collect RGB tomato videos and corresponding depth maps of tomato plants.

[0032] The target recognition unit is used to perform target recognition on each frame of the RGB tomato video and obtain the tomato recognition result corresponding to each frame of the RGB tomato video.

[0033] The target tracking unit is used to obtain the inter-frame correspondence of tomato fruits;

[0034] The maturity calculation unit is used to calculate the maturity of all tomato fruits in each frame based on the tomato recognition results of each frame.

[0035] The occlusion rate calculation unit is used to calculate the occlusion rate of all tomato fruits in each frame;

[0036] The weight calculation unit is used to calculate the weight of all tomato fruits in each frame;

[0037] The future tomato yield calculation unit is used to combine the inter-frame correspondence of fruits and generate the future yield of tomatoes in the detection area based on the maturity, occlusion rate and weight of different tomato fruits in all frames.

[0038] III. A computer device

[0039] The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method for predicting tomato yield considering maturity.

[0040] IV. A computer-readable storage medium

[0041] The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for predicting tomato yield considering maturity.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] 1. Unlike other studies that fix maturity grading levels, this invention combines the three parameters H, S, and V in the HSV color space to evaluate tomato fruit maturity using a numerical quantification method. The grading method is freely defined by the user. The numerical quantification method of maturity proposed in this invention is not affected by human annotation capabilities, has strong generalization ability, can better cope with changes in light and angle, and yields more accurate and reliable results.

[0044] 2. Unlike existing methods that use rectangular target boxes to identify tomato targets, this invention uses a method of finely outlining and marking tomato fruits with polygons to eliminate background clutter interference, making tomato target recognition more accurate, maximizing the identification of fruit pixels, and achieving high information utilization.

[0045] 3. Considering the shading of fruit and the uneven color changes during the ripening process of tomatoes, this invention judges the degree of shading and quantifies the reliability of the ripeness recognition results when identifying the ripeness.

[0046] 4. Based on the quantitative detection of tomato fruit maturity and weight estimation, this invention predicts the maturity status and yield of tomato fruits in the next few days, providing a reference and basis for harvesting and storage.

[0047] 5. The maturity calculation and yield prediction operations involved in this invention are simple, can be processed quickly, have low requirements for equipment performance, can be deployed and applied at low cost, and lower the hardware usage threshold. It is especially suitable for automated tomato harvesting, fine grading and supply chain management in facility agriculture environments (such as greenhouses). Attached Figure Description

[0048] Figure 1 This is a flowchart of the method of the present invention.

[0049] Figure 2 This is a diagram illustrating the annotation of tomato fruit data.

[0050] Figure 3 This is an example image of the tomato recognition results.

[0051] Figure 4 This is an example image showing the results of tomato fruit fitting circles and occlusion rate recognition.

[0052] Figure 5 This is an example image showing the results of tomato fruit maturity identification. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0054] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0055] like Figure 1 As shown, the tomato yield prediction method considering maturity proposed in this invention specifically includes the following steps:

[0056] Step 1: Set up the data acquisition platform. The data acquisition platform includes an RGB-D camera, casters, a chassis, and an adjustable camera bracket. The casters are mounted on the chassis, and the RGB-D camera is mounted on the chassis via the adjustable camera bracket. The lens of the RGB-D camera is approximately 0.3m from the tomato growing surface. The RGB-D camera is used to capture clear RGB tomato images and depth maps inside the tomato greenhouse. The chassis is constructed using an aluminum alloy profile support, and the adjustable camera bracket is a swivel arm. When filming tomato inspection videos, the platform's operating speed is controlled below 0.05m / s to ensure stable operation of the RGB-D camera.

[0057] RGB tomato videos and corresponding depth maps of tomato plants were collected using a data acquisition platform; the shooting angles of different frames of RGB tomato images are somewhat different.

[0058] Step 2: Perform target recognition and target tracking on each frame of the RGB tomato video to obtain the tomato recognition results corresponding to each frame of the RGB tomato image and the inter-frame correspondence of the fruit.

[0059] In one feasible implementation, target recognition is performed on each frame of RGB tomato images in the RGB tomato video, specifically including:

[0060] A tomato target recognition model based on deep learning instance segmentation is used to identify targets in each frame of RGB tomato images. The model outputs tomato recognition results, which include tomato fruit instance segmentation results and masks. The tomato fruit instance segmentation results are saved in TXT format, and the masks are in image format, such as... Figure 3 As shown in the tomato recognition results, the same tomato fruit is marked in the same recognition box, and the recognition box does not contain stems, leaves or other debris. The recognition box is a polygon.

[0061] Using the polygon annotation tool in Labelme data annotation software, select and annotate the outline of the target tomato in the foreground of the acquired RGB tomato image. The annotation range only includes the fruit part, excluding the stem, leaves, and other debris. Figure 2 As shown. If the continuity of a tomato fruit is interrupted by an obstruction, the control polygon range can be selected during annotation to annotate different parts of the same fruit within a single polygon, thus obtaining the annotated dataset.

[0062] In one feasible implementation, the tomato target recognition model based on deep learning instance segmentation is the Yolov8-Seg model. The labeled dataset is divided into a training set and a validation set. During training, the Yolov8-Seg model is trained using the training set to obtain the tomato target recognition model based on deep learning instance segmentation.

[0063] Step 3: Based on the tomato recognition results of each frame, calculate the maturity of all tomato fruits in that frame; then, combine the depth map and maturity of each frame to calculate the occlusion rate and weight of all tomato fruits in that frame; traverse all frames to obtain the occlusion rate and weight of different tomato fruits in all frames.

[0064] In one feasible implementation, based on the tomato recognition results of each frame, the ripeness of all tomato fruits in that frame is calculated, including:

[0065] The masks in the tomato recognition results are converted to HSV space. Then, the mode of the corresponding values ​​of the three channels of the mask for each tomato fruit in HSV space is determined and denoted as the mode H of the hue channel. mode Saturation channel mode S mode Mode V of the brightness channel mode Then, the ripeness of the tomato fruit is calculated based on the mode of the three channels, such as... Figure 5 As shown.

[0066] In one feasible implementation, an appropriate number of bunches of tomatoes covering various ripening stages are selected. A colorimeter is used to perform contact measurements on the bunches of tomatoes to collect the surface color of the fruit. This data is then used to calibrate the ripeness of each tomato fruit and denoted as MA. contact The maturity value ranges from [0, 100], where 0 represents immature and 100 represents fully mature. Using a camera, images of an appropriate number of tomato fruits under different lighting conditions and angles were captured on the tomato plants in a greenhouse setting. The mode H of the hue channel for each tomato fruit was determined. mode Saturation channel mode S mode and the mode V of the brightness channel mode H mode The value range of S is [0, 179]. mode The value range of V is [0, 255]. mode The value range is [0, 255]. The mode H is determined by the hue channels of several tomato fruits. mode Saturation channel mode S mode Mode of the brightness channel V mode The maturity level and its corresponding maturity level form a maturity fitting dataset. A fitting function is selected, and each fitting function is solved based on the maturity fitting dataset, and then each fitting function is evaluated.

[0067] In this embodiment, the types of fitting functions used are shown in Table 1. The R function in Table 1... 2 The coefficient of determination represents the fit; the closer it is to 1, the better the model fit. The p-value is used to test the significance of the model; the smaller the p-value, the more reliable the fit result.

[0068] Table 1. Correspondence between Fitting Function and Fit Quality

[0069]

[0070] As can be seen from the fitting quality in Table 1, this invention selects R... 2 The interaction term with the largest and smallest p-value was regressed to 1. Therefore, the tomato fruit maturity MA used in this invention... contact The calculation formula is as follows:

[0071] MA contact =θ0+θ1H mode +θ2S mode +θ3V mode +θ4(H mode ·S mode )+θ5(V mode ·S mode )+θ6(H mode ·V mode )

[0072] The data obtained from the actual experiment were fitted, and the specific values ​​of the six fitting parameters are as follows:

[0073] The first fitting coefficient θ0 = 0.4772, the second fitting coefficient θ1 = -0.0076, the third fitting coefficient θ2 = 0.0014, the fourth fitting coefficient θ3 = 0.0001, and the fifth fitting coefficient θ4 = -4.1 × 10⁻⁶. -5 The sixth fitting coefficient θ5 = 0, and the seventh fitting coefficient θ6 = -3.7 × 10⁻⁶. -5 .

[0074] Experiments have verified that, based on the maturity grading results of agronomic experts, the accuracy rate of tomato maturity classification calculated using this formula reaches 95%.

[0075] In one feasible implementation, by combining the depth map and maturity level of each frame, the occlusion rate and weight of all tomato fruits in that frame are calculated, including:

[0076] The weight of each tomato fruit is calculated as follows:

[0077] The color change of tomatoes is not uniform during the ripening process, and multiple colors often appear at the same time. In addition, there may be varying degrees of occlusion in the image, all of which affect the visual recognition of the ripeness of tomatoes.

[0078] First, based on the morphological characteristics of tomato fruits, this invention summarizes the complete state of a tomato fruit in an image as a circle, and uses the completeness of the fitted circle of the tomato fruit in the image to assess the degree of occlusion of the tomato fruit. Specifically, it fits an approximate circle to the recognition area of ​​the irregular polygon of each tomato fruit in the tomato recognition result, and obtains the radius of the fitted circle of that tomato fruit.

[0079] Next, the ratio of the identified region to the area of ​​the fitted circle is calculated based on the radius of the fitted circle and denoted as the occlusion rate (Ratio). This ratio is used to quantify the degree of occlusion of the current fruit. The closer the Ratio is to 1, the less occlusion there is, and the more completely visible the fruit is. The fitted circle effect and occlusion rate calculation results are illustrated below. Figure 4 As shown.

[0080] Then, the volume of the tomato fruit is calculated based on the camera intrinsic parameters, the depth map of the frame, and the radius of the fitted circle.

[0081] Finally, the weight of the tomato fruit is calculated based on its volume and ripeness.

[0082] In one feasible implementation, the Canny edge detection method is used to extract the edge contour of a binary mask for each tomato fruit, obtaining an edge contour map. Specifically, the Canny edge detection method uses two thresholds (low threshold 50, high threshold 150) to filter gradient magnitudes, thereby generating the edge contour map. Then, the Hough transform is used to perform minimum bounding circle detection on the edge contour map to obtain the radius of the fitted circle.

[0083] In one feasible implementation, Green's formula is used to calculate the area of ​​the recognition region of the irregular polygon.

[0084] In one feasible implementation, the volume of the tomato fruit is calculated based on camera intrinsic parameters, the depth map of the frame, and the radius of the fitted circle, including:

[0085] First, the depth map of the frame is filtered to obtain a filtered depth map. Based on the filtered depth map, the depth value of the center position of each tomato fruit is determined. Next, the radius of the fitted circle is converted into the true radius of the fruit using camera intrinsic parameters and then corrected to obtain the corrected radius. Finally, the volume of the tomato fruit is calculated based on the corrected radius.

[0086] In one feasible implementation, the true radius R of the fruit 2D The calculation formula is as follows:

[0087] R 2D =r·α·D center / f

[0088] Where r is the radius of the fitted circle, α is the pixel size of the camera, and D center The depth value at the center of each tomato fruit, where f is the camera focal length.

[0089] In one feasible implementation, assuming the tomato is a sphere, the three-dimensional radius is corrected using depth variations, as shown in the following formula:

[0090] R 3D =R 2D ·(1+(△D / D center ) 2 ) 0.5

[0091] Among them, R 3D ΔD represents the corrected radius; ΔD is the standard deviation of the tomato surface depth variation, which was obtained through experimental calculation.

[0092] The weight of the tomato fruit is calculated based on its volume and ripeness, including:

[0093] In the relationship between tomato maturity and density, experimental data usually show that the higher the maturity (the greater the MA value), the lower the tomato density (due to increased intercellular spaces and changes in water distribution).

[0094] The density value is dynamically adjusted based on maturity level (MA) using the following formula:

[0095] ρ adjusted =ρ based ·(1-β·MA / 100)

[0096] Where, ρ based ρ is the basic density of tomato fruit; β is the correction factor, obtained experimentally; adjusted Corrected density for tomato fruit;

[0097] Finally, the weight of the tomato fruit is calculated based on the volume and corrected density of the tomato fruit.

[0098] In one feasible implementation, when processing continuous video stream data, a target tracking algorithm based on Kalman Filter (KF) is used to continuously track individual tomato fruits in the video sequence and assign them unique numbers.

[0099] Step 4: Based on the inter-frame correspondence of the fruits, generate the future yield of tomatoes within the detection range according to the maturity, occlusion rate and weight of different tomato fruits in all frames.

[0100] In one feasible implementation, the future maturity and weight of a tomato fruit are generated based on the inter-frame correspondence of the fruit, according to the maturity, occlusion rate, and weight of the same tomato fruit in all frames. Specifically, this includes:

[0101] First, the maturity weight of the tomato fruit in each frame is calculated based on the occlusion rate of the tomato fruit. Then, the weighted maturity and weight of the tomato fruit in the frame are calculated based on the maturity weight. Next, the weighted maturity and weight of the tomato fruit in all frames are summed to obtain the weighted maturity sum and weight sum. Then, the weighted maturity sum and weight sum are divided by the maturity weight to obtain the final maturity and weight of the tomato fruit. Finally, the future maturity and weight of the tomato fruit are generated based on the final maturity and weight of the tomato fruit.

[0102] In one feasible implementation, the ripening process of tomato fruit can be simulated and described using a Logistic growth model, reflecting its initial rapid growth, gradual ripening, and subsequent stabilization of maturity. This invention uses the maturity rate coefficient and final maturity to experimentally calculate the percentage of maturity (%) of a single tomato fruit over the next d days, obtaining the following formula for calculating the future maturity of tomato fruit:

[0103] MA(d) = MA final +(100- MA final )·(1-e-λd )

[0104] Where MA(d) represents the maturity of the tomato fruit in the next d days; MA final λ represents the final maturity of the tomato fruit (0~100%); λ represents the ripening rate coefficient, which is obtained through experimental measurement.

[0105] The Logistic growth curve formula describes the weight of a single tomato over the next d days. It is known that once a tomato fruit enters the green ripening stage, its size growth becomes extremely slow. The formula for calculating the future weight of a tomato fruit is as follows:

[0106] Weight(d) = Weight final / (1+e -k(d-dm) )·(1+γ·(MA(d)-MA mg ) / 100)

[0107] dm=ln((100-MA final )-(MA mg -MA final )) / λ

[0108] Where Weight(d) represents the weight of the tomato fruit in the next d days; final MA represents the final weight of the tomato fruit. mg This represents the maturity value corresponding to the green-ripe stage of tomatoes, determined by the user based on the tomato variety. k is the weight gain rate before the green-ripe stage, determined experimentally. γ represents the growth coefficient after the green-ripe stage. dm is the time point corresponding to the green-ripe stage.

[0109] To facilitate harvesting, users can set a harvestable maturity value (MA). pick Harvesting is permitted when MA(d) exceeds this value, i.e., when MA(d) ≥ MA. pick The harvest status of tomato fruit i in the next d days P harvest,i(d) =1; otherwise P harvest,i(d) =0; therefore, estimate the total output Weight on day d. total (d):

[0110] Weight total (d)=∑ i P harvest,i(d) Weight i (d)

[0111] Among them, Weight i (d) represents the weight of tomato fruit i in the next d days.

[0112] This invention also proposes a tomato yield prediction system that considers maturity, the system comprising:

[0113] The data acquisition platform is used to collect RGB tomato videos and corresponding depth maps of tomato plants.

[0114] The target recognition unit is used to perform target recognition on each frame of the RGB tomato video and obtain the tomato recognition result corresponding to each frame of the RGB tomato video.

[0115] The target tracking unit is used to obtain the inter-frame correspondence of tomato fruits;

[0116] The maturity calculation unit is used to calculate the maturity of all tomato fruits in each frame based on the tomato recognition results of each frame.

[0117] The occlusion rate calculation unit is used to calculate the occlusion rate of all tomato fruits in each frame;

[0118] The weight calculation unit is used to calculate the weight of all tomato fruits in each frame;

[0119] The future tomato yield calculation unit is used to combine the inter-frame correspondence of fruits and generate the future yield of tomatoes within the detection range based on the maturity, occlusion rate and weight of different tomato fruits in all frames.

[0120] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a method for predicting tomato yield taking into account maturity.

[0121] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for predicting tomato yield taking into account maturity.

Claims

1. A method for predicting tomato yield considering maturity, characterized in that, Includes the following steps: Step 1: Collect RGB video of the tomato plant and the corresponding depth map; Step 2: Perform target recognition and target tracking on each frame of the RGB tomato video to obtain the tomato recognition results corresponding to each frame of the RGB tomato video and the inter-frame correspondence of the fruit; Step 3: Based on the tomato recognition results of each frame, calculate the maturity of all tomato fruits in that frame; Then, by combining the depth map and maturity of each frame, the occlusion rate and weight of all tomato fruits in that frame are calculated. Iterate through all frames to obtain the occlusion rate and weight of different tomato fruits in each frame. In step 3, based on the tomato recognition results of each frame, the maturity of all tomato fruits in that frame is calculated, including: The mask in the tomato recognition result is converted to HSV space. Then, the mode of the three channel values ​​of the mask corresponding to each tomato fruit in HSV space is determined. Then, the ripeness of the tomato fruit is calculated based on the mode of the three channels. Combining the depth map and maturity level of each frame, calculate the occlusion rate and weight of all tomato fruits in that frame, including: The weight of each tomato fruit is calculated as follows: First, an approximate circle is fitted to the recognition area of ​​each tomato fruit in the tomato recognition results to obtain the radius of the fitted circle for that tomato fruit. Next, the ratio of the area of ​​the recognition area to the area of ​​the fitted circle is calculated based on the fitted circle radius and recorded as the occlusion rate Ratio. Then, the volume of the tomato fruit is calculated based on the camera intrinsic parameters, the depth map of the frame, and the fitted circle radius. Finally, the weight of the tomato fruit is calculated based on its volume and maturity. Step 4: Based on the inter-frame correspondence of the fruits, generate the future yield of tomatoes within the detection range according to the maturity, occlusion rate and weight of different tomato fruits in all frames.

2. The method for predicting tomato yield considering maturity according to claim 1, characterized in that, In step 2, target recognition is performed on each frame of the RGB tomato image in the RGB tomato video, specifically including: A tomato target recognition model based on deep learning instance segmentation is used to perform target recognition on each frame of RGB tomato image, and the model outputs tomato recognition results; in the tomato recognition results, the same tomato fruit is marked in the same recognition box, and the recognition box is a polygon.

3. The method for predicting tomato yield considering maturity according to claim 1, characterized in that, The calculation of the tomato fruit volume based on camera intrinsic parameters, the depth map of the frame, and the radius of the fitted circle includes: First, the depth value of the center position of each tomato fruit is determined based on the depth map; then, the radius of the fitted circle is converted into the true radius of the fruit by using camera intrinsic parameters and then corrected to obtain the corrected radius; finally, the volume of the tomato fruit is calculated based on the corrected radius.

4. The method for predicting tomato yield considering maturity according to claim 1, characterized in that, The calculation of the tomato fruit's weight based on its volume and ripeness includes: The density value is dynamically adjusted based on maturity level (MA) using the following formula: r adjusted =p based ·(1-β·MA / 100) Where, ρ based ρ is the base density of the tomato fruit; β is the correction factor; adjusted Corrected density for tomato fruits; Then, calculate the weight of the tomato fruit based on its volume and corrected density.

5. The method for predicting tomato yield considering maturity according to claim 1, characterized in that, In step 4, combining the inter-frame correspondence of the fruit, the future maturity and weight of the tomato fruit are generated based on the maturity, occlusion rate, and weight of the same tomato fruit in all frames. Specifically, this includes: First, calculate the maturity weight of the tomato fruit in each frame based on its occlusion rate. Then, calculate the weighted maturity and weight of the tomato fruit in that frame based on the maturity weight. Next, sum the weighted maturity and weight of the tomato fruit in all frames to obtain the final weighted maturity and weight sum of the tomato fruit. Then, divide the final weighted maturity and weight sum of the tomato fruit by the maturity weight to obtain the final maturity and weight of the tomato fruit. Finally, generate the future maturity and weight of the tomato fruit based on its final maturity and weight.

6. A tomato yield prediction system that takes maturity into account, characterized in that, include: The data acquisition platform is used to collect RGB tomato videos and corresponding depth maps of tomato plants. The target recognition unit is used to perform target recognition on each frame of the RGB tomato video and obtain the tomato recognition result corresponding to each frame of the RGB tomato video. The target tracking unit is used to obtain the inter-frame correspondence of tomato fruits; The maturity calculation unit is used to calculate the maturity of all tomato fruits in each frame based on the tomato recognition results of each frame. In the maturity calculation unit, the mask in the tomato recognition result is converted to HSV space, then the mode of the three channel values ​​of the mask corresponding to each tomato fruit in HSV space is determined, and then the maturity of the tomato fruit is calculated based on the mode of the three channels. The occlusion rate calculation unit is used to calculate the occlusion rate of all tomato fruits in each frame; In the occlusion rate calculation unit, the weight of each tomato fruit is calculated as follows: First, an approximate circle is fitted to the recognition area of ​​each tomato fruit in the tomato recognition results to obtain the radius of the fitted circle for that tomato fruit. Next, the ratio of the area of ​​the recognition area to the area of ​​the fitted circle is calculated based on the fitted circle radius and recorded as the occlusion rate Ratio. Then, the volume of the tomato fruit is calculated based on the camera intrinsic parameters, the depth map of the frame, and the fitted circle radius. Finally, the weight of the tomato fruit is calculated based on its volume and maturity. The weight calculation unit is used to calculate the weight of all tomato fruits in each frame; The future tomato yield calculation unit is used to combine the inter-frame correspondence of fruits and generate the future yield of tomatoes in the detection area based on the maturity, occlusion rate and weight of different tomato fruits in all frames.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the tomato yield prediction method considering maturity as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the tomato yield prediction method considering maturity as described in any one of claims 1 to 5.

Images